End-To-End Optimization of LiDAR Beam Configuration

Overview

End-To-End Optimization of LiDAR Beam Configuration

arXiv | IEEE Xplore

This repository is the official implementation of the paper:

End-To-End Optimization of LiDAR Beam Configuration for 3D Object Detection and Localization

Niclas Vödisch, Ozan Unal, Ke Li, Luc Van Gool, and Dengxin Dai.

To appear in RA-L.

Overview of 3D object detection

If you find our work useful, please consider citing our paper:

to be added after publication

📔 Abstract

Pre-determined beam configurations of low-resolution LiDARs are task-agnostic, hence simply using can result in non-optimal performance. In this work, we propose to optimize the beam distribution for a given target task via a reinforcement learning-based learning-to-optimize (RL-L2O) framework. We design our method in an end-to-end fashion leveraging the final performance of the task to guide the search process. Due to the simplicity of our approach, our work can be integrated with any LiDAR-based application as a simple drop-in module. In this repository, we provide the code for the exemplary task of 3D object detection.

🏗️ ️ Setup

To clone this repository and all submodules run:

git clone --recurse-submodules -j8 [email protected]:vniclas/lidar_beam_selection.git

⚙️ Installation

To install this code, please follow the steps below:

  1. Create a conda environment: conda create -n beam_selection python=3.8
  2. Activate the environment: conda activate beam_selection
  3. Install dependencies: pip install -r requirements.txt
  4. Install cudatoolkit (change to the used CUDA version):
    conda install cudnn cudatoolkit=10.2
  5. Install spconv (change to the used CUDA version):
    pip install spconv-cu102
  6. Install OpenPCDet (linked as submodule):
    cd third_party/OpenPCDet && python setup.py develop && cd ../..
  7. Install Pseudo-LiDAR++ (linked as submodule):
    pip install -r third_party/Pseudo_Lidar_V2/requirements.txt
    pip install pillow==8.3.2 (avoid runtime warnings)

💾 Data Preparation

  1. Download KITTI 3D Object Detection dataset and extract the files:
    1. Left color images image_2
    2. Right color images image_3
    3. Velodyne point clouds velodyne
    4. Camera calibration matrices calib
    5. Training labels label_2
  2. Predict the depth maps:
    1. Download pretrained model (training+validation)
    2. Generate the data:
    cd third_party/Pseudo_Lidar_V2  
    python ./src/main.py -c src/configs/sdn_kitti_train.config \
    --resume PATH_TO_CHECKPOINTS/sdn_kitti_object_trainval.pth --datapath PATH_TO_KITTI/training/ \
    --data_list ./split/trainval.txt --generate_depth_map --data_tag trainval \
    --save_path PATH_TO_DATA/sdn_kitti_train_set
    Note: Please adjust the paths PATH_TO_CHECKPOINTS, PATH_TO_KITTI, and PATH_TO_DATA to match your setup.
  3. Rename training/velodyne to training/velodyne_original
  4. Symlink the KITTI folders to PCDet:
    • ln -s PATH_TO_KITTI/training third_party/OpenPCDet/data/kitti/training
    • ln -s PATH_TO_KITTI/testing third_party/OpenPCDet/data/kitti/testing

🏃 Running 3D Object Detection

  1. Adjust paths in main.py. Further available parameters are listed in rl_l2o/eps_greedy_search.py and can be added in main.py.
  2. Adjust the number of epochs of the 3D object detector in (we used 40 epochs):
  3. Adjust the training scripts of the utilized detector to match your setup, e.g., object_detection/scripts/train_pointpillar.sh.
  4. Initiate the search: python main.py
    Note: Since we keep intermediate results to easily re-use them in later iterations, running the script will create a lot of data in the output_dir specified in main.py. You might want to manually delete some folders from time to time.

🔧 Adding more Tasks

Due to the design of the RL-L2O framework, it can be used as a simple drop-in module for many LiDAR applications. To apply the search algorithm to another task, just implement a custom RewardComputer, e.g., see object_detection/compute_reward.py. Additionally, you will have to prepare a set of features for each LiDAR beam. For the KITTI 3D Object Detection dataset, we provide the features as presented in the paper in object_detection/data/features_pcl.pkl.

👩‍⚖️ License

Creative Commons License
This software is made available for non-commercial use under a Creative Commons Attribution-NonCommercial 4.0 International License. A summary of the license can be found on the Creative Commons website.

Owner
Niclas
PhD student
Niclas
GANTheftAuto is a fork of the Nvidia's GameGAN

Description GANTheftAuto is a fork of the Nvidia's GameGAN, which is research focused on emulating dynamic game environments. The early research done

Harrison 801 Dec 27, 2022
A project to build an AI voice assistant using Python . The Voice assistant interacts with the humans to perform basic tasks.

AI_Personal_Voice_Assistant_Using_Python A project to build an AI voice assistant using Python . The Voice assistant interacts with the humans to perf

Chumui Tripura 1 Oct 30, 2021
This repository contains the code for designing risk bounded motion plans for car-like robot using Carla Simulator.

Nonlinear Risk Bounded Robot Motion Planning This code simulates the bicycle dynamics of car by steering it on the road by avoiding another static car

8 Sep 03, 2022
A simple PyTorch Implementation of Generative Adversarial Networks, focusing on anime face drawing.

AnimeGAN A simple PyTorch Implementation of Generative Adversarial Networks, focusing on anime face drawing. Randomly Generated Images The images are

Jie Lei 雷杰 1.2k Jan 03, 2023
Code for our paper 'Generalized Category Discovery'

Generalized Category Discovery This repo is a placeholder for code for our paper: Generalized Category Discovery Abstract: In this paper, we consider

107 Dec 28, 2022
Spherical Confidence Learning for Face Recognition, accepted to CVPR2021.

Sphere Confidence Face (SCF) This repository contains the PyTorch implementation of Sphere Confidence Face (SCF) proposed in the CVPR2021 paper: Shen

Maths 70 Dec 09, 2022
Pytorch implementation of VAEs for heterogeneous likelihoods.

Heterogeneous VAEs Beware: This repository is under construction 🛠️ Pytorch implementation of different VAE models to model heterogeneous data. Here,

Adrián Javaloy 35 Nov 29, 2022
A script that trains a model to recognize handwritten digits using the MNIST data set.

handwritten-digits-recognition A script that trains a model to recognize handwritten digits using the MNIST data set. Then it loads external files and

Hamza Sayih 1 Oct 30, 2021
A deep learning network built with TensorFlow and Keras to classify gender and estimate age.

Convolutional Neural Network (CNN). This repository contains a source code of a deep learning network built with TensorFlow and Keras to classify gend

Pawel Dziemiach 1 Dec 18, 2021
An inofficial PyTorch implementation of PREDATOR based on KPConv.

PREDATOR: Registration of 3D Point Clouds with Low Overlap An inofficial PyTorch implementation of PREDATOR based on KPConv. The code has been tested

ZhuLifa 14 Aug 03, 2022
Do Smart Glasses Dream of Sentimental Visions? Deep Emotionship Analysis for Eyewear Devices

EMOShip This repository contains the EMO-Film dataset described in the paper "Do Smart Glasses Dream of Sentimental Visions? Deep Emotionship Analysis

1 Nov 18, 2022
Visual Tracking by TridenAlign and Context Embedding

Visual Tracking by TridentAlign and Context Embedding (TACT) Test code for "Visual Tracking by TridentAlign and Context Embedding" Janghoon Choi, Juns

Janghoon Choi 32 Aug 25, 2021
Official PyTorch implementation of "Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble" (NeurIPS'21)

Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble This is the code for reproducing the results of the paper Uncertainty-Bas

43 Nov 23, 2022
Breast Cancer Detection 🔬 ITI "AI_Pro" Graduation Project

BreastCancerDetection - This program is designed to predict two severity of abnormalities associated with breast cancer cells: benign and malignant. Mammograms from MIAS is preprocessed and features

6 Nov 29, 2022
Notebooks for my "Deep Learning with TensorFlow 2 and Keras" course

Deep Learning with TensorFlow 2 and Keras – Notebooks This project accompanies my Deep Learning with TensorFlow 2 and Keras trainings. It contains the

Aurélien Geron 1.9k Dec 15, 2022
Official codebase for running the small, filtered-data GLIDE model from GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models.

GLIDE This is the official codebase for running the small, filtered-data GLIDE model from GLIDE: Towards Photorealistic Image Generation and Editing w

OpenAI 2.9k Jan 04, 2023
converts nominal survey data into a numerical value based on a dictionary lookup.

SWAP RATE Converts nominal survey data into a numerical values based on a dictionary lookup. It allows the user to switch nominal scale data from text

Jake Rhodes 1 Jan 18, 2022
YOLTv5 rapidly detects objects in arbitrarily large aerial or satellite images that far exceed the ~600×600 pixel size typically ingested by deep learning object detection frameworks

YOLTv5 rapidly detects objects in arbitrarily large aerial or satellite images that far exceed the ~600×600 pixel size typically ingested by deep learning object detection frameworks.

Adam Van Etten 145 Jan 01, 2023
This is project is the implementation of the DeepShift: Towards Multiplication-Less Neural Networks paper

DeepShift This is project is the implementation of the DeepShift: Towards Multiplication-Less Neural Networks paper, that aims to replace multiplicati

Mostafa Elhoushi 88 Dec 23, 2022
Train CNNs for the fruits360 data set in NTOU CS「Machine Vision」class.

CNNs fruits360 Train CNNs for the fruits360 data set in NTOU CS「Machine Vision」class. CNN on a pretrained model Build a CNN on a pretrained model, Res

Ricky Chuang 1 Mar 07, 2022