MinkLoc++: Lidar and Monocular Image Fusion for Place Recognition

Overview

MinkLoc++: Lidar and Monocular Image Fusion for Place Recognition

Paper: MinkLoc++: Lidar and Monocular Image Fusion for Place Recognition accepted for International Joint Conference on Neural Networks (IJCNN) 2021 ArXiv

Jacek Komorowski, Monika Wysoczańska, Tomasz Trzciński

Warsaw University of Technology

Our other projects

  • MinkLoc3D: Point Cloud Based Large-Scale Place Recognition (WACV 2021): MinkLoc3D
  • Large-Scale Topological Radar Localization Using Learned Descriptors (ICONIP 2021): RadarLoc
  • EgonNN: Egocentric Neural Network for Point Cloud Based 6DoF Relocalization at the City Scale (IEEE Robotics and Automation Letters April 2022): EgoNN

Introduction

We present a discriminative multimodal descriptor based on a pair of sensor readings: a point cloud from a LiDAR and an image from an RGB camera. Our descriptor, named MinkLoc++, can be used for place recognition, re-localization and loop closure purposes in robotics or autonomous vehicles applications. We use late fusion approach, where each modality is processed separately and fused in the final part of the processing pipeline. The proposed method achieves state-of-the-art performance on standard place recognition benchmarks. We also identify dominating modality problem when training a multimodal descriptor. The problem manifests itself when the network focuses on a modality with a larger overfit to the training data. This drives the loss down during the training but leads to suboptimal performance on the evaluation set. In this work we describe how to detect and mitigate such risk when using a deep metric learning approach to train a multimodal neural network.

Overview

Citation

If you find this work useful, please consider citing:

@INPROCEEDINGS{9533373,  
   author={Komorowski, Jacek and Wysoczańska, Monika and Trzcinski, Tomasz},  
   booktitle={2021 International Joint Conference on Neural Networks (IJCNN)},   
   title={MinkLoc++: Lidar and Monocular Image Fusion for Place Recognition},   
   year={2021},  
   doi={10.1109/IJCNN52387.2021.9533373}
}

Environment and Dependencies

Code was tested using Python 3.8 with PyTorch 1.9.1 and MinkowskiEngine 0.5.4 on Ubuntu 20.04 with CUDA 10.2.

The following Python packages are required:

  • PyTorch (version 1.9.1)
  • MinkowskiEngine (version 0.5.4)
  • pytorch_metric_learning (version 1.0 or above)
  • tensorboard
  • colour_demosaicing

Modify the PYTHONPATH environment variable to include absolute path to the project root folder:

export PYTHONPATH=$PYTHONPATH:/home/.../MinkLocMultimodal

Datasets

MinkLoc++ is a multimodal descriptor based on a pair of inputs:

  • a 3D point cloud constructed by aggregating multiple 2D LiDAR scans from Oxford RobotCar dataset,
  • a corresponding RGB image from the stereo-center camera.

We use 3D point clouds built by authors of PointNetVLAD: Deep Point Cloud Based Retrieval for Large-Scale Place Recognition paper (link). Each point cloud is built by aggregating 2D LiDAR scans gathered during the 20 meter vehicle traversal. For details see PointNetVLAD paper or their github repository (link). You can download training and evaluation point clouds from here (alternative link).

After downloading the dataset, you need to edit config_baseline_multimodal.txt configuration file (in config folder). Set dataset_folder parameter to point to a root folder of PointNetVLAD dataset with 3D point clouds. image_path parameter must be a folder where downsampled RGB images from Oxford RobotCar dataset will be saved. The folder will be created by generate_rgb_for_lidar.py script.

Generate training and evaluation tuples

Run the below code to generate training pickles (with positive and negative point clouds for each anchor point cloud) and evaluation pickles. Training pickle format is optimized and different from the format used in PointNetVLAD code.

cd generating_queries/ 

# Generate training tuples for the Baseline Dataset
python generate_training_tuples_baseline.py --dataset_root 
   
    

# Generate training tuples for the Refined Dataset
python generate_training_tuples_refine.py --dataset_root 
    
     

# Generate evaluation tuples
python generate_test_sets.py --dataset_root 
     

     
    
   

is a path to dataset root folder, e.g. /data/pointnetvlad/benchmark_datasets/. Before running the code, ensure you have read/write rights to , as training and evaluation pickles are saved there.

Downsample RGB images and index RGB images linked with each point cloud

RGB images are taken directly from Oxford RobotCar dataset. First, you need to download stereo camera images from Oxford RobotCar dataset. See dataset website for details (link). After downloading Oxford RobotCar dataset, run generate_rgb_for_lidar.py script. The script finds 20 closest RGB images in RobotCar dataset for each 3D point cloud, downsamples them and saves them in the target directory (image_path parameter in config_baseline_multimodal.txt). During the training an input to the network consists of a 3D point cloud and one RGB image randomly chosen from these 20 corresponding images. During the evaluation, a network input consists of a 3D point cloud and one RGB image with the closest timestamp.

cd scripts/ 

# Generate training tuples for the Baseline Dataset
python generate_rgb_for_lidar.py --config ../config/config_baseline_multimodal.txt --oxford_root 
   

   

Training

MinkLoc++ can be used in unimodal scenario (3D point cloud input only) and multimodal scenario (3D point cloud + RGB image input). To train MinkLoc++ network, download and decompress the 3D point cloud dataset and generate training pickles as described above. To train the multimodal model (3D+RGB) download the original Oxford RobotCar dataset and extract RGB images corresponding to 3D point clouds as described above. Edit the configuration files:

  • config_baseline_multimodal.txt when training a multimodal (3D+RGB) model
  • config_baseline.txt and config_refined.txt when train unimodal (3D only) model

Set dataset_folder parameter to the dataset root folder, where 3D point clouds are located. Set image_path parameter to the path with RGB images corresponding to 3D point clouds, extracted from Oxford RobotCar dataset using generate_rgb_for_lidar.py script (only when training a multimodal model). Modify batch_size_limit parameter depending on the available GPU memory. Default limits requires 11GB of GPU RAM.

To train the multimodal model (3D+RGB), run:

cd training

python train.py --config ../config/config_baseline_multimodal.txt --model_config ../models/minklocmultimodal.txt

To train a unimodal model (3D only) model run:

cd training

# Train unimodal (3D only) model on the Baseline Dataset
python train.py --config ../config/config_baseline.txt --model_config ../models/minkloc3d.txt

# Train unimodal (3D only) model on the Refined Dataset
python train.py --config ../config/config_refined.txt --model_config ../models/minkloc3d.txt

Pre-trained Models

Pretrained models are available in weights directory

  • minkloc_multimodal.pth multimodal model (3D+RGB) trained on the Baseline Dataset with corresponding RGB images
  • minkloc3d_baseline.pth unimodal model (3D only) trained on the Baseline Dataset
  • minkloc3d_refined.pth unimodal model (3D only) trained on the Refined Dataset

Evaluation

To evaluate pretrained models run the following commands:

cd eval

# To evaluate the multimodal model (3D+RGB only) trained on the Baseline Dataset
python evaluate.py --config ../config/config_baseline_multimodal.txt --model_config ../models/minklocmultimodal.txt --weights ../weights/minklocmultimodal_baseline.pth

# To evaluate the unimodal model (3D only) trained on the Baseline Dataset
python evaluate.py --config ../config/config_baseline.txt --model_config ../models/minkloc3d.txt --weights ../weights/minkloc3d_baseline.pth

# To evaluate the unimodal model (3D only) trained on the Refined Dataset
python evaluate.py --config ../config/config_refined.txt --model_config ../models/minkloc3d.txt --weights ../weights/minkloc3d_refined.pth

Results

MinkLoc++ performance (measured by Average [email protected]%) compared to the state of the art:

Multimodal model (3D+RGB) trained on the Baseline Dataset extended with RGB images

Method Oxford ([email protected]) Oxford ([email protected]%)
CORAL [1] 88.9 96.1
PIC-Net [2] 98.2
MinkLoc++ (3D+RGB) 96.7 99.1

Unimodal model (3D only) trained on the Baseline Dataset

Method Oxford ([email protected]%) U.S. ([email protected]%) R.A. ([email protected]%) B.D ([email protected]%)
PointNetVLAD [3] 80.3 72.6 60.3 65.3
PCAN [4] 83.8 79.1 71.2 66.8
DAGC [5] 87.5 83.5 75.7 71.2
LPD-Net [6] 94.9 96.0 90.5 89.1
EPC-Net [7] 94.7 96.5 88.6 84.9
SOE-Net [8] 96.4 93.2 91.5 88.5
NDT-Transformer [10] 97.7
MinkLoc3D [9] 97.9 95.0 91.2 88.5
MinkLoc++ (3D-only) 98.2 94.5 92.1 88.4

Unimodal model (3D only) trained on the Refined Dataset

Method Oxford ([email protected]%) U.S. ([email protected]%) R.A. ([email protected]%) B.D ([email protected]%)
PointNetVLAD [3] 80.1 94.5 93.1 86.5
PCAN [4] 86.4 94.1 92.3 87.0
DAGC [5] 87.8 94.3 93.4 88.5
LPD-Net [6] 94.9 98.9 96.4 94.4
SOE-Net [8] 96.4 97.7 95.9 92.6
MinkLoc3D [9] 98.5 99.7 99.3 96.7
MinkLoc++ (RGB-only) 98.4 99.7 99.3 97.4
  1. Y. Pan et al., "CORAL: Colored structural representation for bi-modal place recognition", preprint arXiv:2011.10934 (2020)
  2. Y. Lu et al., "PIC-Net: Point Cloud and Image Collaboration Network for Large-Scale Place Recognition", preprint arXiv:2008.00658 (2020)
  3. M. A. Uy and G. H. Lee, "PointNetVLAD: Deep Point Cloud Based Retrieval for Large-Scale Place Recognition", 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
  4. W. Zhang and C. Xiao, "PCAN: 3D Attention Map Learning Using Contextual Information for Point Cloud Based Retrieval", 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
  5. Q. Sun et al., "DAGC: Employing Dual Attention and Graph Convolution for Point Cloud based Place Recognition", Proceedings of the 2020 International Conference on Multimedia Retrieval
  6. Z. Liu et al., "LPD-Net: 3D Point Cloud Learning for Large-Scale Place Recognition and Environment Analysis", 2019 IEEE/CVF International Conference on Computer Vision (ICCV)
  7. L. Hui et al., "Efficient 3D Point Cloud Feature Learning for Large-Scale Place Recognition" preprint arXiv:2101.02374 (2021)
  8. Y. Xia et al., "SOE-Net: A Self-Attention and Orientation Encoding Network for Point Cloud based Place Recognition", 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
  9. J. Komorowski, "MinkLoc3D: Point Cloud Based Large-Scale Place Recognition", Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), (2021)
  10. Z. Zhou et al., "NDT-Transformer: Large-scale 3D Point Cloud Localisation Using the Normal Distribution Transform Representation", 2021 IEEE International Conference on Robotics and Automation (ICRA)
  • J. Komorowski, M. Wysoczanska, T. Trzcinski, "MinkLoc++: Lidar and Monocular Image Fusion for Place Recognition", accepted for International Joint Conference on Neural Networks (IJCNN), (2021)

License

Our code is released under the MIT License (see LICENSE file for details).

Based on the paper "Geometry-aware Instance-reweighted Adversarial Training" ICLR 2021 oral

Geometry-aware Instance-reweighted Adversarial Training This repository provides codes for Geometry-aware Instance-reweighted Adversarial Training (ht

Jingfeng 47 Dec 22, 2022
Reproduced Code for Image Forgery Detection papers.

Image Forgery Detection With over 4.5 billion active internet users, the amount of multimedia content being shared every day has surpassed everyone’s

Umar Masud 15 Dec 06, 2022
The versatile ocean simulator, in pure Python, powered by JAX.

Veros is the versatile ocean simulator -- it aims to be a powerful tool that makes high-performance ocean modeling approachable and fun. Because Veros

TeamOcean 245 Dec 20, 2022
CSAW-M: An Ordinal Classification Dataset for Benchmarking Mammographic Masking of Cancer

CSAW-M This repository contains code for CSAW-M: An Ordinal Classification Dataset for Benchmarking Mammographic Masking of Cancer. Source code for tr

Yue Liu 7 Oct 11, 2022
[AAAI22] Reliable Propagation-Correction Modulation for Video Object Segmentation

Reliable Propagation-Correction Modulation for Video Object Segmentation (AAAI22) Preview version paper of this work is available at: https://arxiv.or

Xiaohao Xu 70 Dec 04, 2022
RTS3D: Real-time Stereo 3D Detection from 4D Feature-Consistency Embedding Space for Autonomous Driving

RTS3D: Real-time Stereo 3D Detection from 4D Feature-Consistency Embedding Space for Autonomous Driving (AAAI2021). RTS3D is efficiency and accuracy s

71 Nov 29, 2022
The Power of Scale for Parameter-Efficient Prompt Tuning

The Power of Scale for Parameter-Efficient Prompt Tuning Implementation of soft embeddings from https://arxiv.org/abs/2104.08691v1 using Pytorch and H

Kip Parker 208 Dec 30, 2022
rliable is an open-source Python library for reliable evaluation, even with a handful of runs, on reinforcement learning and machine learnings benchmarks.

Open-source library for reliable evaluation on reinforcement learning and machine learning benchmarks. See NeurIPS 2021 oral for details.

Google Research 529 Jan 01, 2023
Easy way to add GoogleMaps to Flask applications. maintainer: @getcake

Flask Google Maps Easy to use Google Maps in your Flask application requires Jinja Flask A google api key get here Contribute To contribute with the p

Flask Extensions 611 Dec 05, 2022
PyTorch Implementation of NCSOFT's FastPitchFormant: Source-filter based Decomposed Modeling for Speech Synthesis

FastPitchFormant - PyTorch Implementation PyTorch Implementation of FastPitchFormant: Source-filter based Decomposed Modeling for Speech Synthesis. Qu

Keon Lee 63 Jan 02, 2023
An educational AI robot based on NVIDIA Jetson Nano.

JetBot Looking for a quick way to get started with JetBot? Many third party kits are now available! JetBot is an open-source robot based on NVIDIA Jet

NVIDIA AI IOT 2.6k Dec 29, 2022
A semismooth Newton method for elliptic PDE-constrained optimization

sNewton4PDEOpt The Python module implements a semismooth Newton method for solving finite-element discretizations of the strongly convex, linear ellip

2 Dec 08, 2022
constructing maps of intellectual influence from publication data

Influencemap Project @ ANU Influence in the academic communities has been an area of interest for researchers. This can be seen in the popularity of a

CS Metrics 13 Jun 18, 2022
Repositório criado para abrigar os notebooks com a listas de exercícios propostos pelo professor Gustavo Guanabara do canal Curso em Vídeo do YouTube durante o Curso de Python 3

Curso em Vídeo - Exercícios de Python 3 Sobre o repositório Este repositório contém os notebooks com a listas de exercícios propostos pelo professor G

João Pedro Pereira 9 Oct 15, 2022
This repository contains source code for the Situated Interactive Language Grounding (SILG) benchmark

SILG This repository contains source code for the Situated Interactive Language Grounding (SILG) benchmark. If you find this work helpful, please cons

Victor Zhong 17 Nov 27, 2022
PyTorch Implementation of Realtime Multi-Person Pose Estimation project.

PyTorch Realtime Multi-Person Pose Estimation This is a pytorch version of Realtime_Multi-Person_Pose_Estimation, origin code is here Realtime_Multi-P

Dave Fang 157 Nov 12, 2022
Implementation of Google Brain's WaveGrad high-fidelity vocoder

WaveGrad Implementation (PyTorch) of Google Brain's high-fidelity WaveGrad vocoder (paper). First implementation on GitHub with high-quality generatio

Ivan Vovk 363 Dec 27, 2022
This MVP data web app uses the Streamlit framework and Facebook's Prophet forecasting package to generate a dynamic forecast from your own data.

📈 Automated Time Series Forecasting Background: This MVP data web app uses the Streamlit framework and Facebook's Prophet forecasting package to gene

Zach Renwick 42 Jan 04, 2023
Source code for "Taming Visually Guided Sound Generation" (Oral at the BMVC 2021)

Taming Visually Guided Sound Generation • [Project Page] • [ArXiv] • [Poster] • • Listen for the samples on our project page. Overview We propose to t

Vladimir Iashin 226 Jan 03, 2023
Easy-to-use,Modular and Extendible package of deep-learning based CTR models .

DeepCTR DeepCTR is a Easy-to-use,Modular and Extendible package of deep-learning based CTR models along with lots of core components layers which can

浅梦 6.6k Jan 08, 2023