MS-SVConv : 3D Point Cloud Registration with Multi-Scale Architecture and Self-supervised Fine-tuning

Compute features for 3D point cloud registration. The article is available on Arxiv. It relies on:

A multi scale sparse voxel architecture
Self-supervised fine-tuning The combination of both allows better generalization capabilities and transfer across different datasets.

The code is available on the torch-points3d repository. This repository is to show how to launch the code for training and testing.

Demo

If you want to try MS-SVConv without installing anything on your computer, A Google colab notebook is available here (it takes few minutes to install everything). In the colab, we compute features using MS-SVConv and use Ransac (implementation of Open3D) to compute the transformation. You can try on 3DMatch on ETH. With this notebook, you can directly use the pretrained model on your project !

Installation

The code have been tried on an NVDIA RTX 1080 Ti with CUDA version 10.1. The OS was Ubuntu 18.04.

Installation for training and evaluation

This installation step is necessary if you want to train and evaluate MS-SVConv.

first you need, to clone the torch-points3d repository

git clone https://github.com/nicolas-chaulet/torch-points3d.git

Torch-points3d uses poetry to manage the packages. after installing Poetry, run :

poetry install --no-root

Activate the environnement

poetry shell

If you want to train MS-SVConv on 3DMatch, you will need pycuda (It's optional for testing).

pip install pycuda

You will also need to install Minkowski Engine and torchsparse Finally, you will need TEASER++ for testing.

If you have problems with installation (espaecially with pytorch_geometric), please visit the Troubleshooting section of torch-points3d page.

Training

registration

If you want to train MS-SVConv with 3 heads starting at the scale 2cm, run this command:

poetry run python train.py task=registration model_type=ms_svconv_base model_name=MS_SVCONV_B2cm_X2_3head dataset=fragment3dmatch training=sparse_fragment_reg tracker_options.make_submission=True training.epochs=200 eval_frequency=10

automatically, the code will call the right yaml file in conf/data/registration for the dataset and conf/model/registration for the model. If you just want to train MS-SVConv with 1 head, run this command

poetry run python train.py task=registration models=registration/ms_svconv_base model_name=MS_SVCONV_B2cm_X2_1head data=registration/fragment3dmatch training=sparse_fragment_reg tracker_options.make_submission=True epochs=200 eval_frequency=10

You can modify some hyperparameters directly on the command line. For example, if you want to change the learning rate of 1e-2, you can run:

poetry run python train.py task=registration models=registration/ms_svconv_base model_name=MS_SVCONV_B2cm_X2_1head data=registration/fragment3dmatch training=sparse_fragment_reg tracker_options.make_submission=True epochs=200 eval_frequency=10 optim.base_lr=1e-2

To resume training:

poetry run python train.py task=registration models=registration/ms_svconv_base model_name=MS_SVCONV_B2cm_X2_3head data=registration/fragment3dmatch training=sparse_fragment_reg tracker_options.make_submission=True epochs=200 eval_frequency=10 checkpoint_dir=/path/of/directory/containing/pretrained/model

WARNING : On 3DMatch, you will need a lot of disk space because the code will download the RGBD image on 3DMatch and build the fragments from scratch. Also the code takes time (few hours).

For 3DMatch, it was supervised training because the pose is necessary. But we can also fine-tune in a self-supervised fashion (without needing the pose).

To train on Modelnet run this command:

poetry run python train.py task=registration models=registration/ms_svconv_base model_name=MS_SVCONV_B2cm_X2_3head data=registration/modelnet_sparse_ss training=sparse_fragment_reg tracker_options.make_submission=True epochs=200 eval_frequency=10

To fine-tune on ETH run this command (First, download the pretrained model from 3DMatch here):

poetry run python train.py task=registration models=registration/ms_svconv_base model_name=MS_SVCONV_B4cm_X2_3head data=registration/eth_base training=sparse_fragment_reg_finetune tracker_options.make_submission=True epochs=200 eval_frequency=10 models.path_pretrained=/path/to/your/pretrained/model.pt

To fine-tune on TUM, run this command:

poetry run python train.py task=registration models=registration/ms_svconv_base model_name=MS_SVCONV_B4cm_X2_3head data=registration/testtum_ss training=sparse_fragment_reg_finetune tracker_options.make_submission=True epochs=200 eval_frequency=10 models.path_pretrained=/path/to/your/pretrained/model.pt

For all these command, it will save in outputs directory log of the training, it will save a .pt file which is the weights of

semantic segmentation

You can also train MS-SVConv on scannet for semantic segmentation. To do this simply run:

poetry run python train.py task=segmentation models=segmentation/ms_svconv_base model_name=MS_SVCONV_B4cm_X2_3head lr_scheduler.params.gamma=0.9922 data=segmentation/scannet-sparse training=minkowski_scannet tracker_options.make_submission=False tracker_options.full_res=False data.process_workers=1 wandb.log=True eval_frequency=10 batch_size=4

And you can easily transfer from registration to segmantation, with this command:

poetry run python train.py task=segmentation models=segmentation/ms_svconv_base model_name=MS_SVCONV_B4cm_X2_3head lr_scheduler.params.gamma=0.9922 data=segmentation/scannet-sparse training=minkowski_scannet tracker_options.make_submission=False tracker_options.full_res=False data.process_workers=1 wandb.log=True eval_frequency=10 batch_size=4 models.path_pretrained=/path/to/your/pretrained/model.pt

Evaluation

If you want to evaluate the models on 3DMatch, download the model here and run:

poetry run python scripts/test_registration_scripts/evaluate.py task=registration models=registration/ms_svconv_base model_name=MS_SVCONV_B2cm_X2_3head data=registration/fragment3dmatch training=sparse_fragment_reg cuda=True data.sym=True checkpoint_dir=/directory/of/the/models/

on ETH (model here),

poetry run python scripts/test_registration_scripts/evaluate.py task=registration models=registration/ms_svconv_base model_name=MS_SVCONV_B4cm_X2_3head data=registration/eth_base training=sparse_fragment_reg cuda=True data.sym=True checkpoint_dir=/directory/of/the/models/

on TUM (model here),

poetry run python scripts/test_registration_scripts/evaluate.py task=registration models=registration/ms_svconv_base model_name=MS_SVCONV_B2cm_X2_3head data=registration/testtum_ss training=sparse_fragment_reg cuda=True data.sym=True checkpoint_dir=/directory/of/the/models/

You can also visualize matches, you can run:

python scripts/test_registration_scripts/see_matches.py task=registration models=registration/ms_svconv_base model_name=MS_SVCONV_B4cm_X2_3head data=registration/eth_base training=sparse_fragment_reg cuda=True data.sym=True checkpoint_dir=/directory/of/the/models/ data.first_subsampling=0.04 +ind=548 +t=22

You should obtain this image

Model Zoo

You can find all the pretrained model (More will be added in the future)

citation

If you like our work, please cite it :

@inproceedings{horache2021mssvconv,
      title={3D Point Cloud Registration with Multi-Scale Architecture and Self-supervised Fine-tuning},
      author={Sofiane Horache and Jean-Emmanuel Deschaud and François Goulette},
      year={2021},
      journal={arXiv preprint arXiv:2103.14533}
}

And if you use ETH, 3DMatch, TUM or ModelNet as dataset, please cite the respective authors.

TODO

Add other pretrained models on the model zoo
Add others datasets such as KITTI Dataset

Compute descriptors for 3D point cloud registration using a multi scale sparse voxel architecture

Related tags

Overview

MS-SVConv : 3D Point Cloud Registration with Multi-Scale Architecture and Self-supervised Fine-tuning

Demo

Installation

Installation for training and evaluation

Training

registration

semantic segmentation

Evaluation

Model Zoo

citation

TODO

Owner

Pytorch Lightning Distributed Accelerators using Ray

PyTorch common framework to accelerate network implementation, training and validation

PyTorch implementation of some learning rate schedulers for deep learning researcher.

Bare bones use-case for deploying a containerized web app (built in streamlit) on AWS.

PASTRIE: A Corpus of Prepositions Annotated with Supersense Tags in Reddit International English

ICLR 2021 i-Mix: A Domain-Agnostic Strategy for Contrastive Representation Learning

pip install python-office

Sparse Progressive Distillation: Resolving Overfitting under Pretrain-and-Finetune Paradigm

OBG-FCN - implementation of 'Object Boundary Guided Semantic Segmentation'

Official repo for AutoInt: Automatic Integration for Fast Neural Volume Rendering in CVPR 2021

code for "AttentiveNAS Improving Neural Architecture Search via Attentive Sampling"

WORD: Revisiting Organs Segmentation in the Whole Abdominal Region

A Broader Picture of Random-walk Based Graph Embedding

(Preprint) Official PyTorch implementation of "How Do Vision Transformers Work?"

LeViT a Vision Transformer in ConvNet's Clothing for Faster Inference

This repository contains the code for the CVPR 2020 paper "Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision"

The codes and related files to reproduce the results for Image Similarity Challenge Track 2.

pytorch implementation of dftd2 & dftd3

Code for "ShineOn: Illuminating Design Choices for Practical Video-based Virtual Clothing Try-on", accepted at WACV 2021 Generation of Human Behavior Workshop.

code for paper"A High-precision Semantic Segmentation Method Combining Adversarial Learning and Attention Mechanism"