Code for the RA-L (ICRA) 2021 paper "SeqNet: Learning Descriptors for Sequence-Based Hierarchical Place Recognition"

Last update: Dec 12, 2022

Overview

SeqNet: Learning Descriptors for Sequence-Based Hierarchical Place Recognition

[ArXiv+Supplementary] [IEEE Xplore RA-L 2021] [ICRA 2021 YouTube Video]

and

SeqNetVLAD vs PointNetVLAD: Image Sequence vs 3D Point Clouds for Day-Night Place Recognition

[ArXiv] [CVPR 2021 Workshop 3DVR]

Sequence-Based Hierarchical Visual Place Recognition.

News:

Jun 23: CVPR 2021 Workshop 3DVR paper, "SeqNetVLAD vs PointNetVLAD", now available on arXiv. Oxford dataset to be released soon.

Jun 02: SeqNet code release with the Nordland dataset.

Setup (One time)

Conda

conda create -n seqnet python=3.8 mamba -c conda-forge -y
conda activate seqnet
mamba install numpy pytorch=1.8.0 torchvision tqdm scikit-learn faiss tensorboardx h5py -c conda-forge -y

Download

Run bash download.sh to download single image NetVLAD descriptors (3.4 GB) for the Nordland-clean dataset ^[a] and corresponding model files (1.5 GB) ^[b].

Run

Train

To train sequential descriptors through SeqNet:

python main.py --mode train --pooling seqnet --dataset nordland-sw --seqL 10 --w 5 --outDims 4096 --expName "w5"

To (re-)train single descriptors through SeqNet:

python main.py --mode train --pooling seqnet --dataset nordland-sw --seqL 1 --w 1 --outDims 4096 --expName "w1"

Test

python main.py --mode test --pooling seqnet --dataset nordland-sf --seqL 5 --split test --resume ./data/runs/Jun03_15-22-44_l10_w5/

The above will reproduce results for SeqNet (S5) as per Supp. Table III on Page 10.

To obtain other results from the same table, expand this.

# Raw Single (NetVLAD) Descriptor
python main.py --mode test --pooling single --dataset nordland-sf --seqL 1 --split test

# SeqNet (S1)
python main.py --mode test --pooling seqnet --dataset nordland-sf --seqL 1 --split test --resume ./data/runs/Jun03_15-07-46_l1_w1/

# Raw + Smoothing
python main.py --mode test --pooling smooth --dataset nordland-sf --seqL 5 --split test

# Raw + Delta
python main.py --mode test --pooling delta --dataset nordland-sf --seqL 5 --split test

# Raw + SeqMatch
python main.py --mode test --pooling single+seqmatch --dataset nordland-sf --seqL 5 --split test

# SeqNet (S1) + SeqMatch
python main.py --mode test --pooling s1+seqmatch --dataset nordland-sf --seqL 5 --split test --resume ./data/runs/Jun03_15-07-46_l1_w1/

# HVPR (S5 to S1)
# Run S5 first and save its predictions by specifying `resultsPath`
python main.py --mode test --pooling seqnet --dataset nordland-sf --seqL 5 --split test --resume ./data/runs/Jun03_15-22-44_l10_w5/ --resultsPath ./data/results/
# Now run S1 + SeqMatch using results from above (the timestamp of `predictionsFile` would be different in your case)
python main.py --mode test --pooling s1+seqmatch --dataset nordland-sf --seqL 5 --split test --resume ./data/runs/Jun03_15-07-46_l1_w1/ --predictionsFile ./data/results/Jun03_16-07-36_l5_0.npz

Acknowledgement

The code in this repository is based on Nanne/pytorch-NetVlad. Thanks to Tobias Fischer for his contributions to this code during the development of our project QVPR/Patch-NetVLAD.

Citation

@article{garg2021seqnet,
  title={SeqNet: Learning Descriptors for Sequence-based Hierarchical Place Recognition},
  author={Garg, Sourav and Milford, Michael},
  journal={IEEE Robotics and Automation Letters},
  volume={6},
  number={3},
  pages={4305-4312},
  year={2021},
  publisher={IEEE},
  doi={10.1109/LRA.2021.3067633}
}

@misc{garg2021seqnetvlad,
  title={SeqNetVLAD vs PointNetVLAD: Image Sequence vs 3D Point Clouds for Day-Night Place Recognition},
  author={Garg, Sourav and Milford, Michael},
  howpublished={CVPR 2021 Workshop on 3D Vision and Robotics (3DVR)},
  month={Jun},
  year={2021},
}

Other Related Projects

Patch-NetVLAD (2021); Delta Descriptors (2020); CoarseHash (2020); seq2single (2019); LoST (2018)

[a] This is the clean version of the dataset that excludes images from the tunnels and red lights, exact image names can be obtained from here.

[b] These will automatically save to ./data/, you can modify this path in download.sh and get_datasets.py to specify your workdir.

Code for the RA-L (ICRA) 2021 paper "SeqNet: Learning Descriptors for Sequence-Based Hierarchical Place Recognition"

Related tags

Overview

SeqNet: Learning Descriptors for Sequence-Based Hierarchical Place Recognition

SeqNetVLAD vs PointNetVLAD: Image Sequence vs 3D Point Clouds for Day-Night Place Recognition

News:

Setup (One time)

Conda

Download

Run

Train

Test

Acknowledgement

Citation

Other Related Projects

Owner

Sourav Garg

Neural Turing Machines (NTM) - PyTorch Implementation

Deep learning image registration library for PyTorch

PyTorch evaluation code for Delving Deep into the Generalization of Vision Transformers under Distribution Shifts.

Does Pretraining for Summarization Reuqire Knowledge Transfer?

Automatic voice-synthetised summaries of latest research papers on arXiv

Awesome Graph Classification - A collection of important graph embedding, classification and representation learning papers with implementations.

Python implementation of Bayesian optimization over permutation spaces.

VLGrammar: Grounded Grammar Induction of Vision and Language

This project implements "virtual speed" from heart rate monito

YoloV5 implemented by TensorFlow2 , with support for training, evaluation and inference.

Tiny Kinetics-400 for test

Compute FID scores with PyTorch.

Solution of Kaggle competition: Sartorius - Cell Instance Segmentation

WSDM2022 "A Simple but Effective Bidirectional Extraction Framework for Relational Triple Extraction"

Official repository for Few-shot Image Generation via Cross-domain Correspondence (CVPR '21)

Pytorch implementation of "ARM: Any-Time Super-Resolution Method"

Implementation of CVPR'2022:Surface Reconstruction from Point Clouds by Learning Predictive Context Priors

A high-performance distributed deep learning system targeting large-scale and automated distributed training.

A Temporal Extension Library for PyTorch Geometric

A embed able annotation tool for end to end cross document co-reference