Current state of supervised and unsupervised depth completion methods

Related tags

Deep Learningbenchmarkmachine-learningawesomecomputer-visionvoidsupervised-learningunsupervised-learning3d-reconstruction3d-visionkittiself-supervised-learningdepth-completion
Overview

Awesome Depth Completion

Table of Contents

About Sparse-to-Dense Depth Completion

In the sparse-to-dense depth completion problem, one wants to infer the dense depth map of a 3-D scene given an RGB image and its corresponding sparse reconstruction in the form of a sparse depth map obtained either from computational methods such as SfM (Strcuture-from-Motion) or active sensors such as lidar or structured light sensors.

Example 1: VOID dataset (indoor VIO)

Input RGB image Sparse point cloud Output point cloud from KBNet

Example 2: KITTI dataset (outdoor lidar)

Input RGB image Output point cloud from ScaffNet

Current State of Depth Completion Methods

Here we compile both unsupervised/self-supervised (monocular and stereo) and supervised methods published in recent conferences and journals on the VOID (Wong et. al., 2020) and KITTI (Uhrig et. al., 2017) depth completion benchmarks. Our ranking considers all four metrics rather than just RMSE.

Quick Links

Unsupervised VOID Depth Completion Benchmark

Paper Publication Code MAE RMSE iMAE iRMSE
Unsupervised Depth Completion with Calibrated Backprojection Layers ICCV 2021 PyTorch 39.80 95.86 21.16 49.72
Learning Topology from Synthetic Data for Unsupervised Depth Completion RA-L & ICRA 2021 Tensorflow 60.68 122.01 35.24 67.34
Unsupervised Depth Completion from Visual Inertial Odometry RA-L & ICRA 2020 Tensorflow 85.05 169.79 48.92 104.02
Dense depth posterior (ddp) from single image and sparse range CVPR 2019 Tensorflow 151.86 222.36 74.59 112.36
Self-supervised Sparse-to-Dense: Self- supervised Depth Completion from LiDAR and Monocular Camera ICRA 2019 PyTorch 178.85 243.84 80.12 107.69

Supervised VOID Depth Completion Benchmark

Paper Publication Code MAE RMSE iMAE iRMSE
Scanline Resolution-Invariant Depth Completion Using a Single Image and Sparse LiDAR Point Cloud RA-L & IROS 2021 N/A 59.40 181.42 19.37 46.56

Unsupervised KITTI Depth Completion Benchmark

Paper Publication Code MAE RMSE iMAE iRMSE
Unsupervised Depth Completion with Calibrated Backprojection Layers ICCV 2021 PyTorch 256.76 1069.47 1.02 2.95
Learning Topology from Synthetic Data for Unsupervised Depth Completion RA-L & ICRA 2021 Tensorflow 280.76 1121.93 1.15 3.30
Project to Adapt: Domain Adaptation for Depth Completion from Noisy and Sparse Sensor Data ACCV 2020 PyTorch 280.42 1095.26 1.19 3.53
Unsupervised Depth Completion from Visual Inertial Odometry RA-L & ICRA 2020 Tensorflow 299.41 1169.97 1.20 3.56
A Surface Geometry Model for LiDAR Depth Completion RA-L & ICRA 2021 Tensorflow 298.3 1239.84 1.21 3.76
Dense depth posterior (ddp) from single image and sparse range CVPR 2019 Tensorflow 343.46 1263.19 1.32 3.58
DFuseNet: Deep Fusion of RGB and Sparse Depth Information for Image Guided Dense Depth Completion ITSC 2019 PyTorch 429.93 1206.66 1.79 3.62
In Defense of Classical Image Processing: Fast Depth Completion on the CPU CRV 2018 Python 302.60 1288.46 1.29 3.78
Self-supervised Sparse-to-Dense: Self- supervised Depth Completion from LiDAR and Monocular Camera ICRA 2019 PyTorch 350.32 1299.85 1.57 4.07
Semantically Guided Depth Upsampling GCPR 2016 N/A 605.47 2312.57 2.05 7.38

Supervised KITTI Depth Completion Benchmark

Paper Publication Code MAE RMSE iMAE iRMSE
Non-Local Spatial Propagation Network for Depth Completion ECCV 2020 PyTorch 199.5 741.68 0.84 1.99
CSPN++: Learning Context and Resource Aware Convolutional Spatial Propagation Networks for Depth Completion AAAI 2020 N/A 209.28 743.69 0.90 2.07
Dense depth posterior (ddp) from single image and sparse range CVPR 2019 Tensorflow 203.96 832.94 0.85 2.10
Adaptive context-aware multi-modal network for depth completion TIP 2021 PyTorch 206.80 732.99 0.90 2.08
PENet: Towards Precise and Efficient Image Guided Depth Completion ICRA 2021 PyTorch 210.55 730.08 0.94 2.17
FCFR-Net: Feature Fusion based Coarse- to-Fine Residual Learning for Depth Completion AAAI 2021 N/A 217.15 735.81 0.98 2.20
Learning Guided Convolutional Network for Depth Completion TIP 2020 PyTorch 218.83 736.24 0.99 2.25
DenseLiDAR: A Real-Time Pseudo Dense Depth Guided Depth Completion Network ICRA 2021 N/A 214.13 755.41 0.96 2.25
A Multi-Scale Guided Cascade Hourglass Network for Depth Completion WACV 2020 PyTorch 220.41 762.19 0.98 2.30
Sparse and noisy LiDAR completion with RGB guidance and uncertainty MVA 2019 PyTorch 215.02 772.87 0.93 2.19
A Multi-Scale Guided Cascade Hourglass Network for Depth Completion WACV 2020 N/A 220.41 762.19 0.98 2.30
Learning Joint 2D-3D Representations for Depth Completion ICCV 2019 N/A 221.19 752.88 1.14 2.34
DeepLiDAR: Deep Surface Normal Guided Depth Prediction for Outdoor Scene From Sparse LiDAR Data and Single Color Image CVPR 2019 PyTorch 226.50 758.38 1.15 2.56
Depth Completion from Sparse LiDAR Data with Depth-Normal Constraints ICCV 2019 N/A 235.17 777.05 1.13 2.42
Scanline Resolution-Invariant Depth Completion Using a Single Image and Sparse LiDAR Point Cloud RA-L & IROS 2021 N/A 233.34 809.09 1.06 2.57
Confidence propagation through cnns for guided sparse depth regression PAMI 2019 PyTorch 233.26 829.98 1.03 2.60
Self-supervised Sparse-to-Dense: Self- supervised Depth Completion from LiDAR and Monocular Camera ICRA 2019 PyTorch 249.95 814.73 1.21 2.80
Uncertainty-Aware CNNs for Depth Completion: Uncertainty from Beginning to End CVPR 2020 PyTorch 251.77 960.05 1.05 3.37
Sparse and Dense Data with CNNs: Depth Completion and Semantic Segmentation 3DV 2019 N/A 234.81 917.64 0.95 2.17
Depth coefficients for depth completion CVPR 2019 N/A 252.21 988.38 1.13 2.87
Depth estimation via affinity learned with convolutional spatial propagation network ECCV 2018 N/A 279.46 1019.64 1.15 2.93
Learning morphological operators for depth completion ACIVS 2019 N/A 310.49 1045.45 1.57 3.84
Sparsity Invariant CNNs 3DV 2017 Tensorflow 416.14 1419.75 1.29 3.25
Deep Convolutional Compressed Sensing for LiDAR Depth Completion ACCV 2018 Tensorflow 439.48 1325.37 3.19 59.39
Owner
I am a post-doctoral researcher at the UCLA Vision Lab under the supervision of Professor Stefano Soatto.
GitHub Repository
Official code of ICCV2021 paper "Residual Attention: A Simple but Effective Method for Multi-Label Recognition"

CSRA This is the official code of ICCV 2021 paper: Residual Attention: A Simple But Effective Method for Multi-Label Recoginition Demo, Train and Vali

163 Dec 22, 2022
Toward Realistic Single-View 3D Object Reconstruction with Unsupervised Learning from Multiple Images (ICCV 2021)

Table of Content Introduction Getting Started Datasets Installation Experiments Training & Testing Pretrained models Texture fine-tuning Demo Toward R

VinAI Research 42 Dec 05, 2022
Face Synthetics dataset is a collection of diverse synthetic face images with ground truth labels.

The Face Synthetics dataset Face Synthetics dataset is a collection of diverse synthetic face images with ground truth labels. It was introduced in ou

Microsoft 608 Jan 02, 2023
Improving 3D Object Detection with Channel-wise Transformer

"Improving 3D Object Detection with Channel-wise Transformer" Thanks for the OpenPCDet, this implementation of the CT3D is mainly based on the pcdet v

Hualian Sheng 107 Dec 20, 2022
Modifications of the official PyTorch implementation of StyleGAN3. Let's easily generate images and videos with StyleGAN2/2-ADA/3!

Alias-Free Generative Adversarial Networks (StyleGAN3) Official PyTorch implementation of the NeurIPS 2021 paper Alias-Free Generative Adversarial Net

Diego Porres 185 Dec 24, 2022
The official pytorch implementation of our paper "Is Space-Time Attention All You Need for Video Understanding?"

TimeSformer This is an official pytorch implementation of Is Space-Time Attention All You Need for Video Understanding?. In this repository, we provid

Facebook Research 1k Dec 31, 2022
Resources for the Ki testnet challenge

Ki Testnet Challenge This repository hosts ki-testnet-challenge. A set of scripts and resources to be used for the Ki Testnet Challenge What is the te

Ki Foundation 23 Aug 08, 2022
Official NumPy Implementation of Deep Networks from the Principle of Rate Reduction (2021)

Deep Networks from the Principle of Rate Reduction This repository is the official NumPy implementation of the paper Deep Networks from the Principle

Ryan Chan 49 Dec 16, 2022
A Re-implementation of the paper "A Deep Learning Framework for Character Motion Synthesis and Editing"

What is This This is a simple re-implementation of the paper "A Deep Learning Framework for Character Motion Synthesis and Editing"(1). Only Sections

102 Dec 14, 2022
Code for our CVPR 2021 paper "MetaCam+DSCE"

Joint Noise-Tolerant Learning and Meta Camera Shift Adaptation for Unsupervised Person Re-Identification (CVPR'21) Introduction Code for our CVPR 2021

FlyingRoastDuck 59 Oct 31, 2022
Qimera: Data-free Quantization with Synthetic Boundary Supporting Samples

Qimera: Data-free Quantization with Synthetic Boundary Supporting Samples This repository is the official implementation of paper [Qimera: Data-free Q

Kanghyun Choi 21 Nov 03, 2022
Autoregressive Models in PyTorch.

Autoregressive This repository contains all the necessary PyTorch code, tailored to my presentation, to train and generate data from WaveNet-like auto

Christoph Heindl 41 Oct 09, 2022
Example Of Fine-Tuning BERT For Named-Entity Recognition Task And Preparing For Cloud Deployment Using Flask, React, And Docker

Example Of Fine-Tuning BERT For Named-Entity Recognition Task And Preparing For Cloud Deployment Using Flask, React, And Docker This repository contai

Nikita 12 Dec 14, 2022
Use CLIP to represent video for Retrieval Task

A Straightforward Framework For Video Retrieval Using CLIP This repository contains the basic code for feature extraction and replication of results.

Jesus Andres Portillo Quintero 54 Dec 22, 2022
An official TensorFlow implementation of “CLCC: Contrastive Learning for Color Constancy” accepted at CVPR 2021.

CLCC: Contrastive Learning for Color Constancy (CVPR 2021) Yi-Chen Lo*, Chia-Che Chang*, Hsuan-Chao Chiu, Yu-Hao Huang, Chia-Ping Chen, Yu-Lin Chang,

Yi-Chen (Howard) Lo 58 Dec 17, 2022
Official implementation of GraphMask as presented in our paper Interpreting Graph Neural Networks for NLP With Differentiable Edge Masking.

GraphMask This repository contains an implementation of GraphMask, the interpretability technique for graph neural networks presented in our ICLR 2021

Michael Schlichtkrull 29 Sep 02, 2022
Auxiliary Raw Net (ARawNet) is a ASVSpoof detection model taking both raw waveform and handcrafted features as inputs, to balance the trade-off between performance and model complexity.

Overview This repository is an implementation of the Auxiliary Raw Net (ARawNet), which is ASVSpoof detection system taking both raw waveform and hand

6 Jul 08, 2022
Anomaly detection in multi-agent trajectories: Code for training, evaluation and the OpenAI highway simulation.

Anomaly Detection in Multi-Agent Trajectories for Automated Driving This is the official project page including the paper, code, simulation, baseline

12 Dec 02, 2022
ChineseBERT: Chinese Pretraining Enhanced by Glyph and Pinyin Information

ChineseBERT: Chinese Pretraining Enhanced by Glyph and Pinyin Information This repository contains code, model, dataset for ChineseBERT at ACL2021. Ch

413 Dec 01, 2022
Official repository of the paper 'Essentials for Class Incremental Learning'

Essentials for Class Incremental Learning Official repository of the paper 'Essentials for Class Incremental Learning' This Pytorch repository contain

33 Nov 27, 2022