Depth-supervised NeRF: Fewer Views and Faster Training for Free

Project | Paper | YouTube

Pytorch implementation of our method for learning neural radiance fields that takes advantage of depth supervised by 3D point clouds. It can be used to train NeRF models given only very few input views.

Depth-supervised NeRF: Fewer Views and Faster Training for Free

arXiv 2107.02791, 2021

Kangle Deng¹, Andrew Liu², Jun-Yan Zhu¹, Deva Ramanan^1,3,

¹CMU, ²Google, ³Argo AI

We propose DS-NeRF (Depth-supervised Neural Radiance Fields), a model for learning neural radiance fields that takes advantage of depth supervised by 3D point clouds.

NeRF trained with 2 views:

DS-NeRF trained with 2 views:

Quick Start

Dependencies

Install requirements:

pip install -r requirements.txt

You will also need COLMAP installed to compute poses if you want to run on your data.

Data

Download data for the example scene: fern_2v

bash download_example_data.sh

To play with other scenes presented in the paper, download the data here.

Pre-trained Models

You can download the pre-trained models here. Place the downloaded directory in ./logs in order to test it later. See the following directory structure for an example:

├── logs 
│   ├── fern_2v    # downloaded logs
│   ├── flower_2v  # downloaded logs

How to Run?

Generate camera poses and sparse depth information using COLMAP (optional)

This step is necessary only when you want to run on your data.

First, place your scene directory somewhere. See the following directory structure for an example:

├── data
│   ├── fern_2v
│   ├── ├── images
│   ├── ├── ├── image001.png
│   ├── ├── ├── image002.png

To generate the poses and sparse point cloud:

python imgs2poses.py <your_scenedir>

Testing

Once you have the experiment directory (downloaded or trained on your own) in ./logs,

• to render a video:

python run_nerf.py --config configs/fern_dsnerf.txt --render_only

Training

To train a DS-NeRF on the example fern dataset:

python run_nerf.py --config configs/fern_dsnerf.txt

You can create your own experiment configuration to try other datasets.

Citation

If you find this repository useful for your research, please cite the following work.

@article{kangle2021dsnerf,
  title={Depth-supervised NeRF: Fewer Views and Faster Training for Free},
  author={Kangle Deng, Andrew Liu, Jun-Yan Zhu, and Deva Ramanan},
  journal={arXiv preprint arXiv:2107.02791},
  year={2021}
}

Credits

This code borrows heavily from nerf-pytorch.