NPBG++: Accelerating Neural Point-Based Graphics

Related tags

Deep Learningnpbgpp
Overview

[CVPR 2022] NPBG++: Accelerating Neural Point-Based Graphics

Project Page | Paper

This repository contains the official Python implementation of the paper.

The repository also contains faithful implementation of NPBG.

We introduce the pipelines working with following datasets: ScanNet, NeRF-Synthetic, H3DS, DTU.

We follow the PyTorch3D convention for coordinate systems and cameras.

Changelog

  • [April 27, 2022] Added more example data and point clouds
  • [April 5, 2022] Initial code release

Dependencies

python -m venv ~/.venv/npbgplusplus
source ~/.venv/npbgplusplus/bin/activate
pip install -r requirements.txt

# install pytorch3d
curl -LO https://github.com/NVIDIA/cub/archive/1.10.0.tar.gz
tar xzf 1.10.0.tar.gz
export CUB_HOME=$PWD/cub-1.10.0
pip install "git+https://github.com/facebookresearch/[email protected]" --no-cache-dir --verbose

# install torch_scatter (2.0.8)
pip install torch-scatter -f https://pytorch-geometric.com/whl/torch-1.9.1+${CUDA}.html
# where ${CUDA} should be replaced by either cpu, cu101, cu102, or cu111 depending on your PyTorch installation.
# {CUDA} must match with torch.version.cuda (not with runtime or driver version)
# using 1.7.1 instead of 1.7.0 produces "incompatible cuda version" error

python setup.py build develop

Below you can see the examples on how to run the particular stages of different models on different datasets.

How to run NPBG++

Checkpoints and example data are available here.

Run training
python train_net.py trainer.gpus=4 hydra.run.dir=experiments/npbgpp_scannet datasets=scannet_pretrain datasets.n_point=6e6 system=npbgpp_sphere system.visibility_scale=0.5 trainer.max_epochs=39 dataloader.train_data_mode=each trainer.reload_dataloaders_every_n_epochs=1
python train_net.py trainer.gpus=4 hydra.run.dir=experiments/npbgpp_nerf datasets=nerf_blender_pretrain system=npbgpp_sphere system.visibility_scale=1.0 trainer.max_epochs=24 dataloader.train_data_mode=each weights_path=experiments/npbgpp_scannet/checkpoints/epoch38.ckpt
python train_net.py trainer.gpus=4 hydra.run.dir=experiments/npbgpp_h3ds datasets=h3ds_pretrain system=npbgpp_sphere system.visibility_scale=1.0 trainer.max_epochs=24 dataloader.train_data_mode=each trainer.reload_dataloaders_every_n_epochs=1 weights_path=experiments/npbgpp_scannet/checkpoints/epoch38.ckpt
python train_net.py trainer.gpus=4 hydra.run.dir=experiments/npbgpp_dtu datasets=dtu_pretrain system=npbgpp_sphere system.visibility_scale=1.0 trainer.max_epochs=36 dataloader.train_data_mode=each trainer.reload_dataloaders_every_n_epochs=1  weights_path=experiments/npbgpp_scannet/checkpoints/epoch38.ckpt
Run testing
python train_net.py trainer.gpus=1 hydra.run.dir=experiments/npbgpp_eval_scan118 datasets=dtu_one_scene datasets.data_root=$\{hydra:runtime.cwd\}/example/DTU_masked datasets.scene_name=scan118 system=npbgpp_sphere system.visibility_scale=1.0 weights_path=./checkpoints/npbgpp_dtu_nm_mvs_ft_epoch35.ckpt eval_only=true dataloader=small
Run finetuning of coefficients
python train_net.py trainer.gpus=1 hydra.run.dir=experiments/npbgpp_5ae021f2805c0854_ft datasets=h3ds_one_scene datasets.data_root=$\{hydra:runtime.cwd\}/example/H3DS datasets.selection_count=0 datasets.train_num_samples=2000 datasets.train_image_size=null datasets.train_random_shift=false datasets.train_random_zoom=[0.5,2.0] datasets.scene_name=5ae021f2805c0854 system=coefficients_ft system.max_points=1e6 system.descriptors_save_dir=$\{hydra:run.dir\}/descriptors trainer.max_epochs=20 system.descriptors_pretrained_dir=experiments/npbgpp_eval_5ae021f2805c0854/descriptors weights_path=$\{hydra:runtime.cwd\}/checkpoints/npbgpp_h3ds.ckpt dataloader=small
Run testing with finetuned coefficients
python train_net.py trainer.gpus=1 hydra.run.dir=experiments/npbgpp_5ae021f2805c0854_test datasets=h3ds_one_scene datasets.data_root=$\{hydra:runtime.cwd\}/example/H3DS datasets.selection_count=0 datasets.scene_name=5ae021f2805c0854 system=coefficients_ft system.max_points=1e6 system.descriptors_save_dir=$\{hydra:run.dir\}/descriptors system.descriptors_pretrained_dir=experiments/npbgpp_5ae021f2805c0854_ft/descriptors weights_path=experiments/npbgpp_5ae021f2805c0854_ft/checkpoints/last.ckpt dataloader=small eval_only=true

How to run NPBG

Run pretraining
python train_net.py trainer.gpus=4 hydra.run.dir=experiments/npbg_scannet datasets=scannet_pretrain datasets.train_random_zoom=[0.5,2.0] datasets.train_image_size=512 datasets.selection_count=0 system=npbg system.descriptors_save_dir=experiments/npbg_scannet/result/descriptors trainer.max_epochs=39 dataloader.train_data_mode=each trainer.reload_dataloaders_every_n_epochs=1 trainer.limit_val_batches=0 system.max_points=11e6
python train_net.py trainer.gpus=4 hydra.run.dir=experiments/npbg_nerf datasets=nerf_blender_pretrain datasets.train_random_zoom=[0.5,2.0] datasets.train_image_size=512 datasets.selection_count=0 system=npbg system.descriptors_save_dir=experiments/npbg_nerf/result/descriptors trainer.max_epochs=24 dataloader.train_data_mode=each trainer.reload_dataloaders_every_n_epochs=1 trainer.limit_val_batches=0 system.max_points=4e6
python train_net.py trainer.gpus=4 hydra.run.dir=experiments/npbg_h3ds datasets=h3ds_pretrain datasets.train_random_zoom=[0.5,2.0] datasets.train_image_size=null datasets.train_random_shift=false datasets.selection_count=0 system=npbg system.descriptors_save_dir=experiments/npbg_h3ds/result/descriptors trainer.max_epochs=24 dataloader.train_data_mode=each trainer.reload_dataloaders_every_n_epochs=1 trainer.limit_val_batches=0 system.max_points=3e6  # Submitted batch job 1175175
python train_net.py trainer.gpus=4 hydra.run.dir=experiments/npbg_dtu_nm datasets=dtu_pretrain datasets.train_random_zoom=[0.5,2.0] datasets.train_image_size=512 datasets.selection_count=0 system=npbg system.descriptors_save_dir=experiments/npbg_dtu_nm/result/descriptors trainer.max_epochs=36 dataloader.train_data_mode=each trainer.reload_dataloaders_every_n_epochs=1 trainer.limit_val_batches=0 system.max_points=3e6
Run fine-tuning on 1 scene
python train_net.py trainer.gpus=4 hydra.run.dir=experiments/npbg_scannet_0045 datasets=scannet_one_scene datasets.scene_name=scene0045_00 datasets.n_point=6e6 datasets.train_random_zoom=[0.5,2.0] datasets.train_image_size=512 datasets.selection_count=0 system=npbg system.descriptors_save_dir=experiments/npbg_scannet_0045/result/descriptors system.max_scenes_per_train_epoch=1 trainer.max_epochs=20 weights_path=experiments/npbg_scannet/result/checkpoints/epoch38.ckpt system.max_points=6e6
python train_net.py trainer.gpus=4 hydra.run.dir=experiments/npbg_nerf_hotdog datasets=nerf_blender_one_scene datasets.scene_name=hotdog datasets.train_random_zoom=[0.5,2.0] datasets.train_image_size=512 datasets.selection_count=0 system=npbg system.descriptors_save_dir=npbgplusplus/experiments/npbg_nerf_hotdog/result/descriptors system.max_scenes_per_train_epoch=1 trainer.max_epochs=20 weights_path=experiments/npbg_nerf/result/checkpoints/epoch23.ckpt system.max_points=4e6
python train_net.py trainer.gpus=4 hydra.run.dir=experiments/npbg_h3ds_5ae021f2805c0854 datasets=h3ds_one_scene datasets.scene_name=5ae021f2805c0854 datasets.train_random_zoom=[0.5,2.0] datasets.train_image_size=null datasets.train_random_shift=false datasets.selection_count=0 system=npbg system.descriptors_save_dir=experiments/npbg_h3ds_5ae021f2805c0854/result/descriptors system.max_scenes_per_train_epoch=1 trainer.max_epochs=20 weights_path=experiments/npbg_h3ds/result/checkpoints/epoch23.ckpt system.max_points=3e6
python train_net.py trainer.gpus=4 hydra.run.dir=experiments/npbg_dtu_nm_scan110 datasets=dtu_one_scene datasets.scene_name=scan110 datasets.train_random_zoom=[0.5,2.0] datasets.train_image_size=512 datasets.selection_count=0 system=npbg system.descriptors_save_dir=experiments/npbg_dtu_nm_scan110/result/descriptors system.max_scenes_per_train_epoch=1 trainer.max_epochs=20 weights_path=experiments/npbg_dtu_nm/result/checkpoints/epoch35.ckpt system.max_points=3e6

Citation

If you find our work useful in your research, please consider citing:

@article{rakhimov2022npbg++,
  title={NPBG++: Accelerating Neural Point-Based Graphics},
  author={Rakhimov, Ruslan and Ardelean, Andrei-Timotei and Lempitsky, Victor and Burnaev, Evgeny},
  journal={arXiv preprint arXiv:2203.13318},
  year={2022}
}

License

See the LICENSE for more details.

Owner
Ruslan Rakhimov
Ruslan Rakhimov
Official implementation of the ICCV 2021 paper "Conditional DETR for Fast Training Convergence".

The DETR approach applies the transformer encoder and decoder architecture to object detection and achieves promising performance. In this paper, we handle the critical issue, slow training convergen

281 Dec 30, 2022
Funnels: Exact maximum likelihood with dimensionality reduction.

Funnels This repository contains the code needed to reproduce the experiments from the paper: Funnels: Exact maximum likelihood with dimensionality re

2 Apr 21, 2022
Server files for UltimateLabeling

UltimateLabeling server files Server files for UltimateLabeling. git clone https://github.com/alexandre01/UltimateLabeling_server.git cd UltimateLabel

Alexandre Carlier 4 Oct 10, 2022
CT Based COVID 19 Diagnose by Image Processing and Deep Learning

This project proposed the deep learning and image processing method to undertake the diagnosis on 2D CT image and 3D CT volume.

1 Feb 08, 2022
Public implementation of "Learning from Suboptimal Demonstration via Self-Supervised Reward Regression" from CoRL'21

Self-Supervised Reward Regression (SSRR) Codebase for CoRL 2021 paper "Learning from Suboptimal Demonstration via Self-Supervised Reward Regression "

19 Dec 12, 2022
This is the official code of L2G, Unrolling and Recurrent Unrolling in Learning to Learn Graph Topologies.

Learning to Learn Graph Topologies This is the official code of L2G, Unrolling and Recurrent Unrolling in Learning to Learn Graph Topologies. Requirem

Stacy X PU 16 Dec 09, 2022
Learning Energy-Based Models by Diffusion Recovery Likelihood

Learning Energy-Based Models by Diffusion Recovery Likelihood Ruiqi Gao, Yang Song, Ben Poole, Ying Nian Wu, Diederik P. Kingma Paper: https://arxiv.o

Ruiqi Gao 41 Nov 22, 2022
Novel Instances Mining with Pseudo-Margin Evaluation for Few-Shot Object Detection

Novel Instances Mining with Pseudo-Margin Evaluation for Few-Shot Object Detection (NimPme) The official implementation of Novel Instances Mining with

12 Sep 08, 2022
A working implementation of the Categorical DQN (Distributional RL).

Categorical DQN. Implementation of the Categorical DQN as described in A distributional Perspective on Reinforcement Learning. Thanks to @tudor-berari

Florin Gogianu 98 Sep 20, 2022
EXplainable Artificial Intelligence (XAI)

EXplainable Artificial Intelligence (XAI) This repository includes the codes for different projects on eXplainable Artificial Intelligence (XAI) by th

4 Nov 28, 2022
RGB-stacking ๐Ÿ›‘ ๐ŸŸฉ ๐Ÿ”ท for robotic manipulation

RGB-stacking ๐Ÿ›‘ ๐ŸŸฉ ๐Ÿ”ท for robotic manipulation BLOG | PAPER | VIDEO Beyond Pick-and-Place: Tackling Robotic Stacking of Diverse Shapes, Alex X. Lee*,

DeepMind 95 Dec 23, 2022
Dyalog-apl-docset - Dyalog APL Dash Docset Generator

Dyalog APL Dash Docset Generator o alasa e kili sona kepeken tenpo lili a A Dash

Maciej Goszczycki 1 Jan 10, 2022
Contra is a lightweight, production ready Tensorflow alternative for solving time series prediction challenges with AI

Contra AI Engine A lightweight, production ready Tensorflow alternative developed by Styvio styvio.com ยป How to Use ยท Report Bug ยท Request Feature Tab

styvio 14 May 25, 2022
Implementation of Deep Deterministic Policy Gradiet Algorithm in Tensorflow

ddpg-aigym Deep Deterministic Policy Gradient Implementation of Deep Deterministic Policy Gradiet Algorithm (Lillicrap et al.arXiv:1509.02971.) in Ten

Steven Spielberg P 247 Dec 07, 2022
a pytorch implementation of auto-punctuation learned character by character

Learning Auto-Punctuation by Reading Engadget Articles Link to Other of my work ๐ŸŒŸ Deep Learning Notes: A collection of my notes going from basic mult

Ge Yang 137 Nov 09, 2022
Official pytorch implementation for Learning to Listen: Modeling Non-Deterministic Dyadic Facial Motion (CVPR 2022)

Learning to Listen: Modeling Non-Deterministic Dyadic Facial Motion This repository contains a pytorch implementation of "Learning to Listen: Modeling

50 Dec 17, 2022
OpenABC-D: A Large-Scale Dataset For Machine Learning Guided Integrated Circuit Synthesis

OpenABC-D: A Large-Scale Dataset For Machine Learning Guided Integrated Circuit Synthesis Overview OpenABC-D is a large-scale labeled dataset generate

NYU Machine-Learning guided Design Automation (MLDA) 31 Nov 22, 2022
PyTorch implementation of "Supervised Contrastive Learning" (and SimCLR incidentally)

PyTorch implementation of "Supervised Contrastive Learning" (and SimCLR incidentally)

Yonglong Tian 2.2k Jan 08, 2023
Job Assignment System by Real-time Emotion Detection

Emotion-Detection Job Assignment System by Real-time Emotion Detection Emotion is the essential role of facial expression and it could provide a lot o

1 Feb 08, 2022
CoMoGAN: continuous model-guided image-to-image translation. CVPR 2021 oral.

CoMoGAN: Continuous Model-guided Image-to-Image Translation Official repository. Paper CoMoGAN: continuous model-guided image-to-image translation [ar

166 Dec 31, 2022