Neural Geometric Level of Detail: Real-time Rendering with Implicit 3D Shapes (CVPR 2021 Oral)

Related tags

Deep Learningnglod
Overview

Neural Geometric Level of Detail: Real-time Rendering with Implicit 3D Surfaces

Official code release for NGLOD. For technical details, please refer to:

Neural Geometric Level of Detail: Real-time Rendering with Implicit 3D Surfaces
Towaki Takikawa*, Joey Litalien*, Kangxue Xin, Karsten Kreis, Charles Loop, Derek Nowrouzezahrai, Alec Jacobson, Morgan McGuire, and Sanja Fidler
In Computer Vision and Pattern Recognition (CVPR), 2021 (Oral)
[Paper] [Bibtex] [Project Page]

If you find this code useful, please consider citing:

@article{takikawa2021nglod,
    title = {Neural Geometric Level of Detail: Real-time Rendering with Implicit {3D} Shapes}, 
    author = {Towaki Takikawa and
              Joey Litalien and 
              Kangxue Yin and 
              Karsten Kreis and 
              Charles Loop and 
              Derek Nowrouzezahrai and 
              Alec Jacobson and 
              Morgan McGuire and 
              Sanja Fidler},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    year = {2021},
}

New: Sparse training code with Kaolin now available in app/spc! Read more about it here

Directory Structure

sol-renderer contains our real-time rendering code.

sdf-net contains our training code.

Within sdf-net:

sdf-net/lib contains all of our core codebase.

sdf-net/app contains standalone applications that users can run.

Getting started

Python dependencies

The easiest way to get started is to create a virtual Python 3.8 environment:

conda create -n nglod python=3.8
conda activate nglod
pip install --upgrade pip
pip install -r ./infra/requirements.txt

The code also relies on OpenEXR, which requires a system library:

sudo apt install libopenexr-dev 
pip install pyexr

To see the full list of dependencies, see the requirements.

Building CUDA extensions

To build the corresponding CUDA kernels, run:

cd sdf-net/lib/extensions
chmod +x build_ext.sh && ./build_ext.sh

The above instructions were tested on Ubuntu 18.04/20.04 with CUDA 10.2/11.1.

Training & Rendering

Note. All following commands should be ran within the sdf-net directory.

Download sample data

To download a cool armadillo:

wget https://raw.githubusercontent.com/alecjacobson/common-3d-test-models/master/data/armadillo.obj -P data/

To download a cool matcap file:

wget https://raw.githubusercontent.com/nidorx/matcaps/master/1024/6E8C48_B8CDA7_344018_A8BC94.png -O data/matcap/green.png

Training from scratch

python app/main.py \
    --net OctreeSDF \
    --num-lods 5 \
    --dataset-path data/armadillo.obj \
    --epoch 250 \
    --exp-name armadillo

This will populate _results with TensorBoard logs.

Rendering the trained model

If you set custom network parameters in training, you need to also reflect them for the renderer.

For example, if you set --feature-dim 16 above, you need to set it here too.

python app/sdf_renderer.py \
    --net OctreeSDF \
    --num-lods 5 \
    --pretrained _results/models/armadillo.pth \
    --render-res 1280 720 \
    --shading-mode matcap \
    --lod 4

By default, this will populate _results with the rendered image.

If you want to export a .npz model which can be loaded into the C++ real-time renderer, add the argument --export path/file.npz. Note that the renderer only supports the base Neural LOD configuration (the default parameters with OctreeSDF).

Core Library Development Guide

To add new functionality, you will likely want to make edits to the files in lib.

We try our best to keep our code modular, such that key components such as trainer.py and renderer.py need not be modified very frequently to add new functionalities.

To add a new network architecture for an example, you can simply add a new Python file in lib/models that inherits from a base class of choice. You will probably only need to implement the sdf method which implements the forward pass, but you have the option to override other methods as needed if more custom operations are needed.

By default, the loss function used are defined in a CLI argument, which the code will automatically parse and iterate through each loss function. The network architecture class is similarly defined in the CLI argument; simply use the exact class name, and don't forget to add a line in __init__.py to resolve the namespace.

App Development Guide

To make apps that use the core library, add the sdf-net directory into the Python sys.path, so the modules can be loaded correctly. Then, you will likely want to inherit the same CLI parser defined in lib/options.py to save time. You can then add a new argument group app to the parser to add custom CLI arguments to be used in conjunction with the defaults. See app/sdf_renderer.py for an example.

Examples of things that are considered apps include, but are not limited to:

  • visualizers
  • training code
  • downstream applications

Third-Party Libraries

This code includes code derived from 3 third-party libraries, all distributed under the MIT License:

https://github.com/zekunhao1995/DualSDF

https://github.com/rogersce/cnpy

https://github.com/krrish94/nerf-pytorch

Acknowledgements

We would like to thank Jean-Francois Lafleche, Peter Shirley, Kevin Xie, Jonathan Granskog, Alex Evans, and Alex Bie at NVIDIA for interesting discussions throughout the project. We also thank Peter Shirley, Alexander Majercik, Jacob Munkberg, David Luebke, Jonah Philion and Jun Gao for their help with paper editing.

We also thank Clement Fuji Tsang for his help with the code release.

The structure of this repo was inspired by PIFu: https://github.com/shunsukesaito/PIFu

pq is a jq-like Pickle file viewer

pq PQ is a jq-like viewer/processing tool for pickle files. howto # pq '' file.pkl {'other': 456, 'test': 123} # pq 'table' file.pkl |other|test| | 45

3 Mar 15, 2022
Fibonacci Method Gradient Descent

An implementation of the Fibonacci method for gradient descent, featuring a TKinter GUI for inputting the function / parameters to be examined and a matplotlib plot of the function and results.

Emma 1 Jan 28, 2022
Code for Max-Margin Contrastive Learning - AAAI 2022

Max-Margin Contrastive Learning This is a pytorch implementation for the paper Max-Margin Contrastive Learning accepted to AAAI 2022. This repository

Anshul Shah 12 Oct 22, 2022
DeepHyper: Scalable Asynchronous Neural Architecture and Hyperparameter Search for Deep Neural Networks

What is DeepHyper? DeepHyper is a software package that uses learning, optimization, and parallel computing to automate the design and development of

DeepHyper Team 214 Jan 08, 2023
[IJCAI'21] Deep Automatic Natural Image Matting

Deep Automatic Natural Image Matting [IJCAI-21] This is the official repository of the paper Deep Automatic Natural Image Matting. Introduction | Netw

Jizhizi_Li 316 Jan 06, 2023
Detection of PCBA defect

Detection_of_PCBA_defect Detection_of_PCBA_defect Use yolov5 to train. $pip install -r requirements.txt Detect.py will detect file(jpg,mp4...) in cu

6 Nov 28, 2022
ROCKET: Exceptionally fast and accurate time series classification using random convolutional kernels

ROCKET + MINIROCKET ROCKET: Exceptionally fast and accurate time series classification using random convolutional kernels. Data Mining and Knowledge D

298 Dec 26, 2022
AI Toolkit for Healthcare Imaging

Medical Open Network for AI MONAI is a PyTorch-based, open-source framework for deep learning in healthcare imaging, part of PyTorch Ecosystem. Its am

Project MONAI 3.7k Jan 07, 2023
Pytorch GUI(demo) for iVOS(interactive VOS) and GIS (Guided iVOS)

GUI for iVOS(interactive VOS) and GIS (Guided iVOS) GUI Implementation of CVPR2021 paper "Guided Interactive Video Object Segmentation Using Reliabili

Yuk Heo 13 Dec 09, 2022
StyleGAN2 Webtoon / Anime Style Toonify

StyleGAN2 Webtoon / Anime Style Toonify Korea Webtoon or Japanese Anime Character Stylegan2 base high Quality 1024x1024 / 512x512 Generate and Transfe

121 Dec 21, 2022
Just playing with getting CLIP Guided Diffusion running locally, rather than having to use colab.

CLIP-Guided-Diffusion Just playing with getting CLIP Guided Diffusion running locally, rather than having to use colab. Original colab notebooks by Ka

Nerdy Rodent 336 Dec 09, 2022
Code for ACM MM2021 paper "Complementary Trilateral Decoder for Fast and Accurate Salient Object Detection"

CTDNet The PyTorch code for ACM MM2021 paper "Complementary Trilateral Decoder for Fast and Accurate Salient Object Detection" Requirements Python 3.6

CVTEAM 28 Oct 20, 2022
Python wrapper to access the amazon selling partner API

PYTHON-AMAZON-SP-API Amazon Selling-Partner API If you have questions, please join on slack Contributions very welcome! Installation pip install pytho

Michael Primke 330 Jan 06, 2023
Code for "Neural Parts: Learning Expressive 3D Shape Abstractions with Invertible Neural Networks", CVPR 2021

Neural Parts: Learning Expressive 3D Shape Abstractions with Invertible Neural Networks This repository contains the code that accompanies our CVPR 20

Despoina Paschalidou 161 Dec 20, 2022
SOFT: Softmax-free Transformer with Linear Complexity, NeurIPS 2021 Spotlight

SOFT: Softmax-free Transformer with Linear Complexity SOFT: Softmax-free Transformer with Linear Complexity, Jiachen Lu, Jinghan Yao, Junge Zhang, Xia

Fudan Zhang Vision Group 272 Dec 25, 2022
Full Stack Deep Learning Labs

Full Stack Deep Learning Labs Welcome! Project developed during lab sessions of the Full Stack Deep Learning Bootcamp. We will build a handwriting rec

Full Stack Deep Learning 1.2k Dec 31, 2022
TensorFlow ROCm port

Documentation TensorFlow is an end-to-end open source platform for machine learning. It has a comprehensive, flexible ecosystem of tools, libraries, a

ROCm Software Platform 622 Jan 09, 2023
Easy to use Audio Tagging in PyTorch

Audio Classification, Tagging & Sound Event Detection in PyTorch Progress: Fine-tune on audio classification Fine-tune on audio tagging Fine-tune on s

sithu3 15 Dec 22, 2022
Official Pytorch implementation of "Beyond Static Features for Temporally Consistent 3D Human Pose and Shape from a Video", CVPR 2021

TCMR: Beyond Static Features for Temporally Consistent 3D Human Pose and Shape from a Video Qualtitative result Paper teaser video Introduction This r

Hongsuk Choi 215 Jan 06, 2023