Official Implementation of Neural Splines

Overview

Neural Splines: Fitting 3D Surfaces with Inifinitely-Wide Neural Networks

Neural Splines Teaser This repository contains the official implementation of the CVPR 2021 (Oral) paper Neural Splines: Fitting 3D Surfaces with Infinitely-Wide Neural Networks.

Setup Instructions

System Requirements

Neural Splines uses FALKON, a state-of-the-art kernel ridge regression solver to fit surfaces on one or more GPUs. We thus require at least one GPU to run Neural Splines. We additionally require a working version of the CUDA compiler nvcc. We recommend running this code on a machine with a lot of memory if you want to reconstruct large point clouds since Neural Splines stores an MxM preconditioner matrix in CPU memory (where M is the number of Nystrom samples).

Installing Dependencies

Neural splines has several dependencies which must be installed before it can be used. Some of these dependencies must be built and take time to install. There are three ways to install dependencies:

Installing Dependencies with conda

Simply run

conda env create -f environment.yml

and then go grab a coffee . When you get back, you will have a conda environment called neural-splines with the right dependencies installed.

Installing Dependencies with pip

We include several requirement-*.txt files in the requirements directory depending on your version of cuda. Choose the right file for your installation then run

pip install -r requirements/requirements-cuda<VERSION>.txt

and then go grab a coffee .

Installing Dependencies Manually (Not Recommended)

You will need to install the following dependencies manually to use this repository:

You will also need to build the following dependencies from source. The easiest way to do this is with pip (see commands below), but you can also clone the linked repositories and run setup.py install:

Testing Your Installation

⚠️ WARNING ⚠️ Due to a bug in KeOps, the first time you use any code in this repository will throw a ModuleNotFoundError. All subsequent invocations of Neural Splines should work.

  1. Download and unzip the example point clouds here
  2. Unzip the file, in the directory of this repository, which should produce a directory named demo_data
  3. Run python fit.py demo_data/bunny.ply 10_000 128 On the first run this will fail (see above, just rerun it). On the second run it will compile some kernels and then produce a file called recon.ply which should be a reconstructed Stanford Bunny. The image below shows the input points and reconstruction for the bunny,
  4. Run python fit-grid.py demo_data/living_room_33_500_per_m2.ply 10_000 512 8 which will produce another recon.ply mesh, this time of a full room as shown below.

A reconstructed Stanford Bunny A reconstruced living room

Using Neural Splines from the Command Line

There are two scripts in this repository to fit surfaces from the command line:

  • fit.py fits an input point cloud using a single Neural Spline. This method is good for smaller inputs without too much geometric complexity.
  • fit_grid.py fits an input point cloud in chunks using a different Neural Spline per chunk. This method is better for very large scenes with a lot of geometric complexity.

Reconstructing a point cloud with fit.py

fit.py fits an input point cloud using a single Neural Spline. This approach works best for relatively small inputs which don't have too much geometric complexity. fit.py takes least the following arguments

fit.py <INPUT_POINT_CLOUD> <NUM_NYSTROM_SAMPLES> <GRID_SIZE>

where

  • <INPUT_POINT_CLOUD> is a path to a PLY file containing 3D points and corresponding normals
  • <EPS> is a spacing parameter used for finite difference approximation of the gradient term in the kernel. To capture all surface details this should be less than half the smallest distance between two points. Generally setting this to values smalelr than 0.5/grid_size is reasonable for this parameter
  • <NUM_NYSTROM_SAMPLES> is the number of points to use as basis centers. A larger number of Nystrom samples will yield a more accurate reconstruction but increase runtime and CPU memory usage. Generally good values for this are between 10*sqrt(N) and 100*sqrt(N) where N is the number of input points.
  • <GRID_SIZE> is the number of voxel cells along the longest axis of the bounding box on which the reconstructed function gets sampled. For example if <grid_size> is 128 and the bounding box of the input pointcloud has dimensions [1, 0.5, 0.5], then we will sample the function on a 128x64x64 voxel grid before extracting a mesh.

Reconstructing very large point clouds with fit-grid.py

fit-grid.py fits an input point cloud in chunks using a different Neural Spline per chunk. This approach works well when the input point cloud is large or has a lot of geometric complexity. fit-grid.py takes the following required arguments

fit-grid.py <INPUT_POINT_CLOUD> <NUM_NYSTROM_SAMPLES> <GRID_SIZE> <CELLS_PER_AXIS>

where

  • <INPUT_POINT_CLOUD> is a path to a PLY file containing 3D points and corresponding normals
  • <NUM_NYSTROM_SAMPLES> is the number of points to use as basis centers within each chunk. A larger number of Nystrom samples will yield a more accurate reconstruction but increase runtime and CPU memory usage.
  • <GRID_SIZE> is the number of voxel cells along the longest axis of the bounding box on which the reconstructed function gets sampled. For example if <GRID_SIZE> is 128 and the bounding box of the input pointcloud has dimensions [1, 0.5, 0.5], then we will sample the function on a 128x64x64 voxel grid before extracting a mesh.
  • <CELLS_PER_AXIS> is an integer specifying the number of chunks to use along each axis. E.g. if <cells-per-axis> is 8, we will reconstruct the surface using 8x8x8 chunks.

Furthermore, fit-grid.py accepts the following optional arguments:

  • --overlap <OVERLAP> optionally specify the fraction by which cells overlap. The default value is 0.25. If this value is too small, there may be artifacts in the output at the boundary of cells.
  • --weight-type <WEIGHT_TYPE> How to interpolate predictions in overlapping cells. Must be one of 'trilinear' or 'none', where 'trilinear' interpolates using a partition of unity defined using a bicubic spline and 'none' does not interpolate overlapping cells. Default is 'trilinear'.
  • --min-pts-per-cell <MIN_PTS_PER_CELL> Ignore cells with fewer points than this value. Default is 0.

Additional arguments to fit.py and fit-grid.py

Additionally, both fit.py and fit-grid.py accept the following optional arguments which can alter the behavior and performance of the fitting process:

  • --scale <SCALE>: Reconstruct the surface in a bounding box whose diameter is --scale times bigger than the diameter of the bounding box of the input points. Defaults is 1.1.
  • --regularization <REGULARIZATION>: Regularization penalty for kernel ridge regression. Default is 1e-10.
  • --nystrom-mode <NYSTROM_MODE>: How to generate nystrom samples. Default is 'blue-noise'. Must be one of
    • 'random': choose Nyström samples at random from the input
    • 'blue-noise': downsample the input with blue noise to get Nyström samples
    • 'k-means': use k-means clustering to generate Nyström samples
  • --trim <TRIM>: If set to a positive value, trim vertices of the reconstructed mesh whose nearest point in the input is greater than this value. The units of this argument are voxels (where the grid_size determines the size of a voxel) Default is -1.0.
  • --eps <EPS>: Perturbation amount for finite differencing in voxel units. i.e. we perturb points by eps times the diagonal length of a voxel (where the grid_size determines the size of a voxel). To approximate the gradient of the function, we sample points +/- eps along the normal direction.
  • --voxel-downsample-threshold <VOXEL_DOWNSAMPLE_THRESHOLD>: If the number of input points is greater than this value, downsample it by averaging points and normals within voxels on a grid. The size of the voxel grid is determined via the --grid-size argument. Default is 150_000.NOTE: This can massively speed up reconstruction for very large point clouds and generally won't throw away any details.
  • --kernel <KERNEL>: Which kernel to use. Must be one of 'neural-spline', 'spherical-laplace', or 'linear-angle'. Default is 'neural-spline'.NOTE: The spherical laplace is a good approximation to the neural tangent kernel (see this paper for details)
  • --seed <SEED>: Random number generator seed to use.
  • --out <OUT>: Path to file to save reconstructed mesh in.
  • --save-grid: If set, save the function evaluated on a voxel grid to {out}.grid.npy where out is the value of the --out argument.
  • --save-points: If set, save the tripled input points, their occupancies, and the Nyström samples to an npz file named {out}.pts.npz where out is the value of the --out argument.
  • --cg-max-iters <CG_MAX_ITERS>: Maximum number of conjugate gradient iterations. Default is 20.
  • --cg-stop-thresh <CG_STOP_THRESH>: Stop threshold for the conjugate gradient algorithm. Default is 1e-5.
  • --dtype DTYPE: Scalar type of the data. Must be one of 'float32' or 'float64'. Warning: float32 only works for very simple inputs.
  • --outer-layer-variance <OUTER_LAYER_VARIANCE>: Variance of the outer layer of the neural network from which the neural spline kernel arises from. Default is 0.001.
  • --verbose: If set, spam your terminal with debug information

Trimming Reconstructed Meshes

Neural Splines can sometimes add surface sheets far away from input points, to remove these, we include a surface trimming script (similar to Poisson Surface Reconstruction), which trims mesh faces away from the input points. To trim a surface, simply run:

python trim-surface.py <INPUT_POINT_CLOUD> <RECONSTRUCTED_MESH> <GRID_SIZE> <DISTANCE_THRESHOLD> --out <OUT_FILE>

where:

  • <INPUT_POINT_CLOUD> is a path to the input point cloud to the reconstruction algorithm
  • <RECONSTRUCTED_MESH> is a path to the mesh reconstructed by neural splines
  • <GRID_SIZE> is the size of the voxel grid used to reconstruct the mesh (the same value as the <GRID_SIZE> argument to fit.py or fit-grid.py)
  • <DISTANCE_THRESHOLD> is the distance (in voxels) above which faces should be discarded (e.g. passing 2.5 will discard any surface which is greater than 3 voxels away from an input point.
  • --out <OUT_FILE> is an optional path to save the trimmed mesh to. By default it is trimmed.ply.

Using Neural Splines in Python

Neural Splines can be used directly from within python by importing the neural_splines module in this repository.

To reconstruct a surface using Neural Splines, use the function neural_splines.fit_model_to_pointcloud. It returns a model object with the same API as Skikit-Learn. NOTE: neural_splines.fit_model_to_pointcloud can additionally accept other optional arguments. Run help(neural_splines.fit_model_to_pointcloud) for details.

from neural_splines import fit_model_to_point_cloud

# x is a point cloud stored in a torch tensor of shape [N, 3]
# n is a tensor of unit normals (one per point) of shape [N, 3]
# num_ny is the number of Nystrom samples to use 
# eps is the finite differencing coefficient (see documentation above)
model = fit_model_to_pointcloud(x, n, num_ny, eps)

# Evaluate the neural spline at a point p
p = torch.tensor([[0.5, 0.5, 0.5]]).to(x)
f_p = model.predict(p)

To evaluate a fitted Neural Spline on a grid of points, you can use the function neural_splines.eval_model_on_grid. NOTE: neural_splines.eval_model_on_grid can also accept other optional arguments, run help(neural_splines.eval_model_on_grid) for details.

from neural_splines import eval_model_on_grid

# Assume model is a Neural Spline fitted with fit_model_to_point_cloud

# Bounding box of the point cloud x represented as a tuple (origin, size)
bbox = x.min(0)[0], x.max(0)[0] - x.min(0)[0]
grid_res = torch.tensor([128, 128, 128]).to(torch.int32)

recon = eval_model_on_grid(model, bbox, voxel_grid_size)  # a [128, 128, 128] shaped tensor representing the neural spline evaluated on a grid.
Owner
Francis Williams
I am a PhD student in the Math and Data Group and the Geometric Computing Lab at NYU advised by Joan Bruna and Denis Zorin.
Francis Williams
Hierarchical Memory Matching Network for Video Object Segmentation (ICCV 2021)

Hierarchical Memory Matching Network for Video Object Segmentation Hongje Seong, Seoung Wug Oh, Joon-Young Lee, Seongwon Lee, Suhyeon Lee, Euntai Kim

Hongje Seong 72 Dec 14, 2022
A simple python library for fast image generation of people who do not exist.

Random Face A simple python library for fast image generation of people who do not exist. For more details, please refer to the [paper](https://arxiv.

Sergei Belousov 170 Dec 15, 2022
Rayvens makes it possible for data scientists to access hundreds of data services within Ray with little effort.

Rayvens augments Ray with events. With Rayvens, Ray applications can subscribe to event streams, process and produce events. Rayvens leverages Apache

CodeFlare 32 Dec 25, 2022
Multi-Person Extreme Motion Prediction

Multi-Person Extreme Motion Prediction Implementation for paper Wen Guo, Xiaoyu Bie, Xavier Alameda-Pineda, Francesc Moreno-Noguer, Multi-Person Extre

GUO-W 38 Nov 15, 2022
Full Resolution Residual Networks for Semantic Image Segmentation

Full-Resolution Residual Networks (FRRN) This repository contains code to train and qualitatively evaluate Full-Resolution Residual Networks (FRRNs) a

Toby Pohlen 274 Oct 27, 2022
MRQy is a quality assurance and checking tool for quantitative assessment of magnetic resonance imaging (MRI) data.

Front-end View Backend View Table of Contents Description Prerequisites Running Basic Information Measurements User Interface Feedback and usage Descr

Center for Computational Imaging and Personalized Diagnostics 58 Dec 02, 2022
LEAP: Learning Articulated Occupancy of People

LEAP: Learning Articulated Occupancy of People Paper | Video | Project Page This is the official implementation of the CVPR 2021 submission LEAP: Lear

Neural Bodies 60 Nov 18, 2022
This repository contains code for the paper "Disentangling Label Distribution for Long-tailed Visual Recognition", published at CVPR' 2021

Disentangling Label Distribution for Long-tailed Visual Recognition (CVPR 2021) Arxiv link Blog post This codebase is built on Causal Norm. Install co

Hyperconnect 85 Oct 18, 2022
Official Chainer implementation of GP-GAN: Towards Realistic High-Resolution Image Blending (ACMMM 2019, oral)

GP-GAN: Towards Realistic High-Resolution Image Blending (ACMMM 2019, oral) [Project] [Paper] [Demo] [Related Work: A2RL (for Auto Image Cropping)] [C

Wu Huikai 402 Dec 27, 2022
Code for ACL2021 paper Consistency Regularization for Cross-Lingual Fine-Tuning.

xTune Code for ACL2021 paper Consistency Regularization for Cross-Lingual Fine-Tuning. Environment DockerFile: dancingsoul/pytorch:xTune Install the f

Bo Zheng 42 Dec 09, 2022
Metric learning algorithms in Python

metric-learn: Metric Learning in Python metric-learn contains efficient Python implementations of several popular supervised and weakly-supervised met

1.3k Dec 28, 2022
This repository contains codes of ICCV2021 paper: SO-Pose: Exploiting Self-Occlusion for Direct 6D Pose Estimation

SO-Pose This repository contains codes of ICCV2021 paper: SO-Pose: Exploiting Self-Occlusion for Direct 6D Pose Estimation This paper is basically an

shangbuhuan 52 Nov 25, 2022
Automatically erase objects in the video, such as logo, text, etc.

Video-Auto-Wipe Read English Introduction:Here   本人不定期的基于生成技术制作一些好玩有趣的算法模型,这次带来的作品是“视频擦除”方向的应用模型,它实现的功能是自动感知到视频中我们不想看见的部分(譬如广告、水印、字幕、图标等等)然后进行擦除。由于图标擦

seeprettyface.com 141 Dec 26, 2022
Deep Ensembling with No Overhead for either Training or Testing: The All-Round Blessings of Dynamic Sparsity

[ICLR 2022] Deep Ensembling with No Overhead for either Training or Testing: The All-Round Blessings of Dynamic Sparsity by Shiwei Liu, Tianlong Chen, Zahra Atashgahi, Xiaohan Chen, Ghada Sokar, Elen

VITA 18 Dec 31, 2022
This repository provides the official code for GeNER (an automated dataset Generation framework for NER).

GeNER This repository provides the official code for GeNER (an automated dataset Generation framework for NER). Overview of GeNER GeNER allows you to

DMIS Laboratory - Korea University 50 Nov 30, 2022
DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generative Transformers

DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generative Transformers Authors: Jaemin Cho, Abhay Zala, and Mohit Bansal (

Jaemin Cho 98 Dec 15, 2022
Modeling Category-Selective Cortical Regions with Topographic Variational Autoencoders

Modeling Category-Selective Cortical Regions with Topographic Variational Autoencoders

1 Oct 11, 2021
Session-based Recommendation, CoHHN, price preferences, interest preferences, Heterogeneous Hypergraph, Co-guided Learning, SIGIR2022

This is our implementation for the paper: Price DOES Matter! Modeling Price and Interest Preferences in Session-based Recommendation Xiaokun Zhang, Bo

Xiaokun Zhang 27 Dec 02, 2022
Integrated Semantic and Phonetic Post-correction for Chinese Speech Recognition

Integrated Semantic and Phonetic Post-correction for Chinese Speech Recognition | paper | dataset | pretrained detection model | Authors: Yi-Chang Che

Yi-Chang Chen 1 Aug 23, 2022
State of the Art Neural Networks for Generative Deep Learning

pyradox-generative State of the Art Neural Networks for Generative Deep Learning Table of Contents pyradox-generative Table of Contents Installation U

Ritvik Rastogi 8 Sep 29, 2022