Rendering Point Clouds with Compute Shaders

Overview

Compute Shader Based Point Cloud Rendering

This repository contains the source code to our techreport:
Rendering Point Clouds with Compute Shaders and Vertex Order Optimization
Markus Schütz, Bernhard Kerbl, Michael Wimmer. (not peer-reviewed, currently in submission)

  • Compute shaders can render point clouds up to an order of magnitude faster than GL_POINTS.
  • With a combination of warp-wide deduplication and early-z, compute shaders able to render 796 million points (12.7GB) at stable 62 to 64 frames per second in various different viewpoints on an RTX 3090. This corresponds to a memory bandwidth utilization of about 802GB/s, or a throughput of about 50 billion points per second.
  • The vertex order also strongly affects the performance. Some locality of points that are consecutive in memory is beneficial, but excessive locality can result in drastic slowdowns if it leads to thousands of GPU threads attempting to update a single pixel. As such, neither Morton ordered nor shuffled buffers are optimal. However combining both by first sorting by Morton code, and then shuffling batches of 128 points but leaving points within a batch in order, results in an improved ordering that ensures high performance with our compute approaches, and it also increases the performance of GL_POINTS by up to 5 times.

About the Framework

This framework is written in C++ and JavaScript (using V8). Most of the rendering is done in JavaScript with bindings to OpenGL 4.5 functions. It is written with live-coding in mind, so many javascript files are immediately executed at runtime as soon as they are saved by any text editor. As such, code has to be written with repeated execution in mind.

Getting Started

  • Compile Skye.sln project with Visual Studio.
  • Open the workspace in vscode.
  • Open "load_pointcloud.js" (quick search files via ctrl + e).
    • Adapt the path to the correct location of the las file.
    • Adapt position and lookAt to a viewpoint that fits your point cloud.
    • Change window.x to something that fits your monitor setup, e.g., 0 if you've got a single monitor, or 2540 if you've got two monitors and your first one has a with of 2540 pixels.
  • Press "Ctrl + Shift + B" to start the app. You should be seing an empty green window. (window.x is not yet applied)
  • Once you save "load_pointcloud.js" via ctrl+s, it will be executed, the window will be repositioned, and the point cloud will be loaded.
  • You can change position and lookAt at runtime and apply them by simply saving load_pointcloud.js again. The pointcloud will not be loaded again - to do so, you'll need to restart first.

After loading the point cloud, you should be seeing something like the screenshot below. The framework includes an IMGUI window with frame times, and another window that lets you switch between various rendering methods. Best try with data sets with tens of millions or hundreds of millions of points!

sd

Code Sections

Code for the individual rendering methods is primarily found in the modules/compute_<methods> folders.

Method Location
atomicMin ./modules/compute
reduce ./modules/compute_ballot
early-z ./modules/compute_earlyDepth
reduce & early-z ./modules/compute_ballot_earlyDepth
dedup ./modules/compute_ballot_earlyDepth_dedup
HQS ./modules/compute_hqs
HQS1R ./modules/compute_hqs_1x64bit_fast
busy-loop ./modules/compute_guenther
just-set ./modules/compute_just_set

Results

Frame times when rendering 796 million points on an RTX 3090 in a close-up viewpoint. Times in milliseconds, lower is better. The compute methods reduce (with early-z) and dedup (with early-z) yield the best results with Morton order (<16.6ms, >60fps). The shuffled Morton order greatly improves performance of GL_POINTS and some compute methods, and it is usually either the fastest or within close margins of the fastest combinations of rendering method and ordering.

Not depicted is that the dedup method is the most stable approach that continuously maintains >60fps in all viewpoints, while the performance of the reduce method varies and may drop to 50fps in some viewpoints. As such, we would recomend to use dedup in conjunction with Morton order if the necessary compute features are available, and reduce (with early-z) for wider support.

Comments
  • Can provide some dataset to test the demo (like default Candi Banyunibo data set)?

    Can provide some dataset to test the demo (like default Candi Banyunibo data set)?

    Hi, just found this paper & project via graphics weekly news.. just compiled the demo, and seems uses that dataset by default (banyunibo_inside_morton).. anyway to obtain that dataset? if not can you provide some download link to some huge & equivalent data set used by the demo like retz,eclepens,etc.. are this datasets under non "open" licenses? just wanted to test performance on my Titan V compared to a 3090.. thanks..

    opened by oscarbg 11
  • invisible window

    invisible window

    If I compile in DEBUG (VS2019) it crashes in void V8Helper::setupGL in this line: setupV8GLExtBindings(tpl);

    If I compile in RELEASE it compiles and the console shows no error but the windows is invisible, I can only see the console (and if I click keys I see them in the log).

    I have a 1070

    opened by jagenjo 3
  • A question of render.cs

    A question of render.cs

    image in render.cs there's vec2 variable called imgPos, i don't know why it times 0.5 and plus 0.5, what does it mean? Thank you very much if you can answer my questions. : )

    opened by UMR19 2
  • Questions about point cloud display when zoom in

    Questions about point cloud display when zoom in

    Hi, Thanks for your excellent job, just i compiled the demo and modified the setting to load my point cloud, it loaded successfully and have a better performance. but when i roll the mouse wheel to zoom in to look at the detail, lots of point missed, but when i use potree to display my point cloud, it display perfect. I am a beginner in computer graphics,may be it's point size is too small? image In the potree image

    Thanks for you reply:)

    opened by UMR19 0
  • ssRG and ssBA not bound to resolve?

    ssRG and ssBA not bound to resolve?

    In https://github.com/m-schuetz/compute_rasterizer/blob/master/compute_hqs/render.js

    Both buffers are bound in the render attribute pass, but neither is explicitly bound in the resolve pass. Always assumed that bindBufferBase affects the currently bound shader, but apparently the binding caries over to the next shader? Just a reminder to look this up to clarify my understanding of how they work.

    gl.bindBufferBase(gl.SHADER_STORAGE_BUFFER, 3, ssRG);
    gl.bindBufferBase(gl.SHADER_STORAGE_BUFFER, 4, ssBA);
    
    opened by m-schuetz 0
Releases(build_laz_crf)
Owner
Markus Schütz
Markus Schütz
Localized representation learning from Vision and Text (LoVT)

Localized Vision-Text Pre-Training Contrastive learning has proven effective for pre- training image models on unlabeled data and achieved great resul

Philip Müller 10 Dec 07, 2022
In this project, we develop a face recognize platform based on MTCNN object-detection netcwork and FaceNet self-supervised network.

模式识别大作业——人脸检测与识别平台 本项目是一个简易的人脸检测识别平台,提供了人脸信息录入和人脸识别的功能。前端采用 html+css+js,后端采用 pytorch,

Xuhua Huang 5 Aug 02, 2022
Classifying cat and dog images using Kaggle dataset

PyTorch Image Classification Classifies an image as containing either a dog or a cat (using Kaggle's public dataset), but could easily be extended to

Robert Coleman 74 Nov 22, 2022
Pytorch implementations of Bayes By Backprop, MC Dropout, SGLD, the Local Reparametrization Trick, KF-Laplace, SG-HMC and more

Bayesian Neural Networks Pytorch implementations for the following approximate inference methods: Bayes by Backprop Bayes by Backprop + Local Reparame

1.4k Jan 07, 2023
End-to-end image segmentation kit based on PaddlePaddle.

English | 简体中文 PaddleSeg PaddleSeg has released the new version including the following features: Our team won the 6.2k Jan 02, 2023

A PyTorch implementation of EventProp [https://arxiv.org/abs/2009.08378], a method to train Spiking Neural Networks

Spiking Neural Network training with EventProp This is an unofficial PyTorch implemenation of EventProp, a method to compute exact gradients for Spiki

Pedro Savarese 35 Jul 29, 2022
Multi-task Multi-agent Soft Actor Critic for SMAC

Multi-task Multi-agent Soft Actor Critic for SMAC Overview The CARE formulti-task: Multi-Task Reinforcement Learning with Context-based Representation

RuanJingqing 8 Sep 30, 2022
😇A pyTorch implementation of the DeepMoji model: state-of-the-art deep learning model for analyzing sentiment, emotion, sarcasm etc

------ Update September 2018 ------ It's been a year since TorchMoji and DeepMoji were released. We're trying to understand how it's being used such t

Hugging Face 865 Dec 24, 2022
Official implementation of the paper ``Unifying Nonlocal Blocks for Neural Networks'' (ICCV'21)

Spectral Nonlocal Block Overview Official implementation of the paper: Unifying Nonlocal Blocks for Neural Networks (ICCV'21) Spectral View of Nonloca

91 Dec 14, 2022
A PyTorch implementation of Implicit Q-Learning

IQL-PyTorch This repository houses a minimal PyTorch implementation of Implicit Q-Learning (IQL), an offline reinforcement learning algorithm, along w

Garrett Thomas 30 Dec 12, 2022
Implementations of orthogonal and semi-orthogonal convolutions in the Fourier domain with applications to adversarial robustness

Orthogonalizing Convolutional Layers with the Cayley Transform This repository contains implementations and source code to reproduce experiments for t

CMU Locus Lab 36 Dec 30, 2022
TensorLight - A high-level framework for TensorFlow

TensorLight is a high-level framework for TensorFlow-based machine intelligence applications. It reduces boilerplate code and enables advanced feature

Benjamin Kan 10 Jul 31, 2022
Veri Setinizi Yolov5 Formatına Dönüştürün

Veri Setinizi Yolov5 Formatına Dönüştürün! Bu Repo da Neler Var? Xml Formatındaki Veri Setini .Txt Formatına Çevirme Xml Formatındaki Dosyaları Silme

Kadir Nar 4 Aug 22, 2022
ALFRED - A Benchmark for Interpreting Grounded Instructions for Everyday Tasks

ALFRED A Benchmark for Interpreting Grounded Instructions for Everyday Tasks Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han,

ALFRED 204 Dec 15, 2022
Source-to-Source Debuggable Derivatives in Pure Python

Tangent Tangent is a new, free, and open-source Python library for automatic differentiation. Existing libraries implement automatic differentiation b

Google 2.2k Jan 01, 2023
Brain Tumor Detection with Tensorflow Neural Networks.

Brain-Tumor-Detection A convolutional neural network model built with Tensorflow & Keras to detect brain tumor and its different variants. Data of the

404ErrorNotFound 5 Aug 23, 2022
Metric learning algorithms in Python

metric-learn: Metric Learning in Python metric-learn contains efficient Python implementations of several popular supervised and weakly-supervised met

1.3k Jan 02, 2023
The codebase for our paper "Generative Occupancy Fields for 3D Surface-Aware Image Synthesis" (NeurIPS 2021)

Generative Occupancy Fields for 3D Surface-Aware Image Synthesis (NeurIPS 2021) Project Page | Paper Xudong Xu, Xingang Pan, Dahua Lin and Bo Dai GOF

xuxudong 97 Nov 10, 2022
This repository contains all the code and materials distributed in the 2021 Q-Programming Summer of Qode.

Q-Programming Summer of Qode This repository contains all the code and materials distributed in the Q-Programming Summer of Qode. If you want to creat

Sammarth Kumar 11 Jun 11, 2021
The code of paper 'Learning to Aggregate and Personalize 3D Face from In-the-Wild Photo Collection'

Learning to Aggregate and Personalize 3D Face from In-the-Wild Photo Collection Pytorch implemetation of paper 'Learning to Aggregate and Personalize

Tencent YouTu Research 136 Dec 29, 2022