Source code for "OmniPhotos: Casual 360° VR Photography"

Overview

OmniPhotos: Casual 360° VR Photography

Project Page | Video | Paper | Demo | Data

This repository contains the source code for creating and viewing OmniPhotos – a new approach for casual 360° VR photography using a consumer 360° video camera.

OmniPhotos: Casual 360° VR Photography
Tobias Bertel, Mingze Yuan, Reuben Lindroos, Christian Richardt
ACM Transactions on Graphics (SIGGRAPH Asia 2020)

Demo

The quickest way to try out OmniPhotos is via our precompiled demo (610 MB). Download and unzip to get started. Documentation for the precompiled binaries, which can also be downloaded separately (25 MB), can be found in the downloaded demo directory.

For the demo to run smoothly, we recommend a recently updated Windows 10 machine with a discrete GPU.

Additional OmniPhotos

We provide 31 OmniPhotos for download:

  • 9 preprocessed datasets that are ready for viewing (3.2 GB zipped, 12.8 GB uncompressed)
  • 31 unprocessed datasets with their input videos, camera poses etc.; this includes the 9 preprocessed datasets (17.4 GB zipped, 17.9 GB uncompressed)

Note: A few of the .insv files are missing for the 5.7k datasets. If you need to process these from scratch (using the insv files) these files can be found here.

How to view OmniPhotos

OmniPhotos are viewed using the "Viewer" executable, either in windowed mode (default) or in a compatible VR headset (see below). To run the viewer executable on the preprocessed datasets above, run the command:

Viewer.exe path-to-datasets/Preprocessed/

with paths adjusted for your machine. The viewer will automatically load the first dataset in the directory (in alphabetical order) and give you the option to load any of the datasets in the directory.

If you would like to run the viewer with VR enabled, please ensure that the firmware for your HMD is updated, you have SteamVR installed on your machine, and then run the command:

Viewer.exe --vr path-to-datasets/Preprocessed/

The OmniPhotos viewer can also load a specific single dataset directly:

Viewer.exe [--vr] path-to-datasets/Preprocessed/Temple3/Config/config-viewer.yaml

How to preprocess datasets

If you would like to preprocess additional datasets, for example "Ship" in the "Unprocessed" directory, run the command:

Preprocessing.exe path-to-datasets/Unpreprocessed/Ship/Config/config-viewer.yaml

This will preprocess the dataset according to the options specified in the config file. Once the preprocessing is finished, the dataset can be opened in the Viewer.

For processing new datasets from scratch, please follow the detailed documentation at Python/preprocessing/readme.md.

Compiling from source

The OmniPhotos Preprocessing and Viewer applications are written in C++11, with some Python used for preparing datasets.

Both main applications and the included libraries use CMake as build system generator. We recommend CMake 3.16 or newer, but older 3.x versions might also work.

Our code has been developed and tested with Microsoft Visual Studio 2015 and 2019 (both 64 bit).

Required dependencies

  1. GLFW 3.3 (version 3.3.1 works)
  2. Eigen 3.3 (version 3.3.2 works)
    • Please note: Ceres (an optional dependency) requires Eigen version "3.3.90" (~Eigen master branch).
  3. OpenCV 4.2
    • OpenCV 4.2 includes DIS flow in the main distribution, so precompiled OpenCV can be used.
    • OpenCV 4.1.1 needs to be compiled from source with the optflow contrib package (for DIS flow).
    • We also support the CUDA Brox flow from the cudaoptflow module, if it is compiled in. In this case, tick USE_CUDA_IN_OPENCV in CMake.
  4. OpenGL 4.1: provided by the operating system
  5. glog (newer than 0.4.0 master works)
  6. gflags (version 2.2.2 works)

Included dependencies (in /src/3rdParty/)

  1. DearImGui 1.79: included automatically as a git submodule.
  2. GL3W
  3. JsonCpp 1.8.0: almalgamated version
  4. nlohmann/json 3.6.1
  5. OpenVR 1.10.30: enable with WITH_OPENVR in CMake.
  6. TCLAP
  7. tinyfiledialogs 3.3.8

Optional dependencies

  1. Ceres (with SuiteSparse) is required for the scene-adaptive proxy geometry fitting. Enable with USE_CERES in CMake.
  2. googletest (master): automatically added when WITH_TEST is enabled in CMake.

Citation

Please cite our paper if you use this code or any of our datasets:

@article{OmniPhotos,
  author    = {Tobias Bertel and Mingze Yuan and Reuben Lindroos and Christian Richardt},
  title     = {{OmniPhotos}: Casual 360° {VR} Photography},
  journal   = {ACM Transactions on Graphics},
  year      = {2020},
  volume    = {39},
  number    = {6},
  pages     = {266:1--12},
  month     = dec,
  issn      = {0730-0301},
  doi       = {10.1145/3414685.3417770},
  url       = {https://richardt.name/omniphotos/},
}

Acknowledgements

We thank the reviewers for their thorough feedback that has helped to improve our paper. We also thank Peter Hedman, Ana Serrano and Brian Cabral for helpful discussions, and Benjamin Attal for his layered mesh rendering code.

This work was supported by EU Horizon 2020 MSCA grant FIRE (665992), the EPSRC Centre for Doctoral Training in Digital Entertainment (EP/L016540/1), RCUK grant CAMERA (EP/M023281/1), an EPSRC-UKRI Innovation Fellowship (EP/S001050/1), a Rabin Ezra Scholarship and an NVIDIA Corporation GPU Grant.

Comments
  • Documentation pipeline update

    Documentation pipeline update

    Pipeline to automatically create documenation on read the docs using doxygen.

    • [ ] link to github page on index.html (mainpage.hpp)
    • [ ] add documentation convention to docs\README.md
    • [ ] fork branch from cr333/main and apply changes to that
    • [ ] create new PR from forked branch
    opened by reubenlindroos 1
  • Adds a progress bar when circleselector is running

    Adds a progress bar when circleselector is running

    Also improves speed of the circleselector module by ~50%

    Todo:

    • [x] add tqdm to requirements.txt
    • [x] np.diffs for find_path_length
    • [x] atomic lock for incrementing the progress bar?
    opened by reubenlindroos 0
  • Circle Selector

    Circle Selector

    • [x] clean up requirements.txt
    • [x] save plot of heatmaps to the cache/dataset directory.
    • [x] move json file to capture directory
    • [x] update the template with option to switch off circlefitting
    • [x] update template to remove some of the options (e.g op_filename_expression)
    • [x] Update README.md with automatic circle selection (section 2.2)
    • [x] Update documentation for installation?
    • [x] Linting (spacing), comment convention, Pep convention
    • [x] sort imports
    • [x] replace op_filename_epression with original_filename_expression

    cv_utils

    • [x] remove extra copy of computeColor
    • [x] change pjoin to os.path.join
    • [x] add more documentation for parameters in cv_utils (change lookatang to look_at_angle)
    • [x] 'nxt' to 'next'
    • [x] comment on line 100 (slice_equirect)

    datatypes

    • [x] more comments on some of the methods in PointDict
    opened by reubenlindroos 0
  • Documentation pipeline update

    Documentation pipeline update

    Pipeline to automatically create documenation on read the docs using doxygen.

    • [x] link to github page on index.html (mainpage.hpp)
    • [x] add documentation convention to docs\README.md
    • [x] fork branch from cr333/main and apply changes to that
    • [x] create new PR from forked branch
    • [x] remove documentation for header comment block in docs/README.md
    • [x] cleanup index.rst (try removing, see if sphinx can build anyway)
    • [ ] mainpage.hpp cleanup (capitalise, centralise)
    • [x] clarify line 48 in README.md
    opened by reubenlindroos 0
  • Demo updated

    Demo updated

    Converts demo documentation files from rst and based in sphinx to be hosted in Github. The sites markdown API now renders the documentation files rather than using sphinx + rtd.

    opened by reubenlindroos 0
  • Adds build test to master branch on push and PR

    Adds build test to master branch on push and PR

    build

    • [ ] change actions to not send email for every build
    • [x] fix requested changes
    • [x] make into squash merge to not mess with main branch history
    • [x] group build steps (building dependencies which ahve been left seperate for debugging purposes)
    • [x] check glog build variables in cmake (e.g BUILD_TEST should not be enabled)
    • [x] check eigen warnings in build log
    • [x] check if precompiled headers might speed up build
    • [x] check if multithread build could be used
    • [x] remove verbose flag from extraction of opencv

    test

    • [x] add test data download
    • [x] reduce size of test dataset
    • [x] check what happens on failure
    • [ ] check if we can "publish" test results (xml?)
    opened by reubenlindroos 0
  • Get problems while preprocessing

    Get problems while preprocessing

    I did download all of those binary files from here:https://github.com/cr333/OmniPhotos/releases/download/v1.1/OmniPhotos-v1.1-win10-x64.zip

    And I did put ffmpeg.exe into system Path. However, Im getting errors saying this below:

    $ ./preproc/preproc.exe -c preproc-config-template.yaml [23276] Failed to execute script 'main' due to unhandled exception! Traceback (most recent call last): File "main.py", line 24, in File "preproc_app.py", line 39, in init File "data_preprocessor.py", line 32, in init File "abs_preprocessor.py", line 70, in init File "abs_preprocessor.py", line 225, in load_origin_data_info File "ffmpeg_probe.py", line 20, in probe File "subprocess.py", line 800, in init File "subprocess.py", line 1207, in _execute_child FileNotFoundError: [WinError 2]

    image

    opened by BlairLeng 2
Releases(v1.1)
Owner
Christian Richardt
Christian Richardt
Capstone-Project-2 - A game program written in the Python language

Capstone-Project-2 My Pygame Game Information: Description This Pygame project i

Nhlakanipho Khulekani Hlophe 1 Jan 04, 2022
RM Operation can equivalently convert ResNet to VGG, which is better for pruning; and can help RepVGG perform better when the depth is large.

RM Operation can equivalently convert ResNet to VGG, which is better for pruning; and can help RepVGG perform better when the depth is large.

184 Jan 04, 2023
A Demo server serving Bert through ONNX with GPU written in Rust with <3

Demo BERT ONNX server written in rust This demo showcase the use of onnxruntime-rs on BERT with a GPU on CUDA 11 served by actix-web and tokenized wit

Xavier Tao 28 Jan 01, 2023
《Geo Word Clouds》paper implementation

《Geo Word Clouds》paper implementation

Russellwzr 2 Jan 28, 2022
pytorch implementation of dftd2 & dftd3

torch-dftd pytorch implementation of dftd2 [1] & dftd3 [2, 3] Install # Install from pypi pip install torch-dftd # Install from source (for developer

33 Nov 28, 2022
Incremental Cross-Domain Adaptation for Robust Retinopathy Screening via Bayesian Deep Learning

Incremental Cross-Domain Adaptation for Robust Retinopathy Screening via Bayesian Deep Learning Update (September 18th, 2021) A supporting document de

Taimur Hassan 1 Mar 16, 2022
Tutorial for the PERFECTING FACTORY 5.0 WITH EDGE-POWERED AI workshop

Workshop Advantech Jetson Nano This tutorial has been designed for the PERFECTING FACTORY 5.0 WITH EDGE-POWERED AI workshop in collaboration with Adva

Edge Impulse 18 Nov 22, 2022
CRNN With PyTorch

CRNN-PyTorch Implementation of https://arxiv.org/abs/1507.05717

Vadim 4 Sep 01, 2022
PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation

PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Created by Charles R. Qi, Hao Su, Kaichun Mo, Leonidas J. Guibas from Sta

Charles R. Qi 4k Dec 30, 2022
This is a collection of all challenges in HKCERT CTF 2021

香港網絡保安新生代奪旗挑戰賽 2021 (HKCERT CTF 2021) This is a collection of all challenges (and writeups) in HKCERT CTF 2021 Challenges ID Chinese name Name Score S

10 Jan 27, 2022
The official implementation for "FQ-ViT: Fully Quantized Vision Transformer without Retraining".

FQ-ViT [arXiv] This repo contains the official implementation of "FQ-ViT: Fully Quantized Vision Transformer without Retraining". Table of Contents In

132 Jan 08, 2023
Segcache: a memory-efficient and scalable in-memory key-value cache for small objects

Segcache: a memory-efficient and scalable in-memory key-value cache for small objects This repo contains the code of Segcache described in the followi

TheSys Group @ CMU CS 78 Jan 07, 2023
[NeurIPS 2021] “Improving Contrastive Learning on Imbalanced Data via Open-World Sampling”,

Improving Contrastive Learning on Imbalanced Data via Open-World Sampling Introduction Contrastive learning approaches have achieved great success in

VITA 24 Dec 17, 2022
A fast Evolution Strategy implementation in Python

Evostra: Evolution Strategy for Python Evolution Strategy (ES) is an optimization technique based on ideas of adaptation and evolution. You can learn

Mika 251 Dec 08, 2022
Seq2seq - Sequence to Sequence Learning with Keras

Seq2seq Sequence to Sequence Learning with Keras Hi! You have just found Seq2Seq. Seq2Seq is a sequence to sequence learning add-on for the python dee

Fariz Rahman 3.1k Dec 18, 2022
Customised to detect objects automatically by a given model file(onnx)

LabelImg LabelImg is a graphical image annotation tool. It is written in Python and uses Qt for its graphical interface. Annotations are saved as XML

Heeone Lee 1 Jun 07, 2022
PyTorch implementation of the paper: "Preference-Adaptive Meta-Learning for Cold-Start Recommendation", IJCAI, 2021.

PAML PyTorch implementation of the paper: "Preference-Adaptive Meta-Learning for Cold-Start Recommendation", IJCAI, 2021. (Continuously updating ) Int

15 Nov 18, 2022
A Low Complexity Speech Enhancement Framework for Full-Band Audio (48kHz) based on Deep Filtering.

DeepFilterNet A Low Complexity Speech Enhancement Framework for Full-Band Audio (48kHz) based on Deep Filtering. libDF contains Rust code used for dat

Hendrik Schröter 292 Dec 25, 2022
Official repo for AutoInt: Automatic Integration for Fast Neural Volume Rendering in CVPR 2021

AutoInt: Automatic Integration for Fast Neural Volume Rendering CVPR 2021 Project Page | Video | Paper PyTorch implementation of automatic integration

Stanford Computational Imaging Lab 149 Dec 22, 2022
Noise Conditional Score Networks (NeurIPS 2019, Oral)

Generative Modeling by Estimating Gradients of the Data Distribution This repo contains the official implementation for the NeurIPS 2019 paper Generat

451 Dec 26, 2022