Source code for "OmniPhotos: Casual 360° VR Photography"

Overview

OmniPhotos: Casual 360° VR Photography

Project Page | Video | Paper | Demo | Data

This repository contains the source code for creating and viewing OmniPhotos – a new approach for casual 360° VR photography using a consumer 360° video camera.

OmniPhotos: Casual 360° VR Photography
Tobias Bertel, Mingze Yuan, Reuben Lindroos, Christian Richardt
ACM Transactions on Graphics (SIGGRAPH Asia 2020)

Demo

The quickest way to try out OmniPhotos is via our precompiled demo (610 MB). Download and unzip to get started. Documentation for the precompiled binaries, which can also be downloaded separately (25 MB), can be found in the downloaded demo directory.

For the demo to run smoothly, we recommend a recently updated Windows 10 machine with a discrete GPU.

Additional OmniPhotos

We provide 31 OmniPhotos for download:

  • 9 preprocessed datasets that are ready for viewing (3.2 GB zipped, 12.8 GB uncompressed)
  • 31 unprocessed datasets with their input videos, camera poses etc.; this includes the 9 preprocessed datasets (17.4 GB zipped, 17.9 GB uncompressed)

Note: A few of the .insv files are missing for the 5.7k datasets. If you need to process these from scratch (using the insv files) these files can be found here.

How to view OmniPhotos

OmniPhotos are viewed using the "Viewer" executable, either in windowed mode (default) or in a compatible VR headset (see below). To run the viewer executable on the preprocessed datasets above, run the command:

Viewer.exe path-to-datasets/Preprocessed/

with paths adjusted for your machine. The viewer will automatically load the first dataset in the directory (in alphabetical order) and give you the option to load any of the datasets in the directory.

If you would like to run the viewer with VR enabled, please ensure that the firmware for your HMD is updated, you have SteamVR installed on your machine, and then run the command:

Viewer.exe --vr path-to-datasets/Preprocessed/

The OmniPhotos viewer can also load a specific single dataset directly:

Viewer.exe [--vr] path-to-datasets/Preprocessed/Temple3/Config/config-viewer.yaml

How to preprocess datasets

If you would like to preprocess additional datasets, for example "Ship" in the "Unprocessed" directory, run the command:

Preprocessing.exe path-to-datasets/Unpreprocessed/Ship/Config/config-viewer.yaml

This will preprocess the dataset according to the options specified in the config file. Once the preprocessing is finished, the dataset can be opened in the Viewer.

For processing new datasets from scratch, please follow the detailed documentation at Python/preprocessing/readme.md.

Compiling from source

The OmniPhotos Preprocessing and Viewer applications are written in C++11, with some Python used for preparing datasets.

Both main applications and the included libraries use CMake as build system generator. We recommend CMake 3.16 or newer, but older 3.x versions might also work.

Our code has been developed and tested with Microsoft Visual Studio 2015 and 2019 (both 64 bit).

Required dependencies

  1. GLFW 3.3 (version 3.3.1 works)
  2. Eigen 3.3 (version 3.3.2 works)
    • Please note: Ceres (an optional dependency) requires Eigen version "3.3.90" (~Eigen master branch).
  3. OpenCV 4.2
    • OpenCV 4.2 includes DIS flow in the main distribution, so precompiled OpenCV can be used.
    • OpenCV 4.1.1 needs to be compiled from source with the optflow contrib package (for DIS flow).
    • We also support the CUDA Brox flow from the cudaoptflow module, if it is compiled in. In this case, tick USE_CUDA_IN_OPENCV in CMake.
  4. OpenGL 4.1: provided by the operating system
  5. glog (newer than 0.4.0 master works)
  6. gflags (version 2.2.2 works)

Included dependencies (in /src/3rdParty/)

  1. DearImGui 1.79: included automatically as a git submodule.
  2. GL3W
  3. JsonCpp 1.8.0: almalgamated version
  4. nlohmann/json 3.6.1
  5. OpenVR 1.10.30: enable with WITH_OPENVR in CMake.
  6. TCLAP
  7. tinyfiledialogs 3.3.8

Optional dependencies

  1. Ceres (with SuiteSparse) is required for the scene-adaptive proxy geometry fitting. Enable with USE_CERES in CMake.
  2. googletest (master): automatically added when WITH_TEST is enabled in CMake.

Citation

Please cite our paper if you use this code or any of our datasets:

@article{OmniPhotos,
  author    = {Tobias Bertel and Mingze Yuan and Reuben Lindroos and Christian Richardt},
  title     = {{OmniPhotos}: Casual 360° {VR} Photography},
  journal   = {ACM Transactions on Graphics},
  year      = {2020},
  volume    = {39},
  number    = {6},
  pages     = {266:1--12},
  month     = dec,
  issn      = {0730-0301},
  doi       = {10.1145/3414685.3417770},
  url       = {https://richardt.name/omniphotos/},
}

Acknowledgements

We thank the reviewers for their thorough feedback that has helped to improve our paper. We also thank Peter Hedman, Ana Serrano and Brian Cabral for helpful discussions, and Benjamin Attal for his layered mesh rendering code.

This work was supported by EU Horizon 2020 MSCA grant FIRE (665992), the EPSRC Centre for Doctoral Training in Digital Entertainment (EP/L016540/1), RCUK grant CAMERA (EP/M023281/1), an EPSRC-UKRI Innovation Fellowship (EP/S001050/1), a Rabin Ezra Scholarship and an NVIDIA Corporation GPU Grant.

Comments
  • Documentation pipeline update

    Documentation pipeline update

    Pipeline to automatically create documenation on read the docs using doxygen.

    • [ ] link to github page on index.html (mainpage.hpp)
    • [ ] add documentation convention to docs\README.md
    • [ ] fork branch from cr333/main and apply changes to that
    • [ ] create new PR from forked branch
    opened by reubenlindroos 1
  • Adds a progress bar when circleselector is running

    Adds a progress bar when circleselector is running

    Also improves speed of the circleselector module by ~50%

    Todo:

    • [x] add tqdm to requirements.txt
    • [x] np.diffs for find_path_length
    • [x] atomic lock for incrementing the progress bar?
    opened by reubenlindroos 0
  • Circle Selector

    Circle Selector

    • [x] clean up requirements.txt
    • [x] save plot of heatmaps to the cache/dataset directory.
    • [x] move json file to capture directory
    • [x] update the template with option to switch off circlefitting
    • [x] update template to remove some of the options (e.g op_filename_expression)
    • [x] Update README.md with automatic circle selection (section 2.2)
    • [x] Update documentation for installation?
    • [x] Linting (spacing), comment convention, Pep convention
    • [x] sort imports
    • [x] replace op_filename_epression with original_filename_expression

    cv_utils

    • [x] remove extra copy of computeColor
    • [x] change pjoin to os.path.join
    • [x] add more documentation for parameters in cv_utils (change lookatang to look_at_angle)
    • [x] 'nxt' to 'next'
    • [x] comment on line 100 (slice_equirect)

    datatypes

    • [x] more comments on some of the methods in PointDict
    opened by reubenlindroos 0
  • Documentation pipeline update

    Documentation pipeline update

    Pipeline to automatically create documenation on read the docs using doxygen.

    • [x] link to github page on index.html (mainpage.hpp)
    • [x] add documentation convention to docs\README.md
    • [x] fork branch from cr333/main and apply changes to that
    • [x] create new PR from forked branch
    • [x] remove documentation for header comment block in docs/README.md
    • [x] cleanup index.rst (try removing, see if sphinx can build anyway)
    • [ ] mainpage.hpp cleanup (capitalise, centralise)
    • [x] clarify line 48 in README.md
    opened by reubenlindroos 0
  • Demo updated

    Demo updated

    Converts demo documentation files from rst and based in sphinx to be hosted in Github. The sites markdown API now renders the documentation files rather than using sphinx + rtd.

    opened by reubenlindroos 0
  • Adds build test to master branch on push and PR

    Adds build test to master branch on push and PR

    build

    • [ ] change actions to not send email for every build
    • [x] fix requested changes
    • [x] make into squash merge to not mess with main branch history
    • [x] group build steps (building dependencies which ahve been left seperate for debugging purposes)
    • [x] check glog build variables in cmake (e.g BUILD_TEST should not be enabled)
    • [x] check eigen warnings in build log
    • [x] check if precompiled headers might speed up build
    • [x] check if multithread build could be used
    • [x] remove verbose flag from extraction of opencv

    test

    • [x] add test data download
    • [x] reduce size of test dataset
    • [x] check what happens on failure
    • [ ] check if we can "publish" test results (xml?)
    opened by reubenlindroos 0
  • Get problems while preprocessing

    Get problems while preprocessing

    I did download all of those binary files from here:https://github.com/cr333/OmniPhotos/releases/download/v1.1/OmniPhotos-v1.1-win10-x64.zip

    And I did put ffmpeg.exe into system Path. However, Im getting errors saying this below:

    $ ./preproc/preproc.exe -c preproc-config-template.yaml [23276] Failed to execute script 'main' due to unhandled exception! Traceback (most recent call last): File "main.py", line 24, in File "preproc_app.py", line 39, in init File "data_preprocessor.py", line 32, in init File "abs_preprocessor.py", line 70, in init File "abs_preprocessor.py", line 225, in load_origin_data_info File "ffmpeg_probe.py", line 20, in probe File "subprocess.py", line 800, in init File "subprocess.py", line 1207, in _execute_child FileNotFoundError: [WinError 2]

    image

    opened by BlairLeng 2
Releases(v1.1)
Owner
Christian Richardt
Christian Richardt
This repository contains code released by Google Research.

This repository contains code released by Google Research.

Google Research 26.6k Dec 31, 2022
Multi-Horizon-Forecasting-for-Limit-Order-Books

Multi-Horizon-Forecasting-for-Limit-Order-Books This jupyter notebook is used to demonstrate our work, Multi-Horizon Forecasting for Limit Order Books

Zihao Zhang 116 Dec 23, 2022
Code needed to reproduce the examples found in "The Temporal Robustness of Stochastic Signals"

The Temporal Robustness of Stochastic Signals Code needed to reproduce the examples found in "The Temporal Robustness of Stochastic Signals" Case stud

0 Oct 28, 2021
Implementation of the paper ''Implicit Feature Refinement for Instance Segmentation''.

Implicit Feature Refinement for Instance Segmentation This repository is an official implementation of the ACM Multimedia 2021 paper Implicit Feature

Lufan Ma 17 Dec 28, 2022
code for "Self-supervised edge features for improved Graph Neural Network training",

Self-supervised edge features for improved Graph Neural Network training Data availability: Here is a link to the raw data for the organoids dataset.

Neal Ravindra 23 Dec 02, 2022
A spherical CNN for weather forecasting

DeepSphere-Weather - Deep Learning on the sphere for weather/climate applications. The code in this repository provides a scalable and flexible framew

DeepSphere 47 Dec 25, 2022
Pytorch implementation of the paper Progressive Growing of Points with Tree-structured Generators (BMVC 2021)

PGpoints Pytorch implementation of the paper Progressive Growing of Points with Tree-structured Generators (BMVC 2021) Hyeontae Son, Young Min Kim Pre

Hyeontae Son 9 Jun 06, 2022
Source Code for our paper: Understand me, if you refer to Aspect Knowledge: Knowledge-aware Gated Recurrent Memory Network

KaGRMN-DSG_ABSA This repository contains the PyTorch source Code for our paper: Understand me, if you refer to Aspect Knowledge: Knowledge-aware Gated

XingBowen 4 May 20, 2022
Sudoku solver - A sudoku solver with python

sudoku_solver A sudoku solver What is Sudoku? Sudoku (Japanese: 数独, romanized: s

Sikai Lu 0 May 22, 2022
Official implementation of GraphMask as presented in our paper Interpreting Graph Neural Networks for NLP With Differentiable Edge Masking.

GraphMask This repository contains an implementation of GraphMask, the interpretability technique for graph neural networks presented in our ICLR 2021

Michael Schlichtkrull 29 Sep 02, 2022
Cl datasets - PyTorch image dataloaders and utility functions to load datasets for supervised continual learning

Continual learning datasets Introduction This repository contains PyTorch image

berjaoui 5 Aug 28, 2022
[CVPR'21] Locally Aware Piecewise Transformation Fields for 3D Human Mesh Registration

Locally Aware Piecewise Transformation Fields for 3D Human Mesh Registration This repository contains the implementation of our paper Locally Aware Pi

sfwang 70 Dec 19, 2022
Another pytorch implementation of FCN (Fully Convolutional Networks)

FCN-pytorch-easiest Trying to be the easiest FCN pytorch implementation and just in a get and use fashion Here I use a handbag semantic segmentation f

Y. Dong 158 Dec 21, 2022
A bare-bones TensorFlow framework for Bayesian deep learning and Gaussian process approximation

Aboleth A bare-bones TensorFlow framework for Bayesian deep learning and Gaussian process approximation [1] with stochastic gradient variational Bayes

Gradient Institute 127 Dec 12, 2022
Efficient neural networks for analog audio effect modeling

micro-TCN Efficient neural networks for audio effect modeling

Christian Steinmetz 94 Dec 29, 2022
A robust camera and Lidar fusion based velocity estimator to undistort the pointcloud.

Lidar with Velocity A robust camera and Lidar fusion based velocity estimator to undistort the pointcloud. related paper: Lidar with Velocity : Motion

ISEE Research Group 164 Dec 30, 2022
Transfer SemanticKITTI labeles into other dataset/sensor formats.

LiDAR-Transfer Transfer SemanticKITTI labeles into other dataset/sensor formats. Content Convert datasets (NUSCENES, FORD, NCLT) to KITTI format Minim

Photogrammetry & Robotics Bonn 64 Nov 21, 2022
Implementation of Stochastic Image-to-Video Synthesis using cINNs.

Stochastic Image-to-Video Synthesis using cINNs Official PyTorch implementation of Stochastic Image-to-Video Synthesis using cINNs accepted to CVPR202

CompVis Heidelberg 135 Dec 28, 2022
YOLOX Win10 Project

Introduction 这是一个用于Windows训练YOLOX的项目,相比于官方项目,做了一些适配和修改: 1、解决了Windows下import yolox失败,No such file or directory: 'xxx.xml'等路径问题 2、CUDA out of memory等显存不

5 Jun 08, 2022
A little Python application to auto tag your photos with the power of machine learning.

Tag Machine A little Python application to auto tag your photos with the power of machine learning. Report a bug or request a feature Table of Content

Florian Torres 14 Dec 21, 2022