Implementation of the ICCV'21 paper Temporally-Coherent Surface Reconstruction via Metric-Consistent Atlases

Overview

Temporally-Coherent Surface Reconstruction via Metric-Consistent Atlases [Papers 1, 2][Project page] [Video]

The implementation of the papers

Install

The framework was tested with Python 3.8, PyTorch 1.7.0. and CUDA 11.0. The easiest way to work with the code is to create a new virtual Python environment and install the required packages.

  1. Install the virtualenvwrapper.
  2. Create a new environment and install the required packages.
mkvirtualenv --python=python3.8 tcsr
pip install -r requirements.txt
  1. Install Pytorch3d.
cd ~
curl -LO https://github.com/NVIDIA/cub/archive/1.10.0.tar.gz
tar xzf 1.10.0.tar.gz
export CUB_HOME=$PWD/cub-1.10.0
pip install git+https://github.com/facebookresearch/[email protected]
  1. Get the code and prepare the environment as follows:
git clone [email protected]:bednarikjan/temporally_coherent_surface_reconstruction.git
git submodule update --init --recursive
export PYTHONPATH="{PYTHONPATH}:path/to/dir/temporally_coherent_surface_reconstruction"

Get the Data

The project was tested on 6 base datasets (and their derivatives). Each datasets has to be processed so as to generate the input point clouds for training, the GT correspondences for evauluation and other auxilliary data. To do so, please use the individual scripts in tcsr/process_datasets. For each dataset, follow these steps:

  1. Download the data (links below).
  2. Open the script <dataset_name>.py and set the input/output paths.
  3. Run the script: python <dataset_name>.py

1. ANIM

  • Download the sequences horse gallop, horse collapse, camel gallop, camel collapse, and elephant gallop.
  • Download the sequence walking cat.

2. AMA

  • Download all 10 sequences, meshes only.

3. DFAUST

4. CAPE

  • Request the access to the raw scans and download it.
  • At the time of writing the paper (September 2021) four subjects (00032, 00096, 00159, 03223) were available and used in the paper.

5. INRIA

  • Request the access to the dataset and download it.
  • At the time of writing the paper (September 2021), four subjects (s1, s2, s3, s6) were available and used in the paper.

6. CMU

Train

The provided code allows for training our proposed method (OUR) but also the other atlas based approaches Differential Surface Representation (DSR) and AtlasNet (AN). The training is configured using the *.yaml configuration scripts in tcsr/train/configs.

There are 9 sample configuration files our_<dataset_name>.yaml which train OUR on each individual dataset and 2 sample configuration files an_anim.yaml, dsr_anim.yaml which train AN and DSR respectivelly on ANIM dataset.

By default, the trainin uses the exact settings as in the paper, namely it trains for 200'000 iterations using SGD, learning rate of 0.001 and batch size of 4. This can be altered in the configuration files.

Before starting the training, follow these steps:

  • Open the source file tcsr/data/data_loader.py and set the paths to the datasets in each dataset class.
  • Open the desired training configuration *.yaml file in tcsr/train/configs/ and set the output path for the training run data in the attribute path_train_run.

Start the training usint the script tcsr/train/train.py:

python train.py --conf configs/<file_name>.yaml

By default the script saves the training progress each 2000 iterations so you can safely kill it at any point and resume the trianing later using:

python train.py --cont path/to/training_run/root_dir

Evaluate

To evaluate a trianed model on the dense correspondence prediction task, use the script tcsr/evaluate/eval_dataset.py which allows for evaluation of multiple sequences (i.e. individual training runs within one dataset) at once. Please have a look at the command line arguments in the file.

An example of how to run the evaluation for the training runs contained in the root directory train_runs_root corresponding to 2 training runs run for the sequences cat_walk and horse_gallop within ANIM dataset:

python eval_dataset.py /path/to/train_runs_root --ds anim --include_seqs cat_walk horse_gallop  

The script produces a *.csv file in train_runs_root with the 4 measured metrics (see the paper).

Visualize

There are currently two ways to visualize the predictions.

1. Tensorboard

By default, the training script saves the GT and the predicted point clouds (for a couple of random data samples) each 2000 iterations. These can be viewed within Tensorboard. Each patch is visualized with a different color. This visualization is mostly useful as a sanity check during the trianing to see that the model is converging as expected.

  • Navigate to the root directory of the trianing runs and run:
tensorboard --logdir=. --port=8008 --bind_all
  • Open your browser and navigate to http://localhost:8008/

2. Per-sequence reconstruction GIF

You can view the reconstructed surfaces as a patch-wise textured mesh as a video within a GIF file. For this purpose, use the IPython Notebook file tcsr/visualize/render_uv.ipynb and open it in jupyterlab which allows for viewing the GIF right after running the code.

The rendering parameters (such as the camera location, texturing mode, gif speed etc.) are set usin the configuration file tcsr/visualize/conf_patches.yaml. There are sample configurations for the sequence cat_walk, which can be used to write configurations for other sequences/datasets.

Before running the cells, set the variables in the second cell (paths, models, data).

Citation

@inproceedings{bednarik2021temporally_coherent,
   title = {Temporally-Coherent Surface Reconstruction via Metric-Consistent Atlases},
   author = {Bednarik, Jan and Kim, Vladimir G. and Chaudhuri, Siddhartha and Parashar, Shaifali and Salzmann, Mathieu and Fua, Pascal and Aigerman, Noam},
   booktitle = {Proceedings of IEEE International Conference on Computer Vision (ICCV)},
   year = {2021}
}

@inproceedings{bednarik2021temporally_consistent,
   title = {Temporally-Consistent Surface Reconstruction via Metrically-Consistent Atlases},
   author = {Bednarik, Jan and Aigerman, Noam and Kim, Vladimir G. and Chaudhuri, Siddhartha and Parashar, Shaifali and Salzmann, Mathieu and Fua, Pascal},
   booktitle = {arXiv},
   year = {2021}
}

Acknowledgements

This work was partially done while the main author was an intern at Adobe Research.

TODO

  • Add support for visualizing the correspondence error heatmap on the GT mesh.
  • Add support for visualizing the colorcoded correspondences on the GT mesh.
  • Add the support for generating the pre-aligned AMAa dataset using ICP.
  • Add the code for the nonrigid ICP experiments.
Robustness via Cross-Domain Ensembles

Robustness via Cross-Domain Ensembles [ICCV 2021, Oral] This repository contains tools for training and evaluating: Pretrained models Demo code Traini

Visual Intelligence & Learning Lab, Swiss Federal Institute of Technology (EPFL) 27 Dec 23, 2022
Implements Gradient Centralization and allows it to use as a Python package in TensorFlow

Gradient Centralization TensorFlow This Python package implements Gradient Centralization in TensorFlow, a simple and effective optimization technique

Rishit Dagli 101 Nov 01, 2022
An abstraction layer for mathematical optimization solvers.

MathOptInterface Documentation Build Status Social An abstraction layer for mathematical optimization solvers. Replaces MathProgBase. Citing MathOptIn

JuMP-dev 284 Jan 04, 2023
Pytorch implementation for our ICCV 2021 paper "TRAR: Routing the Attention Spans in Transformers for Visual Question Answering".

TRAnsformer Routing Networks (TRAR) This is an official implementation for ICCV 2021 paper "TRAR: Routing the Attention Spans in Transformers for Visu

Ren Tianhe 49 Nov 10, 2022
Contains a bunch of different python programm tasks

py_tasks Contains a bunch of different python programm tasks Armstrong.py - calculate Armsrong numbers in range from 0 to n with / without cache and c

Dmitry Chmerenko 1 Dec 17, 2021
The repository for our EMNLP 2021 paper "Finnish Dialect Identification: The Effect of Audio and Text"

Finnish Dialect Identification The repository for our EMNLP 2021 paper "Finnish Dialect Identification: The Effect of Audio and Text". We present a te

Rootroo Ltd 2 Dec 25, 2021
Resources for the "Evaluating the Factual Consistency of Abstractive Text Summarization" paper

Evaluating the Factual Consistency of Abstractive Text Summarization Authors: Wojciech Kryściński, Bryan McCann, Caiming Xiong, and Richard Socher Int

Salesforce 165 Dec 21, 2022
[TNNLS 2021] The official code for the paper "Learning Deep Context-Sensitive Decomposition for Low-Light Image Enhancement"

CSDNet-CSDGAN this is the code for the paper "Learning Deep Context-Sensitive Decomposition for Low-Light Image Enhancement" Environment Preparing pyt

Jiaao Zhang 17 Nov 05, 2022
Learning Off-Policy with Online Planning, CoRL 2021

LOOP: Learning Off-Policy with Online Planning Accepted in Conference of Robot Learning (CoRL) 2021. Harshit Sikchi, Wenxuan Zhou, David Held Paper In

Harshit Sikchi 24 Nov 22, 2022
The MATH Dataset

Measuring Mathematical Problem Solving With the MATH Dataset This is the repository for Measuring Mathematical Problem Solving With the MATH Dataset b

Dan Hendrycks 267 Dec 26, 2022
An All-MLP solution for Vision, from Google AI

MLP Mixer - Pytorch An All-MLP solution for Vision, from Google AI, in Pytorch. No convolutions nor attention needed! Yannic Kilcher video Install $ p

Phil Wang 784 Jan 06, 2023
Everything about being a TA for ITP/AP course!

تی‌ای بودن! تی‌ای یا دستیار استاد از نقش‌های رایج بین دانشجویان مهندسی است، این ریپوزیتوری قرار است نکات مهم درمورد تی‌ای بودن و تی ای شدن را به ما نش

<a href=[email protected]"> 14 Sep 10, 2022
PyTorch implementation of Munchausen Reinforcement Learning based on DQN and SAC. Handles discrete and continuous action spaces

Exploring Munchausen Reinforcement Learning This is the project repository of my team in the "Advanced Deep Learning for Robotics" course at TUM. Our

Mohamed Amine Ketata 10 Mar 10, 2022
BARTScore: Evaluating Generated Text as Text Generation

This is the Repo for the paper: BARTScore: Evaluating Generated Text as Text Generation Updates 2021.06.28 Release online evaluation Demo 2021.06.25 R

NeuLab 196 Dec 17, 2022
Retinal vessel segmentation based on GT-UNet

Retinal vessel segmentation based on GT-UNet Introduction This project is a retinal blood vessel segmentation code based on UNet-like Group Transforme

Kent0n 27 Dec 18, 2022
Official repository for MixFaceNets: Extremely Efficient Face Recognition Networks

MixFaceNets This is the official repository of the paper: MixFaceNets: Extremely Efficient Face Recognition Networks. (Accepted in IJCB2021) https://i

Fadi Boutros 51 Dec 13, 2022
Generate Contextual Directory Wordlist For Target Org

PathPermutor Generate Contextual Directory Wordlist For Target Org This script generates contextual wordlist for any target org based on the set of UR

8 Jun 23, 2021
Simulating an AI playing 2048 using the Expectimax algorithm

2048-expectimax Simulating an AI playing 2048 using the Expectimax algorithm The base game engine uses code from here. The AI player is modeled as a m

Subha Ramesh 2 Jan 31, 2022