Official repository of the paper Learning to Regress 3D Face Shape and Expression from an Image without 3D Supervision

Overview

RingNet

alt text

This is an official repository of the paper Learning to Regress 3D Face Shape and Expression from an Image without 3D Supervision. The project was formerly referred by RingNet. The codebase consists of the inference code, i.e. give an face image using this code one can generate a 3D mesh of a complete head with the face region. For further details on the method please refer to the following publication,

Learning to Regress 3D Face Shape and Expression from an Image without 3D Supervision
Soubhik Sanyal, Timo Bolkart, Haiwen Feng, Michael J. Black
CVPR 2019

More details on our NoW benchmark dataset, 3D face reconstruction challenge can be found in our project page. A pdf preprint is also available on the project page.

  • Update: We have released the evaluation code for NoW Benchmark challenge here.

  • Update: Add demo to build a texture for the reconstructed mesh from the input image.

  • Update: NoW Dataset is divided into Test set and Validation Set. Ground Truth scans are available for the Validation Set. Please Check our project page for more details.

  • Update: We have released a PyTorch implementation of the decoder FLAME with dynamic conture loading which can be directly used for training networks. Please check FLAME_PyTorch for the code.

Installation

The code uses Python 2.7 and it is tested on Tensorflow gpu version 1.12.0, with CUDA-9.0 and cuDNN-7.3.

Setup RingNet Virtual Environment

virtualenv --no-site-packages 
   
    /.virtualenvs/RingNet
source 
    
     /.virtualenvs/RingNet/bin/activate
pip install --upgrade pip==19.1.1

    
   

Clone the project and install requirements

git clone https://github.com/soubhiksanyal/RingNet.git
cd RingNet
pip install -r requirements.txt
pip install opendr==0.77
mkdir model

Install mesh processing libraries from MPI-IS/mesh. (This now only works with python 3, so donot install it)

  • Update: Please install the following fork for working with the mesh processing libraries with python 2.7

Download models

  • Download pretrained RingNet weights from the project website, downloads page. Copy this inside the model folder
  • Download FLAME 2019 model from here. Copy it inside the flame_model folder. This step is optional and only required if you want to use the output Flame parameters to play with the 3D mesh, i.e., to neutralize the pose and expression and only using the shape as a template for other methods like VOCA (Voice Operated Character Animation).
  • Download the FLAME_texture_data and unpack this into the flame_model folder.

Demo

RingNet requires a loose crop of the face in the image. We provide two sample images in the input_images folder which are taken from CelebA Dataset.

Output predicted mesh rendering

Run the following command from the terminal to check the predictions of RingNet

python -m demo --img_path ./input_images/000001.jpg --out_folder ./RingNet_output

Provide the image path and it will output the predictions in ./RingNet_output/images/.

Output predicted mesh

If you want the output mesh then run the following command

python -m demo --img_path ./input_images/000001.jpg --out_folder ./RingNet_output --save_obj_file=True

It will save a *.obj file of the predicted mesh in ./RingNet_output/mesh/.

Output textured mesh

If you want the output the predicted mesh with the image projected onto the mesh as texture then run the following command

python -m demo --img_path ./input_images/000001.jpg --out_folder ./RingNet_output --save_texture=True

It will save a *.obj, *.mtl, and *.png file of the predicted mesh in ./RingNet_output/texture/.

Output FLAME and camera parameters

If you want the predicted FLAME and camera parameters then run the following command

python -m demo --img_path ./input_images/000001.jpg --out_folder ./RingNet_output --save_obj_file=True --save_flame_parameters=True

It will save a *.npy file of the predicted flame and camera parameters and in ./RingNet_output/params/.

Generate VOCA templates

If you want to play with the 3D mesh, i.e. neutralize pose and expression of the 3D mesh to use it as a template in VOCA (Voice Operated Character Animation), run the following command

python -m demo --img_path ./input_images/000013.jpg --out_folder ./RingNet_output --save_obj_file=True --save_flame_parameters=True --neutralize_expression=True

License

Free for non-commercial and scientific research purposes. By using this code, you acknowledge that you have read the license terms (https://ringnet.is.tue.mpg.de/license.html), understand them, and agree to be bound by them. If you do not agree with these terms and conditions, you must not use the code. For commercial use please check the website (https://ringnet.is.tue.mpg.de/license.html).

Referencing RingNet

Please cite the following paper if you use the code directly or indirectly in your research/projects.

@inproceedings{RingNet:CVPR:2019,
title = {Learning to Regress 3D Face Shape and Expression from an Image without 3D Supervision},
author = {Sanyal, Soubhik and Bolkart, Timo and Feng, Haiwen and Black, Michael},
booktitle = {Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)},
month = jun,
year = {2019},
month_numeric = {6}
}

Contact

If you have any questions you can contact us at [email protected] and [email protected].

Acknowledgement

  • We thank Ahmed Osman for his support in the tensorflow implementation of FLAME.
  • We thank Raffi Enficiaud and Ahmed Osman for pushing the release of psbody.mesh.
  • We thank Benjamin Pellkofer and Jonathan Williams for helping with our RingNet project website.
Owner
Soubhik Sanyal
Currently Applied Scientist at Amazon Research PhD Student
Soubhik Sanyal
COIN the currently largest dataset for comprehensive instruction video analysis.

COIN Dataset COIN is the currently largest dataset for comprehensive instruction video analysis. It contains 11,827 videos of 180 different tasks (i.e

86 Dec 28, 2022
Code for EMNLP'21 paper "Types of Out-of-Distribution Texts and How to Detect Them"

ood-text-emnlp Code for EMNLP'21 paper "Types of Out-of-Distribution Texts and How to Detect Them" Files fine_tune.py is used to finetune the GPT-2 mo

Udit Arora 19 Oct 28, 2022
Sarus implementation of classical ML models. The models are implemented using the Keras API of tensorflow 2. Vizualization are implemented and can be seen in tensorboard.

Sarus published models Sarus implementation of classical ML models. The models are implemented using the Keras API of tensorflow 2. Vizualization are

Sarus Technologies 39 Aug 19, 2022
DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generative Transformers

DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generative Transformers Authors: Jaemin Cho, Abhay Zala, and Mohit Bansal (

Jaemin Cho 98 Dec 15, 2022
Complex-Valued Neural Networks (CVNN)Complex-Valued Neural Networks (CVNN)

Complex-Valued Neural Networks (CVNN) Done by @NEGU93 - J. Agustin Barrachina Using this library, the only difference with a Tensorflow code is that y

youceF 1 Nov 12, 2021
Information Gain Filtration (IGF) is a method for filtering domain-specific data during language model finetuning. IGF shows significant improvements over baseline fine-tuning without data filtration.

Information Gain Filtration Information Gain Filtration (IGF) is a method for filtering domain-specific data during language model finetuning. IGF sho

4 Jul 28, 2022
Official PyTorch implementation of the paper: Improving Graph Neural Network Expressivity via Subgraph Isomorphism Counting.

Improving Graph Neural Network Expressivity via Subgraph Isomorphism Counting Official PyTorch implementation of the paper: Improving Graph Neural Net

Giorgos Bouritsas 58 Dec 31, 2022
Reproduction of Vision Transformer in Tensorflow2. Train from scratch and Finetune.

Vision Transformer(ViT) in Tensorflow2 Tensorflow2 implementation of the Vision Transformer(ViT). This repository is for An image is worth 16x16 words

sungjun lee 42 Dec 27, 2022
Source code for our paper "Improving Empathetic Response Generation by Recognizing Emotion Cause in Conversations"

Source code for our paper "Improving Empathetic Response Generation by Recognizing Emotion Cause in Conversations" this repository is maintained by bo

Yuhan Liu 24 Nov 29, 2022
Jremesh-tools - Blender addon for quad remeshing

JRemesh Tools Blender 2.8 - 3.x addon for quad remeshing. Currently it is a wrap

Jayanam 89 Dec 30, 2022
PyTorch-centric library for evaluating and enhancing the robustness of AI technologies

Responsible AI Toolbox A library that provides high-quality, PyTorch-centric tools for evaluating and enhancing both the robustness and the explainabi

24 Dec 22, 2022
Crawl & visualize ICLR papers and reviews

Crawl and Visualize ICLR 2022 OpenReview Data Descriptions This Jupyter Notebook contains the data crawled from ICLR 2022 OpenReview webpages and thei

Federico Berto 75 Dec 05, 2022
The repository offers the official implementation of our paper in PyTorch.

Cloth Interactive Transformer (CIT) Cloth Interactive Transformer for Virtual Try-On Bin Ren1, Hao Tang1, Fanyang Meng2, Runwei Ding3, Ling Shao4, Phi

Bingoren 49 Dec 01, 2022
Train Dense Passage Retriever (DPR) with a single GPU

Gradient Cached Dense Passage Retrieval Gradient Cached Dense Passage Retrieval (GC-DPR) - is an extension of the original DPR library. We introduce G

Luyu Gao 92 Jan 02, 2023
Official code for "Decoupling Zero-Shot Semantic Segmentation"

Decoupling Zero-Shot Semantic Segmentation This is the official code for the arxiv. ZegFormer is the first framework that decouple the zero-shot seman

Jian Ding 108 Dec 30, 2022
FedML: A Research Library and Benchmark for Federated Machine Learning

FedML: A Research Library and Benchmark for Federated Machine Learning 📄 https://arxiv.org/abs/2007.13518 News 2021-02-01 (Award): #NeurIPS 2020# Fed

FedML-AI 2.3k Jan 08, 2023
Multi-Output Gaussian Process Toolkit

Multi-Output Gaussian Process Toolkit Paper - API Documentation - Tutorials & Examples The Multi-Output Gaussian Process Toolkit is a Python toolkit f

GAMES 113 Nov 25, 2022
[ICCV 2021] Official Pytorch implementation for Discriminative Region-based Multi-Label Zero-Shot Learning SOTA results on NUS-WIDE and OpenImages

Discriminative Region-based Multi-Label Zero-Shot Learning (ICCV 2021) [arXiv][Project page coming soon] Sanath Narayan*, Akshita Gupta*, Salman Kh

Akshita Gupta 54 Nov 21, 2022
Code to train models from "Paraphrastic Representations at Scale".

Paraphrastic Representations at Scale Code to train models from "Paraphrastic Representations at Scale". The code is written in Python 3.7 and require

John Wieting 71 Dec 19, 2022
A Simple Example for Imitation Learning with Dataset Aggregation (DAGGER) on Torcs Env

Imitation Learning with Dataset Aggregation (DAGGER) on Torcs Env This repository implements a simple algorithm for imitation learning: DAGGER. In thi

Hao 66 Nov 23, 2022