Code for "Reconstructing 3D Human Pose by Watching Humans in the Mirror", CVPR 2021 oral

Overview

Reconstructing 3D Human Pose by Watching Humans in the Mirror

report
Qi Fang*, Qing Shuai*, Junting Dong, Hujun Bao, Xiaowei Zhou
CVPR 2021 Oral


The videos are from Youtube and Douyin. Please contact us for any copyright issue.

News

  • We build a website for a fast preview of our dataset. The whole dataset will be released later.

Features

In this paper, we introduce the new task of reconstructing 3D human pose from a single image in which we can see the person and the person’s image through a mirror.

This implementation:

  • has the demo of our optimization-based approach implemented purely in PyTorch.
  • provides a method to estimate the surface normal of the mirror from vanishing points.
  • provides an annotator to label the mirror edges for the vanishing points.
  • can estimate the focal length of the Internet mirror images.

Installation

This repo has a close relation with EasyMocap. Please refer to our EasyMocap project for installation.

Demo

Download our zju-m-test.zip and run the following code:

# set the data path
data=<path_to_sample>/zju-m-demo
out=<path_to_sample>/zju-m-demo-output
# extract the video frames
python3 scripts/preprocess/extract_video.py ${data}
# Run demo on videos
python3 apps/demo/1v1p_mirror.py ${data} --out ${out} --vis_smpl --video

Mirrored-Human Dataset (Coming Soon)

Due to the license limitation, we cannot share the raw data directly. We are working hard to organize the Mirrored-Human dataset in terms of url links and timestamps.

See Build Your Internet Dataset if you can't wait for our release.

Annotator

We also provide the annotator metioned in our paper.

The first row shows that we label the edges of the mirror and calculate the vanishing point by the human body automaticly. The intrisic camera parameter can be calculated by this two vanishing points.

The second row shows that to obtain a more accurate vanishing points and camera parameters, we can label the parallel lines in the scene, for example the door, the grid in the ground, and the door.


See EasyMocap/apps/annotator for more instructions.

Build Custom Internet Dataset

See doc/internet.md for more instructions.

Build Custom Evaluation Dataset (Multi-View)

This part is provided for the researchers who want to:

  1. capture more accurate human motion with multiple cameras and a mirror
  2. build a different evaluation dataset

See doc/custom.md for more instructions.

Evaluation

To evaluate the reconstruction part in our paper, see doc/evaluation.md.

Contact

Please open an issue if you have any questions. We appreciate all contributions to improve our project.

If you find some videos that we didn't notice, please tell us.

Citation

@inproceedings{fang2021mirrored,
  title={Reconstructing 3D Human Pose by Watching Humans in the Mirror},
  author={Fang, Qi and Shuai, Qing and Dong, Junting and Bao, Hujun and Zhou, Xiaowei},
  booktitle={CVPR},
  year={2021}
}

Acknowledgement

This project is build on our EasyMocap. We also would like to thank Jianan Zhen and Yuhao Chen for their advice for the paper. Sincere thanks to the performers (Yuji Chen and Hao Xu) in the evaluation dataset and people who uploaded the mirror-human videos to the Internet.

Recommendations to other works from our group

Welcome to checkout our work on learning-based feature matching (LoFTR) and reconstruction (NeuralBody and NeuralRecon) in CVPR 2021.

Owner
ZJU3DV
ZJU3DV is a research group of State Key Lab of CAD&CG, Zhejiang University. We focus on the research of 3D computer vision, SLAM and AR.
ZJU3DV
TRIQ implementation

TRIQ Implementation TF-Keras implementation of TRIQ as described in Transformer for Image Quality Assessment. Installation Clone this repository. Inst

Junyong You 115 Dec 30, 2022
Supplementary materials for ISMIR 2021 LBD paper "Evaluation of Latent Space Disentanglement in the Presence of Interdependent Attributes"

Evaluation of Latent Space Disentanglement in the Presence of Interdependent Attributes Supplementary materials for ISMIR 2021 LBD submission: K. N. W

Karn Watcharasupat 2 Oct 25, 2021
Learn about Spice.ai with in-depth samples

Samples Learn about Spice.ai with in-depth samples ServerOps - Learn when to run server maintainance during periods of low load Gardener - Intelligent

Spice.ai 16 Mar 23, 2022
Users can free try their models on SIDD dataset based on this code

SIDD benchmark 1 Train python train.py If you want to train your network, just modify the yaml in the options folder. 2 Validation python validation.p

Yuzhi ZHAO 2 May 20, 2022
This is the repository of the NeurIPS 2021 paper "Curriculum Disentangled Recommendation withNoisy Multi-feedback"

Curriculum_disentangled_recommendation This is the repository of the NeurIPS 2021 paper "Curriculum Disentangled Recommendation with Noisy Multi-feedb

14 Dec 20, 2022
Aiming at the common training datsets split, spectrum preprocessing, wavelength select and calibration models algorithm involved in the spectral analysis process

Aiming at the common training datsets split, spectrum preprocessing, wavelength select and calibration models algorithm involved in the spectral analysis process, a complete algorithm library is esta

Fu Pengyou 50 Jan 07, 2023
(JMLR'19) A Python Toolbox for Scalable Outlier Detection (Anomaly Detection)

Python Outlier Detection (PyOD) Deployment & Documentation & Stats Build Status & Coverage & Maintainability & License PyOD is a comprehensive and sca

Yue Zhao 6.6k Jan 03, 2023
EdMIPS: Rethinking Differentiable Search for Mixed-Precision Neural Networks

EdMIPS is an efficient algorithm to search the optimal mixed-precision neural network directly without proxy task on ImageNet given computation budgets. It can be applied to many popular network arch

Zhaowei Cai 47 Dec 30, 2022
Utilities to bridge Canvas-generated course rosters with GitLab's API.

gitlab-canvas-utils A collection of scripts originally written for CSE 13S. Oversees everything from GitLab course group creation, student repository

Eugene Chou 5 Jun 08, 2022
NICE-GAN — Official PyTorch Implementation Reusing Discriminators for Encoding: Towards Unsupervised Image-to-Image Translation

NICE-GAN-pytorch - Official PyTorch implementation of NICE-GAN: Reusing Discriminators for Encoding: Towards Unsupervised Image-to-Image Translation

Runfa Chen 208 Nov 25, 2022
Application of the L2HMC algorithm to simulations in lattice QCD.

l2hmc-qcd 📊 Slides Recent talk on Training Topological Samplers for Lattice Gauge Theory from the Machine Learning for High Energy Physics, on and of

Sam Foreman 37 Dec 14, 2022
Tech Resources for Academic Communities

Free tech resources for faculty, students, researchers, life-long learners, and academic community builders for use in tech based courses, workshops, and hackathons.

Microsoft 2.5k Jan 04, 2023
Real-world Anomaly Detection in Surveillance Videos- pytorch Re-implementation

Real world Anomaly Detection in Surveillance Videos : Pytorch RE-Implementation This repository is a re-implementation of "Real-world Anomaly Detectio

seominseok 62 Dec 08, 2022
Graph Attention Networks

GAT Graph Attention Networks (Veličković et al., ICLR 2018): https://arxiv.org/abs/1710.10903 GAT layer t-SNE + Attention coefficients on Cora Overvie

Petar Veličković 2.6k Jan 05, 2023
PyContinual (An Easy and Extendible Framework for Continual Learning)

PyContinual (An Easy and Extendible Framework for Continual Learning) Easy to Use You can sumply change the baseline, backbone and task, and then read

Zixuan Ke 176 Jan 05, 2023
Agile SVG maker for python

Agile SVG Maker Need to draw hundreds of frames for a GIF? Need to change the style of all pictures in a PPT? Need to draw similar images with differe

SemiWaker 4 Sep 25, 2022
Video-face-extractor - Video face extractor with Python

Python face extractor Setup Create the srcvideos and faces directories Put your

2 Feb 03, 2022
PyTorch implementation of "PatchGame: Learning to Signal Mid-level Patches in Referential Games" to appear in NeurIPS 2021

PatchGame: Learning to Signal Mid-level Patches in Referential Games This repository is the official implementation of the paper - "PatchGame: Learnin

Kamal Gupta 22 Mar 16, 2022
Pca-on-genotypes - Mini bioinformatics project - PCA on genotypes

Mini bioinformatics project: PCA on genotypes This repo contains the code from t

Maria Nattestad 8 Dec 04, 2022
GeoTransformer - Geometric Transformer for Fast and Robust Point Cloud Registration

Geometric Transformer for Fast and Robust Point Cloud Registration PyTorch imple

Zheng Qin 220 Jan 05, 2023