Unsupervised Learning of Probably Symmetric Deformable 3D Objects from Images in the Wild

Demo | Project Page | Video | Paper

Shangzhe Wu, Christian Rupprecht, Andrea Vedaldi, Visual Geometry Group, University of Oxford. In CVPR 2020 (Best Paper Award).

We propose a method to learn weakly symmetric deformable 3D object categories from raw single-view images, without ground-truth 3D, multiple views, 2D/3D keypoints, prior shape models or any other supervision.

Setup (with Anaconda)

1. Install dependencies:

conda env create -f environment.yml

OR manually:

conda install -c conda-forge scikit-image matplotlib opencv moviepy pyyaml tensorboardX

2. Install PyTorch:

conda install pytorch==1.2.0 torchvision==0.4.0 cudatoolkit=9.2 -c pytorch

Note: The code is tested with PyTorch 1.2.0 and CUDA 9.2 on CentOS 7. A GPU version is required for training and testing, since the neural_renderer package only has GPU implementation. You are still able to run the demo without GPU.

3. Install neural_renderer:

This package is required for training and testing, and optional for the demo. It requires a GPU device and GPU-enabled PyTorch.

pip install neural_renderer_pytorch

Note: It may fail if you have a GCC version below 5. If you do not want to upgrade your GCC, one alternative solution is to use conda's GCC and compile the package from source. For example:

conda install gxx_linux-64=7.3
git clone https://github.com/daniilidis-group/neural_renderer.git
cd neural_renderer
python setup.py install

4. (For demo only) Install facenet-pytorch:

This package is optional for the demo. It allows automatic human face detection.

pip install facenet-pytorch

Datasets

1. CelebA face dataset. Please download the original images (img_celeba.7z) from their website and run celeba_crop.py in data/ to crop the images.
2. Synthetic face dataset generated using Basel Face Model. This can be downloaded using the script download_synface.sh provided in data/.
3. Cat face dataset composed of Cat Head Dataset and Oxford-IIIT Pet Dataset (license). This can be downloaded using the script download_cat.sh provided in data/.
4. Synthetic car dataset generated from ShapeNet cars. The images are rendered from with random viewpoints from the top, where the cars are primarily oriented vertically. This can be downloaded using the script download_syncar.sh provided in data/.

Please remember to cite the corresponding papers if you use these datasets.

Pretrained Models

Download pretrained models using the scripts provided in pretrained/, eg:

cd pretrained && sh download_pretrained_celeba.sh

Demo

python -m demo.demo --input demo/images/human_face --result demo/results/human_face --checkpoint pretrained/pretrained_celeba/checkpoint030.pth

Options:

• --gpu: enable GPU
• --detect_human_face: enable automatic human face detection and cropping using MTCNN provided in facenet-pytorch. This only works on human face images. You will need to manually crop the images for other objects.
• --render_video: render 3D animations using neural_renderer (GPU is required)

Training and Testing

Check the configuration files in experiments/ and run experiments, eg:

python run.py --config experiments/train_celeba.yml --gpu 0 --num_workers 4

Citation

@InProceedings{Wu_2020_CVPR,
  author = {Shangzhe Wu and Christian Rupprecht and Andrea Vedaldi},
  title = {Unsupervised Learning of Probably Symmetric Deformable 3D Objects from Images in the Wild},
  booktitle = {CVPR},
  year = {2020}
}