3D-aware GANs based on NeRF (arXiv).

Hi guy, There is an issue CUDA error: out of memory (even with batch size = 1) when I try to run training script with this command CUDA_VISIBLE_DEVICES=2 python -c "import sys; sys.path.append('./'); from exp.tests.test_cips3d import Testing_ffhq_exp; Testing_ffhq_exp().test_train_ffhq(debug=False)" --tl_opts batch_size 1 img_size 32 total_iters 80000

I try to run on V100 GPU with 32Gb mem. What should I do? Btw, really appreciate your work, a great paper. 👏

Hi, PeterouZh,

I'm reproducing your results at the same pace with you. Honestly speaking, this model takes about 40 hours to reach 64x64 at FID 15.97 with 8 A100 gpus. While I change the resolution to 128x128, the FID reach to 23.58. I'm still traning it and it reach FID 20.03 yet.

How can this model reach FID 6.XX as you described in paper? Do we miss some key things? It looks that this model can only reach 10+ FID in 256 resolutions because the performance increases very lowly when the FID reach 16 at 64x64 resolution.

By the way, I try to reproduce your results few weeks ago but I met problems about moxing. Does moxing provide very important tricks for this work?

Hello,

Thanks for sharing your source code and pre-trained weights. I am trying to generate high-quality images from FFHQ pre-trained model. However, the quality of generated images is not as good enough as stated in the paper. I could not reproduce the results.

I am using the pre-trained weights from here https://github.com/PeterouZh/CIPS-3D/releases/tag/v0.0.2

The command I tried: python exp/cips3d/scripts/sample_images.py --tl_config_file exp/cips3d/configs/ffhq_exp.yaml --tl_command sample_images

Generated images:

Do you have any idea regarding the problem?

Hi! You did a great job, thanks for such a great paper and promptly published CIPS-3D code.

I've already gotten good results with your pipeline, but for images with resolution 64x64. Now I'm waiting the results of generating images with a resolution of 128x128. And I will further train for higher resolution images.

I understand correctly, in order to get 512x512 images, I need to convert the original FFHQ dataset once again through your script dataset_tool.py, but specifying the resize for 512 in it? And after I need to run training pipeline with lower values for generator learning rate and discriminator learning rate?

Thanks!

I want test some other image on your model. But I dont konw how to do it. If I have image sequence with pose data，how to test？

1. Align the images in the way of StyleGAN. You can refer to this script align_images.py.
2. Project the aligned images into the W space, also known as GAN inversion. Different from the common 2D inversion, you'd better set an appropriate yaw/pitch/fov for the CIPS-3D generator to make the initial pose of G(w) and the image to be inverted consistent.
3. After you get the w of the image, you can reconstruct images of different styles using G'(w). G' can be obtained by interpolating generators of different domains.

Hope this helps.

Originally posted by @PeterouZh in https://github.com/PeterouZh/CIPS-3D/issues/7#issuecomment-963163677

Hi,good job!

I have a problem, please help me.

pip install -e torch_fidelity_lib ERROR: File "setup.py" or "setup.cfg" not found. Directory cannot be installed in editable mode: /media/sdb/wd/test_code/CIPS-3D/torch_fidelity_lib

I load the FFHQ pretrained model from Pre-trained checkpoints. And change the finetune_dir as Pre-trained checkpoints in finetune_photo2cartoon.sh. But it seems not to work. I want to know if the pre-trained model can be used in finetune_photo2cartoon.sh?

Dear Dr.Zhou, Thanks for sharing your great job and congratulations on your graduating Ph.D ! I have a few questions and hoping for your reply.

1、I found a command in another issue https://github.com/PeterouZh/CIPS-3D/issues/31#issue-1196645855 python exp/cips3d/scripts/sample_images.py --tl_config_file exp/cips3d/configs/ffhq_exp.yaml --tl_command sample_images But I can't find those arguments in sample_images.py and confuse about why he knows how to use. And I also found some packages import from tl2 library, but failed to find any documentation. I wonder if there are any instruction i miss in addition to README. 2、I saw two generators file in /CIPS-3D/exp/cips3d/models generator.py and generator_v1.py, which one should I use ? 3、Which class in generator files indicates the complete generator module cause I want to do some inversion tests and not sure whether it's class GeneratorNerfINR ? And the G_ema.pth or generator.pth in ckpt is the corresponding parametors to the generator which I can directly load into, am i right? 4、What is the use of state_dict.pth in ckpt.

By the way, I think using Chinese is more convenience for us. Thanks!

Hi there,

I try to output the image with the gradient. However, I found that if I use your default testing code, it will call whole_grad_forward (https://github.com/PeterouZh/CIPS-3D/blob/aee40251a02c34e58d3002bcb845151c41b538f0/exp/dev/nerf_inr/models/generator_nerf_inr_v16.py#L1395), and will remove the gradient. If I comment out the torch.no_grad(), it would be out of memory. Is there a way to output the image with gradient? Thanks

Hi,

Thanks for the great work. I am trying to inverse the image into w/z using the pretrained model. So would you release the pretrained discriminator to enable the inversion feature? Thanks

I know nerf is a view-dependent synthesis method due to a direction input. However, in your code. I find you don't use it. Why can cips3d still work? just input the world coordinate can achieve new view synthesis? why?

您好，感谢您的开源代码。

在Readme中您有说明，生成高分辨率时的训练流程是32->64->128->256, 每次训练都基于前一分辨率得到的model进行finetune。 这样的训练策略的确会比直接训练要容易得多，那请问您试过直接训练256分辨率吗，调整训练参数是否也能得到类似的效果？

How to view G model effects？

run web_demo.py like this ， web only display 3 same pictures，1picture display nothing（be black）.

web_demo.py like below :

Hello, thank you for your contribution. I try to run your web_ Demo. I saw you say"Thus current stylization is limited to randomly generated images. To edit a real image, we need to project the image to the latent space of the generator. ".So I can't import other face images to produce the effect like the demo-video? Thank you.

Thanks for your excellent work. I am curious why you set the ray_near and ray_end to 0.88 and 1.12? (and for other variables like h_stddev etc.) Is that set empirically?

Hi, would you be interested in adding CIPS-3D to Hugging Face? The Hub offers free hosting, and it would make your work more accessible and visible to the rest of the ML community.

Example from other organizations: Keras: https://huggingface.co/keras-io Microsoft: https://huggingface.co/microsoft Facebook: https://huggingface.co/facebook

Example spaces with repos: github: https://github.com/salesforce/BLIP Spaces: https://huggingface.co/spaces/salesforce/BLIP

github: https://github.com/facebookresearch/omnivore Spaces: https://huggingface.co/spaces/akhaliq/omnivore

and here are guides for adding spaces/models/datasets to your org

How to add a Space: https://huggingface.co/blog/gradio-spaces how to add models: https://huggingface.co/docs/hub/adding-a-model uploading a dataset: https://huggingface.co/docs/datasets/upload_dataset.html

Please let us know if you would be interested and if you have any questions, we can also help with the technical implementation.