[CVPR 2021] Pytorch implementation of Hijack-GAN: Unintended-Use of Pretrained, Black-Box GANs

Last update: Sep 05, 2022

Overview

Hijack-GAN: Unintended-Use of Pretrained, Black-Box GANs

In this work, we propose a framework HijackGAN, which enables non-linear latent space traversal and gain high-level controls, e.g., attributes, head poses, and landmarks, over unconditional image generation GANs in a fully black-box setting. It opens up the possibility of reusing GANs while raising concerns about unintended usage.

[Paper (CVPR 2021)][Project Page]

Prerequisites

Install required packages

pip install -r requirements.txt

Download pretrained GANs

Download the CelebAHQ pretrained weights of ProgressiveGAN [paper][code] and StyleGAN [paper][code], and then put those weights in ./models/pretrain. For example,

pretrain/
├── Pretrained_Models_Should_Be_Placed_Here
├── karras2018iclr-celebahq-1024x1024.pkl
├── karras2019stylegan-celebahq-1024x1024.pkl
├── pggan_celebahq_z.pt
├── stylegan_celebahq_z.pt
├── stylegan_headpose_z_dp.pt
└── stylegan_landmark_z.pt

Quick Start

Specify number of images to edit, a model to generate images, some parameters for editting.

LATENT_CODE_NUM=1
python edit.py \
    -m pggan_celebahq \
    -b boundaries/ \
    -n "$LATENT_CODE_NUM" \
    -o results/stylegan_celebahq_eyeglasses \
    --step_size 0.2 \
    --steps 40 \
    --attr_index 0 \
    --task attribute \
    --method ours

Usage

Important: For different given images (initial points), different step size and steps may be considered. In the following examples, we provide the parameters used in our paper. One could adjust them for better performance.

Specify Number of Samples

LATENT_CODE_NUM=1

Unconditional Modification

python edit.py \
    -m pggan_celebahq \
    -b boundaries/ \
    -n "$LATENT_CODE_NUM" \
    -o results/stylegan_celebahq_smile_editing \
    --step_size 0.2 \
    --steps 40 \
    --attr_index 0\
    --task attribute

Conditional Modification

python edit.py \
    -m pggan_celebahq \
    -b boundaries/ \
    -n "$LATENT_CODE_NUM" \
    -o results/stylegan_celebahq_smile_editing \
    --step_size 0.2 \
    --steps 40 \
    --attr_index 0\
    --condition\
    -i codes/pggan_cond/age.npy
    --task attribute

Head pose

Pitch

python edit.py \
    -m stylegan_celebahq \
    -b boundaries/ \
    -n "$LATENT_CODE_NUM" \
    -o results/ \
    --task head_pose \
    --method ours \
    --step_size 0.01 \
    --steps 2000 \
    --attr_index 1\
    --condition\
    --direction -1 \
    --demo

Yaw

python edit.py \
    -m stylegan_celebahq \
    -b boundaries/ \
    -n "$LATENT_CODE_NUM" \
    -o results/ \
    --task head_pose \
    --method ours \
    --step_size 0.1 \
    --steps 200 \
    --attr_index 0\
    --condition\
    --direction 1\
    --demo

Landmarks

Parameters for reference: (attr_index, step_size, steps) (4: 0.005 400) (5: 0.01 100), (6: 0.1 200), (8 0.1 200)

CUDA_VISIBLE_DEVICES=0 python edit.py \
    -m stylegan_celebahq \
    -b boundaries/ \
    -n "$LATENT_CODE_NUM" \
    -o results/ \
    --task landmark \
    --method ours \
    --step_size 0.1 \
    --steps 200 \
    --attr_index 6\
    --condition\
    --direction 1 \
    --demo

Generate Balanced Data

This a templeate showing how we generated balanced data for attribute manipulation (16 attributes in our internal experiments). You can modify it to fit your task better. Please first refer to here and replace YOUR_TASK_MODEL with your own classification model, and then run:

NUM=500000
CUDA_VISIBLE_DEVICES=0 python generate_balanced_data.py -m stylegan_celebahq \
    -o ./generated_data -K ./generated_data/indices.pkl -n "$NUM" -SI 0 --no_generated_imgs

[CVPR 2021] Pytorch implementation of Hijack-GAN: Unintended-Use of Pretrained, Black-Box GANs

Related tags

Overview

Hijack-GAN: Unintended-Use of Pretrained, Black-Box GANs

Prerequisites

Install required packages

Download pretrained GANs

Quick Start

Usage

Specify Number of Samples

Unconditional Modification

Conditional Modification

Head pose

Pitch

Yaw

Landmarks

Generate Balanced Data

Evaluations

TO-DO

Acknowledgment

Owner

Hui-Po Wang

Layered Neural Atlases for Consistent Video Editing

A Number Recognition algorithm

Source code of all the projects of Udacity Self-Driving Car Engineer Nanodegree.

Offical implementation for "Trash or Treasure? An Interactive Dual-Stream Strategy for Single Image Reflection Separation".

We envision models that are pre-trained on a vast range of domain-relevant tasks to become key for molecule property prediction

Code for our ALiBi method for transformer language models.

This is a simple framework to make object detection dataset very quickly

🔥 Real-time Super Resolution enhancement (4x) with content loss and relativistic adversarial optimization 🔥

Beancount-mercury - Beancount importer for Mercury Startup Checking

An implementation of IMLE-Net: An Interpretable Multi-level Multi-channel Model for ECG Classification

Official repository with code and data accompanying the NAACL 2021 paper "Hurdles to Progress in Long-form Question Answering" (https://arxiv.org/abs/2103.06332).

ByteTrack: Multi-Object Tracking by Associating Every Detection Box

Locally Most Powerful Bayesian Test for Out-of-Distribution Detection using Deep Generative Models

Picasso: a methods for embedding points in 2D in a way that respects distances while fitting a user-specified shape.

Official repository of OFA. Paper: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework

Official repository for the ICLR 2021 paper Evaluating the Disentanglement of Deep Generative Models with Manifold Topology

CLOCs: Camera-LiDAR Object Candidates Fusion for 3D Object Detection

Point Cloud Denoising input segmentation output raw point-cloud valid/clear fog rain de-noised Abstract Lidar sensors are frequently used in environme

Morphable Detector for Object Detection on Demand

Code base for reproducing results of I.Schubert, D.Driess, O.Oguz, and M.Toussaint: Learning to Execute: Efficient Learning of Universal Plan-Conditioned Policies in Robotics. NeurIPS (2021)