Sandwich Batch Normalization

Code for Sandwich Batch Normalization.

Introduction

We present Sandwich Batch Normalization (SaBN), an extremely easy improvement of Batch Normalization (BN) with only a few lines of code changes.

We demonstrate the prevailing effectiveness of SaBN as a drop-in replacement in four tasks:

1. conditional image generation,
2. neural architecture search,
3. adversarial training,
4. arbitrary neural style transfer.

Usage

Check each of them for more information:

1. GAN
2. NAS
3. Adv
4. NST

Main Results

1. Conditional Image Generation

Using SaBN in conditional generation task enables an immediate performance boost. Evaluation results on CIFAR-10 are shown below:

Visual results on ImageNet (128*128 resolution):

SNGAN	SNGAN-SaBN (ours)

2. Neural Architecture Search

We adopted DARTS as the baseline search algorithm. Results on NAS-Bench-201 are presented below:

CIFAR-100	ImageNet16-120

3. Adversarial Training

Evaluation results:

4. Arbitrary Neural Style Transfer

The model equipped with the proposed SaAdaIN achieves lower style & content loss on both training and testing set.

Training loss:

Training style loss	Training content loss

Validation loss:

Validation style loss	Validation content loss

Acknowledgement

1. NAS codebase from NAS-Bench-201.
2. NST codebase from AdaIN.