Test-Agnostic Long-Tailed Recognition

This repository is the official Pytorch implementation of Test-Agnostic Long-Tailed Recognition by Test-Time Aggregating Diverse Experts with Self-Supervision.

• TADE (our method) innovates the expert training scheme by introducing diversity-promoting expertise-guided losses, which train different experts to handle distinct class distributions. In this way, the learned experts would be more diverse than existing multi-expert methods, leading to better ensemble performance, and aggregatedly simulate a wide spectrum of possible class distributions.
• TADE develops a new self-supervised method, namely prediction stability maximization, to adaptively aggregate these experts for better handling unknown test distribution, using unlabeled test class data.

Results

ImageNet-LT (ResNeXt-50)

Long-tailed recognition with uniform test class distribution:

Test-agnostic long-tailed recognition:

CIFAR100-Imbalance ratio 100 (ResNet-32)

Long-tailed recognition with uniform test class distribution：

Test-agnostic long-tailed recognition：

Places-LT (ResNet-152)

Long-tailed recognition with uniform test class distribution:

Test-agnostic long-tailed recognition:

iNaturalist 2018 (ResNet-50)

Long-tailed recognition with uniform test class distribution:

Test-agnostic long-tailed recognition:

Requirements

• To install requirements:

pip install -r requirements.txt

Hardware requirements

8 GPUs with >= 11G GPU RAM are recommended. Otherwise the model with more experts may not fit in, especially on datasets with more classes (the FC layers will be large). We do not support CPU training, but CPU inference could be supported by slight modification.

Datasets

Four bechmark datasets

• Please download these datasets and put them to the /data file.
• ImageNet-LT and Places-LT can be found at here.
• iNaturalist data should be the 2018 version from here.
• CIFAR-100 will be downloaded automatically with the dataloader.

data
├── ImageNet_LT
│   ├── test
│   ├── train
│   └── val
├── CIFAR100
│   └── cifar-100-python
├── Place365
│   ├── data_256
│   ├── test_256
│   └── val_256
└── iNaturalist 
    ├── test2018
    └── train_val2018

Txt files

• We provide txt files for test-agnostic long-tailed recognition for ImageNet-LT, Places-LT and iNaturalist 2018. CIFAR-100 will be generated automatically with the code.
• For iNaturalist 2018, please unzip the iNaturalist_train.zip.

data_txt
├── ImageNet_LT
│   ├── ImageNet_LT_backward2.txt
│   ├── ImageNet_LT_backward5.txt
│   ├── ImageNet_LT_backward10.txt
│   ├── ImageNet_LT_backward25.txt
│   ├── ImageNet_LT_backward50.txt
│   ├── ImageNet_LT_forward2.txt
│   ├── ImageNet_LT_forward5.txt
│   ├── ImageNet_LT_forward10.txt
│   ├── ImageNet_LT_forward25.txt
│   ├── ImageNet_LT_forward50.txt
│   ├── ImageNet_LT_test.txt
│   ├── ImageNet_LT_train.txt
│   ├── ImageNet_LT_uniform.txt
│   └── ImageNet_LT_val.txt
├── Places_LT_v2
│   ├── Places_LT_backward2.txt
│   ├── Places_LT_backward5.txt
│   ├── Places_LT_backward10.txt
│   ├── Places_LT_backward25.txt
│   ├── Places_LT_backward50.txt
│   ├── Places_LT_forward2.txt
│   ├── Places_LT_forward5.txt
│   ├── Places_LT_forward10.txt
│   ├── Places_LT_forward25.txt
│   ├── Places_LT_forward50.txt
│   ├── Places_LT_test.txt
│   ├── Places_LT_train.txt
│   ├── Places_LT_uniform.txt
│   └── Places_LT_val.txt
└── iNaturalist18
    ├── iNaturalist18_backward2.txt
    ├── iNaturalist18_backward3.txt
    ├── iNaturalist18_forward2.txt
    ├── iNaturalist18_forward3.txt
    ├── iNaturalist18_train.txt
    ├── iNaturalist18_uniform.txt
    └── iNaturalist18_val.txt 

Pretrained models

• For the training on Places-LT, we follow previous method and use the pre-trained model.
• Please download the checkpoint. Unzip and move the checkpoint files to /model/pretrained_model_places/.

Script

ImageNet-LT

Training

• To train the expertise-diverse model, run this command:

python train.py -c configs/config_imagenet_lt_resnext50_tade.json

Evaluate

• To evaluate expertise-diverse model on the uniform test class distribution, run:

python test.py -r checkpoint_path

• To evaluate expertise-diverse model on agnostic test class distributions, run:

python test_all_imagenet.py -r checkpoint_path

Test-time training

• To test-time train the expertise-diverse model for agnostic test class distributions, run:

python test_train_imagenet.py -c configs/test_time_imagenet_lt_resnext50_tade.json -r checkpoint_path

CIFAR100-LT

Training

• To train the expertise-diverse model, run this command:

python train.py -c configs/config_cifar100_ir100_tade.json

• One can change the imbalance ratio from 100 to 10/50 by changing the config file.

Evaluate

• To evaluate expertise-diverse model on the uniform test class distribution, run:

python test.py -r checkpoint_path

• To evaluate expertise-diverse model on agnostic test class distributions, run:

python test_all_cifar.py -r checkpoint_path

Test-time training

• To test-time train the expertise-diverse model for agnostic test class distributions, run:

python test_train_cifar.py -c configs/test_time_cifar100_ir100_tade.json -r checkpoint_path

• One can change the imbalance ratio from 100 to 10/50 by changing the config file.

Places-LT

Training

• To train the expertise-diverse model, run this command:

python train.py -c configs/config_places_lt_resnet152_tade.json

Evaluate

• To evaluate expertise-diverse model on the uniform test class distribution, run:

python test_places.py -r checkpoint_path

• To evaluate expertise-diverse model on agnostic test class distributions, run:

python test_all_places.py -r checkpoint_path

Test-time training

• To test-time train the expertise-diverse model for agnostic test class distributions, run:

python test_train_places.py -c configs/test_time_places_lt_resnet152_tade.json -r checkpoint_path

iNaturalist 2018

Training

• To train the expertise-diverse model, run this command:

python train.py -c configs/config_iNaturalist_resnet50_tade.json

Evaluate

• To evaluate expertise-diverse model on the uniform test class distribution, run:

python test.py -r checkpoint_path

• To evaluate expertise-diverse model on agnostic test class distributions, run:

python test_all_inat.py -r checkpoint_path

Test-time training

• To test-time train the expertise-diverse model for agnostic test class distributions, run:

python test_train_inat.py -c configs/test_time_iNaturalist_resnet50_tade.json -r checkpoint_path

Citation

If you find our work inspiring or use our codebase in your research, please cite our work.

@article{zhang2021test,
  title={Test-Agnostic Long-Tailed Recognition by Test-Time Aggregating Diverse Experts with Self-Supervision},
  author={Zhang, Yifan and Hooi, Bryan and Hong, Lanqing and Feng, Jiashi},
  journal={arXiv},
  year={2021}
}

Acknowledgements

This is a project based on this pytorch template.

The mutli-expert framework are based on RIDE. The data generation of agnostic test class distributions takes references from LADE.