BossNAS
This repository contains PyTorch evaluation code, retraining code and pretrained models of our paper: BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised Neural Architecture Search. [pdf link]
Illustration of the Siamese supernets training with ensemble bootstrapping.
Illustration of the fabric-like Hybrid CNN-transformer Search Space with flexible down-sampling positions.
Our Results and Trained Models
-
Here is a summary of our searched models:
Model MAdds Steptime Top-1 (%) Top-5 (%) Url BossNet-T0 w/o SE 3.4B 101ms 80.5 95.0 checkpoint BossNet-T0 3.4B 115ms 80.8 95.2 checkpoint BossNet-T0^ 5.7B 147ms 81.6 95.6 same as above BossNet-T1 7.9B 156ms 81.9 95.6 checkpoint BossNet-T1^ 10.5B 165ms 82.2 95.7 same as above -
Here is a summary of architecture rating accuracy of our method:
Search space Dataset Kendall tau Spearman rho Pearson R MBConv ImageNet 0.65 0.78 0.85 NATS-Bench Ss Cifar10 0.53 0.73 0.72 NATS-Bench Ss Cifar100 0.59 0.76 0.79
Usage
1. Requirements
- Install PyTorch 1.7.0+ and torchvision 0.8.1+, for example:
conda install -c pytorch pytorch torchvision
- Install pytorch-image-models 0.3.2, for example:
pip install timm==0.3.2
-
Download the ImageNet dataset from http://image-net.org/, and move validation images to labeled subfolders
- To do this, you can use the following script: https://raw.githubusercontent.com/soumith/imagenetloader.torch/master/valprep.shvalprep.sh
2. Retrain or Evaluate our BossNet-T models
-
First, move to retraining code directory to perform Retraining or Evaluation.
cd HyTra_retrainingOur retraining code of BossNet-T is based on DeiT repository.
-
You can evaluate our BossNet-T models with the following command:
python -m torch.distributed.launch --nproc_per_node=4 --use_env main.py --model bossnet_T0 --input-size 224 --batch-size 128 --data-path /PATH/TO/ImageNet --num_workers 8 --eval --resume PATH/TO/BossNet-T0-80_8.pth
python -m torch.distributed.launch --nproc_per_node=4 --use_env main.py --model bossnet_T1 --input-size 224 --batch-size 128 --data-path /PATH/TO/ImageNet --num_workers 8 --eval --resume PATH/TO/BossNet-T1-81_9.pth
Please download our checkpoint files from the result table. Please change the
--nproc_per_nodeoption to suit your GPU numbers, and change the--data-path,--resumeand--input-sizeaccordingly. -
You can retrain our BossNet-T models with the following command:
Please change the
--nproc_per_nodeand--data-pathaccordingly. Note that the learning rate will be automatically scaled according to the GPU numbers and batchsize. We recommend training with 128 batchsize and 8 GPUs.python -m torch.distributed.launch --nproc_per_node=8 --use_env main.py --model bossnet_T0 --input-size 224 --batch-size 128 --data-path /PATH/TO/ImageNet --num_workers 8
python -m torch.distributed.launch --nproc_per_node=8 --use_env main.py --model bossnet_T1 --input-size 224 --batch-size 128 --data-path /PATH/TO/ImageNet --num_workers 8
Architecture of our BossNet-T0
3. Evaluate architecture rating accuracy of BossNAS
-
You can get the ranking correlations of BossNAS on MBConv search space with the following commands:
cd MBConv_ranking python get_model_score_mbconv.py
- You can get the ranking correlations of BossNAS on NATS-Bench Ss with the following commands:
cd NATS_SS_ranking python get_model_score_nats.py
Citation
@article{li2021bossnas,
author = {Li, Changlin and
Tang, Tao and
Wang, Guangrun and
Peng, Jiefeng and
Wang, Bing and
Liang, Xiaodan and
Chang, Xiaojun},
title = {BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised Neural Architecture Search},
journal = {arXiv:2103.12424},
year = 2021,
}
TODO
Searching code will be released later.
268 Jan 01, 2023
32 Dec 30, 2022
1.5k Dec 27, 2022
209 Jan 05, 2023
110 Jan 02, 2023
187 Oct 01, 2022
75 Dec 13, 2022
3.3k Jan 06, 2023
310 Dec 13, 2022
95 Dec 30, 2022
2 Aug 28, 2022
4.6k Jan 05, 2023
123 Dec 23, 2022
4 Dec 31, 2022
44 Dec 27, 2022
2.5k Dec 30, 2022
615 Dec 25, 2022
34 Dec 29, 2022
4 Aug 20, 2021
17 Nov 01, 2022