PyTorch implementation for "Sharpness-aware Quantization for Deep Neural Networks".

Last update: Dec 19, 2022

Related tags

Deep Learning SAQ

Overview

Sharpness-aware Quantization for Deep Neural Networks

Recent Update

2021.11.23: We release the source code of SAQ.

Setup the environments

Clone the repository locally:

git clone https://github.com/zhuang-group/SAQ

Install pytorch 1.8+, tensorboard and prettytable

conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch
pip install tensorboard
pip install prettytable

Data preparation

ImageNet

Download the ImageNet 2012 dataset from here, and prepare the dataset based on this script.
Change the dataset path in link_imagenet.py and link the ImageNet-100 by

python link_imagenet.py

CIFAR-100

Download the CIFAR-100 dataset from here.

After downloading ImageNet and CIFAR-100, the file structure should look like:

dataset
├── imagenet
    ├── train
    │   ├── class1
    │   │   ├── img1.jpeg
    │   │   ├── img2.jpeg
    │   │   └── ...
    │   ├── class2
    │   │   ├── img3.jpeg
    │   │   └── ...
    │   └── ...
    └── val
        ├── class1
        │   ├── img4.jpeg
        │   ├── img5.jpeg
        │   └── ...
        ├── class2
        │   ├── img6.jpeg
        │   └── ...
        └── ...
├── cifar100
    ├── cifar-100-python
    │   ├── meta
    │   ├── test
    │   ├── train
    │   └── ...
    └── ...

Training

Fixed-precision quantization

Download the pre-trained full-precision models from the model zoo.
Train low-precision models.

To train low-precision ResNet-20 on CIFAR-100, run:

sh script/train_qsam_cifar_r20.sh

To train low-precision ResNet-18 on ImageNet, run:

sh script/train_qsam_imagenet_r18.sh

Mixed-precision quantization

Download the pre-trained full-precision models from the model zoo.
Train the configuration generator.

To train the configuration generator of ResNet-20 on CIFAR-100, run:

sh script/train_generator_cifar_r20.sh

To train the configuration generator on ImageNet, run:

sh script/train_generator_imagenet_r18.sh

After training the configuration generator, run following commands to fine-tune the resulting models with the obtained bitwidth configurations on CIFAR-100 and ImageNet.

sh script/finetune_cifar_r20.sh

sh script/finetune_imagenet_r18.sh

Results on CIFAR-100

Network	Method	Bitwidth	BOPs (M)	Top-1 Acc. (%)	Top-5 Acc. (%)
ResNet-20	SAQ	4	674.6	68.7	91.2
ResNet-20	SAMQ	MP	659.3	68.7	91.2
ResNet-20	SAQ	3	392.1	67.7	90.8
ResNet-20	SAMQ	MP	374.4	68.6	91.2
MobileNetV2	SAQ	4	1508.9	75.6	93.7
MobileNetV2	SAMQ	MP	1482.1	75.5	93.6
MobileNetV2	SAQ	3	877.1	74.4	93.2
MobileNetV2	SAMQ	MP	869.5	75.5	93.7

Results on ImageNet

Network	Method	Bitwidth	BOPs (G)	Top-1 Acc. (%)	Top-5 Acc. (%)
ResNet-18	SAQ	4	34.7	71.3	90.0
ResNet-18	SAMQ	MP	33.7	71.4	89.9
ResNet-18	SAQ	2	14.4	67.1	87.3
MobileNetV2	SAQ	4	5.3	70.2	89.4
MobileNetV2	SAMQ	MP	5.3	70.3	89.4

License

This repository is released under the Apache 2.0 license as found in the LICENSE file.

Acknowledgement

This repository has adopted codes from SAM, ASAM and ESAM, we thank the authors for their open-sourced code.

You might also like...

Objective of the repository is to learn and build machine learning models using Pytorch. 30DaysofML Using Pytorch

30 Days Of Machine Learning Using Pytorch Objective of the repository is to learn and build machine learning models using Pytorch. List of Algorithms

119 Nov 24, 2022

Pretrained SOTA Deep Learning models, callbacks and more for research and production with PyTorch Lightning and PyTorch

1.4k Jan 1, 2023

Amazon Forest Computer Vision: Satellite Image tagging code using PyTorch / Keras with lots of PyTorch tricks

Amazon Forest Computer Vision Satellite Image tagging code using PyTorch / Keras Here is a sample of images we had to work with Source: https://www.ka

360 Dec 10, 2022

The Incredible PyTorch: a curated list of tutorials, papers, projects, communities and more relating to PyTorch.

This is a curated list of tutorials, projects, libraries, videos, papers, books and anything related to the incredible PyTorch. Feel free to make a pu

9.2k Jan 2, 2023

Amazon Forest Computer Vision: Satellite Image tagging code using PyTorch / Keras with lots of PyTorch tricks

Amazon Forest Computer Vision Satellite Image tagging code using PyTorch / Keras Here is a sample of images we had to work with Source: https://www.ka

359 Jan 5, 2023

A bunch of random PyTorch models using PyTorch's C++ frontend

PyTorch Deep Learning Models using the C++ frontend Gettting started Clone the repo 1. https://github.com/mrdvince/pytorchcpp 2. cd fashionmnist or

0 Jul 13, 2021

PyTorch Autoencoders - Implementing a Variational Autoencoder (VAE) Series in Pytorch.

PyTorch Autoencoders Implementing a Variational Autoencoder (VAE) Series in Pytorch. Inspired by this repository Model List check model paper conferen

8 Nov 21, 2022

PyTorch-LIT is the Lite Inference Toolkit (LIT) for PyTorch which focuses on easy and fast inference of large models on end-devices.

PyTorch-LIT PyTorch-LIT is the Lite Inference Toolkit (LIT) for PyTorch which focuses on easy and fast inference of large models on end-devices. With

157 Dec 11, 2022

A general framework for deep learning experiments under PyTorch based on pytorch-lightning

torchx Torchx is a general framework for deep learning experiments under PyTorch based on pytorch-lightning. TODO list gan-like training wrapper text

6 Mar 17, 2022

Comments

Quantize_first_last_layer

Hi! I noticed that in your code, you set bits_weights=8 and bits_activations=32 for first layer as default, it's not what is claimed in your paper " For the first and last layers of all quantized models, we quantize both weights and activations to 8-bit. " And I see an accuracy drop if I adjust the bits_activations to 8 for the first layer, could u please explain what is the reason? Thanks!

opened by mmmiiinnnggg 0
代码问题请求帮助

你好，带佬的代码写的很好，有部分代码不太懂，想请教一下， parser.add_argument( "--arch_bits", type=lambda s: [float(item) for item in s.split(",")] if len(s) != 0 else "", default=" ", help="bits configuration of each layer",

if len(args.arch_bits) != 0: if args.wa_same_bit: set_wae_bits(model, args.arch_bits) elif args.search_w_bit: set_w_bits(model, args.arch_bits) else: set_bits(model, args.arch_bits) show_bits(model) logger.info("Set arch bits to: {}".format(args.arch_bits)) logger.info(model) 这个arch_bits主要是做什么的呢，卡在这里有段时间了

opened by LKAMING97 0

PyTorch implementation for "Sharpness-aware Quantization for Deep Neural Networks".

Related tags

Overview

Sharpness-aware Quantization for Deep Neural Networks

Recent Update

Setup the environments

Data preparation

ImageNet

CIFAR-100

Training

Fixed-precision quantization

Mixed-precision quantization

Results on CIFAR-100

Results on ImageNet

License

Acknowledgement

You might also like...

Objective of the repository is to learn and build machine learning models using Pytorch. 30DaysofML Using Pytorch

Pretrained SOTA Deep Learning models, callbacks and more for research and production with PyTorch Lightning and PyTorch

Amazon Forest Computer Vision: Satellite Image tagging code using PyTorch / Keras with lots of PyTorch tricks

The Incredible PyTorch: a curated list of tutorials, papers, projects, communities and more relating to PyTorch.

Amazon Forest Computer Vision: Satellite Image tagging code using PyTorch / Keras with lots of PyTorch tricks

A bunch of random PyTorch models using PyTorch's C++ frontend

PyTorch Autoencoders - Implementing a Variational Autoencoder (VAE) Series in Pytorch.

PyTorch-LIT is the Lite Inference Toolkit (LIT) for PyTorch which focuses on easy and fast inference of large models on end-devices.

A general framework for deep learning experiments under PyTorch based on pytorch-lightning

Comments

Quantize_first_last_layer

代码问题请求帮助

Releases(v0.1.1)

v0.1.1(Nov 23, 2021)

v0.1(Nov 23, 2021)

Owner

Zhuang AI Group

[ICCV'21] Pri3D: Can 3D Priors Help 2D Representation Learning?

A neuroanatomy-based augmented reality experience powered by computer vision. Features 3D visuals of the Atlas Brain Map slices.

face_recognization (FaceNet) + TFHE (HNP) + hand_face_detection (Mediapipe)

Research shows Google collects 20x more data from Android than Apple collects from iOS. Block this non-consensual telemetry using pihole blocklists.

[CVPR 2022] "The Principle of Diversity: Training Stronger Vision Transformers Calls for Reducing All Levels of Redundancy" by Tianlong Chen, Zhenyu Zhang, Yu Cheng, Ahmed Awadallah, Zhangyang Wang

This repository contains code demonstrating the methods outlined in Path Signature Area-Based Causal Discovery in Coupled Time Series presented at Causal Analysis Workshop 2021.

TyXe: Pyro-based BNNs for Pytorch users

Tensorflow implementation of Swin Transformer model.

The codes and related files to reproduce the results for Image Similarity Challenge Track 1.

Auto HMM: Automatic Discrete and Continous HMM including Model selection

Code for our CVPR2021 paper coordinate attention

We utilize deep reinforcement learning to obtain favorable trajectories for visual-inertial system calibration.

This is an unofficial PyTorch implementation of Meta Pseudo Labels

NeuPy is a Tensorflow based python library for prototyping and building neural networks

Code for "The Intrinsic Dimension of Images and Its Impact on Learning" - ICLR 2021 Spotlight

Buffon’s needle: one of the oldest problems in geometric probability

Tensorflow implementation of ID-Unet: Iterative Soft and Hard Deformation for View Synthesis.

A novel framework to automatically learn high-quality scanning of non-planar, complex anisotropic appearance.

Bringing Characters to Life with Computer Brains in Unity

Tools for investing in Python