PyTorch implementation for "Sharpness-aware Quantization for Deep Neural Networks".

Last update: Dec 19, 2022

Related tags

Deep Learning SAQ

Overview

Sharpness-aware Quantization for Deep Neural Networks

Recent Update

2021.11.23: We release the source code of SAQ.

Setup the environments

Clone the repository locally:

git clone https://github.com/zhuang-group/SAQ

Install pytorch 1.8+, tensorboard and prettytable

conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch
pip install tensorboard
pip install prettytable

Data preparation

ImageNet

Download the ImageNet 2012 dataset from here, and prepare the dataset based on this script.
Change the dataset path in link_imagenet.py and link the ImageNet-100 by

python link_imagenet.py

CIFAR-100

Download the CIFAR-100 dataset from here.

After downloading ImageNet and CIFAR-100, the file structure should look like:

dataset
├── imagenet
    ├── train
    │   ├── class1
    │   │   ├── img1.jpeg
    │   │   ├── img2.jpeg
    │   │   └── ...
    │   ├── class2
    │   │   ├── img3.jpeg
    │   │   └── ...
    │   └── ...
    └── val
        ├── class1
        │   ├── img4.jpeg
        │   ├── img5.jpeg
        │   └── ...
        ├── class2
        │   ├── img6.jpeg
        │   └── ...
        └── ...
├── cifar100
    ├── cifar-100-python
    │   ├── meta
    │   ├── test
    │   ├── train
    │   └── ...
    └── ...

Training

Fixed-precision quantization

Download the pre-trained full-precision models from the model zoo.
Train low-precision models.

To train low-precision ResNet-20 on CIFAR-100, run:

sh script/train_qsam_cifar_r20.sh

To train low-precision ResNet-18 on ImageNet, run:

sh script/train_qsam_imagenet_r18.sh

Mixed-precision quantization

Download the pre-trained full-precision models from the model zoo.
Train the configuration generator.

To train the configuration generator of ResNet-20 on CIFAR-100, run:

sh script/train_generator_cifar_r20.sh

To train the configuration generator on ImageNet, run:

sh script/train_generator_imagenet_r18.sh

After training the configuration generator, run following commands to fine-tune the resulting models with the obtained bitwidth configurations on CIFAR-100 and ImageNet.

sh script/finetune_cifar_r20.sh

sh script/finetune_imagenet_r18.sh

Results on CIFAR-100

Network	Method	Bitwidth	BOPs (M)	Top-1 Acc. (%)	Top-5 Acc. (%)
ResNet-20	SAQ	4	674.6	68.7	91.2
ResNet-20	SAMQ	MP	659.3	68.7	91.2
ResNet-20	SAQ	3	392.1	67.7	90.8
ResNet-20	SAMQ	MP	374.4	68.6	91.2
MobileNetV2	SAQ	4	1508.9	75.6	93.7
MobileNetV2	SAMQ	MP	1482.1	75.5	93.6
MobileNetV2	SAQ	3	877.1	74.4	93.2
MobileNetV2	SAMQ	MP	869.5	75.5	93.7

Results on ImageNet

Network	Method	Bitwidth	BOPs (G)	Top-1 Acc. (%)	Top-5 Acc. (%)
ResNet-18	SAQ	4	34.7	71.3	90.0
ResNet-18	SAMQ	MP	33.7	71.4	89.9
ResNet-18	SAQ	2	14.4	67.1	87.3
MobileNetV2	SAQ	4	5.3	70.2	89.4
MobileNetV2	SAMQ	MP	5.3	70.3	89.4

License

This repository is released under the Apache 2.0 license as found in the LICENSE file.

Acknowledgement

This repository has adopted codes from SAM, ASAM and ESAM, we thank the authors for their open-sourced code.

You might also like...

Objective of the repository is to learn and build machine learning models using Pytorch. 30DaysofML Using Pytorch

30 Days Of Machine Learning Using Pytorch Objective of the repository is to learn and build machine learning models using Pytorch. List of Algorithms

119 Nov 24, 2022

Pretrained SOTA Deep Learning models, callbacks and more for research and production with PyTorch Lightning and PyTorch

1.4k Jan 1, 2023

Amazon Forest Computer Vision: Satellite Image tagging code using PyTorch / Keras with lots of PyTorch tricks

Amazon Forest Computer Vision Satellite Image tagging code using PyTorch / Keras Here is a sample of images we had to work with Source: https://www.ka

360 Dec 10, 2022

The Incredible PyTorch: a curated list of tutorials, papers, projects, communities and more relating to PyTorch.

This is a curated list of tutorials, projects, libraries, videos, papers, books and anything related to the incredible PyTorch. Feel free to make a pu

9.2k Jan 2, 2023

Amazon Forest Computer Vision: Satellite Image tagging code using PyTorch / Keras with lots of PyTorch tricks

Amazon Forest Computer Vision Satellite Image tagging code using PyTorch / Keras Here is a sample of images we had to work with Source: https://www.ka

359 Jan 5, 2023

A bunch of random PyTorch models using PyTorch's C++ frontend

PyTorch Deep Learning Models using the C++ frontend Gettting started Clone the repo 1. https://github.com/mrdvince/pytorchcpp 2. cd fashionmnist or

0 Jul 13, 2021

PyTorch Autoencoders - Implementing a Variational Autoencoder (VAE) Series in Pytorch.

PyTorch Autoencoders Implementing a Variational Autoencoder (VAE) Series in Pytorch. Inspired by this repository Model List check model paper conferen

8 Nov 21, 2022

PyTorch-LIT is the Lite Inference Toolkit (LIT) for PyTorch which focuses on easy and fast inference of large models on end-devices.

PyTorch-LIT PyTorch-LIT is the Lite Inference Toolkit (LIT) for PyTorch which focuses on easy and fast inference of large models on end-devices. With

157 Dec 11, 2022

A general framework for deep learning experiments under PyTorch based on pytorch-lightning

torchx Torchx is a general framework for deep learning experiments under PyTorch based on pytorch-lightning. TODO list gan-like training wrapper text

6 Mar 17, 2022

Comments

Quantize_first_last_layer

Hi! I noticed that in your code, you set bits_weights=8 and bits_activations=32 for first layer as default, it's not what is claimed in your paper " For the first and last layers of all quantized models, we quantize both weights and activations to 8-bit. " And I see an accuracy drop if I adjust the bits_activations to 8 for the first layer, could u please explain what is the reason? Thanks!

opened by mmmiiinnnggg 0
代码问题请求帮助

你好，带佬的代码写的很好，有部分代码不太懂，想请教一下， parser.add_argument( "--arch_bits", type=lambda s: [float(item) for item in s.split(",")] if len(s) != 0 else "", default=" ", help="bits configuration of each layer",

if len(args.arch_bits) != 0: if args.wa_same_bit: set_wae_bits(model, args.arch_bits) elif args.search_w_bit: set_w_bits(model, args.arch_bits) else: set_bits(model, args.arch_bits) show_bits(model) logger.info("Set arch bits to: {}".format(args.arch_bits)) logger.info(model) 这个arch_bits主要是做什么的呢，卡在这里有段时间了

opened by LKAMING97 0

PyTorch implementation for "Sharpness-aware Quantization for Deep Neural Networks".

Related tags

Overview

Sharpness-aware Quantization for Deep Neural Networks

Recent Update

Setup the environments

Data preparation

ImageNet

CIFAR-100

Training

Fixed-precision quantization

Mixed-precision quantization

Results on CIFAR-100

Results on ImageNet

License

Acknowledgement

You might also like...

Objective of the repository is to learn and build machine learning models using Pytorch. 30DaysofML Using Pytorch

Pretrained SOTA Deep Learning models, callbacks and more for research and production with PyTorch Lightning and PyTorch

Amazon Forest Computer Vision: Satellite Image tagging code using PyTorch / Keras with lots of PyTorch tricks

The Incredible PyTorch: a curated list of tutorials, papers, projects, communities and more relating to PyTorch.

Amazon Forest Computer Vision: Satellite Image tagging code using PyTorch / Keras with lots of PyTorch tricks

A bunch of random PyTorch models using PyTorch's C++ frontend

PyTorch Autoencoders - Implementing a Variational Autoencoder (VAE) Series in Pytorch.

PyTorch-LIT is the Lite Inference Toolkit (LIT) for PyTorch which focuses on easy and fast inference of large models on end-devices.

A general framework for deep learning experiments under PyTorch based on pytorch-lightning

Comments

Quantize_first_last_layer

代码问题请求帮助

Releases(v0.1.1)

v0.1.1(Nov 23, 2021)

v0.1(Nov 23, 2021)

Owner

Zhuang AI Group

BackgroundRemover lets you Remove Background from images and video with a simple command line interface

gym-anm is a framework for designing reinforcement learning (RL) environments that model Active Network Management (ANM) tasks in electricity distribution networks.

auto-tuning momentum SGD optimizer

Artifacts for paper "MMO: Meta Multi-Objectivization for Software Configuration Tuning"

Approaches to modeling terrain and maps in python

Semi-Supervised Signed Clustering Graph Neural Network (and Implementation of Some Spectral Methods)

A real-time approach for mapping all human pixels of 2D RGB images to a 3D surface-based model of the body

Automated image registration. Registrationimation was too much of a mouthful.

A set of simple scripts to process the Imagenet-1K dataset as TFRecords and make index files for NVIDIA DALI.

CCNet: Criss-Cross Attention for Semantic Segmentation (TPAMI 2020 & ICCV 2019).

LogAvgExp - Pytorch Implementation of LogAvgExp

U-Time: A Fully Convolutional Network for Time Series Segmentation

Official Code for "Non-deep Networks"

CodeContests is a competitive programming dataset for machine-learning

This repository contains code accompanying the paper "An End-to-End Chinese Text Normalization Model based on Rule-Guided Flat-Lattice Transformer"

A Demo server serving Bert through ONNX with GPU written in Rust with <3

PyTorch implementation of "Image-to-Image Translation Using Conditional Adversarial Networks".

Official Implementation for HyperStyle: StyleGAN Inversion with HyperNetworks for Real Image Editing

Fully convolutional deep neural network to remove transparent overlays from images

Official code base for the poster "On the use of Cortical Magnification and Saccades as Biological Proxies for Data Augmentation" published in NeurIPS 2021 Workshop (SVRHM)