Efficient Sharpness-aware Minimization for Improved Training of Neural Networks

Code for “Efficient Sharpness-aware Minimization for Improved Training of Neural Networks”

Requisite

This code is implemented in PyTorch, and we have tested the code under the following environment settings:

python = 3.8.8
torch = 1.8.0
torchvision = 0.9.0

What is in this repository

Codes for our ESAM on CIFAR10/CIFAR100 datasets.

How to use it

from utils.layer_dp_sam import ESAM
base_optimizer = torch.optim.SGD(model.parameters(),lr=args.learning_rate,momentum=0.9,weight_decay=args.weight_decay)
optimizer = ESAM(paras, base_optimizer, rho=args.rho, weight_dropout=args.weight_dropout,adaptive=args.isASAM,nograd_cutoff=args.nograd_cutoff,opt_dropout = args.opt_dropout,temperature=args.temperature)

--beta the SWP hyperparameter

--gamma the SDS hyperparameter

During training loss_fct should have reduction="none", to return instance-wise losses. defined_backward is the function used for DDP and mixed precision backward

loss_fct = torch.nn.CrossEntropyLoss(reduction="none")
def defined_backward():
    if args.fp16:
    with amp.scale_loss(loss, optimizer0) as scaled_loss:
        scaled_loss.backward()
    else:
        loss.backward()

paras = [inputs,targets,loss_fct,model,defined_backward]
optimizer.paras = paras
optimizer.step()
predictions_logits,loss = optimizer.returnthings

Example

bash run.sh

Reference Code

[1] SAM

Efficient Sharpness-aware Minimization for Improved Training of Neural Networks

Related tags

Overview

Efficient Sharpness-aware Minimization for Improved Training of Neural Networks

Requisite

What is in this repository

How to use it

Example

Reference Code

Owner

Angusdu

[LREC] MMChat: Multi-Modal Chat Dataset on Social Media

object detection; robust detection; ACM MM21 grand challenge; Security AI Challenger Phase VII

Code for Reciprocal Adversarial Learning for Brain Tumor Segmentation: A Solution to BraTS Challenge 2021 Segmentation Task

FastReID is a research platform that implements state-of-the-art re-identification algorithms.

An implementation for the loss function proposed in Decoupled Contrastive Loss paper.

MakeItTalk: Speaker-Aware Talking-Head Animation

Keyword2Text This repository contains the code of the paper: "A Plug-and-Play Method for Controlled Text Generation"

A collection of Google research projects related to Federated Learning and Federated Analytics.

This is the code of NeurIPS'21 paper "Towards Enabling Meta-Learning from Target Models".

Apply a perspective transformation to a raster image inside Inkscape (no need to use an external software such as GIMP or Krita).

Roadmap to becoming a machine learning engineer in 2020

QA-GNN: Question Answering using Language Models and Knowledge Graphs

Project code for weakly supervised 3D object detectors using wide-baseline multi-view traffic camera data: WIBAM.

TensorFlow-based implementation of "ICNet for Real-Time Semantic Segmentation on High-Resolution Images".

Existing Literature about Machine Unlearning

Deep motion generator collections

Net2net - Network-to-Network Translation with Conditional Invertible Neural Networks

Deep GPs built on top of TensorFlow/Keras and GPflow

[CVPR 2021] Forecasting the panoptic segmentation of future video frames

This repository contains code demonstrating the methods outlined in Path Signature Area-Based Causal Discovery in Coupled Time Series presented at Causal Analysis Workshop 2021.