[ICML'21] Estimate the accuracy of the classifier in various environments through self-supervision

Overview

What Does Rotation Prediction Tell Us about Classifier Accuracy under Varying Testing Environments?

[Paper] [ICML'21 Project]

PyTorch Implementation

This repository contains:

  • the PyTorch implementation of AutoEavl.
  • the example on CIFAR-10 setup (use imgaug)
  • linear regression

Please follow the instruction below to install it and run the experiment demo.

Prerequisites

  • Linux (tested on Ubuntu 16.04LTS)
  • NVIDIA GPU + CUDA CuDNN (tested on GTX 2080 Ti)
  • CIFAR-10 (download and unzip to PROJECT_DIR/data/)
  • CIFAR10.1 (download and unzip to PROJECT_DIR/data/CIFAR-10.1)
  • Please use PyTorch1.5 to avoid compilation errors (other versions should be good)
  • You might need to change the file paths, and please be sure you change the corresponding paths in the codes as well

Getting started

  1. Install dependencies
    # Imgaug (or see https://imgaug.readthedocs.io/en/latest/source/installation.html)
    conda config --add channels conda-forge
    conda install imgaug
  2. Creat synthetic sets
    # By default it creates 500 synthetic sets
    python generate_synthetic_sets.py
  3. Learn classifier on CIFAR-10 (DenseNet-10-12)
    # Save as "PROJECT_DIR/DenseNet-40-12-ss/checkpoint.pth.tar"
    # Modified based on the wonderful github of https://github.com/andreasveit/densenet-pytorch
    python train.py --layers 40 --growth 12 --no-bottleneck --reduce 1.0
  4. Test classifier on synthetic sets
    # 1) Get "PROJECT_DIR/accuracy_cls_dense_aug.npy" file
    # 2) Get "PROJECT_DIR/accuracy_ss_dense_aug.npy" file
    # 3) You will see Rank correlation and Pearsons correlation
    # 4) The absolute error of linear regression is also shown
    python test_many.py --layers 40 --growth 12 --no-bottleneck --reduce 1.0
  5. Correlation study
    # You will see correlation.pdf;
    python analyze_correlation.py
        

Citation

If you use the code in your research, please cite:

    @inproceedings{Deng:ICML2021,
      author    = {Weijian Deng and
                   Stephen Gould and
                   Liang Zheng},
      title     = {What Does Rotation Prediction Tell Us about Classifier Accuracy under Varying Testing Environments?},
      booktitle = {ICML},
      year      = {2021}
    }

License

MIT

Owner
Third-year PhD student at ANU.
H&M Fashion Image similarity search with Weaviate and DocArray

H&M Fashion Image similarity search with Weaviate and DocArray This example shows how to do image similarity search using DocArray and Weaviate as Doc

Laura Ham 18 Aug 11, 2022
FuseDream: Training-Free Text-to-Image Generationwith Improved CLIP+GAN Space OptimizationFuseDream: Training-Free Text-to-Image Generationwith Improved CLIP+GAN Space Optimization

FuseDream This repo contains code for our paper (paper link): FuseDream: Training-Free Text-to-Image Generation with Improved CLIP+GAN Space Optimizat

XCL 191 Dec 31, 2022
机器学习、深度学习、自然语言处理等人工智能基础知识总结。

说明 机器学习、深度学习、自然语言处理基础知识总结。 目前主要参考李航老师的《统计学习方法》一书,也有一些内容例如XGBoost、聚类、深度学习相关内容、NLP相关内容等是书中未提及的。

Peter 445 Dec 12, 2022
This repository contains code accompanying the paper "An End-to-End Chinese Text Normalization Model based on Rule-Guided Flat-Lattice Transformer"

FlatTN This repository contains code accompanying the paper "An End-to-End Chinese Text Normalization Model based on Rule-Guided Flat-Lattice Transfor

THUHCSI 74 Nov 28, 2022
The project is an official implementation of our CVPR2019 paper "Deep High-Resolution Representation Learning for Human Pose Estimation"

Deep High-Resolution Representation Learning for Human Pose Estimation (CVPR 2019) News [2020/07/05] A very nice blog from Towards Data Science introd

Leo Xiao 3.9k Jan 05, 2023
Semi Supervised Learning for Medical Image Segmentation, a collection of literature reviews and code implementations.

Semi-supervised-learning-for-medical-image-segmentation. Recently, semi-supervised image segmentation has become a hot topic in medical image computin

Healthcare Intelligence Laboratory 1.3k Jan 03, 2023
Model Serving Made Easy

The easiest way to build Machine Learning APIs BentoML makes moving trained ML models to production easy: Package models trained with any ML framework

BentoML 4.4k Jan 08, 2023
Code for "Hierarchical Skills for Efficient Exploration" HSD-3 Algorithm and Baselines

Hierarchical Skills for Efficient Exploration This is the source code release for the paper Hierarchical Skills for Efficient Exploration. It contains

Facebook Research 38 Dec 06, 2022
Website which uses Deep Learning to generate horror stories.

Creepypasta - Text Generator Website which uses Deep Learning to generate horror stories. View Demo · View Website Repo · Report Bug · Request Feature

Dhairya Sharma 5 Oct 14, 2022
Introduction to AI assignment 1 HCM University of Technology, term 211

Sokoban Bot Introduction to AI assignment 1 HCM University of Technology, term 211 Abstract This is basically a solver for Sokoban game using Breadth-

Quang Minh 4 Dec 12, 2022
This is the face keypoint train code of project face-detection-project

face-key-point-pytorch 1. Data structure The structure of landmarks_jpg is like below: |--landmarks_jpg |----AFW |------AFW_134212_1_0.jpg |------AFW_

I‘m X 3 Nov 27, 2022
Official PyTorch Code of GrooMeD-NMS: Grouped Mathematically Differentiable NMS for Monocular 3D Object Detection (CVPR 2021)

GrooMeD-NMS: Grouped Mathematically Differentiable NMS for Monocular 3D Object Detection GrooMeD-NMS: Grouped Mathematically Differentiable NMS for Mo

Abhinav Kumar 76 Jan 02, 2023
To model the probability of a soccer coach leave his/her team during Campeonato Brasileiro for 10 chosen teams and considering years 2018, 2019 and 2020.

To model the probability of a soccer coach leave his/her team during Campeonato Brasileiro for 10 chosen teams and considering years 2018, 2019 and 2020.

Larissa Sayuri Futino Castro dos Santos 1 Jan 20, 2022
Code for MSc Quantitative Finance Dissertation

MSc Dissertation Code ReadMe Sector Volatility Prediction Performance Using GARCH Models and Artificial Neural Networks Curtis Nybo MSc Quantitative F

2 Dec 01, 2022
Official implementation of "Refiner: Refining Self-attention for Vision Transformers".

RefinerViT This repo is the official implementation of "Refiner: Refining Self-attention for Vision Transformers". The repo is build on top of timm an

101 Dec 29, 2022
PyVideoAI: Action Recognition Framework

This reposity contains official implementation of: Capturing Temporal Information in a Single Frame: Channel Sampling Strategies for Action Recognitio

Kiyoon Kim 22 Dec 29, 2022
(ICCV 2021 Oral) Re-distributing Biased Pseudo Labels for Semi-supervised Semantic Segmentation: A Baseline Investigation.

DARS Code release for the paper "Re-distributing Biased Pseudo Labels for Semi-supervised Semantic Segmentation: A Baseline Investigation", ICCV 2021

CVMI Lab 58 Jan 01, 2023
LV-BERT: Exploiting Layer Variety for BERT (Findings of ACL 2021)

LV-BERT Introduction In this repo, we introduce LV-BERT by exploiting layer variety for BERT. For detailed description and experimental results, pleas

Weihao Yu 14 Aug 24, 2022
Sudoku solver - A sudoku solver with python

sudoku_solver A sudoku solver What is Sudoku? Sudoku (Japanese: 数独, romanized: s

Sikai Lu 0 May 22, 2022
An official source code for "Augmentation-Free Self-Supervised Learning on Graphs"

Augmentation-Free Self-Supervised Learning on Graphs An official source code for Augmentation-Free Self-Supervised Learning on Graphs paper, accepted

Namkyeong Lee 59 Dec 01, 2022