Implementation for paper "Towards the Generalization of Contrastive Self-Supervised Learning"

Last update: Nov 30, 2022

Overview

Contrastive Self-Supervised Learning on CIFAR-10

Paper

"Towards the Generalization of Contrastive Self-Supervised Learning", Weiran Huang, Mingyang Yi and Xuyang Zhao, arXiv:2111.00743, 2021.

The most critical argument we made in our paper is that the quality of data augmentation exhibits great impact on the quality of contrastive-learned encoder. The data augmentation with sharper intra-class concentration enables the model to have better generalization on downstream tasks. We verify it through a variety of experiments in this repository.

Supported methods

SimCLR
Barlow Twins
MoCo
SimSiam

Installation

pip install -r requirement.txt

Dependencies

torch==1.4.0
torchvision==0.5.0
pytorch-lightning==1.3.8
lightly==1.0.8 (important!)

Evaluation

KNN evaluation protocol. Code from here.

Results

ResNet-18 trained by SimCLR and Barlow Twins over various data augmentation combinations.

Example: python main.py --model=twins --epoch=800 --batch=512 --round=3 --augs=abcde

(a)	(b)	(c)	(d)	(e)	SimCLR	Barlow Twins
✓	✓	✓	✓	✓	89.92 ± 0.05	83.93 ± 0.57
✓	✓	✓	✓	×	88.41 ± 0.11	83.37 ± 0.43
✓	✓	✓	×	×	83.62 ± 0.19	73.70 ± 0.99
✓	✓	×	×	×	62.91 ± 0.25	49.56 ± 0.11
✓	×	×	×	×	62.37 ± 0.09	48.54 ± 0.29

Augmentation operations include:

(a) random cropping with a scaling factor chosen in [0.08, 1.0];

(b) random Gaussian blur with a probability 0.5;

(d) color distortion with a probability of 0.8 and with strength of [0.4, 0.4, 0.4, 0.1];

(e) random horizontal flipping with a probability of 0.5.

ResNet18 trained by SimCLR and Barlow Twins over various color distortion strengths.

Example: python main.py --model=simclr --epoch=800 --batch=512 --round=3 --augs=color --strength=1

Color Distortion Strength	SimCLR	Barlow Twins
1/8	73.60 ± 0.11	61.13 ± 2.81
1/4	76.25 ± 0.16	68.30 ± 0.15
1/2	78.49 ± 0.09	72.76 ± 1.50
1	82.64 ± 0.57	78.79 ± 0.54

Acknowledgement

This code is based on:

Citation

If you find our work useful in your research, please consider citing:

@misc{huang2021generalization,
      title={Towards the Generalization of Contrastive Self-Supervised Learning}, 
      author={Weiran Huang and Mingyang Yi and Xuyang Zhao},
      year={2021},
      eprint={2111.00743},
      archivePrefix={arXiv}
}

You might also like...

Implementation of Nyström Self-attention, from the paper Nyströmformer

Nyström Attention Implementation of Nyström Self-attention, from the paper Nyströmformer. Yannic Kilcher video Install $ pip install nystrom-attention

95 Jan 2, 2023

Implementation of SETR model, Original paper: Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers.

SETR - Pytorch Since the original paper (Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers.) has no official

112 Dec 16, 2022

Official implementation of the paper Image Generators with Conditionally-Independent Pixel Synthesis https://arxiv.org/abs/2011.13775

CIPS -- Official Pytorch Implementation of the paper Image Generators with Conditionally-Independent Pixel Synthesis Requirements pip install -r requi

Multimodal Lab @ Samsung AI Center Moscow

201 Dec 21, 2022

Official pytorch implementation of paper

Implementation for paper "Towards the Generalization of Contrastive Self-Supervised Learning"

Related tags

Overview

Contrastive Self-Supervised Learning on CIFAR-10

Paper

Supported methods

Installation

Dependencies

Evaluation

Results

ResNet-18 trained by SimCLR and Barlow Twins over various data augmentation combinations.

ResNet18 trained by SimCLR and Barlow Twins over various color distortion strengths.

Acknowledgement

Citation

You might also like...

Implementation of Nyström Self-attention, from the paper Nyströmformer

Implementation of SETR model, Original paper: Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers.

Official implementation of the paper Image Generators with Conditionally-Independent Pixel Synthesis https://arxiv.org/abs/2011.13775

Official pytorch implementation of paper "Image-to-image Translation via Hierarchical Style Disentanglement".

PyTorch implementation of paper "Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes", CVPR 2021

Implementation of Barlow Twins paper

Official pytorch implementation of paper "Inception Convolution with Efficient Dilation Search" (CVPR 2021 Oral).

Official implementation of our paper "LLA: Loss-aware Label Assignment for Dense Pedestrian Detection" in Pytorch.

Functional TensorFlow Implementation of Singular Value Decomposition for paper Fast Graph Learning

Releases(v0.2.1)

v0.2.1(Jun 16, 2022)

v0.2.0(Jun 4, 2022)

v0.1.0(Jun 4, 2022)

Owner

Weiran Huang

9th place solution

Dealing With Misspecification In Fixed-Confidence Linear Top-m Identification

Natural Posterior Network: Deep Bayesian Predictive Uncertainty for Exponential Family Distributions

Neon: an add-on for Lightbulb making it easier to handle component interactions

EgoNN: Egocentric Neural Network for Point Cloud Based 6DoF Relocalization at the City Scale

The aim of the game, as in the original one, is to find a specific image from a group of different images of a person's face

Topic Discovery via Latent Space Clustering of Pretrained Language Model Representations

🏅 Top 5% in 제2회 연구개발특구 인공지능 경진대회 AI SPARK 챌린지

Using machine learning to predict and analyze high and low reader engagement for New York Times articles posted to Facebook.

A new data augmentation method for extreme lighting conditions.

Tacotron 2 - PyTorch implementation with faster-than-realtime inference

Pytorch implementation of SELF-ATTENTIVE VAD, ICASSP 2021

ACAV100M: Automatic Curation of Large-Scale Datasets for Audio-Visual Video Representation Learning. In ICCV, 2021.

Code for the Lovász-Softmax loss (CVPR 2018)

这个开源项目主要是对经典的时间序列预测算法论文进行复现，模型主要参考自GluonTS，框架主要参考自Informer

PyTorch Connectomics: segmentation toolbox for EM connectomics

Implementation of paper "Self-supervised Learning on Graphs:Deep Insights and New Directions"

Spatial Temporal Graph Convolutional Networks (ST-GCN) for Skeleton-Based Action Recognition in PyTorch

DFFNet: An IoT-perceptive Dual Feature Fusion Network for General Real-time Semantic Segmentation

Fantasy Points Prediction and Dream Team Formation