TensorFlow implementation of Barlow Twins (Barlow Twins: Self-Supervised Learning via Redundancy Reduction)

Last update: Sep 14, 2022

Overview

Barlow-Twins-TF

This repository implements Barlow Twins (Barlow Twins: Self-Supervised Learning via Redundancy Reduction) in TensorFlow and demonstrates it on the CIFAR10 dataset.

Summary:

With a ResNet20 as a trunk and a 3-layer MLP (each layer containing 2048 units) and 100 epochs of pre-training, this training notebook can give 62.61% accuracy on the CIFAR10 test set. The pre-training total takes ~23 minutes on a single Tesla V100. There are minor differences from the original implementation. However, the original loss function and the other minor details like having a big enough projection dimension have been maintained.

For details on Barlow Twins, I suggest reading the original paper, it's really well-written.

Loss progress during pre-training

Other notes

Pre-trained model is available here.
To follow the original implementation details as closely as possible, a WarmUpCosine learning rate schedule has been used during pre-training:
During linear evaluation, Cosine Decay has been used.

Acknowledgements

Thanks to Stéphane Deny (one of the authors of the paper) for helping me catch a pesky bug.

Releases(v1.0.0)

v1.0.0(May 10, 2021)

The archive contains a ResNet20 encoder (trunk + projection head) trained using Barlow Twins for 100 epochs with CIFAR10.
Source code(tar.gz)
Source code(zip)
barlow_twins.tar.gz(38.07 MB)

Public implementation of "Learning from Suboptimal Demonstration via Self-Supervised Reward Regression" from CoRL'21

Self-Supervised Reward Regression (SSRR) Codebase for CoRL 2021 paper "Learning from Suboptimal Demonstration via Self-Supervised Reward Regression "

19 Dec 12, 2022

An official PyTorch implementation of the TKDE paper "Self-Supervised Graph Representation Learning via Topology Transformations".

Self-Supervised Graph Representation Learning via Topology Transformations This repository is the official PyTorch implementation of the following pap

2 Oct 31, 2022

Twins: Revisiting the Design of Spatial Attention in Vision Transformers

Twins: Revisiting the Design of Spatial Attention in Vision Transformers Very recently, a variety of vision transformer architectures for dense predic

482 Dec 18, 2022

This repository is the official implementation of Unleashing the Power of Contrastive Self-Supervised Visual Models via Contrast-Regularized Fine-Tuning (NeurIPS21).

Core-tuning This repository is the official implementation of ``Unleashing the Power of Contrastive Self-Supervised Visual Models via Contrast-Regular

18 Dec 17, 2022

PyTorch implementation of the paper Deep Networks from the Principle of Rate Reduction

A repository that shares tuning results of trained models generated by TensorFlow / Keras. Post-training quantization (Weight Quantization, Integer Quantization, Full Integer Quantization, Float16 Quantization), Quantization-aware training. TensorFlow Lite. OpenVINO. CoreML. TensorFlow.js. TF-TRT. MediaPipe. ONNX. [.tflite,.h5,.pb,saved_model,tfjs,tftrt,mlmodel,.xml/.bin, .onnx]

PINTO_model_zoo Please read the contents of the LICENSE file located directly under each folder before using the model. My model conversion scripts ar

2.4k Jan 5, 2023

TensorFlow implementation of Barlow Twins (Barlow Twins: Self-Supervised Learning via Redundancy Reduction)

Related tags

Overview

Barlow-Twins-TF

Loss progress during pre-training

Other notes

Acknowledgements

You might also like...

Public implementation of "Learning from Suboptimal Demonstration via Self-Supervised Reward Regression" from CoRL'21

An official PyTorch implementation of the TKDE paper "Self-Supervised Graph Representation Learning via Topology Transformations".

Twins: Revisiting the Design of Spatial Attention in Vision Transformers

This repository is the official implementation of Unleashing the Power of Contrastive Self-Supervised Visual Models via Contrast-Regularized Fine-Tuning (NeurIPS21).

PyTorch implementation of the paper Deep Networks from the Principle of Rate Reduction

Official NumPy Implementation of Deep Networks from the Principle of Rate Reduction (2021)

Model-based 3D Hand Reconstruction via Self-Supervised Learning, CVPR2021

TLDR: Twin Learning for Dimensionality Reduction

Releases(v1.0.0)

v1.0.0(May 10, 2021)

Owner

Sayak Paul

The code for the CVPR 2021 paper Neural Deformation Graphs, a novel approach for globally-consistent deformation tracking and 3D reconstruction of non-rigid objects.

Official implement of Paper：A deeply supervised image fusion network for change detection in high resolution bi-temporal remote sening images

D2LV: A Data-Driven and Local-Verification Approach for Image Copy Detection

TorchIO is a Medical image preprocessing and augmentation toolkit for deep learning. Part of the PyTorch Ecosystem.

Deep learning model, heat map, data prepo

All-in-one Docker container that allows a user to explore Nautobot in a lab environment.

[ICRA2021] Reconstructing Interactive 3D Scene by Panoptic Mapping and CAD Model Alignment

A PyTorch implementation of a Factorization Machine module in cython.

Pytorch implementation for the EMNLP 2020 (Findings) paper: Connecting the Dots: A Knowledgeable Path Generator for Commonsense Question Answering

[AAAI 2022] Separate Contrastive Learning for Organs-at-Risk and Gross-Tumor-Volume Segmentation with Limited Annotation

Code release for paper: The Boombox: Visual Reconstruction from Acoustic Vibrations

Artifacts for paper "MMO: Meta Multi-Objectivization for Software Configuration Tuning"

Jingju baseline - A baseline model of our project of Beijing opera script generation

Modeling Category-Selective Cortical Regions with Topographic Variational Autoencoders

A Comparative Review of Recent Kinect-Based Action Recognition Algorithms (TIP2020, Matlab codes)

Syllabus del curso IIC2115 - Programación como Herramienta para la Ingeniería 2022/I

The Codebase for Causal Distillation for Language Models.

Easy-to-use,Modular and Extendible package of deep-learning based CTR models .

Create Data & AI apps in 20 lines of code with Shimoku

Region-aware Contrastive Learning for Semantic Segmentation, ICCV 2021