Implementation of TransGanFormer, an all-attention GAN that combines the finding from the recent GanFormer and TransGan paper

Last update: Dec 06, 2022

Overview

TransGanFormer (wip)

Implementation of TransGanFormer, an all-attention GAN that combines the finding from the recent GansFormer and TransGan paper. It will also contain a bunch of tricks I have picked up building transformers and GANs for the last year or so, including efficient linear attention and pixel level attention.

Install

$ pip install transganformer

Usage

$ transganformer --data ./path/to/data

Citations

@misc{jiang2021transgan,
    title   = {TransGAN: Two Transformers Can Make One Strong GAN}, 
    author  = {Yifan Jiang and Shiyu Chang and Zhangyang Wang},
    year    = {2021},
    eprint  = {2102.07074},
    archivePrefix = {arXiv},
    primaryClass = {cs.CV}
}

@misc{hudson2021generative,
    title   = {Generative Adversarial Transformers}, 
    author  = {Drew A. Hudson and C. Lawrence Zitnick},
    year    = {2021},
    eprint  = {2103.01209},
    archivePrefix = {arXiv},
    primaryClass = {cs.CV}
}

You might also like...

Finding all things on-prem Microsoft for password spraying and enumeration.

msprobe About Installing Usage Examples Coming Soon Acknowledgements About Finding all things on-prem Microsoft for password spraying and enumeration.

205 Jan 9, 2023

Implementation of the 😇 Attention layer from the paper, Scaling Local Self-Attention For Parameter Efficient Visual Backbones

HaloNet - Pytorch Implementation of the Attention layer from the paper, Scaling Local Self-Attention For Parameter Efficient Visual Backbones. This re

189 Nov 22, 2022

Implementation of a memory efficient multi-head attention as proposed in the paper, "Self-attention Does Not Need O(n²) Memory"

Memory Efficient Attention Pytorch Implementation of a memory efficient multi-head attention as proposed in the paper, Self-attention Does Not Need O(

180 Jan 5, 2023

Implementation of Deformable Attention in Pytorch from the paper "Vision Transformer with Deformable Attention"

Deformable Attention Implementation of Deformable Attention from this paper in Pytorch, which appears to be an improvement to what was proposed in DET

128 Dec 24, 2022

A curated (most recent) list of resources for Learning with Noisy Labels

321 Jan 9, 2023

The official implementation of NeurIPS 2021 paper: Finding Optimal Tangent Points for Reducing Distortions of Hard-label Attacks

11 Nov 27, 2022

The official pytorch implementation of our paper "Is Space-Time Attention All You Need for Video Understanding?"

TimeSformer This is an official pytorch implementation of Is Space-Time Attention All You Need for Video Understanding?. In this repository, we provid

1k Dec 31, 2022

Unofficial PyTorch implementation of Fastformer based on paper "Fastformer: Additive Attention Can Be All You Need"."

Fastformer-PyTorch Unofficial PyTorch implementation of Fastformer based on paper Fastformer: Additive Attention Can Be All You Need. Usage : import t

126 Dec 6, 2022

Unofficial Tensorflow-Keras implementation of Fastformer based on paper [Fastformer: Additive Attention Can Be All You Need](https://arxiv.org/abs/2108.09084).

Fastformer-Keras Unofficial Tensorflow-Keras implementation of Fastformer based on paper Fastformer: Additive Attention Can Be All You Need. Tensorflo

10 Jan 30, 2022

Comments

kmeans_iters in attention

Hello first at all thank you for taking the time and port all of theses models to pytorch, you are really helping out people like me who have tensorflow as a second language :D

in line 689 of https://github.com/dorarad/gansformer/blob/main/training/network.py there is a loop over kmeans-iter in the transformerlayer, but i can t find this loop in your implementation of Attention, is it implicitly implemented with these einops - i am sadly not fluent with einops or tensorflow :/

regards martin

opened by martinpflaum 0
dual_contrast_loss argument - error while training

When trying to train a model, the dual_contrast_loss argument spits up an error on lightweight-gan.py. My guess is that it's not implemented yet.

At first I thought it was a mismatch (should it be dual_contrastive_loss btw like in lightweight-gan?) but apparently needed to be commented out for training to start successfully.

Very exciting by the way! Training seems to be super fast, about 28h for 150,000 iterations for a 256p model on my 2080 (although it's a very small dataset). I'll share some closer-to-final results once I have them.

Best, Theodore.

opened by TheodoreGalanos 2

Releases(0.0.17)

0.0.17(Apr 27, 2021)

Source code(tar.gz)
Source code(zip)
0.0.16(Apr 15, 2021)

Source code(tar.gz)
Source code(zip)
0.0.15(Apr 13, 2021)

Source code(tar.gz)
Source code(zip)
0.0.14(Apr 13, 2021)

Source code(tar.gz)
Source code(zip)
0.0.12(Apr 12, 2021)

Source code(tar.gz)
Source code(zip)
0.0.11(Apr 12, 2021)

Source code(tar.gz)
Source code(zip)
0.0.10(Apr 11, 2021)

Source code(tar.gz)
Source code(zip)
0.0.9(Apr 11, 2021)

Source code(tar.gz)
Source code(zip)
0.0.8(Apr 11, 2021)

Source code(tar.gz)
Source code(zip)
0.0.7(Apr 9, 2021)

Source code(tar.gz)
Source code(zip)
0.0.6(Apr 9, 2021)

Source code(tar.gz)
Source code(zip)
0.0.5(Apr 9, 2021)

Source code(tar.gz)
Source code(zip)
0.0.4(Apr 7, 2021)

Source code(tar.gz)
Source code(zip)
0.0.3(Apr 7, 2021)

Source code(tar.gz)
Source code(zip)
0.0.2(Apr 5, 2021)

Source code(tar.gz)
Source code(zip)
0.0.1(Apr 5, 2021)

Source code(tar.gz)
Source code(zip)

Owner

Phil Wang

Working with Attention. It's all we need.

GitHub Repository

Generic template to bootstrap your PyTorch project with PyTorch Lightning, Hydra, W&B, and DVC.

NN Template Generic template to bootstrap your PyTorch project. Click on Use this Template and avoid writing boilerplate code for: PyTorch Lightning,

520 Dec 30, 2022

This is the repository for our paper SimpleTrack: Understanding and Rethinking 3D Multi-object Tracking

SimpleTrack This is the repository for our paper SimpleTrack: Understanding and Rethinking 3D Multi-object Tracking. We are still working on writing t

189 Dec 26, 2022

Self-Supervised Generative Style Transfer for One-Shot Medical Image Segmentation

Self-Supervised Generative Style Transfer for One-Shot Medical Image Segmentation This repository contains the Pytorch implementation of the proposed

19 Nov 10, 2022

This is the official implementation of TrivialAugment and a mini-library for the application of multiple image augmentation strategies including RandAugment and TrivialAugment.

Trivial Augment This is the official implementation of TrivialAugment (https://arxiv.org/abs/2103.10158), as was used for the paper. TrivialAugment is

94 Dec 30, 2022

Tensors and Dynamic neural networks in Python with strong GPU acceleration

PyTorch is a Python package that provides two high-level features: Tensor computation (like NumPy) with strong GPU acceleration Deep neural networks b

61.4k Jan 04, 2023

Paper: De-rendering Stylized Texts

Paper: De-rendering Stylized Texts Wataru Shimoda1, Daichi Haraguchi2, Seiichi Uchida2, Kota Yamaguchi1 1CyberAgent.Inc, 2 Kyushu University Accepted

55 Dec 18, 2022

Machine Learning Model deployment for Container (TensorFlow Serving)

try_tf_serving ├───dataset │ ├───testing │ │ ├───paper │ │ ├───rock │ │ └───scissors │ └───training │ ├───paper │ ├───rock

5 Jan 07, 2022

CoaT: Co-Scale Conv-Attentional Image Transformers

CoaT: Co-Scale Conv-Attentional Image Transformers Introduction This repository contains the official code and pretrained models for CoaT: Co-Scale Co

191 Dec 03, 2022

Pytorch Implementation for Dilated Continuous Random Field

DilatedCRF Pytorch implementation for fully-learnable DilatedCRF. If you find my work helpful, please consider our paper: @article{Mo2022dilatedcrf,

3 Nov 13, 2022

Locally Enhanced Self-Attention: Rethinking Self-Attention as Local and Context Terms

LESA Introduction This repository contains the official implementation of Locally Enhanced Self-Attention: Rethinking Self-Attention as Local and Cont

20 Dec 31, 2021

Official Pytorch implementation of "Learning Debiased Representation via Disentangled Feature Augmentation (Neurips 2021, Oral)"

Learning Debiased Representation via Disentangled Feature Augmentation (Neurips 2021, Oral): Official Project Webpage This repository provides the off

68 Dec 17, 2022

Implementation of TransGanFormer, an all-attention GAN that combines the finding from the recent GanFormer and TransGan paper

Related tags

Overview

TransGanFormer (wip)

Install

Usage

Citations

You might also like...

Finding all things on-prem Microsoft for password spraying and enumeration.

Implementation of the 😇 Attention layer from the paper, Scaling Local Self-Attention For Parameter Efficient Visual Backbones

Implementation of a memory efficient multi-head attention as proposed in the paper, "Self-attention Does Not Need O(n²) Memory"

Implementation of Deformable Attention in Pytorch from the paper "Vision Transformer with Deformable Attention"

A curated (most recent) list of resources for Learning with Noisy Labels

The official implementation of NeurIPS 2021 paper: Finding Optimal Tangent Points for Reducing Distortions of Hard-label Attacks

The official pytorch implementation of our paper "Is Space-Time Attention All You Need for Video Understanding?"

Unofficial PyTorch implementation of Fastformer based on paper "Fastformer: Additive Attention Can Be All You Need"."

Unofficial Tensorflow-Keras implementation of Fastformer based on paper [Fastformer: Additive Attention Can Be All You Need](https://arxiv.org/abs/2108.09084).

Comments

kmeans_iters in attention

dual_contrast_loss argument - error while training

Releases(0.0.17)

0.0.17(Apr 27, 2021)

0.0.16(Apr 15, 2021)

0.0.15(Apr 13, 2021)

0.0.14(Apr 13, 2021)

0.0.12(Apr 12, 2021)

0.0.11(Apr 12, 2021)

0.0.10(Apr 11, 2021)

0.0.9(Apr 11, 2021)

0.0.8(Apr 11, 2021)

0.0.7(Apr 9, 2021)

0.0.6(Apr 9, 2021)

0.0.5(Apr 9, 2021)

0.0.4(Apr 7, 2021)

0.0.3(Apr 7, 2021)

0.0.2(Apr 5, 2021)

0.0.1(Apr 5, 2021)

Owner

Phil Wang

Generic template to bootstrap your PyTorch project with PyTorch Lightning, Hydra, W&B, and DVC.

This is the repository for our paper SimpleTrack: Understanding and Rethinking 3D Multi-object Tracking

Self-Supervised Generative Style Transfer for One-Shot Medical Image Segmentation

This is the official implementation of TrivialAugment and a mini-library for the application of multiple image augmentation strategies including RandAugment and TrivialAugment.

Tensors and Dynamic neural networks in Python with strong GPU acceleration

Paper: De-rendering Stylized Texts

Machine Learning Model deployment for Container (TensorFlow Serving)

CoaT: Co-Scale Conv-Attentional Image Transformers

Pytorch Implementation for Dilated Continuous Random Field

Locally Enhanced Self-Attention: Rethinking Self-Attention as Local and Context Terms

Official Pytorch implementation of "Learning Debiased Representation via Disentangled Feature Augmentation (Neurips 2021, Oral)"

[NeurIPS 2020] Semi-Supervision (Unlabeled Data) & Self-Supervision Improve Class-Imbalanced / Long-Tailed Learning

Robust Instance Segmentation through Reasoning about Multi-Object Occlusion [CVPR 2021]

Emotion Recognition from Facial Images

A python package for generating, analyzing and visualizing building shadows

This is the pytorch code for the paper Curious Representation Learning for Embodied Intelligence.

NaturalCC is a sequence modeling toolkit that allows researchers and developers to train custom models

Justmagic - Use a function as a method with this mystic script, like in Nim

Using BERT+Bi-LSTM+CRF

Pytorch implementation of MLP-Mixer with loading pre-trained models.