The code for two papers: Feedback Transformer and Expire-Span.

Last update: Dec 25, 2022

Related tags

Deep Learning transformer-sequential

Overview

transformer-sequential

This repo contains the code for two papers:

Feedback Transformer
Expire-Span

The training code is structured for long sequential modeling with Transformer-like architectures.

Requirements

You will need a CUDA-enabled GPU to run the code.

Setup

Run the following:

pip install -r requirements.txt

Feedback Transformer

Introduced in Addressing Some Limitations of Transformers with Feedback Memory.

Running Experiments from the Paper

enwik8

Model	Params	Valid	Test
Feedback Transformer	77M	0.984	0.962

Numbers are Bits-Per-Character

bash experiments/feedback/enwik8.sh

Algorithmic

Model	3 Variable	5 Variable
Transformer	33.7	37.5
Feedback Transformer	99.1	92.6

Numbers are % Accuracy on Test

bash experiments/feedback/algorithmic_3var.sh
bash experiments/feedback/algorithmic_5var.sh

Expire-Span

Introduced in Not All Memories are Created Equal: Learning to Expire.

Running Experiments from the Paper

enwik8

Model	Params	Valid	Test
Expire-Span 12L	38M	1.014	0.994

Numbers are Bits-Per-Character

bash experiments/expire_span/enwik8.sh

Object Collision

Model	Maximum Span	Test Error (%)
Expire-Span	16k	52.2
Expire-Span	32k	36.7
Expire-Span	64k	26.7

bash experiments/expire_span/object_collision_16k.sh
bash experiments/expire_span/object_collision_32k.sh
bash experiments/expire_span/object_collision_64k.sh

License

The code is licensed under CC-BY-NC license. See the LICENSE file for more details.

The code for two papers: Feedback Transformer and Expire-Span.

Related tags

Overview

transformer-sequential

Requirements

Setup

Feedback Transformer

Running Experiments from the Paper

enwik8

Algorithmic

Expire-Span

Running Experiments from the Paper

enwik8

Object Collision

License

Owner

Facebook Research

Unofficial PyTorch reimplementation of the paper Swin Transformer V2: Scaling Up Capacity and Resolution

HairCLIP: Design Your Hair by Text and Reference Image

A really easy-to-use and powerful sudoku solver.

A repository with exploration into using transformers to predict DNA ↔ transcription factor binding

Implementation of StyleSpace Analysis: Disentangled Controls for StyleGAN Image Generation in PyTorch

Novel and high-performance medical image classification pipelines are heavily utilizing ensemble learning strategies

Linear algebra python - Number of operations and problems in Linear Algebra and Numerical Linear Algebra

A script written in Python that returns a consensus string and profile matrix of a given DNA string(s) in FASTA format.

Official PyTorch Implementation of Rank & Sort Loss [ICCV2021]

Unsupervised Video Interpolation using Cycle Consistency

Benchmarks for Object Detection in Aerial Images

Self-Supervised depth kalilia

MicRank is a Learning to Rank neural channel selection framework where a DNN is trained to rank microphone channels.

ConvMAE: Masked Convolution Meets Masked Autoencoders

git《USD-Seg:Learning Universal Shape Dictionary for Realtime Instance Segmentation》(2020) GitHub: [fig2]

(Py)TOD: Tensor-based Outlier Detection, A General GPU-Accelerated Framework

Object tracking implemented with YOLOv4, DeepSort, and TensorFlow.

Training a deep learning model on the noisy CIFAR dataset

GAN-STEM-Conv2MultiSlice - Exploring Generative Adversarial Networks for Image-to-Image Translation in STEM Simulation

ChebLieNet, a spectral graph neural network turned equivariant by Riemannian geometry on Lie groups.