Long Expressive Memory for Sequence Modeling
This repository contains the implementation to reproduce the numerical experiments of the paper Long Expressive Memory for Sequence Modeling
Requirements
To setup the environment, install the requirements using python 3.7.10 with
pip install -r requirements.txt
Experiments
This repository contains the codes to reproduce the results of the following experiments for the proposed LEM:
- Adding Task
- EigenWorms
- FitzHugh-Nagumo
- Google12 (V2)
- Heart rate prediction
- noisy CIFAR-10
- Permuted Sequential MNIST
- PennTree Bank (word- and char-level)
- Sequential MNIST
All datasets get either synthetically generated or downloaded automatically, providing a full data pipline for each experiment.
Note that the PTB experiments are based on the open-source tensorflow project LAnguage Modelling Benchmarks (lamb) for tuning and testing Tensorflow language models. In order to reproduce these experiments, please follow the steps explained in the README of the PTB directory.
Results
The results of LEM for each of the experiments are:
| Experiment | Result |
| EigenWorms | 92.3% mean test accuracy |
| FitzHugh-Nagumo | 0.002 test L^2 loss |
| Google12 (V2) | 95.7% test accuracy |
| Heart rate prediction | 0.85 test L^2 loss |
| noisy CIFAR-10 | 60.5% test accuracy |
| Permuted Sequential MNIST | 96.6% test accuracy |
| PennTree Bank char-level (single layer) | 1.25 test bits-per-character |
| PennTree Bank word-level (single layer) | 72.8 test perplexity |
| Sequential MNIST | 99.5% test accuracy |
35 Dec 30, 2022
9 Jul 17, 2022
39 Dec 28, 2022
6 May 03, 2022
5 May 26, 2022
8 Oct 07, 2022
502 Jan 08, 2023
5 Aug 25, 2020
37 Dec 24, 2022
10 Oct 16, 2022
1 Jan 18, 2022
125 Dec 22, 2022
22 Oct 28, 2022
33 Dec 05, 2022
5.4k Jan 02, 2023
9 Dec 23, 2022
2 Aug 22, 2022
25 Aug 30, 2022
42 Dec 13, 2022
101 Sep 07, 2022