tldr-transformers

The tl;dr on a few notable transformer/language model papers + other papers (alignment, memorization, etc).

Models: GPT- *, * BERT *, Adapter- *, * T5, etc.

Each set of notes includes links to the paper, the original code implementation (if available) and the Huggingface 🤗 implementation.

Here is an example: t5.

The transformers papers are presented somewhat chronologically below. Go to the " 👉 Notes 👈 " column below to find the notes for each paper.

This repo also includes a table quantifying the differences across transformer papers all in one table.

Contents

• Quick Note
• Motivation
• Papers::Transformer Papers
• Papers::1 Table To Rule Them All
• Papers::Alignment Papers
• Papers::Scaling Law Papers
• Papers::LM Memorization Papers
• Papers::Limited Label Learning Papers
• How To Contribute
• How To Point Our Errors
• Citation
• License

Quick_Note

This is not an intro to deep learning in NLP. If you are looking for that, I recommend one of the following: Fast AI's course, one of the Coursera courses, or maybe this old thing. Come here after that.

Motivation

With the explosion in papers on all things Transformers the past few years, it seems useful to catalog the salient features/results/insights of each paper in a digestible format. Hence this repo.

Models

BigTable

All of the table summaries found ^ collapsed into one really big table here.

Alignment

Scaling

Memorization

FewLabels

Contribute

If you are interested in contributing to this repo, feel free to do the following:

1. Fork the repo.
2. Create a Draft PR with the paper of interest (to prevent "in-flight" issues).
3. Use the suggested template to write your "tl;dr". If it's an architecture paper, you may also want to add to the larger table here.
4. Submit your PR.

Errata

Undoubtedly there is information that is incorrect here. Please open an Issue and point it out.

Citation

@misc{cliff-notes-transformers,
  author = {Thompson, Will},
  url = {https://github.com/will-thompson-k/cliff-notes-transformers},
  year = {2021}
}

For the notes above, I've linked the original papers.

License

MIT