pip3 install transformer-tricks

This repo contains code (Python and LaTeX) for the Transformer Tricks papers.

• Flash normalization:

  • arXiv paper: https://arxiv.org/abs/2407.09577
  • See python folder for code to convert LLMs to FlashNorm
  • Notebook example of converting an LLM to flashNorm:
  • Notebook for paper:
  • HuggingFace repo

• Approximate attention [work in progress]:

  • PDF here

• Removing weights for skipless transformers:

  • arXiv paper: https://arxiv.org/abs/2404.12362
  • Notebook:

• Precomputing the first layer:

  • arXiv paper: https://arxiv.org/abs/2402.13388