Transformers_from_scratch

Implementation of Vanilla Transformers from scratch (Attention is all you need)

1. Embeddings: representing text in lower dimensional representation. converting text into numerical : learnable parameter.

2. Positional Embedding: Convey positional information to the model Sin function: Even position of embedding cos function: odd position of embedding stays the same : nonlearnable parameter.

3. Attention Mechanism:

Resources

1. Umar Jamil: Transfomers from scratch
2. Analytics Vidhya: Transformers from scratch
3. Stanford Online: Transformers United Series
4. Andrej Karpathy: Neural Network Zero to Hero: Build GPT