Scaled Dot-Product Attention 🔗

This mini-experiment demonstrates the core attention mechanism used in Transformers by implementing scaled dot-product attention from scratch in PyTorch. It shows how query, key, and value tensors interact to compute attention weights using batched tensor operations and softmax normalization, and how these weights are used to produce context-aware output representations. The experiment focuses purely on understanding the mechanics of attention by operating on randomly generated sequences, making it easy to observe how information is aggregated across time steps. By inspecting the attention score matrix and the final output embeddings, this example builds intuition for sequence-to-sequence interactions, contextual mixing, and the fundamental idea behind self-attention, serving as a practical foundation for extending to multi-head attention and full Transformer models.