LLM Self-Attention by Hand

The mechanism that lets a ChatGPT-style model understand a word differently depending on the words around it, rebuilt from nothing but NumPy arrays, in a world small enough that you can check every single number with pencil and paper.

Audience. You can read Python and have met NumPy arrays. No machine learning background is assumed. The workshop builds every mathematical tool it uses, and there are only three of them: writing a word as a list of numbers, a similarity score between two such lists, and a rule that turns scores into percentages.

Time. Around 90 minutes to 2 hours across four modules.

What you will be able to do at the end. Explain what self-attention computes and why it is built the way it is, trace a three-word sentence through the entire mechanism by hand, read the standard attention formula the way you would read code, and verify a from-scratch implementation against PyTorch's official one.

Final check. Take the auto-graded exam: 18 questions, 70 points to pass.

The whole story in one picture

The word "bank" means one thing in "the river bank" and something else in "the money bank". But the first thing a language model does with a word is look it up in a fixed table: same word, same list of numbers, no matter the sentence. Something downstream has to repair that, and that something is self-attention.

One embedding, two context-aware outputs

This picture is the workshop's destination, and every point on it is computed by code you will have read line by line. The gray dot is the one fixed entry for "bank" in the lookup table. The two red diamonds are what self-attention turns it into: pulled toward "river" in one sentence and toward "money" in the other. Same word, same table entry, two different meanings, recovered from context alone.

Along the way, with a measurement at every step:

Module 1 builds a complete, working version of the idea in three lines of NumPy, with no trained parameters at all, and it is already enough to tell the two banks apart.
Module 2 finds two precise ways those three lines fall short, then fixes both with two small hand-built matrices that give each word the power to ask a question that is different from "who looks like me?". You will read every entry of those matrices and know why it is there.
Module 2 also uncovers a trap: at the sizes real models run at, the arithmetic quietly collapses so that a single word soaks up nearly all of the attention, and one carefully chosen constant prevents it. You will measure the collapse and the rescue.
Module 3 proves the from-scratch code and PyTorch's official implementation compute the same function, agreeing to the last bit of rounding by which two ordinary-sized double-precision numbers can differ at all. It then adds the one extra rule that text-generating models need: no looking at words that have not been written yet.

How this workshop is organized

Use the sidebar to move between pages (the menu button on a phone). Each module ends with a short quiz; attempt it before opening the collapsible answer key. The modules build on each other, so go in order.

Page	What you will learn
Module 0: The three tools	Words as lists of numbers, the dot product as a similarity meter, and softmax as a scores-to-shares converter, each pinned down by a hand-checkable measurement.
Module 1: The simplest attention that works	The lookup-table problem, two failed repairs that corner the fix, then the fix itself: every word's vector becomes a relevance-weighted blend of its neighbors'. Three lines of NumPy, zero parameters, and "bank" comes apart into its two meanings.
Module 2: Queries, keys, and one carefully chosen constant	The two measured limitations of Module 1, the query/key machinery that fixes them, and a measurement of why every real model divides its scores by the same magic number.
Module 3: The real thing	The full head in seven lines, verified against PyTorch, the causal mask that stops a generator from peeking ahead, and where all of this sits inside GPT-2.

Prerequisites

To run the experiment yourself (optional; every result is reproduced on the page):

Python 3 with numpy, torch, and matplotlib (pip install numpy torch matplotlib).
A CPU is plenty. The biggest computation in this workshop is a few thousand dot products.

Everything numeric on these pages comes from one deterministic run of a single script, experiment.py (fixed seed 0), and every result is reproduced inline on the module pages: you never need to leave the page you are reading. For the curious, two companion pages exist purely for cross-checking: the Source page holds the full script, ready to copy, and the Log page holds its captured output with line numbers; small "(cross-check: ...)" notes link the two as we go. The exact environment is recorded in the reproducing section of Module 3.

Start with Module 0: The three tools.