Module 1: The simplest attention that works

The problem: the lookup table cannot see the sentence

Module 0 left us with an uncomfortable fact about the embedding table (the fixed word-to-vector lookup a language model starts with): it stores exactly one row per vocabulary word. One row means one fixed meaning, chosen before any sentence exists: the lookup hands "bank" the same four numbers in "the river bank" and in "the money bank", because looking a word up asks only which word is this, never which sentence is it in. That is the problem in the title: the lookup table cannot see the sentence.

The experiment script makes this concrete with a helper that takes one sentence and stacks its looked-up rows into a matrix, one row per word of that sentence:

def embed(sentence):
    """Stack the embedding rows for a sentence into a matrix X.

    X has one ROW per word and one COLUMN per axis, so a 3-word sentence
    in our 4-dimensional space gives a 3 x 4 matrix.
    """
    return np.stack([EMBED[w] for w in sentence])

For the two test sentences, the helper stacks these rows, straight from Module 0's embedding table:

X_river, for "the river bank":

X_money, for "the money bank":

Only the middle row differs between the two matrices. The "bank" row is visibly the same four numbers in both: the problem this module exists to fix, sitting in plain sight. The script's first act is to build both matrices and check that fact exactly (cross-check: results.log line 28):

    X_river = embed(SENT_RIVER)
    X_money = embed(SENT_MONEY)

    # First the input fact the whole workshop turns on: the embedding row
    # for 'bank' is the same array in both sentences.
    same = bool(np.array_equal(X_river[2], X_money[2]))
    log(f"  input check: 'bank' row identical in both sentences -> {same}")

  input check: 'bank' row identical in both sentences -> True

Row 2 of each matrix (rows count from 0, so that is the third word, "bank") is the same four numbers, [0.3, 0.3, 0.9, 0.0], in both sentences. Whatever "understanding the sentence" is going to mean, it cannot live in this matrix. It has to be computed.

Why blend at all? Ruling out the alternatives

Do not take blending on faith; arrive at it by watching the alternatives fail.

Attempt 0: no mixing. Keep computing on each word's own vector, just harder: more layers, cleverer math, whatever you like, as long as bank's new vector is computed from bank's old vector alone. This cannot work, as a matter of logic: bank's input row is the same four numbers in both sentences, and any function fed identical inputs returns identical outputs. A "bank" processed alone will mean the same thing next to "river" as next to "money", forever. Whatever the repair is, it must take the other words' vectors as inputs too. Information has to flow between positions; the only question is how to combine it.

Attempt 1: the simplest combination, an equal average.

Predict. Suppose we replace every word's vector with the plain average of all three vectors in the sentence. One real problem gets fixed, and a new, fatal one appears. Try to name both before reading on.

For "the river bank", the equal average is plain arithmetic on Module 0's embedding table (redo it on paper: add the three rows, divide by 3):

The part that worked: this vector knows the sentence's topic. It tilts water over money 0.40 to 0.10, a tilt that bank's own row (balanced at 0.3 each) never had, and in the money sentence the same arithmetic comes out money-flavored instead. Mixing genuinely repairs context-blindness; that much of the idea survives all the way to GPT-2.

The part that is fatal: whose new vector is this? Everyone's. The averaging never looked at who was asking, so "the", "river", and "bank" are all replaced by this same sentence-summary vector, and the three words become indistinguishable from each other. We cured "same word across sentences" and contracted "every word within a sentence identical". The shares are wrong too: the content-free filler word "the" got exactly as much say in the blend as the topic word "river", a full 1/3 each.

Those two failures spell out the requirement. The combination must be an average with weights, where the weights (a) belong to the word doing the looking, a different set per word, so word identities survive, and (b) give bigger shares to more relevant words and smaller shares to filler. Those per-word mixing weights are the attention in self-attention: how much each word attends to each of its neighbors while rebuilding itself.

The idea: a relevance-weighted blend

What is an average with weights, concretely? The weights are a blending recipe: they say how much of each ingredient goes into the mix. With weights that are positive and sum to 1, blending runs slot by slot. Blend [2, 0] and [4, 10] with weights 0.9 and 0.1: the first slot is 0.9 x 2 + 0.1 x 4 = 2.2, the second is 0.9 x 0 + 0.1 x 10 = 1.0, so the result is [2.2, 1.0]. Three properties of this, because the rest of the module leans on them: the weights decide whose information survives into the result; because the weights sum to 1, the result lands between the ingredients, so a blend of embeddings still looks like an embedding; and no slot ever receives anything the ingredients did not bring, so a blend moves information between vectors, never invents it.

So the design question is: where do each word's weights come from? We want relevant words to get big weights, and Module 0 built exactly the two tools needed:

1. Score every pair of words with a dot product (the similarity meter).
2. Softmax each word's row of scores into shares that sum to 1.
3. Blend: each word's new vector is the shares-weighted average of all the vectors.

That three-step pipeline, score then normalize then blend, is attention. Everything in Modules 2 and 3 is refinement.

Three lines of NumPy

Here is the whole mechanism as the script defines it, one line per step. Its docstring boasts "no learned parts at all", and the contrast is this: a real attention layer carries extra matrices of numbers that training adjusts, its learned parts, also known as its parameters (Module 2 introduces them, hand-wired so every number stays readable). This version has none. Its only ingredients are the raw embedding rows, and its three steps are fixed arithmetic, so whatever "bank" manages by the end of this module, none of it will have come from training:

def simple_attention(X):
    """Attention with no learned parts at all.

    Each word scores every word in the sentence with a plain dot product
    of raw embeddings, softmaxes its row of scores into weights, and
    replaces its own vector with the weighted average of ALL the vectors.
    """
    # scores[i, j] = how similar word i's embedding is to word j's.
    scores = X @ X.T

    # Each row of scores becomes a row of weights that sums to 1.
    weights = softmax(scores)

    # Each output row is a weights-blended average of the embedding rows.
    return weights @ X, weights

Reading it with Module 0's tools: X @ X.T is every pairwise dot product at once (a 3x3 score table for a 3-word sentence). softmax runs on each row independently, so each word gets its own blending shares. And weights @ X computes all three weighted averages in one matrix multiplication: row i of the output is word i's shares applied to the embedding rows. Watch the shapes tell the story: X is words x axes (3 x 4); X @ X.T sums over the axes and leaves words x words (3 x 3, who resembles whom); weights @ X sums over the ingredient words and lands back on words x axes (3 x 4), the same shape as X, one context-aware row per word. This is called self-attention because the sentence scores and blends itself: the same matrix X plays both the asker and the answerer.

Run it on "the river bank"

The script runs simple_attention on both sentences, and recomputes the intermediate score matrix so we can watch the mechanism think:

    out_river, W_river = simple_attention(X_river)
    out_money, W_money = simple_attention(X_money)

    S_river = X_river @ X_river.T

Here is that matrix, along with a symmetry check (cross-check: results.log lines 29-34):

  score matrix S = X @ X.T for "the river bank":
                the   river    bank
    the       1.000   0.000   0.000
    river     0.000   0.900   0.540
    bank      0.000   0.540   0.990
  symmetric? True (S[i,j] always equals S[j,i] here)

Let's go deeper. Each row belongs to the word doing the looking; each column to the word being looked at. Bank's row holds exactly the three dot products you hand-checked in Module 0: 0.000 against "the", 0.540 against "river", 0.990 against itself. The diagonal (1.000, 0.900, 0.990) is every word's self-match, the biggest entry in every row, just as Module 0's "a vector always agrees with itself" fact predicted. And the table is symmetric: "river looking at bank" equals "bank looking at river" (both 0.540), because a dot product does not care about order; the capture's last line is the run checking exactly that. File both facts away; they become Module 2's opening problems.

Softmaxing each row turns the scores into the attention weights, for both sentences (cross-check: results.log lines 35-44):

  attention weights (each row softmaxed) for "the river bank":
                the   river    bank
    the       0.576   0.212   0.212
    river     0.193   0.475   0.332
    bank      0.185   0.317   0.498
  attention weights for "the money bank":
                the   money    bank
    the       0.576   0.212   0.212
    money     0.213   0.483   0.305
    bank      0.195   0.280   0.525

Let's go deeper on these two tables, because almost everything that matters is visible in them:

• Rows sum to 1, columns do not. Each row is one word's blending recipe; the columns are just where the shares happened to land.
• Bank's river-sentence row is [0.185, 0.317, 0.498]: exactly the softmax([0.000, 0.540, 0.990]) you verified by hand in Module 0. The mechanism contains no arithmetic you have not already done on paper.
• Bank's recipe differs between the sentences: 31.7% from "river" in one, 28.0% from "money" in the other. The recipe adapts because the scores came from whoever is actually in the sentence. This is the entire trick.
• Every word's largest share is still itself ("the": 57.6%, bank: 49.8% and 52.5%). The self-match fact caps how much a word can borrow. Hold that thought.

Predict. Bank's river-sentence recipe is 18.5% "the", 31.7% "river", 49.8% itself, and bank walks in with water and money perfectly balanced at 0.3 each. Before reading on: which axis wins in bank's output, and roughly how lopsidedly?

The payoff: one row in, two meanings out

Time to run the last line, weights @ X: the 3 x 3 recipe table you just read, applied to the 3 x 4 matrix of embedding rows from the top of the module. Out come three new vectors, one per word, still on the four named axes; row i is word i's recipe applied to the embedding rows. Bank's is the one we care about, for both sentences (cross-check: results.log lines 45-49):

  bank's output vector (water, money, place, filler):
    in "the river bank": water=0.435  money=0.149  place=0.543  filler=0.185
    in "the money bank": water=0.158  money=0.409  place=0.501  filler=0.195
  water vs money, river sentence: 0.435 vs 0.149
  water vs money, money sentence: 0.158 vs 0.409

The prediction lands: bank went in balanced at 0.3 water, 0.3 money, and came out watery in the river sentence (0.435 vs 0.149, nearly 3 to 1) and money-ish in the money sentence (0.409 vs 0.158). The figure tells the same story geometrically: the one gray lookup-table entry splits into two red diamonds, each dragged toward the topic word that was actually present.

The blend in slow motion, term by term

Where exactly did bank's new water-ness come from? Walk the river-sentence blend one term at a time, starting from what went in: bank's row of the embedding table, the same row in every sentence it will ever appear in:

Read as a sentence, this row says: "I might be the water kind of bank, I might be the money kind, no evidence either way (0.3 each), and I am definitely a place."

Bank's recipe is [0.185, 0.317, 0.498] over ["the", "river", "bank"] (its row of the weight table above), and the ingredients are the three rows of X_river. Computing bank's row of weights @ X takes exactly two steps, both doable on paper:

1. Scale. Multiply each ingredient's whole row by bank's weight for that word. River contributes 31.7% of everything it is: 31.7% of its 0.9 water, 31.7% of its 0.3 place, 31.7% of its zero money. Each scaled row is one row of the table below.
2. Add. Sum the three scaled rows column by column. The bottom row of the table is that sum: bank's new vector, z_bank.

Column by column:

• water: the only water on offer besides bank's own comes from river's 0.9, and 31.7% of it (0.285) pours in; bank's own 0.3 survives as 0.149. River supplies about two-thirds of the output's water-ness.
• money: nobody in this sentence brings money-ness except bank itself, and bank kept only 49.8% of its own vector, so its 0.3 is diluted to 0.149. Money-ness had no ally, so it shrank: an axis no ingredient reinforces can only fade.
• place: both river (0.095) and bank (0.448) chip in; bank stays strongly a place.
• filler: the 0.185 is exactly "the"'s share of the recipe, since [0, 0, 0, 1] has nothing else to give. It is the tax softmax paid to a word with zero content overlap.

(The same twelve multiplications can be grouped the other way: each cell of z_bank is bank's recipe dotted with one column of the ingredients. Water column [0.0, 0.9, 0.3]: 0.185 x 0.0 + 0.317 x 0.9 + 0.498 x 0.3 = 0.435. Rows first or columns first, matrix multiplication is nothing but these twelve products.)

One note on the last digit: the weights are shown rounded to three decimals, so a hand-recomputed column can drift by one (water: 0.285 + 0.149 = 0.434 against the full-precision 0.435). The exact sums are the run's output, the row you already saw in the payoff capture above.

Now the before and the after, side by side:

v_bank said "ambiguously both kinds of bank"; z_bank says "given that 'river' is standing next to me, probably the water kind". The lookup row stores the word's general identity, identical in every sentence ever written; the blended row stores that identity mixed with information from this particular sentence. A vector that depends on context is called a contextual representation, and you have just computed one with three lines of NumPy and no training whatsoever.

To make the pattern yours, rebuild this table for "the money bank": bank's recipe there is [0.195, 0.280, 0.525] over ["the", "money", "bank"] (its row of the money-sentence weight table above), against the rows of X_money. Check your column sums against the money-sentence line of the payoff capture; quiz question 6 asks about what you find.

What is still missing

This mechanism works, but look again at the weight tables and two shortfalls stand out, both of which we just measured:

1. Self-focus. Every word's biggest share is itself, always, because a vector cannot lose a dot-product contest against itself. "bank" could only manage 31.7% on "river". A word can never learn to look mostly elsewhere, no matter how much the sentence demands it.
2. Symmetry. river-looking-at-bank always equals bank-looking-at-river. But real relevance is not symmetric: "river" massively changes what "bank" means, while "bank" barely changes what "river" means.

Both problems trace to the same root: a word can only ask "who looks like me?", because its one embedding vector plays every role in the score. Module 2 gives each word two extra faces, a question it asks and an advertisement it displays, and both problems fall.

Next: Module 2: Queries, keys, and one carefully chosen constant