All technical posts
All personal posts

« Back to Home

Why attention has an output projection: heads add into the residual stream

21 Nov 2026

In multi-head attention, each head produces its own output, the outputs are concatenated, and the result is multiplied by an output projection called WO. Why is that matrix needed? One way to see it is as mixing the heads: every output coordinate is a learned combination of all the heads. Another way is as additive writes, and the two views are the same matrix multiplication. The short version: the heads are independent and additive. Each head adds its own contribution into the residual stream, and WO only decides where in the stream that contribution lands.

This post works through it with explicit shapes and a small numeric example (4-dimensional model, two heads, two tokens, checked with numpy). The framing follows A Mathematical Framework for Transformer Circuits by Elhage et al.

Download this post as a PDF

The big picture

Residual stream (X, n × d): every head reads it, and every head adds its own write back into it X Xnext = X + Σh Zh·WO,h head 1 own Q, K, V weights pattern P1 (n × n) Z1 = P1·X·Wv1 n × k slice WO,1 k × d + head 2 own Q, K, V weights pattern P2 (n × n) Z2 = P2·X·Wv2 n × k slice WO,2 k × d + … head H own Q, K, V weights pattern PH ZH = PH·X·WvH n × k slice WO,H k × d + The single matrix WO ((H·k) × d) is just these slices stacked on top of each other: WO,1 (rows 1 to k) WO,2 (next k rows) WO,H (last k rows) concatenate-then-project, [Z1 | Z2 | … | ZH]·WO, is the same as each head writing Zh·WO,h and adding the writes up. Each head works on its own inside the layer; the heads meet in the residual stream.
Every head reads the residual stream, computes its own output Zh, projects it with its own slice of WO, and adds the result back into the stream. Inside the layer each head works on its own; the heads meet in the stream.

Contents

Notation (read this first)

Convention: one row per token. Matrices multiply on the right: a token's vector is a row, and X · W applies W to every row.

SymbolMeaningShape
nnumber of tokens in the sequence–
dmodel width (the size of the residual stream)–
H, knumber of heads; width of one head (k = d / H in most models, so H·k = d)–
Xlayer input: one row per token (the residual stream at this layer)n × d
Wqh, Wkh, Wvhhead h's query, key and value weightsd × k
Phhead h's attention pattern: softmax(X·Wqh · (X·Wkh)T / √k + causal mask). Row i = how token i spreads its attention over earlier tokens. Depends on head h's Q and K only.n × n
Zhhead h's output: Ph · X · Wvh (a weighted average of value vectors)n × k
[Z1 | Z2 | …]concatenation: place the Z's side by side (columns)n × (H·k)
WOthe output projection, one matrix for the whole layer(H·k) × d
WO,hrows (h-1)·k+1 … h·k of WO: head h's own slice; stacking WO,1 on top of WO,2 … gives WOk × d
OVhWvh · WO,h: what head h does to a token vector, end to endd × d, rank ≤ k

The layer, standard form and the equivalent form

Standard form (how it is implemented, one big matmul):
Out = [Z1 | Z2 | … | ZH] · WO   (n × d), then added to the residual stream: Xnext = X + Out.
Equivalent form (the key identity):
Out = Z1·WO,1 + Z2·WO,2 + … + ZH·WO,H = Σh Ph · X · OVh

Why it holds (2 heads): splitting a product by blocks. A row [a b] times a stacked matrix [U; V] equals a·U + b·V, because each output entry is a sum over the inputs' positions, and the sum over the first k positions and the sum over the last k positions just add. So [Z1|Z2]·[WO,1; WO,2] = Z1WO,1 + Z2WO,2. The second equality uses Zh WO,h = Ph X (Wvh WO,h).

Toy numbers (d = 4, H = 2, k = 2, n = 2 tokens)

Head 1 is blue (columns 1-2 of the concatenation, rows 1-2 of WO), head 2 is orange. Patterns are given rather than computed: P1 lets token 2 average tokens 1 and 2; P2 lets each token look only at itself.

1021
0112
X
10
0.50.5
P1
10
01
P2
10
01
11
00
Wv1 (4x2)
01
10
00
11
Wv2 (4x2)

Head outputs Zh = Ph·X·Wvh (each n × k):

32
22
Z1
12
32
Z2

Way A, concatenate then project: [Z1|Z2] · WO

3212
2232
[Z1 | Z2]
·
1010
0101
1100
001-1
W_O (4x4): top 2 rows = W_O,1, bottom 2 rows = W_O,2
=
4350
5540
Out

Way B, each head writes its own contribution, then add: Z1WO,1 + Z2WO,2

3232
2222
Z1 · W_O,1
+
112-2
332-2
Z2 · W_O,2
=
4350
5540
Out (same as A)

Way C, with the single OV matrices: Wvh·WO,h (4 × 4, but rank 2 each):

1010
0101
1111
0000
OV1
001-1
1100
0000
111-1
OV2

and P1·X·OV1 + P2·X·OV2 also gives Out exactly (verified).

What this means

  • Each head works on its own inside a layer, and WO combines them. Head h's pattern uses only its own Q and K, and its output goes into the sum linearly. Read through WO, the sum is also a mixing of the heads' outputs: every output coordinate is a learned combination of all the heads. The heads meet in the residual stream, where a later head reads what earlier heads wrote (the paper calls this Q-, K- and V-composition).
  • WO decides where each head writes. Without WO, the concatenation puts head 1 in coordinates 1-2 and head 2 in coordinates 3-4, always. In the toy: head 1's write would be Z1 in columns 1-2 only. With WO,1 it is [3 2 3 2; 2 2 2 2]: a different 2-dimensional slice of the 4-dimensional residual stream, free to overlap with other heads' slices and with what the MLPs and embeddings write.
  • Only the product OVh = Wvh·WO,h matters. You can replace Wvh by Wvh·R and WO,h by R-1·WO,h for any invertible k × k matrix R and the model is identical (verified in the toy with R = [[1,1],[0,1]]). So "V" and "O" are one linear map split in two for dimension bookkeeping: d → k, then k → d.
  • Same split on the other side: the QK circuit is Wqh·WkhT (d × d, rank ≤ k). It decides where head h looks; the OV circuit decides what it moves.

Two intuitions, scored

IdeaVerdict
"With separate queries per head there is less cross-mixing, so WO compensates."WO does mix the heads' outputs into one vector per token, and this is the same thing as each head adding its own write. Cross-head effects on the attention patterns arise in later layers, which read the residual stream.
"It is like the last layer of an MLP: heads are hidden neurons, WO is the down-projection."Half right. Both are parallel units summed into one shared space through a learned output matrix. But a head writes a rank-k chunk whose content depends on the attention pattern, not a scalar times a fixed vector.

Caveats

  • I did not find (and am not citing) a clean experiment measuring how much quality is lost if WO is dropped. In pure linear algebra a later reader could absorb a change of basis, so WO is a freedom to choose the write subspace, not a mathematical necessity. The shared basis matters in practice because MLP activations and normalisation are not basis-independent.
  • The paper analyses attention-only transformers and mostly ignores MLPs and LayerNorm. It does not cover RoPE, GQA, MLA or KV-cache cost.
  • In llama.cpp, WO is the single attn_output matmul on the concatenated head outputs; the per-head split above is for understanding only.

Suggested reading: the paper's sections on the residual stream and on "attention heads are independent and additive", and the QK/OV circuit decomposition. All toy numbers were checked in numpy: Way A, B and C agree exactly, and the R-gauge check passes.

« Back to Home


Comments