Why attention has an output projection: heads add into the residual stream
21 Nov 2026
In multi-head attention, each head produces its own output, the outputs are concatenated, and the result is multiplied by an output projection called WO. Why is that matrix needed? One way to see it is as mixing the heads: every output coordinate is a learned combination of all the heads. Another way is as additive writes, and the two views are the same matrix multiplication. The short version: the heads are independent and additive. Each head adds its own contribution into the residual stream, and WO only decides where in the stream that contribution lands.
This post works through it with explicit shapes and a small numeric example (4-dimensional model, two heads, two tokens, checked with numpy). The framing follows A Mathematical Framework for Transformer Circuits by Elhage et al.
Every head reads the residual stream, computes its own output Zh, projects it with its own slice of WO, and adds the result back into the stream. Inside the layer each head works on its own; the heads meet in the stream.
Convention: one row per token. Matrices multiply on the right: a token's vector is a row, and X · W applies W to every row.
Symbol
Meaning
Shape
n
number of tokens in the sequence
–
d
model width (the size of the residual stream)
–
H, k
number of heads; width of one head (k = d / H in most models, so H·k = d)
–
X
layer input: one row per token (the residual stream at this layer)
n × d
Wqh, Wkh, Wvh
head h's query, key and value weights
d × k
Ph
head h's attention pattern: softmax(X·Wqh · (X·Wkh)T / √k + causal mask). Row i = how token i spreads its attention over earlier tokens. Depends on head h's Q and K only.
n × n
Zh
head h's output: Ph · X · Wvh (a weighted average of value vectors)
n × k
[Z1 | Z2 | …]
concatenation: place the Z's side by side (columns)
n × (H·k)
WO
the output projection, one matrix for the whole layer
(H·k) × d
WO,h
rows (h-1)·k+1 … h·k of WO: head h's own slice; stacking WO,1 on top of WO,2 … gives WO
k × d
OVh
Wvh · WO,h: what head h does to a token vector, end to end
d × d, rank ≤ k
The layer, standard form and the equivalent form
Standard form (how it is implemented, one big matmul): Out = [Z1 | Z2 | … | ZH] · WO (n × d), then added to the residual stream: Xnext = X + Out.
Equivalent form (the key identity): Out = Z1·WO,1 + Z2·WO,2 + … + ZH·WO,H = Σh Ph · X · OVh
Why it holds (2 heads): splitting a product by blocks. A row [a b] times a stacked matrix [U; V] equals a·U + b·V, because each output entry is a sum over the inputs' positions, and the sum over the first k positions and the sum over the last k positions just add. So [Z1|Z2]·[WO,1; WO,2] = Z1WO,1 + Z2WO,2. The second equality uses Zh WO,h = Ph X (Wvh WO,h).
Toy numbers (d = 4, H = 2, k = 2, n = 2 tokens)
Head 1 is blue (columns 1-2 of the concatenation, rows 1-2 of WO), head 2 is orange. Patterns are given rather than computed: P1 lets token 2 average tokens 1 and 2; P2 lets each token look only at itself.
Way B, each head writes its own contribution, then add:Z1WO,1 + Z2WO,2
3
2
3
2
2
2
2
2
Z1 · W_O,1
+
1
1
2
-2
3
3
2
-2
Z2 · W_O,2
=
4
3
5
0
5
5
4
0
Out (same as A)
Way C, with the single OV matrices:Wvh·WO,h (4 × 4, but rank 2 each):
1
0
1
0
0
1
0
1
1
1
1
1
0
0
0
0
OV1
0
0
1
-1
1
1
0
0
0
0
0
0
1
1
1
-1
OV2
and P1·X·OV1 + P2·X·OV2 also gives Out exactly (verified).
What this means
Each head works on its own inside a layer, and WO combines them. Head h's pattern uses only its own Q and K, and its output goes into the sum linearly. Read through WO, the sum is also a mixing of the heads' outputs: every output coordinate is a learned combination of all the heads. The heads meet in the residual stream, where a later head reads what earlier heads wrote (the paper calls this Q-, K- and V-composition).
WO decides where each head writes. Without WO, the concatenation puts head 1 in coordinates 1-2 and head 2 in coordinates 3-4, always. In the toy: head 1's write would be Z1 in columns 1-2 only. With WO,1 it is [3 2 3 2; 2 2 2 2]: a different 2-dimensional slice of the 4-dimensional residual stream, free to overlap with other heads' slices and with what the MLPs and embeddings write.
Only the product OVh = Wvh·WO,h matters. You can replace Wvh by Wvh·R and WO,h by R-1·WO,h for any invertible k × k matrix R and the model is identical (verified in the toy with R = [[1,1],[0,1]]). So "V" and "O" are one linear map split in two for dimension bookkeeping: d → k, then k → d.
Same split on the other side: the QK circuit is Wqh·WkhT (d × d, rank ≤ k). It decides where head h looks; the OV circuit decides what it moves.
Two intuitions, scored
Idea
Verdict
"With separate queries per head there is less cross-mixing, so WO compensates."
WO does mix the heads' outputs into one vector per token, and this is the same thing as each head adding its own write. Cross-head effects on the attention patterns arise in later layers, which read the residual stream.
"It is like the last layer of an MLP: heads are hidden neurons, WO is the down-projection."
Half right. Both are parallel units summed into one shared space through a learned output matrix. But a head writes a rank-k chunk whose content depends on the attention pattern, not a scalar times a fixed vector.
Caveats
I did not find (and am not citing) a clean experiment measuring how much quality is lost if WO is dropped. In pure linear algebra a later reader could absorb a change of basis, so WO is a freedom to choose the write subspace, not a mathematical necessity. The shared basis matters in practice because MLP activations and normalisation are not basis-independent.
The paper analyses attention-only transformers and mostly ignores MLPs and LayerNorm. It does not cover RoPE, GQA, MLA or KV-cache cost.
In llama.cpp, WO is the single attn_output matmul on the concatenated head outputs; the per-head split above is for understanding only.
Suggested reading: the paper's sections on the residual stream and on "attention heads are independent and additive", and the QK/OV circuit decomposition. All toy numbers were checked in numpy: Way A, B and C agree exactly, and the R-gauge check passes.