Log: results.log
The complete captured output of one deterministic run (seed 0) of
the experiment script. Every number in the modules is already
reproduced on the module pages; this page exists for cross-checking. The line numbers on
the left match the modules' "(cross-check: results.log lines 22-23)" notes, so any of
them can be looked up directly. The raw file lives at captures/results.log in the
workshop directory, and re-running the script regenerates it byte for byte.
1 ========================================================================
2 LLM Self-Attention by Hand -- experiment log
3 torch 2.12.1+cpu | numpy 2.2.6 | matplotlib 3.10.6 | python 3.10.12 | seed 0
4 toy world: 4-word vocabulary, 4 hand-named embedding axes
5 ========================================================================
6
7 [S1] the toy world
8 embedding axes (hand-named): water, money, place, filler
9 word water money place filler
10 the 0.0 0.0 0.0 1.0
11 river 0.9 0.0 0.3 0.0
12 money 0.0 0.9 0.1 0.0
13 bank 0.3 0.3 0.9 0.0
14 test sentences: "the river bank" / "the money bank"
15 note: 'bank' has exactly ONE row here, whichever sentence it is in
16
17 [S2] dot product and softmax, on their own
18 bank . river = 0.3*0.9 + 0.3*0.0 + 0.9*0.3 + 0.0*0.0 = 0.540
19 bank . money = 0.360
20 bank . the = 0.000
21 bank . bank = 0.990
22 softmax([0.000, 0.540, 0.990]) = [0.185, 0.317, 0.498] (sum = 1.000000)
23 softmax(same scores x 5) = [0.006, 0.095, 0.899] (one clear winner)
24 [plot saved: diagrams/dot_product_arrows.png]
25 [plot saved: diagrams/softmax_in_action.png]
26
27 [S3] attention with raw embeddings (no learned parts)
28 input check: 'bank' row identical in both sentences -> True
29 score matrix S = X @ X.T for "the river bank":
30 the river bank
31 the 1.000 0.000 0.000
32 river 0.000 0.900 0.540
33 bank 0.000 0.540 0.990
34 symmetric? True (S[i,j] always equals S[j,i] here)
35 attention weights (each row softmaxed) for "the river bank":
36 the river bank
37 the 0.576 0.212 0.212
38 river 0.193 0.475 0.332
39 bank 0.185 0.317 0.498
40 attention weights for "the money bank":
41 the money bank
42 the 0.576 0.212 0.212
43 money 0.213 0.483 0.305
44 bank 0.195 0.280 0.525
45 bank's output vector (water, money, place, filler):
46 in "the river bank": water=0.435 money=0.149 place=0.543 filler=0.185
47 in "the money bank": water=0.158 money=0.409 place=0.501 filler=0.195
48 water vs money, river sentence: 0.435 vs 0.149
49 water vs money, money sentence: 0.158 vs 0.409
50 [plot saved: diagrams/raw_attention_heatmaps.png]
51 [plot saved: diagrams/bank_water_money_plane.png]
52
53 [S4] a hand-wired query/key head
54 queries and keys for "the river bank" (2 channels: topic, place):
55 the q = [ 0.00, 1.50] k = [ 0.00, 0.00]
56 river q = [ 0.60, 0.00] k = [ 0.90, 0.30]
57 bank q = [ 1.80, 0.00] k = [ 0.60, 0.90]
58 score matrix Q @ K.T:
59 the river bank
60 the 0.000 0.450 1.350
61 river 0.000 0.540 0.360
62 bank 0.000 1.620 1.080
63 symmetric? False (bank->river = 1.620, river->bank = 0.360)
64 weights after dividing by sqrt(2) and softmaxing each row:
65 the river bank
66 the 0.201 0.276 0.522
67 river 0.266 0.390 0.344
68 bank 0.159 0.500 0.341
69 bank's row, raw vs wired: [0.185, 0.317, 0.498] -> [0.159, 0.500, 0.341]
70 'the's row, raw vs wired: [0.576, 0.212, 0.212] -> [0.201, 0.276, 0.522]
71 bank now takes half its update from 'river'; 'the' now looks at its noun
72 bank's output vector: water=0.552 money=0.102 place=0.457 filler=0.159
73 [plot saved: diagrams/qk_vs_raw_heatmaps.png]
74
75 [S5] why scores are divided by sqrt(d_k)
76 random q, k entries drawn from N(0,1); 8 keys per row; 2000 rows per size
77 d = 4: dot-product std = 1.97 (sqrt(d) = 2.00) | avg top weight: unscaled = 0.522, scaled = 0.338
78 d = 16: dot-product std = 4.02 (sqrt(d) = 4.00) | avg top weight: unscaled = 0.750, scaled = 0.356
79 d = 64: dot-product std = 8.01 (sqrt(d) = 8.00) | avg top weight: unscaled = 0.879, scaled = 0.362
80 d = 256: dot-product std = 16.03 (sqrt(d) = 16.00) | avg top weight: unscaled = 0.941, scaled = 0.366
81 unscaled top weight heads for 1.0 (winner-take-all); scaled stays flat
82 [plot saved: diagrams/scaling_max_weight.png]
83
84 [S6] the general head, checked against PyTorch
85 ours vs torch.nn.functional.scaled_dot_product_attention:
86 max |difference| = 2.22e-16
87 same comparison with the causal mask on both sides:
88 max |difference| = 2.22e-16
89 wired head with a causal mask (upper right forced to 0):
90 the river bank
91 the 1.000 0.000 0.000
92 river 0.406 0.594 0.000
93 bank 0.159 0.500 0.341
94 every row still sums to 1: 1.000000, 1.000000, 1.000000
95 order-blindness: bank's output in "the river bank" equals its
96 output in "bank river the" -> True (real LLMs add position information)
97 one GPT-2 sized head: d_model = 768, d_k = 64 -> W_q + W_k + W_v = 147,456 numbers
98 12 heads x 12 layers (+ output projections) -> 28,311,552 attention parameters of GPT-2's ~124M
99 [plot saved: diagrams/causal_mask.png]
100