Final Exam: LLM Self-Attention by Hand

1. [Module 0] A large positive dot product between two word vectors tells you that:

The two words usually appear next to each other in text The two vectors have large values on the same axes, i.e. they point in similar directions Both vectors contain only positive numbers The two vectors have the same number of dimensions

2. [Module 0] Four words all score exactly 7.0. What share does softmax give each of them? (decimal or fraction both fine, e.g. 0.10 or 1/10)

3. [Module 0] Which statement about softmax is FALSE?

The output shares always sum to exactly 1 A bigger score always gets a bigger share A score of exactly 0 gets a share of exactly 0 Multiplying all scores by 5 makes the largest share bigger whenever the scores differ

4. [Module 0] In the toy world, bank's highest dot product (0.990) is with itself, beating even "river" (0.540). Why?

"bank" happens to be the longest vector in the table A vector dotted with itself sums the squares of its entries, so every axis matches perfectly The table was hand-designed to make this happen; learned embeddings behave differently Softmax boosts the diagonal of every score matrix

5. [Module 1] In simple_attention, the expression X @ X.T computes:

Every pairwise dot-product score between words in the sentence The blended output vectors The attention weights, ready to use A normalized copy of the embeddings

6. [Module 1] True or false: in "the money bank", the input embedding row for "bank" is different from its row in "the river bank".

True False

7. [Module 1] After simple attention on "the river bank", bank's output was watery (water 0.435 vs money 0.149). Where did the extra water-ness come from?

The embedding table's "bank" row was updated for this sentence The blend mixed in 31.7% of river's vector, which carries 0.9 on the water axis Softmax suppressed the money axis of every word The water axis is listed first, so it receives the largest weight

8. [Module 1] In an attention weight matrix, every row sums to 1 but columns do not. Why the rows?

The embeddings are normalized to length 1 before scoring Each row is one word's scores passed through softmax, which always produces shares summing to 1 The score matrix is symmetric, which forces the rows to balance NumPy's @ operator normalizes its output rows

9. [Module 2] With queries and keys, the score q_i . k_j is best read as:

How similar the identities of words i and j are How well word j's advertisement answers word i's question How close words i and j are in the sentence How likely word j is to follow word i in real text

10. [Module 2] True or false: even with separate query and key matrices, the score matrix must stay symmetric (score(i,j) = score(j,i)).

True False

11. [Module 2] Under the hand-wired query/key head, which word receives the LARGEST share of bank's attention in "the river bank"?

bank itself, as always river the It splits exactly evenly

12. [Module 2] The real reason attention divides scores by sqrt(d_k) is:

To keep the weights positive To make the score matrix invertible Dot-product spread grows like sqrt(d_k), and unscaled scores push softmax into winner-take-all; the division cancels the growth To make the computation faster on GPUs

13. [Module 2] Queries and keys have 100 dimensions with unit-variance entries. By the workshop's measured growth law, the typical spread (standard deviation) of the raw scores is about: (a whole number)

14. [Module 3] The causal mask sets future scores to negative infinity rather than 0 because:

A score of 0 still gets a positive softmax share (e^0 = 1); only negative infinity produces an exactly-zero share Zero would divide by zero inside softmax Negative infinity is faster for the hardware to process It prevents the exponential from overflowing

15. [Module 3] Under a causal mask, how many words (including itself) can the FIRST word of any sentence attend to? (a whole number)

16. [Module 3] True or false: the measured 2.22e-16 difference between the NumPy head and PyTorch's implementation indicates a small bug in one of them.

True False

17. [Module 3] The workshop measured that attention alone gives "bank" the identical output in "the river bank" and "bank river the". The root cause is:

Softmax sorts the scores before normalizing them Scores depend only on the content of the vectors; no part of the computation uses a word's position The causal mask removes word-order information The toy embeddings are too small to encode order

18. [Module 3] Inside a transformer block, the only place where information moves BETWEEN words is:

The MLP The tokenizer The self-attention blend The final next-token softmax