Module 3: A network that cannot learn

The prediction

Module 2 measured layer 1's gradient at 3,015x smaller than the output layer's before training began. Whatever learning rate we pick, the early layers will move thousands of times more slowly than the late ones — so the features the late layers depend on stay, for practical purposes, frozen at random. The prediction: the network cannot beat chance, and its loss will sit at the ln 2 ≈ 0.693 "learned nothing" score that Module 0 measured for a network outputting p = 0.5 everywhere.

The run

The training recipe, from experiments/experiment.py — plain minibatch SGD, nothing exotic:

opt = torch.optim.SGD(net.parameters(), lr=0.5)
for step in range(1, 4001):
    idx = torch.randint(0, N_TRAIN, (64,), generator=g)
    loss = bce(net(X_train[idx]), y_train[idx])
    opt.zero_grad()
    loss.backward()
    opt.step()

[S6] training runs (SGD, lr = 0.5, batch = 64, 4000 steps)
plain             final train loss = 0.6926, final test acc = 51.5%, first >=90%: never
with batch norm   final train loss = 0.0053, final test acc = 100.0%, first >=90%: step 100
chance accuracy = 50.0% | 'learned nothing' loss = ln 2 = 0.6931

(Captured in captures/results.log lines 63–66.) Read the plain row against the baseline row below it: after 4,000 optimization steps — 320 passes over the training set — the plain network's loss is 0.6926. The know-nothing score is 0.6931. It has clawed back 0.0005 of loss. Its test accuracy, 51.5%, is a coin flip. The batch-norm row is the same architecture, same starting weights (Module 1's checksum), same recipe — it is on this page only as a spoiler; Module 4 gives it its due.

The red line is Module 0's prediction made visible: flat at ln 2 from the first hundred steps to the last. This is the fingerprint to remember. A loss stuck at ln 2 on a balanced binary task (or ln K on a K-class task) does not mean "slow convergence" — it means no gradient is reaching the parts of the network that would have to change.

One honest note about the red accuracy curve: it flickers around 50% and twice spikes toward ~79% before collapsing back within a few evaluations. The output layer's handful of parameters do receive healthy gradients (they are at the top of the relay), so they occasionally latch onto something — but the features beneath them never improve, so the gains do not hold. Do not read those blips as learning; the loss curve shows none happened.

What did each network actually build? Evaluating both trained networks over a grid of the input plane makes it concrete:

On the right, batch norm carved the S-shaped boundary the moons demand. On the left, the plain network's output is a nearly uniform p ≈ 0.5 across the entire plane — so flat that the contour plotter finds almost nothing to draw. That washed-out field is the geometric meaning of "stuck at ln 2."

The tempting fix: just push harder

If gradients are 3,000x too small, why not a bigger learning rate? Let us actually try it — same plain network, learning rates 0.5, 5, and 50:

depth 4 (the net from S6):
  plain, lr = 5.0    final train loss = 0.0017, final test acc = 100.0%, first >=90%: step 1050
  plain, lr = 50.0   final train loss = 5.7024, final test acc = 48.5%, first >=90%: never
  (lr = 0.5 = the S6 run: final test acc = 51.5%)

(Captured in captures/results.log lines 72–75.) An honest result, and it cuts both ways. At lr = 5, the plain 4-layer network does eventually learn — 100% accuracy. Vanishing gradients at this depth mean slow, not impossible: with 10x more force, the top layers start moving, their weights grow, and growing weights raise the very hop factors that were strangling the signal (Module 2's W shrink column is not a constant of nature — it tracks the weights), so the network slowly bootstraps itself out of the hole. But look at the cost. It first touches 90% at step 1050 — batch norm got there at step 100 at the original learning rate — and the curve is a cliffhanger:

Nothing happens for a thousand steps, then the orange line jumps. And the window is narrow: at lr = 50 the training loss rises to 5.70 — far above the know-nothing 0.693 — because steps that large fling the well-conditioned top layers past every minimum. 48.5% accuracy: the crank has a cliff on both sides.

Why brute force cannot scale: depth 8

The real verdict comes from making the problem modestly deeper. Module 2's mechanism says every added layer multiplies another ~0.1 into the gradient ratio, so depth 8 should be roughly 10,000x worse than depth 4 — beyond any learning rate's reach. Measured:

depth 8 (same width, same recipe):
  init gradient ratio L1/out: plain = 4.97e-07, batch norm = 3.37e-01
  plain, lr = 0.5    final train loss = 0.6932, final test acc = 51.5%, first >=90%: never
  plain, lr = 5.0    final train loss = 0.7033, final test acc = 48.5%, first >=90%: never
  plain, lr = 50.0   final train loss = 5.1390, final test acc = 48.5%, first >=90%: never
  batch norm, lr = 0.5  final train loss = 0.0175, final test acc = 100.0%, first >=90%: step 50

(Captured in captures/results.log lines 78–83.) Read it with me: at depth 8 the plain network's first layer starts with gradients two million times smaller than its output layer (ratio 4.97e-07 — close to the 10⁻⁶-ish extrapolation). The learning rate that rescued depth 4 now does nothing (lr = 5: loss 0.7033, still the know-nothing plateau); the bigger crank still explodes (lr = 50: loss 5.14). Meanwhile batch norm's starting ratio is 0.34 — the depth barely dented it — and the same lr = 0.5 recipe crosses 90% within 50 steps:

This is the module's takeaway, and it is the difference between a workaround and a fix. The learning-rate crank fights the symptom (small numbers) and loses to the exponent: every two extra layers cost another ~100x. Batch norm changes the mechanism — it holds the per-layer factor near 1 (Module 4 measures exactly how), so depth stops compounding at all.