Module 2: Watch the gradients vanish

First, why is there a "backward" signal at all?

Training asks one question of every weight in the network: if this weight nudged up a little, would the loss go down? Nudge in whichever direction shrinks the loss, repeat. The gradient is just the vector of answers.

For the output layer the answer is direct — Module 0 measured it: the loss responds to the final logit at rate p − y. But consider a weight in hidden layer 1. Its nudge changes layer 1's output, which changes layer 2's output, which changes layer 3's, and so on to the loss. The influence is real but indirect, relayed through every layer above.

Indirect influence multiplies, and it pays to see why with plain numbers before any formulas. Suppose a 1-unit wiggle in layer 3's activation moves layer 4's activation by 0.2 units (layer 4 responds at rate 0.2), and a 1-unit wiggle in layer 4 moves the loss by 0.1. Then a 1-unit wiggle in layer 3 moves the loss by 0.2 × 0.1 = 0.02 — the rates multiply, exactly like currency conversions chained through two exchanges. That is all the chain rule is. In derivative notation, for our stack where each layer's activation h_k feeds the next:

∂L/∂h₍ₖ₋₁₎ = ∂L/∂hₖ · ∂hₖ/∂h₍ₖ₋₁₎

The backward pass starts at the loss with the healthy signal p − y and works down, multiplying in one layer-crossing rate ∂h_k/∂h_{k−1} per layer. Each crossing is a sigmoid layer, h_k = σ(W_k h_{k−1} + b_k), so each rate contains two ingredients you have already met:

• the sigmoid's slope σ′(z_k) — at most 0.25 (measured, Module 0), and
• the weight matrix W_k — whose entries started at std 0.31 (measured, Module 1).

A slope that cannot exceed 0.25 times a smallish weight matrix makes each layer-crossing rate well below 1. And sub-unity factors compound geometrically. Before measuring the real network, here is the pure arithmetic of that compounding — no network involved, just factor ** depth:

factor 0.9: after 4 layers -> 6.56e-01, after 10 layers -> 3.49e-01
factor 0.5: after 4 layers -> 6.25e-02, after 10 layers -> 9.77e-04
factor 0.25: after 4 layers -> 3.91e-03, after 10 layers -> 9.54e-07
factor 0.1: after 4 layers -> 1.00e-04, after 10 layers -> 1.00e-10

(Captured in captures/results.log lines 57–60.) Read the bottom row: if each layer keeps only a tenth of the signal, four layers keep a ten-thousandth, and ten layers keep a ten-billionth. On the plot's log scale each factor is a straight line — geometric decay means the exponent grows with depth. Hold onto the 0.1 row; the plain network is about to land almost exactly on it.

How we measure it

Everything below comes from one forward and one backward pass of the untrained plain network on the full 800-point training batch — no optimizer, no training, just autograd. Two details make the per-layer readout possible. After loss.backward(), PyTorch stores gradients on parameters automatically (weight.grad), which gives us the per-layer weight gradient ||dL/dW_k||. But the traveling signal — the gradient with respect to each layer's activation, ||dL/dh_k|| — lives on intermediate tensors, and autograd normally discards those gradients to save memory. Asking it to keep them is one call, made inside the network's forward pass when recording is on:

z = nrm(lin(h))
a = self.act(z)
if record:
    z.retain_grad()
    a.retain_grad()

Then the measurement is a single pass (from measure_at_init in experiments/experiment.py):

out = net(X_train, record=True)
loss = bce(out, y_train)
loss.backward()
# per layer: net.hidden[k].weight.grad.norm(), rec["h"][k].grad.norm(), ...

The measurement: a cliff

PLAIN (Linear -> sigmoid): loss at init = 0.7784
  layer    ||dL/dW||    ||dL/dh||  std(z_pre)   std(z)   std(h)  mean slope  W shrink
      1     6.63e-05     1.75e-05      0.3513   0.3513   0.0848      0.2420     0.767
      2     5.37e-04     1.83e-04      0.4514   0.4514   0.1087      0.2379     0.604
      3     4.01e-03     1.75e-03      0.5744   0.5744   0.1366      0.2313     0.623
      4     3.99e-02     1.47e-02      0.2321   0.2321   0.0575      0.2467     0.470
 output     2.00e-01
gradient ratio, layer 1 vs output: 3.32e-04  (3015x smaller)
backward shrink factor per layer (||dL/dh_k|| / ||dL/dh_k+1||): 0.096, 0.104, 0.120

(Captured in captures/results.log lines 33–41.) This table is the core measurement of the workshop, so let us read it column by column. ||dL/dW|| is the size of the gradient the optimizer will actually use to update that layer's weights: 0.2 at the output, 4e-2 at hidden layer 4, then 4e-3, 5e-4, 6.6e-5 — dropping by roughly 10x per layer, to a layer-1 signal 3,015 times smaller than the output's. ||dL/dh|| is the traveling backward signal itself, and the last line divides consecutive entries to expose the per-layer factor directly: 0.096, 0.104, 0.120. The relay loses about 90% of the signal at every hop.

On the log axis the plain net's line (red) is close to straight: near-constant factor per layer, exactly the geometric decay of the arithmetic plot above, landing on its 0.1 row. (The green line is the batch-norm network — same starting weights, measured in the same pass. It is barely a line at all; it is nearly flat. It waits for Module 4.)

The same cliff, drawn as the relay it is — each box holds the measured signal size ||dL/dh_k|| from the table, each arrow the measured hop factor:

Where does 0.1 per hop come from?

The chain-rule crossing for h_k = σ(z_k), z_k = W_k h_{k−1} + b_k is (entry-wise slope, then matrix):

∂hₖ/∂h₍ₖ₋₁₎ = diag(σ′(zₖ)) · Wₖ

So each hop's factor is roughly (mean sigmoid slope) × (how much W_k scales a vector). Both are in the table. Take the hop from h2 down to h1, which crosses layer 2: mean slope 0.2379, and the W shrink column — the root-mean-square singular value of W_2, i.e. the typical scaling a random direction receives — is 0.604. Their product predicts a factor of about 0.14; the measured factor is 0.096. Same ballpark (the estimate is crude because a random 4-wide matrix shrinks most specific directions somewhat more than its average suggests), and the structure of the answer is what matters:

hop factor ≈ slope × weight scale ≈ 0.24 × 0.5 ≈ 0.1

Neither ingredient alone would be fatal at this depth — 0.6 per layer over four layers is only a 7x total loss. It is the product of two sub-unity numbers, compounded over four layers, that digs the 3,000x hole.

The honest surprise: it is not saturation

Module 0 planted a suspicion: sigmoids have flat, saturated zones where the slope is ~0, and the standard textbook account of vanishing gradients blames them. If saturation were the culprit here, the mean slope column would show values near zero.

Look at it again: 0.2420, 0.2379, 0.2313, 0.2467. The theoretical maximum is 0.25. Our sigmoids are operating within 8% of the steepest they can possibly be — the exact opposite of saturated. The reason is in the std(z) column: the pre-activations have spread 0.23–0.57, so nearly all of them sit in the sigmoid's steep central region around z = 0. (Why so central? Small weights — std 0.31 — times order-one inputs give small z. And it compounds: std(h) shows each layer's output spread collapsing to 0.06–0.14, because a sigmoid near z = 0 has slope ≈ 1/4, squashing the already-small spread further before the next small-weight layer shrinks it again. The forward signal is dying in parallel with the backward one.)

So for this network, at initialization, the mechanism is scale, not saturation: maximally steep sigmoids still multiply in at most 0.25 per layer, and small random weights multiply in another ~0.5. The distinction is not pedantry — it predicts which fixes will work. If saturation were the problem, batch norm's job would be to move z's toward the steep region; Module 4 will show it does something quite different (and measurably makes the slopes smaller, yet fixes the network anyway).