Module 2: Watch the gradients vanish

First, why is there a "backward" signal at all?

Training asks one question of every weight in the network: if this weight nudged up a little, would the loss go down? Nudge in whichever direction shrinks the loss, repeat. The gradient is just the vector of answers.

For the output layer the answer is direct; Module 0 measured it: the loss responds to the final logit at rate p − y. But consider a weight in hidden layer 1. Its nudge changes layer 1's output, which changes layer 2's output, which changes layer 3's, and so on to the loss. The influence is real but indirect, relayed through every layer above.

Indirect influence multiplies, and it pays to see why with plain numbers before any formulas. Suppose a 1-unit wiggle in layer 3's activation moves layer 4's activation by 0.2 units (layer 4 responds at rate 0.2), and a 1-unit wiggle in layer 4 moves the loss by 0.1. Then a 1-unit wiggle in layer 3 moves the loss by 0.2 × 0.1 = 0.02: the rates multiply, exactly like currency conversions chained through two exchanges. That is all the chain rule is. In derivative notation, for our stack where each layer's activation h_k feeds the next:

$$\frac{\partial L}{\partial h_{k-1}}=\frac{\partial L}{\partial h_k}·\frac{\partial h_k}{\partial h_{k-1}}$$

The backward pass starts at the loss with the healthy signal p − y and works down, multiplying in one layer-crossing rate ∂h_k/∂h_{k−1} per layer. Each crossing is a sigmoid layer, h_k = σ(W_k h_{k−1} + b_k), so each rate contains two ingredients you have already met:

• the sigmoid's slope σ′(z_k), at most 0.25 (measured, Module 0), and
• the weight matrix W_k, whose entries started at std 0.31 (measured, Module 1).

Make a prediction

A slope that cannot exceed 0.25, times a smallish weight matrix, makes each layer-crossing rate well below 1. And sub-unity factors compound geometrically. Before any measurement, commit to a number:

Predict. Suppose each layer crossing keeps about one tenth of the incoming signal. What fraction of the signal survives four crossings? Write it down (a power of ten is enough) before reading on.

The compounding is nothing but repeated multiplication:

0.1 after 1 crossing   -> 10%
0.1 after 4 crossings  -> 0.01%   (one ten-thousandth)
0.1 after 10 crossings -> one ten-billionth

Section 5's pass of the experiment script runs this same arithmetic for a range of factors (no network involved, just factor ** depth) and logs it (captures/results.log lines 58–61):

factor 0.9: after 4 layers -> 6.56e-01, after 10 layers -> 3.49e-01
factor 0.5: after 4 layers -> 6.25e-02, after 10 layers -> 9.77e-04
factor 0.25: after 4 layers -> 3.91e-03, after 10 layers -> 9.54e-07
factor 0.1: after 4 layers -> 1.00e-04, after 10 layers -> 1.00e-10

Read the bottom row: if each layer keeps only a tenth of the signal, four layers keep a ten-thousandth, and ten layers keep a ten-billionth. On the plot's log scale each factor is a straight line: geometric decay means the exponent grows with depth. Hold onto the 0.1 row; the plain network is about to land almost exactly on it.

Run the measurement

Everything below comes from one forward and one backward pass of the untrained plain network on the full 800-point training batch: no optimizer, no training steps, just autograd. The pass itself is four lines, verbatim from the script's measure_at_init (bce is the fused sigmoid-plus-cross-entropy loss from Module 0):

    net.zero_grad()
    out = net(X_train, record=True)
    loss = bce(out, y_train)
    loss.backward()

The record=True flag is the measurement hook, and its whole idea fits in three steps:

1. save each hidden layer's activation tensor as the forward pass computes it,
2. ask autograd to keep that tensor's gradient (for tensors in the middle of the network it is discarded by default), and
3. after loss.backward(), read the gradient sitting on each saved activation.

That yields the traveling signal: the size of ∂L/∂h_k at every depth. The per-layer weight gradients ∂L/∂W_k need no hook at all, because PyTorch always keeps gradients on parameters (weight.grad).

h = torch.sigmoid(layer(h))   # one hidden layer's activation
h.retain_grad()               # keep dL/dh here when backward() runs
saved.append(h)

# ... the forward pass continues up to the loss ...
loss.backward()

for h in saved:
    print(h.grad.norm())      # the backward signal at this layer

Observe the relay

When Section 4's pass of the experiment script runs the one-pass measurement above on the untrained plain network, this is the relay it finds. Each box holds the measured backward signal at that depth, each arrow the measured hop factor:

The same numbers as a table, read in the direction the signal travels (top row first). Each "retained" entry is that row's signal divided by the row above's: the measured per-hop conversion rate. (Distilled from the table Section 4's pass logs at captures/results.log lines 33–41; the full readout appears in the diagnosis below.)

Your prediction lands: the relay keeps roughly 10% per hop, losing roughly 90% at every crossing. (If you rerun the script in the exact environment of the reproducing section, seed 0 makes the log match byte for byte; change anything, the seed, the width, the batch, and every digit moves. The finding is the ratio structure, the roughly tenfold drop per hop, not the particular norms.)

The optimizer never sees ∂L/∂h directly; what it uses to update layer k is the weight gradient ∂L/∂W_k, and that inherits the same cliff, because it is assembled from the traveling signal arriving at layer k (same log table, ||dL/dW|| column):

The script divides the ends of that column for us (results.log line 40): layer 1's weight gradient is 3.32e-04 of the output layer's, 3,015 times smaller, before training has taken a single step. On a log axis the decline is a near-straight line, the signature of a near-constant factor per layer, landing on the 0.1 row of the prediction plot:

Where does 0.1 per hop come from?

The chain-rule crossing for h_k = σ(z_k), z_k = W_k h_{k−1} + b_k is (entry-wise slope, then matrix):

$$\frac{\partial h_k}{\partial h_{k-1}}=\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}({\sigma }^{\prime }(z_k))W_k$$

Translated: every hop multiplies the signal by an activation-slope effect (each neuron's σ′, a number at most 0.25) and a weight-matrix effect (whatever scaling W_k applies to the vector passing through). Two knobs per hop, nothing else. So if the relay loses 90% per hop, one of those two knobs, or their product, must be responsible.

Diagnose: saturation or scale?

Both knobs can strangle the relay in principle, and they predict different measurements. Module 0 planted the first suspicion: sigmoids have flat, saturated zones where the slope is ~0, and the standard textbook account of vanishing gradients blames them.

• Suspect A, saturation: pre-activations have drifted into the sigmoid's flat zones. Predicts a mean slope near zero, a sizable fraction of units with slope near zero, and a large spread of z (that is what pushes units into the flat zones).
• Suspect B, scale contraction: the sigmoids are fine, operating near their maximum slope of 0.25, but 0.25 times a sub-unity weight gain still multiplies the signal by ~0.1 per hop. Predicts healthy slopes and a small spread of z.

Now the full instrument readout this module has been quoting, from the same one-pass measurement (captures/results.log lines 33–41). You have already met the first two columns; the diagnosis lives in the last four:

PLAIN (Linear -> sigmoid): loss at init = 0.7784
  layer    ||dL/dW||    ||dL/dh||  std(z_pre)   std(z)   std(h)  mean slope  %slope<.05  W gain
      1     6.63e-05     1.75e-05      0.3513   0.3513   0.0848      0.2420        0.0%   0.767
      2     5.37e-04     1.83e-04      0.4514   0.4514   0.1087      0.2379        0.0%   0.604
      3     4.01e-03     1.75e-03      0.5744   0.5744   0.1366      0.2313        0.0%   0.623
      4     3.99e-02     1.47e-02      0.2321   0.2321   0.0575      0.2467        0.0%   0.470
 output     2.00e-01
gradient ratio, layer 1 vs output: 3.32e-04  (3015x smaller)
backward shrink factor per layer (||dL/dh_k|| / ||dL/dh_k+1||): 0.096, 0.104, 0.120

(One column pair needs a word: std(z_pre) is the spread before the network's normalizer slot and std(z) after it. In the plain network that slot is an identity, so the two match exactly; keep the pair in mind, because Module 4's batch-norm table is where they split apart.)

Read the verdict off the slope columns. mean slope is 0.2420, 0.2379, 0.2313, 0.2467 against a theoretical maximum of 0.25: these sigmoids operate within 8% of the steepest they can possibly be. And a mean can hide a saturated tail, so the %slope<.05 column counts the units whose slope actually sits in a flat zone: 0.0% at every layer. Not one unit in the network is saturated. Suspect A is measured out of the case.

Suspect B's prediction is what the table shows instead. The std(z) column says the pre-activations have spread 0.23–0.57, so nearly all of them sit in the sigmoid's steep central region around z = 0. Why so central? Small weights (std 0.31) times order-one inputs give small z. And note what is, and is not, dying on the forward side: a sigmoid fed z near 0 outputs values near 0.5, so the activations themselves stay comfortably mid-range. What collapses is their variation across examples: std(h) shrinks to 0.06–0.14, meaning each layer passes along less and less of what distinguished one input from another, even though the values look healthy. The forward signal's information content and the backward signal are dying together.

Put the hop together from the table's own columns. Take the hop from h2 down to h1, which crosses layer 2: mean slope 0.2379, and the W gain column (the root-mean-square singular value of W_2, i.e. the typical scaling a random direction receives) is 0.604. Their product predicts a factor of about 0.14; the measured factor is 0.096. Same ballpark (the estimate is crude because a random 4-wide matrix shrinks most specific directions somewhat more than its average suggests), and the structure of the answer is what matters:

hop factor ≈ slope × weight gain ≈ 0.24 × 0.5 ≈ 0.1

How much does each ingredient contribute? The sigmoid's slope ceiling is the heavyweight: 0.24⁴ ≈ 0.0033, a ~300x loss over four layers even if the weights preserved scale perfectly. The small weights multiply in another 0.5⁴ to 0.6⁴, roughly a further 8–16x. Compounded together they land near 10⁻⁴, the 3,000x hole the table measured. The lesson is multiplicative: the weights did not create the problem, they multiplied an already serious sigmoid contraction into a catastrophe.

So for this network, at initialization, the mechanism is scale, not saturation: maximally steep sigmoids still multiply in at most 0.25 per layer, and small random weights multiply in another ~0.5. The distinction is not pedantry; it predicts which fixes will work. If saturation were the problem, batch norm's job would be to move z's toward the steep region; Module 4 will show it does something quite different (and measurably makes the slopes smaller, yet fixes the network anyway).

Turn one knob yourself

A measured mechanism earns its keep by making predictions. Every knob lives at the top of the experiment script (DEPTH, WIDTH, and the norm= / act= options of TinyNet), and later passes of the same script already measure three turns of them, so you can predict first and check immediately:

1. Depth 4 → 8. Four more hops at ~0.1 each. Predict the layer-1-to-output gradient ratio before peeking; the measured answer is on results.log line 80.
2. Sigmoid → ReLU. ReLU's slope is exactly 1 for every active unit, so the activation-slope knob stops contracting; only the weight gain remains. Predict whether the cliff gets shallower or steeper, then check the measured init ratio on line 93.
3. Weights 10x larger. Would a W gain well above 1 cancel the 0.25 slope ceiling? That one is quiz question 3.

Module 3 shows what the cliff does to actual training, and Module 4 repeats this module's measurement, identical starting weights, with exactly one change: a batch-norm step between each Linear layer and its sigmoid. The cliff disappears, and the measurement explaining why is not the one the textbook story predicts.