Batch Norm vs Vanishing Gradients

A hands-on tour of one of the classic failure modes in deep learning, using the smallest network that can exhibit it. Every number on these pages was measured, not asserted; you can re-run the one script that produced them and get the same numbers.

Audience. You have trained a neural network before (a PyTorch tutorial counts) and you know what a derivative is. We build up everything else, including exactly why backpropagation multiplies one factor per layer.

Time. Around two hours across six pages.

What you will be able to do at the end. Measure gradient flow in any network, recognize the fingerprint of vanishing gradients in a training run, explain the actual mechanism (which is not the saturation story most textbooks lead with, as you will verify), and show quantitatively why batch normalization fixes it while brute-force alternatives do not scale.

Final check. Take the auto-graded exam: 17 questions, 70 to pass.

The whole story in one picture

We train a deliberately tiny network to classify the classic "two moons" dataset:

The two-moons task

Each dot is one example with two features (its x and y coordinates). The two classes interleave like crescent moons, so no straight line separates them; the network has to bend a boundary through the gap. That requires depth to matter — which is exactly what we want to stress-test.

The network is as small as we could make it while keeping the failure visible: four hidden layers of four sigmoid neurons each, 77 parameters in total.

The network

Here is what you will prove for yourself over the next six pages, with measurements at every step:

Built plainly, this network's first layer receives gradients 3,015 times smaller than its last layer, before training even starts.
Trained for 4,000 steps it stays at 51.5% test accuracy — a coin flip.
Adding one BatchNorm1d block per layer — same weights, same data, same training — takes it to 100%.
A 10x learning rate can rescue the 4-layer version (slowly). At 8 layers, no learning rate we tried rescues it, while batch norm reaches 90% accuracy within 50 steps.
And the standard explanation for why batch norm helps ("it keeps the sigmoid out of its saturated region") is measurably wrong for this network. The real mechanism is better.

How this workshop is organized

Use the sidebar to move between pages (the menu button on a phone). Each page ends with a short quiz — attempt it before opening the collapsible answer key. The pages build on each other, so go in order.

Page	What you will learn
Module 0: The two functions everything hinges on	The sigmoid and the cross-entropy loss, pinned down by measurement: max slope 0.25, the ln 2 baseline, and the p − y gradient.
Module 1: The smallest network that can fail	The two-moons task, the 2 → [4]×4 → 1 network, and the controls that make the comparison fair.
Module 2: Watch the gradients vanish	Build the backprop-as-a-relay intuition, then measure the per-layer gradient cliff: a factor of ~0.1 per layer.
Module 3: A network that cannot learn	The flatlined training run, the brute-force learning-rate rescue, and why brute force stops working at depth 8.
Module 4: Batch norm brings it back	What BN computes, the re-measured gradients, and the honest mechanism — including where the textbook story fails.
Module 5: The other fixes and one classic bug	ReLU, Xavier init, residual connections, LayerNorm — all measured — plus the model.eval() bug, demonstrated.

Prerequisites

To run the experiment yourself (optional — every result is reproduced on the page):

Python 3 with torch, numpy, and matplotlib (pip install torch numpy matplotlib).
A CPU is plenty. The biggest network in this workshop has 8 hidden layers of 4 neurons.

Everything numeric on these pages comes from one deterministic run of a single script, experiments/experiment.py (fixed seed 0). Its full output is captured in captures/results.log, and numbers are cited to that file by line as we go. The exact environment is recorded in the reproducing section of Module 5.

Start with Module 0: The two functions everything hinges on.