Batch Norm vs Vanishing Gradients

A hands-on tour of one of the classic failure modes in deep learning, using the smallest network that can exhibit it. Every number on these pages was measured, not asserted; you can re-run the one script that produced them and get the same numbers.

Audience. You have trained a neural network before (a PyTorch tutorial counts) and you know what a derivative is. We build up everything else, including exactly why backpropagation multiplies one factor per layer.

Time. Around two hours across six pages.

What you will be able to do at the end. Measure gradient flow in any network, recognize the fingerprint of vanishing gradients in a training run, explain the actual mechanism (and debunk, with a measurement of your own, the explanation most textbooks lead with), and show quantitatively why batch normalization fixes it while brute-force alternatives do not scale.

Final check. Take the auto-graded exam: 17 questions, 70 to pass.

The whole story in one picture

We train a deliberately tiny network to classify the classic "two moons" dataset:

The two-moons task

Each dot is one example with two features (its x and y coordinates). The two classes interleave like crescent moons, so no straight line separates them; the network has to bend a boundary through the gap. That requires depth to matter, which is exactly what we want to stress-test.

The network is as small as we could make it while keeping the failure visible: four hidden layers of four sigmoid neurons each, 77 parameters in total.

The network

Here is what you will prove for yourself over the next six pages, with measurements at every step:

Built plainly, the learning signal reaching this network's first layer is 3,015 times weaker than at its last layer, before training even starts.
Trained for 4,000 steps it stays at 51.5% test accuracy: a coin flip.
Adding one standard normalization step per layer (a single line of code), with the same weights, same data, and same training, takes it to 100%.
A 10x learning rate can rescue the 4-layer version (slowly). At 8 layers, no learning rate we tried rescues it, while batch normalization reaches 90% accuracy within 50 steps.
Along the way you will test the most common textbook explanation for why the fix works, and find, by direct measurement, that it is wrong for this very network. The real story is more interesting.

How this workshop is organized

Use the sidebar to move between pages (the menu button on a phone). Each page ends with a short quiz; attempt it before opening the collapsible answer key. The pages build on each other, so go in order.

Page	What you will learn
Module 0: The two functions everything hinges on	The sigmoid and the cross-entropy loss, pinned down by measurement, including the two specific properties every later argument leans on.
Module 1: The smallest network that can fail	The two-moons task, the 2 → [4]×4 → 1 network, and the controls that make the comparison fair.
Module 2: Watch the gradients vanish	Build the backprop-as-a-relay intuition, then measure how much learning signal survives each layer and diagnose what destroys it.
Module 3: A network that cannot learn	The flatlined training run, the brute-force learning-rate rescue, and why brute force stops working at depth 8.
Module 4: Batch norm brings it back	What BN computes, the re-measured gradients, and the honest mechanism, including where the textbook story fails.
Module 5: The other fixes and one classic bug	ReLU, Xavier init, residual connections, LayerNorm (all measured), plus the model.eval() bug, demonstrated.

Prerequisites

To run the experiment yourself (optional; every result is reproduced on the page):

Python 3 with torch, numpy, and matplotlib (pip install torch numpy matplotlib).
A CPU is plenty. The biggest network in this workshop has 8 hidden layers of 4 neurons.

Everything numeric on these pages comes from one deterministic run of a single script, experiment.py (fixed seed 0). The full listing, ready to copy, is on the Source page. Its output is captured in captures/results.log, and every number is cited back to the pass of the script that produced it, by log line, as we go. The exact environment is recorded in the reproducing section of Module 5.

Start with Module 0: The two functions everything hinges on.