Final Exam — Batch Norm vs Vanishing Gradients

1. [Module 0] A gradient passes backward through a single sigmoid neuron that happens to be at its steepest possible operating point. By what factor is the gradient scaled?

1.0 — a sigmoid at its steepest passes gradients unchanged 0.5 0.25 0.017

2. [Module 0] On a balanced binary task, a training-loss curve that flatlines at a specific value tells you the network has learned nothing at all. What is that value, to three decimal places?

3. [Module 0] The network predicts p = 0.9 for an example whose true label is y = 1. What gradient does the loss send to the output logit, and what does its size mean?

+0.9 — a strong correction is needed −0.1 — a small nudge; the prediction is already nearly right −0.9 — the sign flips because y = 1 0 — correct-side predictions send no gradient

4. [Module 1] True or false: the plain network and the batch-norm network in this workshop start from different random weights, so some of the accuracy gap could be initialization luck.

True False

5. [Module 1] A single 16-neuron hidden layer would solve two moons easily. Why does the workshop use four 4-neuron layers instead?

Deeper networks always generalize better on small data Fewer parameters per layer prevents overfitting the noise The two-moons boundary cannot be represented with one hidden layer The failure being studied compounds per layer, so it only shows up with depth

6. [Module 2] For the plain 4-layer network at initialization, roughly what fraction of the backward signal survives each layer crossing, as measured?

About 0.9 — each layer costs only a little About 0.1 — each layer keeps a tenth of the signal About 0.25 — exactly the sigmoid’s maximum slope About 0.5 — the weight matrices’ typical scaling

7. [Module 2] At initialization, the plain 4-layer network’s first hidden layer receives weight gradients how much smaller than its output layer’s?

About 4× smaller About 30× smaller About 3,000× smaller About 2,000,000× smaller

8. [Module 2] The plain network’s measured mean sigmoid slopes at initialization are 0.23–0.25. What does this measurement rule out as the cause of its vanishing gradients?

Saturation — the sigmoids are near their steepest, not in their flat zones Small weights — the slopes prove the weights are large enough The chain rule — multiplication of factors cannot occur at these slopes Nothing — slopes near 0.25 are exactly what saturation looks like

9. [Module 2] The per-layer factor that strangles the plain network is a product of exactly two ingredients. Which pair?

Learning rate × batch size Activation slope × input noise level Activation slope × weight-matrix scale Weight-matrix scale × number of neurons per layer

10. [Module 3] A colleague’s deep sigmoid classifier sits at loss 0.693 for 300 epochs. Per this workshop, what is the best first action?

Train longer — deep networks converge slowly Measure per-layer gradient norms with one forward/backward pass Increase the learning rate 10× and rerun Add more neurons per layer to increase capacity

11. [Module 3] True or false: since lr = 5 rescued the plain 4-layer network, a large enough learning rate also rescued the plain 8-layer network in this workshop’s measurements.

True False

12. [Module 3] At lr = 50 the plain network’s final training loss is 5.70 — far above the 0.693 plateau. What happened?

The gradients vanished even faster at the higher learning rate The loss function overflowed numerically The well-conditioned top layers were flung past minima, leaving the net confidently wrong The network memorized the training set and the value reflects overfitting

13. [Module 4] Comparing measured mean sigmoid slopes at identical weights: what did adding batch norm do to them, and why does it matter?

Raised them toward 0.25, confirming BN works by avoiding saturation Lowered them — while gradients grew ~800× — refuting the saturation explanation Left them exactly unchanged, since the weights are identical Drove them to zero, which is why γ and β are needed

14. [Module 4] Mechanically, how does batch norm restore the backward signal?

Its backward pass divides by the small batch std — a per-layer gain boost that pins the hop factor near 1 It clips gradients below a threshold so they cannot shrink further It adds γ and β, whose gradients are large and propagate downward It skips the sigmoid during backpropagation

15. [Module 4] BN standardizes to mean 0, spread 1 — then immediately lets learned γ, β re-scale and re-shift. What did the trained network’s γ = 4.70 on one neuron demonstrate?

A training instability: γ should stay near 1 in a healthy network That γ compensates for the learning rate being too small The network chose a much wider input spread for that neuron — deliberately buying saturation for a switch-like response That γ converges to the batch std σ_B

16. [Module 5] Which property of ReLU attacks the vanishing-gradient factor?

It is bounded between 0 and 1, like the sigmoid but cheaper Its slope is exactly 1 wherever the unit is active — zero attenuation on active paths It normalizes each batch to mean 0 and spread 1 Its outputs are always positive, which keeps weights growing

17. [Module 5] A deployed BN model aces offline evaluation (one large batch) but gives erratic, batch-size-dependent predictions on live traffic arriving in ones and twos. What is the diagnosis?

Distribution shift between offline data and live traffic The learning rate was too high in the final epochs The model was left in train mode — BN is normalizing each tiny batch by its own statistics; call model.eval() Dead ReLU neurons activated by the smaller batches