Module 0: The two functions everything hinges on

You have likely met both functions before. The point of this page is not to introduce them but to measure the properties we will lean on, so that when Module 2 says "the gradient shrinks because a slope multiplies in at every layer," you have already seen exactly which slope and how big it can possibly be.

The sigmoid, and the one number to remember about it

A neuron in our network computes a weighted sum of its inputs, z = w·x + b, and then applies the sigmoid to it:

$$\sigma (z)=\frac{1}{1+e^{-z}}$$

The sigmoid squashes any real number into (0, 1). That squashing is what makes the network nonlinear: a stack of purely linear layers would collapse into one linear layer, and could never bend a boundary around the moons.

For gradient flow, though, what matters about any function is not its value but its slope: when the input z nudges a little, how much does the output move? For the sigmoid the slope has a famous closed form. Writing s = σ(z):

$${\sigma }^{\prime }(z)=s(1-s)$$

Both s and (1 − s) live in (0, 1), and a product p(1 − p) is maximized at p = 1/2, where it equals 1/4. So the sigmoid's slope is never larger than 0.25, achieved exactly where the sigmoid crosses 1/2, i.e. at z = 0. Far from zero the slope decays to nothing; that flat zone is called saturation.

Do not take the algebra on faith. Section 2's pass of the experiment script (the part that pins down the sigmoid and loss facts by measurement) checks it numerically over a fine grid and logs what it finds at captures/results.log lines 14–15:

# Evaluate the sigmoid on a fine grid.
zs = torch.linspace(-8, 8, 2001)
sig = torch.sigmoid(zs)

# Its slope has the closed form sigma'(z) = sigma(z) * (1 - sigma(z)).
dsig = sig * (1 - sig)

# Find the largest slope anywhere on the grid -- the claim is 0.25, at z = 0.
i = int(torch.argmax(dsig))
log(f"  max sigmoid slope     = {dsig[i]:.6f}  at z = {zs[i]:+.4f}")

max sigmoid slope     = 0.250000  at z = -0.0000
slope at z = +/-4     = 0.017663

Let's go deeper into the two logged lines: the largest slope found anywhere on the grid is exactly 0.250000, at z = 0 (matching the algebra), and by z = ±4 the slope has already collapsed to about 0.018, fourteen times smaller than the peak. Here is the whole picture:

The blue curve is σ(z); the orange curve is its slope σ′(z), peaking at 0.25 and dying off in the shaded saturation zones.

Why care about the slope at all? Because a slope is a conversion rate for small changes: if z nudges by a tiny amount δ, the sigmoid's output moves by roughly σ′(z) · δ: whatever passes through the sigmoid has its small changes scaled by that factor. And when functions are chained, as layers in a network are, the conversion rates chain by multiplying: if a nudge somewhere upstream reaches this sigmoid's input already scaled by some factor a, it leaves the sigmoid scaled by a · σ′(z). Concretely: a nudge δ crossing one sigmoid at its very steepest comes out as 0.25 δ; crossing a second sigmoid at its steepest leaves 0.0625 δ. That is the chain rule, nothing more, and it applies equally to the training signal flowing backward, since a gradient is exactly a "how much does a nudge here move the loss" conversion rate. Module 2 builds this into the full layer-by-layer relay and measures it; the multiplication itself is the seed of this whole workshop's story.

So, two facts to carry forward:

1. Every trip through a sigmoid multiplies a gradient by at most 0.25, usually less.
2. In saturation the multiplier is nearly zero. (Keep this one in your pocket: Module 2 will test whether saturation is actually what bites our network, and the answer will surprise you.)

The loss: how wrong is the network, in one number

The network's final layer produces a single number (a logit), which a last sigmoid converts to a probability p that the input belongs to class 1. Training needs to compress "how good are the predictions right now?" into a single number it can push downhill, step by step.

The obvious candidate, accuracy, is useless for this. Nudge a weight slightly and almost no prediction flips, so accuracy almost never moves: it is a staircase, flat nearly everywhere, and you cannot follow a slope that is zero. Training needs a score that moves smoothly every time p moves, costs nothing when the model is right, and punishes being confidently wrong.

The score that classifiers use is binary cross-entropy (BCE), and the idea fits in one sentence: look up the probability the model assigned to the correct answer, and charge its negative logarithm as the penalty. If the true label is y = 1, the model gave the correct answer probability p; if y = 0, it gave the correct answer probability 1 − p.

Why a logarithm, and why the minus sign? Watch what −ln does across the range of "probability you gave the right answer" (plain arithmetic; every row is a point on the measured curve below):

Three properties fall out, and each is exactly something training needs. Being right costs nothing. Any change in p changes the loss, so there is always a slope to follow; the staircase problem is gone. And the penalty grows without bound as the model gets more confidently wrong: being 99% sure of the wrong answer costs 6.6 times more than shrugging, and it keeps climbing from there. The training signal shouts loudest exactly where the model is most broken. (Hold onto that last property: in Module 3 you will meet a network whose loss is far above the shrug value, and this table is what makes that number readable; it is not ignorant, it is confidently wrong.)

The textbook formula packs the two cases into one line using an old trick, multiplying each branch by something that equals 1 when that branch applies and 0 when it does not:

$$L=-[y\ln p+(1-y)\ln (1-p)]$$

When y = 1, the factor (1 − y) is 0, so the second term vanishes and −ln p remains. When y = 0, the factor y kills the first term instead, leaving −ln(1 − p). That is all the formula is: the "charge −ln of the probability on the correct answer" rule, written without an if-statement.

The plot (computed over the full range of p by Section 2's pass, like every figure on this page) shows both branches: the blue y = 1 branch is −ln p, zero at p = 1 and exploding toward p = 0; the red y = 0 branch mirrors it. The dotted line marks the one value from the table worth memorizing, which Section 2's pass also computes directly and logs at line 16:

log(f"  BCE loss at p = 0.5   = {-math.log(0.5):.6f}   (ln 2 = {math.log(2):.6f})")

BCE loss at p = 0.5   = 0.693147   (ln 2 = 0.693147)

A network that always outputs p = 0.5 (that is, one that shrugs at every example) scores exactly ln 2 ≈ 0.693. Remember that number: when a training-loss curve flatlines at 0.693, it is not "converging," it is telling you the network has learned nothing at all. You will see precisely that curve in Module 3.

The gradient at the output: p − y

One more fact makes everything measurable. When the loss is BCE and the output nonlinearity is a sigmoid, the derivative of the loss with respect to the logit z collapses to something remarkably clean:

$$\frac{\partial L}{\partial z}=p-y$$

Deriving it is two applications of the chain rule (the −1/p from the log cancels against the p(1 − p) from the sigmoid). Rather than derive it, let us have PyTorch's autograd (the same machinery that computes every gradient in this workshop) check it on a concrete example. Section 2's pass runs exactly this check and logs the result at lines 17–20:

# Verify dL/dz = p - y on a concrete example, letting autograd do the math:
# a logit of 2.0 scored against true label y = 0.
z = torch.tensor(2.0, requires_grad=True)
p = torch.sigmoid(z)
loss = nn.functional.binary_cross_entropy(p, torch.tensor(0.0))
loss.backward()

autograd check at z = 2.0, y = 0:
  p = sigmoid(2.0)    = 0.880797
  formula p - y       = +0.880797
  autograd dL/dz      = +0.880797

Let's go deeper: at logit z = 2 the model predicts p = 0.881; the true label is y = 0, so the formula says the gradient should be p − y = +0.881; and z.grad, computed by autograd with no knowledge of our formula, is +0.880797, identical to six decimal places.

Two reasons this matters for the rest of the workshop:

• It anchors the relay. The backward pass has to start somewhere. It starts at the output with the honest, healthy-sized signal p − y (order 0.1 to 1). Whatever happens to gradients deeper in the network happens on the way down from there, which is exactly what Module 2 measures, layer by layer.
• It calibrates your trust in autograd. Every measurement ahead is an autograd readout. You have now seen it agree exactly with a hand-derived formula.

What to carry forward

• σ′(z) ≤ 0.25, with equality only at z = 0; slopes near ±4 are already ~0.018.
• A know-nothing classifier scores loss ln 2 ≈ 0.693 and 50% accuracy.
• BCE punishes confident wrongness without bound: a loss far above 0.693 means confidently wrong, not ignorant.
• The backward pass starts at the output with gradient p − y.