Module 0: The two functions everything hinges on

You have likely met both functions before. The point of this page is not to introduce them but to measure the properties we will lean on, so that when Module 2 says "the gradient shrinks because a slope multiplies in at every layer," you have already seen exactly which slope and how big it can possibly be.

The sigmoid, and the one number to remember about it

A neuron in our network computes a weighted sum of its inputs, z = w·x + b, and then applies the sigmoid to it:

σ(z) = 1 / (1 + e^(−z))

The sigmoid squashes any real number into (0, 1). That squashing is what makes the network nonlinear — a stack of purely linear layers would collapse into one linear layer, and could never bend a boundary around the moons.

For gradient flow, though, what matters about any function is not its value but its slope: when the input z nudges a little, how much does the output move? For the sigmoid the slope has a famous closed form. Writing s = σ(z):

σ′(z) = s · (1 − s)

Both s and (1 − s) live in (0, 1), and a product p(1 − p) is maximized at p = 1/2, where it equals 1/4. So the sigmoid's slope is never larger than 0.25, achieved exactly where the sigmoid crosses 1/2, i.e. at z = 0. Far from zero the slope decays to nothing — that flat zone is called saturation.

Do not take the algebra on faith; the experiment script checks it numerically over a fine grid:

zs = torch.linspace(-8, 8, 2001)
sig = torch.sigmoid(zs)
dsig = sig * (1 - sig)            # sigma'(z) = sigma(z) * (1 - sigma(z))
i = int(torch.argmax(dsig))

max sigmoid slope     = 0.250000  at z = -0.0000
slope at z = +/-4     = 0.017663

(Captured in captures/results.log lines 14–15.) Read it with me: the largest slope found anywhere on the grid is exactly 0.250000, at z = 0 — matching the algebra — and by z = ±4 the slope has already collapsed to about 0.018, fourteen times smaller than the peak. Here is the whole picture:

The blue curve is σ(z); the orange curve is its slope σ′(z), peaking at 0.25 and dying off in the shaded saturation zones. Carry two facts forward:

1. Every trip through a sigmoid multiplies a gradient by at most 0.25 — usually less.
2. In saturation the multiplier is nearly zero. (Keep this one in your pocket: Module 2 will test whether saturation is actually what bites our network, and the answer will surprise you.)

The loss: how wrong is the network, in one number

The network's final layer produces a single number (a logit), which a last sigmoid converts to a probability p that the input belongs to class 1. Training needs a score for how bad p is against the true label y (0 or 1). We use binary cross-entropy (BCE):

L = −[ y · ln p + (1 − y) · ln(1 − p) ]

Which is just: "the negative log of the probability you assigned to the correct answer."

The plot shows both branches: when y = 1 the loss is −ln p (blue), which is 0 if the network is confidently right and explodes if it is confidently wrong; the y = 0 branch (red) mirrors it. The dotted line marks the value we will care about most later:

BCE loss at p = 0.5   = 0.693147   (ln 2 = 0.693147)

(Captured in captures/results.log line 16.) A network that always outputs p = 0.5 — that is, one that shrugs at every example — scores exactly ln 2 ≈ 0.693. Remember that number: when a training-loss curve flatlines at 0.693, it is not "converging," it is telling you the network has learned nothing at all. You will see precisely that curve in Module 3.

The gradient at the output: p − y

One more fact makes everything measurable. When the loss is BCE and the output nonlinearity is a sigmoid, the derivative of the loss with respect to the logit z collapses to something remarkably clean:

∂L/∂z = p − y

Deriving it is two applications of the chain rule (the −1/p from the log cancels against the p(1 − p) from the sigmoid). Rather than derive it, let us have PyTorch's autograd — the same machinery that computes every gradient in this workshop — check it on a concrete example:

z = torch.tensor(2.0, requires_grad=True)
p = torch.sigmoid(z)
loss = nn.functional.binary_cross_entropy(p, torch.tensor(0.0))
loss.backward()

autograd check at z = 2.0, y = 0:
  p = sigmoid(2.0)    = 0.880797
  formula p - y       = +0.880797
  autograd dL/dz      = +0.880797

(Captured in captures/results.log lines 17–20.) Read it with me: at logit z = 2 the model predicts p = 0.881; the true label is y = 0, so the formula says the gradient should be p − y = +0.881; and z.grad, computed by autograd with no knowledge of our formula, is +0.880797 — identical to six decimal places.

Two reasons this matters for the rest of the workshop:

• It anchors the relay. The backward pass has to start somewhere. It starts at the output with the honest, healthy-sized signal p − y (order 0.1 to 1). Whatever happens to gradients deeper in the network happens on the way down from there — which is exactly what Module 2 measures, layer by layer.
• It calibrates your trust in autograd. Every measurement ahead is an autograd readout. You have now seen it agree exactly with a hand-derived formula.

What to carry forward

• σ′(z) ≤ 0.25, with equality only at z = 0; slopes near ±4 are already ~0.018.
• A know-nothing classifier scores loss ln 2 ≈ 0.693 and 50% accuracy.
• The backward pass starts at the output with gradient p − y.