Log: results.log
The complete captured output of one deterministic run (seed 0) of
the experiment script. Every number quoted in the modules cites a
line of this log; the line numbers on the left match those citations, so a reference like
"lines 25 and 29" can be looked up directly. The raw file lives at captures/results.log in
the workshop directory, and re-running the script regenerates it byte for byte.
1 ========================================================================
2 Batch Norm vs Vanishing Gradients -- experiment log
3 torch 2.12.1+cpu | numpy 2.2.6 | seed 0
4 network: 4 hidden layers x 4 neurons, sigmoid activations
5 ========================================================================
6
7 [S1] two-moons dataset
8 samples: 1000 (500 per class), noise sigma = 0.1
9 split: 800 train / 200 test
10 input feature std: x1 = 0.890, x2 = 0.511
11 [plot saved: diagrams/two_moons.png]
12
13 [S2] sigmoid and loss facts
14 max sigmoid slope = 0.250000 at z = -0.0000
15 slope at z = +/-4 = 0.017663
16 BCE loss at p = 0.5 = 0.693147 (ln 2 = 0.693147)
17 autograd check at z = 2.0, y = 0:
18 p = sigmoid(2.0) = 0.880797
19 formula p - y = +0.880797
20 autograd dL/dz = +0.880797
21 [plot saved: diagrams/sigmoid_and_deriv.png]
22 [plot saved: diagrams/bce_loss.png]
23
24 [S3] the networks
25 plain net parameters: 77
26 hidden-weight checksum, plain = 18.770746
27 hidden-weight checksum, bn = 18.770746
28 identical starting weights: True
29 layer-2 weight stats: std = 0.3114, max|w| = 0.4777 (PyTorch default init, fan_in = 4)
30
31 [S4] at-init measurements (full train batch, one backward pass)
32
33 PLAIN (Linear -> sigmoid): loss at init = 0.7784
34 layer ||dL/dW|| ||dL/dh|| std(z_pre) std(z) std(h) mean slope %slope<.05 W gain
35 1 6.63e-05 1.75e-05 0.3513 0.3513 0.0848 0.2420 0.0% 0.767
36 2 5.37e-04 1.83e-04 0.4514 0.4514 0.1087 0.2379 0.0% 0.604
37 3 4.01e-03 1.75e-03 0.5744 0.5744 0.1366 0.2313 0.0% 0.623
38 4 3.99e-02 1.47e-02 0.2321 0.2321 0.0575 0.2467 0.0% 0.470
39 output 2.00e-01
40 gradient ratio, layer 1 vs output: 3.32e-04 (3015x smaller)
41 backward shrink factor per layer (||dL/dh_k|| / ||dL/dh_k+1||): 0.096, 0.104, 0.120
42
43 BATCHNORM (Linear -> BN -> sigmoid): loss at init = 0.7914
44 layer ||dL/dW|| ||dL/dh|| std(z_pre) std(z) std(h) mean slope %slope<.05 W gain
45 1 5.39e-02 8.25e-03 0.3513 1.0001 0.2180 0.2025 0.0% 0.767
46 2 9.34e-02 1.01e-02 0.4858 0.9996 0.2192 0.2020 0.0% 0.604
47 3 1.15e-01 1.44e-02 0.5569 0.9999 0.2215 0.2009 0.0% 0.623
48 4 2.07e-01 1.48e-02 0.2728 0.9995 0.2195 0.2018 0.0% 0.470
49 output 2.36e-01
50 gradient ratio, layer 1 vs output: 2.29e-01 (4x smaller)
51 backward shrink factor per layer (||dL/dh_k|| / ||dL/dh_k+1||): 0.816, 0.703, 0.972
52 [plot saved: diagrams/grad_norms_by_layer.png]
53 [plot saved: diagrams/grad_norms_plain.png]
54 [plot saved: diagrams/signal_decay.png]
55 [plot saved: diagrams/act_std_by_layer.png]
56
57 [S5] geometric decay illustration (pure arithmetic)
58 factor 0.9: after 4 layers -> 6.56e-01, after 10 layers -> 3.49e-01
59 factor 0.5: after 4 layers -> 6.25e-02, after 10 layers -> 9.77e-04
60 factor 0.25: after 4 layers -> 3.91e-03, after 10 layers -> 9.54e-07
61 factor 0.1: after 4 layers -> 1.00e-04, after 10 layers -> 1.00e-10
62 [plot saved: diagrams/geometric_decay.png]
63
64 [S6] training runs (SGD, lr = 0.5, batch = 64, 4000 steps)
65 plain final train loss = 0.6926, final test acc = 51.5%, first >=90%: never
66 with batch norm final train loss = 0.0053, final test acc = 100.0%, first >=90%: step 100
67 chance accuracy = 50.0% | 'learned nothing' loss = ln 2 = 0.6931
68 [plot saved: diagrams/loss_curves.png]
69 [plot saved: diagrams/accuracy_curves.png]
70 [plot saved: diagrams/decision_boundaries.png]
71
72 [S7] brute force vs depth (4000 steps each)
73 depth 4 (the net from S6):
74 plain, lr = 5.0 final train loss = 0.0017, final test acc = 100.0%, first >=90%: step 1050
75 plain, lr = 50.0 final train loss = 5.7024, final test acc = 48.5%, first >=90%: never
76 (lr = 0.5 = the S6 run: final test acc = 51.5%)
77 [plot saved: diagrams/lr_crank.png]
78
79 depth 8 (same width, same recipe):
80 init gradient ratio L1/out: plain = 4.97e-07, batch norm = 3.37e-01
81 plain, lr = 0.5 final train loss = 0.6932, final test acc = 51.5%, first >=90%: never
82 plain, lr = 5.0 final train loss = 0.7033, final test acc = 48.5%, first >=90%: never
83 plain, lr = 50.0 final train loss = 5.1390, final test acc = 48.5%, first >=90%: never
84 batch norm, lr = 0.5 final train loss = 0.0175, final test acc = 100.0%, first >=90%: step 50
85 [plot saved: diagrams/depth8_rescue.png]
86
87 [S8] the other fixes (same init where applicable, same training recipe)
88 xavier layer-2 weight std = 0.5155 (default was 0.3114)
89
90 variant init grad ratio L1/out final test acc
91 plain sigmoid 3.32e-04 51.5%
92 batch norm 2.29e-01 100.0%
93 ReLU 2.26e-01 93.0%
94 Xavier init 1.59e-02 100.0%
95 residual 3.24e-02 93.0%
96 layer norm 2.76e-03 100.0%
97 [plot saved: diagrams/variants_accuracy.png]
98 [plot saved: diagrams/relu_and_deriv.png]
99
100 [S9] the model.eval() bug (using the trained batch-norm net from S6)
101 test accuracy, model.eval() (correct) = 100.0%
102 test accuracy, train mode, full test batch = 100.0%
103 test accuracy, train mode, batches of 2 = 71.5%
104 learned gamma (BN layer 1) = ['1.42', '4.70', '0.82', '0.87']
105 learned beta (BN layer 1) = ['-0.10', '0.46', '0.16', '0.04']
106
107 done. every figure written to diagrams/, every number above is citable.