Training & Backpropagation

In the previous lesson, you traced a CNN score while its kernel values stayed fixed. A useful detector can't be supplied by hand for every crack, barcode smear, or torn label in a shipment photo. The model has to discover parameter values from examples.

We will make that discovery visible using a small regression problem: predict extra delivery delay in hours from a route-disruption score. A real CNN has far more parameters, but each kernel weight changes for the same reason as the one weight below: changing it changes a prediction, which changes a loss.

One update has four distinct jobs

A model learns by repeating four steps:

Suppose a shipment has route-disruption score x = 2 and later arrives y = 6 hours late. Start with this model:

$$\hat{y} = wx + b$$

Set w = 1 and b = 0. The predicted extra delay is 2 hours, far below the observed 6. For this numerical lesson, use half squared error:

$$L = \frac{1}{2}(\hat{y} - y)^2$$

The factor 1/2 doesn't change which parameters are good; it cancels the 2 when we differentiate the square. Here the initial loss is:

$$L = \frac{1}{2}(2 - 6)^2 = 8$$

The derivative of loss with respect to the prediction is -4: increasing the predicted delay would reduce this mistake. Because w is multiplied by x = 2, its gradient is twice as large:

$$\frac{\partial L}{\partial \hat{y}} = \hat{y} - y = -4, \qquad \frac{\partial L}{\partial w} = (\hat{y} - y)x = -8, \qquad \frac{\partial L}{\partial b} = \hat{y} - y = -4$$

With learning rate $\eta = 0.1$, subtract those gradients:

$$w_{\text{new}} = 1 - 0.1(-8) = 1.8, \qquad b_{\text{new}} = 0 - 0.1(-4) = 0.4$$

The revised prediction is 1.8 * 2 + 0.4 = 4.0 hours, and loss falls from 8.0 to 2.0.

One update helps, but training is a repeated correction:

Step size can rescue or wreck learning

The gradient says which direction lowers loss locally. The learning rate says how far to move. A very small rate makes progress slowly. A useful rate settles toward a low-loss setting. A large rate can jump past it and grow the error on every oscillation.

The same one-example problem makes this visible because we can hold everything except learning rate constant:

Don't assume that one loss decrease proves the rate is safe. Real batches disagree with one another, and deep networks can have steep regions in some directions and shallow ones in others. Training code should log loss and gradient norms so a bad run becomes visible quickly.

Backpropagation assigns credit through a graph

Gradient descent needs dL/dw for every trainable parameter. Calculating it by nudging each parameter separately is useful as a check for a tiny formula, but it would repeat forward evaluation for every parameter direction. Backpropagation uses reverse-mode automatic differentiation: it reuses the forward computation graph and moves derivatives backward from one scalar loss through each local operation.^[1]

Rumelhart, Hinton, and Williams showed that this error-driven weight adjustment could learn useful hidden representations in multilayer networks in their 1986 Nature paper.^[2] For our small shipment-delay model, the graph is already enough to see the mechanism:

The chain rule follows paths from loss back to parameters:

$$\frac{\partial L}{\partial w} = \frac{\partial L}{\partial \hat{y}} \frac{\partial \hat{y}}{\partial w} = (-4)(2) = -8$$ $$\frac{\partial L}{\partial b} = \frac{\partial L}{\partial \hat{y}} \frac{\partial \hat{y}}{\partial b} = (-4)(1) = -4$$

A useful engineering habit is to compare an analytical gradient with a centered finite difference. The check is slow for training, but excellent for finding a missing factor or wrong sign in a new backward formula.

Build a scalar autograd engine

Andrej Karpathy's Micrograd makes the backward sweep inspectable: each object stores one scalar, its accumulated gradient, the values that produced it, and a local backward function.^[3] Tensor libraries apply the same principle to matrix operations; a scalar version keeps the accounting readable.

For prediction = w * x + b and loss = 0.5 * (prediction - target) ** 2, every operation leaves a small backward rule:

The following compact engine builds the computation graph during the forward pass, orders nodes so children run before parents during the backward pass, and accumulates gradients with +=.

One rule is easy to miss: if a value affects loss through two paths, its gradient is the sum of those paths. The simple expression a * a exposes an incorrect engine immediately.

In a tensor framework, parameter gradient buffers also accumulate until they are cleared. That makes intentional gradient accumulation possible, but it also creates a common bug: forgetting to clear gradients between independent optimizer steps.^[4]

Move from one shipment to a batch

One damaged shipment doesn't represent a delivery network. Training normally estimates a useful update from a mini-batch, a subset of training examples processed before one parameter update. For mean loss, the batch gradient is the average of the example gradients.

This tiny dataset contains only three shipments, so the code below uses all three at once: technically, it is full-batch gradient descent. The arithmetic is the same one a mini-batch uses; a larger run uses a subset before each update. Here the true relationship is extra_delay = 2 * disruption + 1. Starting from zero parameters:

An epoch means one complete pass through the training examples. LLM pretraining runs are commonly described in optimizer steps and tokens processed because the dataset is very large; the batch contract remains the same: forward, average loss, backward, update.

Validation loss tests whether learning transfers

Training loss only measures examples used for updates. A validation set contains held-out examples used for evaluation, not for backpropagation. If training loss drops while validation loss rises, the new parameters fit the training set better but transfer worse. That pattern is a warning, not a diagnosis: investigate data mismatch, leakage, noise, model capacity and regularization before deciding why it happened.

This deliberately mismatched example makes the warning visible. Training records follow a peak-season delay pattern; validation records follow a normal-week pattern. Starting from a model that already matches normal weeks, fitting peak season improves train loss and damages validation loss:

For a real run, compare train and validation curves at consistent checkpoints and log enough context to reproduce the run: data version, split policy, batch size, learning rate, optimizer, seed and checkpoint identifier.

See the same responsibilities in PyTorch

Libraries remove manual derivative bookkeeping, not the conceptual separation. In the code below, multiplying PyTorch's mean squared error by 0.5 keeps the same half-squared-error convention used above. loss.backward() fills parameter gradients and optimizer.step() applies an update. The loss drops after one step on the same three-example batch.^[4]

Because gradients accumulate, two backward calls without clearing produce twice the gradient for the same loss:

Why the next loss function matters

We used half squared error because predicting delay hours is a regression problem and its derivative is easy to inspect. A CNN that selects damaged, intact, or needs-review, and an LLM that selects the next token, face a classification problem: the output layer produces competing logits rather than one continuous hour estimate.

The backward-pass mechanics won't disappear. The next lesson supplies the missing objective: softmax converts logits into a distribution, cross-entropy scores the correct choice, and their gradient becomes the signal sent backward through the model.

Mastery check

Key concepts

• Training separates forward computation, loss evaluation, backward differentiation and parameter update.
• A gradient measures loss sensitivity; gradient descent moves in the opposite direction.
• Backpropagation applies local chain-rule derivatives in reverse order through a computation graph.
• A finite-difference check verifies a formula but isn't an efficient training method.
• Shared graph paths and repeated backward calls require gradient accumulation.
• With mean loss, mini-batches provide one averaged update from a subset of training examples.
• Validation loss measures transfer to held-out data and can reveal a failing run.

Evaluation rubric

• Foundational: Compute prediction, half squared loss, dL/dw, dL/db and one SGD update for the shipment-delay example.
• Intermediate: Explain why reverse-mode differentiation is preferable to nudging every parameter, and implement a small scalar graph that accumulates gradients correctly.
• Advanced: Use mini-batch and validation output to diagnose a failed run, then map manual gradient steps to PyTorch's backward(), zero_grad() and optimizer update.

Common pitfalls

• Symptom: Parameters move in the wrong direction. Cause: Raw loss was subtracted instead of the gradient, or a derivative sign is wrong. Fix: Compare analytical gradients with centered finite differences on a tiny case.
• Symptom: A reused value such as a * a receives half its expected gradient. Cause: Graph paths overwrite rather than accumulate contributions. Fix: Update parent gradients with +=.
• Symptom: PyTorch gradients unexpectedly double across steps. Cause: An independent batch started before gradient buffers were cleared. Fix: Run optimizer.zero_grad() before backward() unless accumulation is deliberate.
• Symptom: Training loss falls while validation loss rises. Cause: The model is fitting a training-specific pattern, or data and split policy don't match evaluation. Fix: Inspect held-out examples, leakage, distribution shift and regularization before continuing.
• Symptom: Loss oscillates or grows rapidly. Cause: Learning rate may be too large for the local curvature. Fix: Lower step size and inspect gradient norms and numeric values.

Follow-up questions