Training & Backpropagation

In the previous lesson, you traced a CNN score while its kernel values stayed fixed. A useful detector can't be supplied by hand for every visual pattern, document layout, or runtime trace. The model has to discover parameter values from examples.

You already derived gradients and parameter updates for a latency model and then studied optimizer rules. Here those ideas reconnect with dense layers, ReLU, and max pooling from the two network lessons, then expand into batches and held-out validation.

Make that discovery visible with a small regression problem: predict extra decode latency from a prompt-complexity score. A real neural network has far more parameters, but each weight changes for the same reason as the one weight below: changing it changes a prediction, which changes a loss.

One update has four distinct jobs

A model learns by repeating four steps:

Suppose a request has prompt-complexity score x = 2 and later measures y = 6 extra milliseconds of decode latency. Start with this model:

$$\hat{y} = wx + b$$

Set w = 1 and b = 0. The predicted extra delay is 2 ms, far below the observed 6. For this numerical lesson, use half squared error:

$$L = \frac{1}{2}(\hat{y} - y)^2$$

The factor 1/2 doesn't change which parameters are good; it cancels the 2 when we differentiate the square. Here the initial loss is:

$$L = \frac{1}{2}(2 - 6)^2 = 8$$

The derivative of loss with respect to the prediction is -4: increasing the predicted delay would reduce this mistake. Because w is multiplied by x = 2, its gradient is twice as large:

$$\frac{\partial L}{\partial \hat{y}} = \hat{y} - y = -4, \qquad \frac{\partial L}{\partial w} = (\hat{y} - y)x = -8, \qquad \frac{\partial L}{\partial b} = \hat{y} - y = -4$$

With learning rate $\eta = 0.1$, subtract those gradients:

$$w_{\text{new}} = 1 - 0.1(-8) = 1.8, \qquad b_{\text{new}} = 0 - 0.1(-4) = 0.4$$

The revised prediction is 1.8 * 2 + 0.4 = 4.0 ms, and loss falls from 8.0 to 2.0.

One update helps, but training is a repeated correction:

Step size can rescue or wreck learning

The gradient says which direction lowers loss locally, and the learning rate says how far to move. A very small rate makes slow progress; a useful rate settles toward a low-loss setting; and a large rate can jump past the target, growing the error on every oscillation.

The same one-example problem makes this visible because every value except learning rate stays constant:

Don't assume that one loss decrease proves the rate is safe. Real batches disagree with one another, and deep networks can have steep regions in some directions and shallow ones in others. Training code should log loss and gradient norms so a bad run becomes visible quickly.

Backpropagation assigns credit through a graph

Gradient descent needs dL/dw for every trainable parameter. Calculating it by nudging each parameter separately is useful as a check for a tiny formula, but it would repeat forward evaluation for every parameter direction. Backpropagation uses reverse-mode automatic differentiation: it reuses the forward computation graph and moves derivatives backward from one scalar loss through each local operation.^{[1]Reference 1Deep Learning.https://www.deeplearningbook.org/}

Rumelhart, Hinton, and Williams showed that this error-driven weight adjustment could learn useful hidden representations in multilayer networks in their 1986 Nature paper.^{[2]Reference 2Learning Representations by Back-Propagating Errors.https://doi.org/10.1038/323533a0} For our small decode-latency model, the graph is already enough to see the mechanism:

The chain rule follows paths from loss back to parameters:

$$\frac{\partial L}{\partial w} = \frac{\partial L}{\partial \hat{y}} \frac{\partial \hat{y}}{\partial w} = (-4)(2) = -8$$ $$\frac{\partial L}{\partial b} = \frac{\partial L}{\partial \hat{y}} \frac{\partial \hat{y}}{\partial b} = (-4)(1) = -4$$

A useful engineering habit is to compare an analytical gradient with a centered finite difference. The check is slow for training, but excellent for finding a missing factor or wrong sign in a new backward formula.

Build a scalar autograd engine

Andrej Karpathy's Micrograd makes the backward sweep inspectable: each object stores one scalar, its accumulated gradient, the values that produced it, and a local backward function.^{[3]Reference 3micrograd.https://github.com/karpathy/micrograd} Tensor libraries apply the same principle to matrix operations; a scalar version keeps the accounting readable.

For prediction = w * x + b and loss = 0.5 * (prediction - target) ** 2, every operation leaves a small backward rule:

This compact engine builds the computation graph during the forward pass, orders nodes so children run before parents during the backward pass, and accumulates gradients with +=.

One rule is easy to miss: if a value affects loss through two paths, its gradient is the sum of those paths. The simple expression a * a exposes an incorrect engine immediately.

In a tensor framework, parameter gradient buffers also accumulate until they are cleared. That makes intentional gradient accumulation possible, but it also creates a common bug: forgetting to clear gradients between independent optimizer steps.^{[4]Reference 4Optimizing Model Parameters.https://docs.pytorch.org/tutorials/beginner/basics/optimization_tutorial.html}

The same local rules stack through multilayer networks: each operation only needs its local derivative and the upstream error signal $\delta$. The scalar engine above is the bookkeeping pattern; the next three rules are the ones you meet right after the dense and CNN forward passes.

Local rules for dense layers, ReLU, and max pooling

For one dense unit $z = w^\top x + b$ with upstream signal $\delta = \partial L / \partial z$:

$$\frac{\partial L}{\partial w} = \delta\, x,\qquad \frac{\partial L}{\partial b} = \delta,\qquad \frac{\partial L}{\partial x} = \delta\, w$$

In matrix form with $z = Wx + b$ and $\delta = \partial L/\partial z$ shaped like $z$, the weight gradient is the outer product $dW = \delta x^\top$ (equivalently np.outer(delta, x) when both are 1-D), and the input gradient is $dx = W^\top \delta$.

ReLU is $h = \max(0, z)$. Backward multiplies the upstream signal by a mask: pass $\delta$ only where $z > 0$, and send zero where the forward pass clipped.

Max pooling sends the pooled region's entire upstream gradient to the winner saved during the forward pass, and zero to the other cells in that region. This backward contract requires the winner coordinates saved by the CNN lesson.

Libraries implement these as fused kernels, but the accounting is the same: outer-product weight grads, mask for ReLU, winner-only routing for max pool. Classification heads swap the half-SE $\delta$ for the softmax cross-entropy signal $p - y$ in the next lesson; the dense / ReLU / pool rules above stay unchanged.

Move from one request to a batch

One request doesn't represent a serving workload. Training normally estimates a useful update from a mini-batch, a subset of training examples processed before one parameter update. For mean loss, the batch gradient is the average of the example gradients.

The dataset contains only three requests, so the first pass uses all three at once: technically, it's full-batch gradient descent. The arithmetic is the same one a mini-batch uses; a larger run uses a subset before each update. Here the true relationship is extra_latency = 2 * complexity + 1. Starting from zero parameters:

An epoch means one complete pass through the training examples. LLM pretraining runs are commonly described in optimizer steps and tokens processed because the dataset is very large; the batch contract remains the same: forward, average loss, backward, update.

Validation loss tests whether learning transfers

Training loss only measures examples used for updates. A validation set contains held-out examples used for evaluation, not for backpropagation. If training loss drops while validation loss rises, the new parameters fit the training set better but transfer worse. That pattern is a warning, not a diagnosis: investigate data mismatch, leakage, noise, model capacity and regularization before deciding why it happened.

This deliberately mismatched example makes the warning visible. Training records follow a peak-season delay pattern; validation records follow a normal-week pattern. Starting from a model that already matches normal weeks, fitting peak season improves train loss and damages validation loss:

For a real run, compare train and validation curves at consistent checkpoints and log enough context to reproduce the run: data version, split policy, batch size, learning rate, optimizer, seed and checkpoint identifier.

See the same responsibilities in PyTorch

Libraries remove manual derivative bookkeeping, not the conceptual separation. Multiplying PyTorch's mean squared error by 0.5 keeps the same half-squared-error convention used above. loss.backward() fills parameter gradients and optimizer.step() applies an update. The loss drops after one step on the same three-example batch.^{[4]Reference 4Optimizing Model Parameters.https://docs.pytorch.org/tutorials/beginner/basics/optimization_tutorial.html}

Because gradients accumulate, two backward calls without clearing produce twice the gradient for the same loss:

Why the next loss function matters

We used half squared error because predicting decode latency in milliseconds is a regression problem and its derivative is easy to inspect. A CNN that selects damaged, intact, or needs-review, and an LLM that selects the next token, face a classification problem: the output layer produces competing logits rather than one continuous millisecond estimate.

The backward-pass mechanics won't disappear. The next lesson supplies the missing objective: softmax converts logits into a distribution, cross-entropy scores the correct choice, and their gradient becomes the signal sent backward through the model.

Mastery check

Key concepts

• Training separates forward computation, loss evaluation, backward differentiation and parameter update.
• A gradient measures loss sensitivity; gradient descent moves in the opposite direction.
• Backpropagation applies local chain-rule derivatives in reverse order through a computation graph.
• A finite-difference check verifies a formula but isn't an efficient training method.
• Shared graph paths and repeated backward calls require gradient accumulation.
• With mean loss, mini-batches provide one averaged update from a subset of training examples.
• Validation loss measures transfer to held-out data and can reveal a failing run.

Evaluation rubric

• Foundational: Compute prediction, half squared loss, dL/dw, dL/db and one SGD update for the decode-latency example.
• Intermediate: Explain why reverse-mode differentiation is preferable to nudging every parameter, and implement a small scalar graph that accumulates gradients correctly.
• Advanced: Use mini-batch and validation output to diagnose a failed run, then map manual gradient steps to PyTorch's backward(), zero_grad() and optimizer update.

Common pitfalls

• Symptom: Parameters move in the wrong direction. Cause: Raw loss was subtracted instead of the gradient, or a derivative sign is wrong. Fix: Compare analytical gradients with centered finite differences on a tiny case.
• Symptom: A reused value such as a * a receives half its expected gradient. Cause: Graph paths overwrite rather than accumulate contributions. Fix: Update parent gradients with +=.
• Symptom: PyTorch gradients unexpectedly double across steps. Cause: An independent batch started before gradient buffers were cleared. Fix: Run optimizer.zero_grad() before backward() unless accumulation is deliberate.
• Symptom: Training loss falls while validation loss rises. Cause: The model is fitting a training-specific pattern, or data and split policy don't match evaluation. Fix: Inspect held-out examples, leakage, distribution shift and regularization before continuing.
• Symptom: Loss oscillates or grows rapidly. Cause: Learning rate may be too large for the local curvature. Fix: Lower step size and inspect gradient norms and numeric values.

Follow-up questions

The latency model predicts 2 ms when the target is 6 ms, using half squared error and x = 2. Why are dL/dw and dL/db different?

Answer

The loss slope with respect to the prediction is -4. Bias shifts the prediction one-for-one, so dL/db is -4. Weight is multiplied by x = 2, so a one-unit weight change moves prediction by two milliseconds and dL/dw is -8.

Why is a finite-difference gradient check useful but unsuitable as the training algorithm for a large model?

Answer

It checks a derivative by evaluating loss at a small positive and negative parameter nudge, so it can reveal a wrong analytic formula on a small graph. Repeating that process for every parameter direction requires many forward evaluations. Backpropagation reuses the computation graph and computes the full gradient from one scalar loss in a reverse sweep.

Your scalar autograd engine returns d(a * a)/da = 3 when a = 3. What should it return, and what code pattern fixes the bug?

Answer

It should return 6 because a influences the product through both inputs: one contribution of 3 from each path. Each backward rule must add into parent gradients with += rather than replace a previous contribution.

A training run's loss improves while its held-out validation loss gets steadily worse. What does that prove, and what does it not prove?

Answer

That pattern proves current updates fit training examples better while performance on held-out examples deteriorates. It doesn't by itself identify the cause. Check split leakage, distribution shift, label quality, model capacity, learning rate and regularization before deciding on a fix.

When is an update full-batch gradient descent rather than mini-batch gradient descent?

Answer

A full-batch update uses every training example to compute one gradient. A mini-batch update uses a subset. Both can average example losses and gradients, but mini-batches reduce work per update and can produce different subset-based updates as training proceeds.