Gradients and Backprop

Measuring engineering work turns vague performance into numbers. Training builds on the same instinct, but with a different problem: the system begins with poor numbers and must learn better ones from examples.

Suppose a coding assistant predicts response latency from prompt length. A request used 200 input tokens and finished in 5 seconds, but the current rule predicts 6. Which stored number should change, in which direction, and by how much? Gradients answer that question.

A model with one adjustable number

A parameter is a number a model is allowed to change while it learns. Start with one parameter, w, representing predicted seconds per 100 input tokens.

For one completed request:

The loss is one non-negative number. Zero means the prediction matches this row. Larger numbers mean the miss is worse.

That code evaluates a model. It doesn't yet learn. Learning requires knowing whether to move w upward or downward.

Nudge the weight to estimate a slope

Change w by a tiny amount and recompute the loss:

Loss increased by 0.004004 when the weight increased by 0.001. Divide those changes:

$$\text{slope estimate} = \frac{1.004004 - 1.000000}{0.001} = 4.004$$

A positive slope means increasing w makes loss worse near this point. To lower the loss, move w down.

This is a finite difference: a practical approximation to a derivative. It's slow if you repeat it for millions of parameters, but it's an excellent check for a small formula.

The derivative gives the exact local slope

A derivative is the slope as the nudge becomes infinitesimally small. Here the loss is:

$$L(w) = (2w - 5)^2$$

Apply the chain rule in two pieces:

$$\frac{dL}{dw} = 2(2w - 5) \times 2$$

The first factor differentiates the square. The final 2 comes from the prediction 2w: changing w by one changes the prediction by two seconds for this row.

At w = 3:

$$\frac{dL}{dw} = 2(6 - 5) \times 2 = 4$$

The finite-difference estimate was 4.004; the derivative is exactly 4 at this point.

A centered finite difference nudges left and right, so it's usually a better numerical check than a one-sided nudge.

Gradient descent takes a downhill step

The derivative tells you uphill. Gradient descent moves the opposite way. A learning rate is a small chosen step scale:

$$w_{\text{new}} = w - \text{learning rate} \times \frac{dL}{dw}$$

With w = 3, derivative 4, and learning rate 0.1:

$$w_{\text{new}} = 3 - 0.1 \times 4 = 2.6$$

Now the prediction is 2.6 * 2 = 5.2, and loss is (5.2 - 5)² = 0.04. One update reduced the loss from 1.0 to 0.04.

At w = 3, the derivative is positive 4. Why does the update subtract it?

Answer

The derivative points toward increasing loss. Subtracting a positive derivative moves w downward, from 3.0 to 2.6, and lowers this row's loss from 1.0 to 0.04.

A correct gradient can still take a bad step

Calculus supplies direction. It doesn't choose a safe learning rate for you.

From the same starting point, a learning rate of 1.0 produces:

$$w_{\text{new}} = 3 - 1.0 \times 4 = -1$$

The new prediction is -2 seconds and loss becomes (-2 - 5)² = 49. The derivative was right; the step jumped straight past the good region.

In a real training loop, watch for non-finite loss values such as NaN or infinity. They can signal an unstable step or invalid input. Later optimization chapters will teach adaptive optimizers and schedules; first you need a backward calculation you can trust.

Several parameters mean several partial derivatives

Response latency doesn't depend on prompt length alone. Add a second parameter for queue delay:

$$\hat{y} = w_p x_p + w_q x_q$$

Here x_p is prompt length in hundreds of tokens, and x_q is queue delay in seconds. For one request:

A partial derivative changes one parameter while holding the other fixed:

$$\frac{\partial L}{\partial w_p} = 2(\hat{y} - y)x_p = 2 \times 1 \times 2 = 4$$ $$\frac{\partial L}{\partial w_q} = 2(\hat{y} - y)x_q = 2 \times 1 \times 1 = 2$$

Collect those slopes in order and you have the gradient: [4, 2]. The next article gives that list its full vector-and-shape language. For now, read it as one correction signal per stored parameter.

For this one row, the update lands exactly on the target, so loss drops to 0. That is lucky with a single example. One row can illustrate an update, but it can't uniquely determine what prompt length and queue delay should contribute across all future requests. Learning useful weights requires several observed requests.

Backpropagation is chain-rule bookkeeping

The two-parameter gradient came from a chain of small operations:

1. Predict: prediction = w_p * x_p + w_q * x_q
2. Subtract target: error = prediction - y
3. Square error: loss = error ** 2

During the forward pass, these operations produce prediction = 6, error = 1, and loss = 1. During the backward pass, begin at the scalar loss and move backward:

Multiply along each path. Prompt length gets 2 * 1 * 2 = 4; queue delay gets 2 * 1 * 1 = 2.

Backpropagation efficiently repeats this reverse chain-rule calculation through a layered model. Rumelhart, Hinton, and Williams used back-propagated error derivatives to learn internal representations in multilayer networks in their 1986 Nature paper.^{[1]Reference 1Learning Representations by Back-Propagating Errors.https://doi.org/10.1038/323533a0}

The same bookkeeping works when a layer sits between the weight and the loss. Deep models insert nonlinearities such as ReLU so intermediate units can turn on or off. Those local derivatives (1 or 0 for ReLU) join the chain just like the feature multipliers above.

Take a three-node path: scale the prompt length, apply ReLU, then square the residual against a target activation:

$$h = w_1 x,\quad z = \mathrm{ReLU}(h),\quad L = (z - y)^2$$

With x = 2, w_1 = 1.5, and y = 1:

Multiply backward. The intermediate activation gradient is ∂L/∂h = 4 × 1 = 4. The weight gradient is ∂L/∂w_1 = 4 × 2 = 8. If h had been negative, ReLU's local derivative would be 0, so the weight would receive no update for this example even when the residual is nonzero.

Later deep-learning lessons stack many such local factors. The rule stays the same: store forward values, multiply local derivatives on the way back, and use zero local derivatives where a unit is inactive.

In h = wx, z = ReLU(h), and L = (z-y)^2, what happens to dL/dw when h < 0?

Answer

It becomes zero for that example because ReLU's local derivative is zero. The upstream loss signal is multiplied by that zero before it reaches w.

When paths meet, add their signals

The example above sends the loss signal to two different parameters. A computation graph can also use the same parameter more than once. When backward paths meet at one parameter, add their contributions.

Use an intentionally simple latency rule to isolate that idea. The same weight w scales both the prompt-length estimate and the queue-delay estimate:

$$\hat{y} = w x_p + w x_q$$

With w = 2, x_p = 2, x_q = 1, and y = 5, prediction is 6 and error is 1. The prompt path contributes 2 * 1 * 2 = 4. The queue path contributes 2 * 1 * 1 = 2. Both paths reach w, so its gradient is 4 + 2 = 6.

One weight is used on two computation paths whose gradient contributions are 4 and 2. What gradient should the optimizer receive for that weight?

Answer

6. When backward paths meet at one parameter, add their contributions rather than choosing one path or multiplying them together.

This scalar rule is deliberately small. Larger models reuse parameters across many rows or token positions, and branches can merge inside a graph. The next batch example combines row contributions too, then divides by the number of rows because its loss is a mean.

Build it: learn from a small latency table

Now use four completed requests rather than one. Each row has prompt length and recorded queue delay; the target is observed response seconds. You already used NumPy arrays earlier in the curriculum, so X @ weights computes all four predictions at once.

The finite-value guard doesn't make training stable by itself. It stops the loop before a NaN or infinity update silently corrupts the weights. If it fires, inspect the learning rate and input scale first.

This is still a tiny model, not a production latency service. Its value is transparency: every prediction, loss, derivative, and update fits into code you can explain.

Gradient checking catches backward bugs

A manual backward formula can silently omit a factor, transpose the wrong array, or reuse the wrong value. Compare it to centered finite differences before trusting it.

For each parameter, nudge only that one value by eps, compute the two losses, and compare the numerical slope to the formula from your backward pass.

Finite differences are far too expensive for normal training because they need repeated forward computations per parameter. They are ideal for testing a small backward formula.

Catastrophic cancellation in single precision

Gradient checks usually need double precision (float64). PyTorch's default gradcheck tolerances are designed for double-precision inputs; lower-precision inputs often fail even when the analytical gradient is correct.^{[2]Reference 2Gradcheck mechanicshttps://docs.pytorch.org/docs/stable/notes/gradcheck.html}

A centered difference subtracts two nearly identical losses: loss(w + eps) - loss(w - eps). When these values agree to many decimal places, subtracting them cancels the matching leading digits. This catastrophic cancellation leaves only a few bits of precision, which are dominated by round-off noise.

With float32, an eps around 1e-5 to 1e-6 can leave little signal after subtraction. This check uses PyTorch's default-sized perturbation to show why double precision is the safer default:

PyTorch computes the same backward pass

Reverse-mode automatic differentiation propagates chain-rule products backward from outputs to inputs. It's especially useful when one scalar loss depends on many parameters, which is the training shape we have here.^{[3]Reference 3Automatic Differentiation in Machine Learning: a Surveyhttps://www.jmlr.org/papers/v18/17-468.html}

PyTorch autograd records operations involving tensors that request gradients, then exposes the reverse pass through a call to backward().^{[4]Reference 4PyTorch: An Imperative Style, High-Performance Deep Learning Library.https://arxiv.org/abs/1912.01703} This cell repeats the small latency table and checks PyTorch against the manual NumPy derivative. It also runs backward twice to expose one important default: PyTorch adds each new result into the leaf tensor's .grad buffer.

The second backward pass doubles the stored numbers because it adds the same gradient again. A training loop normally calls optimizer.zero_grad() before the next batch, or sets buffers to None, unless accumulating across several batches is intentional.^{[5]Reference 5Optimizing Model Parameters.https://docs.pytorch.org/tutorials/beginner/basics/optimization_tutorial.html} Frameworks save you from writing every derivative. They don't remove your responsibility to understand why a gradient has its sign, shape, and size.

PyTorch autograd traps: in-place writes and shared storage

PyTorch tracks operations on tensors to construct the computation graph for backpropagation. In-place writes can invalidate saved values or mutate aliased tensors.^{[6]Reference 6Autograd mechanicshttps://docs.pytorch.org/docs/stable/notes/autograd.html}

In-place mutation of saved variables

To compute gradients, PyTorch must save certain forward-pass activation values. If you mutate one of these variables in place before calling .backward() (using operations like +=, relu_(), or direct slice assignment), PyTorch raises a runtime error because the saved value needed to compute the gradient has been overwritten.

Shared-storage mutations

Tensors created via operations like .view(), .slice(), or .expand() share memory storage with their parent tensors. Mutating a view in place silently alters the parent tensor. In particular, .expand() creates a view with a stride of 0 along the expanded dimensions, meaning multiple indices refer to the exact same physical memory location. PyTorch prevents in-place writes on these expanded views to avoid corrupting the gradient graph.

This cell triggers both protections:

Practice with pencil before code

Use one latency row with x_prompt = 3, x_queue = 1, actual seconds y = 7, and weights [w_prompt, w_queue] = [2, 2].

1. Compute prediction, error, and squared loss.
2. Compute both partial derivatives by multiplying 2 * error by the relevant feature.
3. Apply one gradient-descent update with learning rate 0.05. What is the new prediction and loss?
4. If a centered finite-difference check for the prompt parameter returns 5.99999 while your manual derivative says 6, is that evidence of a bug?
5. If loss becomes NaN after a very large update, what two things should you inspect first?

Solution sketches

1. Prediction is 2*3 + 2*1 = 8, error is 8 - 7 = 1, and loss is 1.
2. The prompt partial is 2*1*3 = 6; the queue partial is 2*1*1 = 2. The gradient is [6, 2].
3. New weights are [2 - 0.05*6, 2 - 0.05*2] = [1.7, 1.9]. Prediction is 1.7*3 + 1.9 = 7.0, so loss is 0.
4. No. A numerical derivative is approximate, and agreement within a small tolerance with the same sign is a passing check.
5. Inspect the learning rate and the scale or validity of input values. A huge step or non-finite data can make the forward loss invalid before a later optimizer can help.

From gradients to optimizer state

Training should now read as a concrete process:

We deliberately stopped before optimizer algorithms such as momentum or Adam. They control how to use gradients over many steps; a later chapter teaches them once vectors, matrices, and deeper linear algebra are available.

Gradient debugging checklist

Before trusting an update, verify that the forward loss is finite, the gradient has the expected sign and shape, a centered finite difference agrees within tolerance, and one small step lowers loss on the checked example. If the check passes but training diverges, inspect learning rate and input scale before rewriting backpropagation.