Gradients and Backprop

A request takes 5 seconds, but a model predicts 6. The model's rule is simple: multiply the number of 100-token blocks by an adjustable number. For this 200-token request, that number is currently 3. Should we increase it or decrease it to make a better prediction?

The NumPy lesson taught how to organize and calculate with model inputs. Training adds another step: use a prediction's error to decide how the model's stored numbers should change. Start with just one number so you can trace the whole update by hand.

A model with one adjustable number

A parameter is a stored number a model adjusts as it learns from data. Start with a single parameter, w: predicted response seconds per 100 input tokens. Tokens are the text chunks processed by a language model. This latency predictor is intentionally stripped down so you can trace every moving piece without getting lost in tensor bookkeeping.

For one completed request:

The loss measures prediction error with one number. Squaring makes both a one-second overestimate and a one-second underestimate cost 1; an error of two seconds costs 4. Zero means a perfect prediction for this row. This squared-error loss has squared-seconds units rather than seconds. Lowering it improves our prediction of latency, not the service's actual response time.

Before running the cell, predict its two printed values. The table already gives you enough: the rule predicts 3 * 2 = 6, so the error is one second.

That code scores a model, but it doesn't yet learn. The loss tells us how wrong its prediction is; the next question is directional: should w go up or down?

Read a slope by nudging one weight

Change only w by a tiny amount and recompute the loss. Before looking at the second row, predict what should happen: the current prediction is already too high, so an upward nudge should make the miss worse.

Loss increased by 0.004004 when the weight increased by 0.001. Divide the change in loss by the change in weight:

$$\text{slope estimate} = \frac{1.004004 - 1.000000}{0.001} = 4.004$$

A positive slope means increasing w makes loss worse near this point. To lower the loss, move w down. The sign gives a direction; the size estimates the change in loss per unit change in weight. Check that ratio with the same nudge in Python:

This is a finite difference, a practical approximation to a derivative. A forward evaluation means running the prediction and loss calculation at chosen parameters. A one-sided check needs a new evaluation for each parameter you nudge; a centered check needs two. That becomes expensive for large models, but it's useful for checking a small formula.

The derivative gives the exact local slope

The nudge estimated 4.004. What would remain if we made the nudge smaller and smaller? A derivative is the slope that the difference ratio approaches, when that limit exists.^{[1]Reference 1Calculus Volume 1: Defining the Derivativehttps://openstax.org/books/calculus-volume-1/pages/3-1-defining-the-derivative} At our current w = 3, a nudge of size h gives prediction 6 + 2h, error 1 + 2h, and loss (1 + 2h)². Expand the square and divide the loss change by h:

$$\frac{(1 + 2h)^2 - 1}{h} = \frac{4h + 4h^2}{h} = 4 + 4h \qquad (h \ne 0)$$

For h = 0.001, this is 4.004, exactly the estimate above. As h approaches zero, 4h vanishes and the slope approaches 4. We never divide by zero; we ask what the ratio approaches for nonzero nudges.

Now write a rule that works at other weights too. Let L name the loss:

$$L(w) = (2w - 5)^2$$

For any error e, the same expansion gives ((e + h)² - e²) / h = 2e + h, which approaches 2e. That's why the derivative of a square is twice its input. Our error is e = 2w - 5, which changes twice as fast as w. Multiply those two sensitivities:

$$\frac{dL}{dw} = 2(2w - 5) \times 2$$

Read dL/dw as "the derivative of loss with respect to weight." Differentiating the square gives the first factor. The final 2 comes from the prediction 2w: changing w by a small amount changes the prediction by twice that amount. This multiplication is the chain rule: combine sensitivities along a sequence of dependent calculations.^{[2]Reference 2Calculus Volume 1: The Chain Rulehttps://openstax.org/books/calculus-volume-1/pages/3-6-the-chain-rule}

At w = 3:

$$\frac{dL}{dw} = 2(6 - 5) \times 2 = 4$$

The finite-difference estimate was 4.004; the derivative is exactly 4 at this point. Check it by comparing losses on both sides of w. The two weights are 2 * eps apart, so divide by that distance:

A centered finite difference uses the losses at w + eps and w - eps. For this quadratic, the extra nudge terms cancel exactly in real arithmetic; the remaining discrepancy comes from floating-point rounding. For general smooth functions, it usually reduces approximation error compared with the same-sized one-sided nudge.

Gradient descent takes a downhill step

With one parameter, the derivative is the one-entry gradient. Its sign tells whether increasing w raises or lowers loss locally. Gradient descent moves opposite that slope. A learning rate is the number we choose to scale the update.

Here the slope is positive. Adding the slope would move right, toward higher loss, so the update must subtract it.

$$w_{\text{new}} = w - \text{learning rate} \times \frac{dL}{dw}$$

With w = 3, derivative 4, and learning rate 0.1:

$$w_{\text{new}} = 3 - 0.1 \times 4 = 2.6$$

Now the prediction is 2.6 * 2 = 5.2, and loss is (5.2 - 5)² = 0.04. One update reduced the loss from 1.0 to 0.04. Repeat the calculation, recomputing the slope at the new weight each time. Each printed row shows the state before its update:

Read down the rows rather than only the final weight. Each update lowers the loss, and the gradient shrinks as w approaches 2.5. At that weight, the prediction is exactly 5, loss is zero, and the slope is zero. This is a minimum because squared loss can't be negative. In other functions, a zero derivative alone doesn't prove you've found a minimum.

The slope is local. It doesn't tell you how far to walk.

A correct gradient can still take a bad step

Calculus supplies direction. It doesn't choose a safe learning rate for you.

Keep the starting point and gradient fixed, then change only the learning rate. Predict the failure before calculating it: a rate ten times larger follows the same downhill direction, but travels much farther.

From the same starting point, a learning rate of 1.0 produces:

$$w_{\text{new}} = 3 - 1.0 \times 4 = -1$$

The new prediction is -2 seconds and loss becomes (-2 - 5)² = 49. The derivative was right; the step jumped straight past the good region.

Why 0.125? The minimum is 0.5 weight units below the start, so the required step satisfies learning_rate * 4 = 0.5. Try 0.25 in the cell: it moves to w = 2, where loss is back to 1. Following the correct gradient direction doesn't guarantee that a finite step lowers loss.

In a real training loop, treat a non-finite loss such as NaN or infinity as a diagnosis signal. It can come from an unstable step or invalid input. Later optimization chapters will cover adaptive optimizers and schedules; first you need a backward calculation you can trust.

One weight was enough to see a downhill step. A latency model still needs to separate prompt work from queue delay.

Two weights, two slopes

The same request has one second of recorded queue delay. Add a second parameter to scale that delay, and reset the two weights to [2, 2]. The prediction is 2 * 2 + 2 * 1 = 6, still one second too high. In notation, let x_p count prompt blocks, x_q record queue seconds, and put a hat on y to distinguish the prediction from the observed time:

$$\hat{y} = w_p x_p + w_q x_q$$

Here w_p has units of seconds per prompt block, while w_q is a unitless multiplier on queue seconds. The feature values are 2 and 1, so in these chosen units the same loss signal produces a prompt slope twice as large. A larger numerical gradient doesn't by itself mean a more important feature: changing the feature's units changes its gradient.

A partial derivative measures sensitivity to one parameter while holding the others fixed. The symbol ∂ replaces d to show that the loss has several adjustable inputs:

$$\frac{\partial L}{\partial w_p} = 2(\hat{y} - y)x_p = 2 \times 1 \times 2 = 4$$ $$\frac{\partial L}{\partial w_q} = 2(\hat{y} - y)x_q = 2 \times 1 \times 1 = 2$$

Collect those slopes in parameter order and you have the gradient: [4, 2]. This ordered list is a vector; no extra machinery is needed yet. Apply the same update rule separately to each weight:

For this row and learning rate, the update lands exactly on the target. But [2, 1] and [1.5, 2] would both predict 5 too. One row can't tell us how much prompt length and queue delay should each contribute across future requests. We'll need more rows after we've checked how these two slopes were calculated.

The two slopes aren't a special two-weight trick. They are the chain rule, written backward.

Backpropagation is chain-rule bookkeeping

The two-parameter gradient came from a chain of small operations:

1. Predict: prediction = w_p * x_p + w_q * x_q
2. Subtract target: error = prediction - y
3. Square error: loss = error ** 2

During the forward pass, these operations produce prediction = 6, error = 1, and loss = 1. Now predict the first signal sent backward: differentiating error² should give 2 * error = 2. Start at the scalar loss and move backward:

Multiply local factors along each path. Prompt length gets 2 * 1 * 2 = 4; queue delay gets 2 * 1 * 1 = 2. The route tells you which feature scales each parameter's correction. At the loss itself, start with dL/dL = 1: a unit change in loss changes loss by one unit.

Backpropagation is that reverse walk through the computation. Operations retain the forward values their backward rules need, then multiply the arriving signal by a local derivative. Rumelhart, Hinton, and Williams used back-propagated error derivatives to learn internal representations in multilayer networks in their 1986 Nature paper.^{[3]Reference 3Learning Representations by Back-Propagating Errors.https://doi.org/10.1038/323533a0} The explicit calculation below keeps each factor visible:

The same bookkeeping works when an extra operation sits between a weight and the loss. So far, every predicted term changes proportionally with its weight. An operation with different behavior on different parts of its input range is nonlinear.

For example, wrap the prompt term in ReLU (rectified linear unit), max(0, h). It passes a positive input and returns zero for a negative input. Name the value before ReLU h and the value after it z:

$$h = w_p x_p,\quad z = \mathrm{ReLU}(h),\quad \hat{y} = z + w_q x_q,\quad L = (\hat{y} - y)^2$$

ReLU's local derivative is 1 when h > 0: a small increase passes through unchanged. It's 0 when h < 0: a small increase still leaves the output at zero. At h = 0, the left and right slopes disagree, so there's no ordinary derivative. This code uses a backward value of 0 at the corner, matching PyTorch's convention.^{[4]Reference 4Autograd mechanicshttps://docs.pytorch.org/docs/stable/notes/autograd.html}

Keep the same latency row, with w_p = 2 so h = 4 stays positive. The ReLU is a pass-through, and the prompt weight still receives 4.

Now flip the prompt weight to -0.5. h goes negative, ReLU outputs 0 seconds from the prompt path, and that weight gets no update. Predict that zero before running the cell.

The printed -0.0 compares equal to zero; its sign comes from multiplying a negative loss signal by zero. Later deep-learning lessons stack many such local factors. The rule stays the same: store needed forward values and multiply local derivatives on the way back.

When paths meet, add their signals

The example above sends the loss signal to two different parameters. A computation graph can also use the same parameter more than once. Changing a shared parameter changes both paths, so their contributions to the loss must add.

For this example only, tie the two numerical coefficients to one stored value w. Treat the inputs as normalized numbers: x_p is prompt length divided by 100 tokens, x_q is queue time divided by one second, and the output is measured in seconds. Both inputs are then unitless, so a shared seconds-valued coefficient is consistent:

$$\hat{y} = w x_p + w x_q$$

With w = 2, x_p = 2, x_q = 1, and y = 5, prediction is 6 and error is 1. The prompt path contributes 2 * 1 * 2 = 4; the queue path contributes 2 * 1 * 1 = 2. Both paths reach w, so its gradient is 4 + 2 = 6.

You can also combine the forward terms first: w * 2 + w * 1 = w * 3. Its local derivative is 3, so the complete derivative is 2 * error * 3 = 6. The two routes agree. Verify the separate contributions in code:

This scalar rule is deliberately small. Larger models reuse parameters across many rows or token positions, and branches can merge inside a graph. The next batch example combines row contributions too, then divides by the number of rows because its loss is a mean.

Four completed requests

Return to two independent weights and use four completed requests. Each row has prompt length and queue delay, and the target is observed response seconds. These are constructed data: targets follow the rule 2 * prompt_blocks + queue_seconds exactly.

Use the mean squared error: compute each row's squared error, add them, then divide by four. Its gradient follows the same pattern. Compute 2 * error * feature for each row, add the contributions for each weight, and divide by four.

At initial weights [0, 0], all predictions are zero. The prompt-weight contributions are [-5, -20, -39, -80], whose mean is -36. The queue-weight contributions are [-2.5, -10, -6.5, -40], whose mean is -14.75. With learning rate 0.03, the first updated weights must be [1.08, 0.4425].

In NumPy, store the four rows as X with shape (4, 2). From the earlier array lesson, X @ weights multiplies each row's two features by the corresponding weights and adds them, giving four predictions. The gradient below uses explicit feature columns, so you can match it to the arithmetic above:

The known weights are [2, 1], but 101 updates stop near that solution rather than landing on it. The loss falls quickly at first, then more slowly. A fixed iteration count is a stopping choice, not a promise that optimization has finished. Close predictions on these four training rows also don't tell us how well the rule predicts unseen requests.

You'll often see the two column means written compactly as (2 / len(X)) * (X.T @ errors). Here X.T has shape (2, 4): each row now holds one feature's values across the four requests. Multiplying by the four errors adds feature-weighted errors, then 2 / len(X) supplies the square's factor and the mean. Both versions compute the same two slopes; the next linear-algebra lessons develop that notation further.

A finite-value guard doesn't make training stable by itself. It stops the loop before a NaN or infinity update silently corrupts the weights. If it fires, inspect the learning rate and input scale first.

This is still a tiny model, not a production latency service. Its value is transparency: every prediction, loss, derivative, and update fits into code you can explain.

A manual backward formula can silently omit a factor. Before trusting it on a larger batch, compare each parameter's formula to a nudge.

Gradient checking catches backward bugs

For each parameter, nudge only that value by eps, compute the two losses, and compare the numerical slope to the formula from your backward pass. Use weights [1.3, 1.1], away from the minimum so the check tests nonzero slopes. The two gradients should agree component by component, not just in overall direction.

Try removing the factor 2 from manual_gradient and rerun the cell. The assertion should fail: numerical differences are evaluating the actual squared loss, not your proposed derivative. Restore the factor before continuing. Passing at one point supports the formula for that case; it isn't a proof for every input.

That comparison is only trustworthy in enough precision. Single-precision floats can make a correct backward rule look wrong.

Catastrophic cancellation in single precision

Use double precision (float64) for these small gradient checks. The check subtracts nearly equal loss values, and those values have already been rounded by floating-point arithmetic. After the matching leading digits cancel, the tiny difference can contain a large relative error. This is catastrophic cancellation.

Making eps smaller reduces approximation error only until rounding starts to dominate. Keep the one-weight latency loss and use the same eps = 1e-6 in two dtypes. Both analytical derivatives are 4, but the numerical checks needn't be equally accurate:

The formula hasn't changed, only its numerical evaluation. Framework gradient checkers face the same limit: PyTorch's gradcheck defaults are intended for double-precision inputs.^{[5]Reference 5Gradcheck mechanicshttps://docs.pytorch.org/docs/stable/notes/gradcheck.html} Also avoid testing exactly at ReLU's corner, where no ordinary derivative exists.

Let PyTorch repeat the backward pass

The manual check agrees. Now let a framework repeat the same calculation. Reverse-mode automatic differentiation propagates chain-rule products backward from outputs to inputs. It's especially useful when one scalar loss depends on many parameters, which is the training shape we have here.^{[6]Reference 6Automatic Differentiation in Machine Learning: a Surveyhttps://www.jmlr.org/papers/v18/17-468.html}

PyTorch arrays are called tensors. Its autograd system records differentiable operations on tensors that request gradients, then computes the reverse pass when you call backward().^{[4]Reference 4Autograd mechanicshttps://docs.pytorch.org/docs/stable/notes/autograd.html} Setting requires_grad=True on the weights asks for their derivatives. This cell repeats the four-row latency table and checks PyTorch against the manual NumPy derivative.

The weights are a leaf tensor: we created them directly, rather than calculating them from another tracked tensor. PyTorch stores their derivatives in .grad. Calling backward again adds to that buffer; it doesn't replace it. Predict the second value before running the cell. The detach().clone() calls below save independent snapshots of the gradient without tracking the copies themselves.

The second backward pass doubles the stored numbers because it adds the same gradient again. A training loop normally calls optimizer.zero_grad() before the next batch, or sets buffers to None, unless accumulating across several batches is intentional.^{[7]Reference 7Optimizing Model Parameters.https://docs.pytorch.org/tutorials/beginner/basics/optimization_tutorial.html}

Adding contributions within one backward pass follows the chain rule. Keeping contributions across separate backward calls is a framework accumulation policy. Both use addition, but clearing .grad controls only the second behavior. Also notice that backward() hasn't updated w: an optimizer or explicit subtraction must apply the gradient afterward.

In-place writes can break the backward pass

PyTorch tracks operations on tensors to construct the computation graph for backpropagation. In-place writes can invalidate saved values or mutate aliased tensors.^{[4]Reference 4Autograd mechanicshttps://docs.pytorch.org/docs/stable/notes/autograd.html}

To compute some derivatives, autograd saves forward-pass values. Overwriting a saved value before .backward() can make that derivative impossible to compute correctly, so PyTorch checks for changes and raises when the backward pass needs an invalidated value. Not every in-place operation is forbidden; the dependency on the overwritten value determines the problem.

Views are a second trap, similar to the shared buffers in NumPy. Expanding a size-1 axis with .expand() uses an array stride of 0, so several output indices refer to the same memory location. Updating all those entries at once with add_() would write repeatedly to that location. PyTorch rejects this overlapping write; use .clone() first when you need independent entries.^{[8]Reference 8torch.Tensor.expandhttps://docs.pytorch.org/docs/stable/generated/torch.Tensor.expand.html}

The two cells below isolate storage behavior rather than fit latency. The first uses sigmoid, a smooth function that squashes a number into the range (0, 1). If its output is b, its derivative is b * (1 - b), so backward needs the original b. Changing that output breaks the dependency. The second cell writes through a view whose entries share storage. Read each error as a diagnosis of a different problem.

Test the loop on another request

Try one more latency row before opening the solution sketches: x_prompt = 3, x_queue = 1, actual seconds y = 7, and weights [w_prompt, w_queue] = [2, 2].

1. Compute prediction, error, and squared loss.
2. Compute both partial derivatives by multiplying 2 * error by the relevant feature.
3. Apply one gradient-descent update with learning rate 0.05. What is the new prediction and loss?
4. If a centered finite-difference check for the prompt parameter returns 5.99999 while your manual derivative says 6, is that evidence of a bug?
5. If loss becomes NaN after a very large update, what two things should you inspect first?

Solution sketches

1. Prediction is 2*3 + 2*1 = 8, error is 8 - 7 = 1, and loss is 1.
2. The prompt partial is 2*1*3 = 6; the queue partial is 2*1*1 = 2. The gradient is [6, 2].
3. New weights are [2 - 0.05*6, 2 - 0.05*2] = [1.7, 1.9]. Prediction is 1.7*3 + 1.9 = 7.0, so loss is 0.
4. Not by itself. The absolute difference is about 0.00001, or roughly 0.00017% of the derivative. Compare it with your chosen absolute and relative tolerances; matching signs alone would not be enough. The NumPy check above accepts this difference with its default relative tolerance.
5. Inspect the learning rate and the scale or validity of input values. A huge step or non-finite data can make the forward loss invalid before a later optimizer can help.

Keep evidence from each update

When a training step misbehaves, leave a trail you can inspect:

Before trusting an update, verify four things: the forward loss is finite, the gradient has the expected sign and shape, a centered finite difference agrees within tolerance, and one small step lowers loss on the checked example.

If those checks pass but training diverges, inspect the learning rate and input scale before rewriting backpropagation. Practice the sequence on the latency table: compute one finite-difference check, then confirm that the step lowers loss.

Momentum, Adam, and learning-rate schedules control how to use gradients over many steps. Those tools wait until vectors, matrices, and deeper linear algebra are in place. Next you need names for the lists you just computed.