Training & Backpropagation

The CNN lesson computed feature maps while its kernel weights stayed frozen. Now we let the weights change. Start with a concrete model: it predicts 2 milliseconds of extra time to generate an LLM token when the server actually measures 6. How much should its weight and bias change after that mistake?

The correction has four linked jobs: compute the prediction, turn its miss into one scalar loss, send that loss slope back through the computation graph, and move parameters by a chosen step size. You already computed a slope and a downhill step for a latency model, then saw optimizer rules that reuse those slopes. Here, one running example lets you watch that arithmetic turn into a production training loop, then follow the same error through dense layers, ReLU activations, max-pool winners, and deep multi-layer stacks.

We'll use synthetic latency measurements so every single calculation is verifiable by hand. The input is a prompt-complexity score x; latency values are numerical measurements in milliseconds, and half-squared loss has units of milliseconds squared. These values teach optimization from first principles, letting you inspect every intermediate tensor and gradient buffer.

One update has four distinct jobs

When a model predicts poorly, four questions keep the parameter update interpretable: what did the current parameters predict, how wrong was that prediction, which parameters contributed to the error, and how far should they move?

Suppose a request has prompt-complexity score x = 2.0 and later measures y = 6.0 extra milliseconds of decode latency. We start with a linear latency model:

$$\hat{y} = wx + b$$

Here $\hat{y}$ ("y hat") is the predicted delay, $w$ is the weight, and $b$ is the bias. Set initial parameters w = 1.0 and b = 0.0:

$$\hat{y} = (1.0)(2.0) + 0.0 = 2.0\text{ ms}$$

The prediction is 4.0 ms below the true target of 6.0 ms. Because the prediction is too low and x is positive, increasing either parameter will push the prediction closer to the target. Score the error with half squared error: square the -4.0 ms residual and multiply by 1/2 to get 8.0 square milliseconds. That extra 1/2 doesn't change the optimal $(w, b)$; it cancels the exponent when we differentiate:

$$L = \frac{1}{2}(\hat{y} - y)^2$$

The initial loss is:

$$L = \frac{1}{2}(2.0 - 6.0)^2 = \frac{1}{2}(-4.0)^2 = 8.0$$

What does a negative slope mean here? Nudge the prediction slightly upward, from 2.0 to 2.01 ms. Loss becomes 0.5 * (2.01 - 6.0)**2 = 7.96005, dropping by about 0.04 for a 0.01 ms nudge. The ratio of change is about -4.0: near this prediction point, each extra millisecond reduces loss by four square milliseconds.

Now compare how the weight and the bias affect that prediction. A 0.01 increase in bias moves the prediction by 0.01 ms. The same numerical increase in weight moves the prediction by x * 0.01 = 0.02 ms. The weight gradient, or loss sensitivity, is exactly twice the bias gradient:

$$\frac{\partial L}{\partial \hat{y}} = \hat{y} - y = -4.0, \qquad \frac{\partial L}{\partial w} = (\hat{y} - y)x = (-4.0)(2.0) = -8.0, \qquad \frac{\partial L}{\partial b} = \hat{y} - y = -4.0$$

The gradient is a local slope, not the step itself. Gradient descent subtracts a scaled gradient vector from the current parameters. When we estimate the full-dataset gradient from one randomly sampled example or mini-batch, the method is called stochastic gradient descent (SGD). With learning rate $\eta = 0.1$, the parameter updates are:

$$w_{\text{new}} = 1.0 - 0.1(-8.0) = 1.8, \qquad b_{\text{new}} = 0.0 - 0.1(-4.0) = 0.4$$

The revised prediction is 1.8 * 2.0 + 0.4 = 4.0 ms, and the loss falls from 8.0 to 0.5 * (4.0 - 6.0)**2 = 2.0. The two-millisecond prediction boost combines the weight change (0.8 * 2.0 = 1.6) and the bias change (0.4). We're updating internal parameters, not subtracting 0.1 * (-4.0) directly from the output prediction. Run this complete calculation in Python:

One update cuts our error in half. If we run the update rule repeatedly, the prediction continues approaching 6.0 and the loss continues shrinking toward zero:

The error shrinks each step because each new prediction changes the gradient slope. The learning rate stayed fixed; the error signal being scaled became progressively smaller. That makes the choice of learning rate our next critical question.

Step size can rescue or wreck learning

The gradient points uphill toward increasing loss; its negative points locally downhill. The learning rate scales that downhill step. A tiny rate crawls. A balanced rate settles cleanly. An aggressive rate jumps past the minimum and causes the loss to explode.

We can analyze this mathematically for our single-example model. Updating both parameters changes the prediction by:

$$\Delta \hat{y} = \Delta w \cdot x + \Delta b = -\eta e x^2 - \eta e = -\eta e (x^2 + 1)$$

where $e = \hat{y} - y$ is the prediction residual. The new residual $e_{\text{next}}$ becomes:

$$e_{\text{next}} = e + \Delta \hat{y} = e \left(1 - \eta(x^2 + 1)\right)$$

For $x = 2.0$, $x^2 + 1 = 5.0$, so $e_{\text{next}} = (1 - 5\eta)e$. The contraction factor $1 - 5\eta$ dictates convergence:

• If $\eta = 0.1$, $1 - 5(0.1) = 0.5$: the residual halves every single step.
• If $\eta = 0.2$, $1 - 5(0.2) = 0.0$: the model lands on the exact target in one step.
• If $\eta = 0.4$, $1 - 5(0.4) = -1.0$: the residual flips sign but maintains its magnitude (-4.0 becomes +4.0), oscillating indefinitely without improving.
• If $\eta = 0.5$, $1 - 5(0.5) = -1.5$: the residual flips sign and expands by $1.5\times$ every iteration, diverging to infinity.

Let's test all three behaviors in code:

A single training point can't pin down unique parameters: every pair satisfying $2w + b = 6$ yields zero loss. We'll introduce batches of varied requests shortly to break that symmetry.

Backpropagation assigns credit through a graph

One weight and one bias are easy to differentiate by hand. Real models can contain billions of parameters, so perturbing each parameter separately to compute numerical slopes would require at least one additional forward evaluation per parameter. That's intractable. Backpropagation is reverse-mode automatic differentiation: run the forward pass once to record intermediate values, then push derivatives backward from one scalar loss to every parameter in a single sweep.^{[1]Reference 1Automatic Differentiation in Machine Learning: a Surveyhttps://www.jmlr.org/papers/v18/17-468.html[2]Reference 2Autograd mechanicshttps://docs.pytorch.org/docs/stable/notes/autograd.html}

With one scalar loss and millions of parameters, reverse mode shares intermediate activations across all parameter paths. Its compute cost scales with the operations in the forward graph, avoiding a separate forward pass per parameter.^{[1]Reference 1Automatic Differentiation in Machine Learning: a Surveyhttps://www.jmlr.org/papers/v18/17-468.html} Rumelhart, Hinton, and Williams showed how error-driven reverse propagation trains multi-layer representations that would otherwise be impossible to coordinate.^{[3]Reference 3Learning Representations by Back-Propagating Errors.https://doi.org/10.1038/323533a0[4]Reference 4Deep Learning.https://www.deeplearningbook.org/}

Before tracking backward derivatives, inspect what the forward pass stores: the intermediate product m = w * x, the prediction $\hat{y}$, and the residual $e = \hat{y} - y$. The decode-latency graph lets us inspect each node directly:

Start at the loss. The seed derivative at the root is $\partial L/\partial L = 1.0$. The chain rule multiplies that seed by each local derivative along the path to every parameter:

$$\frac{\partial L}{\partial w} = \frac{\partial L}{\partial e} \frac{\partial e}{\partial \hat{y}} \frac{\partial \hat{y}}{\partial m} \frac{\partial m}{\partial w} = (e)(1)(1)(x) = (-4.0)(1)(1)(2.0) = -8.0$$ $$\frac{\partial L}{\partial b} = \frac{\partial L}{\partial e} \frac{\partial e}{\partial \hat{y}} \frac{\partial \hat{y}}{\partial b} = (e)(1)(1) = (-4.0)(1)(1) = -4.0$$

Whenever you write a custom backward pass, test your analytical derivatives against centered finite differences. Numerical differentiation is too slow for actual training, but it's the standard tool to verify analytical gradients before deploying them:^{[5]Reference 5Gradcheck mechanicshttps://docs.pytorch.org/docs/stable/notes/gradcheck.html}

$$\frac{\partial L}{\partial \theta} \approx \frac{L(\theta + \epsilon) - L(\theta - \epsilon)}{2\epsilon}$$

Finite differences require numerical discipline: a perturbation that's too large samples surface curvature rather than the tangent slope, while one that's too tiny loses precision to floating-point roundoff. Use double precision (float64) for gradient checks and avoid points where derivatives are discontinuous (like ReLU at zero).^{[6]Reference 6torch.autograd.gradcheck.gradcheckhttps://docs.pytorch.org/docs/main/generated/torch.autograd.gradcheck.gradcheck.html}

Build a scalar autograd engine

Andrej Karpathy's Micrograd captures autograd bookkeeping in a compact executable engine: each object stores a scalar value, its accumulated gradient, its parent references, and a local backward closure.^{[7]Reference 7micrograd.https://github.com/karpathy/micrograd} Tensor libraries apply this exact pattern across high-dimensional arrays. Building a scalar engine keeps the mechanics transparent.

For our model $\hat{y} = wx + b$ and half-squared loss $L = 0.5(\hat{y} - y)^2$, every operation implements one local derivative rule:

Two structural rules govern the engine:

1. Topological ordering: The backward sweep must visit an output node before visiting any of its parent inputs. We build a directed graph from the loss root, perform a depth-first search, and reverse the topological order.
2. Gradient accumulation: Parent gradients must use +=, not =. When a variable feeds multiple downstream operations, its total gradient is the sum of derivatives along all paths (multivariable chain rule).

The autograd engine matches our hand calculations exactly. Now consider a shared variable: in $s = a \cdot a$ with $a = 3.0$, what is $ds/da$? Mathematically, $d(a^2)/da = 2a = 6.0$. Because $a$ feeds both multiplication inputs, the backward closure fires for both operands. An engine that assigns with = would overwrite the first contribution with 3.0. Using += sums both paths to 6.0:

Tensor libraries follow this exact accumulation contract. Parameter .grad buffers persist until explicitly cleared. That behavior enables intentional gradient accumulation across micro-batches, but it means un-cleared buffers will silently corrupt future training steps.

Local backward rules for dense layers, activations, and pooling

Multilayer networks compose simple vector operations. Each layer receives an upstream error gradient $\delta$ from the layer above and computes two items: gradients for its own parameters, and the downstream error signal to pass to the preceding layer.

Dense layer backward

For a dense linear transformation $z = Wx + b$ with incoming gradient $\delta = \partial L/\partial z$:

• Weight gradient is the outer product: $\partial L/\partial W = \delta x^\top$ (shape matches $W$).
• Bias gradient is the upstream signal itself: $\partial L/\partial b = \delta$.
• Input gradient routes signal backward: $\partial L/\partial x = W^\top \delta$ (shape matches $x$).^{[4]Reference 4Deep Learning.https://www.deeplearningbook.org/}

ReLU backward

ReLU is $h = \max(0, z)$.^{[8]Reference 8Rectified Linear Units Improve Restricted Boltzmann Machines.https://www.cs.toronto.edu/~fritz/absps/reluICML.pdf} Its backward rule checks whether the input was active during the forward pass:

$$\frac{\partial L}{\partial z} = \frac{\partial L}{\partial h} \cdot \mathbf{1}_{z > 0}$$

If $z > 0$, the gradient passes through unchanged. If $z \le 0$, the gradient is multiplied by zero. A unit that remains negative across all training examples is a "dead ReLU" and receives zero gradient updates.

Max-pooling backward

Max pooling extracts the maximum value within each spatial window and saves the coordinates of that winner during the forward pass. The backward rule routes the incoming pooled gradient exclusively to the saved winning coordinate. All non-winning locations receive a gradient of zero:

In the CNN pipeline from the previous lesson, these local rules connect in reverse order. The dense head sends its input gradient to max-pooling, max-pooling routes that signal to its saved winners, and ReLU passes it only through positions that were positive during the forward pass.

The five-phase PyTorch training loop state machine

Production training orchestrates five distinct operations per batch: zero gradients, forward pass, loss calculation, backward pass, and optimizer step. Misordering these steps stalls training, accumulates phantom gradients, or causes silent convergence bugs.

Here is the exact state transition contract for each phase:

1. optimizer.zero_grad(set_to_none=True): PyTorch accumulates gradients by default. Before launching a new backward pass, we must reset parameter .grad attributes. Passing set_to_none=True leaves those attributes as None instead of filling existing tensors with zeros. PyTorch documents a generally lower memory footprint and a modest performance benefit, with the important behavior that optimizers skip parameters whose gradients remain None.^{[9]Reference 9Optimizer.zero_gradhttps://docs.pytorch.org/docs/stable/generated/torch.optim.Optimizer.zero_grad.html}
2. Forward pass (predictions = model(inputs)): Autograd operates as a dynamic "define-by-run" tape. As tensors flow through layers, PyTorch records mathematical operations, builds a directed acyclic graph (DAG), and automatically saves the tensors each backward formula needs. Custom autograd functions request those saved tensors explicitly with ctx.save_for_backward; built-in operations manage them internally.^{[2]Reference 2Autograd mechanicshttps://docs.pytorch.org/docs/stable/notes/autograd.html}
3. Loss evaluation (loss = criterion(predictions, targets)): The criterion reduces this batch's prediction errors into one scalar. That scalar tensor forms the root of the autograd DAG, with its grad_fn attribute pointing to the final operation (for example, <MseBackward0>).
4. Backward pass (loss.backward()): Calling .backward() on the root scalar traverses the DAG in reverse topological order. PyTorch executes local vector-Jacobian products, multiplies incoming gradients along all paths, and accumulates partial derivatives directly into leaf parameter .grad buffers. Once backward finishes, the intermediate activation graph is freed by default (retain_graph=False).
5. Optimizer step (optimizer.step()): The optimizer updates registered parameters in place. For the SGD optimizer used here, the rule is $\theta \leftarrow \theta - \eta \cdot \theta.\text{grad}$; adaptive optimizers transform the stored gradient before updating. The optimizer doesn't clear gradients, so that responsibility remains with zero_grad().

Let's test this complete loop on our latency dataset:

What happens if you omit zero_grad()? Each backward pass adds new gradients to existing ones. Running two consecutive backward calls doubles the gradient magnitude and takes an unintentional double-sized step:

Evaluation mode and no_grad

During validation and inference, we don't compute parameter updates. We use two complementary controls:

• model.eval(): Switches module behavior for layers like Dropout (which deactivates mask generation) and BatchNorm (which freezes running population statistics). It doesn't disable autograd tracking.
• with torch.no_grad():: Disables reverse-mode gradient recording within the current thread and block. Results have requires_grad=False, so PyTorch doesn't retain the saved tensors needed only for a later backward pass. That reduces memory use for computations that would otherwise track gradients.^{[2]Reference 2Autograd mechanicshttps://docs.pytorch.org/docs/stable/notes/autograd.html}

Batching dynamics: full batch, mini-batch, and online stochastic

How many examples should contribute to a single parameter update? Optimization operates across three distinct regimes:

Full-batch gradient descent

Full-batch gradient descent evaluates every example in the dataset before taking a step:

$$g_{\text{full}} = \frac{1}{N} \sum_{i=1}^N \nabla_\theta L(x_i, y_i; \theta)$$

For fixed parameters and a fixed dataset, this is the exact empirical-risk gradient. A large dataset usually can't be materialized as one device batch, although the same exact gradient can be accumulated across smaller chunks before one update. The main cost is update frequency: the optimizer waits for all $N$ examples before taking one step.^{[4]Reference 4Deep Learning.https://www.deeplearningbook.org/}

Online stochastic gradient descent (pure SGD)

Pure SGD uses exactly one sample per update ($B = 1$). Each update is cheap, but its gradient is a noisy estimate of the full gradient. Processing samples one by one also tends to underuse parallel matrix hardware, so many cheap updates can still take longer in wall-clock time.

Mini-batch gradient descent

Mini-batch training trades between update frequency, gradient noise, and hardware utilization. Partitioning data into batches of size $B$ converts sample-by-sample work into batched tensor operations, but the best $B$ depends on the model, hardware, optimizer, and validation behavior.

Why does reduction="mean" matter? If examples are sampled uniformly, dividing by batch size $B$ makes the mini-batch gradient an unbiased estimate of the full-dataset gradient:

$$g_{\text{batch}} = \frac{1}{B} \sum_{i=1}^B \nabla_\theta L(x_i, y_i; \theta) \implies \mathbb{E}[g_{\text{batch}}] = g_{\text{full}}$$

Averaging prevents the gradient from doubling merely because the batch contains twice as many examples. Changing $B$ still changes sampling noise and the number of optimizer updates per epoch, so the learning rate usually needs fresh validation. Linear and square-root scaling recipes work only under additional optimizer and workload assumptions; neither is a universal batch-size rule.

Let's compare the three batching regimes on a 6-sample dataset:

Each run processes 60 example presentations, but the update counts differ: 10 full-batch updates, 30 mini-batch updates, and 60 online updates. The smaller batches happen to reach lower training loss faster on this ordered, noiseless line. That isn't a universal quality ranking. Real training chooses a batch size by measuring validation behavior and accelerator throughput, not by assuming the noisiest gradient wins.

Vanishing and exploding gradients in deep networks

In a multi-layer feedforward network with $L$ layers:

$$h_l = \sigma(W_l h_{l-1} + b_l), \quad l = 1, \dots, L$$

Write $g_l = \partial L/\partial h_l$ for the gradient arriving at layer $l$. One reverse step is:

$$g_{l-1} = W_l^\top\left(g_l \odot \sigma'(z_l)\right)$$

Here $\odot$ is element-wise multiplication, and $\sigma'(z_l)$ is the activation derivative evaluated at the saved preactivation. Repeating this rule from layer $L$ down to layer $1$ multiplies many layer Jacobians, exposing the core instability of deep backpropagation:

1. Vanishing gradients: If the repeated layer Jacobians contract the directions carrying the error signal (for example, because of saturating activations or undersized weights), the gradient product can decay exponentially: $\left\|\frac{\partial L}{\partial h_1}\right\| \propto \gamma^{L-1} \to 0 \quad (\gamma < 1)$ The earliest layers receive near-zero gradients. Their weights freeze, and depth provides no representational benefit over a shallow model.
2. Exploding gradients: If the repeated Jacobians expand those directions, the product can grow exponentially: $\left\|\frac{\partial L}{\partial h_1}\right\| \propto \gamma^{L-1} \to \infty \quad (\gamma > 1)$ Gradients explode into thousands or millions. Parameter updates fling weights far across the loss surface, causing arithmetic overflow (NaN or Inf).

Gradient norm clipping

Pascanu, Mikolov, and Bengio proposed gradient norm clipping to stabilize training when gradients explode.^{[10]Reference 10On the Difficulty of Training Recurrent Neural Networkshttps://proceedings.mlr.press/v28/pascanu13.html} Unlike value clipping (clip_grad_value_), which truncates individual gradient coordinates independently and changes the update direction, norm clipping rescales the entire gradient vector when its Euclidean norm exceeds a threshold $C$:

$$\mathbf{g} \leftarrow \mathbf{g} \cdot \min\left(1, \frac{C}{\|\mathbf{g}\|_2 + \epsilon}\right)$$

If $\|\mathbf{g}\|_2 \le C$, the gradient stays untouched. If $\|\mathbf{g}\|_2 > C$, the vector length is clamped to $C$ while preserving its search direction. In PyTorch, call clip_grad_norm_ after backward() and before optimizer.step():

Even though deep linear multiplication produced an unclipped gradient norm of $36,635.74$, norm clipping reduced the total gradient magnitude to $5.00$ while preserving its direction. The learning rate still controls how far the parameters move after that bound is applied.

Weight initialization: preserving variance across depth

Why can't we initialize weights to zeros or standard normal random numbers?

• Identical initialization: If every neuron in a layer starts with the same weights and bias, each computes the same activation and receives the same gradient. The neurons remain identical across training steps, preventing them from learning distinct features. Setting every weight and bias to zero is the clearest example of this symmetry failure.
• Unscaled $\mathcal{N}(0, 1)$ initialization: Consider a linear transformation $z = \sum_{i=1}^{n_{\text{in}}} w_i x_i$. If inputs $x_i$ and weights $w_i$ are independent with zero mean: $\operatorname{Var}(z) = \sum_{i=1}^{n_{\text{in}}} \operatorname{Var}(w_i x_i) = n_{\text{in}} \operatorname{Var}(w) \operatorname{Var}(x)$ If $\operatorname{Var}(w) = 1$ and $n_{\text{in}} = 256$, the activation variance expands by $256\times$ per layer! Within three layers, activations blow up to extreme magnitudes.

To keep $\operatorname{Var}(z) = \operatorname{Var}(x)$, we must scale weight variance inversely with the number of input connections (fan-in):

$$\operatorname{Var}(w) = \frac{1}{n_{\text{in}}}$$

Xavier / Glorot initialization

For symmetric activations that are approximately linear near zero (such as $\tanh$ and $\text{sigmoid}$), Glorot and Bengio balanced the forward pass variance ($\operatorname{Var}(w) = 1/n_{\text{in}}$) with the backward pass gradient variance ($\operatorname{Var}(w) = 1/n_{\text{out}}$) by using their harmonic mean:^{[4]Reference 4Deep Learning.https://www.deeplearningbook.org/}

$$\operatorname{Var}(w) = \frac{2}{n_{\text{in}} + n_{\text{out}}}$$

For a normal distribution, use standard deviation $\sqrt{2/(n_{\text{in}} + n_{\text{out}})}$. For a uniform distribution, sample from $\mathcal{U}\left(-\sqrt{\frac{6}{n_{\text{in}} + n_{\text{out}}}}, \sqrt{\frac{6}{n_{\text{in}} + n_{\text{out}}}}\right)$.

He / Kaiming initialization

For a symmetric, zero-mean preactivation $z$, ReLU zeroes the negative half of the distribution. Its second moment becomes:

$$\mathbb{E}[\operatorname{ReLU}(z)^2] = \int_0^\infty z^2 p(z)\, dz = \frac{1}{2}\mathbb{E}[z^2]$$

The equation concerns expected squared activation, not variance: ReLU outputs have a positive mean, so those quantities differ. Under the initialization assumptions, compensating for the halved second moment gives He/Kaiming's fan-in rule:^{[4]Reference 4Deep Learning.https://www.deeplearningbook.org/}

$$\operatorname{Var}(w) = \frac{2}{n_{\text{in}}}$$

Let's test activation standard deviations across a 10-layer network in PyTorch:

With unscaled weights, activation standard deviation exploded to $2.53 \times 10^{10}$ within 10 layers. With undersized weights, it collapsed to $3.41 \times 10^{-10}$. Kaiming initialization finished at $0.78$, which remains on the input's order of magnitude rather than guaranteeing an exact value of 1.0 in a finite random network.

Train, validation, and test splits with early stopping

A model that memorizes training data but fails on fresh queries is useless in production. Rigorous evaluation partitions data into three disjoint sets:

1. Training set (typically 70-80%): Used by the optimizer to compute gradients and update weights.
2. Validation set (typically 10-15%): Held out from backpropagation. Evaluated at regular checkpoint intervals to measure generalization, tune hyperparameters (learning rate, weight decay), and trigger early stopping.
3. Test set (typically 10-15%): Locked in a vault until model development concludes. Evaluated exactly once to report unbiased generalization performance before shipping.

Underfitting vs overfitting

Comparing loss curves between training and validation reveals model capacity dynamics:

• Underfitting (high bias): Both training loss and validation loss remain high and plateau early. The model lacks the representational capacity or training time to capture the underlying pattern. Remedy: increase network width/depth, add relevant features, or reduce regularization.
• Overfitting (high variance): Training loss continues descending toward zero while validation loss plateaus and starts climbing upward. The model is memorizing noise specific to the training set. Remedy: early stopping, dropout, weight decay, or acquiring more diverse training data.

Early stopping with checkpointing

Early stopping monitors validation loss after every epoch. If validation loss fails to improve by a threshold min_delta for patience consecutive epochs, training terminates and restores the weights from the lowest-loss checkpoint:

At epoch 4, validation loss reached its minimum of 1.80. Over epochs 5, 6, and 7, validation loss deteriorated while training loss continued falling. The early stopping monitor halted training after patience = 3 worsening evaluations and rolled back parameters to the checkpoint from epoch 4.

Why the next loss function matters

Half squared error works well for continuous regression targets like latency in milliseconds. But classification problems, like classifying an issue into bug, docs, or security, and language models predicting the next token from a large vocabulary have a different output contract: the model outputs unnormalized real-valued logits.

The underlying backpropagation engine remains unchanged. Only the output head and loss function adapt: softmax normalizes raw logits into a probability distribution, cross-entropy penalizes probability assigned to incorrect classes, and their combined gradient $\hat{p} - y$ initiates the reverse sweep. Raw logits carry one more risk this lesson doesn't solve: naively exponentiating a large score can overflow floating point to inf, then produce NaN during normalization. The next lesson derives the max-subtraction fix that leaves the mathematical softmax unchanged while keeping every exponential at or below 1.0.

Diagnose a broken update

When training behaves erratically, reduce your run to a single reproducible batch and inspect tensor states:

Checkpoints & practice

Keep the original residual -4.0 but change the input to x = -2.0 by starting at w = -1.0, b = 0.0. With learning rate 0.1, which parameter moves down, and does the prediction still rise?

Answer

dL/dw = (-4.0) * (-2.0) = 8.0, while dL/db = -4.0. The updated parameters are w = -1.0 - 0.1(8.0) = -1.8 and b = 0.0 - 0.1(-4.0) = 0.4. The prediction becomes (-1.8)(-2.0) + 0.4 = 4.0. The weight decreases, but because x is negative, a lower weight produces a higher prediction. The input sign dictates the parameter update direction.

At x = 2.0, what happens to the residual if the learning rate is exactly 0.2? What happens if it's 0.4?

Answer

The residual update factor is 1 - 5*eta. When eta = 0.2, 1 - 5(0.2) = 0.0, zeroing the residual in a single step. When eta = 0.4, 1 - 5(0.4) = -1.0: the residual flips sign (-4 becomes +4) but keeps its exact magnitude, so the loss never improves.

Why does PyTorch default to accumulating gradients across backward() calls instead of overwriting them?

Answer

Gradient accumulation lets you simulate large mini-batch sizes that exceed GPU memory capacity. You can execute multiple forward and backward passes across sequential micro-batches, accumulating gradients into param.grad, and only call optimizer.step() once. That's why you must explicitly call zero_grad() when starting a fresh batch.

Why does gradient norm clipping preserve the update direction while coordinate-wise gradient value clipping can change it?

Answer

Value clipping clamps every individual coordinate to [-C, C], altering the angle and direction of the gradient vector. Norm clipping scales the entire vector by C / ||g||, preserving the exact search direction while capping the overall update magnitude.