LearnPreparation & PrerequisitesSoftmax, Cross-Entropy & Optimization

📝EasyNLP Fundamentals

Softmax, Cross-Entropy & Optimization

Turn raw class scores into stable probabilities and a useful learning signal, then apply the same loss to next-token predictions.

13 min read

Learning path

Step 21 of 177 in the full curriculum

Training & Backpropagation RNNs, LSTMs, GRUs, and Sequence Modeling

Personalize this lesson

Adapt explanations and teaching visuals to your background and preferred voice.

In the previous lesson, a model predicted one number: next-token latency in milliseconds. Its loss could say "increase the prediction." A code-review classifier now faces a different job. Given a short review note, should it label the issue bug, docs, or security?

Suppose the evidence says docs is correct, but the model currently favors bug. We need a score for how bad that choice is, and a gradient that says which competing scores to raise or lower. Softmax turns raw scores into probability shares. Cross-entropy measures how little probability reached the correct choice. Together they form the standard categorical output-and-loss pair taught for neural classifiers and language models.^{[1]Reference 1Deep Learning.https://www.deeplearningbook.org/}

A decision begins as three raw scores

The final layer of a classifier emits one number per possible label. These numbers are logits: unconstrained scores, not probabilities.

Label	Current logit	What the review note says
`bug`	`3.0`	Model's current favorite, but wrong
`docs`	`1.0`	Correct supervised label
`security`	`0.0`	Possible, but not correct here

A larger logit means the model prefers a label relative to its competitors. It doesn't mean 3.0 is 300%, and a logit can be negative without creating a negative probability.

This first snippet makes the mistake visible: the scores select a favorite, but don't satisfy probability rules.

inspect-raw-logits.py

labels = ["bug", "docs", "security"]
logits = [3.0, 1.0, 0.0]

best_index = max(range(len(logits)), key=logits.__getitem__)
print("largest logit:", labels[best_index])
print("raw score sum:", sum(logits))
print("valid probability distribution:", all(0 <= z <= 1 for z in logits) and sum(logits) == 1)

Output

largest logit: bug
raw score sum: 4.0
valid probability distribution: False

Two bar charts show logits 3, 1, and 0 becoming exponential weights 20.09, 2.72, and 1.00, then a stacked probability bar splits one unit into 84.4%, 11.4%, and 4.2%. — Exponentiation expands the score gaps into positive weights. Dividing every weight by the same `23.81` total preserves their proportions and packs them into one unit of probability mass.

Softmax assigns probability shares

Softmax takes every logit $z_i$ , exponentiates it, and divides by the sum of all exponentiated logits:

p_i = \frac{e^{z_i}}{\sum_{j=1}^{K} e^{z_j}}

Here $p_i$ is the probability assigned to label $i$ , $z_i$ is its raw logit, and $K$ is the number of choices. Exponentiation makes each share positive. Dividing by their total makes the shares sum to one.

For our three class scores, the arithmetic is small enough to do by hand:

Action	Logit $z_i$	$e^{z_i}$	Probability $p_i$
`bug`	`3.0`	`20.09`	`0.844`
`docs`	`1.0`	`2.72`	`0.114`
`security`	`0.0`	`1.00`	`0.042`
Total		`23.81`	`1.000`

The model places only 0.114 probability on the correct docs label. That's the quantity the loss must expose.

Use NumPy to reproduce the table. This version is already stable because it subtracts the largest logit before taking exponentials.

stable-softmax.py

import numpy as np

def stable_softmax(logits: np.ndarray) -> np.ndarray:
    shifted = logits - np.max(logits)
    exp_shifted = np.exp(shifted)
    return exp_shifted / exp_shifted.sum()

labels = ["bug", "docs", "security"]
logits = np.array([3.0, 1.0, 0.0])
probabilities = stable_softmax(logits)

for label, probability in zip(labels, probabilities):
    print(f"{label:8s} {probability:.3f}")
print("sum     ", round(float(probabilities.sum()), 3))

Output

bug      0.844
docs     0.114
security 0.042
sum      1.0

Stable arithmetic matters before scale does

Softmax depends on differences between logits, not on their absolute offset. Adding or subtracting the same constant from every score leaves every probability unchanged:

\frac{e^{z_i-c}}{\sum_j e^{z_j-c}} = \frac{e^{z_i}e^{-c}}{e^{-c}\sum_j e^{z_j}} = \frac{e^{z_i}}{\sum_j e^{z_j}}

That cancellation gives us a safety rule: subtract the largest logit before exponentiating. The largest exponent becomes $e^0 = 1$ , while the relative differences stay intact.

This failure case uses the same score gaps shifted upward by 997. Direct exponentiation overflows; max-shifting produces the original distribution.

overflow-then-stable.py

import numpy as np

def stable_softmax(logits: np.ndarray) -> np.ndarray:
    shifted = logits - np.max(logits)
    exp_shifted = np.exp(shifted)
    return exp_shifted / exp_shifted.sum()

large_logits = np.array([1000.0, 998.0, 997.0])
with np.errstate(over="ignore", invalid="ignore"):
    naive = np.exp(large_logits) / np.exp(large_logits).sum()

stable = stable_softmax(large_logits)
print("naive finite:", np.isfinite(naive).all())
print("stable:", np.round(stable, 3))
print("stable sum:", round(float(stable.sum()), 3))

Output

naive finite: False
stable: [0.844 0.114 0.042]
stable sum: 1.0

Cross-entropy grades the correct label

The supervised label for this example is docs. Written as a one-hot target vector in the same label order, it's:

y = [0,\ 1,\ 0]

One-hot means that exactly one class receives target weight one. Cross-entropy compares this target with the predicted probability vector:

L = -\sum_i y_i \log(p_i)

Every term except docs is multiplied by zero, so the loss reduces to:

L = -\log(p_{\text{docs}}) = -\log(0.114) \approx 2.170

Had the label been the model's preferred bug, the loss would have been only $-\log(0.844) \approx 0.170$ . The loss is high because the supervised answer received little probability, not because the top choice merely happened to be wrong.

For numerical stability, compute that same cross-entropy straight from logits instead of first materializing a tiny probability. For a correct class index $y$ :

L = \log\left(\sum_j e^{z_j}\right) - z_y

The first term is called log-sum-exp. Its stable implementation uses the same maximum $m = \max_j z_j$ introduced for stable softmax:

\log\left(\sum_j e^{z_j}\right) = m + \log\left(\sum_j e^{z_j-m}\right)

This code computes the loss for the correct docs label directly from logits. It works for ordinary scores and extremely large offsets.

cross-entropy-from-logits.py

import numpy as np

def cross_entropy_from_logits(logits: np.ndarray, target_index: int) -> float:
    maximum = np.max(logits)
    logsumexp = maximum + np.log(np.exp(logits - maximum).sum())
    return float(logsumexp - logits[target_index])

ordinary = np.array([3.0, 1.0, 0.0])
shifted_high = ordinary + 1000.0

print("ordinary docs loss:", round(cross_entropy_from_logits(ordinary, 1), 3))
print("large-offset loss:   ", round(cross_entropy_from_logits(shifted_high, 1), 3))

Output

ordinary docs loss: 2.17
large-offset loss:    2.17

For a shorter arithmetic exercise, consider a second example with logits [1.0, 0.0, -2.0] in the same label order. bug still leads, but by a smaller margin. Compare the loss under two possible labels.

Now compute it:

compare-label-losses.py

import numpy as np

def stable_softmax(logits):
    shifted = logits - np.max(logits)
    exp_logits = np.exp(shifted)
    return exp_logits / exp_logits.sum()

logits = np.array([1.0, 0.0, -2.0])  # bug, docs, security
probabilities = stable_softmax(logits)

print("docs loss", round(float(-np.log(probabilities[1])), 3))
print("bug loss", round(float(-np.log(probabilities[0])), 3))

Output

docs loss 1.349
bug loss 0.349

Why squared probability error is a weak fit here

The earlier latency lesson used half squared error for a continuous target in milliseconds. We could attach softmax to a classifier and square its probability error, but the softmax derivative then shrinks the signal when a wrong class is already saturated near probability one. Cross-entropy paired with softmax avoids that extra shrinkage: its logit gradient will be $p-y$ .^{[1]Reference 1Deep Learning.https://www.deeplearningbook.org/}

This example makes the difference visible. The logits are extremely sure about the wrong bug label while the target is docs.

confidently-wrong-gradients.py

import numpy as np

def softmax(logits: np.ndarray) -> np.ndarray:
    shifted = logits - np.max(logits)
    exps = np.exp(shifted)
    return exps / exps.sum()

def squared_probability_loss(logits: np.ndarray, target: np.ndarray) -> float:
    error = softmax(logits) - target
    return float(0.5 * np.sum(error ** 2))

logits = np.array([8.0, 0.0, 0.0])
target = np.array([0.0, 1.0, 0.0])
probabilities = softmax(logits)
ce_gradient = probabilities - target

epsilon = 1e-5
mse_gradient = np.array([
    (squared_probability_loss(logits + np.eye(3)[i] * epsilon, target)
     - squared_probability_loss(logits - np.eye(3)[i] * epsilon, target))
    / (2 * epsilon)
    for i in range(3)
])

print("probabilities:", np.round(probabilities, 4))
print("cross-entropy gradient:", np.round(ce_gradient, 4))
print("squared-probability gradient:", np.round(mse_gradient, 4))

Output

probabilities: [9.993e-01 3.000e-04 3.000e-04]
cross-entropy gradient: [ 9.993e-01 -9.997e-01  3.000e-04]
squared-probability gradient: [ 0.001  -0.0007 -0.0003]

Cross-entropy isn't the only possible classification objective, but it gives a direct categorical likelihood objective and a useful correction even when a wrong option dominates.

The gradient says exactly what to correct

For one labeled example, combine stable softmax with cross-entropy:

L = \log\left(\sum_j e^{z_j}\right) - z_y

Differentiate with respect to any logit $z_k$ :

\frac{\partial L}{\partial z_k} = \frac{e^{z_k}}{\sum_j e^{z_j}} - \mathbf{1}[k=y] = p_k - y_k

This derivative is commonly summarized component-wise as $p_i - y_i$ : predicted probability share minus target share.

The indicator $\mathbf{1}[k=y]$ equals one only for the correct class. For our docs example:

Label	Probability $p$	Target $y$	Gradient $p-y$	Gradient descent effect
`bug`	`0.844`	`0`	`+0.844`	Lowers wrong favorite
`docs`	`0.114`	`1`	`-0.886`	Raises correct label
`security`	`0.042`	`0`	`+0.042`	Lowers competitor slightly

Three probability number lines show p minus y as an arrow from one-hot target to prediction: bug plus 0.844, docs minus 0.886, and security plus 0.042. An action column shows that gradient descent subtracts this signal, lowering bug and security logits while raising the correct docs logit. — Arrow direction shows the signed gradient `p − y`; SGD moves in the opposite direction by subtracting it. That lowers the high wrong `bug` logit, raises the low correct `docs` logit, and gives `security` only a small downward correction.

Logits themselves are outputs, not normally the parameters an optimizer stores. Updating them below is a diagnostic shortcut: it shows the direction that upstream weight updates are trying to produce on the next forward pass.

one-logit-signal-step.py

import numpy as np

def probabilities_and_loss(logits: np.ndarray, target_index: int):
    shifted = logits - np.max(logits)
    log_probs = shifted - np.log(np.exp(shifted).sum())
    return np.exp(log_probs), float(-log_probs[target_index])

labels = ["bug", "docs", "security"]
logits = np.array([3.0, 1.0, 0.0])
target_index = labels.index("docs")
target = np.eye(3)[target_index]

before, before_loss = probabilities_and_loss(logits, target_index)
gradient = before - target
after_logits = logits - 0.5 * gradient
after, after_loss = probabilities_and_loss(after_logits, target_index)

print("gradient:", np.round(gradient, 3))
print("docs probability:", round(float(before[1]), 3), "->", round(float(after[1]), 3))
print("loss:", round(before_loss, 3), "->", round(after_loss, 3))

Output

gradient: [ 0.844 -0.886  0.042]
docs probability: 0.114 -> 0.23
loss: 2.17 -> 1.469

As in the previous lesson, a finite-difference test catches sign mistakes in a new loss implementation. Nudging each logit by a tiny amount should agree with $p-y$ .

verify-cross-entropy-gradient.py

import numpy as np

def loss(logits: np.ndarray, target_index: int) -> float:
    shifted = logits - np.max(logits)
    return float(np.log(np.exp(shifted).sum()) - shifted[target_index])

logits = np.array([3.0, 1.0, 0.0])
target_index = 1
shifted = logits - np.max(logits)
probabilities = np.exp(shifted) / np.exp(shifted).sum()
analytic = probabilities - np.eye(3)[target_index]

epsilon = 1e-5
numeric = np.array([
    (loss(logits + np.eye(3)[i] * epsilon, target_index)
     - loss(logits - np.eye(3)[i] * epsilon, target_index))
    / (2 * epsilon)
    for i in range(3)
])

print("analytic:", np.round(analytic, 6))
print("numeric: ", np.round(numeric, 6))
print("match:", np.allclose(analytic, numeric, atol=1e-6))

Output

analytic: [ 0.843795 -0.885805  0.04201 ]
numeric:  [ 0.843795 -0.885805  0.04201 ]
match: True

PyTorch receives logits directly

Training code shouldn't implement this loss from scratch unless you're testing your understanding or building a custom variant. In PyTorch, nn.CrossEntropyLoss receives raw logits for class-index targets and computes the equivalent of log-softmax followed by negative log-likelihood internally. Passing already-softmaxed probabilities changes the function being optimized.^{[2]Reference 2Optimizing Model Parameters.https://docs.pytorch.org/tutorials/beginner/basics/optimization_tutorial.html}

First, confirm that PyTorch returns the same loss and $p-y$ gradient as our hand calculation:

pytorch-cross-entropy.py

import torch
from torch import nn

logits = torch.tensor([[3.0, 1.0, 0.0]], requires_grad=True)
target = torch.tensor([1])  # docs
loss_fn = nn.CrossEntropyLoss()

loss = loss_fn(logits, target)
loss.backward()
probabilities = torch.softmax(logits.detach(), dim=1)

print("probabilities:", probabilities.numpy().round(3))
print("loss:", round(loss.item(), 3))
print("gradient:", logits.grad.numpy().round(3))

Output

probabilities: [[0.844 0.114 0.042]]
loss: 2.17
gradient: [[ 0.844 -0.886  0.042]]

Now reproduce a common bug. The incorrect call passes probabilities where the loss expects logits, effectively normalizing an already normalized output again.

do-not-softmax-before-cross-entropy.py

import torch
from torch import nn

target = torch.tensor([1])  # docs
loss_fn = nn.CrossEntropyLoss()

correct_input = torch.tensor([[3.0, 1.0, 0.0]], requires_grad=True)
correct_loss = loss_fn(correct_input, target)
correct_loss.backward()

wrong_input = torch.tensor([[3.0, 1.0, 0.0]], requires_grad=True)
wrong_loss = loss_fn(torch.softmax(wrong_input, dim=1), target)
wrong_loss.backward()

print("raw logits loss:", round(correct_loss.item(), 3))
print("probabilities passed as logits:", round(wrong_loss.item(), 3))
print("correct docs gradient:", round(correct_input.grad[0, 1].item(), 3))
print("distorted docs gradient:", round(wrong_input.grad[0, 1].item(), 3))

Output

raw logits loss: 2.17
probabilities passed as logits: 1.387
correct docs gradient: -0.886
distorted docs gradient: -0.127

Shapes won't warn you about this bug. Keep the model's final classification layer linear during training and feed those raw logits into cross-entropy.

Temperature changes a distribution, not a label

Softmax can be made sharper or flatter by dividing logits by a positive temperature $T$ :

p_i(T) = \frac{e^{z_i/T}}{\sum_j e^{z_j/T}}

When $T$ is below one, logit differences grow and the leading option takes more probability. When $T$ is above one, differences shrink and more probability remains on alternatives. Hinton, Vinyals, and Dean use this operation to reveal softer class relationships during knowledge distillation.^{[3]Reference 3Distilling the Knowledge in a Neural Network.https://arxiv.org/abs/1503.02531} In text generation, the same mathematical scaling can shape the distribution a decoding policy samples from. Softmax still doesn't choose a token by itself.

Temperature	`bug`	`docs`	`security`	Interpretation
`0.5`	`0.980`	`0.018`	`0.002`	Wrong favorite becomes harder to escape
`1.0`	`0.844`	`0.114`	`0.042`	Original model distribution
`2.0`	`0.629`	`0.231`	`0.140`	Alternatives receive more mass

Use the same stable function to check each distribution:

temperature-scales-logits.py

import numpy as np

def softmax_at_temperature(logits: np.ndarray, temperature: float) -> np.ndarray:
    scaled = logits / temperature
    shifted = scaled - np.max(scaled)
    exponentials = np.exp(shifted)
    return exponentials / exponentials.sum()

logits = np.array([3.0, 1.0, 0.0])
for temperature in (0.5, 1.0, 2.0):
    probabilities = softmax_at_temperature(logits, temperature)
    print(f"T={temperature:.1f}", np.round(probabilities, 3))

Output

T=0.5 [0.98  0.018 0.002]
T=1.0 [0.844 0.114 0.042]
T=2.0 [0.629 0.231 0.14 ]

A sequence supplies many supervised positions

Our classifier example produced one loss. A language model trains on the same categorical idea many times at once: after each context position, it produces logits for the next token, and the observed next token supplies the class label. A sequence model still needs a mechanism for history, which is the question the next lesson will address.

To make the language-model version literal, use a tiny vocabulary: bug, docs, and today. Suppose a two-token reply should be docs today. At each supervised position, the model produces a new logit vector over that same vocabulary:

Supervised position	Correct token	Probability on token	Loss
Reply token 1	`docs`	`0.114`	`2.170`
Reply token 2	`today`	`0.736`	`0.306`
Mean loss			`1.238`

Bar chart comparing per-token cross-entropy losses 2.170 for missed docs and 0.306 for correct today, with an 88 to 12 contribution split and mean loss 1.238 feeding one backward pass. — The missed docs position contributes 2.170 of the 2.476 total loss, about 88%, while today contributes 0.306. Averaging gives 1.238, but the larger miss still dominates the shared gradient.

This final NumPy example applies the same stable loss across two supervised reply positions. Each row holds logits over the same tiny vocabulary; averaging yields the one scalar sent backward.

average-position-losses.py

import numpy as np

def per_position_cross_entropy(logits: np.ndarray, targets: np.ndarray) -> np.ndarray:
    shifted = logits - np.max(logits, axis=1, keepdims=True)
    log_probs = shifted - np.log(np.exp(shifted).sum(axis=1, keepdims=True))
    return -log_probs[np.arange(len(targets)), targets]

# Vocabulary order: bug, docs, today.
# Row 1 target is docs. Row 2 target is today.
logits = np.array([[3.0, 1.0, 0.0], [0.5, 0.0, 2.0]])
targets = np.array([1, 2])
losses = per_position_cross_entropy(logits, targets)

print("position losses:", np.round(losses, 3))
print("mean loss:", round(float(losses.mean()), 3))

Output

position losses: [2.17  0.306]
mean loss: 1.238

Debugging checklist

Symptom	Likely cause	Check or fix
Loss becomes `NaN` in custom NumPy code	Exponentiating large logits directly	Subtract row maximum, then compute log-sum-exp
Training loss is plausible but gradients are weak or wrong	Probabilities were passed into `CrossEntropyLoss`	Pass raw logits into the loss
Model is confidently wrong on a class	Correct label has tiny softmax mass	Inspect $p-y$ and confirm correct logit receives negative gradient
Generation samples unwanted alternatives	Sampling distribution is too flat	Inspect temperature and decoding policy separately from training
Mean sequence loss hides one severe miss	Mean reduction combines easy and hard positions	Print per-position losses before averaging

Mastery check

Key concepts

Logits are unconstrained scores whose differences determine probability shares.
Stable softmax subtracts the maximum logit before exponentiation.
Cross-entropy with a class label is the negative log probability of that label.
Softmax plus cross-entropy produces the logit gradient $p-y$ .
PyTorch categorical cross-entropy receives raw logits, not probabilities.
Temperature rescales logits; decoding decides how a token is selected.
Sequence training averages this same categorical loss across supervised positions.

Evaluation rubric

Foundational: Compute the stable softmax probabilities and cross-entropy loss for the docs example by hand.
Intermediate: Explain why max shifting leaves probabilities unchanged and diagnose an overflowing implementation.
Intermediate: Derive $p-y$ and explain the sign of each label gradient when bug is wrongly preferred.
Advanced: Verify the gradient numerically, use the PyTorch interface correctly, and connect per-position loss to next-token sequence training.

Common pitfalls

Symptom: A logit is reported as a confidence percentage. Cause: Raw output scores were confused with normalized probabilities. Fix: Apply softmax for interpretation and treat calibration as a separate evaluation question.
Symptom: Custom loss returns NaN on large scores. Cause: Direct exponentiation overflowed. Fix: Compute max-shifted softmax or stable log-sum-exp.
Symptom: nn.CrossEntropyLoss runs but learns poorly. Cause: Model probabilities were passed as if they were logits. Fix: Pass the final linear layer's raw scores.
Symptom: Lower temperature reinforces an incorrect option. Cause: Temperature sharpened existing score order; it doesn't learn a correction. Fix: Use training gradients to repair logits and tune decoding separately.
Symptom: Mean sequence loss looks acceptable despite a serious wrong token. Cause: Easy token positions dilute one high-loss event. Fix: Inspect unreduced position losses during debugging.

Follow-up questions

Next Step

Continue to RNNs, LSTMs, GRUs, and Sequence Modeling

You can now turn competing logits into a stable loss and a correction signal at each supervised position. Next you'll build the first model in this path that carries ordered history forward so those predictions can depend on what came before.

PreviousTraining & Backpropagation

Share this article

X Facebook LinkedIn Bluesky Reddit Hacker News Email

References

Deep Learning.

Goodfellow, I., Bengio, Y., Courville, A. · 2016

https://www.deeplearningbook.org/

Optimizing Model Parameters.

PyTorch Contributors · 2026 · Official tutorial

https://docs.pytorch.org/tutorials/beginner/basics/optimization_tutorial.html

Distilling the Knowledge in a Neural Network.

Hinton, G., Vinyals, O., & Dean, J. · 2015

https://arxiv.org/abs/1503.02531

Discussion

Questions and insights from fellow learners.

Discussion loads when you reach this section.

Softmax, Cross-Entropy & Optimization

A decision begins as three raw scores

Softmax assigns probability shares

The correct label is `docs`, but its probability is `0.114`. Why can't the model fix this by raising only the displayed probability number after softmax?

Stable arithmetic matters before scale does

Cross-entropy grades the correct label

Another example gets `0.80` probability on its correct label. Which example contributes more loss: that example or our `docs` example at `0.114`?

Why squared probability error is a weak fit here

The gradient says exactly what to correct

PyTorch receives logits directly

Temperature changes a distribution, not a label

If `docs` is correct but `bug` is currently the highest logit, does lowering temperature repair the model?

A sequence supplies many supervised positions

Debugging checklist

Mastery check

Key concepts

Evaluation rubric

Common pitfalls

Follow-up questions

Why does max-shifted softmax return the same probabilities as ordinary softmax?

For probabilities `[0.844, 0.114, 0.042]` and target `docs`, why is the gradient `[0.844, -0.886, 0.042]` useful?

Why should a PyTorch model omit `softmax` immediately before `nn.CrossEntropyLoss`?

What new question remains after every next-token prediction has a score?

Mastery Check

Discussion

A decision begins as three raw scores

Softmax assigns probability shares

The correct label is docs, but its probability is 0.114. Why can't the model fix this by raising only the displayed probability number after softmax?

The correct label is docs, but its probability is 0.114. Why can't the model fix this by raising only the displayed probability number after softmax?

Stable arithmetic matters before scale does

Cross-entropy grades the correct label

Another example gets 0.80 probability on its correct label. Which example contributes more loss: that example or our docs example at 0.114?

Another example gets 0.80 probability on its correct label. Which example contributes more loss: that example or our docs example at 0.114?

Why squared probability error is a weak fit here

The gradient says exactly what to correct

PyTorch receives logits directly

Temperature changes a distribution, not a label

If docs is correct but bug is currently the highest logit, does lowering temperature repair the model?

If docs is correct but bug is currently the highest logit, does lowering temperature repair the model?

A sequence supplies many supervised positions

Debugging checklist

Mastery check

Key concepts

Evaluation rubric

Common pitfalls

Follow-up questions

Why does max-shifted softmax return the same probabilities as ordinary softmax?

Why does max-shifted softmax return the same probabilities as ordinary softmax?

For probabilities [0.844, 0.114, 0.042] and target docs, why is the gradient [0.844, -0.886, 0.042] useful?

For probabilities [0.844, 0.114, 0.042] and target docs, why is the gradient [0.844, -0.886, 0.042] useful?

Why should a PyTorch model omit softmax immediately before nn.CrossEntropyLoss?

Why should a PyTorch model omit softmax immediately before nn.CrossEntropyLoss?

What new question remains after every next-token prediction has a score?

What new question remains after every next-token prediction has a score?

Mastery Check

Discussion

The correct label is `docs`, but its probability is `0.114`. Why can't the model fix this by raising only the displayed probability number after softmax?

Another example gets `0.80` probability on its correct label. Which example contributes more loss: that example or our `docs` example at `0.114`?

If `docs` is correct but `bug` is currently the highest logit, does lowering temperature repair the model?

For probabilities `[0.844, 0.114, 0.042]` and target `docs`, why is the gradient `[0.844, -0.886, 0.042]` useful?

Why should a PyTorch model omit `softmax` immediately before `nn.CrossEntropyLoss`?