Language Modeling & Next Tokens

The Transformer chapter ended with logits and softmax: a vector of scores becomes a probability distribution over possible next tokens. This chapter explains what that distribution is used for during language modeling.

At the core, a large language model (LLM) does one repeated job: it predicts the next token. This article builds that job from counts into a trainable objective, then shows how a causal Transformer uses it during generation.

Key idea: An LLM doesn't generate a whole paragraph at once. It predicts a single token, appends it to its input, and runs the process again. This is called autoregressive generation.

---

What is a language model?

An autoregressive language model is a probability engine for token sequences: given the tokens so far, it assigns a probability distribution over the vocabulary for what comes next.

Think of it like a warehouse routing system that must choose the next destination for a package. It doesn't treat every destination as equally likely; it ranks them using the package's origin, weight, and delivery promise.

A useful way to make this precise: the probability of a whole sequence factorizes into a product of next-token predictions, one per position. This is just the chain rule of probability.

$$P(x_1, x_2, \dots, x_n) = \prod_{t=1}^{n} P(x_t \mid x_1, \dots, x_{t-1})$$

Read it left to right: the chance of the first token, times the chance of the second given the first, times the chance of the third given the first two, and so on. Each factor is one next-token distribution. That single building block, "distribution over the next token given everything before it," is the entire job. Everything else in this chapter (generation, training, perplexity) is built on it.

Suppose a model receives this prompt: "The package left the"

Its actual probabilities depend on its tokenizer, weights, and preceding context. In a simplified example, it might rank four possible continuations like this:

Probability table

This table is only an excerpt. Tokens omitted from the table receive the remaining probability mass.

Read the distribution before sampling

The table above isn't a list of words the model "knows." It's the model's next-step belief after reading the prompt.

That distinction matters because the same model can behave differently after one extra token.

The model doesn't store one fixed answer for "the package left the." It computes a distribution from the whole context. Change the context, and you change the probabilities.

Try the mental move:

1. Read this prompt.
2. Predict three likely next tokens.
3. Compare your predictions with the options below.

Possible continuations:

All three are plausible. A model doesn't need certainty to generate text. It needs a probability ranking, then the decoding algorithm picks from that ranking.

Tiny vocabulary example

A production vocabulary contains many tokens, with its exact size set by the model's tokenizer. That's too many for a beginner example, so shrink the world to five tokens.

Imagine this prompt:

The support bot answered

The model produces logits, which are raw scores before softmax:

Softmax turns those scores into probabilities:

One line in the code deserves explanation before you read it: logits - logits.max().

Why subtract the maximum logit? Because softmax only cares about differences between logits, not their absolute level. If you subtract the same constant from every score, the ranking and final probabilities stay the same. But the numerics get much safer: the largest shifted logit becomes 0, so its exponential is e^0 = 1 instead of some huge number. On tiny toy logits like [3.0, 1.5, -1.0, ...] this barely matters. On real model logits it prevents overflow and keeps the computation stable.

The exact numbers are less important than the shape:

1. One token is clearly ahead.
2. Another token still has a real chance.
3. Some tokens are valid but unlikely.
4. The probabilities add to 1.

That's next-token prediction in miniature.

From bigram tables to neural n-grams

The smallest context-aware language model is a bigram model. It looks at one previous token and predicts the next token from a table of counts:

If the support-log corpus often contains left -> warehouse, the row for left assigns high probability to warehouse. If it rarely contains left -> coupon, that probability stays tiny.

That table already teaches the core language-modeling contract: read context, produce a probability distribution over the next token. Its weakness is context length. A bigram model can't tell the difference between package left the and customer left the, because it only sees the final token.

Build a count-based language model

Start with six tracking-message fragments. Each adjacent token pair becomes one training event: after left, which token appeared next?

An unseen pair receives probability zero in this unsmoothed table. More data or smoothing helps, but a count table still can't share what it learned about related contexts.

An n-gram model predicts the next token from the previous n - 1 tokens:

For tiny corpora, count tables still work. For real language, the table explodes because most exact contexts are rare. A neural n-gram model addresses that sparsity by learning embeddings and a small MLP (multi-layer perceptron):

1. Look up embeddings for the previous n - 1 tokens.
2. Concatenate them into one vector.
3. Apply a matrix multiply, bias, and a nonlinearity such as the Gaussian Error Linear Unit (GELU).
4. Project to vocabulary logits.
5. Softmax into next-token probabilities.

This is the bridge from count-based autocomplete to modern LLMs. The objective stays next-token prediction. The model family changes: bigram table, neural n-gram MLP, recurrent network, then Transformer. Each step gives the model a better way to use context.

Train a tiny neural n-gram

The next cell trains a small two-token-context, or trigram, model. It isn't a Transformer, but it exposes the same contract: embeddings and shared weights produce logits, and cross-entropy rewards the true next token.

The tiny corpus is deliberately simple, so this cell demonstrates optimization rather than a production-quality model. The important bridge is that a neural model can use one set of learned parameters across contexts instead of storing an isolated probability row for every exact phrase.

Why temperature matters

So far we've ranked likely tokens and used the highest-probability one as the default. In practice, many applications sample from the distribution instead. A temperature parameter rescales the logits before softmax, giving the model more or less freedom to deviate from the top choice.

Why do we divide by temperature instead of adding or subtracting something? Because dividing changes the gap between logits. A temperature below 1 stretches the gaps, so the winner pulls farther ahead after softmax. A temperature above 1 compresses the gaps, so lower-ranked tokens stay more competitive. Adding a constant would not do that; softmax would cancel it out. The formula requires a temperature above 0. Treat a zero-temperature mode as greedy decoding in code instead of dividing logits by zero.

Here's the same logits at three temperatures:

---

At low temperature (0.5), the distribution sharpens. correctly dominates almost completely. This is close to greedy behavior: safe, repetitive, and predictable.

---

At high temperature (2.0), the distribution flattens. slowly becomes much more likely than before, and even odd words like angrily gain real probability. This introduces variety, but also raises the risk of incoherent output.

Production note: Low temperature keeps a support bot consistent and on-brand; higher temperature produces more varied drafts for creative tools. Temperature is one lever among several for shaping generation; a later chapter covers the full decoding toolkit.

Causal language modeling

This specific flavor of language modeling, reading from left to right and predicting the future, is called causal language modeling (or autoregressive modeling). GPT-style models use this objective.^[1][2]

It's called "causal" because the model is bound by causality: each position can use earlier context, but it can't peek at future tokens. In a Transformer block, a token position may attend to itself and earlier positions; the mask blocks later positions.

GPT-style decoder models are causal language models.

(Note: There's another type called masked language modeling, used by Bidirectional Encoder Representations from Transformers (BERT), where a token in the middle of a sentence is hidden and the model looks at both the left and right context to predict the blank.^[3] That objective builds representations from both sides of a blank; the left-to-right loop in this chapter instead needs causal modeling.)

---

How the model learns the distribution

We keep saying the model "predicts" the next token. Where does that skill come from? From training on text where the next token is already known. Every sentence in the training data is a free set of labeled examples: at each position, the correct answer is simply the token that actually came next.

Teacher forcing

During training the model is fed the real previous tokens at every position, not the tokens it would have guessed. This is called teacher forcing. The model never gets to wander off on its own mistakes while learning; each position is scored against ground truth using the true history.

This distinction matters whenever you interpret a training loss or measure generation latency:

Because the true history is known up front, training scores every position of a sequence in a single forward pass. The causal mask is what makes this honest: the logit produced from prefix x_1 ... x_{t-1} can't read its target token x_t.

This side-by-side view helps lock in why training is parallel while generation isn't:

Shift tokens into inputs and labels

For a decoder, one training sequence supplies several labeled predictions. The input is the sequence shifted right; the target is the same sequence shifted left.

Cross-entropy: the training loss

At each position the model outputs a distribution over the vocabulary. We want the probability it assigns to the true next token to be high. The loss for one position is the negative log of that probability, and the loss for a sequence is the average over positions. This is cross-entropy loss, the same objective introduced in the softmax chapter, now applied at every position.

$$\mathcal{L} = -\frac{1}{n} \sum_{t=1}^{n} \log P(x_t \mid x_1, \dots, x_{t-1})$$

If the model gives the true token probability 1.0, that term contributes -log(1) = 0 loss: a perfect prediction. If it gives the true token a tiny probability, -log of a small number is large: a big penalty. Causal pretraining minimizes this average over a large corpus.

The logits at every shifted position can be scored together. This cell calculates stable cross-entropy for the four targets above.

Perplexity: loss in token units

Cross-entropy is measured in nats (because we used natural log), which is hard to feel. Perplexity is the friendlier form: just exponentiate the average loss.

$$\text{Perplexity} = \exp(\mathcal{L})$$

Perplexity has a clean interpretation: it's the effective number of equally likely tokens the model is choosing between at each step. A perplexity of 1 means the model is certain every time. A perplexity of 50,000 on a 50,000-token vocabulary means the model learned nothing and is guessing uniformly. Lower is better.^[4][5]

Work a tiny example by hand. Suppose the model predicts four tokens, assigning the true token these probabilities: 0.5, 0.5, 0.5, 0.125.

A perplexity of about 2.83 means that, averaged across these four steps, the model behaved as if it were choosing between roughly 2.83 equally likely options. The one bad prediction (0.125) drags the average up. That is the whole point of the metric: it tells you, in token units, how surprised the model was by real text.

When you fine-tune a model and watch the loss curve, you're watching average cross-entropy fall. When an eval report uses the same tokenizer, evaluated tokens, and natural-log averaging, perplexity is exp of that same loss. Hold those choices constant before treating a mismatch as a regression.

---

The autoregressive loop

When you ask a chat model to write a shipping-delay update, it doesn't output the whole message in one step. It generates the message one token at a time in a loop.

This is the autoregressive generation loop:

1. Input: The model receives the prompt.
2. Predict: It calculates the probabilities for the very next token.
3. Pick: We take the most likely token or sample from the distribution, optionally adjusting temperature.
4. Append: That chosen token is glued to the end of the prompt.
5. Repeat: The new, longer context determines the distribution for the next token.

This list describes the dependency between generated tokens, not an inefficient serving implementation. Production decode usually reuses earlier attention state through a key-value (KV) cache. It still performs one sequential decode step for each new token, but it doesn't rebuild earlier keys and values. The serving section below makes that tradeoff concrete.

This explains why generation speed is measured in tokens per second (t/s). The model has to run another decoding step for every generated token. A KV cache avoids recomputing earlier keys and values from scratch, but each new token still has to attend to previous context.

A tiny generation loop in code

The code below isn't a neural network. It's a small hand-written language model that shows the same loop: look at current text, choose one next token, append it, and repeat.

Real LLMs replace the hand-written table with a Transformer, but the control flow is the same.

The loop ends when one of these things happens:

Picking a token from the distribution

The loop says "sample a token," but how? The model hands you a distribution; turning it into a single chosen token is the job of a decoding strategy. The simplest is greedy decoding: always take the highest-probability token. It is deterministic, but it can fall into loops, repeating "We apologize for the inconvenience. We apologize for the inconvenience..." because "apologize" keeps scoring highest after "We."

The alternative is to sample: draw a token at random in proportion to its probability, so the second- or third-ranked token sometimes wins. Sampling can reduce repetition and add variety, but careless sampling can also hurt quality. The temperature knob from earlier controls how adventurous that draw is. Greedy versus sampling is the core fork; the full toolkit of decoding methods (and how to tune them) gets a dedicated chapter later, so here we only need the idea that one distribution can be turned into text in more than one way.

Common mistake: Setting a high temperature in a customer-facing support bot and wondering why answers drift off-brand, or setting temperature to 0 in a creative app and wondering why every user gets the same poem. At each step, the current context produces a distribution; the decoding strategy decides how much you gamble on it.

Use a seeded random generator when testing a sampler, so the result is repeatable.

The context window limit

Because the output is constantly being appended to the input, the text the model has to read keeps growing. If you generate a 1000-token support answer, by the end, the model is reading the original prompt plus the 999 tokens it just generated.

This growing sequence must fit into the model's context window, its short-term memory limit. If the context window is 8,000 tokens and the prompt plus generated text reaches 8,001 tokens, the system has to truncate earlier context, reject the request, summarize, or use a longer-context strategy.

What the KV cache changes

Without a cache, the model would recompute attention information for the full prompt at every generation step. That would be wasteful.

The KV cache stores key and value tensors from earlier tokens at each attention layer. New decode queries can attend to these cached tensors instead of rebuilding earlier keys and values at every step.^[6]

Serving systems usually split this into two phases:

• Prefill: process the full prompt in parallel and build the initial KV cache.
• Decode: generate one new token at a time, appending each token and reusing the cache.

This is why serving systems care so much about KV cache memory. The cache speeds generation, but it also consumes GPU memory proportional to context length, number of layers, key/value-head count, head dimension, and batch size.

Production note: If a model seems cheap at 4K context but expensive at 128K context, check KV cache memory before blaming the model weights. Long context changes serving economics.

Estimate cache memory

A simplified cache estimate makes the tradeoff concrete. For each request, layer, token, and key/value head, the server stores one key vector and one value vector. This example uses bfloat16 or float16 values at two bytes each.

This is only the KV cache, not model weights, activations, allocator overhead, or batching metadata. It also shows why fewer key/value heads can reduce cache cost.

---

How can next-token prediction lead to useful capabilities?

A common critique of LLMs is that they're only autocomplete.

If it's autocomplete, how can it write code, solve logic puzzles, or translate languages?

The answer lies in scale, diversity, and optimization. When a neural network predicts the next token across huge mixtures of code, math, dialogue, documentation, and books, simple word association often isn't enough to minimize the loss function.

Worked prompt

Consider this prompt: "An order contains 5 items. After a return of 2 items, the retained item count is"

A simple Markov chain might look at 2 and guess 3 because 2, 3 is common. A larger neural language model can benefit from features that track subtraction-like structure across many related examples, because those features can improve next-token predictions. This is a pressure from the objective, not a guarantee of correct arithmetic.

To predict text from a complex world, the model has pressure to learn useful approximations of that world. Grammar, facts, reasoning patterns, and code structure can become helpful internal representations for minimizing next-token prediction loss.

What this explanation does and doesn't claim

Next-token prediction explains the training signal. It doesn't prove that every model has human-like understanding.

Good language here is careful:

This matters in system design and technical discussions. If you say "it's just next-token prediction," you miss learned structure. If you say "it's basically a human mind," you overclaim. Large generatively pretrained models have shown useful task behavior, but each capability still needs measurement on the task that matters.^[1][7]

---

Practice: trace one completion

Take this prompt:

A shipment event JSON record starts with

Answer these questions before looking down:

1. What are three plausible next tokens?
2. Which token is likely highest probability?
3. What would happen if the model outputs {?
4. Why might a newline or quote still be possible?

One reasonable answer:

Now connect that to code generation. A model that has seen many shipment-event JSON examples can learn syntax regularities: opening braces, quoted keys, colons, commas, values, and closing braces. The same principle applies to Python indentation, SQL clauses, and Markdown tables.

---

Common failure cases

Treating probabilities as facts

The model's top token can be wrong.

Suppose a support bot sees:

The refund policy says annual plans are

A model might assign high probability to non-refundable because that phrase is common in policy text. But if your company's actual policy says annual plans are refundable within 14 days, the statistically likely continuation is still wrong.

The fix isn't to hope the base model "knows" your policy. Put the trusted policy into context, retrieve the right document, or use a tool that checks the source of truth.

The "database" myth

Many beginners assume an LLM contains a hidden database of facts that it looks up during generation. It doesn't. When a model completes Orders marked delivered are eligible for with refund, it isn't querying the current returns-policy table. It's producing a likely continuation from its context and learned parameters.

This distinction matters because:

1. Facts can be outdated. Information stored in model parameters reflects training data. Without fresh context or tools, the model won't know about later events, policy changes, or catalog updates.
2. Facts can be wrong. If the training data contains conflicting information, the model may reproduce the most common version, even if it's inaccurate.
3. Facts can be hallucinated. When the model is uncertain, it may generate a plausible-sounding but false continuation rather than admitting ignorance.

The correct mental model is statistical generation, not retrieval. If you need accurate, up-to-date facts, you must ground the model with retrieval, tools, or explicit context.

---

Summary

• Autoregressive language models output a probability distribution over a vocabulary for the next token; the probability of a whole sequence is the product of these per-token predictions (the chain rule).
• Causal language modeling uses earlier context without peeking at future tokens.
• Teacher forcing trains the model on the true previous tokens, which lets every position be scored in one parallel forward pass; generation instead feeds the model its own outputs.
• The training objective is cross-entropy loss, the averaged negative log-probability of the true next token. Perplexity is exp(loss): the effective number of choices the model faces per step.
• Autoregressive generation is a loop where the predicted token is appended to the input, and the process repeats.
• Bigram, neural n-gram MLP, recurrent, and Transformer language models share the same next-token target; they differ in how much context they can use and how they represent it.
• A decoding strategy turns the distribution into a token. Greedy is deterministic; sampling (tuned by temperature) adds variety. The full toolkit comes in a later chapter.
• The context window is the maximum number of tokens the model can hold in this input loop; the KV cache trades memory for speed during generation.
• Useful capabilities can appear because reusable structure about language, code, and tasks can help reduce next-token prediction error at scale, but they still need evaluation.
• High probability doesn't mean factual truth. LLMs generate, they don't retrieve from a hidden database.

Mastery check

Key concepts

• Language-model objective: next-token prediction
• Chain-rule factorization of sequence probability
• Bigram and neural n-gram models share the next-token target
• Teacher forcing: training on true history in one parallel pass
• Cross-entropy loss as averaged negative log-likelihood
• Perplexity equals exp(loss): effective choices per token
• Autoregressive generation loop
• Causal vs masked language modeling (GPT vs BERT)
• Context window limits and KV-cache serving tradeoffs
• The database myth: LLMs generate instead of retrieving facts

Evaluation rubric

• Foundational: Defines an autoregressive language model in terms of a probability distribution over the next token
• Foundational: Writes sequence probability as a product of per-token conditionals (chain rule)
• Foundational: Explains how a bigram table estimates P(next token | previous token) from counts
• Foundational: Explains the autoregressive generation loop step-by-step
• Foundational: Demonstrates how temperature rescales logits and changes the output distribution
• Intermediate: Describes how a neural n-gram MLP uses embeddings, concatenation, matmul, GELU, and an output projection to predict the next token
• Intermediate: Explains teacher forcing and why training scores all positions in one parallel pass while generation can't
• Intermediate: Connects cross-entropy loss to perplexity via perplexity = exp(loss) and interprets perplexity as effective choices per token
• Intermediate: Contrasts causal (left-to-right) modeling with masked (fill-in-the-blank) modeling
• Intermediate: Distinguishes greedy decoding from sampling as ways to turn one distribution into a token
• Intermediate: Discusses why predicting the next token can reward useful world representations
• Intermediate: Explains why KV cache changes generation speed and serving memory trade-offs
• Intermediate: Identifies why high-probability completions can still be factually wrong without retrieved context
• Intermediate: Corrects the database myth by explaining that LLMs generate rather than retrieve facts

Follow-up questions

Common pitfalls

• Symptom: A creative demo keeps giving nearly identical completions. Cause: Decoding is too deterministic, often from greedy decoding or very low temperature. Fix: Sample from the distribution and raise temperature carefully instead of always taking the top token.
• Symptom: Training loss looks good, but long generations drift after one bad token. Cause: Teacher forcing was confused with real generation, so evaluation never tested the model feeding on its own outputs. Fix: Run autoregressive generation during eval, not loss-only checks.
• Symptom: A model answers a policy question confidently but cites the wrong company rule. Cause: High next-token probability was treated like evidence, even though the model is generating from patterns instead of querying a live source. Fix: Retrieve the current policy or call a tool before trusting the answer.
• Symptom: Generation gets slower or more expensive as outputs get longer. Cause: The key-value cache saves recomputation but still grows with context length, layers, and batch size. Fix: Watch context budgets and cache memory, not only model weight size.
• Symptom: Someone claims perplexity and cross-entropy disagree. Cause: They forgot perplexity is exp(loss) under natural-log averaging, or compared runs with different token accounting. Fix: Hold tokenizer, evaluated tokens, and averaging convention constant before comparing them.
• Symptom: Bigram tables, neural n-grams, and GPT-style models feel unrelated. Cause: Architecture changes were mistaken for objective changes. Fix: Trace all three back to the same job: predict the next token from context.