From GPT to Modern LLMs

The last chapter showed the next-token loop: predict one token, append it, and repeat. This chapter asks how that loop grew from a research architecture into GPT (Generative Pre-trained Transformer) systems and the modern large language models (LLMs) behind assistants, coding tools, and retrieval systems.

The Transformer was introduced in 2017 for machine translation.^[1] Later models specialized its pieces into reading-focused encoders and generation-focused decoders. We'll follow one concrete thread throughout: a customer-support message moving through a decoder-only model, so each historical step answers a practical question.

Key idea: Modern generative LLMs still use the next-token loop from the last chapter. What changed is how models are pretrained, adapted to instructions, routed through parameters, and served within a latency and memory budget.

One generation loop, many tasks

Before we talk about history, let's see why the decoder-only path became so useful for general assistants. A decoder-only model reads text left to right and predicts the next token. That sounds simple, but it turns out to be a remarkably flexible interface.

Imagine you run customer support for an online store. You face three different problems:

1. Classification. A customer writes, "I want my money back." You need the label "Refund."
2. Extraction. A customer pastes a messy paragraph that contains a tracking number buried inside it.
3. Generation. A customer asks, "Where is my order?" You need to write a polite reply that looks up the tracking status and explains the delay.

An encoder-only model like BERT can be fine-tuned for the first task: read a ticket, output a label. A decoder-only model can express all three tasks as "continue the text." Here is what a few-shot classification prompt looks like:

The model reads the prompt from left to right, including the examples. Then it predicts the continuation at the open label slot. A sufficiently capable instruction-tuned model may produce Shipping, but that outcome is something you evaluate, not assume.

This flexibility is why generation became a common product interface. One prompt format can express classification, extraction, summarization, translation, coding, and planning. That universality shaped the field.

The prompt framing itself is simple enough to inspect directly:

The fork: reading and writing split apart

The original 2017 Transformer had two parts: an Encoder (to read the input) and a Decoder (to generate the output). Later models specialized these roles.

1. The encoder-only path (BERT). Google released BERT in 2018.^[2] It used an encoder stack. It conditioned on tokens on both sides of each position and could be fine-tuned for tasks such as classification and extraction. For our support example, a BERT-style classifier can be trained to map a ticket to a label such as Refund. But BERT wasn't built to write long answers token by token.
2. The decoder-only path (GPT). OpenAI released GPT-1 in 2018.^[3] It used a decoder-style Transformer stack with masked self-attention, without the original translation decoder's encoder input. It read a prefix left to right and learned to predict the next token. It could write because generation was baked into the training objective. For our support example, GPT can draft a full reply to the customer, one token at a time.

Encoder-only models became common for language-understanding tasks. Decoder-only models became the common architecture for general assistants. One practical reason is that generation is a flexible interface. If a model can generate text, you can ask for translation, summarization, coding, planning, or classification in the same format: prompt in, text out.

Why the fork matters for products

The fork wasn't just academic. It shaped how products are built today.

The main lesson isn't "BERT bad, GPT good." Encoder-style models are still useful for classification, extraction, reranking, and retrieval-specific embeddings.^[4] GPT-style decoders became the natural assistant interface because a single text-generation loop can express many tasks.

What "decoder-only" means in plain English

Think of a decoder-only model as a writer with a bookmark:

1. It reads everything to the left of the bookmark.
2. It writes the next token.
3. The bookmark moves one token to the right.
4. It repeats the same loop.
5. It can't read future tokens because there are no future tokens yet.

That simple constraint is why the same architecture can write a French translation:

and produce:

or write a support reply:

Customer: Where is my order?

Agent: I'm sorry for the delay. Your tracking number is 1Z999AA10123...

---

No separate model was trained for each of these exact prompts. The model treats the examples and the conversation as context, then continues the pattern.

The attention visibility rule is small enough to inspect. In a causal decoder, a position can attend to itself and earlier positions, never future positions. In a bidirectional encoder, every input position is visible while building a representation.

The 0 for the future end token is the constraint that makes autoregressive generation valid. Real attention learns nonuniform weights over visible positions; this matrix only shows which connections are allowed.

The life of one token

To understand why later innovations matter, let's follow one token through the full decoder-only pipeline using a customer support question: "Where is my order?"

The sentence first goes through tokenization, which breaks it into token IDs (for example, a tokenizer might assign one token representing order an ID such as 1847, while punctuation and spaces may be separate or bundled with nearby text). Those IDs become vectors through an embedding lookup table. Each token ID maps to a learned high-dimensional vector used by the model's layers.

Next, causal self-attention lets each position mix information from allowed positions at or before it. For the representation at "order," that includes "Where," "is," and "my." The model computes Query, Key, and Value projections and uses attention weights to build a context-enriched vector for the current position. After attention, the feed-forward network transforms that vector independently, applying nonlinear transformations that turn raw context into useful features.

---

Finally, after any stack-level final normalization, the refined vector for the last position is projected through a large output matrix to produce scores over the vocabulary. GPT-2, for example, applies a final ln_f before computing logits.^[5] Softmax turns those scores into probabilities, and a decoding policy chooses a next token. During generation, the KV cache stores Key and Value tensors from prior tokens so the model doesn't recompute those projections at each new step.

The same loop runs for every single token the model ever produces. At production scale, even small gains in how attention is computed, how many parameters are active per token, how positions are encoded, or how the KV cache is managed translate into large wins on latency, cost, and the maximum context length you can afford to serve.

A small shape trace makes that interface concrete. This isn't a Transformer implementation: random arrays stand in for learned final normalized hidden states and output weights. It exposes the two production objects to remember: last-position logits and cached prior Key/Value tensors.

Why scaling became a design variable

In 2020, OpenAI published work on scaling laws for neural language models.^[6] The core finding: loss improves predictably with three factors.

1. Parameter count (how big the model is)
2. Dataset size (how much text it reads)
3. Compute (how many training FLOPs, floating-point operations, you spend)

Between GPT-1 and GPT-3, OpenAI released GPT-2 in 2019, a 1.5-billion parameter decoder-only model that demonstrated surprisingly coherent open-ended generation and drew public attention to both the power and risks of scaling up autoregressive language models.^[7]

OpenAI then trained GPT-3, a 175-billion parameter decoder-only model.^[8] It was more than 100 times larger than GPT-2.

GPT-3 demonstrated in-context learning at a surprising scale.^[8] You didn't need to retrain the model for every new task. You could show a few examples in the prompt, and the model often inferred the pattern on the fly. This made prompt engineering a practical developer skill.

Scaling laws with concrete numbers

Scaling laws are often summarized as "bigger is better." That's too sloppy. The useful statement is narrower and more balanced.

Consider two hypothetical training runs:

In this hypothetical table, the 175B model has 25x more parameters, but it was trained on only about 2x more tokens. The Chinchilla paper showed that many large models were undertrained for their size: a smaller model trained on proportionally more data can be far more efficient per parameter.^[9]

This toy calculation makes the imbalance visible. parameters x tokens is only a training-compute proxy, not a quality predictor or a substitute for measured evaluation.

For engineers, this changes how you read model releases. Check four questions:

• How many parameters does it have?
• How much data was it trained on?
• What was the compute budget?
• How was the data filtered and which workload improved?

A smaller model trained well can beat a larger model trained poorly. That's why modern model announcements talk about tokens, data quality, context windows, tool use, and inference efficiency, not only parameter count.

From autocomplete to assistant

GPT-3 was powerful, but it was still a document completer. If you prompted it with "Write a return-policy email", it might continue with another prompt instead of answering directly, because the base objective was "continue the text."

To make models respond to instructions more reliably, InstructGPT used supervised demonstrations followed by human-ranked model outputs, a learned reward model, and reinforcement-learning fine-tuning.^[10] The important distinction is behavioral: a base model learns continuation, while an instruction-tuned model is adapted to answer requests.

Base model, instruct model, and chat model

These words get mixed together, so separate them early.

A base model can be powerful but awkward. An instruct model is easier to use. A chat model adds conversation structure and behavior tuning.

Here is the same user request through three lenses:

This difference is why product teams usually start with an instruct or chat checkpoint instead of raw pre-training weights.

A common pairwise preference record has the same prompt paired with a chosen answer and a rejected answer. InstructGPT collected rankings of model outputs; pairwise records are a useful simplified view of the comparison signal.^[10] The labels capture desired behavior; this code does not train a reward model.

Open weights and local inference

Meta introduced a different distribution model.

In early 2023, Meta introduced LLaMA (Large Language Model Meta AI), a family of foundation models, and released its models to the research community.^[11] That research-focused access gave approved users checkpoints they could evaluate and adapt instead of accessing a model only through a hosted interface. Downloadable weights and unrestricted commercial use are separate questions, which is why the license check below matters.

Researchers also didn't need 175 billion parameters for every useful workload. Chinchilla-style scaling showed that many models were undertrained for their size: smaller models trained on more tokens could be far more compute-efficient.^[9] Independently, downloadable weights enable techniques such as fine-tuning (adapting a pre-trained model on task-specific examples) and quantization (using lower-precision weight representations to reduce memory cost), subject to model quality and license checks.

Open-weight doesn't mean open-source

This distinction matters in real engineering work.

Many model families are open-weight but not fully open-source in the strict software-license sense. That doesn't make them useless. It means you need to read the license before building a business around them.

For a startup, open weights can unlock:

1. Local inference for privacy-sensitive data.
2. Fine-tuning on domain examples.
3. Quantization for cheaper serving.
4. Debugging and evaluation without sending every token to a hosted API.

They also create work:

1. You need GPU capacity.
2. You own uptime.
3. You own safety filters.
4. You own model upgrade testing.
5. You need a license review.

Modern efficiency and inference choices

Beyond raw scaling, model builders can change how much computation and memory each generated token requires. This matters to engineers because a serving budget depends on the active computation path and stored state, not only a headline parameter total.

Mixture of Experts (MoE)

Mixture-of-Experts (MoE) models activate only part of the network for each token. Mixtral, for example, routes each token to a small subset of expert feed-forward networks.^[12] This gives the model more total capacity while keeping active compute lower than a dense model of the same total size.

Think of it like a specialized fulfillment center. You don't send every parcel through every station. A fragile return goes to inspection and repacking, while a normal outbound order goes straight to pick-pack-ship. Most stations stay idle for that parcel. In an MoE model, most expert parameters are inactive for any given token. That lowers active computation relative to a dense model with the same total size, though the full expert pool still consumes memory and routing adds systems tradeoffs.

A toy router shows the mechanism. Its labels are for explanation only: trained experts aren't guaranteed to align neatly with human topics.

Reasoning models and test-time compute

Standard decoder LLMs generate one token at a time. Reasoning-focused models spend additional inference-time computation before producing a final answer. OpenAI's o1-preview described this approach in product form in 2024.^[13] DeepSeek-R1 reported reasoning behavior developed through reinforcement-learning stages in 2025.^[14] For applicable OpenAI reasoning API models, official documentation exposes a model-dependent reasoning-effort control that trades speed and token use against additional reasoning work.^[15] Other model families require their own documentation and evaluation.

For our support example, a standard model might immediately answer a complex refund policy question and get a detail wrong. A reasoning model can spend extra inference-time compute on intermediate reasoning before producing the final answer, which can help it check the order date, the policy window, and the payment method more carefully. Training methods for reasoning and preference tuning get a dedicated chapter later; here you only need the shape of the idea.

The engineering stack that makes it fast

Modern inference relies on several optimizations that sit below the architecture headlines:

• Grouped-query attention (GQA). Instead of every Query head keeping separate Key and Value heads, groups of Query heads share Key and Value heads. GQA aims to retain quality close to multi-head attention while reducing inference cost.^[16]
• Rotary positional embeddings (RoPE). Rather than adding fixed position vectors to token embeddings, RoPE rotates pairs of Query and Key features by a position-dependent angle. This gives self-attention an explicit relative-position signal, though length extrapolation still needs careful training and evaluation.^[17]
• FlashAttention. Standard attention reads and writes large intermediate matrices in GPU memory, which becomes expensive for long sequences. FlashAttention is an exact attention algorithm that tiles the computation to reduce memory traffic between high-bandwidth memory and on-chip SRAM.^[18]

For GQA, the serving consequence can be calculated directly: the KV cache scales with the number of Key/Value heads.

RoPE's rotation also exposes a useful invariant: shifting both token positions by the same amount preserves their relative-position interaction in this two-dimensional example.

GQA, RoPE, and FlashAttention don't change the next-token objective. They change cache cost, position handling, or attention memory traffic, which must be evaluated in the actual serving setup.

How to read a modern model announcement

When a new model appears, avoid one-metric thinking. Read it like an engineer.

This habit keeps you from overreacting to headlines.

Evaluate a released model on your workload

History is useful only if it improves a decision. A published benchmark is evidence about a harness and a date, not proof that a model meets your support workload's quality, latency, privacy, or cost constraints.

Build an evaluation set of representative cases, such as damaged-item returns that require policy citations and delayed shipments that require exact scan times. Measure each candidate under the same prompt and tool setup. A scorecard can make tradeoffs explicit; its weights are product decisions, not universal truths.

Three architectural questions recur across model documentation:

The throughline is durable: the decoder-only next-token loop remained; scale increased capability, instruction tuning adapted behavior, downloadable weights changed deployment choices, and inference techniques changed cost and evaluation questions.

Context attention varies

A common beginner mistake is assuming that a large advertised context window guarantees equally reliable use of evidence at every position. It doesn't.

Research on how language models use long contexts revealed a recurring pattern called "Lost in the Middle." When a model is given a very long prompt, it often performs best on relevant information at the beginning and the end. Relevant evidence placed in the middle of a long input is more likely to be missed or misused.^[19]

For a support system, this has a direct practical implication. If you paste a 50-page return policy into the prompt and bury the customer's specific case details in the middle, the model might miss the case details. The fix is structural: keep important instructions and the most relevant context near strong positions such as the beginning or the end of the prompt, or break long documents into chunks and retrieve only the relevant ones.

---

You can enforce a prompt assembly rule in code. This rule places the governing policy first and the retrieved case evidence next to the question. It doesn't prove the model will answer correctly; evaluate that on held-out cases.

This behavior is one reason to use retrieval-augmented generation (RAG). Instead of dumping every document into the context window, you retrieve only the most relevant passages and place them strategically.

A quick check: match the architecture to the job

Test your understanding by deciding which architecture is a good default for each task.

The table below uses the support and fulfillment examples we have been following, so you can see how the choice changes based on the interface your product needs.

If you chose decoder-only for the ranking task, remember that generation is flexible, but an encoder can still be the better tool for embeddings and classification.

Common misconceptions

These mistakes are easy to make because headlines and marketing tend to oversimplify. If you catch yourself thinking any of the following, pause and check the correction.

These misconceptions are particularly dangerous in team settings, where one person's shorthand can shape architecture decisions for months.

Fixing these misconceptions early will save you from bad architecture decisions and from overpromising to stakeholders when you talk about model capabilities.

Carry these checks forward

After this chapter, you should be able to:

1. Explain why the original Transformer split into encoder-only (BERT) and decoder-only (GPT) paths, and why generation became the broader interface.
2. Walk through the life of a token inside a decoder: tokenization, embedding, attention, feed-forward, softmax, and KV cache reuse.
3. Read a model release with engineer skepticism: check parameters, data, compute, architecture, and license rather than headline claims.
4. Choose between base, instruct, and chat checkpoints based on the product need.
5. Recognize that long context windows aren't uniformly attentive, and place important instructions near strong positions such as the beginning or end of the prompt.
6. Explain why model rankings are workload-specific and what an engineer should measure before picking one.

Mastery check

Key concepts

• The split between Encoder-only (BERT) and Decoder-only (GPT) models
• Scaling laws: why making models bigger actually worked
• The shift from GPT-3 to in-context learning
• Open weights: LLaMA, license checks, local inference, and fine-tuning workflows
• Modern architectural shifts: MoE (Mixture of Experts) and reasoning-time computation
• The life of a token: tokenization, embedding, attention, and KV cache
• Context window behavior: lost in the middle and prompt placement
• Base, instruct, and chat model behavior
• How to read a model announcement without relying on hype

Evaluation rubric

• Foundational: Differentiates between the original Transformer (Encoder-Decoder) and the GPT architecture (Decoder-only)
• Foundational: Explains why decoder-only generation can express classification, extraction, translation, coding, and chat as next-token continuation
• Foundational: Explains the significance of OpenAI's Scaling Laws
• Intermediate: Describes how GPT-3 demonstrated that large models can perform zero-shot and few-shot tasks without fine-tuning
• Intermediate: Identifies how downloadable model weights change evaluation and deployment options, subject to license terms
• Intermediate: Separates base models, instruct models, and chat models by behavior and tuning objective
• Intermediate: Explains why open-weight doesn't automatically mean open-source or unrestricted commercial use
• Intermediate: Describes the lost-in-the-middle effect and why long context windows aren't uniformly useful
• Intermediate: Explains why "best model" is workload-specific and lists the evaluation questions that matter more than a headline score

Follow-up questions

Common pitfalls

• Symptom: You estimate serving cost from total parameters alone. Cause: You didn't check whether a model routes tokens through experts. Fix: Record active parameters, KV-cache requirements, measured throughput, and latency.
• Symptom: You download weights and plan a deployment before license review. Cause: You treated open-weight as unrestricted open-source. Fix: Check license, acceptable-use terms, redistribution, and commercial-use requirements first.
• Symptom: Your support assistant misses decisive case evidence in a long prompt. Cause: You assumed context capacity guaranteed uniform evidence use. Fix: Retrieve relevant passages, place them near the question, and evaluate positional robustness.