From GPT to Modern LLMs

The next-token loop is simple: predict one token, append it, and repeat. Now ask how that loop grew from a research architecture into GPT (Generative Pre-trained Transformer) systems and the modern large language models (LLMs) behind assistants, coding tools, and retrieval systems.

The Transformer was introduced in 2017 for machine translation.^{[1]Reference 1Attention Is All You Need.https://arxiv.org/abs/1706.03762} Later models specialized its pieces into reading-focused encoders and generation-focused decoders. One concrete thread runs throughout: a developer-assistant prompt moving through a decoder-only model, so each historical step answers a practical question.

Same loop, bigger system: Modern generative LLMs still use the next-token loop from the last chapter. What changed is how models are pretrained, adapted to instructions, routed through parameters, and served within a latency and memory budget.

One generation loop, many tasks

Before the history, start with why the decoder-only path became so useful for general assistants. A decoder-only model reads text left to right and predicts the next token. That sounds simple, but it turns out to be a remarkably flexible interface.

A software-team assistant faces three different problems:

1. Classification. A review note says, "This docs page is stale." You need the label Docs.
2. Extraction. A build log contains a run ID buried inside a stack trace.
3. Generation. A teammate asks, "Why did the tests fail?" You need a concise answer grounded in the CI log.

An encoder-only model like BERT can be fine-tuned for the first task: read a review note, output a label. A decoder-only model can express all three tasks as "continue the text." A few-shot classification prompt looks like this:

The model reads the prompt from left to right, including the examples. Then it predicts the continuation at the open label slot. A sufficiently capable instruction-tuned model may produce Docs, but that outcome is something you evaluate, not assume.

This flexibility is why generation became a common product interface. One prompt format can express classification, extraction, summarization, translation, coding, and planning. That universality shaped the field.

The prompt framing itself is simple enough to inspect directly:

Why can a decoder-only model classify a review note even though it was trained to generate text?

Answer

The label is just the next token in a prompt pattern. If the prompt shows examples ending with labels such as Bug, Docs, and Security, the model can continue the pattern by generating the most likely label token for the new note.

The fork: reading and writing split apart

The original 2017 Transformer had two parts: an Encoder (to read the input) and a Decoder (to generate the output). Later models specialized these roles.

1. The encoder-only path (BERT). Google released BERT in 2018.^{[2]Reference 2BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.https://arxiv.org/abs/1810.04805} It used an encoder stack. It conditioned on tokens on both sides of each position and could be fine-tuned for tasks such as classification and extraction. For our developer-assistant example, a BERT-style classifier can be trained to map a review note to a label such as Docs. But BERT wasn't built to write long answers token by token.
2. The decoder-only path (GPT). OpenAI released GPT-1 in 2018.^{[3]Reference 3Improving Language Understanding by Generative Pre-Training.https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf} It used a decoder-style Transformer stack with masked self-attention, without the original translation decoder's encoder input. It read a prefix left to right and learned to predict the next token. It could write because generation was baked into the training objective. For our developer-assistant example, GPT can draft a CI failure explanation, one token at a time.

Encoder-only models became common for language-understanding tasks. Decoder-only models became the common architecture for general assistants. One practical reason is that generation is a flexible interface. If a model can generate text, you can ask for translation, summarization, coding, planning, or classification in the same format: prompt in, text out.

Why the fork matters for products

The fork wasn't just academic. It shaped how products are built today.

The main lesson isn't "BERT bad, GPT good." Encoder-style models are still useful for classification, extraction, reranking, and retrieval-specific embeddings.^{[4]Reference 4Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks.https://arxiv.org/abs/1908.10084} GPT-style decoders became the natural assistant interface because a single text-generation loop can express many tasks.

What "decoder-only" means in plain English

A decoder-only model writes with a bookmark:

1. It reads everything to the left of the bookmark.
2. It writes the next token.
3. The bookmark moves one token to the right.
4. It repeats the same loop.
5. It can't read future tokens because no future tokens exist yet.

That simple constraint is why the same architecture can write a French translation:

and produce:

or write a CI explanation:

Teammate: Why did the tests fail?

Assistant: The auth suite failed because the auth fixture expired...

---

No separate model was trained for each of these exact prompts. The model treats the examples and the conversation as context, then continues the pattern.

The attention visibility rule is small enough to inspect. In a causal decoder, a position can attend to itself and earlier positions, never future positions. In a bidirectional encoder, every input position is visible while building a representation.

The 0 for the future end token is the constraint that makes autoregressive generation valid. Real attention learns nonuniform weights over visible positions; this matrix only shows which connections are allowed.

What makes a decoder-only model different from an encoder-only model in one sentence?

Answer

A decoder-only model reads left to right and generates the next token, while an encoder-only model reads the whole input at once and produces a representation for tasks like classification, retrieval, or extraction.

The life of one token

To understand why later innovations matter, follow one token through the full decoder-only pipeline using a developer-assistant question: "Why did tests fail?"

The sentence first goes through tokenization, which breaks it into token IDs (for example, a tokenizer might assign one token representing tests an ID such as 1847, while punctuation and spaces may be separate or bundled with nearby text). Those IDs become vectors through an embedding lookup table. Each token ID maps to a learned high-dimensional vector used by the model's layers.

Next, causal self-attention lets each position mix information from allowed positions at or before it. For the representation at "fail," that includes "Why," "did," and "tests." The model computes Query, Key, and Value projections and uses attention weights to build a context-enriched vector for the current position. After attention, the feed-forward network transforms that vector independently, applying nonlinear transformations that turn raw context into useful features.

---

Finally, after any stack-level final normalization, the refined vector for the last position is projected through a large output matrix to produce scores over the vocabulary. GPT-2, for example, applies a final ln_f before computing logits.^{[5]Reference 5GPT-2 Source Implementation.https://github.com/openai/gpt-2/blob/master/src/model.py} Softmax turns those scores into probabilities, and a decoding policy chooses a next token. During generation, the KV cache stores Key and Value tensors from prior tokens so the model doesn't recompute those projections at each new step.

The same loop runs for every single token the model ever produces. Once a service handles real traffic, even small gains in how attention is computed, how many parameters are active per token, how positions are encoded, or how the KV cache is managed translate into large wins on latency, cost, and the maximum context length you can afford to serve.

A small shape trace makes that interface concrete. This isn't a Transformer implementation: random arrays stand in for learned final normalized hidden states and output weights. It exposes the two production objects to remember: last-position logits and cached prior Key/Value tensors.

Why scaling became a design variable

In 2020, OpenAI published work on scaling laws for neural language models.^{[6]Reference 6Scaling Laws for Neural Language Modelshttps://arxiv.org/abs/2001.08361} The core finding: loss improves predictably with three factors.

1. Parameter count (how big the model is)
2. Dataset size (how much text it reads)
3. Compute (how many training FLOPs, floating-point operations, you spend)

Between GPT-1 and GPT-3, OpenAI released GPT-2 in 2019, a 1.5-billion parameter decoder-only model that demonstrated surprisingly coherent open-ended generation and drew public attention to both the power and risks of scaling up autoregressive language models.^{[7]Reference 7Language Models are Unsupervised Multitask Learners.https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf}

OpenAI then trained GPT-3, a 175-billion parameter decoder-only model.^{[8]Reference 8Language Models are Few-Shot Learners.https://arxiv.org/abs/2005.14165} It was more than 100 times larger than GPT-2.

GPT-3 demonstrated in-context learning at a surprising scale.^{[8]Reference 8Language Models are Few-Shot Learners.https://arxiv.org/abs/2005.14165} You didn't need to retrain the model for every new task. You could show a few examples in the prompt, and the model often inferred the pattern on the fly. This made prompt engineering a practical developer skill.

Scaling laws with concrete numbers

Scaling laws are often summarized as "bigger is better." That's too sloppy. The useful statement is narrower and more balanced.

Consider two hypothetical training runs:

In this hypothetical table, the 175B model has 25x more parameters, but it was trained on only about 2x more tokens. The Chinchilla paper showed that many large models were undertrained for their size: a smaller model trained on proportionally more data can be far more efficient per parameter.^{[9]Reference 9Training Compute-Optimal Large Language Models.https://arxiv.org/abs/2203.15556}

This toy calculation makes the imbalance visible. parameters x tokens is only a training-compute proxy, not a quality predictor or a substitute for measured evaluation.

For engineers, this changes how you read model releases. Check four questions:

• How many parameters does it have?
• How much data was it trained on?
• What was the compute budget?
• How was the data filtered and which workload improved?

A smaller model trained well can beat a larger model trained poorly. That's why modern model announcements talk about tokens, data quality, context windows, tool use, and inference efficiency, not parameter count alone.

From autocomplete to assistant

GPT-3 was capable, but it was still a document completer. If you prompted it with "Explain this test failure", it might continue with another prompt instead of answering directly, because the base objective was "continue the text."

To make models respond to instructions more reliably, InstructGPT used supervised demonstrations followed by human-ranked model outputs, a learned reward model, and reinforcement-learning fine-tuning.^{[10]Reference 10Training Language Models to Follow Instructions with Human Feedback (InstructGPT).https://arxiv.org/abs/2203.02155} The important distinction is behavioral: a base model learns continuation, while an instruction-tuned model is adapted to answer requests.

Base model, instruct model, and chat model

These words get mixed together, so separate them early.

A base model can be capable but awkward. An instruct model is easier to use. A chat model adds conversation structure and behavior tuning.

The same user request looks different through three lenses:

This difference is why product teams usually start with an instruct or chat checkpoint instead of raw pre-training weights.

A common pairwise preference record has the same prompt paired with a chosen answer and a rejected answer. InstructGPT collected rankings of model outputs; pairwise records are a useful simplified view of the comparison signal.^{[10]Reference 10Training Language Models to Follow Instructions with Human Feedback (InstructGPT).https://arxiv.org/abs/2203.02155} The labels capture desired behavior; this code doesn't train a reward model.

Why is a base model awkward as a developer assistant?

Answer

A base model was trained to continue text from its pre-training distribution. It might complete a transcript, list, or document instead of directly answering the user. Instruction tuning and chat tuning teach the model to treat user requests as tasks and produce useful assistant-style responses.

Open weights and local inference

Meta introduced a different distribution model.

In early 2023, Meta introduced LLaMA (Large Language Model Meta AI), a family of foundation models, and released its models to the research community.^{[11]Reference 11LLaMA: Open and Efficient Foundation Language Models.https://arxiv.org/abs/2302.13971} That research-focused access gave approved users checkpoints they could evaluate and adapt instead of accessing a model only through a hosted interface. Downloadable weights and unrestricted commercial use are separate questions, which is why the license check below matters.

Researchers also didn't need 175 billion parameters for every useful workload. Chinchilla-style scaling showed that many models were undertrained for their size: smaller models trained on more tokens could be far more compute-efficient.^{[9]Reference 9Training Compute-Optimal Large Language Models.https://arxiv.org/abs/2203.15556} Independently, downloadable weights enable techniques such as fine-tuning (adapting a pre-trained model on task-specific examples) and quantization (using lower-precision weight representations to reduce memory cost), subject to model quality and license checks.

Open-weight doesn't mean open-source

This distinction matters in real engineering work.

Many model families are open-weight but not fully open-source in the strict software-license sense. That doesn't make them useless. It means you need to read the license before building a business around them.

Two current examples show why checkpoint access and deployment cost are separate questions:

Both models use MoE, so active parameter count helps explain per-token work but not checkpoint residency. Both also expose impressive vendor benchmark tables. Treat those as candidates for a matched internal eval, not proof that one model fits every workload.^{[13]Reference 13GLM-5.2 Model Cardhttps://huggingface.co/zai-org/GLM-5.2[21]Reference 21GLM-5.2 Model Documentationhttps://docs.z.ai/guides/llm/glm-5.2[16]Reference 16DeepSeek-V4-Flash-0731 Model Cardhttps://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731[20]Reference 20DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencehttps://arxiv.org/abs/2606.19348}

For a fuller deployment decision, use Open-Weight vs Closed API LLMs in 2026.

For a startup, open weights can provide:

1. Local inference for privacy-sensitive data.
2. Fine-tuning on domain examples.
3. Quantization for cheaper serving.
4. Debugging and evaluation without sending every token to a hosted API.

They also create work:

1. You need GPU capacity.
2. You own uptime.
3. You own safety filters.
4. You own model upgrade testing.
5. You need a license review.

Modern efficiency and inference choices

Beyond raw scaling, model builders can change how much computation and memory each generated token requires. This matters to engineers because a serving budget depends on the active computation path and stored state, not a headline parameter total alone.

Mixture of Experts (MoE)

Mixture-of-Experts (MoE) models activate only part of the network for each token. Mixtral, for example, routes each token to a small subset of expert feed-forward networks.^{[22]Reference 22Mixtral of Experts.https://arxiv.org/abs/2401.04088} The model gets more total capacity while keeping active compute lower than a dense model of the same total size.

A compiler pipeline gives a close analogy: a syntax error shouldn't run every optimization pass, and a type error doesn't need the final packaging step. Most passes stay idle for that file. In an MoE model, most expert parameters are inactive for any given token. That lowers active computation relative to a dense model with the same total size, though the full expert pool still consumes memory and routing adds systems tradeoffs.

A toy router shows the mechanism. Its labels are for explanation only: trained experts aren't guaranteed to align neatly with human topics.

Reasoning models and test-time compute

Standard decoder LLMs generate one token at a time. Reasoning-focused models spend additional inference-time computation before producing a final answer. OpenAI's o1-preview described this approach in product form in 2024.^{[23]Reference 23Learning to reason with LLMshttps://openai.com/index/learning-to-reason-with-llms/} DeepSeek-R1 reported reasoning behavior developed through reinforcement-learning stages in 2025.^{[24]Reference 24DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning.https://arxiv.org/abs/2501.12948} For applicable OpenAI reasoning API models, official documentation exposes model-dependent controls such as reasoning.effort (and, on some surfaces, reasoning.mode values like standard/pro) that trade speed and token use against additional reasoning work. Effort values, mode options, and whether the Responses API is preferred are model- and product-dependent; check current provider docs for the model you call.^{[25]Reference 25Reasoning modelshttps://developers.openai.com/api/docs/guides/reasoning} Other model families require their own documentation and evaluation.

For our developer-assistant example, a standard model might immediately answer a complex test failure and blame the wrong file. A reasoning model can spend extra inference-time compute on intermediate reasoning before producing the final answer, which can help it compare stack frames, recent diffs, and failing assertions more carefully. Training methods for reasoning and preference tuning get a dedicated chapter later; here you only need the shape of the idea.

The engineering stack that makes it fast

Modern inference relies on several optimizations that sit below the architecture headlines:

• Grouped-query attention (GQA). Instead of every Query head keeping separate Key and Value heads, groups of Query heads share Key and Value heads. GQA aims to retain quality close to multi-head attention while reducing inference cost.^{[26]Reference 26GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints.https://arxiv.org/abs/2305.13245}
• Rotary positional embeddings (RoPE). Rather than adding fixed position vectors to token embeddings, RoPE rotates pairs of Query and Key features by a position-dependent angle. This gives self-attention an explicit relative-position signal, though length extrapolation still needs careful training and evaluation.^{[27]Reference 27RoFormer: Enhanced Transformer with Rotary Position Embedding.https://arxiv.org/abs/2104.09864}
• FlashAttention. Standard attention reads and writes large intermediate matrices in GPU memory, which becomes expensive for long sequences. FlashAttention is an exact attention algorithm that tiles the computation to reduce memory traffic between high-bandwidth memory and on-chip SRAM.^{[28]Reference 28FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness.https://arxiv.org/abs/2205.14135}

For GQA, the serving consequence can be calculated directly: the KV cache scales with the number of Key/Value heads.

RoPE's rotation also exposes a useful invariant: shifting both token positions by the same amount preserves their relative-position interaction in this two-dimensional example.

GQA, RoPE, and FlashAttention don't change the next-token objective. They change cache cost, position handling, or attention memory traffic, which must be evaluated in the actual serving setup.

How to read a modern model announcement

When a new model appears, avoid one-metric thinking. Read it like an engineer.

This habit keeps you from overreacting to headlines.

Evaluate a released model on your workload

History is useful only if it improves a decision. A published benchmark is evidence about a harness and a date, not proof that a model meets your developer-assistant workload's quality, latency, privacy, or cost constraints.

Build an evaluation set of representative cases, such as failing CI logs that require exact error citation and code-review notes that require correct labels. Measure each candidate under the same prompt and tool setup. A scorecard can make tradeoffs explicit; its weights are product decisions, not universal truths.

A teammate says "we should use whatever benchmark leader is trending this week." What is the engineer's response?

Answer

"Best" is workload- and benchmark-specific. Pick based on your task, held-out evaluations, latency, context needs, price, license, and operational constraints, not on one leaderboard screenshot.

Long context doesn't mean equal attention

A common beginner mistake is assuming that a large advertised context window guarantees equally reliable use of evidence at every position. It doesn't.

Research on how language models use long contexts revealed a recurring pattern called "Lost in the Middle." When a model is given a very long prompt, it often performs best on relevant information at the beginning and the end. Relevant evidence placed in the middle of a long input is more likely to be missed or misused.^{[29]Reference 29Lost in the Middle: How Language Models Use Long Contextshttps://arxiv.org/abs/2307.03172}

For a developer assistant, this has a direct practical implication. If you paste a long incident log into the prompt and bury the failing stack trace in the middle, the model might miss the decisive lines. Keep important instructions and the most relevant context near strong positions such as the beginning or the end of the prompt, or break long documents into chunks and retrieve only the relevant ones.

---

You can enforce a prompt assembly rule in code. This rule places the governing policy first and the retrieved case evidence next to the question. It doesn't prove the model will answer correctly; evaluate that on held-out cases.

This behavior is one reason to use retrieval-augmented generation (RAG). Instead of dumping every document into the context window, you retrieve only the most relevant passages and place them strategically.

Match the architecture to the job

If you chose decoder-only for the ranking task, generation is flexible, but an encoder can still be the better tool for embeddings and classification.

Common misconceptions

Mastery check

Key concepts

• The split between Encoder-only (BERT) and Decoder-only (GPT) models
• Scaling laws: why size, data, and compute must be balanced
• The shift from GPT-3 to in-context learning
• Open weights: LLaMA, license checks, local inference, and fine-tuning workflows
• Modern architectural shifts: MoE (Mixture of Experts) and reasoning-time computation
• The life of a token: tokenization, embedding, attention, and KV cache
• Context window behavior: lost in the middle and prompt placement
• Base, instruct, and chat model behavior
• How to read a model announcement without relying on hype

Evaluation rubric

• Foundational: Differentiates between the original Transformer (Encoder-Decoder) and the GPT architecture (Decoder-only)
• Foundational: Explains why decoder-only generation can express classification, extraction, translation, coding, and chat as next-token continuation
• Foundational: Explains the significance of OpenAI's Scaling Laws
• Intermediate: Describes how GPT-3 demonstrated that large models can perform zero-shot and few-shot tasks without fine-tuning
• Intermediate: Identifies how downloadable model weights change evaluation and deployment options, subject to license terms
• Intermediate: Separates base models, instruct models, and chat models by behavior and tuning objective
• Intermediate: Explains why open-weight doesn't automatically mean open-source or unrestricted commercial use
• Intermediate: Describes the lost-in-the-middle effect and why long context windows aren't uniformly useful
• Intermediate: Explains why "best model" is workload-specific and lists the evaluation questions that matter more than a headline score

Follow-up questions

Why did many general-purpose assistant models use decoder-only architectures instead of encoder-only architectures like BERT?

Answer

BERT (encoder-only) is useful for understanding text for tasks such as classification and extraction because it can condition on tokens on both sides. GPT-style decoder-only models are built for autoregressive generation. Generation became a flexible product interface: you can frame many NLP tasks as text generation, from translation to summarization to coding.

What is a Mixture of Experts (MoE) model?

Answer

Instead of one dense feed-forward block where the same parameters are active for every token, an MoE layer has multiple expert feed-forward networks and a router. For each token, the router selects a small subset of experts, often one or two. Mixtral, for example, routes each token through two of eight feed-forward experts and uses about 12.9B active parameters out of 46.7B total.^[22]

What practical difference should a beginner remember between a base model and a chat model?

Answer

A base model is trained to continue text. A chat model has extra tuning and message formatting so it treats user turns as requests and assistant turns as helpful responses. Product teams usually start with chat or instruct checkpoints for developer assistants instead of raw pre-training checkpoints.

Common pitfalls

• Symptom: You estimate serving cost from total parameters alone. Cause: You didn't check whether a model routes tokens through experts. Fix: Record active parameters, KV-cache requirements, measured throughput, and latency.
• Symptom: You download weights and plan a deployment before license review. Cause: You treated open-weight as unrestricted open-source. Fix: Check license, acceptable-use terms, redistribution, and commercial-use requirements first.
• Symptom: Your developer assistant misses decisive log evidence in a long prompt. Cause: You assumed context capacity guaranteed uniform evidence use. Fix: Retrieve relevant passages, place them near the question, and evaluate positional robustness.