The Bitter Lesson & Compute

If you're trying to teach a computer to play chess, you could spend years writing down every rule a grandmaster knows: "control the center," "don't move the same piece twice," or "connect your rooks." This works for a while. But what if you just gave the computer the basic rules and let it play against itself a billion times on super-fast hardware? It turns out the second approach, relying on raw computation instead of human expertise, always wins in the end. This surprising and consistent outcome is known as "The Bitter Lesson."

In March 2019, Rich Sutton, one of the fathers of reinforcement learning (the branch of AI where agents learn by trial and error), published a short essay that would become one of the most cited and debated pieces in AI history ^[1]. "The Bitter Lesson" makes a simple but profound claim: the biggest lesson that can be read from 70 years of AI research is that general methods that use computation are ultimately the most effective, and by a large margin.

The core thesis

Sutton's argument can be summarized as three claims:

💡 Key insight: Analogy: Power tools vs. hand tools. Imagine two woodworkers: one spends years mastering hand-carving techniques (a "heuristic" or rule-of-thumb approach), creating beautiful but slow work. The other buys a CNC machine and learns to operate it. At first, the hand-carver produces better furniture. But the CNC operator keeps buying faster machines, and within a few years, they're producing furniture that's better and orders of magnitude faster. The Bitter Lesson says AI follows this same pattern: cleverly hand-crafted solutions lose to general methods that scale with compute.

1. •

   AI researchers have repeatedly tried to build human knowledge into their agents, through hand-crafted features, domain-specific heuristics, and expert-designed representations.

2. •

   These approaches work well initially but eventually hit a ceiling, because human-engineered knowledge is complex, brittle, and hard to scale.

3. •

   General methods that use computation, specifically search and learning, consistently win in the long run, because they scale with Moore's Law while human engineering effort doesn't.

The two methods Sutton highlights as scalable are:

• •Search: exploring large spaces of possibilities at inference time (e.g., Monte Carlo Tree Search (MCTS), beam search, test-time compute)
• •Learning: optimizing parameters from data at training time (e.g., gradient descent, reinforcement learning)

Both convert compute into performance. Neither requires encoding human domain knowledge. Both have, historically, displaced every hand-engineered alternative.

Historical evidence: the pattern repeats

The Bitter Lesson isn't just a theory about the future; it's an observation about the past. The same pattern appears again and again across different fields in AI. Early progress often comes from experts encoding their knowledge into systems. But eventually, these hand-crafted systems are completely overtaken by simpler methods that just use massive computation. Let's look at the evidence from five key areas.

Chess: from grandmaster heuristics to brute-force search

In the 1970s-90s, chess AI was dominated by programs with hand-crafted evaluation functions: piece-square tables, pawn structure analysis, king safety metrics, and endgame databases, all designed by chess experts. These programs were impressive but fragile. Each new improvement required painstaking human engineering.

Then came Deep Blue (1997), which used massive hardware parallelism to search ~200 million positions per second. It still used hand-tuned evaluation, but compute was doing most of the work. The culmination was AlphaZero (2017) ^[2], which learned to play chess entirely from self-play using zero human knowledge beyond the rules. Starting from random play, it surpassed every hand-engineered chess program within hours of training on a TPU (Tensor Processing Unit) cluster.

The expert knowledge accumulated over decades of chess programming was made obsolete by a general algorithm (MCTS + neural network evaluation) powered by compute.

Go: the impossible problem solved by learning

Go was considered the frontier of game AI due to its enormous branching factor ($b \approx 250$ vs. chess's $b \approx 35$). For decades, the strongest Go programs used pattern databases hand-designed by Go experts: joseki dictionaries, life-and-death solvers, influence maps.

These programs barely played at amateur level.

AlphaGo (2016) combined deep neural networks with MCTS, initially trained on human expert games and then refined through self-play. AlphaGo Zero (2017) removed even the human games entirely: pure self-play from scratch. It became the strongest Go player in history, playing moves that human experts described as "alien" and "beautiful."

🔬 Research insight: AlphaGo's move 37 in Game 2 against Lee Sedol, a shoulder hit on the fifth line that no human player would consider, was later recognized as genuinely creative. A hand-engineered system would never have discovered it.

Speech recognition: from phoneme rules to end-to-end learning

The history of automatic speech recognition (ASR) follows the same arc:

• •1970s-2000s: Hidden Markov Models (HMMs) with hand-designed phoneme models, pronunciation dictionaries, and language model hierarchies. Building a production ASR system required teams of linguists and phoneticians.
• •2012-2016: Deep neural network acoustic models replaced HMM features, but the pipeline (feature extraction → acoustic model → language model → decoder) remained hand-engineered.
• •2017+: End-to-end models like CTC (Connectionist Temporal Classification), Listen-Attend-Spell, and eventually Whisper (2022) ^[3] collapsed the entire pipeline into a single neural network trained on massive data. Whisper was trained on 680,000 hours of web audio. Its "knowledge" of phonetics, accents, and languages was learned entirely from compute and data.

Every hand-designed component (mel-frequency cepstral coefficients, tri-phone models, n-gram language models) was eventually replaced by a general neural network doing the same job, but better, because it could use more compute and data.

Computer vision: from SIFT to transformers

The vision trajectory is equally clear, and it illustrates an important point about inductive bias. An inductive bias is a prior assumption baked into the model architecture about the structure of the data. Formally, it restricts the hypothesis space $\mathcal{H}$ that the model searches over during training:

• •2000s: Hand-designed feature extractors like SIFT (Scale-Invariant Feature Transform), HOG (Histogram of Oriented Gradients), and SURF (Speeded-Up Robust Features) fed into SVMs (Support Vector Machines) or random forests. Each feature was carefully engineered to capture edges, gradients, or keypoints.
• •1998→2012: LeNet^[4] introduced the convolutional inductive bias: translation equivariance (the same filter detects the same pattern everywhere) and local receptive fields (each neuron only sees a small patch). This structure is genuinely true of images and dramatically reduced the number of learnable parameters. AlexNet^[5] (a deeper CNN trained on ImageNet with GPU compute) demolished hand-crafted features at ILSVRC (ImageNet Large Scale Visual Recognition Challenge), dropping error rates from ~26% to ~16%. The features were learned, not designed.
• •2020: Vision Transformers (ViTs)^[6] removed even the convolutional inductive bias, treating images as sequences of patches and learning everything from data. When trained with sufficient compute and data, ViTs matched or exceeded CNNs, despite having less domain-specific structure.

The progression from hand-designed features → learned features with strong inductive bias (CNNs) → learned features with minimal inductive bias (Transformers) is the Bitter Lesson in action, compressed into two decades. The key insight is that inductive biases accelerate learning within a fixed compute budget, but as compute grows, they become unnecessary constraints.

Natural language processing: from parse trees to language models

Natural Language Processing (NLP) provides a dramatic example:

• •1990s-2000s: NLP systems were built on linguistic engineering: part-of-speech taggers, dependency parsers, named entity recognizers (NER), coreference resolution systems, each hand-designed with linguistic theory. Building a question-answering system required composing dozens of these modules.
• •2013: Word2Vec ^[7] showed that simple neural methods for predicting surrounding words (skip-gram) or predicting a word from its context (Continuous Bag-of-Words, CBOW) could learn rich semantic representations from raw text, with no linguistic annotation required.
• •2018: BERT (Bidirectional Encoder Representations from Transformers) ^[8] and GPT (Generative Pre-trained Transformer) demonstrated that pre-training on large text corpora produced representations that transferred to virtually every NLP task, making most hand-designed NLP pipelines obsolete.
• •2020: GPT-3 ^[9] showed that simply scaling up a language model (175B parameters, 300B tokens) produced a system capable of few-shot learning across an enormous range of tasks (translation, arithmetic, coding, reasoning) without any task-specific architecture or fine-tuning.

The entire NLP community's decades of work on parse trees, semantic role labeling, discourse analysis, and grammar engineering was subsumed by "predict the next token, but with more compute."

The timeline below summarizes this relentless transition from domain-specific engineering to compute-scaled general methods across all major AI subfields. The flowchart maps the historical starting points, like heuristic evaluators in chess or phoneme pipelines in speech, to their eventual convergence on search, learning, and scale. This convergence visually illustrates how general methods ultimately dominate:

Why general methods win

The Bitter Lesson isn't just an observation. There's a structural reason why general methods eventually dominate:

💡 Key insight: Analogy: Language immersion vs. dictionary. Learning a language from a dictionary and grammar book (hand-engineering) gives you decent results fast, but you'll always sound wooden and miss subtle idioms. Immersion, simply listening to millions of conversations (learning from data), is painfully slow at first but eventually produces fluency no textbook can match. The more time (compute) you spend immersed, the better you get. That's why learning scales and engineering doesn't.

The scaling argument

Human engineering effort scales linearly (at best) with researcher-hours. You can hire more linguists, but each one still has to think through problems manually. Compute, by contrast, scales exponentially. Moore's Law (and its modern equivalents in GPU throughput) doubles available compute roughly every 18-24 months. We can represent these two different scaling trajectories conceptually with the following relationships, where inputs (human effort versus compute) drive the resulting system performance:

$$\text{Human engineering: performance} \propto \log(\text{researcher\_hours})$$ $$\text{General methods: performance} \propto \text{compute}^\alpha \quad (\text{where } \alpha > 0)$$

Any method that converts compute into performance will eventually surpass any method that relies on human effort, given enough time. The Bitter Lesson says this crossover has happened repeatedly, and it always will.

The complexity argument

Real-world domains are far more complex than we can fully understand. A language has millions of statistical regularities that no linguist can fully enumerate. A visual scene has billions of pixel-level relationships that no feature engineer can capture. The only way to model this complexity is to learn it from data, and learning from data requires compute.

Code: heuristic vs. compute in action

We can demonstrate the Bitter Lesson with a simple Python simulation. The code below defines a game agent evaluating a board state. It takes a board state (with an explicit value and a hidden complexity factor) as input. We implement two approaches: a heuristic function that applies rigid rules, and a general search function that simulates outcomes. The script outputs the evaluation scores, demonstrating how search uncovers hidden complexity while heuristics hit a hard ceiling.

python
1import random
2
3# A simplified state evaluation problem
4class GameState:
5    def __init__(self, value: int, complexity: int = 0):
6        self.value = value
7        self.complexity = complexity # Hidden factors human rules might miss
8
9    def simulate_step(self) -> float:
10        """Simulate one step forward in the game (requires compute)."""
11        # In a real game, this evolves the state based on rules
12        noise = random.gauss(0, 1)
13        return float(self.value + noise + (self.complexity * 0.1))
14
15# 1. The Heuristic Approach: Human-coded rules
16# Fast, cheap, but capped by the engineer's understanding
17def heuristic_evaluate(state: GameState) -> float:
18    score = 0.0
19    # Rule 1: High value is good
20    if state.value > 10:
21        score += 1.0
22    # Rule 2: Even numbers are preferred (arbitrary human intuition)
23    if state.value % 2 == 0:
24        score += 0.5
25    # The heuristic ignores 'complexity' because the human didn't understand it
26    return score
27
28# 2. The General Approach: Search / Simulation
29# Slow, expensive, but accuracy scales with compute budget
30def search_evaluate(state: GameState, compute_budget: int) -> float:
31    total_outcome = 0.0
32    # Spend compute to simulate the future
33    for _ in range(compute_budget):
34        outcome = state.simulate_step()
35        total_outcome += outcome
36    
37    # The result converges to the truth as compute increases
38    return total_outcome / compute_budget
39
40# Demonstration
41state = GameState(value=12, complexity=50) # A state with hidden complexity
42
43print(f"Heuristic Score: {heuristic_evaluate(state):.2f}")
44# Output: 1.50 (Fixed, fast, but inaccurate)
45
46print(f"Search (Low Compute): {search_evaluate(state, compute_budget=10):.2f}")
47# Output: ~16.50 (Noisy)
48
49print(f"Search (High Compute): {search_evaluate(state, compute_budget=10000):.2f}")
50# Output: ~17.00 (Accurate: captures the hidden complexity)

In this example, the heuristic_evaluate function is static. No matter how much GPU power you buy, it won't get better than its hard-coded logic. The search_evaluate function starts worse (it's noisy and slow), but as you increase compute_budget, it discovers the ground truth, including the hidden complexity term that the human engineer missed.

Why the simulation works

The Python simulation illustrates a fundamental truth about system design:

When compute is expensive and limited, the heuristic wins because it's fast. As compute becomes cheap and abundant, the search approach inevitably surpasses it. The flowchart below visualizes this tradeoff, contrasting human engineering with general methods. It shows how rule-based systems hit a performance ceiling bounded by human comprehension, whereas general methods keep improving as available compute rises.

The generalization argument

Hand-engineered systems tend to be brittle: they work well on the cases the engineer anticipated but fail on unexpected inputs. Learned representations, by contrast, generalize across the distribution of training data. A hand-designed NER system might recognize "New York" and "Paris" but miss "Bengaluru" because no engineer thought to add it. A learned system that has seen enough text will handle all of them.

Train-time vs. inference-time compute

The Bitter Lesson points to learning and search as the two methods that scale arbitrarily with compute. Modern AI systems have split these into two distinct phases of computation, both of which follow the Bitter Lesson.

Train-time compute: learning the foundation

During training, compute is spent on learning (optimizing weights via gradient descent). Throwing more FLOPs (Floating Point Operations) at a larger dataset with a larger model predictably reduces the loss.

This phase builds the model's intuitive "System 1" capabilities. A language model learns to predict the next token with high accuracy. However, this relies entirely on the model's forward pass, meaning the computation per token is fixed. A complex reasoning problem gets the exact same amount of compute as a simple greeting.

Inference-time compute: search and reasoning

At inference time, compute is spent on search (exploring possible outputs). This represents "System 2" thinking. Instead of immediately returning the first predicted token, the system can generate multiple possible reasoning paths, evaluate them, and select the best one.

Modern reasoning models use learned search strategies at inference time to explore multiple reasoning paths before committing to an answer. As Sutton noted with AlphaZero, search scales: if you give the model 10 times more inference compute to think before answering, it will find significantly better solutions. The following diagram illustrates how modern systems split compute into two distinct phases. Train-time compute processes massive data through gradient descent to build base weights, while inference-time compute uses tree search and verification to produce refined output.

By scaling both train-time learning and inference-time search, engineers apply the Bitter Lesson at every stage of the pipeline, avoiding hand-crafted reasoning modules in favor of raw computation.

The counter-arguments: when heuristics still matter

The Bitter Lesson is powerful, but it's not the whole story. Several important nuances deserve attention:

Inductive biases as efficient priors

While general methods eventually win, inductive biases serve as efficient initialization. Convolutions encode translation invariance, which is genuinely true of images. Positional encodings capture the sequential nature of language. These biases don't contradict the Bitter Lesson; they accelerate learning within a fixed compute budget. The lesson predicts they'll be replaced, but doesn't say they're useless now.

💡 Key insight: This creates an interesting tension: the Bitter Lesson favors architectures with minimal inductive bias, but the Transformer's dominance isn't just about generality. The Transformer architecture itself contains an inductive bias: its self-attention mechanism is permutation-equivariant and implicitly assumes that any token can interact with any other token. This is well-suited to GPU hardware (dense matrix multiplies) and to domains where long-range dependencies matter. The Transformer's dominance isn't just about generality; it's also about being hardware-aware, mapping efficiently onto modern accelerator architectures (TPUs, GPUs). Architectures that don't map well to hardware (like tree-structured networks or sparse routing) face practical scaling walls even if they're theoretically more expressive.

Sample efficiency

Not every problem has internet-scale data. In medical imaging, robotics, or rare language NLP, data is scarce and compute alone can't compensate. In these domains, carefully designed architectures and data augmentation (forms of human knowledge) still provide critical advantages. The Bitter Lesson implicitly assumes abundant data, which isn't always available.

Safety and alignment

Perhaps the strongest counter-argument: we may not want fully general methods for safety-critical decisions. A learned reward model might "hack" its objective in unexpected ways, finding adversarial loopholes that satisfy the math but fail the intent. A rule-based safety filter, while brittle, is interpretable and verifiable. The alignment community argues that some problems require human-designed constraints, not just more compute. For instance, Constitutional AI^[10] uses a hybrid approach: a set of human-written rules (the constitution) guides a general learning process. This shows that even in safety, the trend is toward using heuristics only to constrain the search space, while letting compute handle the actual optimization.

The algorithm efficiency counter-argument

Algorithmic progress and compute progress are not the same thing. Hernandez and Brown (2020)^[11] provide an important counterpoint: algorithmic improvements have been just as significant as hardware scaling. They found that the compute required to achieve a fixed performance level on ImageNet halved roughly every 16 months, a rate comparable to Moore's Law. This suggests that clever algorithms (better optimizers, attention mechanisms, normalization techniques) contribute as much to progress as raw FLOPs. The Bitter Lesson may overstate compute's primacy while underweighting the compounding effect of algorithmic research. However, defenders of the Bitter Lesson argue that these algorithmic improvements (like the Transformer itself) are general methods. They don't encode domain knowledge; they just make better use of compute.

Compute as a fixed budget

The Bitter Lesson describes long-term trends, but engineers work within fixed budgets. If you have 8 GPUs and a deadline, using domain knowledge to constrain your search space is pragmatic, not foolish. The lesson applies to research trajectories stretching over decades, not to individual project decisions.

Implications for modern AI engineering

The Bitter Lesson isn't just philosophical. It has concrete implications for how we design AI systems today:

Build for scale, not cleverness

Design architectures that benefit from more compute and data, even if they seem "wasteful" at current scales. The Transformer replaced the LSTM not because it was more elegant, but because its parallelizable attention mechanism could use GPU clusters more effectively. Choose the architecture that scales.

Prefer learning over engineering

When you're tempted to add a hand-designed component (a custom tokenizer heuristic, a rule-based post-processor, a domain-specific feature), ask: "Would a general learned component work if I gave it enough data?" If yes, the Bitter Lesson says invest in data collection and compute rather than engineering.

Scaling laws are the bitter lesson, quantified

The empirical scaling laws discovered by Kaplan et al.^[12] and Hoffmann et al.^[13] are the Bitter Lesson expressed mathematically. They demonstrate that language model loss follows a predictable power law with model parameters and training compute. Kaplan et al. showed that the cross-entropy loss $L$ decreases predictably as a power law of model size $N$: $L(N) \approx \left( N_c / N \right)^{\alpha_N}$, where $N_c$ is a constant and $\alpha_N \approx 0.076$. This means that simply making a model bigger reliably produces better performance, exactly the behavior Sutton highlighted. The scaling laws tell you how much compute to throw at a problem; the Bitter Lesson tells you why this brute-force approach succeeds where elegant hand-engineering fails.

Critically, Hoffmann et al.'s Chinchilla scaling laws showed that the allocation of compute matters: earlier models were dramatically over-parameterized relative to their training data. The compute-optimal frontier for a given FLOPs budget $C$ balances model size $N$ and training tokens $D$ roughly as $N \propto C^{0.5}$ and $D \propto C^{0.5}$. This finding shifted the industry from building ever-larger models to training appropriately-sized models on much more data.

Test-time compute is search reborn

Modern reasoning models apply the Bitter Lesson to inference: instead of engineering complex reasoning pipelines, they spend more compute at test time through learned search strategies. This directly parallels AlphaZero's MCTS, converting compute into intelligence through general search, not hand-designed heuristics.

Snell et al. (2024)^[14] formalized this insight, showing that inference-time compute scaling follows its own power laws analogous to training-time scaling. The key trade-off is between "Verifiers" (reward models that score candidate solutions) and "Generators" (models that propose solutions via beam search or sampling). At fixed total compute, the optimal strategy depends on problem difficulty: easy problems benefit more from a single strong generator, while hard problems benefit from generating many candidates and selecting via a verifier. This "search at test time" approach represents Sutton's search principle applied to modern LLMs.

Your clever trick is temporary

If you develop a technique that works by encoding domain knowledge (a specialized attention mask, a domain-specific loss function, a hand-tuned decoding strategy), expect it to be obsoleted by a simpler, more general approach once sufficient compute is available. This doesn't mean the technique is bad. It means you should design it to be removable.

A practical framework

For engineers deciding between general and specialized approaches, consider this decision matrix:

This matrix highlights that the Bitter Lesson is a statement about asymptotic behavior. It tells us where the field is heading over the long term. However, day-to-day engineering requires balancing these long-term trends against immediate constraints.

If you're building a prototype with zero training data, you have no choice but to use heuristics. You might write a complex prompt with a strict set of extraction rules. But as your product matures and you collect a million user interactions, the Bitter Lesson suggests you should transition away from those complex prompts and instead fine-tune a model on that data.

The most successful AI teams design their systems to smoothly bridge this gap. They use heuristics to bootstrap their initial data flywheel, but they build their infrastructure to eventually replace those same heuristics with learning and search once enough data and compute are available.

Key takeaways

1. •

   The Bitter Lesson is a meta-principle: across 70 years and every subfield of AI, general methods that use computation have displaced hand-engineered approaches.

2. •

   Search and learning are the two scalable methods. Both convert compute into performance without requiring human domain knowledge.

3. •

   The pattern is accelerating: the Transformer's dominance across language, vision, audio, and biology is the Bitter Lesson operating in real-time.

4. •

   Nuance matters: inductive biases, sample efficiency, safety constraints, and fixed compute budgets all create situations where human-designed structure still provides value.

5. •

   For engineers, the practical implication is clear: design systems that benefit from scale (more data, more compute, more parameters) because that's where the long-term wins are.

6. •

   Know when you're fighting the lesson: if your system relies on clever tricks that only work at your current scale, start planning for the general alternative.