The Bitter Lesson & Compute

You just built a conventional ML product path: time-safe features, deployed predictive models, and monitoring gates that preserve held-out evidence and an audit trail. Now comes a harder design question: when your system is wrong, should you write another rule, learn from more examples, or spend more compute searching among possible answers?

Rich Sutton's 2019 essay The Bitter Lesson gives a demanding answer. Across long stretches of AI history, he argues, methods that can use increasing computation eventually outperform systems built around expert-written knowledge. He names two broadly scalable methods: learning, which improves a model from experience, and search, which explores choices before acting.^[1]

This lesson won't treat "more compute" as magic. You'll build a tiny ShopFlow ticket router, see when rules are useful, measure where they fail, and decide what additional budget can honestly buy.

Where effort compounds

Suppose Alex writes ticket #48291: "My FastShip parcel still hasn't arrived. I need my money back." A support system must choose a route: shipping investigation, refund review, or account help.

A rule can say if "refund" in text: route_to("refund"). That works on wording the author anticipated. A learner can fit patterns from labeled tickets such as money back, damaged return, and delivery delayed. A search step can generate several possible actions and check each one against policy evidence before committing.

Those are different ways to spend effort:

Sutton's thesis concerns the long-run direction of research, not a ban on rules. A refund cap or a required human approval can be exactly the right safety boundary. The warning is about making a growing rulebook carry the core intelligence of a messy task.

See a rulebook reach its boundary

Start with a router that a developer can ship in an afternoon. Three keywords cover obvious cases.

The router isn't foolish. It catches exact, high-signal phrases quickly. Its limit is visible: compute won't make those three conditions recognize money back or log in. A human must anticipate and encode every expansion.

Now add rules that fix today's misses, then test tomorrow's wording.

The second result is the treadmill Sutton warns about. More engineer-hours repair yesterday's errors but don't automatically create a method that adapts to tomorrow's distribution.

Replace phrases with evidence from examples

A learned system needs a representation. Before tokenization becomes a full lesson, use the smallest representation possible: lowercase words. A ticket becomes a set of observed terms, and training records which route each term appeared with.

That representation is primitive, but its source of knowledge is different from the rulebook. No engineer chose that return or courier should indicate a route. Labeled examples supplied that association.

Use those counts as a tiny learned classifier: score each route by how many words it shares with a new ticket.

Don't confuse this tiny word-overlap model with an LLM. Its role is to make the principle observable: a general learning procedure can absorb new labeled examples without a developer adding a condition for each phrase.

The correction alone doesn't prove generalization. You'd still evaluate on untouched tickets, just as you did for datasets and training loops. It does show a useful operational property: product feedback can become evidence for the next training run instead of another permanent branch.

Search spends compute after training

Learning isn't the only scalable method in Sutton's essay. Search spends computation at decision time. For a support agent, search might mean proposing multiple actions, retrieving evidence for each, and sending the best supported route forward.

This next miniature isn't a language model. It's a transparent candidate-and-verifier loop that lets you see why additional inference budget helps only when better candidates and a meaningful checker exist.

The first candidate is cheap and weak. With four candidates, the verifier can choose an action tied to carrier and escalation evidence. If every candidate were wrong, or if the verifier rewarded fast refunds without policy support, extra search would amplify error instead.

Snell and collaborators study this question for LLM reasoning: their experiments find that test-time strategies depend on prompt difficulty, and an adaptive allocation can use inference computation more efficiently than a fixed best-of-$N$ baseline on their evaluated tasks.^[2] That result supports a conditional claim, not "thinking longer always works."

Connect the lesson to language-model training

Language models give learning an unusually clean objective: predict the next token. Kaplan and collaborators measured smooth empirical power-law relationships between cross-entropy loss and model size, dataset size, and training computation for the Transformer language models they studied.^[3] That's evidence that a general learning objective can turn additional scale into predictable improvement within an experimental regime.

For a dense decoder-only Transformer, engineers often estimate training compute with:

$$C \approx 6ND$$

Here, $C$ is training floating-point operations (FLOPs), $N$ is the number of model parameters, and $D$ is the number of training tokens. The factor 6 is a planning approximation for forward and backward passes, not a promise about hardware throughput or model quality.

More compute isn't the same as better allocation. Hoffmann and collaborators trained more than 400 models and reported that, under their compute-optimal fits, parameter count and training-token count should scale together: doubling one calls for roughly doubling the other.^[4]

Use the FLOPs approximation to expose the tradeoff under one fixed budget. The following calculation doesn't select the best model; it only tells you how many tokens each model size can afford at that budget.

That calculation matters because "make the model bigger" can consume the budget needed to expose it to sufficient data. A research scientist doesn't ask only how much compute is available. They ask how to allocate it, what evaluation will reveal, and which failure mode an extra unit of compute addresses.

History is evidence, not a slogan

Sutton's essay supports a broad historical pattern. A careful reader should also notice where the evidence stops.

That last distinction is important. Internet text contains human-written knowledge. Training on more of it can be a powerful general procedure while still relying on human-produced data. Treat the Bitter Lesson as a research heuristic: prefer methods that can absorb more evidence and compute, then demand measurement.

Keep rules where they belong

Rules are valuable when they encode a contractual boundary instead of trying to recognize every possible user intent. For ShopFlow, the model can suggest a refund route, but policy can require review for a high-value order or missing evidence.

The rule here isn't pretending to understand language. It protects an action with business and safety consequences. Learning handles messy phrasing; the gate controls what the learned component may execute.

Finally, leave a receipt. A system that consumes more compute but can't report its evaluation score, inference budget, policy gates, and escalation rate isn't ready for a production experiment.

What to carry forward

The Bitter Lesson isn't "throw compute at every problem." It's a test for your design:

1. Can new labeled evidence improve the capability without new handwritten branches?
2. Can additional search or verification budget be measured against a held-out task?
3. Are policy gates protecting actions rather than trying to encode every meaning?
4. Can you report the compute budget and the evaluation that justified it?

Language models begin by converting raw text into discrete pieces. Tokenizers are a perfect next place to ask the same question: which reusable units can be learned from data, and what mistakes are introduced at that boundary?

Mastery check

Checkpoints

Evaluation rubric

• Foundational: State Sutton's thesis and distinguish learning from search.
• Practical: Run the ticket-router examples and explain why a correction is data for a learner but rule debt for a keyword router.
• Quantitative: Estimate dense-model training FLOPs and compare model/token allocations under a fixed budget.
• Production: Design one learned component, one policy gate, and one logged metric for an automated support workflow.

Common pitfalls

"Compute will fix bad data"

Symptom: You enlarge a model or sample more candidates while labels leak or contradict one another. Cause: General methods can amplify the data and objectives they're given. Fix: Keep the dataset-quality and held-out evaluation discipline from the earlier data and production-ML lessons.

"Rules are forbidden"

Symptom: You let a learned router auto-approve high-value refunds without a policy boundary. Cause: You've confused scalable capability with unconstrained execution. Fix: Use rules for auditable gates and learning for messy recognition.

"More search always improves an answer"

Symptom: Cost and latency rise, but accepted answers don't improve. Cause: Candidate generation or verification is too weak for added budget to help. Fix: Evaluate budget levels on held-out tasks and stop spending where gains vanish.