The Bitter Lesson & Compute

An incident router is a small service that sends each failure report to an engineering queue. Today it receives: “Decode latency spiked after yesterday's search deploy,” even though the report never says timeout. A keyword rule can cover today's wording, a learner can absorb labeled reports, or a search procedure can spend more compute comparing candidate actions against evidence. Which choice keeps improving when tomorrow's report uses words nobody anticipated?

The last chapter put freshness checks, delayed-label quality checks, and promotion rules around a job-risk model. Those gates still matter. Here the question moves inside them: what should produce a route when the input is messy language?

Rich Sutton's 2019 essay The Bitter Lesson points to a recurring pattern: as computation becomes cheaper, general methods that can keep using it often overtake systems whose core behavior is hand-authored. He names two such methods: learning, which updates behavior from experience, and search, which spends computation comparing possible choices.^{[1]Reference 1The Bitter Lesson.http://www.incompleteideas.net/IncIdeas/BitterLesson.html}

The lab makes that claim measurable. Watch a three-keyword router miss familiar ideas in unfamiliar words, teach a tiny learner, then spend request-time compute on candidate actions. Judge each upgrade by evidence, not by how sophisticated it sounds.

Rules, learning, and search

Keep that report as the running artifact. Yesterday's search deploy is a reranker, a second-stage model that reorders retrieved documents. The router must choose a queue: serving latency, answer quality, or access/auth.

Rules answer only what someone has anticipated. A check such as if "latency" in text: route_to("latency") maps a known phrase to a queue.

Learning fits patterns from labeled issues. It might see TTFT spike (time to first token, the delay before a serving stack emits its first output token), unsupported citation, and OAuth callback failed, then infer route associations from examples.

Search postpones the decision. It can generate several actions, check each against evaluation evidence, and commit to the best-supported route.

Each method puts a different object in human hands:

The table isn't a verdict against deterministic code. Sutton's claim is about which approach compounds as compute becomes available, not a ban on rules. Keep a promotion gate or human approval when an action carries risk. Don't make a growing rulebook carry the messy task's core interpretation.

See a rulebook reach its boundary

Start with three exact keywords: timeout, citation, and password. The six reports below mix those matches with paraphrases. Predict how many route correctly before running the cell.

Read 3/6 as a boundary, not a mystery. Exact phrases route quickly; tokens arrive slowly and can't authenticate contain the same operational ideas but none of the words this map knows.

A faster CPU can evaluate the same conditions sooner. It can't add an unencoded condition.

Patch the map until all six old reports pass, but keep three new phrasings untouched. Ask yourself whether the added entries recognize TTFT spiked today, response used unsupported claim, or OAuth callback fails again.

On the old set, patching works: 6/6. On the new set, it still scores 0/3. Engineer time bought a memory of six reports, not a procedure that can transfer to unseen wording. If branch additions move only the known-set score, the system is accumulating exceptions.

Replace phrases with evidence from examples

Switch the source of behavior from branches to examples. Before tokenization gets its own lesson, use a deliberately crude representation: lowercase word tokens. Each report becomes a set of observed terms, and the training pass counts which route each term appeared with. No synonym handling yet.

Look at citation: no routing branch declared it special. It appears in quality-labeled reports, so the learner carries it forward as evidence. The representation is primitive and exact-match, but knowledge now enters through examples.

Turn those counts into a tiny classifier. Score each label by the counts of words it shares with a new report. The held-out list includes four paraphrased issues and one deliberate abstention, so a correct score includes knowing when evidence is missing. Predict which report should return manual_review before running the next cell.

Don't mistake this toy for an LLM. Its job is to isolate one change: a general procedure can absorb a labeled example without a developer adding a condition for each phrase. When evidence is absent or tied, abstention keeps uncertainty visible for review instead of turning a list order into a production decision.

Why does bad graph query go to manual_review instead of whichever route appears first?

Answer

None of its words provide route evidence in the training examples, so every route ties at zero. A deterministic tie-break would look reproducible while still inventing a production decision. Abstention preserves the uncertainty for review and future labeling.

Now make the abstention useful. Add one reviewed correction, retrain the same procedure, and predict what happens to bad graph query before running the next cell.

The correction moves bad graph query from manual_review to quality, proving only that this learner can absorb one new association. It doesn't prove generalization: the corrected issue is no longer untouched. Keep evaluating on fresh issues, as you did for datasets and training loops. The useful operational change is that feedback becomes data for a future release instead of another permanent branch.

Feedback now has an owner. Resolved outcomes return to the training set only after the route and its policy or evidence checks have been recorded.

Search spends compute after training

Learning changes a release. Search keeps that release fixed and spends compute after the request arrives. Search is one form of test-time compute: propose several actions, check evidence for each, and commit only to the best-supported route.

Suppose a retrieval-augmented generation (RAG) evaluation, where model answers are grounded in retrieved documents, flags citation_precision on run R42. The search procedure can inspect several candidate actions, some with no evidence and one with both the run ID and metric. Predict what happens when the budget exposes only the first one or two candidates.

The next loop keeps candidate generation and verification explicit. It shows when extra request-time work exposes a better candidate and when it only repeats weak ones. Predict what budgets 1, 2, and 4 will select before running the cell.

Budgets 1 and 2 never expose an evidence-backed action, so the verifier keeps the first quality candidate with score 0. Budget 4 reveals trace-and-block with score 2. If candidate generation misses a valid action or the checker rewards the wrong one, extra search makes the mistake more expensive.

Snell and collaborators tested this trade-off on MATH with PaLM 2 models tuned for revision and verification. Their prompt-dependent allocation sometimes beat best-of-$N$ with about $4\times$ less test-time compute; easier questions benefited more from revisions, while harder questions favored more independent search or tree search. In their equal-compute comparison, harder question bins and heavier inference loads often favored more pretraining. That result is conditional on task, model, and checker, not a license to make every request think longer.^{[2]Reference 2Scaling LLM Test-Time Compute Optimally Can Be More Effective Than Scaling Model Parameters.https://arxiv.org/abs/2408.03314}

Training FLOPs and how to split them

The toy router makes train-time and request-time budgets visible. Language-model pretraining puts the same choice on a larger surface: how much training budget goes into parameters or tokens, and how much request-time budget remains for inference.

Kaplan and collaborators measured smooth empirical power-law relationships between cross-entropy loss, model size, dataset size, and training computation for the Transformer models they studied.^{[3]Reference 3Scaling Laws for Neural Language Modelshttps://arxiv.org/abs/2001.08361} Within that experimental regime, more scale produced predictable loss improvements.

One allocation consequence was striking: their compute-efficient fits favored very large models trained on relatively little data, with dataset size growing as $D \sim C^{0.27}$.^{[3]Reference 3Scaling Laws for Neural Language Modelshttps://arxiv.org/abs/2001.08361} Hoffmann later revisited that split.

Before using the formula, anchor it with a concrete plan: 7B parameters trained on 140B tokens. For a dense decoder-only Transformer, the common planning approximation is six floating-point operations per parameter for each training token, so total work is:

$$C \approx 6ND$$

Here, $C$ is total training floating-point operations (FLOPs), $N$ is the parameter count, and $D$ is the training-token count. Kaplan reports the per-token estimate $C \approx 6N$; multiplying by $D$ yields $6ND$. The factor 6 approximates a forward pass plus a backward pass that costs about twice as much. It estimates arithmetic, not wall-clock time, hardware throughput, or model quality.^{[3]Reference 3Scaling Laws for Neural Language Modelshttps://arxiv.org/abs/2001.08361[4]Reference 4Training Compute-Optimal Large Language Models.https://arxiv.org/abs/2203.15556}

The next cell keeps that arithmetic visible for two plans and prints zettaFLOPs. It doesn't model device throughput. Predict how the number changes when both parameter and token counts grow.

Arithmetic scales with the parameter-token product, so the larger plan uses 49 times more estimated training work. That still says nothing about whether its data or model size is the better use of a fixed budget.

In fits from over 400 models, Hoffmann and collaborators found that parameter count and training-token count should scale together under their compute-optimal fits: doubling one calls for roughly doubling the other, near $N \propto C^{0.50}$ and $D \propto C^{0.50}$, against Kaplan's $N \propto C^{0.73}$ and $D \propto C^{0.27}$.^{[4]Reference 4Training Compute-Optimal Large Language Models.https://arxiv.org/abs/2203.15556} They tested the revised split by training Chinchilla, a 70B-parameter model on 1.4T tokens, at the same training-FLOP budget as Gopher (280B). Chinchilla outperformed the larger Gopher model on the downstream tasks they reported.^{[4]Reference 4Training Compute-Optimal Large Language Models.https://arxiv.org/abs/2203.15556}

Now hold budget equal to the 7B/140B plan. The next cell changes only $N$ and reports how many tokens remain, so it exposes an arithmetic tradeoff without claiming model quality. Predict the 14B row before running it.

At 14B, affordable tokens fall to 70B. Holding the parameter-token product fixed makes the larger model spend its budget faster. Hoffmann's training-optimal fit says this kind of data starvation is usually poor; Kaplan's earlier fit made a different recommendation. Those exponents don't decide a serving objective.

Llama 3 supplies a serving-oriented counterweight. Its 405B flagship was pretrained on 15.6T tokens, and its authors describe that size as approximately compute-optimal for training. They trained smaller models much longer than training-optimal because those models performed better at the same inference budget.^{[5]Reference 5The Llama 3 Herd of Models.https://arxiv.org/abs/2407.21783}

Training-optimal and serving-optimal answer different questions. A research team must ask how to allocate the budget, what evaluation will reveal, and which bottleneck an extra unit of compute addresses: undertrained weights, too little data, or too little search at request time.

History is evidence, not a slogan

Now pressure-test the pattern against history. Sutton's examples support a direction; they don't support every stronger claim people attach to it.

The language-model row marks a boundary. Internet text carries human-produced knowledge, so next-token training can be a scalable general procedure and still depend on human data. Sutton's later view is that LLMs scale with compute up to the limits of internet data, while systems that learn from their own experience might scale further.^{[7]Reference 7Richard Sutton on whether LLMs are a dead end (Dwarkesh Podcast)https://www.dwarkesh.com/p/richard-sutton}

Use that caveat as a design constraint, not a dismissal. Prefer methods that can absorb more evidence and compute, then test whether the extra budget addresses your actual bottleneck.

Keep rules where they belong

Put the human-written rule where it protects an action, not where it has to guess every intent. In an AI release system, a learned component can suggest promote; an explicit policy can require review when an eval gate fails or evidence is missing.

Classify a request along two axes before looking at the matrix: how ambiguous is its meaning, and how costly is a wrong action? Deterministic lookup fits structured, low-risk facts. Free-form, high-consequence actions usually need both learned interpretation and an explicit gate.

The policy gate is deliberately boring. Predict three outputs: a high-scoring promotion with evidence, a low-scoring promotion with evidence, and a high-scoring promotion without evidence.

Read these outputs as ownership boundaries. The model proposes a route; the gate checks consequences and evidence; human review owns the exception. Learning handles varied phrasing, while the rule enforces action policy.

Before leaving the lab, write a receipt. This fixture records a small experiment, not a launch approval. A compute decision isn't ready for production review until it names the evaluated split, training procedure, inference budget, verifier, policy gates, and escalation monitor.

Turn the lesson into a design test

The Bitter Lesson isn't “throw compute at every problem.” Use four questions to audit any compute decision:

1. Can fresh labeled evidence improve capability without a new handwritten branch?
2. Can extra search or verification budget be measured on held-out work?
3. Do policy gates protect an action rather than encode every possible meaning?
4. Can the team report the budget and evaluation that justified it?

One boundary remains. Our learner treated lowercase, whitespace-separated words as the units worth learning. Tokenization is where you can ask the same question again: which reusable pieces emerge from corpus counts, and which mistakes become permanent in the vocabulary?

Mastery check

Checkpoints

What exactly does Sutton name as the two general methods that can make use of large amounts of computation?

Answer

Search and learning. Learning uses computation to improve a model from data or experience; search uses computation while choosing among possible actions or outputs.

Why doesn't a promotion approval threshold contradict the Bitter Lesson?

Answer

A threshold can enforce a policy boundary around a learned component. It doesn't try to recognize every way a teammate can ask to ship a candidate. The brittle pattern is using growing capability rules instead of learning from evaluated examples.

What does $C \approx 6ND$ tell you, and what can't it tell you?

Answer

It gives a rough dense-Transformer training FLOPs estimate from parameter count $N$ and training tokens $D$, with the factor 6 approximating a forward pass plus a more expensive backward pass. It can't tell you model quality, sustained hardware speed, data cleanliness, or which allocation is best without experiments or an empirical scaling model.

Kaplan's compute-efficient fits and Hoffmann's compute-optimal fits disagree about how to split a training budget. What changed, and why might a later production run ignore both?

Answer

Kaplan recommended scaling parameters much faster than data ($D \sim C^{0.27}$). Hoffmann's fits, from over 400 models, said $N$ and $D$ should grow together, and Chinchilla (70B, 1.4T tokens) beat a larger Gopher trained at the same FLOP budget. A serving-heavy lab may still train a smaller model much longer than training-optimal, as Llama 3 did, because inference cost favors the smaller model.

When can extra inference-time search hurt instead of help?

Answer

When candidate generation misses valid actions, the verifier rewards incorrect outputs, or additional latency and cost outweigh measured quality gains. Search needs evaluation and a trustworthy checker.

Evaluation rubric

• Foundational: State Sutton's thesis and distinguish learning from search.
• Practical: Run the incident-router examples and explain why a correction is data for a learner but rule debt for a keyword router.
• Quantitative: Estimate dense-model training FLOPs, compare model/token allocations under a fixed budget, and separate Kaplan, Hoffmann, and serving-optimal splits.
• Production: Design one learned component, one policy gate, and one logged metric for an automated AI-operations workflow.

Common pitfalls

"Compute will fix bad data"

• Symptom: You enlarge a model or sample more candidates while labels leak or contradict one another.
• Cause: General methods can amplify the data and objectives they're given.
• Fix: Keep the dataset-quality and held-out evaluation discipline from the earlier data and production-ML lessons.

"Rules are forbidden"

• Symptom: You let a learned router auto-promote low-score candidates without a policy boundary.
• Cause: You've confused scalable capability with unconstrained execution.
• Fix: Use rules for auditable gates and learning for messy recognition.

"More search always improves an answer"

• Symptom: Cost and latency rise, but accepted answers don't improve.
• Cause: Candidate generation or verification is too weak for added budget to help.
• Fix: Evaluate budget levels on held-out tasks and stop spending where gains vanish.

"A deterministic tie-break is safe"

• Symptom: An unseen issue is sent to the first route in a sorted label list.
• Cause: The classifier turns missing evidence into a reproducible but unsupported action.
• Fix: Abstain when no route has positive evidence or when several routes share the best score.

"Chinchilla's ratio is a law"

• Symptom: You freeze an equal parameter/token split and ignore serving cost, data quality, and later scaling evidence.
• Cause: You've treated one compute-optimal fit as a universal constant.
• Fix: Use Hoffmann's split as a training-budget hypothesis, then measure. Serving can justify training a smaller model longer, as Llama 3 did.