Scaling Laws & Compute-Optimal Training

The last chapter asked how a trained model spends compute while emitting tokens: greedy, beam, or sampling. Before serving, someone has to make a different allocation: with a fixed training-FLOP budget, how many parameters and training tokens should the run use?

For a code assistant, choose between two runs. One candidate spends the budget on a huge model trained on a modest corpus of source code, issues, and documentation. Another uses a smaller model and feeds it far more data. Predict which one wins before seeing a curve: too little capacity leaves patterns unrepresented, while too little data leaves capacity underused. Either mistake can turn a large training bill into a weaker assistant.

Scaling laws turn that choice into an empirical forecast. In pre-training studies, the usual quality metric is cross-entropy loss, or how surprised the model is by the next token.

Kaplan et al. at OpenAI ^{[1]Reference 1Scaling Laws for Neural Language Modelshttps://arxiv.org/abs/2001.08361} measured influential transformer curves. Hoffmann et al. at DeepMind later found a more balanced split of model size and data for compute-optimal dense training, often called Chinchilla scaling.^{[2]Reference 2Training Compute-Optimal Large Language Models.https://arxiv.org/abs/2203.15556}

Neither fit travels unchanged across objectives, data, architectures, or measurement choices.

Three symbols keep the budget legible:

• Parameters ($N$) are learnable weights. They give a model room to store and compose patterns. Papers count $N$ differently, so check whether embeddings and the final unembedding head are included before comparing ratios.
• Training tokens ($D$) are the subword pieces shown during training. They are opportunities to update those weights, not a direct measure of unique knowledge.
• Compute ($C$) is training work measured in floating-point operations (FLOPs). For a dense transformer, a first planning estimate is $C \approx 6ND$: about $2N$ FLOPs per token for a forward pass and $4N$ for backpropagation. Kaplan used this accounting, while later work counted layer and output costs more carefully.^{[1]Reference 1Scaling Laws for Neural Language Modelshttps://arxiv.org/abs/2001.08361[2]Reference 2Training Compute-Optimal Large Language Models.https://arxiv.org/abs/2203.15556} Use it for sizing, then profile the actual architecture.

Check units before trusting any ratio. A 1B-parameter dense model trained on 20B tokens gives approximately 6 × 10^9 × 20 × 10^9 = 1.2 × 10^20 training FLOPs. Doubling both quantities makes that estimate four times larger, even though each one only doubled.

Double parameters, double data, or both

Start with four runs on the same held-out set. A 1-billion-parameter model sees 10 billion tokens, then three follow-up runs change one resource at a time or both together. Before reading the losses, predict which single-axis change will help more, and whether changing both can beat either change alone.

These rows are a synthetic fixture, not a measurement or the later fitting law. They make the bottlenecks visible: either resource helps, but both together lower loss more than either single change. The gains still taper. A 10x increase in parameters doesn't cut loss by 10x; it multiplies the relevant loss term by a smaller, repeatable factor.

That repeatable factor is the useful part of a power law. On a log-log plot, an idealized power-law term becomes a straight line, so a handful of measured runs can estimate how loss changes with parameters, data, or compute. The forecast answers a planning question such as, "If compute grows 10x, what scale should we try?" It doesn't replace a target-scale run, and later we add a loss floor and let parameters and data interact.

Use the fitted parameter exponent to inspect those diminishing returns. This script changes only the parameter-limited term, so its percentages stop short of total loss.

Your team can either double parameters from 1B to 2B while keeping data at 10B tokens, or keep 1B parameters and double data to 20B tokens. Which option gives lower loss in the table?

Answer

Doubling data gives lower loss in this synthetic table: 2.55 instead of 2.60. Data doesn't always win; parameters and tokens are separate bottlenecks, and scaling studies tell you which one is binding for a budget.

Kaplan scaling laws (2020)

The synthetic runs show why one resource can't stand in for the other. Kaplan's 2020 study ^{[1]Reference 1Scaling Laws for Neural Language Modelshttps://arxiv.org/abs/2001.08361} measured each bottleneck separately for transformer language models. Read the three fits as conditional statements: hold the other resources far enough away, then ask how loss $L$ changes.

The three power laws

1. Scaling with parameters (model size $N$)

Suppose data is plentiful and the model is the limiting resource. If you double $N$, should the parameter-limited part of loss fall by half, stay fixed, or move by a smaller factor? The fitted curve answers that question:

$$L(N) \propto N^{-\alpha_N}$$

Here the parameter-limited term shrinks by a fixed multiplier when $N$ is multiplied. With $\alpha_N \approx 0.076$, multiplying parameters by 10 reduces that term by about 16%, not total loss by 16%. Total loss can still be held up by a data term or a floor, which is why this single-variable fit isn't a complete training plan.

2. Scaling with data (dataset size $D$ in tokens)

Now reverse the bottleneck. If the model is already large enough, should another 10x tokens reduce the data-limited term by 10x or by a smaller factor?

$$L(D) \propto D^{-\alpha_D}$$

Hold model size large enough that data is the bottleneck. More training tokens then lower loss along a predictable curve with $\alpha_D \approx 0.095$. The exponent describes the slope of this regime, not a promise that any new token source has the same value.

3. Scaling with compute-optimal training compute ($C_{\min}$)

Finally, allocate both resources together. If compute is spent efficiently, how should the best achievable loss change as the budget grows?

$$L(C_{\min}) \propto C_{\min}^{-\alpha_C}$$

This third curve uses $C_{\min}$, the estimated minimum compute needed to reach a loss when compute is allocated efficiently. It isn't the curve for an arbitrary run with a poor model/data split. Kaplan's fitted exponent is $\alpha_C \approx 0.050$ under the dense-transformer accounting. Actual work also includes attention kernels, optimizer overhead, activation recomputation, sequence length, and architecture choices such as MoE sparsity.

The three slopes don't tell you to compare $\alpha_N$ and $\alpha_D$ and pick the larger one. They describe different controlled regimes. To choose $N$ and $D$ together, put them on one fixed-compute frontier. Kaplan's frontier favored much larger models and relatively little data.

Kaplan's compute-optimal frontier favored larger models

Under Kaplan's fitted regime, extra compute went mostly into model size, with a smaller share going to new data. That recommendation came from the joint frontier, where a relative increase in parameters moved the best achievable loss more than the same relative increase in data.

GPT-3 made that strategy visible: 175B trainable parameters and 300B training tokens, or about 1.7 tokens per parameter.^{[3]Reference 3Language Models are Few-Shot Learners.https://arxiv.org/abs/2005.14165} The ratio describes what GPT-3 used; it doesn't prove that the ratio was optimal for another corpus or objective.

The compute-optimal frontier

Now hold total compute fixed and move along the choices that spend it efficiently. Kaplan reported:

$$N_{\text{opt}} \propto C^{0.73}, \quad D_{\text{opt}} \propto C^{0.27}$$

The question to test is simple: with 10x more compute, does the fitted plan grow $N$ and $D$ at the same rate? The answer is no. These exponents come from loss as a function of model size and training steps, not from comparing $\alpha_N$ and $\alpha_D$ as standalone power laws. Kaplan also fit a joint early-stopped loss surface:

$$L(N, D) = \left[\left(\frac{N_c}{N}\right)^{\alpha_N / \alpha_D} + \frac{D_c}{D}\right]^{\alpha_D}$$

That $L(N,D)$ form describes overfitting when data is scarce. It isn't the derivation of the $0.73$/$0.27$ allocation. In that frontier, 10x more compute means about 5.4x more parameters but only 1.9x more tokens. Chinchilla later re-fit the allocation and found a much more balanced $0.50$/$0.50$ result.

The small script turns those exponents into multipliers. It evaluates the fitted arithmetic only, so its output isn't a hardware benchmark or a claim about wall-clock speed.

Why did Kaplan-era scaling produce parameter-heavy models?

Answer

Kaplan's fitted compute-optimal frontier said that, for a fixed training compute budget, loss improved most by growing parameters much faster than tokens. That didn't come from comparing the single-variable exponents $\alpha_N$ and $\alpha_D$ directly. It came from Kaplan's compute-efficient fits of loss versus model size and training steps, reported as $N_{\rm opt}\propto C^{0.73}$ and $D_{\rm opt}\propto C^{0.27}$.

Use the dense $6ND$ proxy to compare two published parameter-token configurations. The values expose the accounting; rounded counts and architecture-aware FLOPs still differ.

The Chinchilla reallocation (2022)

Kaplan's frontier made a sharp prediction: with one fixed compute budget, a large model trained briefly should beat a smaller model trained longer. Chinchilla asked whether that prediction survived when data and model size were varied together. Before reading the result, choose the likely winner for a code assistant whose corpus still contains useful, unseen examples.

Hoffmann et al. at DeepMind ^{[2]Reference 2Training Compute-Optimal Large Language Models.https://arxiv.org/abs/2203.15556} trained over 400 models, from 70M to more than 16B parameters and from 5B to 500B tokens. Across three fitting approaches, they found that Kaplan's allocation had undervalued data.

Their predicted 70B Chinchilla run used the same reported training compute as 280B Gopher while seeing four times as many tokens.

The disagreement wasn't evidence for two incompatible kinds of transformer. Porian et al. ^{[4]Reference 4Resolving Discrepancies in Compute-Optimal Scaling of Language Modelshttps://arxiv.org/abs/2406.19146} reproduced Kaplan-style experiments and isolated three setup differences: Kaplan omitted the final decoding-layer computation, used a fixed warmup that was too long for small models, and didn't tune optimizer settings as scale changed.

After those corrections, their fitted frontier moved close to Chinchilla's. They also found that careful learning-rate decay wasn't the central explanation Hoffmann et al. had proposed. Compute accounting and fitting choices can move an exponent before any architecture changes.

Chinchilla-optimal training

Hoffmann et al. report three fitting approaches, not one magic exponent. Each route asks where loss is lowest while keeping training FLOPs fixed:

The ~20 tokens-per-parameter rule tracks Approaches 1 and 2, and it's the ratio used for the 70B Chinchilla run on 1.4T tokens. The $0.46$ number belongs to Approach 3's parametric frontier, not to a separate 20:1 recipe. Keeping those outputs distinct prevents a fitting detail from turning into a false constant.

For planning, the common result is easier to remember: under this fixed training-compute objective, grow model size and data at roughly equal rates. Double compute, then start near $\sqrt{2}\times$ the parameters and $\sqrt{2}\times$ the tokens, before checking your own data and stack.

For Chinchilla-style compute-optimal training, how many tokens does a 70B dense model need?

Answer

Use approximately 20 tokens per parameter: $70B \times 20 = 1.4T$ tokens. This is a fitted rule of thumb for the dense-transformer setup Hoffmann et al. studied, not a universal constant.

Treat 20:1 as a candidate starting point, not a constant.^{[2]Reference 2Training Compute-Optimal Large Language Models.https://arxiv.org/abs/2203.15556} Proxy runs on your corpus tell you whether data or capacity is binding before you commit to a larger run.

The next function does that first-pass arithmetic. It takes a dense training-FLOP budget, substitutes $D=20N$ into $C\approx6ND$, and returns a starting value for $N$ and $D$.

Chinchilla vs. Gopher: a concrete example

The Gopher comparison makes the reallocation concrete. Both rows below use the paper's reported training budget, but they spend it in opposite ways:

The FLOP column is the matched budget reported by Hoffmann et al. It won't equal $6ND$ exactly when you multiply the rounded counts: $6ND$ is a planning approximation, while the paper uses architecture-aware accounting.^{[2]Reference 2Training Compute-Optimal Large Language Models.https://arxiv.org/abs/2203.15556}

On the paper's reported downstream evaluations, Chinchilla beat Gopher despite using one quarter as many parameters and about 4.7x as many tokens. The source also reports lower inference cost and memory for the smaller model.^{[2]Reference 2Training Compute-Optimal Large Language Models.https://arxiv.org/abs/2203.15556} That comparison is bounded by its setup: the models were trained with the paper's TPUv3/TPUv4 and JAX/Haiku stack, and quality came from stated benchmark protocols. It isn't a measured serving-throughput or dollar comparison for your hardware.

Why was Chinchilla also a serving-cost breakthrough?

Answer

Chinchilla matched Gopher's reported training compute with a much smaller model trained on more tokens. It scored better on the reported evaluations, and its lower parameter count implies lower dense FLOPs per inference token under the same proxy. The paper's result isn't a promise about latency, utilization, or dollars on a different serving stack.

Later dense-model token ratios

Published model reports show token-to-parameter ratios, but a ratio alone can't tell you which objective selected it.

Meta reports 15.6T pre-training tokens for the Llama 3 flagship.^{[5]Reference 5The Llama 3 Herd of Models.https://arxiv.org/abs/2407.21783} For 405B, that's about $15.6T / 405B \approx 38.5$ tokens per parameter, above Chinchilla's roughly 20:1 fit.

Meta's own IsoFLOP fit, using its data and a $3.8 \times 10^{25}$ FLOP budget, suggested a 402B model on 16.55T tokens. The 405B / 15.6T run landed close to that forecast.^{[5]Reference 5The Llama 3 Herd of Models.https://arxiv.org/abs/2407.21783} Those facts support a model-size choice under Meta's fitted training objective; they don't show that lifetime inference cost selected the flagship ratio.

The smaller Llama 3 models show overtraining more clearly. If each sees the same reported 15.6T-token corpus, 8B works out to about $1{,}950$ tokens per parameter and 70B to about $223$. Meta reports that its smaller models were trained much longer than compute-optimal and performed better than compute-optimal models at the same inference budget.^{[5]Reference 5The Llama 3 Herd of Models.https://arxiv.org/abs/2407.21783} Treat the ratios as published configurations, then ask whether extra training pays back under your demand forecast. Don't assume every size should sit at 20:1.

You want to train a 10B-parameter model at Chinchilla-style ratios. About how many tokens and FLOPs do you need?

Answer

Tokens: $D \approx 20 \times 10B = 200B$. Training compute: $C \approx 6ND = 6 \times 10^{10} \times 2 \times 10^{11} = 1.2 \times 10^{22}$ FLOPs. An estimate in that range captures the planning rule, not a measured cluster bill.

Beyond Chinchilla: inference-aware scaling

Chinchilla picks the lowest pre-training loss for a fixed training-compute budget. Deployment adds another bill. If two candidates reach the same quality bar, the smaller one requires fewer dense FLOPs per request under the same proxy, even if it consumed more training FLOPs before launch. Whether that saves money depends on serving stack and demand.

That changes the question from “How should one training budget be split?” to “Which model reaches this quality target at the lowest lifetime cost for this demand?” An inference-aware objective counts both the original training run and all future tokens processed while the model serves users.

The inference cost problem

A model that serves enough traffic can accumulate inference FLOPs that rival or exceed its original training run. Sardana et al. ^{[6]Reference 6Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Lawshttps://arxiv.org/abs/2401.00448} extend a Chinchilla-style loss fit with that deployment demand.

Under their modeled demand of roughly 1B requests, the fitted objective favors smaller models trained longer. They also train 47 models, ranging from 150M to 6B parameters, at token-to-parameter ratios up to 10,000. Quality kept improving in that tested range, although the sweep wasn't complete for every size.^{[6]Reference 6Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Lawshttps://arxiv.org/abs/2401.00448}

This is evidence about their architecture, data, and loss proxy, not a universal serving rule.

$$\min_{N, D} \; C_{\text{train}}(N, D) + C_{\text{inference}}(N, T_{\text{served}}) \quad \text{subject to } \quad L(N, D) \le \ell$$

Read the objective from left to right. Training costs scale with model size times data ($6ND$ FLOPs). Inference costs scale with model size times all served tokens ($2N \cdot T_{\text{served}}$) for a dense decoder-only model. Once $T_{\text{served}}$ is large enough, extra training for a smaller model can be repaid by cheaper requests. Here:

$$C_{\text{train}} \approx 6ND, \quad C_{\text{inference}} \approx 2N \cdot T_{\text{served}}$$

Here $T_{\text{served}}$ means total input plus output tokens across the model's deployment lifetime. This is a FLOP proxy. It doesn't specify an accelerator, software stack, batch or sequence distribution, precision, kernel baseline, or correctness test, so it can't establish wall-clock latency, throughput, utilization, or dollars. Those need measurements from the serving stack.

What question does inference-aware scaling ask that Chinchilla doesn't?

Answer

Chinchilla asks how to minimize pre-training loss under a fixed training-compute budget. Inference-aware scaling asks which model minimizes total lifetime cost at a target quality, including both training FLOPs and all future served-token FLOPs.

Concrete break-even example

Suppose two candidates can reach the same pre-training loss $\ell$. Predict before calculating: which candidate should win at low demand, and which one should win after enough requests? The numbers below are an arithmetic illustration, not two measured models from Sardana et al. A real comparison first has to show that both candidates hit the quality target.

• Model A (Chinchilla-style): 70B parameters trained on 1.4T tokens. Training cost ≈ $6 \times 70\text{B} \times 1.4\text{T} \approx 5.88 \times 10^{23}$ FLOPs. Inference cost per token ≈ $2 \times 70\text{B} = 140$B FLOPs/token.
• Model B (inference-aware, smaller + over-trained): 30B parameters trained on ~4T tokens (heavier over-training to match quality). Training cost ≈ $6 \times 30\text{B} \times 4\text{T} \approx 7.2 \times 10^{23}$ FLOPs (slightly higher upfront). Inference cost per token ≈ $2 \times 30\text{B} = 60$B FLOPs/token, less than half of Model A.

The break-even point is where Model B's extra training work has been repaid by its cheaper requests. Under the equal-quality assumption, Model A is cheaper below about 1.65T served tokens; Model B wins above it. This fixture has no hardware, framework, precision, batching, kernel baseline, or correctness measurement, so it demonstrates the accounting trade-off rather than a deployment speedup or dollar saving.

The script below computes the break-even point from the stated FLOP proxy. It measures arithmetic only, with neither model measurement nor equal-quality evidence.

The first script finds the crossing algebraically. Now test both sides of that crossing with the same proxy: at 1T served tokens, training cost still dominates; at 3T, repeated serving makes the smaller model cheaper.

What the inference-aware objective predicts

The arithmetic example gives one crossover. Sardana et al.'s fitted objective generalizes the direction: as expected demand rises, the best point shifts toward fewer parameters and more pre-training tokens at a fixed modeled loss target. That's a prediction under an explicit loss model and demand estimate, not evidence that a particular released model used that objective.

For each candidate, state the quality target and the evidence you can actually measure:

The exact optimum depends on the loss target, architecture, data distribution, inference forecast, and serving stack. Don't infer a training objective from a released model's token-to-parameter ratio alone.

Public-text supply as a planning constraint

More tokens help only while useful tokens are available. Chinchilla and inference-aware fits both assume a data supply that can grow with the planned run. Public human-generated text is finite, and the size of that stock is uncertain.

Villalobos et al. ^{[7]Reference 7Will We Run Out of Data? Limits of LLM Scaling Based on Human-Generated Datahttps://arxiv.org/abs/2211.04325} estimate an effective public-human-text stock of roughly $4\times 10^{14}$ tokens after quality filtering and multi-epoch adjustments. Under their assumed dataset-growth trends, full utilization falls between 2026 and 2032, with a median projection of 2028.

Their assumed 5x over-training policy moves the intersection earlier. Those are scenario forecasts, not observations about the current stock, and the paper's estimate depends on how quality and repeated passes are counted.

Once the data regime changes, $B/D^\beta$ is no longer a safe extrapolation. A team may filter and deduplicate harder, reuse data across controlled epochs, transfer from another domain, or add synthetic data. Each choice can add signal, but each also changes what the fitted $D$ term means.

The next calculation compares one reported dataset with the projected stock. It measures dataset size only, not unique-text consumption or model quality.

In the Villalobos et al. forecast, why does assumed over-training move full utilization earlier?

Answer

A longer-trained model uses a larger token budget per parameter than a Chinchilla-style baseline. Under a fixed effective-stock estimate and continued growth assumptions, larger dataset requirements intersect the projected stock earlier. This is a property of the scenario, not proof that an exhaustion date has occurred.

A new scaling axis: test-time compute

Training scale fixes the weights. Serving still gives you a per-query knob: how much work should the model spend before returning one answer?

A system can generate a longer trace, sample several candidates, or search with a verifier. Snell et al. ^{[8]Reference 8Scaling LLM Test-Time Compute Optimally Can Be More Effective Than Scaling Model Parameters.https://arxiv.org/abs/2408.03314} study those revision and verifier-search strategies in a FLOP-matched evaluation on MATH.

When a smaller base model already solves some items, adaptively allocated test-time compute can beat a roughly 14x larger pretrained model on easier and intermediate items. Harder items favor more pre-training in their comparison. Read that finding as task- and method-specific quality evidence under a stated FLOP budget, not as latency or throughput on a production engine.

DeepSeek-R1 is a related post-training example, not a replacement for pre-training scale. R1-Zero uses reinforcement learning without supervised fine-tuning first; the final R1 pipeline adds cold-start data, supervised fine-tuning, and further RL stages.^{[9]Reference 9DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learninghttps://arxiv.org/abs/2501.12948} Post-training can change how many tokens a deployed model spends per query. It doesn't mean pre-training data or compute stopped mattering.

For sizing, this adds a second trade-off. At one quality target, you might choose a larger or longer-trained model that answers in one pass, or a smaller model that spends more compute on hard queries. More candidates and longer responses also multiply $T_{\text{served}}$, so they feed back into the lifetime-cost objective.

The small calculation below isolates generated-token cost for sampled candidates. It has no accelerator, batching, precision, kernel baseline, or verifier, so its output shows linear FLOP accounting rather than a production performance result. A real request also includes prompt tokens and any verifier work.

Why is test-time compute a separate scaling axis rather than just a larger fixed model?

Answer

Parameters and pre-training tokens are fixed once the model is trained. Test-time compute is chosen per query: the same deployed model can spend more generation or multiple sampled attempts on harder inputs. That creates an additional serving-cost trade-off, while the attainable quality gain still depends on the trained model and task.

When the shortcut breaks

Scaling laws become useful when you know what each shortcut leaves out. Five failure patterns are worth recognizing because each one points to a different experiment.

A ratio isn't a constant

Suppose a 1B model trained on 20B tokens performs poorly on a messy internal-code corpus. The tempting response is to memorize the ratio more firmly and scale the model, but 20:1 was fitted for a particular dense-transformer and general-web-text regime. Data quality, tokenizer design, optimizer settings, repetition, and architecture can move the useful ratio.

Start with 20:1 as a baseline, then run 100M to 1B proxy models on the actual corpus. Fit the trend you observe. That small experiment tells you whether data or capacity is binding before you scale either one by orders of magnitude.

FLOP estimates aren't cluster bills

If a spreadsheet gives two runs the same $C \approx 6ND$ but the cluster bills differ, the estimate did its job as a first check and then reached its limit. The dense rule leaves out attention-kernel details, optimizer state, activation checkpointing, sequence length, utilization, distributed communication, data-pipeline stalls, and sparse routing.

Replace the proxy with measurements from your stack before committing budget: accelerator model and count, framework and kernel versions, token and sequence shapes, batch and concurrency, precision, exact baseline implementation, and a correctness check. Then report wall-clock throughput or dollars per token with those conditions attached. A bare “2x faster” is not evidence another team can reproduce.

Low loss isn't deployed capability

An excellent cross-entropy forecast can still lead to a coding assistant that hallucinates APIs, ignores instructions, or fails safety checks. The curve predicts next-token compression on a training distribution. It doesn't directly measure factuality, downstream tasks, instruction following, alignment, or reasoning.

Reserve budget for instruction tuning, evaluation, and alignment. Instruction tuning and RLHF ^{[10]Reference 10Training Language Models to Follow Instructions with Human Feedback (InstructGPT).https://arxiv.org/abs/2203.02155} can change behaviors that the pre-training loss curve never observes. Keep those checks in the plan even when the loss forecast looks smooth.

Training-optimal isn't lifetime-optimal

A 70B Chinchilla-style model can be the right answer for a training-loss objective and the wrong answer for a heavily used service. Chinchilla spends less upfront compute than a longer-trained small model, while the small model spends fewer FLOPs on every request under the dense proxy.

Estimate served-token demand before choosing a size. Compare $6ND + 2N \cdot T_{\text{served}}$ across candidates that meet a measured quality target, then validate the crossover on the actual serving stack. The formula predicts a direction; it doesn't supply a latency, utilization, or dollar number by itself.

Extrapolation needs a middle rung

A clean line through 100M to 1B proxy runs can still miss badly at 100B. Outside the measured range, optimizer instability, numerical precision, data exhaustion, or distributed communication can introduce a new bottleneck.

Run at least one mid-scale validation before booking the full target. For a 100M to 1B proxy range and a 100B target, a 10B run is a useful check. Treat the extrapolation as a hypothesis, and refit when measured loss leaves its forecast band.

Scaling law breakdowns and limitations

The curve can be smooth while its assumptions change underneath. Scaling laws are empirical fits, not physical laws: architecture, data quality, tokenizer, and optimizer regime can all move the exponents. Carry the measured range and objective with every forecast.

Emergent abilities

Wei et al. ^{[11]Reference 11Emergent Abilities of Large Language Models.https://arxiv.org/abs/2206.07682} highlighted tasks where measured performance stayed near zero and then jumped at larger scales, which looked like a phase transition. Schaeffer et al. ^{[12]Reference 12Are Emergent Abilities of Large Language Models a Mirage?https://arxiv.org/abs/2304.15004} argued that many jumps came from the metric: replacing exact-match thresholds with continuous scores often made the same progress look smoother.

When a benchmark suddenly turns upward, inspect the scoring rule before inventing a new scaling law. Keep the thresholded task result if it matters to users, but pair it with a continuous metric that shows whether the underlying behavior changed gradually.

Task-specific ceilings

Downstream tasks introduce their own ceilings. When pre-training loss keeps falling but factuality, instruction-following, or safety metrics plateau, more pre-training alone isn't a fix. A coding assistant still needs data-quality checks, post-training, evaluation, and alignment after the pre-training budget is chosen. Instruction tuning and RLHF ^{[10]Reference 10Training Language Models to Follow Instructions with Human Feedback (InstructGPT).https://arxiv.org/abs/2203.02155} belong in that later budget; they aren't a free byproduct of a lower $L(N,D)$.

Why can a scaling study predict great loss but still produce a bad coding assistant?

Answer

Scaling laws predict pre-training loss, not factual accuracy, instruction following, domain reliability, or safety behavior. A coding assistant still needs data-quality checks, post-training, evaluation, and alignment work after the pre-training budget is chosen.

Architecture sensitivity

Before applying a dense fit to another architecture, ask what $N$ means for both capacity and per-token work. That shortcut breaks for:

• Mixture-of-Experts (MoE) (lesson): MoE models activate only some parameters for each input, separating total parameter count from per-token compute. A dense fit can't tell you how those two quantities should scale together.
• State Space Models (SSMs): SSMs process sequences recurrently rather than with quadratic attention. Their exponents need independent measurements.
• Hybrid architectures: A model combining attention with SSMs or another mechanism needs a fresh fit for its combined capacity and compute paths.

Dense-transformer curves remain useful context, but they can't choose the allocation for these systems. Measure total and active parameters, data, compute, and quality on the architecture you plan to train.

Why can't you blindly apply dense-transformer scaling laws to MoE models?

Answer

Dense scaling laws treat parameter count as the main size variable. MoE models split total parameters from active parameters per token. Total parameters affect memory and routing; active parameters affect compute. That means you need a separate empirical scaling fit.

From theory to practice: running a scaling study

At this point, the question is operational: how do you turn a curve into a safe decision? A scaling study uses small, controlled proxy runs to forecast a larger run before anyone commits its full budget. The forecast earns trust in stages. If a mid-scale check leaves the forecast band, stop and refit instead of spending through the mismatch.

A complementary tool is µ-Transfer (Tensor Programs V) ^{[13]Reference 13Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transferhttps://arxiv.org/abs/2203.03466}. Under Maximal Update Parametrization (µP), many optimal hyperparameters stay stable as width changes, so a small proxy can tune settings for a larger target. Yang et al. test this on Transformer and ResNet experiments, transferring from 13M parameters to BERT-large and from 40M parameters to a 6.7B GPT-3 model while cutting tuning cost.

Those are results for the tested setups, not a reason to skip mid-scale validation. Hyperparameter transfer, the loss fit, the data mix, and the hardware stack can fail independently.

Fitting the parametric loss function

To see what a scaling study actually fits, use Chinchilla Approach 3 as a small worked surface:^{[2]Reference 2Training Compute-Optimal Large Language Models.https://arxiv.org/abs/2203.15556}

If both model capacity and data are finite, which penalties should remain in the measured loss?

$$L(N, D) = E + \frac{A}{N^{\alpha}} + \frac{B}{D^{\beta}}$$

As $N$ and $D$ grow, the two penalties shrink and the fitted floor $E$ is what remains in this measured regime. $A/N^\alpha$ is the penalty for finite parameters, and $B/D^\beta$ is the penalty for finite training data. $E$ is an extrapolated fit parameter, not a measurement of language's irreducible entropy.

Before fitting noisy runs, practice reading a known surface. The table below is synthetic and was generated from

$$L(N, D) = 1.10 + \frac{2.80}{N^{0.070}} + \frac{7.80}{D^{0.098}}$$

then rounded to three decimals. A real study would estimate all constants from noisy runs. Starting from a known law keeps this arithmetic inspectable.

Read across a row and down a column. More tokens lower loss when $N$ stays fixed; more parameters lower loss when $D$ stays fixed. The three-term formula captures both penalties at once, so a forecast must say which resource changed.

Now test whether a fit can recover its own generating law. The script holds out the 1B / 100B run, searches a coarse grid for $\alpha$ and $\beta$ using the other eight points, and forecasts 70B / 1.4T. $E$, $A$, and $B$ stay fixed so the search fits on one screen. A real fit would estimate all five constants, report uncertainty, and still require a mid-scale check.

The hold-out residual is tiny because the table came from the same law. Real runs add noise and setup drift. Compare the 70B forecast with a mid-scale run using the planned tokenizer, data mix, sequence length, and training stack; stop or refit when measured loss leaves the forecast band. A smooth extrapolation still has to earn the budget.

You fit a loss surface on 100M to 1B proxy runs and forecast 2.091 at 70B / 1.4T. What should you do before booking that run?

Answer

Keep a hold-out residual, then run a mid-scale check on the real tokenizer, mix, and stack. If measured loss leaves the forecast band, refit. Don't treat the 70B number as measured.