Scaling Laws & Compute-Optimal Training

After decoding, we move from "how a trained model emits tokens" to "how a lab decides what model to train in the first place." Scaling laws describe how model loss changes as you add parameters, tokens, and compute. The practical question behind the math is how to spend a fixed training budget without overbuilding the wrong part of the system.

A fixed training budget for a code-assistant model creates a simple-sounding choice: build a huge model and train it on a modest amount of source, issue, and documentation data, or build a smaller model and train it on much more data. Get this balance wrong, and you can spend millions of dollars on a model that underperforms a smaller, better-trained alternative.

This is the problem scaling laws address. They are empirical relationships that predict how a model's performance improves as you increase its size, its training data, or both. Kaplan et al. at OpenAI ^{[1]Reference 1Scaling Laws for Neural Language Modelshttps://arxiv.org/abs/2001.08361} quantified influential transformer scaling curves, and the Chinchilla paper ^{[2]Reference 2Training Compute-Optimal Large Language Models.https://arxiv.org/abs/2203.15556} later found a more balanced allocation of model size and data for compute-optimal dense training. These fits help plan expensive runs, but each fit applies only within its objective, data, architecture, and measurement assumptions.

Before going further, define three basic terms used throughout:

• Parameters (N): The learnable numbers inside a model (weights and biases). They set how much structure the model can store and compose.
• Training tokens (D): The individual words or subword pieces the model sees during training. They provide the examples the model learns from.
• Compute (C): The total number of floating-point operations (FLOPs) needed for training. For a standard dense transformer, a useful rule of thumb is $C \approx 6ND$ (approximately 2N FLOPs per token for the forward pass and 4N for backpropagation). This approximation originates in Kaplan et al. and is refined in the Chinchilla work.^{[1]Reference 1Scaling Laws for Neural Language Modelshttps://arxiv.org/abs/2001.08361[2]Reference 2Training Compute-Optimal Large Language Models.https://arxiv.org/abs/2203.15556}

Why scaling laws matter

Frontier pre-training runs consume large compute budgets. In the early days of deep learning, architecture design and hyperparameter tuning drove many performance improvements. Today, scale is one of the main levers. But scaling isn't as simple as making everything bigger. If you misallocate your compute budget between model size and data volume, you can spend a large budget on a model that underperforms a smaller, better-trained counterpart.

Scaling laws address this problem by providing predictive models that guide resource allocation. They map the relationship between the scale of your inputs (parameters, tokens, and compute) and the expected performance of the output model. In pre-training studies, that performance is usually cross-entropy loss, which measures how well the model predicts the next token. By studying these relationships, engineers can estimate how a model will perform before committing to a costly, months-long training run.

These empirical laws allow teams to:

• Forecast performance before committing to a full training run.
• Optimize compute allocation between parameters and training tokens.
• Make economic trade-offs between training cost and inference cost in production environments.
• Extrapolate from small-scale proxy experiments to production-scale models, reducing the risk of expensive surprises at scale.

They are planning tools, not guarantees. A good scaling study reduces uncertainty before a large run, but the final decision still has to account for data quality, architecture, hardware efficiency, post-training, and deployment cost.

Building intuition with numbers

Before any formulas, a tiny concrete example shows the shape of scaling. Suppose you train a 1-billion-parameter model on 10 billion tokens and measure its cross-entropy loss. Then you run three follow-up experiments:

These numbers are synthetic, but they capture a real pattern: doubling parameters or data lowers loss, but doubling both together lowers it more than either alone. The improvement is smooth and predictable, but it follows diminishing returns. A 10x increase in parameters doesn't cut loss by 10x; it cuts it by a smaller, fixed multiplier.

This predictable curve is a power law. In everyday language, an idealized power-law term becomes a straight line when you plot its logarithm against log(parameters) or log(data). That line estimates the scaling trend. It lets engineers answer questions like: "If I have 10x more compute, how much better will my model get?" without training the model first. Later sections add an asymptotic floor and fit parameters and data together.

Your team can either double parameters from 1B to 2B while keeping data at 10B tokens, or keep 1B parameters and double data to 20B tokens. Which option gives lower loss in the table?

Answer

Doubling data gives lower loss in this synthetic table: 2.55 instead of 2.60. Data doesn't always win; parameters and tokens are separate bottlenecks, and scaling studies tell you which one is binding for a budget.

Kaplan scaling laws (2020)

OpenAI's 2020 work ^{[1]Reference 1Scaling Laws for Neural Language Modelshttps://arxiv.org/abs/2001.08361} established three power-law relationships for transformer language models. Each describes a different bottleneck regime for loss $L$:

The three power laws

1. Scaling with parameters (model size $N$)

$$L(N) \propto N^{-\alpha_N}$$

Reading the formula: In the parameter-limited Kaplan form, the parameter-limited loss term decreases as a power law when you make the model bigger. Double the parameters, and that term drops by a fixed multiplier. The exponent $\alpha_N \approx 0.076$ means the improvement is smooth and predictable, but diminishing returns are steep: a 10x increase in parameters reduces the parameter-limited term by about 16% (not total loss by 16%). When you use a joint parametric form later (for example Chinchilla's $E + A N^{-\alpha} + B D^{-\beta}$), total $L$ also has a floor and a data-limited term.

2. Scaling with data (dataset size $D$ in tokens)

$$L(D) \propto D^{-\alpha_D}$$

Reading the formula: The same pattern holds for data. More training tokens means lower loss, following a predictable power curve with $\alpha_D \approx 0.095$.

3. Scaling with compute-optimal training compute ($C_{\min}$)

$$L(C_{\min}) \propto C_{\min}^{-\alpha_C}$$

Reading the formula: Kaplan's compute curve is written in terms of optimally allocated training compute, not arbitrary training runs. Using the standard dense-transformer budgeting rule $C \approx 6ND$, the fitted exponent is $\alpha_C \approx 0.050$. The factor of 6 comes from a rough FLOP accounting: about $2N$ per token for the forward pass and about $4N$ per token for backpropagation. It's a planning heuristic, not an exact profiler. Real training runs also depend on attention kernels, optimizer overhead, activation recomputation, and architecture choices such as MoE sparsity.

Kaplan's influential result wasn't that $\alpha_N$ was larger than $\alpha_D$; it wasn't. The operational takeaway came from the compute-optimal frontier: for a fixed compute budget, the fitted optimum favored much larger models and relatively little data.

Kaplan's compute-optimal frontier favored larger models

Kaplan's team concluded that, under their fitted scaling regime, you should spend most new compute on model size and only a smaller fraction on additional data. In that fitted regime, adding parameters moved the compute-optimal frontier more than adding the same relative amount of data.

This led to the initial strategy of training very large models on relatively modest data budgets. GPT-3 made that pattern visible: 175B parameters on roughly 300B tokens, a 1.7:1 ratio.^{[3]Reference 3Language Models are Few-Shot Learners.https://arxiv.org/abs/2005.14165}

The compute-optimal frontier

When compute is the binding constraint, Kaplan suggested allocating the budget such that:

$$N_{\text{opt}} \propto C^{0.73}, \quad D_{\text{opt}} \propto C^{0.27}$$

Where these exponents come from: Kaplan didn't get these exponents by comparing $\alpha_N$ and $\alpha_D$ directly. He first fit a joint loss surface,

$$L(N, D) = \left[\left(\frac{N_c}{N}\right)^{\alpha_N / \alpha_D} + \frac{D_c}{D}\right]^{\alpha_D}$$

and then solved for the best allocation under the transformer training-cost approximation $C \approx 6ND$. That optimization produced the strongly parameter-heavy split above. Chinchilla later re-fit the compute-optimal frontier and got the much more balanced $0.50$/$0.50$ result.

Kaplan-style scaling: If you get 10x more compute, spend most of it on a bigger model (about 5x bigger) and only a little more data (about 2x more). This led to the strategy of building very large models on modest datasets.

Why did Kaplan-era scaling produce parameter-heavy models?

Answer

Kaplan's fitted compute-optimal frontier said that, for a fixed training compute budget, loss improved most by growing parameters much faster than tokens. That didn't come from comparing single-variable exponents directly; it came from fitting a joint loss surface and optimizing it under the $C \approx 6ND$ compute rule.

The Chinchilla reallocation (2022)

Two training teams get the same compute budget. Team A trains a much larger model on a thin token budget. Team B trains a smaller model on a much richer token budget. Which system performs better? DeepMind ran this experiment with AI models, and the results changed how teams thought about model size.

Hoffmann et al. at DeepMind ^{[2]Reference 2Training Compute-Optimal Large Language Models.https://arxiv.org/abs/2203.15556} challenged the Kaplan prescription by training over 400 models ranging from 70M to over 16B parameters on 5B to 500B tokens. Their central finding was that the Kaplan allocation had underestimated the value of data.

Why did two careful studies disagree so sharply? Later analysis traced the gap to methodology rather than a new law of transformers. Porian et al. ^{[4]Reference 4Resolving Discrepancies in Compute-Optimal Scaling of Language Modelshttps://arxiv.org/abs/2406.19146} reproduce Kaplan-style scaling experiments and identify three contributors: omitting final decoding-layer computation, an excessively long fixed warmup for small models, and optimizer hyperparameters that weren't tuned as a function of scale. Correcting those factors produces close agreement with the Chinchilla allocation; their experiments also find that tailored learning-rate decay isn't the central explanation. A scaling exponent is only as trustworthy as its compute accounting and fitting procedure.

Chinchilla-optimal training

Hoffmann et al. report multiple fitting approaches, not one unique law. A common classroom summary is equal scaling of parameters and tokens under fixed training FLOPs:

$$N_{\text{opt}} \propto C^{0.50}, \quad D_{\text{opt}} \propto C^{0.50}$$

That $0.50$/$0.50$ form is the practical Approach 3-style story used for the ~20 tokens/parameter rule of thumb. Other IsoFLOP-style fits in the same paper can yield slightly unbalanced exponents (often quoted near ~0.46 / ~0.54). When someone says "Chinchilla says 0.46," they are usually citing a different approach from the equal-allocation summary, not proving the 20:1 recipe wrong.

In plain terms: the correction people act on is that model size and data should grow at roughly equal rates under a fixed training-compute budget. If you double compute, make the model about $\sqrt{2}\times$ bigger and use about $\sqrt{2}\times$ more data. The practical rule:

For Chinchilla-style compute-optimal training, how many tokens does a 70B dense model need?

Answer

Use approximately 20 tokens per parameter: $70B \times 20 = 1.4T$ tokens. This is a fitted rule of thumb for the dense-transformer setup Hoffmann et al. studied, not a universal constant.

That 20:1 ratio is a fitted result for the dense-transformer setup Hoffmann et al. studied, not a universal constant.^{[2]Reference 2Training Compute-Optimal Large Language Models.https://arxiv.org/abs/2203.15556}

Chinchilla vs. Gopher: a concrete example

DeepMind demonstrated this dramatically with Gopher and Chinchilla:

The FLOP column is the matched training budget reported by Hoffmann et al. It won't equal $6ND$ exactly if you plug in the displayed parameter and token counts: $6ND$ is a planning approximation, the paper uses architecture-aware FLOP accounting, and the displayed counts are rounded.^{[2]Reference 2Training Compute-Optimal Large Language Models.https://arxiv.org/abs/2203.15556}

With the same reported training budget, Chinchilla outperformed Gopher across the evaluation suite DeepMind reported. The 4x smaller model trained on about 4.7x more data delivered better downstream accuracy while also being cheaper to fine-tune and serve.^{[2]Reference 2Training Compute-Optimal Large Language Models.https://arxiv.org/abs/2203.15556}

Why was Chinchilla also a serving-cost breakthrough?

Answer

Chinchilla matched Gopher's reported training compute with a much smaller model trained on more tokens. The smaller model was better on the reported evaluations and cheaper to serve per token because dense inference cost scales roughly with parameter count.

Later dense-model token ratios

Published model reports make it possible to observe the token-to-parameter ratio used. That observation doesn't by itself reveal which objective selected the configuration.

Meta reports that the Llama 3 family was pre-trained on 15.6T text tokens across sizes.^{[5]Reference 5The Llama 3 Herd of Models.https://arxiv.org/abs/2407.21783} For the 405B flagship that is about $15.6T / 405B \approx 38.5$ tokens per parameter, above Chinchilla's roughly 20:1 fitted rule. The paper also says 405B is approximately compute-optimal under scaling laws fitted on Meta's data and its $3.8 \times 10^{25}$ FLOP training budget.^{[5]Reference 5The Llama 3 Herd of Models.https://arxiv.org/abs/2407.21783} Those facts don't establish lifetime inference cost as the cause of the ratio.

The same 15.6T corpus is the clearer overtraining story for the smaller members: Llama 3 8B sees about $15.6T / 8B \approx 1{,}950$ tokens per parameter, and 70B about $15.6T / 70B \approx 223$ tokens per parameter. Those ratios are far above Chinchilla-style compute-optimal training for the same sizes. Teams often do this deliberately so a small, cheap-to-serve model keeps improving on extra unique tokens after a larger sibling would have exhausted the compute-optimal path. Treat those numbers as published configuration facts, then evaluate them with the lifetime-cost objective in the next section rather than assuming every size should sit at 20:1.

You want to train a 10B-parameter model at Chinchilla-style ratios. About how many tokens and FLOPs do you need?

Answer

Tokens: $D \approx 20 \times 10B = 200B$. Training compute: $C \approx 6ND = 6 \times 10^{10} \times 2 \times 10^{11} = 1.2 \times 10^{22}$ FLOPs. If your estimate is in that ballpark, you have the main rule of thumb.

Beyond Chinchilla: inference-aware scaling

Chinchilla tells you how to train the model with the lowest pre-training loss for a fixed training-compute budget, but that's only half the story. If two models meet the same quality bar, the smaller one may be cheaper to serve on every request. The relevant economic objective becomes lifetime cost, not pre-training loss at the end of the run alone.

Kaplan and Chinchilla both ask: "Given a fixed pre-training compute budget, how should I split it between parameters and tokens?" Inference-aware scaling changes the objective: "Given a target quality and expected demand, which model minimizes total lifetime cost?" Chinchilla-optimal minimizes training loss for a fixed training compute budget. If an operator instead optimizes total deployment-lifetime cost, the objective includes training cost plus inference cost over the model's deployment lifetime.

The inference cost problem

A model that serves enough traffic can accumulate inference FLOPs that rival or exceed the original training cost. Sardana et al. ^{[6]Reference 6Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Lawshttps://arxiv.org/abs/2401.00448} extend a Chinchilla-style objective to account for deployment demand.

Under their fitted objective and demand assumptions, Sardana et al. report that researchers expecting roughly 1B requests should prefer smaller models trained for longer. They validate the analysis with 47 trained models and report continued improvement at token-to-parameter ratios as high as 10,000 in the measured regime.^{[6]Reference 6Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Lawshttps://arxiv.org/abs/2401.00448}

$$\min_{N, D} \; C_{\text{train}}(N, D) + C_{\text{inference}}(N, T_{\text{served}}) \quad \text{subject to } \quad L(N, D) \le \ell$$

Reading the formula: Training costs scale with model size times data ($6ND$ FLOPs). Inference costs scale with model size times total processed inference tokens ($2N \cdot T_{\text{served}}$) for a dense decoder-only model. Once $T_{\text{served}}$ gets large enough, a smaller model that's trained longer can match the target quality while costing less over its lifetime. Here:

$$C_{\text{train}} \approx 6ND, \quad C_{\text{inference}} \approx 2N \cdot T_{\text{served}}$$

Here $T_{\text{served}}$ represents total input plus output tokens processed across inference requests during the model's deployment lifetime. The proxy ignores differences in hardware utilization, latency, and attention cost, so production sizing still needs measurements from the serving stack.

What question does inference-aware scaling ask that Chinchilla doesn't?

Answer

Chinchilla asks how to minimize pre-training loss under a fixed training-compute budget. Inference-aware scaling asks which model minimizes total lifetime cost at a target quality, including both training FLOPs and all future served-token FLOPs.

Concrete break-even example

Suppose you target a fixed pre-training loss $\ell$ that two candidate models can achieve. The numbers below are an arithmetic illustration, not two measured models from Sardana et al.; a real comparison first has to show that both candidates hit the quality target.

• Model A (Chinchilla-style): 70B parameters trained on 1.4T tokens. Training cost ≈ $6 \times 70\text{B} \times 1.4\text{T} \approx 5.88 \times 10^{23}$ FLOPs. Inference cost per token ≈ $2 \times 70\text{B} = 140$B FLOPs/token.
• Model B (inference-aware, smaller + over-trained): 30B parameters trained on ~4T tokens (heavier over-training to match quality). Training cost ≈ $6 \times 30\text{B} \times 4\text{T} \approx 7.2 \times 10^{23}$ FLOPs (slightly higher upfront). Inference cost per token ≈ $2 \times 30\text{B} = 60$B FLOPs/token, less than half of Model A.

The break-even point occurs when Model B's extra training FLOPs are recovered by its lower inference FLOPs. Under these equal-quality assumptions, Model A is cheaper below about 1.65T served tokens and Model B is cheaper above that point. A deployment decision must also replace this FLOP proxy with measured quality, latency, utilization, and hardware cost.

What the inference-aware objective predicts

Under the Sardana et al. objective, increasing expected demand shifts the fitted optimum toward fewer parameters and more pre-training tokens while holding the modeled loss target fixed. That's a prediction under an explicit quality model and demand estimate, not evidence that any particular released model was selected using that objective.

For a given quality target, compare candidates according to the question you can evaluate:

The exact optimum depends on the loss target, architecture, data distribution, inference forecast, and serving stack. Don't infer a training objective from a released model's token-to-parameter ratio alone.

Public-text supply as a planning constraint

Both Chinchilla and inference-aware scaling require access to more useful training tokens as their recommended data budget grows. The supply of public human-generated text is finite, but a projection about that supply isn't evidence that it has already been exhausted.

In their analysis, Villalobos et al. ^{[7]Reference 7Will We Run Out of Data? Limits of LLM Scaling Based on Human-Generated Datahttps://arxiv.org/abs/2211.04325} estimate an effective public-human-text stock of about 320T tokens after quality and multi-epoch adjustments. Under continued dataset-growth trends, they forecast full utilization between 2026 and 2032, with a median year of 2028; their assumed 5x over-training scenario shifts that median one year earlier. These are scenario-dependent forecasts, not observed 2026 facts. The projection doesn't establish that full utilization has already occurred.

This projected constraint changes how you read every scaling law here. The $B/D^\beta$ term can only be extrapolated confidently while the data regime remains comparable to the fit. When fresh high-quality text is scarce, practitioners may evaluate better filtering and deduplication, controlled multi-epoch reuse, transfer from other domains, and synthetic data; each changes the assumptions behind the fitted curve.

In the Villalobos et al. forecast, why does assumed over-training move full utilization earlier?

Answer

A longer-trained model uses a larger token budget per parameter than a Chinchilla-style baseline. Under a fixed effective-stock estimate and continued growth assumptions, larger dataset requirements intersect the projected stock earlier. This is a property of the scenario, not proof that an exhaustion date has occurred.

A new scaling axis: test-time compute

So far, "scale" meant training compute split between parameters and tokens. A third allocation question is how much compute the model spends at inference time while answering a single query.

Instead of producing one answer in a fixed number of forward passes, a model can generate longer responses, sample multiple candidates, or search with a verifier. Snell et al. ^{[8]Reference 8Scaling LLM Test-Time Compute Optimally Can Be More Effective Than Scaling Model Parameters.https://arxiv.org/abs/2408.03314} evaluate revision and verifier-search strategies with PaLM 2-S* on MATH. In their FLOP-matched comparison, adaptively allocated test-time compute with a smaller model can outperform a roughly 14x larger model on problems where the smaller model already attains non-trivial success; on the hardest problems or under higher inference workloads, additional pre-training is more effective.

DeepSeek-R1 provides a related post-training example: its R1-Zero model applies reinforcement learning without supervised fine-tuning before RL, while the final R1 pipeline integrates cold-start data, supervised fine-tuning, and reinforcement-learning stages.^{[9]Reference 9DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learninghttps://arxiv.org/abs/2501.12948} It demonstrates that post-training can change reasoning behavior; it doesn't show that pre-training data or compute no longer matters.

This matters for sizing decisions in two ways. First, it adds a knob: for a quality target, you can trade a bigger or longer-trained model against a smaller model that spends more compute per query. Second, it interacts with the inference-aware objective from earlier, because additional generated candidates or longer responses multiply $T_{\text{served}}$, raising the serving term.

The small calculation below isolates generated-token cost for sampled candidates. A real request also includes prompt tokens and any verifier work.

Why is test-time compute a genuinely separate scaling axis rather than just bigger inference?

Answer

Parameters and pre-training tokens are fixed once the model is trained. Test-time compute is chosen per query: the same deployed model can spend more generation or multiple sampled attempts on harder inputs. That creates an additional serving-cost trade-off, while the attainable quality gain still depends on the trained model and task.

Common failures and fixes

Even experienced engineers trip over scaling laws. These five mistakes are frequent enough to check during review.

Mistake 1: treating the 20:1 rule as universal

• Symptom: You read the Chinchilla paper, memorize "20 tokens per parameter," and apply it to every model you build. A 1B-parameter model trained on 20B tokens underperforms on your messy internal-code corpus, and you can't figure out why.

• Cause: The 20:1 ratio is a fitted result for dense transformers on general web text. It shifts with data quality, tokenizer design, optimizer regime, and architecture. It's a starting point, not a physical constant.

• Fix: Treat 20:1 as a baseline, then run small proxy experiments (100M to 1B parameters) on your actual data to fit your own exponents. Data quality, repetition, tokenizer, optimizer, and architecture can move the fitted optimum in either direction.

Mistake 2: treating $C \approx 6ND$ as exact accounting

• Symptom: Your spreadsheet says two candidate runs have the same training FLOPs, but the real cluster bills differ materially.

• Cause: $6ND$ is a dense-transformer rule of thumb. It ignores attention-kernel details, optimizer state, activation checkpointing, sequence length, hardware utilization, distributed communication, data pipeline stalls, and sparse-routing behavior.

• Fix: Use $6ND$ for first-pass sizing, then replace it with measured hardware FLOPs, wall-clock throughput, and dollars per token from your actual stack before committing budget.

Mistake 3: confusing loss with deployed capability

• Symptom: Your scaling study predicts excellent cross-entropy loss, but the deployed model hallucinates facts, ignores instructions, or fails safety checks.

• Cause: Scaling laws predict pre-training loss (how well the model compresses the training distribution), not downstream task performance, alignment quality, or reasoning ability. A model can have great loss and still be useless in production.

• Fix: Budget separately for the post-training pipeline. Techniques such as instruction tuning and RLHF ^{[10]Reference 10Training Language Models to Follow Instructions with Human Feedback (InstructGPT).https://arxiv.org/abs/2203.02155} can improve instruction behavior and preference alignment that a pre-training loss curve doesn't measure. Don't skip evaluation or post-training because your loss curve looks good.

Mistake 4: ignoring inference costs when sizing a model

• Symptom: You train a Chinchilla-optimal 70B model, deploy it to production, and discover that your serving bill exceeds your training budget within three months.

• Cause: Chinchilla optimizes training loss, not total lifetime cost. A smaller model trained longer costs more upfront in training compute but saves money on each inference call.

• Fix: Before you commit to a model size, estimate your expected inference volume. Compare total cost = $6ND$ (train) + $2N \cdot T_{\text{served}}$ (inference) across candidates that meet a measured quality target. At sufficiently high demand, a smaller model trained for longer can be cheaper under this proxy; verify the crossover on your serving stack.

Mistake 5: extrapolating scaling laws far beyond the training regime

• Symptom: Your proxy experiments span models from 100M to 1B parameters. You fit a beautiful straight line, extrapolate it to predict the loss of a 100B model, and the actual result is way off.

• Cause: Power-law fits work well inside the range where they were measured. Outside that range, new bottlenecks appear: optimizer instability, numerical precision issues, data exhaustion, or hardware communication overhead that didn't exist at small scale.

• Fix: Validate with at least one mid-scale experiment before committing to the full target scale. If your proxy range is 100M to 1B, run a 10B validation before you trust the curve at 100B. Treat extrapolation as a hypothesis, not a guarantee.

Scaling law breakdowns and limitations

Scaling laws are empirical fits, not physical laws. Exponents shift with architecture, data quality, tokenizer, and optimizer regime, so one fitted curve shouldn't be treated as a universal law.

Emergent abilities

Wei et al. ^{[11]Reference 11Emergent Abilities of Large Language Models.https://arxiv.org/abs/2206.07682} highlighted benchmark tasks where performance appears to stay near zero and then jump at larger scales, creating the impression of a phase transition in capability. Schaeffer et al. ^{[12]Reference 12Are Emergent Abilities of Large Language Models a Mirage?https://arxiv.org/abs/2304.15004} later argued that many of these jumps are measurement artifacts: when you replace exact-match thresholds with continuous metrics, the same capabilities often return to smoother scaling curves.

The point isn't that emergence is fake or guaranteed. The useful lesson is measurement design: thresholded metrics can make smooth changes look abrupt, so teams should inspect continuous metrics before treating a capability jump as a new law of scale.

Task-specific ceilings

Scaling laws predict pre-training loss, not downstream task performance. A model that achieves excellent perplexity may still fail at:

• Factual accuracy (hallucinations)
• Instruction following
• Safety alignment
• Specific domain tasks

Because scaling laws only measure the model's ability to compress and predict the training distribution, teams must allocate separate compute budgets for the post-training pipeline. Techniques such as instruction tuning and RLHF ^{[10]Reference 10Training Language Models to Follow Instructions with Human Feedback (InstructGPT).https://arxiv.org/abs/2203.02155} are used to improve instruction behavior and preference alignment, which must be evaluated separately from pre-training loss.

Why can a scaling study predict great loss but still produce a bad coding assistant?

Answer

Scaling laws predict pre-training loss, not factual accuracy, instruction following, domain reliability, or safety behavior. A coding assistant still needs data-quality checks, post-training, evaluation, and alignment work after the pre-training budget is chosen.

Architecture sensitivity

Scaling laws derived for dense transformers don't directly transfer to:

• Mixture-of-Experts (MoE) (lesson): MoE models selectively activate subsets of parameters for each input, decoupling total parameters from compute per token. This makes it difficult to apply standard dense scaling laws, as active parameters and total parameters scale differently.
• State Space Models (SSMs): SSMs process sequences recurrently rather than using quadratic attention. These alternative architectures have their own scaling exponents and require independent empirical fitting.
• Hybrid architectures: Models that combine attention with SSMs or other mechanisms must have their scaling behavior re-characterized from scratch.

For these architectures, dense-transformer scaling laws are useful context, not a substitute for new measurements. Each architecture needs empirical scaling studies to determine its own parameter and data allocation.

Why can't you blindly apply dense-transformer scaling laws to MoE models?

Answer

Dense scaling laws treat parameter count as the main size variable. MoE models split total parameters from active parameters per token. Total parameters affect memory and routing; active parameters affect compute. That means you need a separate empirical scaling fit.

From theory to practice: running a scaling study

When planning a large training run, teams conduct scaling studies (small-scale experiments that predict large-scale performance) before committing resources. The practical loop is:

1. Train proxy models, usually across several parameter counts and token budgets.
2. Fit scaling exponents for the loss surface.
3. Extrapolate to the target frontier: model size, token budget, and expected loss.
4. Run a mid-scale validation experiment to check the extrapolation.
5. Commit to the full training run only after the mid-scale result lands near the fitted curve.

The µ-Transfer approach

To estimate the performance of a large model before committing to its full run, engineers use proxy models. A complementary tool is µ-Transfer (Tensor Programs V) ^{[13]Reference 13Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transferhttps://arxiv.org/abs/2203.03466}, which uses the Maximal Update Parametrization (µP) so selected hyperparameters tuned on a smaller proxy can be transferred to a larger target model.

Yang et al. show that many optimal hyperparameters remain stable as model size changes in µP, and they verify transfer on Transformer and ResNet experiments. Their reported examples transfer from 13M parameters to BERT-large and from 40M parameters to a 6.7B GPT-3 model while reducing tuning cost. This is evidence for the tested setups, not permission to copy every setting without validation.

Used alongside a scaling study, the workflow is:

1. Train smaller proxy models across multiple data budgets to sample the loss surface.
2. In a µP setup, tune hyperparameters covered by the transfer method at small scale and transfer candidate settings to larger models.
3. Fit the power-law scaling exponents ($\alpha$, $\beta$, and the constants) using the observed losses from proxy runs.
4. Extrapolate the fitted curve to the target scale to predict final loss.
5. Validate with a single medium-scale experiment before committing to the full run.

The µ-Transfer paper shows that zero-shot hyperparameter transfer can work across large changes in model scale in its tested setups. A medium-scale validation still matters because transferability, loss fits, data mixture, and hardware behavior are separate failure modes.

Fitting the parametric loss function

One common parametric fit for model size and data, used in Chinchilla-style scaling studies, is:^{[2]Reference 2Training Compute-Optimal Large Language Models.https://arxiv.org/abs/2203.15556}

$$L(N, D) = E + \frac{A}{N^{\alpha}} + \frac{B}{D^{\beta}}$$

Reading the formula: Loss equals three fitted terms: (1) $E$, the asymptotic floor in this model of the measured regime, (2) $A/N^\alpha$, the fitted penalty associated with finite parameters, and (3) $B/D^\beta$, the fitted penalty associated with finite training data. Making the model bigger reduces term 2; more data reduces term 3. Treat $E$ as an extrapolated fit parameter, not proof that you have measured the irreducible entropy of language.

To see how this works in practice, consider a tiny synthetic dataset of proxy runs:

Notice the pattern: fixing N and increasing D lowers loss; fixing D and increasing N also lowers loss. The parametric formula above captures both effects at once.

This Python script fits the scaling constants with SciPy's curve_fit and then predicts loss at a larger target scale:

Turn the fit into a compute decision

Keep at least one proxy run out of the fit, report its residual and parameter uncertainty in a decision table, then compare the forecast with that held-out result. Before committing the target budget, run a mid-scale checkpoint on the planned tokenizer, data mixture, sequence length, and training stack. Stop or refit when measured loss leaves the forecast band; a smooth extrapolation isn't permission to spend.