Scaling Laws & Compute-Optimal Training

Your team has a training budget of $10^{24}$ floating-point operations (FLOPs). One proposal spends it on a large model and a short run; another trains a smaller model for longer. Which proposal should you approve, and what evidence would change your choice?

The tradeoff depends on the objective. A parameter-heavy run can spend compute inefficiently for its available token budget. A smaller model may keep improving with more tokens yet miss the lowest loss attainable at the same training compute. It may still be the better model to serve. “Not training-optimal” doesn't mean “no longer learning.”

Scaling laws are empirical fits that help compare these proposals. In studied regimes, power laws summarize how cross-entropy loss changes with compute, parameters, and tokens. Their usefulness depends on the data, architecture, training recipe, and range over which they were checked.

Kaplan et al.'s influential 2020 study fitted a parameter-heavy allocation.^{[1]Reference 1Scaling Laws for Neural Language Modelshttps://arxiv.org/abs/2001.08361} Hoffmann et al.'s 2022 Chinchilla study found approximately balanced growth in parameters and tokens.^{[2]Reference 2Training Compute-Optimal Large Language Models.https://arxiv.org/abs/2203.15556} Later work investigated why the fits differed, and how future inference demand changes the objective.^{[3]Reference 3Resolving Discrepancies in Compute-Optimal Scaling of Language Modelshttps://arxiv.org/abs/2406.19146[4]Reference 4Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Lawshttps://arxiv.org/abs/2401.00448} The historical Llama models illustrate long training runs at several parameter sizes; they aren't evidence for one universal inference-optimal ratio.^{[5]Reference 5The Llama 3 Herd of Models.https://arxiv.org/abs/2407.21783}

Three fundamental symbols anchor every scaling formula:

• Parameters ($N$): A declared weight count. Verify whether a paper includes embedding and output-projection weights. Tying those weights changes storage, but the vocabulary projection still costs arithmetic. Parameter count alone isn't a measurement of useful capacity.
• Training tokens ($D$): The number of subword token presentations processed during training, usually grouped into batches for updates. Distinguish this from unique corpus size. Repeated presentations can help, but a fresh-data fit needn't describe multi-epoch learning.
• Compute ($C$): A stated training-FLOP budget or estimate. Distinguish useful model arithmetic, recomputation, accelerator peak capacity, elapsed time, and dollars.

The first-principles derivation of 6ND

Dense transformer training costs follow an elegant rule of thumb: $C \approx 6ND$ FLOPs. To understand where the constant $6$ comes from, break down matrix multiplications across forward and backward passes.

For a batch of token vectors $X$ and a dense matrix $W \in \mathbb{R}^{d_{\text{in}} \times d_{\text{out}}}$, the forward operation is $Y=XW$. Counting a multiply-accumulate as two FLOPs gives approximately $2d_{\text{in}}d_{\text{out}}$ FLOPs per token. This is the conventional GEMM estimate, not an exact count of the final dot-product addition. Summing the large dense projections gives:

For trainable dense matrices with gradients propagated to their inputs, backpropagation has two large contractions:

1. Activation gradients: $\nabla_X L=(\nabla_Y L)W^T$, approximately another $2N$ FLOPs per token across the counted matrices.
2. Weight gradients: $\nabla_W L=X^T(\nabla_Y L)$, approximately another $2N$ FLOPs per token, including accumulation over the batch.

Adding activation gradients ($2N$) and weight gradients ($2N$) gives $4N$ FLOPs per token for the backward pass:

Summing forward ($2N$) and backward ($4N$) passes yields $6N$ FLOPs per token. Multiplied across a training run of $D$ tokens, total compute lands at:

Recomputing approximately one full counted forward pass through activation checkpointing would raise this estimate to $8ND$. Selective checkpointing adds a different amount. Inference has no backward pass, so its dense-projection proxy is $2N$ FLOPs per processed token.

These proxies omit attention's query/key and attention/value products, nonlinearities, normalization, optimizer work, and other overhead. For long contexts, attention arithmetic can matter substantially. Include the vocabulary head when it isn't negligible; using a storage count that omits its operation can distort small-model comparisons.^{[3]Reference 3Resolving Discrepancies in Compute-Optimal Scaling of Language Modelshttps://arxiv.org/abs/2406.19146} FLOPs don't directly determine decode latency, which can be limited by memory traffic.

Check your arithmetic carefully. Training a 1-billion-parameter model on 20 billion tokens requires approximately $6 \times (1 \times 10^9) \times (20 \times 10^9) = 1.2 \times 10^{20}$ FLOPs. Doubling both parameters and tokens quadruples compute to $4.8 \times 10^{20}$ FLOPs.

Double parameters, double data, or both

Consider four invented measurements under a shared evaluation setup. A 1-billion-parameter baseline trains on 10 billion tokens. Three follow-up runs double model size, double data volume, or double both:

The middle rows cost the same under $6ND$. In this fixture, spending that increment on tokens gives 0.05 lower loss than spending it on parameters. This supports one local choice; without repeat runs or uncertainty, it doesn't establish a general bottleneck or scaling exponent. Doubling both lowers loss further at four times the baseline compute.

Always hold evaluation conditions constant: tokenizer vocabulary, dataset distribution, context length, and loss units. Cross-entropy loss measured in natural logarithms is expressed in nats per token. Changing tokenizers alters average token length, rendering raw per-token losses incomparable.

If $L=KC^{-\alpha}$, plotting $\log L$ against $\log C$ gives a straight line. If the law has a floor, $L=E+KC^{-\alpha}$, the straight-line quantity is $L-E$. Guessing the floor can change both exponent and forecast. Proxy runs support a forecast to test, not guaranteed extrapolation.

Inspect how diminishing returns manifest under power-law scaling using Kaplan's parameter exponent ($\alpha_N \approx 0.076$):

alpha_n = 0.076

for parameter_multiplier in [2, 10, 100]:
    scaled_term = parameter_multiplier ** (-alpha_n)
    reduction = 1 - scaled_term
    print(
        f"{parameter_multiplier:>3}x parameters -> "
        f"{scaled_term:.3f}x fitted-regime loss "
        f"({reduction:.1%} reduction)"
    )

2x parameters -> 0.949x fitted-regime loss (5.1% reduction)
 10x parameters -> 0.839x fitted-regime loss (16.1% reduction)
100x parameters -> 0.705x fitted-regime loss (29.5% reduction)

Kaplan scaling laws (2020)

Kaplan et al. ^{[1]Reference 1Scaling Laws for Neural Language Modelshttps://arxiv.org/abs/2001.08361} established empirical scaling laws for dense decoder-only transformers. By isolating individual training constraints, they demonstrated that performance follows precise power-law relationships across several orders of magnitude.

The three power laws

1. Scaling with parameters (model size $N$)

When training data is abundant and compute is unrestricted, model capacity forms the primary bottleneck. As parameter count $N$ increases, loss decreases as a power law:

Kaplan fitted $\alpha_N\approx0.076$ using non-embedding parameter counts. Its displayed single-axis law fits loss itself without an additive floor, so the 16% calculation concerns fitted-regime loss. Don't silently reinterpret it as a reduction in $L-E$ from a different model, or extrapolate it to zero entropy.^{[1]Reference 1Scaling Laws for Neural Language Modelshttps://arxiv.org/abs/2001.08361}

2. Scaling with data (dataset size $D$ in tokens)

When model capacity is arbitrarily large and training halts before overfitting, available tokens form the bottleneck:

Kaplan reported $\alpha_D \approx 0.095$. Here $D$ describes dataset token positions, not the number of distinct vocabulary IDs. The fresh-data relationship doesn't prescribe the value of repeated epochs; those need their own checks.

3. Scaling with compute-optimal training compute ($C_{\min}$)

When allocating compute efficiently between parameters and tokens, minimum attainable loss scales as:

Kaplan estimated $\alpha_C \approx 0.050$ for its compute-efficient frontier. $C_{\min}$ includes the paper's adjustment for inefficient large batches; it isn't interchangeable with any run's measured FLOPs. The relationship concerns efficient allocation, not simply spending more on one axis.

Kaplan's compute allocation: why parameters grew faster than data

When holding compute fixed and optimizing $N$ and $D$ jointly, Kaplan et al. arrived at an unexpected conclusion:

Under Kaplan's frontier, when total compute grows by $10\times$, parameters should increase by $10^{0.73} \approx 5.37\times$, while tokens should grow by only $10^{0.27} \approx 1.86\times$. For a $100\times$ compute surge, parameters expand by $28.84\times$ while tokens increase by just $3.47\times$.

This allocation shaped early frontier LLMs. OpenAI trained GPT-3 ^{[6]Reference 6Language Models are Few-Shot Learners.https://arxiv.org/abs/2005.14165} with 175 billion parameters on only 300 billion tokens, yielding a token-to-parameter ratio of roughly 1.7:1. DeepMind followed a similar path with Gopher, building a 280-billion-parameter model on 300 billion tokens (1.07:1).

Compare the allocation multipliers directly in Python:

frontiers = {
    "Kaplan": (0.73, 0.27),
    "Chinchilla": (0.50, 0.50),
}

for compute_multiplier in [10, 100]:
    print(f"{compute_multiplier}x training compute")
    for name, (parameter_exp, token_exp) in frontiers.items():
        parameters = compute_multiplier ** parameter_exp
        tokens = compute_multiplier ** token_exp
        print(f"  {name:<10} N={parameters:5.2f}x, D={tokens:5.2f}x")

10x training compute
  Kaplan     N= 5.37x, D= 1.86x
  Chinchilla N= 3.16x, D= 3.16x
100x training compute
  Kaplan     N=28.84x, D= 3.47x
  Chinchilla N=10.00x, D=10.00x

Compare the dominant-matmul estimates for two historical configurations using the $6ND$ proxy:

def dense_training_flops(parameters, tokens):
    return 6 * parameters * tokens

gpt3_style = dense_training_flops(175e9, 300e9)
small_data_rich = dense_training_flops(70e9, 1.4e12)

print(f"GPT-3-style training FLOPs: {gpt3_style:.2e}")
print(f"70B / 1.4T-token training FLOPs: {small_data_rich:.2e}")

GPT-3-style training FLOPs: 3.15e+23
70B / 1.4T-token training FLOPs: 5.88e+23

The Chinchilla reallocation (2022)

In 2022, Hoffmann et al. at DeepMind ^{[2]Reference 2Training Compute-Optimal Large Language Models.https://arxiv.org/abs/2203.15556} found that several earlier models used too few training tokens relative to the allocation their loss fits favored at the same compute.

Why the allocation studies disagreed

Why did Kaplan et al. conclude that data mattered so much less than parameters?

Hoffmann et al. highlighted a schedule concern: an intermediate checkpoint in a long cosine-decay run hasn't received the cooldown of a run scheduled to finish at that token budget. They matched training horizons to schedules and investigated larger proxy models. This was an explanation to investigate, not proof that decay alone caused the different allocation exponent.^{[2]Reference 2Training Compute-Optimal Large Language Models.https://arxiv.org/abs/2203.15556}

Porian et al.'s later reproduction identified three influential factors: counting output-head FLOPs, shortening excessive warmup for small runs, and tuning learning rate, batch size, and AdamW $\beta_2$ across scale. In their experiments, matched cosine decay improved losses but had limited effect on the allocation exponent; tuned runs with a constant learning rate could recover approximately balanced allocation.^{[3]Reference 3Resolving Discrepancies in Compute-Optimal Scaling of Language Modelshttps://arxiv.org/abs/2406.19146}

The distinction matters when designing your study. A change can lower every run's loss without greatly moving the best parameter/token split. Check the optimum along equal-compute profiles, not just whether one final checkpoint improved.

Chinchilla's three methodologies and the equal-scaling derivation

Hoffmann et al. trained over 400 models spanning 70 million to 16 billion parameters across 5 billion to 500 billion tokens. They deployed three independent fitting approaches:

Approach 3 lets us derive the allocation from a fitted loss surface. Assume positive coefficients and exponents:

Where:

• $E$ is a fitted floor, interpreted in this model as the ideal data-distribution loss. Its fitted value isn't an independently measured entropy.
• $A/N^\alpha$ is the penalty for finite parameters.
• $B/D^\beta$ is the penalty for finite training tokens.

To find the compute-optimal allocation for a fixed budget $C$, substitute $D = \frac{C}{6N}$ into the loss function:

Differentiating with respect to $N$ and setting the derivative to zero:

Multiplying through by $N$ produces an elegant balance condition:

At the optimum, the loss reductions per proportional increase in either resource balance. These are derivatives with respect to $\log N$ and $\log D$, not equal gains per one parameter and one token:

Solving for $N^*(C)$ and $D^*(C)$ in terms of compute $C$:

Notice the compute-scaling exponents:

In DeepMind's empirical fit, $\alpha \approx 0.34$ and $\beta \approx 0.28$. Substituting those exponents gives:

The rounded $\alpha,\beta$ calculation is approximately balanced; the paper reports Approach 3 as $a=0.46,b=0.54$. Approach 1 reports $0.50/0.50$ and Approach 2 $0.49/0.51$. With the simplified equal-growth model, both resources scale with the square root of compute:

The often-used 20:1 token-to-parameter approximation is consistent with the paper's Approach 1 projections: roughly 20.2B tokens for 1B parameters, or 205.1B for 10B. It also describes the nominal 70B/1.4T Chinchilla configuration. It doesn't follow as a constant ratio from the unequal-exponent parametric fit above:

The general fit gives $D^*/N^*\propto C^{(\alpha-\beta)/(\alpha+\beta)}$. Even equal exponents $\alpha=\beta=\gamma$ only give a constant ratio $(B/A)^{1/\gamma}$, not a mandatory value of 20. The coefficients and their counting units matter. Use 20:1 as a declared sizing approximation, then test it against your own curves.^{[2]Reference 2Training Compute-Optimal Large Language Models.https://arxiv.org/abs/2203.15556}

Try changing just the coefficients. These are authored unitless toy counts, with $\gamma=0.5$ and fixed $ND=10^6$, not a fitted language model:

import math

gamma = 0.5
product = 1e6
for a, b in ((1.0, 1.0), (2.0, 8.0)):
    ratio = (b / a) ** (1 / gamma)
    n = math.sqrt(product / ratio)
    d = ratio * n
    print(f"A={a:.0f}, B={b:.0f}: N={n:.0f}, D={d:.0f}, D/N={ratio:.0f}")

A=1, B=1: N=1000, D=1000, D/N=1
A=2, B=8: N=250, D=4000, D/N=16

Both laws allocate growth equally as compute rises, yet their optimal ratios differ. With unequal exponents, even the ratio changes with budget: the rounded $0.34/0.28$ fit makes it grow by about 1.562 when compute increases 100-fold.

Size a candidate from a target FLOP budget using a declared 20:1 approximation:

def chinchilla_style_size(training_flops, tokens_per_parameter=20):
    import math
    if not all(math.isfinite(x) and x > 0 for x in (training_flops, tokens_per_parameter)):
        raise ValueError("FLOP budget and token/parameter ratio must be finite and positive")
    parameters = (training_flops / (6 * tokens_per_parameter)) ** 0.5
    tokens = tokens_per_parameter * parameters
    return parameters, tokens

parameters, tokens = chinchilla_style_size(1e24)
print(f"Parameters at 1e24 FLOPs: {parameters / 1e9:.1f}B")
print(f"Tokens at 20:1: {tokens / 1e12:.2f}T")

Parameters at 1e24 FLOPs: 91.3B
Tokens at 20:1: 1.83T

Chinchilla vs. Gopher: a frontier-scale test

DeepMind reported training nominally 70B-parameter Chinchilla on 1.4T tokens at Gopher's training-FLOP budget. The paper's estimated optimal-size range was 40B–70B, and it selected the larger end partly for data and computational efficiency. This was an empirical test, rather than proof of an exact global optimum.^{[2]Reference 2Training Compute-Optimal Large Language Models.https://arxiv.org/abs/2203.15556}

The shared budget above is the figure used for the paper's frontier projections. Appendix F also reports a revised estimate of $6.3\times10^{23}$ with its more detailed accounting, versus the earlier $5.76\times10^{23}$. The rounded $6ND$ proxy doesn't reproduce either exactly; counts and accounting formulas aren't interchangeable. Chinchilla improved broad evaluation results, including average MMLU, BIG-bench, and LAMBADA, but not every individual task: it underperformed Gopher on four MMLU tasks. The models also differed in optimizer, tokenizer, and data mixture, so this comparison doesn't isolate only the allocation.^{[2]Reference 2Training Compute-Optimal Large Language Models.https://arxiv.org/abs/2203.15556}

At the same weight precision, one-fourth the parameters implies roughly one-fourth the weight storage and dense-projection arithmetic. Total serving memory also includes KV cache and workspace; elapsed latency and dollars require profiling.

Compute a 10-billion-parameter Chinchilla-style training budget:

def chinchilla_tokens(parameters, tokens_per_parameter=20):
    return parameters * tokens_per_parameter

def dense_training_flops(parameters, tokens):
    return 6 * parameters * tokens

parameters = 10e9
tokens = chinchilla_tokens(parameters)
flops = dense_training_flops(parameters, tokens)

print(f"Chinchilla-style tokens for 10B parameters: {tokens:.2e}")
print(f"Dense training FLOPs: {flops:.2e}")

Chinchilla-style tokens for 10B parameters: 2.00e+11
Dense training FLOPs: 1.20e+22

Beyond Chinchilla: inference-aware scaling

Chinchilla scaling solves a specific optimization problem: finding the lowest pre-training loss for a fixed training compute budget ($C_{\text{train}}$). Its objective excludes serving cost; it doesn't assume the model is literally never used.

Serving can eventually dominate lifetime cost, but token counts alone need arithmetic. Under the same $6ND$ training and $2NQ$ inference proxies, inference exceeds training at $Q>3D$. For a run with 1.4T training tokens, that means more than 4.2T processed inference tokens, not automatically “hundreds of billions.”

Here overtraining means training beyond a declared training-optimal token allocation for that size. We use 20:1 as a simple historical baseline, not the optimum of every fit. It doesn't mean overfitting, and it doesn't guarantee validation loss falls smoothly forever. It can improve a compact model while using more training compute than a different size would need to reach the same fitted loss.

The lifetime economic break-even equation

Sardana et al. ^{[4]Reference 4Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Lawshttps://arxiv.org/abs/2401.00448} formalized inference-aware scaling by modeling the total lifetime compute bill:

Where:

• $Q$ counts lifetime processed inference tokens, treating prompt and completion tokens alike in this simplified proxy. Reused cached prefixes don't require a new full forward pass for every billed input token.
• $2N$ counts the large dense projections. Context-dependent attention and serving overhead aren't included.

Suppose two candidates achieve the same held-out pre-training cross-entropy under a shared evaluation setup. This doesn't automatically establish equal deployed task quality:

• Model 1 (Chinchilla-optimal): Larger model ($N_1$) trained on fewer tokens ($D_1 \approx 20N_1$).
• Model 2 (Overtrained compact model): Smaller model ($N_2 < N_1$) overtrained on a massive token corpus ($D_2 \gg 20N_2$).

If Model 1 is training-optimal for that target loss in the fit, Model 2 needs at least as much training compute. For a general pair of candidates, measure the difference rather than assuming it's positive:

However, Model 2 saves compute on every single served token during deployment:

The break-even processed-token volume ($Q_{\text{break-even}}$), not number of requests, is where accumulated inference savings offset a positive training premium:

Above a positive break-even, the smaller model wins on proxy FLOPs, assuming equal acceptable quality and the same demand. If its training difference is nonpositive, it is already no more expensive at zero demand. If parameter counts are equal, this proxy gives no per-token saving to divide by.

Dollar break-even may differ. With train/inference prices per useful FLOP $p_t,p_i$, the crossing is $Q=p_t\Delta C_{\rm train}/(p_i\Delta C_{\rm infer/token})$. Sardana et al. separately model real-world costs and different prompt/decode utilization. Batch size, memory traffic, caching, precision, and latency targets can change those costs.^{[4]Reference 4Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Lawshttps://arxiv.org/abs/2401.00448}

Concrete break-even example

Consider two authored configurations and assume they meet the same task-quality requirement. The arithmetic below doesn't establish that assumption; actual evaluations must:

• Model A (Chinchilla-style): 70B parameters trained on 1.4T tokens. Training cost $\approx 6 \times 70\text{B} \times 1.4\text{T} \approx 5.88 \times 10^{23}$ FLOPs. Inference cost per token $\approx 2 \times 70\text{B} = 140$B FLOPs.
• Model B (Inference-aware overtrained): 30B parameters trained on 4.0T tokens. Training cost $\approx 6 \times 30\text{B} \times 4.0\text{T} \approx 7.20 \times 10^{23}$ FLOPs. Inference cost per token $\approx 2 \times 30\text{B} = 60$B FLOPs.

Model B requires $1.32 \times 10^{23}$ extra training FLOPs upfront. But it saves $140\text{B} - 60\text{B} = 80\text{B}$ FLOPs on every served token.

Calculate the exact break-even token volume in Python:

def train_flops(parameters, tokens):
    return 6 * parameters * tokens

def inference_flops(parameters, served_tokens):
    return 2 * parameters * served_tokens

model_a_train = train_flops(70e9, 1.4e12)
model_b_train = train_flops(30e9, 4e12)
extra_train = model_b_train - model_a_train
savings_per_token = 2 * (70e9 - 30e9)
break_even_tokens = extra_train / savings_per_token

print(f"Model A training FLOPs: {model_a_train:.2e}")
print(f"Model B training FLOPs: {model_b_train:.2e}")
print(f"Break-even served tokens: {break_even_tokens:.2e}")

Model A training FLOPs: 5.88e+23
Model B training FLOPs: 7.20e+23
Break-even served tokens: 1.65e+12

Evaluate total lifetime compute at 1 trillion versus 3 trillion served tokens:

def total_flops(parameters, training_tokens, served_tokens):
    return 6 * parameters * training_tokens + 2 * parameters * served_tokens

models = {
    "70B / 1.4T": (70e9, 1.4e12),
    "30B / 4.0T": (30e9, 4.0e12),
}

for demand in [1e12, 3e12]:
    costs = {
        name: total_flops(parameters, tokens, demand)
        for name, (parameters, tokens) in models.items()
    }
    cheaper = min(costs, key=costs.get)
    print(f"{demand / 1e12:.0f}T served tokens -> cheaper candidate: {cheaper}")

1T served tokens -> cheaper candidate: 70B / 1.4T
3T served tokens -> cheaper candidate: 30B / 4.0T

Historical configurations: the Llama trajectory

These published configurations show how token budgets can differ from the approximate 20:1 training-only baseline. Ratios below use nominal model sizes, so they are rounded:

• LLaMA 1 (2023) ^{[7]Reference 7LLaMA: Open and Efficient Foundation Language Models.https://arxiv.org/abs/2302.13971}: nominal 7B on 1.0T tokens (about 143:1) and 65B on 1.4T (about 21.5:1).
• Llama 2 (2023) ^{[8]Reference 8Llama 2: Open Foundation and Fine-Tuned Chat Models.https://arxiv.org/abs/2307.09288}: 7B on 2.0T tokens (about 286:1) and 70B on 2.0T (about 28.6:1).
• Original Llama 3 8B (April 2024): its model card reports 15T+ training tokens. That's at least about 1,875 tokens per nominal parameter, or about 94 times the 160B-token budget from assuming $D=20N$.
• Llama 3 405B (2024) ^{[5]Reference 5The Llama 3 Herd of Models.https://arxiv.org/abs/2407.21783}: the paper reports 15.6T tokens for its initial pre-training stage, about 38.5:1. Don't silently assign that precise count to the original 8B release.

Longer training can improve a compact model whose per-token arithmetic is lower. Whether that pays for itself depends on actual quality, demand, and serving costs. The published ratios alone don't establish a lifetime optimum, and changes in tokenizer, data mix, and architecture prevent treating this chronology as a controlled scaling experiment.

Public-text supply as a planning constraint

Overtraining assumes an abundant data supply. However, high-quality human-generated text on the public internet is finite.

Villalobos et al.'s June 2024 forecast ^{[9]Reference 9Will We Run Out of Data? Limits of LLM Scaling Based on Human-Generated Datahttps://arxiv.org/abs/2211.04325v2} projected that growing training-data requirements could meet effective public human-text supply during 2026–2032, with a median around 2028. Its roughly $4 \times 10^{14}$-token figure is a projected effective stock at that intersection, accounting for quality filtering and useful repetition. It isn't a measurement of 400T unique usable tokens available today, or evidence that a bottleneck has already occurred.

Text isn't physically consumed by training. Four runs can reuse the same corpus:

Only the second scenario has four times the unique-data requirement. Even then, dividing raw unique tokens by a quality/repetition-adjusted effective stock mixes different definitions. Access rights, filtering losses, distribution, and the value of repeat epochs matter more than a count of model variants.

Calculate a nominal ratio, then ask what its denominator means:

effective_stock = 4e14  # Villalobos et al.: ~400T effective stock at utilization
llama3_405b_tokens = 15.6e12
chinchilla_style_tokens = 20 * 405e9

print(f"Llama 3 405B reported tokens: {llama3_405b_tokens / 1e12:.1f}T")
print(f"Chinchilla-style 405B tokens: {chinchilla_style_tokens / 1e12:.1f}T")
print(
    "Reported token budget / projected effective stock: "
    f"{llama3_405b_tokens / effective_stock:.1%}"
)
print("Numerical comparison only; not a fraction of unique supply used.")

Llama 3 405B reported tokens: 15.6T
Chinchilla-style 405B tokens: 8.1T
Reported token budget / projected effective stock: 3.9%
Numerical comparison only; not a fraction of unique supply used.

Possible responses include useful repeat epochs, verified synthetic data, and additional domains or modalities. None guarantees that fresh-data scaling continues unchanged. A September 2025 Epoch AI assessment judged a data bottleneck potentially surmountable with synthetic data; that is an updated assessment, not proof of unlimited supply. Test-time reasoning offers another compute allocation, but doesn't create new pre-training evidence by itself.

A new scaling axis: test-time compute

Pre-training fixes model weights. Once trained, serving introduces an orthogonal scaling axis: test-time compute.

Additional inference work can mean longer reasoning traces, multiple sampled candidates, or search scored by an external verifier. These are different mechanisms: DeepSeek-R1's reasoning traces ^{[10]Reference 10DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learninghttps://arxiv.org/abs/2501.12948} don't establish that it executes an external verifier's tree search on every request.

Snell et al. ^{[11]Reference 11Scaling LLM Test-Time Compute Optimally Can Be More Effective Than Scaling Model Parameters.https://arxiv.org/abs/2408.03314} studied MATH with PaLM 2-S* and task-specific revision/verifier training. With compute allocated according to problem difficulty, their smaller model could outperform a single-response model with roughly 14 times as many parameters in some matched-compute scenarios. Harder questions and high inference demand could instead favor the larger pre-trained model. This is a conditional experimental result, not a promise that extra thinking substitutes for any model-size gap. Estimating difficulty and scoring candidates also cost work.

Suppose every candidate generates 512 tokens, with no early stopping. Sixteen candidates produce 16 times as many output tokens. This example counts their dense generation arithmetic; it excludes prompt processing, attention, and any verifier:

def generation_flops(parameters, output_tokens_per_candidate, candidates):
    return 2 * parameters * output_tokens_per_candidate * candidates

parameters = 8e9
output_tokens_per_candidate = 512
for candidates in [1, 4, 16]:
    flops = generation_flops(parameters, output_tokens_per_candidate, candidates)
    print(f"{candidates:>2} candidate(s): {flops:.2e} generation FLOPs per request")

1 candidate(s): 8.19e+12 generation FLOPs per request
 4 candidate(s): 3.28e+13 generation FLOPs per request
16 candidate(s): 1.31e+14 generation FLOPs per request

From theory to practice: running a scaling study

How do you translate theoretical power laws into real-world cluster decisions? A disciplined scaling study executes controlled proxy runs across smaller models to fit loss parameters, extrapolates to the target compute budget, and verifies predictions at an intermediate gate before committing the final cluster budget.

Zero-shot hyperparameter transfer with maximal update parametrization (µP)

An unchanged numerical learning rate can produce different function updates as width grows. Maximal Update Parametrization ($\mu$P) ^{[12]Reference 12Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transferhttps://arxiv.org/abs/2203.03466} controls that dependence through coordinated initialization, multipliers, and optimizer settings. Rules depend on layer role and optimizer; an SGD example is not an Adam recipe for every transformer.

See the mechanism in the paper's two-layer linear setup: $f(x)=V^\top Ux$, with scalar $x=1$ and squared loss $\tfrac12(f-1)^2$. We'll use authored deterministic vectors: alternating $U_i=\pm1$, and equal $V_i$ so the initial prediction is zero. Both sets of gradients must use the old weights. Before running, predict which update will grow with width:

import math

def one_step(u, v, input_lr, output_lr):
    prediction = math.fsum(a * b for a, b in zip(u, v))
    error = prediction - 1.0
    new_u = [a - input_lr * error * b for a, b in zip(u, v)]
    new_v = [b - output_lr * error * a for a, b in zip(u, v)]
    return math.fsum(a * b for a, b in zip(new_u, new_v))

for width in (16, 64, 256):
    u = [1.0 if i % 2 == 0 else -1.0 for i in range(width)]
    standard = one_step(u, [1 / math.sqrt(width)] * width, 1.0, 1.0)
    mup = one_step(u, [1 / width] * width, width, 1 / width)
    print(f"width={width}: standard prediction={standard:.1f}, muP prediction={mup:.1f}")

width=16: standard prediction=17.0, muP prediction=2.0
width=64: standard prediction=65.0, muP prediction=2.0
width=256: standard prediction=257.0, muP prediction=2.0

The standard update grows as $n+1$. The coordinated $\mu$P update stays at 2 because this example scales the input-layer SGD rate up by $n$, the output-layer rate down by $n$, and $V_i$ as $1/n$. It still overshoots the target 1: width stability doesn't select a good base learning rate for you.

The study demonstrated learning-rate transfer from a 40M proxy to 6.7B GPT-3, not a blanket guarantee for 70B runs. Finite-width optima can shift; depth and architecture changes require checks. Regularization settings such as dropout and weight decay needn't transfer. Test the intended width scaling and validate a larger run before spending the full budget.

Fitting the parametric loss surface

In a scaling study, you train a grid of small proxy models (for example, 100M to 1B parameters) across varying token counts and measure validation cross-entropy. You then fit the five constants of Chinchilla Approach 3:

Consider a synthetic benchmark surface generated from:

For this controlled exercise, we supply the true $E$, $A$, and $B$ and search only the two exponents. Hold out the point (1B / 100B), then forecast loss at 70B / 1.4T. Recovering the exponents from their own rounded generating formula is much easier than fitting all five unknowns to noisy training runs:

def scaling_loss(n, d, e, a, alpha, b, beta):
    return e + a / (n ** alpha) + b / (d ** beta)

E, A, B = 1.10, 2.80, 7.80
experiments = [
    (1e8, 1e9, 2.895),
    (1e8, 5e9, 2.745),
    (1e8, 2e10, 2.634),
    (5e8, 1e9, 2.813),
    (5e8, 5e9, 2.663),
    (5e8, 2e10, 2.552),
    (1e9, 5e9, 2.631),
    (1e9, 2e10, 2.520),
    (1e9, 1e11, 2.408),
]
train, held = experiments[:-1], experiments[-1]

def sum_squared_error(alpha, beta):
    total = 0.0
    for n, d, loss in train:
        pred = scaling_loss(n, d, E, A, alpha, B, beta)
        total += (pred - loss) ** 2
    return total

best_error = float("inf")
best = (0.0, 0.0)
for alpha_i in range(40, 121, 2):
    for beta_i in range(40, 121, 2):
        alpha = alpha_i / 1000
        beta = beta_i / 1000
        error = sum_squared_error(alpha, beta)
        if error < best_error:
            best_error = error
            best = (alpha, beta)

alpha_fit, beta_fit = best
n_held, d_held, loss_held = held
hold_pred = scaling_loss(n_held, d_held, E, A, alpha_fit, B, beta_fit)
target = scaling_loss(70e9, 1.4e12, E, A, alpha_fit, B, beta_fit)

print(f"Recovered alpha={alpha_fit:.3f}, beta={beta_fit:.3f}")
print(
    f"Hold-out 1B / 100B: pred={hold_pred:.3f}, "
    f"observed={loss_held:.3f}, residual={hold_pred - loss_held:+.4f}"
)
print(f"Forecast 70B / 1.4T tokens: {target:.3f}")

Recovered alpha=0.070, beta=0.098
Hold-out 1B / 100B: pred=2.408, observed=2.408, residual=+0.0002
Forecast 70B / 1.4T tokens: 2.091

Two measurements can hide very different forecasts

What if the loss floor is unknown? Use normalized compute and fit two authored observations, $L(1)=3$ and $L(100)=2.5$, with $L(C)=E+KC^{-\alpha}$. For each chosen exponent, solve for $K$ and $E$:

for alpha in (0.1, 0.2):
    k = (3.0 - 2.5) / (1.0 - 100.0 ** (-alpha))
    floor = 3.0 - k
    forecast = floor + k * 1e6 ** (-alpha)
    print(f"alpha={alpha:.1f}: floor={floor:.3f}, loss at 1e6={forecast:.3f}")

alpha=0.1: floor=1.645, loss at 1e6=1.985
alpha=0.2: floor=2.169, loss at 1e6=2.222

Both curves hit both observations exactly. Their extrapolated losses differ by about 0.237, and their limiting floors differ by about 0.524. A perfect fit to sparse observations doesn't identify all the constants. A real five-parameter study needs enough variation in both $N$ and $D$, held-out runs, sensitivity to initialization and floor assumptions, and uncertainty estimates that include run-to-run variation.

Turn a fitted surface into a budget allocation

Once you recover $\alpha$ and $\beta$, solve for the compute-optimal parameters $N^*$ and tokens $D^*$ under your target compute budget $C$:

import math

C = 6 * 70e9 * 1.4e12
log_n = (
    math.log(alpha_fit * A / (beta_fit * B))
    + beta_fit * math.log(C / 6)
) / (alpha_fit + beta_fit)
n_opt = math.exp(log_n)
d_opt = C / (6 * n_opt)

def fixed_budget_loss(n):
    return scaling_loss(n, C / (6 * n), E, A, alpha_fit, B, beta_fit)

assert math.isclose(6 * n_opt * d_opt, C, rel_tol=1e-12)
assert fixed_budget_loss(n_opt) < fixed_budget_loss(n_opt / 2)
assert fixed_budget_loss(n_opt) < fixed_budget_loss(n_opt * 2)
print(f"Synthetic optimum: {n_opt / 1e9:.2f}B parameters, {d_opt / 1e12:.2f}T tokens")
print(f"Token/parameter ratio: {d_opt / n_opt:.0f}:1")
print(f"Loss: optimum={fixed_budget_loss(n_opt):.3f}, 70B/1.4T={target:.3f}")

Synthetic optimum: 7.82B parameters, 12.53T tokens
Token/parameter ratio: 1602:1
Loss: optimum=2.074, 70B/1.4T=2.091

That 1602:1 ratio comes from this synthetic surface, not Chinchilla's fitted data. Substituting an unrelated 20:1 rule would optimize a different assumption. Nor does 7.82B / 12.53T establish that a real corpus can supply those useful tokens.

Suppose the usable inventory is only 2T tokens and this proposed run forbids repeat epochs. Now $D\leq D_{\max}$ is a real constraint. Spending the same full compute budget requires $N\geq C/(6D_{\max})$: a smaller model can't spend all the budget without exceeding the inventory. Along this fitted fixed-budget curve, loss decreases until $N_*$ and increases afterward. Restricting the feasible interval therefore gives:

data_cap = 2e12
minimum_n = C / (6 * data_cap)
n_capped = max(n_opt, minimum_n)
d_capped = C / (6 * n_capped)

assert d_opt > data_cap
assert d_capped <= data_cap
assert math.isclose(6 * n_capped * d_capped, C, rel_tol=1e-12)
assert fixed_budget_loss(n_capped) >= fixed_budget_loss(n_opt)
assert fixed_budget_loss(n_capped) < fixed_budget_loss(2 * n_capped)
print(f"No-repeat candidate: {n_capped / 1e9:.2f}B parameters, {d_capped / 1e12:.2f}T tokens")
print(f"Loss: capped={fixed_budget_loss(n_capped):.3f}, unconstrained={fixed_budget_loss(n_opt):.3f}")

No-repeat candidate: 49.00B parameters, 2.00T tokens
Loss: capped=2.086, unconstrained=2.074

The cap moves the candidate to 49B / 2T. This is the optimum of the authored fit with these constraints, not evidence that a 49B real model will achieve the forecast. It assumes model size has no other limit and that the fit remains valid. A memory cap can make full-budget spending infeasible. If you instead permit repeated epochs, distinguish unique inventory from token presentations and check a repetition-aware fit; simply relabeling reused tokens as fresh data doesn't justify the original forecast.

Mid-scale validation gates

Before a large extrapolation, run an intermediate check, for example at 7B or 10B. Predeclare a loss tolerance and a separate throughput requirement. The $\pm 0.02$-nat tolerance below is authored, not a universal standard; choose yours using observed variability and the cost of a wrong decision. If loss misses, inspect the fit, data mix, optimizer, and numerical behavior. If throughput misses, profile communication, kernels, and utilization. Communication overhead alone doesn't explain a changed mathematical loss at the same completed training schedule.

tolerance = 0.02
mid_pred = scaling_loss(10e9, 100e9, E, A, alpha_fit, B, beta_fit)
synthetic_mid_observation = 2.350
residual = synthetic_mid_observation - mid_pred
decision = "proceed_to_further_checks" if abs(residual) <= tolerance else "hold_and_investigate"
assert decision == "hold_and_investigate"
print(f"Mid-scale predicted={mid_pred:.3f}, synthetic observed={synthetic_mid_observation:.3f}")
print(f"Absolute error={abs(residual):.3f}, tolerance={tolerance:.3f}")
print(decision)

Mid-scale predicted=2.310, synthetic observed=2.350
Absolute error=0.040, tolerance=0.020
hold_and_investigate

Failure modes, breakdowns, and boundary conditions

Scaling laws are empirical summaries of specific experimental setups, not unbending laws of physics. Understanding where these models break prevents costly cluster failures.

The 20:1 ratio is not a universal constant

The approximate ratio reflects the studied data, tokenizer, architecture, optimizer, and counting conventions. Code or synthetic mathematics isn't automatically more useful per token for your target task. Measure the intended mixture, including quality and repetition effects. Sardana et al.'s inference-aware study trained 47 models from 150M to 6B with ratios from 10 to 10,000; a fit based only on ratios up to 100 overstated gains at much higher ratios. Extrapolating a fresh-data fit to extreme overtraining needs evidence.^{[4]Reference 4Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Lawshttps://arxiv.org/abs/2401.00448}

Theoretical FLOPs are not actual hardware cluster bills

The $C \approx 6ND$ rule is a dominant-matmul arithmetic proxy. It also omits operations, including attention's sequence-length-dependent work. MFU compares useful model FLOPs per second with a stated hardware peak; its value depends on precision, architecture, batch, sequence length, and stack. Memory limits, communication, and stalls affect wall time. Measure tokens per GPU-second and total cost on the target cluster instead of treating a fixed utilization percentage as a property of all training runs.

Low cross-entropy loss is not deployed capability

Held-out cross-entropy measures next-token prediction on a specified distribution; perplexity is its exponential when loss is in nats. It doesn't directly measure factual reliability, instruction following, or coding success. Budget task evaluations and any needed post-training, such as supervised fine-tuning or preference optimization (RLHF ^{[13]Reference 13Training Language Models to Follow Instructions with Human Feedback (InstructGPT).https://arxiv.org/abs/2203.02155}). A loss target alone can't certify a coding assistant.

Emergent abilities as metric artifacts

Wei et al. ^{[14]Reference 14Emergent Abilities of Large Language Models.https://arxiv.org/abs/2206.07682} popularized the idea of "emergent abilities", reporting tasks where model performance hovered near zero percent accuracy before leaping sharply once scale crossed a threshold.

Schaeffer et al. ^{[15]Reference 15Are Emergent Abilities of Large Language Models a Mirage?https://arxiv.org/abs/2304.15004} showed cases where scoring choices and measurement resolution produced apparent emergence. In an authored independence model, if each of five answer tokens is correct with probability $p$, whole-answer correctness is $p^5$. That function is smooth, although exact-match scores on a small test set can stay at zero and then appear to jump. Partial-credit measures revealed gradual progress in the paper's case studies; they don't guarantee a power law for every capability or rule out every real transition. Inspect scoring and sample size before interpreting a threshold as a new underlying mechanism.

Architecture sensitivity: Mixture-of-Experts and state space models

Different architectures need different size variables and accounting:

• Mixture-of-Experts (MoE) (lesson): total weights drive weight storage, while selected experts plus shared layers drive much of the per-token arithmetic. Bandwidth also depends on which weights a batch touches, caching, and placement. Active-parameter $6ND$ can be a rough proxy, but routing, attention, shared components, and communication still matter. Total parameters alone don't determine compute or task quality.
• State Space Models (SSMs): fixed-size recurrent state changes sequence-memory and compute behavior. Dense attention has quadratic pairwise work during full-sequence processing; cached autoregressive attention reads a growing context per step. These aren't the same comparison as a recurrent update at fixed state size.

Measure architecture-specific loss curves and throughput rather than transferring a dense fit unchanged.