Skip to content
Skip to lesson
Back to Topics
LearnAdvanced Training & AdaptationScaling Laws & Compute-Optimal Training
⚡HardFine-Tuning & Training

Scaling Laws & Compute-Optimal Training

Kaplan, Chinchilla, and later inference-aware fits split a training budget between parameters and tokens, including when a smaller model wins on lifetime cost.

43 min read
Learning path
Step 103 of 199 in the full curriculum

Personalize this lesson

Adapt explanations and teaching visuals to your background and preferred voice.

Your team has a training budget of 102410^{24} floating-point operations (FLOPs). One proposal spends it on a large model and a short run; another trains a smaller model for longer. Which proposal should you approve, and what evidence would change your choice?

The tradeoff depends on the objective. A parameter-heavy run can spend compute inefficiently for its available token budget. A smaller model may keep improving with more tokens yet miss the lowest loss attainable at the same training compute. It may still be the better model to serve. “Not training-optimal” doesn't mean “no longer learning.”

are empirical fits that help compare these proposals. In studied regimes, power laws summarize how loss changes with compute, parameters, and tokens. Their usefulness depends on the data, architecture, training recipe, and range over which they were checked.

Kaplan et al.'s influential 2020 study fitted a parameter-heavy allocation.[1] Hoffmann et al.'s 2022 found approximately balanced growth in parameters and tokens.[2] Later work investigated why the fits differed, and how future inference demand changes the objective.[3][4] The historical Llama models illustrate long training runs at several parameter sizes; they aren't evidence for one universal inference-optimal ratio.[5]

Three symbols show up in every fit below:

  • Parameters (NN): A declared weight count. Verify whether a paper includes and output-projection weights. Tying those weights changes storage, but the vocabulary projection still costs arithmetic. Assess useful capacity through task quality at a stated compute budget, alongside parameter count.
  • Training tokens (DD): The number of token presentations processed during training, usually grouped into batches for updates. Distinguish this from unique corpus size. Repeated presentations can help, but a fresh-data fit needn't describe multi-epoch learning.
  • Compute (CC): A stated training-FLOP budget or estimate. Distinguish useful model arithmetic, recomputation, accelerator peak capacity, elapsed time, and dollars.

The first-principles derivation of 6ND

Dense transformer training has a useful arithmetic shortcut: C≈6NDC \approx 6ND FLOPs. The constant 6 comes from counting the large matrix multiplications in the forward and backward passes.

For a batch of token vectors XX and a dense matrix W∈Rdin×doutW \in \mathbb{R}^{d_{\text{in}} \times d_{\text{out}}}, the forward operation is Y=XWY=XW. Counting a multiply-accumulate as two FLOPs gives approximately 2dindout2d_{\text{in}}d_{\text{out}} FLOPs per token. This conventional GEMM estimate omits the final dot-product addition. Summing the large dense projections gives:

Cforward≈2N FLOPs per tokenC_{\text{forward}} \approx 2N \text{ FLOPs per token}

For trainable dense matrices with gradients propagated to their inputs, backpropagation has two large contractions:

  1. Activation gradients: ∇XL=(∇YL)WT\nabla_X L=(\nabla_Y L)W^T, approximately another 2N2N FLOPs per token across the counted matrices.
  2. Weight gradients: ∇WL=XT(∇YL)\nabla_W L=X^T(\nabla_Y L), approximately another 2N2N FLOPs per token, including accumulation over the batch.

Adding activation gradients (2N2N) and weight gradients (2N2N) gives 4N4N FLOPs per token for the backward pass:

Cbackward≈4N FLOPs per tokenC_{\text{backward}} \approx 4N \text{ FLOPs per token}

Summing forward (2N2N) and backward (4N4N) passes yields 6N6N FLOPs per token. Multiplied across a training run of DD tokens, total compute lands at:

Ctrain≈6ND FLOPsC_{\text{train}} \approx 6ND \text{ FLOPs}

Recomputing approximately one full counted forward pass through would raise this estimate to 8ND8ND. Selective checkpointing adds a different amount. Inference has no backward pass, so its dense-projection proxy is 2N2N FLOPs per processed token.

These proxies omit attention's query/key and attention/value products, nonlinearities, normalization, optimizer work, and other overhead. For long contexts, attention arithmetic can matter substantially. Include the vocabulary head when it isn't negligible; using a storage count that omits its operation can distort small-model comparisons.[3] FLOPs don't directly determine decode latency, which can be limited by memory traffic.

Check your arithmetic carefully. Training a 1-billion-parameter model on 20 billion tokens requires approximately 6×(1×109)×(20×109)=1.2×10206 \times (1 \times 10^9) \times (20 \times 10^9) = 1.2 \times 10^{20} FLOPs. Doubling both parameters and tokens quadruples compute to 4.8×10204.8 \times 10^{20} FLOPs.

Double parameters, double data, or both

Consider four invented measurements under a shared evaluation setup. A 1-billion-parameter baseline trains on 10 billion tokens. Three follow-up runs double model size, double data volume, or double both:

ExperimentParameters (NN)Tokens (DD)Relative compute (6ND6ND)Cross-entropy loss (nats)
Baseline1B10B1x2.80
Double model size2B10B2x2.60
Double data1B20B2x2.55
Double both2B20B4x2.35

The middle rows cost the same under 6ND6ND. In this fixture, spending that increment on tokens gives 0.05 lower loss than spending it on parameters. This supports one local choice; without repeat runs or uncertainty, it doesn't establish a general bottleneck or scaling exponent. Doubling both lowers loss further at four times the baseline compute.

Always hold evaluation conditions constant: tokenizer vocabulary, dataset distribution, context length, and loss units. Cross-entropy loss measured in natural logarithms is expressed in nats per token. Changing tokenizers alters average token length, rendering raw per-token losses incomparable.

If L=KC−αL=KC^{-\alpha}, plotting log⁡L\log L against log⁡C\log C gives a straight line. If the law has a floor, L=E+KC−αL=E+KC^{-\alpha}, the straight-line quantity is L−EL-E. Guessing the floor can change both exponent and forecast. Use proxy runs to form an initial forecast, then validate it at larger scales.

One log plot for the synthetic law L(C)=2+C to the negative 0.05 shows losses 3.000, 2.944, 2.891, 2.841, and 2.794 across equal 3.16-fold compute steps. Four bars below use a shared 0.060 scale and show the successive loss drops shrinking from 0.056 to 0.047.
Each 3.16× step moves an equal distance along the logarithmic compute axis. The drop bars share a 0.060 reference scale, so their shortening shows the diminishing absolute gain.

Inspect how diminishing returns manifest under power-law scaling using Kaplan's parameter exponent (αN≈0.076\alpha_N \approx 0.076):

power-law-diminishing-returns.py
alpha_n = 0.076

for parameter_multiplier in [2, 10, 100]:
    scaled_term = parameter_multiplier ** (-alpha_n)
    reduction = 1 - scaled_term
    print(
        f"{parameter_multiplier:>3}x parameters -> "
        f"{scaled_term:.3f}x fitted-regime loss "
        f"({reduction:.1%} reduction)"
    )
Output
  2x parameters -> 0.949x fitted-regime loss (5.1% reduction)
 10x parameters -> 0.839x fitted-regime loss (16.1% reduction)
100x parameters -> 0.705x fitted-regime loss (29.5% reduction)

Kaplan scaling laws (2020)

Kaplan et al. [1] fitted loss for dense decoder-only transformers while holding other constraints aside. Across the scales they measured, loss followed a power law in parameters, in tokens, and along the compute-efficient frontier.

The three power laws

1. Scaling with parameters (model size NN)

When training data is abundant and compute is unrestricted, model capacity forms the primary bottleneck. As parameter count NN increases, loss decreases as a power law:

L(N)∝N−αNL(N) \propto N^{-\alpha_N}

Kaplan fitted αN≈0.076\alpha_N\approx0.076 using non-embedding parameter counts. Its displayed single-axis law fits loss itself without an additive floor, so the 16% calculation concerns fitted-regime loss. Don't silently reinterpret it as a reduction in L−EL-E from a different model, or extrapolate it to zero entropy.[1]

2. Scaling with data (dataset size DD in tokens)

When model capacity is arbitrarily large and training halts before overfitting, available tokens form the bottleneck:

L(D)∝D−αDL(D) \propto D^{-\alpha_D}

Kaplan reported αD≈0.095\alpha_D \approx 0.095. Here DD describes dataset token positions, not the number of distinct vocabulary IDs. The fresh-data relationship doesn't prescribe the value of repeated epochs; those need their own checks.

3. Scaling with compute-optimal training compute (Cmin⁡C_{\min})

When allocating compute efficiently between parameters and tokens, minimum attainable loss scales as:

L(Cmin⁡)∝Cmin⁡−αCL(C_{\min}) \propto C_{\min}^{-\alpha_C}

Kaplan estimated αC≈0.050\alpha_C \approx 0.050 for its compute-efficient frontier. Cmin⁡C_{\min} includes the paper's adjustment for inefficient large batches; it isn't interchangeable with any run's measured FLOPs. The relationship concerns efficient allocation, not simply spending more on one axis.

Kaplan's compute allocation: why parameters grew faster than data

When holding compute fixed and optimizing NN and DD jointly, Kaplan et al. arrived at an unexpected conclusion:

Nopt∝C0.73,Dopt∝C0.27N_{\text{opt}} \propto C^{0.73}, \qquad D_{\text{opt}} \propto C^{0.27}

Under Kaplan's frontier, when total compute grows by 10×10\times, parameters should increase by 100.73≈5.37×10^{0.73} \approx 5.37\times, while tokens should grow by only 100.27≈1.86×10^{0.27} \approx 1.86\times. For a 100×100\times compute surge, parameters expand by 28.84×28.84\times while tokens increase by just 3.47×3.47\times.

This allocation shaped early frontier LLMs. OpenAI trained GPT-3 [6] with 175 billion parameters on only 300 billion tokens, yielding a token-to-parameter ratio of roughly 1.7:1. DeepMind followed a similar path with Gopher, building a 280-billion-parameter model on 300 billion tokens (1.07:1).

Compare the allocation multipliers directly in Python:

allocation-exponents.py
frontiers = {
    "Kaplan": (0.73, 0.27),
    "Chinchilla": (0.50, 0.50),
}

for compute_multiplier in [10, 100]:
    print(f"{compute_multiplier}x training compute")
    for name, (parameter_exp, token_exp) in frontiers.items():
        parameters = compute_multiplier ** parameter_exp
        tokens = compute_multiplier ** token_exp
        print(f"  {name:<10} N={parameters:5.2f}x, D={tokens:5.2f}x")
Output
10x training compute
  Kaplan     N= 5.37x, D= 1.86x
  Chinchilla N= 3.16x, D= 3.16x
100x training compute
  Kaplan     N=28.84x, D= 3.47x
  Chinchilla N=10.00x, D=10.00x
Normalized allocation geometry at a 100-fold training-compute increase. Equal log-unit axes show Kaplan fitted-growth normalization at 28.84× parameters and 3.47× tokens, Chinchilla fitted-growth normalization at 10× each, and a clearly marked authored alternative at 2.5× parameters and 40× tokens. The shared iso-FLOP contour is N multiplier × D multiplier = 100.
The Kaplan and Chinchilla lines are fitted-growth normalizations at the same 100× compute increase; NN and DD are growth multipliers. Historical model D/ND/N ratios describe a separate quantity. The green 2.5×/40× point is authored to complete the iso-FLOP contour; an inference-optimal point requires a quality fit, demand forecast, and cost model.

Compare the dominant-matmul estimates for two historical configurations using the 6ND6ND proxy:

the-compute-optimal-frontier.py
def dense_training_flops(parameters, tokens):
    return 6 * parameters * tokens

gpt3_style = dense_training_flops(175e9, 300e9)
small_data_rich = dense_training_flops(70e9, 1.4e12)

print(f"GPT-3-style training FLOPs: {gpt3_style:.2e}")
print(f"70B / 1.4T-token training FLOPs: {small_data_rich:.2e}")
Output
GPT-3-style training FLOPs: 3.15e+23
70B / 1.4T-token training FLOPs: 5.88e+23

The proxy gives both runs a compute budget. It does not yet say which split of parameters and tokens minimizes loss.

The Chinchilla reallocation (2022)

In 2022, Hoffmann et al. at DeepMind [2] found that several earlier models used too few training tokens relative to the allocation their loss fits favored at the same compute.

Why the allocation studies disagreed

Why did Kaplan et al. conclude that data mattered so much less than parameters?

Hoffmann et al. proposed a schedule effect: an intermediate checkpoint in a long cosine-decay run hasn't received the cooldown of a run scheduled to finish at that token budget. They matched training horizons to schedules and investigated larger proxy models. Their experiments examine one possible cause of the different allocation exponent; Porian et al.'s later reproduction tests schedule and other optimization choices separately.[2]

Porian et al.'s later reproduction identified three influential factors: counting output-head FLOPs, shortening excessive warmup for small runs, and tuning learning rate, batch size, and AdamW β2\beta_2 across scale. In their experiments, matched cosine decay improved losses but had limited effect on the allocation exponent; tuned runs with a constant learning rate could recover approximately balanced allocation.[3]

The distinction matters when designing your study. A change can lower every run's loss without greatly moving the best parameter/token split. Compare the optimum along equal-compute profiles.

Chinchilla's three methodologies and the equal-scaling derivation

Hoffmann et al. trained over 400 models spanning 70 million to 16 billion parameters across 5 billion to 500 billion tokens. They deployed three independent fitting approaches:

ApproachMethodologyParameter exponent (aa)Token exponent (bb)
1. Empirical envelopeMinimum loss envelope across 400+ runs0.500.500.500.50
2. IsoFLOP profilesSlicing runs at constant compute budgets and finding the loss minimum0.490.490.510.51
3. Parametric loss fitFitting L(N,D)=E+AN−α+BD−βL(N, D) = E + A N^{-\alpha} + B D^{-\beta} via L-BFGS0.460.460.540.54

Approach 3 lets us derive the allocation from a fitted loss surface. Assume positive coefficients and exponents:

L(N,D)=E+ANα+BDβL(N, D) = E + \frac{A}{N^\alpha} + \frac{B}{D^\beta}

Where:

  • EE is a fitted floor, interpreted in this model as the ideal data-distribution loss. Treat its fitted value as a model parameter; independent entropy measurement uses a separate estimate.
  • A/NαA/N^\alpha is the penalty for finite parameters.
  • B/DβB/D^\beta is the penalty for finite training tokens.

To find the compute-optimal allocation for a fixed budget CC, substitute D=C6ND = \frac{C}{6N} into the loss function:

L(N)=E+AN−α+B(C6N)−β=E+AN−α+B(6C)βNβL(N) = E + A N^{-\alpha} + B \left(\frac{C}{6N}\right)^{-\beta} = E + A N^{-\alpha} + B \left(\frac{6}{C}\right)^\beta N^\beta

Differentiating with respect to NN and setting the derivative to zero:

dLdN=−αAN−α−1+βB(6C)βNβ−1=0\frac{dL}{dN} = -\alpha A N^{-\alpha - 1} + \beta B \left(\frac{6}{C}\right)^\beta N^{\beta - 1} = 0

Multiplying through by NN produces an elegant balance condition:

αAN−α=βB(6NC)β=βBD−β\alpha A N^{-\alpha} = \beta B \left(\frac{6N}{C}\right)^\beta = \beta B D^{-\beta}

At the optimum, the loss reductions per proportional increase in either resource balance. These are derivatives with respect to log⁡N\log N and log⁡D\log D, not equal gains per one parameter and one token:

αANα=βBDβ\alpha \frac{A}{N^\alpha} = \beta \frac{B}{D^\beta}

Solving for N∗(C)N^*(C) and D∗(C)D^*(C) in terms of compute CC:

N∗(C)=(αAβB)1α+β(C6)βα+βN^*(C) = \left(\frac{\alpha A}{\beta B}\right)^{\frac{1}{\alpha + \beta}} \left(\frac{C}{6}\right)^{\frac{\beta}{\alpha + \beta}} D∗(C)=(βBαA)1α+β(C6)αα+βD^*(C) = \left(\frac{\beta B}{\alpha A}\right)^{\frac{1}{\alpha + \beta}} \left(\frac{C}{6}\right)^{\frac{\alpha}{\alpha + \beta}}

Notice the compute-scaling exponents:

a=βα+β,b=αα+β,a+b=1a = \frac{\beta}{\alpha + \beta}, \qquad b = \frac{\alpha}{\alpha + \beta}, \qquad a + b = 1

In DeepMind's empirical fit, α≈0.34\alpha \approx 0.34 and β≈0.28\beta \approx 0.28. Substituting those exponents gives:

a=0.280.34+0.28≈0.45,b=0.340.34+0.28≈0.55a = \frac{0.28}{0.34 + 0.28} \approx 0.45, \qquad b = \frac{0.34}{0.34 + 0.28} \approx 0.55

The rounded α,β\alpha,\beta calculation is approximately balanced; the paper reports Approach 3 as a=0.46,b=0.54a=0.46,b=0.54. Approach 1 reports 0.50/0.500.50/0.50 and Approach 2 0.49/0.510.49/0.51. With the simplified equal-growth model, both resources scale with the square root of compute:

N∝C,D∝CN \propto \sqrt{C}, \qquad D \propto \sqrt{C}

The often-used 20:1 token-to-parameter approximation is consistent with the paper's Approach 1 projections: roughly 20.2B tokens for 1B parameters, or 205.1B for 10B. It also describes the nominal 70B/1.4T Chinchilla configuration. It doesn't follow as a constant ratio from the unequal-exponent parametric fit above:

D≈20ND \approx 20N

The general fit gives D∗/N∗∝C(α−β)/(α+β)D^*/N^*\propto C^{(\alpha-\beta)/(\alpha+\beta)}. Even equal exponents α=β=γ\alpha=\beta=\gamma only give a constant ratio (B/A)1/γ(B/A)^{1/\gamma}, not a mandatory value of 20. The coefficients and their counting units matter. Use 20:1 as a declared sizing approximation, then test it against your own curves.[2]

Try changing just the coefficients. These authored unitless toy counts use γ=0.5\gamma=0.5 and fixed ND=106ND=10^6 to illustrate how coefficients change the ratio:

equal-growth-does-not-fix-the-ratio.py
import math

gamma = 0.5
product = 1e6
for a, b in ((1.0, 1.0), (2.0, 8.0)):
    ratio = (b / a) ** (1 / gamma)
    n = math.sqrt(product / ratio)
    d = ratio * n
    print(f"A={a:.0f}, B={b:.0f}: N={n:.0f}, D={d:.0f}, D/N={ratio:.0f}")
Output
A=1, B=1: N=1000, D=1000, D/N=1
A=2, B=8: N=250, D=4000, D/N=16

Both laws allocate growth equally as compute rises, yet their optimal ratios differ. With unequal exponents, even the ratio changes with budget: the rounded 0.34/0.280.34/0.28 fit makes it grow by about 1.562 when compute increases 100-fold.

Equal-unit log axes compare fitted allocation growth with compute: Kaplan parameter growth has slope 0.73, Kaplan token growth slope 0.27, and Chinchilla balanced growth slope 0.50. At 100-fold compute, their normalized multipliers are 28.84/3.47 and 10/10. A separate bar strip shows model-specific D/N ratios of 1.7:1 for GPT-3, 20:1 for Chinchilla, and 38.5:1 for Llama 3 405B.
The lines show fitted growth multipliers ending at a 100× compute increase. The bars show historical D/ND/N snapshots, which describe the training data per parameter in each configuration. A snapshot ratio and a fitted frontier slope answer different questions.

Size a candidate from a target FLOP budget using a declared 20:1 approximation:

compute-budget-to-chinchilla-size.py
def chinchilla_style_size(training_flops, tokens_per_parameter=20):
    import math
    if not all(math.isfinite(x) and x > 0 for x in (training_flops, tokens_per_parameter)):
        raise ValueError("FLOP budget and token/parameter ratio must be finite and positive")
    parameters = (training_flops / (6 * tokens_per_parameter)) ** 0.5
    tokens = tokens_per_parameter * parameters
    return parameters, tokens

parameters, tokens = chinchilla_style_size(1e24)
print(f"Parameters at 1e24 FLOPs: {parameters / 1e9:.1f}B")
print(f"Tokens at 20:1: {tokens / 1e12:.2f}T")
Output
Parameters at 1e24 FLOPs: 91.3B
Tokens at 20:1: 1.83T

Chinchilla vs. Gopher: a frontier-scale test

DeepMind reported training nominally 70B-parameter Chinchilla on 1.4T tokens at Gopher's training-FLOP budget. The paper's estimated optimal-size range was 40B–70B, and it selected the larger end partly for data and computational efficiency. The empirical fit and budget define the tested scope; the exact global optimum remains unknown.[2]

ModelNominal parametersTokensNominal D/ND/NPaper's shared budgetRounded 6ND6ND proxy
Gopher280B300B1.07:15.76×10235.76\times10^{23}5.04×10235.04\times10^{23}
Chinchilla70B1.4T20:15.76×10235.76\times10^{23}5.88×10235.88\times10^{23}

The shared budget above is the figure used for the paper's frontier projections. Appendix F also reports a revised estimate of 6.3×10236.3\times10^{23} with its more detailed accounting, versus the earlier 5.76×10235.76\times10^{23}. The rounded 6ND6ND proxy doesn't reproduce either exactly; counts and accounting formulas aren't interchangeable. Chinchilla improved broad evaluation results, including average MMLU, BIG-bench, and LAMBADA, but not every individual task: it underperformed Gopher on four MMLU tasks. The models also differed in optimizer, tokenizer, and data mixture, so this comparison doesn't isolate only the allocation.[2]

At the same weight precision, one-fourth the parameters implies roughly one-fourth the weight storage and dense-projection arithmetic. Total serving memory also includes KV cache and workspace; elapsed latency and dollars require profiling.

Compute a 10-billion-parameter Chinchilla-style training budget:

chinchilla-style-budget.py
def chinchilla_tokens(parameters, tokens_per_parameter=20):
    return parameters * tokens_per_parameter

def dense_training_flops(parameters, tokens):
    return 6 * parameters * tokens

parameters = 10e9
tokens = chinchilla_tokens(parameters)
flops = dense_training_flops(parameters, tokens)

print(f"Chinchilla-style tokens for 10B parameters: {tokens:.2e}")
print(f"Dense training FLOPs: {flops:.2e}")
Output
Chinchilla-style tokens for 10B parameters: 2.00e+11
Dense training FLOPs: 1.20e+22

Beyond Chinchilla: inference-aware scaling

Chinchilla scaling solves a specific optimization problem: finding the lowest pre-training loss for a fixed training compute budget (CtrainC_{\text{train}}). Its objective excludes serving cost; it doesn't assume the model is literally never used.

Serving can eventually dominate lifetime cost, so compare token counts using the same arithmetic proxies. Under 6ND6ND training and 2NQ2NQ inference, inference exceeds training at Q>3DQ>3D. For a run with 1.4T training tokens, this threshold is more than 4.2T processed inference tokens.

Here overtraining means training beyond a declared training-optimal token allocation for that size. We use 20:1 as a simple historical baseline; the optimal allocation varies across fits. This differs from overfitting, and validation loss may fluctuate as token allocation grows. Overtraining can improve a compact model while using more training compute than a different size would need to reach the same fitted loss.

The lifetime economic break-even equation

Sardana et al. [4] formalized inference-aware scaling by modeling the total lifetime compute bill:

Ctotal=Ctrain+Cinfer=6ND+2N⋅Q=2N(3D+Q)C_{\text{total}} = C_{\text{train}} + C_{\text{infer}} = 6ND + 2N \cdot Q = 2N(3D + Q)

Where:

  • QQ counts lifetime processed inference tokens, treating prompt and completion tokens alike in this simplified proxy. Reused cached prefixes don't require a new full forward pass for every billed input token.
  • 2N2N counts the large dense projections. Context-dependent attention and serving overhead aren't included.

Suppose two candidates achieve the same held-out pre-training cross-entropy under a shared evaluation setup. This doesn't automatically establish equal deployed task quality:

  • Model 1 (Chinchilla-optimal): Larger model (N1N_1) trained on fewer tokens (D1≈20N1D_1 \approx 20N_1).
  • Model 2 (Overtrained compact model): Smaller model (N2<N1N_2 < N_1) overtrained on a massive token corpus (D2≫20N2D_2 \gg 20N_2).

If Model 1 is training-optimal for that target loss in the fit, Model 2 needs at least as much training compute. For a general pair of candidates, measure the difference rather than assuming it's positive:

ΔCtrain=6(N2D2−N1D1)\Delta C_{\text{train}} = 6(N_2 D_2 - N_1 D_1)

However, Model 2 saves compute on every single served token during deployment:

ΔCinfer/token=2(N1−N2) FLOPs per token\Delta C_{\text{infer/token}} = 2(N_1 - N_2) \text{ FLOPs per token}

The break-even processed-token volume (Qbreak-evenQ_{\text{break-even}}), not number of requests, is where accumulated inference savings offset a positive training premium:

6N2D2+2N2Q=6N1D1+2N1Q6N_2 D_2 + 2N_2 Q = 6N_1 D_1 + 2N_1 Q 2(N1−N2)Q=6(N2D2−N1D1)2(N_1 - N_2)Q = 6(N_2 D_2 - N_1 D_1) Qbreak-even=6(N2D2−N1D1)2(N1−N2)=3(N2D2−N1D1)N1−N2=ΔCtrainΔCinfer/tokenQ_{\text{break-even}} = \frac{6(N_2 D_2 - N_1 D_1)}{2(N_1 - N_2)} = \frac{3(N_2 D_2 - N_1 D_1)}{N_1 - N_2} = \frac{\Delta C_{\text{train}}}{\Delta C_{\text{infer/token}}}

Above a positive break-even, the smaller model wins on proxy FLOPs, assuming equal acceptable quality and the same demand. If its training difference is nonpositive, it is already no more expensive at zero demand. If parameter counts are equal, this proxy gives no per-token saving to divide by.

Dollar break-even may differ. With train/inference prices per useful FLOP pt,pip_t,p_i, the crossing is Q=ptΔCtrain/(piΔCinfer/token)Q=p_t\Delta C_{\rm train}/(p_i\Delta C_{\rm infer/token}). Sardana et al. separately model real-world costs and different prompt/decode utilization. Batch size, memory traffic, caching, precision, and latency targets can change those costs.[4]

Concrete break-even example

Consider two authored configurations and assume they meet the same task-quality requirement. The arithmetic below doesn't establish that assumption; actual evaluations must:

  • Model A (Chinchilla-style): 70B parameters trained on 1.4T tokens. Training cost ≈6×70B×1.4T≈5.88×1023\approx 6 \times 70\text{B} \times 1.4\text{T} \approx 5.88 \times 10^{23} FLOPs. Inference cost per token ≈2×70B=140\approx 2 \times 70\text{B} = 140B FLOPs.
  • Model B (Inference-aware overtrained): 30B parameters trained on 4.0T tokens. Training cost ≈6×30B×4.0T≈7.20×1023\approx 6 \times 30\text{B} \times 4.0\text{T} \approx 7.20 \times 10^{23} FLOPs. Inference cost per token ≈2×30B=60\approx 2 \times 30\text{B} = 60B FLOPs.

Model B requires 1.32×10231.32 \times 10^{23} extra training FLOPs upfront. But it saves 140B−60B=80B140\text{B} - 60\text{B} = 80\text{B} FLOPs on every served token.

Assuming equal model quality, total training-plus-serving FLOPs cross at 1.65 trillion served tokens. At the crossing each candidate costs 8.19×10²³ FLOPs: the 70B model is 5.88 training plus 2.31 serving, while the 30B model is 7.20 training plus 0.99 serving. Stacked bars share the same total scale.
Under the equal-quality assumption, both candidates total 8.19 × 10²³ FLOPs at 1.65T served tokens. The 70B candidate spends less upfront and more on every served token; the 30B candidate spends more upfront and less per served token.

Calculate the exact break-even token volume in Python:

concrete-break-even-example.py
def train_flops(parameters, tokens):
    return 6 * parameters * tokens

def inference_flops(parameters, served_tokens):
    return 2 * parameters * served_tokens

model_a_train = train_flops(70e9, 1.4e12)
model_b_train = train_flops(30e9, 4e12)
extra_train = model_b_train - model_a_train
savings_per_token = 2 * (70e9 - 30e9)
break_even_tokens = extra_train / savings_per_token

print(f"Model A training FLOPs: {model_a_train:.2e}")
print(f"Model B training FLOPs: {model_b_train:.2e}")
print(f"Break-even served tokens: {break_even_tokens:.2e}")
Output
Model A training FLOPs: 5.88e+23
Model B training FLOPs: 7.20e+23
Break-even served tokens: 1.65e+12

Evaluate total lifetime compute at 1 trillion versus 3 trillion served tokens:

lifetime-cost-above-and-below-break-even.py
def total_flops(parameters, training_tokens, served_tokens):
    return 6 * parameters * training_tokens + 2 * parameters * served_tokens

models = {
    "70B / 1.4T": (70e9, 1.4e12),
    "30B / 4.0T": (30e9, 4.0e12),
}

for demand in [1e12, 3e12]:
    costs = {
        name: total_flops(parameters, tokens, demand)
        for name, (parameters, tokens) in models.items()
    }
    cheaper = min(costs, key=costs.get)
    print(f"{demand / 1e12:.0f}T served tokens -> cheaper candidate: {cheaper}")
Output
1T served tokens -> cheaper candidate: 70B / 1.4T
3T served tokens -> cheaper candidate: 30B / 4.0T

Historical configurations: the Llama trajectory

These published configurations show how token budgets can differ from the approximate 20:1 training-only baseline. Ratios below use nominal model sizes, so they are rounded:

Longer training can improve a compact model whose per-token arithmetic is lower. Whether that pays for itself depends on actual quality, demand, and serving costs. A lifetime optimum requires those quality and cost measurements for the target workload. Changes in tokenizer, data mix, and architecture also mean this chronology is not a controlled scaling experiment.

Public-text supply as a planning constraint

Overtraining assumes an abundant data supply. However, high-quality human-generated text on the public internet is finite.

Villalobos et al.'s June 2024 forecast [9] projected that growing training-data requirements could meet effective public human-text supply during 2026–2032, with a median around 2028. Its roughly 4×10144 \times 10^{14}-token figure estimates effective stock at that intersection, accounting for quality filtering and useful repetition. The forecast describes a possible future supply boundary; assessing current unique usable text and any present bottleneck requires current data inventories.

Text isn't physically consumed by training. Four runs can reuse the same corpus:

Authored four-run scenarioToken presentations across all runsUnique text in their union
Each uses the same 15T-token corpus once60T15T
Each uses a disjoint 15T-token corpus once60T60T

Only the second scenario has four times the unique-data requirement. Even then, dividing raw unique tokens by a quality/repetition-adjusted effective stock mixes different definitions. Access rights, filtering losses, distribution, and the value of repeat epochs matter more than a count of model variants.

Calculate a nominal ratio, then ask what its denominator means:

projected-public-text-context.py
effective_stock = 4e14  # Villalobos et al.: ~400T effective stock at utilization
llama3_405b_tokens = 15.6e12
chinchilla_style_tokens = 20 * 405e9

print(f"Llama 3 405B reported tokens: {llama3_405b_tokens / 1e12:.1f}T")
print(f"Chinchilla-style 405B tokens: {chinchilla_style_tokens / 1e12:.1f}T")
print(
    "Reported token budget / projected effective stock: "
    f"{llama3_405b_tokens / effective_stock:.1%}"
)
print("Numerical comparison only; not a fraction of unique supply used.")
Output
Llama 3 405B reported tokens: 15.6T
Chinchilla-style 405B tokens: 8.1T
Reported token budget / projected effective stock: 3.9%
Numerical comparison only; not a fraction of unique supply used.

Possible responses include useful repeat epochs, verified synthetic data, and additional domains or modalities. Measure how these sources affect quality and compute before carrying a fresh-data scaling relation forward. A September 2025 Epoch AI assessment judged a data bottleneck potentially surmountable with synthetic data; its conclusion still depends on supply and quality assumptions. Test-time reasoning offers another compute allocation, while pre-training evidence comes from the training data.

A new scaling axis: test-time compute

Pre-training fixes model weights. Once trained, serving introduces an orthogonal scaling axis: .

Additional inference work can mean longer reasoning traces, multiple sampled candidates, or search scored by an external verifier. These are different mechanisms: DeepSeek-R1's reasoning traces [10] don't establish that it executes an external verifier's tree search on every request.

Snell et al. [11] studied MATH with PaLM 2-S* and task-specific revision/verifier training. With compute allocated according to problem difficulty, their smaller model could outperform a single-response model with roughly 14 times as many parameters in some matched-compute scenarios. Harder questions and high inference demand could instead favor the larger pre-trained model. The finding applies to that task, model, training setup, and compute budget; estimating difficulty and scoring candidates also cost work.

Suppose every candidate generates 512 tokens, with no early stopping. Sixteen candidates produce 16 times as many output tokens. This example counts their dense generation arithmetic; it excludes prompt processing, attention, and any verifier:

sampled-candidates-raise-generation-cost.py
def generation_flops(parameters, output_tokens_per_candidate, candidates):
    return 2 * parameters * output_tokens_per_candidate * candidates

parameters = 8e9
output_tokens_per_candidate = 512
for candidates in [1, 4, 16]:
    flops = generation_flops(parameters, output_tokens_per_candidate, candidates)
    print(f"{candidates:>2} candidate(s): {flops:.2e} generation FLOPs per request")
Output
 1 candidate(s): 8.19e+12 generation FLOPs per request
 4 candidate(s): 3.28e+13 generation FLOPs per request
16 candidate(s): 1.31e+14 generation FLOPs per request

From theory to practice: running a scaling study

A power-law fit helps only if a smaller run can predict a larger one. A scaling study fits loss on proxy models, extrapolates to the target budget, and stops at an intermediate gate when the forecast misses.

Start with proxy runs that vary parameter count and token budget, fit a declared loss surface, and record the target-scale forecast. Reserve a measured mid-scale run as the gate: compare its loss with a predeclared tolerance, then either revise the fit or commit to the target run.

Zero-shot hyperparameter transfer with maximal update parametrization (µP)

An unchanged numerical learning rate can produce different function updates as width grows. Maximal Update Parametrization (μ\muP) [12] controls that dependence through coordinated initialization, multipliers, and optimizer settings. Rules depend on layer role and optimizer; an SGD example is not an Adam recipe for every transformer.

See the mechanism in the paper's two-layer linear setup: f(x)=V⊤Uxf(x)=V^\top Ux, with scalar x=1x=1 and squared loss 12(f−1)2\tfrac12(f-1)^2. We'll use authored deterministic vectors: alternating Ui=±1U_i=\pm1, and equal ViV_i so the initial prediction is zero. Both sets of gradients must use the old weights. Before running, predict which update will grow with width:

width-and-a-single-sgd-update.py
import math

def one_step(u, v, input_lr, output_lr):
    prediction = math.fsum(a * b for a, b in zip(u, v))
    error = prediction - 1.0
    new_u = [a - input_lr * error * b for a, b in zip(u, v)]
    new_v = [b - output_lr * error * a for a, b in zip(u, v)]
    return math.fsum(a * b for a, b in zip(new_u, new_v))

for width in (16, 64, 256):
    u = [1.0 if i % 2 == 0 else -1.0 for i in range(width)]
    standard = one_step(u, [1 / math.sqrt(width)] * width, 1.0, 1.0)
    mup = one_step(u, [1 / width] * width, width, 1 / width)
    print(f"width={width}: standard prediction={standard:.1f}, muP prediction={mup:.1f}")
Output
width=16: standard prediction=17.0, muP prediction=2.0
width=64: standard prediction=65.0, muP prediction=2.0
width=256: standard prediction=257.0, muP prediction=2.0

The standard update grows as n+1n+1. The coordinated μ\muP update stays at 2 because this example scales the input-layer SGD rate up by nn, the output-layer rate down by nn, and ViV_i as 1/n1/n. It still overshoots the target 1: width stability doesn't select a good base learning rate for you.

The study demonstrated learning-rate transfer from a 40M proxy to 6.7B GPT-3. At 70B, finite-width optima can shift, and depth or architecture changes require checks. Regularization settings such as dropout and weight decay may also need retuning. Test the intended width scaling and validate a larger run before spending the full budget.

Fitting the parametric loss surface

In a scaling study, you train a grid of small proxy models (for example, 100M to 1B parameters) across varying token counts and measure validation cross-entropy. You then fit the five constants of Chinchilla Approach 3:

L(N,D)=E+ANα+BDβL(N, D) = E + \frac{A}{N^\alpha} + \frac{B}{D^\beta}

Consider a synthetic benchmark surface generated from:

L(N,D)=1.10+2.80N0.070+7.80D0.098L(N, D) = 1.10 + \frac{2.80}{N^{0.070}} + \frac{7.80}{D^{0.098}}
Parameters (NN)Tokens (DD)Synthetic loss (nats)
100M1B2.895
100M5B2.745
100M20B2.634
500M1B2.813
500M5B2.663
500M20B2.552
1B5B2.631
1B20B2.520
1B100B2.408
Log-log map of eight formula-generated proxy points, a distinct held-out point at 1B parameters and 100B tokens, and a forecast marker at 70B parameters and 1.4T tokens. The fit region covers only 0.1B–1B parameters and 1B–20B tokens. The authored fit predicts 2.408 at holdout versus 2.408 observed, residual +0.0002, and forecasts 2.091 at the target.
The dashed box shows the eight proxy inputs; the held-out input at 1B / 100B and the 70B / 1.4T forecast lie outside it. Both values come from the authored surface, so the holdout checks the fitting routine rather than model quality. A real study still needs a measured mid-scale .

For this controlled exercise, we supply the true EE, AA, and BB and search only the two exponents. Hold out the point (1B / 100B), then forecast loss at 70B / 1.4T. Recovering the exponents from their own rounded generating formula is much easier than fitting all five unknowns to noisy training runs:

fit-exponents-and-forecast.py
def scaling_loss(n, d, e, a, alpha, b, beta):
    return e + a / (n ** alpha) + b / (d ** beta)

E, A, B = 1.10, 2.80, 7.80
experiments = [
    (1e8, 1e9, 2.895),
    (1e8, 5e9, 2.745),
    (1e8, 2e10, 2.634),
    (5e8, 1e9, 2.813),
    (5e8, 5e9, 2.663),
    (5e8, 2e10, 2.552),
    (1e9, 5e9, 2.631),
    (1e9, 2e10, 2.520),
    (1e9, 1e11, 2.408),
]
train, held = experiments[:-1], experiments[-1]

def sum_squared_error(alpha, beta):
    total = 0.0
    for n, d, loss in train:
        pred = scaling_loss(n, d, E, A, alpha, B, beta)
        total += (pred - loss) ** 2
    return total

best_error = float("inf")
best = (0.0, 0.0)
for alpha_i in range(40, 121, 2):
    for beta_i in range(40, 121, 2):
        alpha = alpha_i / 1000
        beta = beta_i / 1000
        error = sum_squared_error(alpha, beta)
        if error < best_error:
            best_error = error
            best = (alpha, beta)

alpha_fit, beta_fit = best
n_held, d_held, loss_held = held
hold_pred = scaling_loss(n_held, d_held, E, A, alpha_fit, B, beta_fit)
target = scaling_loss(70e9, 1.4e12, E, A, alpha_fit, B, beta_fit)

print(f"Recovered alpha={alpha_fit:.3f}, beta={beta_fit:.3f}")
print(
    f"Hold-out 1B / 100B: pred={hold_pred:.3f}, "
    f"observed={loss_held:.3f}, residual={hold_pred - loss_held:+.4f}"
)
print(f"Forecast 70B / 1.4T tokens: {target:.3f}")
Output
Recovered alpha=0.070, beta=0.098
Hold-out 1B / 100B: pred=2.408, observed=2.408, residual=+0.0002
Forecast 70B / 1.4T tokens: 2.091

Two measurements can hide very different forecasts

What if the loss floor is unknown? Use normalized compute and fit two authored observations, L(1)=3L(1)=3 and L(100)=2.5L(100)=2.5, with L(C)=E+KC−αL(C)=E+KC^{-\alpha}. For each chosen exponent, solve for KK and EE:

two-anchors-do-not-identify-a-loss-floor.py
for alpha in (0.1, 0.2):
    k = (3.0 - 2.5) / (1.0 - 100.0 ** (-alpha))
    floor = 3.0 - k
    forecast = floor + k * 1e6 ** (-alpha)
    print(f"alpha={alpha:.1f}: floor={floor:.3f}, loss at 1e6={forecast:.3f}")
Output
alpha=0.1: floor=1.645, loss at 1e6=1.985
alpha=0.2: floor=2.169, loss at 1e6=2.222
Two authored power-law curves both pass exactly through the normalized observations L(1)=3 and L(100)=2.5. With alpha 0.1, the fitted floor is 1.645 and loss at 1,000,000 is 1.985; with alpha 0.2, the floor is 2.169 and the forecast is 2.222. The curves separate beyond the two shared anchors.
Both curves are authored to pass through the same normalized observations. Changing α\alpha changes the fitted floor and the one-million-compute forecast. Repeated runs would be needed to estimate empirical variation.

Both curves hit both observations exactly. Their extrapolated losses differ by about 0.237, and their limiting floors differ by about 0.524. A perfect fit to sparse observations doesn't identify all the constants. A real five-parameter study needs enough variation in both NN and DD, held-out runs, sensitivity to initialization and floor assumptions, and uncertainty estimates that include run-to-run variation.

Turn a fitted surface into a budget allocation

Once you recover α\alpha and β\beta, solve for the compute-optimal parameters N∗N^* and tokens D∗D^* under your target compute budget CC:

N∗=[αAβB(C6)β]1/(α+β),D∗=C6N∗N_* = \left[\frac{\alpha A}{\beta B}\left(\frac{C}{6}\right)^\beta\right]^{1/(\alpha+\beta)}, \qquad D_* = \frac{C}{6N_*}
optimize-the-fitted-surface.py
import math

C = 6 * 70e9 * 1.4e12
log_n = (
    math.log(alpha_fit * A / (beta_fit * B))
    + beta_fit * math.log(C / 6)
) / (alpha_fit + beta_fit)
n_opt = math.exp(log_n)
d_opt = C / (6 * n_opt)

def fixed_budget_loss(n):
    return scaling_loss(n, C / (6 * n), E, A, alpha_fit, B, beta_fit)

assert math.isclose(6 * n_opt * d_opt, C, rel_tol=1e-12)
assert fixed_budget_loss(n_opt) < fixed_budget_loss(n_opt / 2)
assert fixed_budget_loss(n_opt) < fixed_budget_loss(n_opt * 2)
print(f"Synthetic optimum: {n_opt / 1e9:.2f}B parameters, {d_opt / 1e12:.2f}T tokens")
print(f"Token/parameter ratio: {d_opt / n_opt:.0f}:1")
print(f"Loss: optimum={fixed_budget_loss(n_opt):.3f}, 70B/1.4T={target:.3f}")
Output
Synthetic optimum: 7.82B parameters, 12.53T tokens
Token/parameter ratio: 1602:1
Loss: optimum=2.074, 70B/1.4T=2.091

That 1602:1 ratio comes from this synthetic surface, not Chinchilla's fitted data. Substituting an unrelated 20:1 rule would optimize a different assumption. Nor does 7.82B / 12.53T establish that a real corpus can supply those useful tokens.

Suppose the usable inventory is only 2T tokens and this proposed run forbids repeat epochs. Now D≤Dmax⁡D\leq D_{\max} is a real constraint. Spending the same full compute budget requires N≥C/(6Dmax⁡)N\geq C/(6D_{\max}): a smaller model can't spend all the budget without exceeding the inventory. Along this fitted fixed-budget curve, loss decreases until N∗N_* and increases afterward. Restricting the feasible interval therefore gives:

Ncapped=max⁡ ⁣(N∗,C6Dmax⁡),Dcapped=C6NcappedN_{\text{capped}}=\max\!\left(N_*,\frac{C}{6D_{\max}}\right),\qquad D_{\text{capped}}=\frac{C}{6N_{\text{capped}}}
allocate-with-a-no-repeat-data-cap.py
data_cap = 2e12
minimum_n = C / (6 * data_cap)
n_capped = max(n_opt, minimum_n)
d_capped = C / (6 * n_capped)

assert d_opt > data_cap
assert d_capped <= data_cap
assert math.isclose(6 * n_capped * d_capped, C, rel_tol=1e-12)
assert fixed_budget_loss(n_capped) >= fixed_budget_loss(n_opt)
assert fixed_budget_loss(n_capped) < fixed_budget_loss(2 * n_capped)
print(f"No-repeat candidate: {n_capped / 1e9:.2f}B parameters, {d_capped / 1e12:.2f}T tokens")
print(f"Loss: capped={fixed_budget_loss(n_capped):.3f}, unconstrained={fixed_budget_loss(n_opt):.3f}")
Output
No-repeat candidate: 49.00B parameters, 2.00T tokens
Loss: capped=2.086, unconstrained=2.074

The cap moves the candidate to 49B / 2T, the optimum of the authored fit with these constraints. Validate that forecast with real training runs; this fit assumes no additional model-size limit and remains valid only under its stated conditions. A memory cap can make full-budget spending infeasible. If you permit repeated epochs, distinguish unique inventory from token presentations and check a repetition-aware fit; relabeling reused tokens as fresh data doesn't justify the original forecast.

Mid-scale validation gates

Before a large extrapolation, run an intermediate check, for example at 7B or 10B. Predeclare a loss tolerance and a separate throughput requirement. The ±0.02\pm 0.02-nat tolerance below is an example policy value; choose one using observed variability and the cost of a wrong decision. If loss misses, inspect the fit, data mix, optimizer, and numerical behavior. If throughput misses, profile communication, kernels, and utilization. Communication overhead alone doesn't explain a changed mathematical loss at the same completed training schedule.

stop-on-a-mid-scale-miss.py
tolerance = 0.02
mid_pred = scaling_loss(10e9, 100e9, E, A, alpha_fit, B, beta_fit)
synthetic_mid_observation = 2.350
residual = synthetic_mid_observation - mid_pred
decision = "proceed_to_further_checks" if abs(residual) <= tolerance else "hold_and_investigate"
assert decision == "hold_and_investigate"
print(f"Mid-scale predicted={mid_pred:.3f}, synthetic observed={synthetic_mid_observation:.3f}")
print(f"Absolute error={abs(residual):.3f}, tolerance={tolerance:.3f}")
print(decision)
Output
Mid-scale predicted=2.310, synthetic observed=2.350
Absolute error=0.040, tolerance=0.020
hold_and_investigate

Failure modes, breakdowns, and boundary conditions

Scaling laws are empirical summaries of specific experimental setups, not unbending laws of physics. Understanding where these models break prevents costly cluster failures.

The 20:1 ratio depends on the training setup

The approximate ratio reflects the studied data, tokenizer, architecture, optimizer, and counting conventions. Code or synthetic mathematics helps when its quality and repetition profile fit the target task. Measure the intended mixture, including those effects. Sardana et al.'s inference-aware study trained 47 models from 150M to 6B with ratios from 10 to 10,000; a fit based only on ratios up to 100 overstated gains at much higher ratios. Extrapolating a fresh-data fit to extreme overtraining needs evidence.[4]

Theoretical FLOPs are not actual hardware cluster bills

The C≈6NDC \approx 6ND rule is a dominant-matmul arithmetic proxy. It also omits operations, including attention's sequence-length-dependent work. MFU compares useful model FLOPs per second with a stated hardware peak; its value depends on precision, architecture, batch, sequence length, and stack. Memory limits, communication, and stalls affect wall time. Measure tokens per GPU-second and total cost on the target cluster instead of treating a fixed utilization percentage as a property of all training runs.

Low cross-entropy loss is not deployed capability

Held-out cross-entropy measures next-token prediction on a specified distribution; perplexity is its exponential when loss is in nats. It doesn't directly measure factual reliability, instruction following, or coding success. Budget task evaluations and any needed post-training, such as supervised fine-tuning or preference optimization ( [13]). A loss target alone can't certify a coding assistant.

Emergent abilities as metric artifacts

Wei et al. [14] popularized the idea of "emergent abilities", reporting tasks where model performance hovered near zero percent accuracy before leaping sharply once scale crossed a threshold.

Schaeffer et al. [15] showed cases where scoring choices and measurement resolution produced apparent emergence. In an authored independence model, if each of five answer tokens is correct with probability pp, whole-answer correctness is p5p^5. That function is smooth, although exact-match scores on a small test set can stay at zero and then appear to jump. Partial-credit measures revealed gradual progress in the paper's case studies; other capabilities may follow different curves, and some thresholds may correspond to real transitions. Inspect scoring and sample size before interpreting a threshold as a new underlying mechanism.

Architecture sensitivity: Mixture-of-Experts and state space models

Different architectures need different size variables and accounting:

  • (lesson): total weights drive weight storage, while selected experts plus shared layers drive much of the per-token arithmetic. Bandwidth also depends on which weights a batch touches, caching, and placement. Active-parameter 6ND6ND can be a rough proxy, but routing, attention, shared components, and communication still matter. Total parameters alone don't determine compute or task quality.
  • State Space Models (SSMs): fixed-size recurrent state changes sequence-memory and compute behavior. Dense attention has quadratic pairwise work during full-sequence processing; cached autoregressive attention reads a growing context per step. These aren't the same comparison as a recurrent update at fixed state size.

Measure architecture-specific loss curves and throughput rather than transferring a dense fit unchanged.

Turn a FLOP budget into a candidate split of parameters and tokens, and name which objective that split optimizes. Do not copy a dense fit onto an architecture or a serving bill it never measured.

Complete the lesson

Mastery Check

Answer every question, then check your score. Score 75% or higher to mark this lesson complete.

1.A dense-projection training proxy counts forward, input-gradient and weight-gradient contractions. A 2B-parameter model processes 30B token presentations. Selective checkpointing is replaced by recomputing one complete counted forward pass. Ignoring other work, what ordinary and recomputed totals follow?
2.An authored law is L=1+3/√N+6/√D with fixed ND=400. Which allocation minimizes this positive half-power surface?
3.A positive two-penalty fit has parameter exponent α=0.6 and token exponent β=0.3. Under the same 6ND constraint and fixed coefficients, compute increases eightfold. What optimal parameter and token multipliers follow?
4.Two authored fits both match L(4)=4 and L(16)=3: F(C)=2+4C^(-1/2), G(C)=8/3+(16/3)C^(-1). What forecasts at C=64 follow, and what do the two exact anchor fits establish?
5.Assume two candidates meet the same deployed quality requirement. A is 40B parameters trained on 800B tokens; B is 20B on 2T. Use 6ND training and 2N per processed inference token. Training costs twice as many dollars per useful FLOP as serving. At what processed demand do proxy FLOPs and modeled dollars break even?
6.A positive fitted surface has its unconstrained minimum at N=10, D=90 along ND=900, and loss rises on either side. Repeats are forbidden and at most 30 token units exist. First allow any model size; then separately impose N≤20 as well. What full-budget outcomes follow?
7.A 6B model generates three complete 256-token candidates per query without early stopping. Count only dense generation arithmetic. A second configuration doubles candidate count at the same length. Which first-configuration FLOPs and second-configuration cost multiplier follow?
8.Use f=ΣUᵢVᵢ at input 1 and loss ½(f−1)². U alternates +1/−1 at even widths. With base η=0.25, standard Vᵢ=1/√n and both rates η. The coordinated alternative uses Vᵢ=1/n, input rate ηn, output rate η/n. Both gradients use old weights. What new predictions at n=32 and n=128 follow?
9.A supplied MoE accounting toy stores 96B total parameters at 2 bytes each. Per token, it executes 8B selected expert parameters plus 4B shared parameters. For 50B presentations, what total weight payload and active dense-projection 6ND proxy follow, excluding routing, attention and communication?
10.In an authored model, four answer tokens are independently correct with probability p each. Compare p=0.7 and p=0.8. A small exact-match evaluation reports 0 then 2 complete successes. Which expected whole-answer probabilities and interpretation follow?

10 questions remaining.

References

Scaling Laws for Neural Language Models

Kaplan et al. · 2020

https://arxiv.org/abs/2001.08361

Training Compute-Optimal Large Language Models.

Hoffmann, J., et al. · 2022 · NeurIPS 2022

https://arxiv.org/abs/2203.15556

Resolving Discrepancies in Compute-Optimal Scaling of Language Models

Porian, T., Wortsman, M., Jitsev, J., Schmidt, L., & Carmon, Y. · 2024

https://arxiv.org/abs/2406.19146

Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws

Nikhil Sardana, Jacob Portes, Sasha Doubov, and Jonathan Frankle · 2024 · ICML 2024

https://arxiv.org/abs/2401.00448

The Llama 3 Herd of Models.

Dubey, A., et al. · 2024 · arXiv preprint

https://arxiv.org/abs/2407.21783

Language Models are Few-Shot Learners.

Brown, T., et al. · 2020 · NeurIPS 2020

https://arxiv.org/abs/2005.14165

LLaMA: Open and Efficient Foundation Language Models.

Touvron, H., et al. · 2023

https://arxiv.org/abs/2302.13971

Llama 2: Open Foundation and Fine-Tuned Chat Models.

Touvron, H., et al. · 2023 · arXiv preprint

https://arxiv.org/abs/2307.09288

Will We Run Out of Data? Limits of LLM Scaling Based on Human-Generated Data

Villalobos, P., et al. (Epoch AI) · 2024 · arXiv preprint

https://arxiv.org/abs/2211.04325v2

DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

DeepSeek-AI · 2025

https://arxiv.org/abs/2501.12948

Scaling LLM Test-Time Compute Optimally Can Be More Effective Than Scaling Model Parameters.

Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar · 2024 · ICLR 2025

https://arxiv.org/abs/2408.03314

Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer

Yang, G., et al. · 2022 · NeurIPS 2021

https://arxiv.org/abs/2203.03466

Training Language Models to Follow Instructions with Human Feedback (InstructGPT).

Ouyang, L., et al. · 2022 · NeurIPS 2022

https://arxiv.org/abs/2203.02155

Emergent Abilities of Large Language Models.

Wei, J., et al. · 2022 · TMLR

https://arxiv.org/abs/2206.07682

Are Emergent Abilities of Large Language Models a Mirage?

Schaeffer, R., et al. · 2023 · NeurIPS 2023

https://arxiv.org/abs/2304.15004

Discussion

Questions and insights from fellow learners.

Discussion loads when you reach this section.