Layer Normalization: Pre-LN vs Post-LN

A decoder keeps one hidden vector for each token as it moves through a stack. That shared per-token state is the residual stream. Suppose one state is [3.0, 1.0, -1.0, 5.0]: each transformer block adds an attention update and a feed-forward update to the same vector. After many additions, one coordinate can dominate later dot products or output scores, while the backward signal becomes uneven across layers.

LayerNorm gives each token a scale checkpoint. It measures that token's feature values, centers them, and rescales their spread; learned scale and shift can then restore useful asymmetry. The placement question is just as important as the arithmetic: does the residual shortcut bypass this checkpoint, or does it cross the checkpoint after every block?

The previous lesson's RoPE and ALiBi encode token position. They don't regulate residual magnitude. Keep the vector visible as we move from LayerNorm's arithmetic to Pre-LN and Post-LN blocks, then use small diagnostics to test what changes at depth.

---

One residual stream, many writes

Start with the block abstraction output = input + sublayer(input). The input takes a skip path, while the sublayer writes an attention or feed-forward update. LayerNorm can inspect the input before that write, or inspect the sum after it.

The distinction matters in both directions. During the forward pass, it controls which scale reaches the next block. During backpropagation, it controls whether the residual shortcut contributes a direct identity term or passes through a normalization Jacobian. Neither layout guarantees stable training under every learning rate or initialization.^{[1]Reference 1On Layer Normalization in the Transformer Architecture.https://arxiv.org/abs/2002.04745}

Residual updates aren't literal multipliers, but a toy multiplicative drift makes the risk easy to see: a scale factor of 1.1 repeated across fifty blocks gives 1.1^50, roughly 117. The real stack adds vectors, so use that number as a warning about repeated transformations, not as a prediction of a model's hidden-state norm.

LayerNorm's job is to re-center and rescale each token's hidden state. Before learned scale and shift are applied, the normalized vector sits near zero with a spread close to one. First, compute that operation for one vector; then the placement choices will have something concrete to act on.

---

LayerNorm by hand

Use the running vector as one token's hidden state:

First find its mean: (3 + 1 + (-1) + 5) / 4 = 2.0. Centering removes this shared offset.

Subtract that mean from each feature, giving [1.0, -1.0, -3.0, 3.0]. The distances between features stay the same; only their origin moves.

Now compute the population variance: (1^2 + (-1)^2 + (-3)^2 + 3^2) / 4 = (1 + 1 + 9 + 9) / 4 = 5.0. LayerNorm divides by the feature count here, rather than by d - 1 as an unbiased sample estimate.

The standard deviation is sqrt(5.0) ≈ 2.236, so dividing the centered vector gives [0.45, -0.45, -1.34, 1.34]. Its mean is zero and its spread is one, up to rounding.

Finally, learned scale and shift can change that standardized result. With γ = [1.0, 1.0, 1.0, 1.0] and β = [0.0, 0.0, 0.0, 0.0], the output stays the same. During training, the model learns γ and β that help it predict the next token.

The operation runs independently for each token. Other tokens in the batch don't enter this token's statistics, so batch-size-one autoregressive generation uses the same rule as larger-batch training.

If you feed 16 token hidden states through LayerNorm, which numbers determine the mean and variance for one token?

Answer

Only that token's own feature values determine its mean and variance. The other 15 tokens in the batch don't affect it, which is why LayerNorm behaves the same during batch-size-1 decoding and during larger-batch training.

Turn that hand calculation into a reusable function. It accepts one vector and optional gamma, beta, and eps values, then checks the rounded result and two edge cases. Predict why setting every scale entry to zero returns an all-zero vector before running it.

The general formula

The function contains no hidden sequence operation. For a vector x ∈ ℝ^d (one token's hidden state), LayerNorm computes:^{[2]Reference 2Layer Normalization.https://arxiv.org/abs/1607.06450}

$\text{LayerNorm}(x) = \gamma \odot \frac{x - \mu}{\sqrt{\sigma^2 + \epsilon}} + \beta$

where:

• μ = (1/d) Σ x_i: mean over the hidden dimension
• σ² = (1/d) Σ (x_i - μ)²: variance over the hidden dimension
• γ, β ∈ ℝ^d: learnable scale and shift parameters
• ε: tiny positive constant to prevent division by zero ($10^{-5}$ in this example; defaults vary by implementation). In mixed precision (FP16 / BF16), keep ε large enough that the variance floor doesn't underflow to zero. Common production floors sit around 1e-5 to 1e-6, not aggressive 1e-12 values that can look fine in FP32 and blow up in half precision.

Subtracting the mean centers the values around zero. Dividing by the standard deviation squeezes them to a similar scale. The learned γ (scale) and β (shift) let the model decide whether the normalized values should be stretched, compressed, or offset for the next layer.

Key distinction from BatchNorm

LayerNorm normalizes across the feature dimension for each token independently. Batch Normalization (BatchNorm) instead computes each feature coordinate's statistics across examples in a mini-batch (and, for some data layouts, additional positions). This makes LayerNorm independent of batch size, which is useful for variable-length sequences and autoregressive generation.

Why is BatchNorm awkward for autoregressive decoding while LayerNorm is natural?

Answer

BatchNorm depends on batch-level statistics or stored running statistics, so behavior can change between training and inference. Autoregressive decoding often uses batch size 1 and variable sequence lengths. LayerNorm normalizes each token independently across its feature dimension, so population statistics are unnecessary.

Hold one token fixed and change only its neighbors. Predict which first output should stay unchanged, then compare it with a batch-normalized feature coordinate.

The LayerNorm row is identical before and after the neighboring token changes. The batch-normalized feature moves from -1.225 to -0.718 because its peers changed. That's the dependency the table summarized.

PyTorch implementation

The hand function operated on one list. Now the PyTorch version accepts a tensor of shape (batch, seq_len, d_model), reduces only its final feature dimension, and broadcasts learned scale and shift over batch and sequence positions. The unbiased=False variance matches the formula above.

Before running it, predict whether the one-token tensor will match [0.45, -0.45, -1.34, 1.34] after rounding.

The colored paths preserve each feature's identity across both operations. Centering translates the whole vector; scaling changes its spread. Learned γ and β can then reshape that standardized vector for the next layer.

---

Where you put LayerNorm matters

The arithmetic is now concrete. The live question is where LayerNorm sits inside a transformer block: before the residual add, or after it. That choice decides which path the next block and the backward pass see.

One block, two placements

There are two placements to compare. Post-LN normalizes after the residual update has been added, so its residual path crosses a normalization operation in every block. Pre-LN normalizes before the sublayer update, so its residual path keeps an identity contribution to the backward Jacobian.

Post-LN: the original transformer

In the original 2017 Transformer and BERT-style encoders, normalization happens after the residual addition:^{[3]Reference 3Attention Is All You Need.https://arxiv.org/abs/1706.03762[4]Reference 4BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.https://arxiv.org/abs/1810.04805}

$x_{l+1} = \text{LayerNorm}(x_l + \text{Sublayer}(x_l))$

In plain terms: compute the sublayer output (attention or FFN), add it to the original input along the skip connection, then normalize the sum. Normalization now sits directly on the main highway, so gradients through the residual path pass through a LayerNorm Jacobian in each Post-LN block.

Pre-LN: the GPT-2 layout

GPT-2 uses the other arrangement: normalization happens before the sublayer.^{[5]Reference 5Language Models are Unsupervised Multitask Learners.https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf}

$x_{l+1} = x_l + \text{Sublayer}(\text{LayerNorm}(x_l))$

Normalize the input first, feed it into the sublayer, then add the result to a direct copy of the input. The residual path from x_l to x_{l+1} keeps an identity contribution to the backward pass.

At the full-model level, GPT-2 and the Pre-LN architecture analyzed by Xiong et al. apply a final normalization before prediction.^{[5]Reference 5Language Models are Unsupervised Multitask Learners.https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf[1]Reference 1On Layer Normalization in the Transformer Architecture.https://arxiv.org/abs/2002.04745} The final operation sets the scale of the accumulated residual stream before the output projection. Dropping it produces a different architecture, so its output scale needs a fresh check. Post-LN doesn't need that extra layer: every block already ends with LayerNorm, so its last hidden state is normalized.

The two full-model paths are easier to compare when the final normalization is visible. The diagram keeps the block formulas from the equations above and shows where the model reaches its logits.

The block graphs below show why that skip path changes the backward Jacobian.

The left graph is Pre-LN: LayerNorm cleans the sublayer input, and the raw input still jumps to the add. The right graph is Post-LN: the sublayer and the skip meet first, then LayerNorm cleans the sum.

In one equation, what is the architectural difference between Post-LN and Pre-LN?

Answer

Post-LN is $x_{l+1} = LayerNorm(x_l + F(x_l))$. Pre-LN is $x_{l+1} = x_l + F(LayerNorm(x_l))$. The important difference is whether normalization sits on the residual highway after the add, or only on the sublayer input before the add.

Hold a sublayer update fixed to isolate placement. Predict which stream has mean zero before looking at the output: Post-LN centers and rescales the updated stream immediately, while Pre-LN lets the updated stream continue along the residual path.

---

A two-layer walkthrough

The forward probe showed one update. Now repeat it twice. Assume each sublayer writes the same 0.5 to every feature, with γ = 1 and β = 0.

The simplification keeps trained attention and feed-forward details out of this pass; those updates are usually nonuniform. Before reading each stack, predict which one preserves the shared offset.

Post-LN stack

• Start: x_0 = [1.0, 1.0, 1.0, 1.0]
• After block 1: x_1 = LayerNorm(x_0 + 0.5) = LayerNorm([1.5, 1.5, 1.5, 1.5]) = [0.0, 0.0, 0.0, 0.0] (mean-centered, then scaled)
• After block 2: x_2 = LayerNorm(x_1 + 0.5) = LayerNorm([0.5, 0.5, 0.5, 0.5]) = [0.0, 0.0, 0.0, 0.0]

The constant offset is reset at every layer. LayerNorm of a constant vector has variance zero, so with γ = 1 and β = 0 the output is the zero vector.

The residual stream therefore doesn't preserve that shared offset in this toy.

Pre-LN stack

• Start: x_0 = [1.0, 1.0, 1.0, 1.0]
• After block 1: x_1 = x_0 + Sublayer(LayerNorm(x_0)) = [1.0, 1.0, 1.0, 1.0] + 0.5 = [1.5, 1.5, 1.5, 1.5]
• After block 2: x_2 = x_1 + Sublayer(LayerNorm(x_1)) = [1.5, 1.5, 1.5, 1.5] + 0.5 = [2.0, 2.0, 2.0, 2.0]

The running sum flows through the network because Pre-LN leaves the residual path outside block normalization. That's also why a Pre-LN stack still wants a final norm before the head: the stream was never reset at block boundaries.

The backward equation below shows why this placement includes an identity gradient route.

Why does the Pre-LN architecture analyzed by Xiong et al. include a final normalization before the output head?

Answer

Inside each Pre-LN block, the residual stream bypasses block-output normalization and accumulates updates. A final norm sets its scale before the vocabulary projection. Removing that norm changes the architecture and requires rechecking output-scale and training behavior.

Now let the sublayer depend on the normalized or raw input instead of returning a fixed offset. The next cell applies two steps of each layout and asserts the same contrast: Pre-LN's stream grows, while Post-LN's output remains centered. Predict which assertion would fail if the two placements were accidentally swapped.

---

Why Pre-LN can train more stably

The forward path explains what gets normalized. Training adds another question: how does a loss signal travel from $x_{l+1}$ back to $x_l$? In Post-LN it crosses the block's normalization Jacobian. In Pre-LN it also has a direct identity route. At depth, multiplying those block Jacobians can change which layers learn fastest.

Post-LN gradient problem

Xiong et al. show that Post-LN produces especially large expected gradients near the output layers at initialization under their analysis.^{[1]Reference 1On Layer Normalization in the Transformer Architecture.https://arxiv.org/abs/2002.04745} For one block, ask where the residual contribution appears after differentiation. The backward Jacobian is:

$\frac{\partial x_{l+1}}{\partial x_l} = J_{\text{LN},l} \cdot (I + J_{F,l})$

Here J_LN,l is the LayerNorm Jacobian and J_F,l is the sublayer Jacobian. Across many layers, the backward pass multiplies terms like this together. In practice that means:

• Gradient scale becomes uneven across depth. Early layers may receive tiny updates while late layers receive huge ones.
• Top layers can receive very large updates at initialization. A single step with a moderately high learning rate can push weights far from a good basin.
• Learning-rate warmup addresses this risk in the evaluated Post-LN setups. Start the learning rate near zero and ramp it up so early updates are smaller.

Pre-LN gradient advantage

For Pre-LN, the same differentiation puts the identity term first:

$\frac{\partial x_{l+1}}{\partial x_l} = I + J_{F,l} \cdot J_{\text{LN},l}$

Every block includes a direct residual contribution to the gradient, so the backward signal doesn't rely entirely on repeated normalization Jacobians. Xiong et al. report that, in their evaluated tasks:

• Pre-LN trains without learning-rate warmup in experiments where Post-LN needed it for stable optimization.^{[1]Reference 1On Layer Normalization in the Transformer Architecture.https://arxiv.org/abs/2002.04745}
• Its initialization-time gradients are better behaved in their analysis.

Use a scalar product to isolate that route. It isn't a training experiment: j_layer_norm and j_sublayer stand in for Jacobians, and the cell compares their repeated products. Predict which twelve-block product falls below one before running it.

The Post-LN factor is 0.825, so its repeated product shrinks to 0.099. Pre-LN's factor is 1.075, and its product grows to 2.382. Real Jacobians are matrices, not fixed scalars, but the toy makes the placement effect visible.

---

A reported Pre-LN risk: representation collapse

Pre-LN isn't free of trade-offs. Analyses including ResiDual report that its residual stream can dominate newer sublayer updates as depth increases. Under that behavior, late layers make smaller relative changes to the hidden representation.^{[6]Reference 6ResiDual: Transformer with Dual Residual Connections.https://arxiv.org/abs/2304.14802}

The ResiDual authors call this reported behavior representation collapse.^{[6]Reference 6ResiDual: Transformer with Dual Residual Connections.https://arxiv.org/abs/2304.14802} Treat it as a design risk to measure, not a guarantee for every Pre-LN model or training run.

Keep the two trade-offs side by side:

There's no universal winner. Xiong et al. demonstrate the Pre-LN optimization advantage in their evaluated tasks. DeepNorm later scales a Post-LN-derived design to a 1,000-layer machine-translation experiment by combining residual scaling with matched initialization.^{[1]Reference 1On Layer Normalization in the Transformer Architecture.https://arxiv.org/abs/2002.04745[7]Reference 7DeepNet: Scaling Transformers to 1,000 Layers.https://arxiv.org/abs/2203.00555}

What is the representation-collapse risk reported for Pre-LN?

Answer

The residual stream can dominate newer sublayer updates as depth grows. Late blocks still run, but their relative changes may become small compared with the accumulated stream. This is an observed and analyzed risk, rather than an inevitable outcome for every Pre-LN run.

Residual magnitude growth across depth

Use a fixed write to make the geometry visible: every block adds [1, 0, 0, 0]. Pre-LN lets the stream accumulate, so that write becomes a smaller fraction of ||x||. Post-LN re-normalizes after each add, so its stream magnitude stays fixed.

DeepNorm is still Post-LN-shaped after the add (LN(α x + G(x))), so its after-LN ||x|| matches Post-LN in this sketch. α changes the mix before LayerNorm by up-scaling the skip (α = (2N)^{1/4} = 2 for this 8-layer encoder-style sketch). The paper's other half, shrinking residual-branch initialization by β, needs weights and isn't represented in this table.

Predict how the Pre-LN write share and the three stream magnitudes will change with depth before running the cell.

Pre-LN ||x|| grows from 2 to 9, while the fixed write's share falls from 0.5 to 0.11. Later blocks still run, but each write accounts for less of the accumulated stream. That's the geometry behind the reported representation-collapse risk.

Post-LN resets its magnitude every block, and DeepNorm matches it after LayerNorm (||x|| ≈ 2 here because a 4-d unit-variance vector has L2 norm $\sqrt{d}$). α appears before LayerNorm: from layer 2 onward ||αx|| = 4 while the write remains 1. The paper then shrinks residual-branch initialization by β so the write doesn't explode at step one.

Real models use learned sublayers rather than a fixed update. Measure ||x_l|| and relative update size on your own stack before calling this behavior collapse.

DeepNorm: scaled residual + matched init

DeepNorm (DeepNet) keeps a Post-LN-shaped block but multiplies the residual branch by a depth-dependent constant α. It also down-scales residual-branch initialization so expected update magnitudes stay controlled at large depth.^{[7]Reference 7DeepNet: Scaling Transformers to 1,000 Layers.https://arxiv.org/abs/2203.00555} The paper's encoder update has the form:

$x_{l+1} = \mathrm{LN}\big(\alpha\, x_l + G_l(x_l)\big)$

Here $G_l$ is the sublayer stack (attention and/or FFN writes). α grows slowly with depth so the residual highway still carries signal, while initialization of $G_l$ is reduced. The paper pairs α with a gain β on residual-branch weights: FFN, value projection, and output projection use it, while query and key keep standard Xavier gain. Encoder-only uses $\alpha=(2N)^{1/4}$ and $\beta=(8N)^{-1/4}$ for $N$ layers. Decoder-only uses the same pair with $M$ in place of $N$; encoder-decoder stacks use different constants on each side. Choose the row that matches your architecture.

DeepNorm is therefore not "plain Post-LN with no residual scaling." The residual scale and matched init are the design, and the 1,000-layer machine-translation experiment is evidence under that recipe, not a free upgrade for an arbitrary Post-LN codebase.

What two mechanisms does DeepNorm add on top of a Post-LN-shaped block?

Answer

A depth-dependent residual scale α in the update $x_{l+1}=\mathrm{LN}(\alpha x_l + G_l(x_l))$, and matched down-scaling of residual-branch initialization so expected update size stays controlled as depth grows.

---

RMSNorm: scale-only normalization

Placement decides where the normalizer sits. Many decoder-only LLMs also change the statistic inside that box.

RMSNorm (Root Mean Square Layer Normalization) replaces LayerNorm's centered standard deviation with a root-mean-square scale. Zhang and Sennrich propose it as a cheaper normalization that keeps rescaling invariance while dropping mean-centering. Their evaluated models match LayerNorm quality, with the speedup depending on the stack.^{[8]Reference 8Root Mean Square Layer Normalization.https://arxiv.org/abs/1910.07467} The Qwen2.5 paper uses RMSNorm with pre-normalization. The Gemma 2 paper uses RMSNorm on both the input and the output of each sublayer.^{[9]Reference 9Qwen2.5 Technical Reporthttps://arxiv.org/abs/2412.15115[10]Reference 10Gemma 2: Improving Open Language Models at a Practical Sizehttps://storage.googleapis.com/deepmind-media/gemma/gemma-2-report.pdf} Copy those layouts only after checking the checkpoint you actually serve.

$\text{RMSNorm}(x) = \gamma \odot \frac{x}{\sqrt{\frac{1}{d}\sum_{i=1}^d x_i^2 + \epsilon}}$

Instead of centering (subtracting the mean) and then scaling (dividing by standard deviation), RMSNorm just divides by the root-mean-square. It doesn't subtract the mean, and many implementations also omit a learned bias term.

The original RMSNorm paper reports comparable quality to LayerNorm on its evaluated tasks and observed run-time reductions from 7% to 64% across different models and implementations.^{[8]Reference 8Root Mean Square Layer Normalization.https://arxiv.org/abs/1910.07467} Fused kernels, hardware, and the fraction of time spent in normalization determine the result in another stack, so those measurements are evidence from the paper rather than a promised speedup.

What does RMSNorm remove from LayerNorm, and what does it keep?

Answer

RMSNorm removes mean-centering and often removes the learned bias. It keeps per-token rescaling by a feature-wise magnitude estimate and keeps a learned scale vector.

The hand example already gives us a useful test. RMSNorm should keep the vector's RMS near one, but its mean needn't be zero. Predict both properties before running the list implementation.

Now apply the same formula to a (batch, seq_len, d_model) tensor. The PyTorch module has one learned weight vector and no bias. Predict why its output mean is nonzero even though its feature RMS is one.

The output has RMS one and mean about 0.67, matching the list calculation. The removed centering step is doing visible work, not only saving an operation.

---

Reported normalization layouts

You can now separate two choices: where normalization sits around the residual add, and which statistic it computes. A third family targets attention logits rather than the residual stream. The table records reported examples; their experiments differ in model size, objective, and optimization setup.

No row is a global winner. Peri-LN names an input-and-output normalization layout and reports experiments up to 3.2B parameters.^{[12]Reference 12Peri-LN: Revisiting Layer Normalization in the Transformer Architecture.https://arxiv.org/abs/2502.02732} Rybakov et al. evaluate attention-specific normalization and capping on an 830M-parameter model driven into instability with high learning rates.^{[11]Reference 11QK-Norm: Improving Transformer Training Stability.https://arxiv.org/abs/2410.16682}

QK-Norm targets attention logits, not the residual stream. Gemma 2 uses a related but different control: logit soft-capping, soft_cap * tanh(logits / soft_cap), on attention logits and the final layer, together with pre+post RMSNorm. That clamps logit range, but it doesn't rescale Q and K.^{[10]Reference 10Gemma 2: Improving Open Language Models at a Practical Sizehttps://storage.googleapis.com/deepmind-media/gemma/gemma-2-report.pdf}

How does QK-Norm reduce attention instability?

Answer

QK-Norm normalizes query and key vectors before their dot product, limiting the contribution from their vector magnitude to attention logits. Rybakov et al. study a LayerNorm-based form and report that combining QK-Norm with softmax capping increases the stable learning-rate range in their evaluated setup. The normalizer and its exact placement relative to positional operations remain architecture choices.

Test the boundary directly. Multiplying a query by twenty changes its raw dot-product logit. RMS-rescaling both query and key should remove that scale-only change while leaving their direction intact. Predict the two pairs of logits before running the cell.

The raw logit grows from 17.5 to 350.0; RMS rescaling holds both cases at 1.807. That controls query-key magnitude only. It doesn't repair a residual stream whose norm or update share is drifting.

Why do hybrid layouts such as Peri-LN or Pre + Post norm exist if Pre-LN is already stable?

Answer

Pre-LN includes a direct identity gradient route, but residual-stream scale can still become a concern. Hybrid layouts add output-side normalization while preserving an input-side norm, then test whether that changes activation and optimization behavior.

Peri-LN and hybrids

Return to Peri-LN, the input-and-output layout in the table. Kim et al. report more balanced variance growth and steadier gradients than their compared layouts in experiments up to 3.2B parameters.^{[12]Reference 12Peri-LN: Revisiting Layer Normalization in the Transformer Architecture.https://arxiv.org/abs/2502.02732} Gemma 2's technical report describes the same input-and-output pattern with RMSNorm without using the Peri-LN name.^{[10]Reference 10Gemma 2: Improving Open Language Models at a Practical Sizehttps://storage.googleapis.com/deepmind-media/gemma/gemma-2-report.pdf} These controls belong around sublayers; QK-Norm belongs inside attention's dot-product path.

---

Debug normalization failures

When a real stack fails, start with the path that is misbehaving. Log residual-stream norms, attention-logit ranges, layerwise gradient norms, the final normalization, and the learning-rate schedule before changing several knobs at once.

---

Practice: build a toggle-switch transformer block

Make the placement choice executable. One block can switch between Post-LN and Pre-LN, and a small stack can report the gradient norm of each sublayer at initialization. The toy uses a linear layer and ReLU instead of attention, so it tests the residual route rather than a full language model.

Predict whether the six gradient norms in each row will be identical before running the cell. Then vary depth and seed.

The profiles aren't identical, even in a six-block toy. That variation is a measurement to inspect, not proof that one layout wins: Xiong et al.'s result concerns their analyzed initialization and evaluated tasks. Use the same logging pattern when a deeper run diverges, then change one architectural or optimization variable at a time.

Mini exercise

1. Change depth from 6 to 12. How do the per-block gradient profiles change?
2. Replace final_norm with nn.Identity() in Pre-LN mode and inspect output scale and loss under several seeds.
3. Try replacing nn.LayerNorm with the RMSNorm class from earlier. Does the gradient norm change meaningfully?

For feedback, compare ranges rather than a single number. Depth changes how many Jacobians are composed; removing the final norm changes the scale reaching the head; switching to RMSNorm changes the statistic but not the residual placement.

---

Mastery check

• Compute LayerNorm by hand, including mean, population variance, epsilon, learned scale, and learned shift.
• Explain why LayerNorm uses one token's feature values while BatchNorm uses statistics across examples or positions.
• Identify Pre-LN and Post-LN from code, then derive $I + J_FJ_{LN}$ versus $J_{LN}(I + J_F)$.
• Connect Post-LN's initialization-time gradient profile to learning-rate warmup without claiming the result applies to every training setup.
• Explain the reported deep Pre-LN representation-collapse risk and name measurements that would reveal it.
• Implement RMSNorm and state exactly which LayerNorm operations it removes.
• Separate residual-stream normalization, QK-Norm, DeepNorm, and input-output hybrid layouts by the mechanism each changes.

Practice checkpoints

Your 48-layer decoder diverges during first few hundred steps, but the same optimizer works on a 12-layer version. What should you inspect first?

Answer

Start with norm placement and residual-path statistics. If the stack is Post-LN, inspect initialization-time layerwise gradients and check whether warmup or initial learning rate differs from a stable setup. If the model follows a Pre-LN architecture with a final norm, confirm that norm remains present; then inspect attention-logit statistics and candidate controls such as QK-Norm or softmax capping.

Why should you avoid treating Pre-LN as a universal replacement for Post-LN?

Answer

Published comparisons are tied to architecture, depth, initialization, schedule, and task. Xiong et al. establish a Pre-LN initialization and warmup advantage in evaluated setups, while DeepNorm demonstrates a stable Post-LN-derived alternative under a different design. Measure the candidate layout under matched conditions.

When does RMSNorm change the model's behavior less than replacing Pre-LN with Post-LN?

Answer

RMSNorm changes the normalization formula inside a broadly similar residual layout. Switching Pre-LN to Post-LN changes where normalization sits relative to the skip path, which directly changes gradient flow and optimization stability. So RMSNorm is a smaller architectural change than moving the norm across the residual addition.