Mechanistic Interpretability

Mechanistic interpretability investigates how a neural network computes its outputs. Sparse autoencoders (SAEs) offer one set of candidate units: reconstruct a dense activation vector as a sparse combination of learned directions, then test how those directions participate in computation. A dictionary is a starting point for an explanation, not a complete algorithm recovered from the model.

Imagine investigating a code assistant that refuses a legitimate password-reset script. Was it responding to credentials, urgent wording, or something else? Suppose neuron 412 activates on credential resets, SQL queries, and urgent instructions, while neuron 789 responds to secret-manager docs and shell syntax. These are invented observations to frame the investigation; neither establishes why the assistant refused.

You have millions of activation numbers, but no clean unit that corresponds to a human concept.

The previous chapter showed how layer normalization changes the scale read by transformer branches. It doesn't tell you what individual coordinate directions compute. Here we'll look for more interpretable candidate directions, connect them into circuit hypotheses, and test edits to the residual stream.

This entanglement is the problem of polysemanticity: an individual neuron responds to multiple unrelated kinds of context. A raw coordinate needn't map cleanly to a single semantic idea.

To explain why polysemanticity happens, researchers rely on the superposition hypothesis: models represent far more features than they have dimensions by packing them into overlapping, non-orthogonal directions.

Mechanistic interpretability researchers train sparse autoencoders (SAEs), networks that reconstruct language model activations through a wide, sparse bottleneck, to search for these underlying feature directions.^{[1]Reference 1Towards Monosemanticity: Decomposing Language Models With Dictionary Learninghttps://transformer-circuits.pub/2023/monosemantic-features/index.html[2]Reference 2Sparse Autoencoders Find Highly Interpretable Features in Language Modelshttps://arxiv.org/abs/2309.08600}

Bricken et al. and Cunningham et al. found features with clearer activation patterns than comparison directions in their experiments.^{[1]Reference 1Towards Monosemanticity: Decomposing Language Models With Dictionary Learninghttps://transformer-circuits.pub/2023/monosemantic-features/index.html[2]Reference 2Sparse Autoencoders Find Highly Interpretable Features in Language Modelshttps://arxiv.org/abs/2309.08600} A candidate might respond to credential-related instructions or urgent wording. Its label and its computational role are separate hypotheses: an intervention can establish an effect without proving that the label captures every context in which the feature fires.

Reconstruction and causality are distinct claims. A feature can help reconstruct activations while tracking a grammatical correlate of the behavior you care about. Controlled patching, ablation, and steering test the consequences of a specified edit. Held-out prompts, control edits, and collateral-capability measurements determine how far that result supports your explanation.

The problem: polysemantic neurons and superposition

Inside a transformer, residual-stream vectors and MLP hidden activations carry dense mixtures of information. If a model wants to track $M$ useful concepts, but its residual stream dimension is $D$ with $M \gg D$, how can it possibly store them?

An ambient space $\mathbb{R}^D$ holds at most $D$ mutually orthogonal unit vectors. That limits an orthogonal linear dictionary at one activation site. It does not limit a model to $D$ concepts over its lifetime: other layers, nonlinear computations, and context-dependent representations also matter.

The Johnson-Lindenstrauss lemma concerns a finite set of $N$ points: a suitable linear map into $D = O(\epsilon^{-2}\log N)$ dimensions preserves all pairwise squared Euclidean distances within factors $1\pm\epsilon$.^{[3]Reference 3Extensions of Lipschitz mappings into a Hilbert space.https://web.stanford.edu/class/cs114/readings/JL-Johnson.pdf} Related concentration arguments let high-dimensional spaces hold many unit directions with small pairwise inner products. For example, draw each direction independently with coordinates $\pm1/\sqrt D$. A concentration bound and a union bound give

$$\Pr\left[\max_{i\ne j}|d_i^Td_j|>\epsilon\right] \le M(M-1)e^{-D\epsilon^2/2}.$$

Keeping the right side below a chosen failure probability lets $M$ grow exponentially with $D$ for fixed $\epsilon$. The bound can be loose or exceed one. It describes this random construction, not the exact capacity of a 4,096-dimensional residual stream or proof that a trained transformer uses such a packing.

Feature sparsity reduces interference in toy models of superposition.^{[4]Reference 4Toy Models of Superpositionhttps://transformer-circuits.pub/2022/toy_model/index.html} It doesn't guarantee interference disappears. For unit directions with coherence $\mu=\max_{i\ne j}|d_i^T d_j|$ and $x=\sum_{i\in S}a_i d_i$, an inactive direction's readout obeys

$|d_j^T x|\le\mu\sum_{i\in S}|a_i|,\qquad j\notin S.$

The contributions accumulate with active magnitudes and support size. A threshold must suppress that leakage while retaining weak real signals. Active-latent counts are properties of a particular SAE recipe, not a measured universal count of concepts in each token.

Predict whether a threshold of $1$ rejects the inactive direction below. Each active direction overlaps it by only $0.1$.

Such packing can produce polysemantic coordinates: several directions contribute to one axis. Polysemanticity alone doesn't establish superposition, however; even a rotated orthogonal dictionary can give mixed coordinates without having more features than dimensions. Residual coordinates and MLP neurons also differ: MLP activation functions introduce a coordinate-specific nonlinearity.

To make that geometric interference measurable, compute the cosine similarity between two candidate directions and construct their combined activation. If the urgency direction isn't orthogonal to the credential direction, predict how much coordinate A increases when both features fire simultaneously.

How sparse autoencoders produce candidate features

A sparse autoencoder attacks the superposition problem with overcomplete dictionary learning. It treats an activation vector $x \in \mathbb{R}^D$ as a sparse linear combination of directions drawn from a much wider learned dictionary with $M \gg D$. If the model packed features into non-orthogonal directions, the SAE's job is to unroll those directions onto separate latent axes.

An SAE consists of an encoder that projects the activation into a high-dimensional sparse code $z$, and a decoder that reconstructs the activation from that code:

• Encoder: $z = f\bigl(W_\text{enc}(x - b_\text{dec}) + b_\text{enc})\bigr)$ where $W_\text{enc} \in \mathbb{R}^{M \times D}$ with $M \gg D$, and $f$ is an activation function such as ReLU or a TopK operator.
• Decoder: $\hat{x} = W_\text{dec} z + b_\text{dec}$ where $W_\text{dec} \in \mathbb{R}^{D \times M}$. Each column $d_i = W_\text{dec}[:, i]$ represents a candidate feature direction in the model's activation space.

In this formulation, subtracting the learned decoder bias gives encoder and decoder a shared reference point.^{[1]Reference 1Towards Monosemanticity: Decomposing Language Models With Dictionary Learninghttps://transformer-circuits.pub/2023/monosemantic-features/index.html} That bias need not equal the exact dataset mean. Other SAE formulations omit pre-encoder subtraction, including the deliberately simple NumPy lab below.

Tied vs untied decoders

Weight tying ($W_\text{dec}=W_\text{enc}^T$) reduces matrix storage. Untied weights let detection and reconstruction directions differ, as in Bricken's SAE and Gao's TopK recipe. Cunningham's study used tied weights and reported no performance reduction on its residual-stream data, with some reduction on MLP data.^{[1]Reference 1Towards Monosemanticity: Decomposing Language Models With Dictionary Learninghttps://transformer-circuits.pub/2023/monosemantic-features/index.html[2]Reference 2Sparse Autoencoders Find Highly Interpretable Features in Language Modelshttps://arxiv.org/abs/2309.08600[5]Reference 5Scaling and Evaluating Sparse Autoencodershttps://cdn.openai.com/papers/sparse-autoencoders.pdf}

The reason comes down to an asymmetry in what the two layers do:

1. The encoder solves a feature detection problem under noise. Because other features in superposition create non-zero dot-product interference, the encoder needs sharp geometric selectivity and negative bias margins to keep false positives low.
2. The decoder adds its learned candidate direction $d_i$ to the reconstructed activation, weighted by $z_i$. The reconstruction lives at the chosen hook, which may be a branch output rather than the residual stream.

Tying restricts that independence, but its effect depends on the data and recipe. Tied initialization is a different choice: Gao initializes encoder weights from the decoder transpose, then allows the two matrices to train independently.^{[5]Reference 5Scaling and Evaluating Sparse Autoencodershttps://cdn.openai.com/papers/sparse-autoencoders.pdf}

The L1 shrinkage problem

The traditional training loss combines a mean squared error (MSE) reconstruction term with an L1 sparsity penalty:

$$\mathcal{L} = \underbrace{\frac{\|x - \hat{x}\|_2^2}{D}}_{\text{reconstruction MSE}} + \lambda \underbrace{\frac{\|z\|_1}{M}}_{\text{sparsity penalty}}$$

For a positive latent, this mean-normalized sparsity term contributes $\lambda/M$ to $\partial\mathcal{L}/\partial z_i$. The optimizer updates encoder parameters through the chain rule, so it doesn't simply subtract $\eta\lambda$ from every latent each step. Nevertheless, the penalty favors smaller magnitudes even when support stays unchanged. With a fixed unit decoder in one dimension, minimizing $(a-z)^2+\lambda z$ over $z\ge0$ gives $z^*=\max(0,a-\lambda/2)$. This is L1 shrinkage. Sum/mean conventions change the coefficient. Unconstrained decoder norms introduce a separate scaling loophole discussed below.

Three widely studied designs address magnitude shrinkage in different ways:

1. TopK SAEs: A nonnegative variant keeps at most $k$ nonzero values: $z=\text{TopK}(\text{ReLU}(W_\text{enc}(x-b_\text{dec})+b_\text{enc}),k)$. It removes the L1 term from the main reconstruction objective, not all reasons an activation could decrease. Selection boundaries are non-smooth, and Gao's full recipe adds AuxK for dead latents. That study trained a 16-million-latent dictionary on GPT-4 activations; its main encoder omits $b_\text{enc}$.^{[5]Reference 5Scaling and Evaluating Sparse Autoencodershttps://cdn.openai.com/papers/sparse-autoencoders.pdf}
2. Gated SAEs: A binary gate selects a nonnegative magnitude estimate. The original recipe shares projection directions, with separate row scales and biases. L1 applies to rectified gate pre-activations. An auxiliary reconstruction through those gate pre-activations supplies a learning signal, with decoder parameters detached in that path. Thus the magnitude estimate has no direct L1 penalty, although shared parameters can still change.^{[6]Reference 6Improving Dictionary Learning with Gated Sparse Autoencodershttps://arxiv.org/abs/2404.16014}
3. JumpReLU SAEs: $\text{JumpReLU}(y,\theta)=y\mathbf{1}(y>\theta)$ keeps the full pre-activation above a learned positive threshold. An L0 penalty controls support. Naively differentiating the threshold gives zero almost everywhere; the paper uses custom windowed straight-through estimators, whose bandwidth affects bias and variance.^{[7]Reference 7Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencodershttps://arxiv.org/abs/2407.14435}

At low sparsity penalty $\lambda$, expect dense mixtures to win; at high $\lambda$, expect sparse codes to win. The small calculation below makes that objective switch visible.

The original "Towards Monosemanticity" study (2023) trained an SAE on a one-layer transformer, decomposing a 512-neuron MLP into 4,096 features in run A/1. Inspected features showed clean specialization for DNA sequences, Hebrew script, base64 text, and legal formatting.^{[1]Reference 1Towards Monosemanticity: Decomposing Language Models With Dictionary Learninghttps://transformer-circuits.pub/2023/monosemantic-features/index.html}

A year later, "Scaling Monosemanticity" applied dictionary learning to a middle-layer residual stream of Claude 3 Sonnet (the March 2024 production model). It scaled dictionary widths to 1 million, 4 million, and 34 million features, isolating abstract concepts like sycophancy, code vulnerabilities, and multilingual entity representations.^{[8]Reference 8Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnethttps://transformer-circuits.pub/2024/scaling-monosemanticity/}

This pipeline separates two distinct questions:

1. Does the dictionary reconstruct the activation accurately while activating only a small handful of coordinates?
2. Do those learned directions correspond to true computational features that predict and control model behavior?

A low reconstruction error and high sparsity answer the first question. They don't automatically answer the second. That distinction determines how you choose hook points and how you validate features causally.

Where to attach SAEs inside a transformer

Researchers attach SAEs at several hook points across a transformer block:

• Residual stream after attention or MLP: The residual stream is the model's shared trunk communication channel. Hooking here captures information accumulated across all preceding layers.
• MLP output update: Captures the specific non-linear update written by that sublayer into the residual stream.
• Attention layer output: Decomposes the cross-token context routing information before it merges into the residual stream.

These sites can use the same SAE architecture, but a pretrained dictionary is tied to its exact checkpoint and hook. Record the layer, token position, pre/post normalization, attention output projection, activation dtype, and any centering or rescaling. Sites can share a width while differing in distribution. Joint dictionaries are possible too, but require an explicit multi-site training design.

Cunningham et al. reported mixed success on MLP data, with many inactive dictionary features in some middle and later layers. Untied weights retained more active features in those experiments. These results don't establish a universal ranking of activation sites: compare reconstruction, sparsity, and downstream behavior at the particular hook you plan to use.^{[2]Reference 2Sparse Autoencoders Find Highly Interpretable Features in Language Modelshttps://arxiv.org/abs/2309.08600}

Marks et al. trained sublayer SAEs in Pythia-70M and used existing Gemma Scope dictionaries for Gemma-2-2B.^{[9]Reference 9Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelshttps://arxiv.org/abs/2403.19647} The hook conventions differ: Gemma's attention dictionary in that study reads before the output projection, and its MLP dictionary reads after the branch output norm. Choosing a hook defines what your directions describe; matching a layer number alone isn't enough.

Feature circuits: from features to algorithms

Once you've trained SAEs across multiple layers, you can move from cataloging isolated features to tracing feature circuits: sparse subgraphs that show how earlier features feed downstream computations to produce the final token prediction.

Attribution methods suggest candidate dependencies across layers, but correlation along an edge isn't causal proof. A circuit hypothesis becomes credible only when intervention experiments (ablation, feature clamping, or activation patching) shift behavior in the exact direction the circuit predicts.^{[9]Reference 9Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelshttps://arxiv.org/abs/2403.19647}

Circuit studies investigate model- and task-specific mechanisms, with varying completeness:

• Induction heads that complete patterns such as [A][B] … [A] → [B]. Olsson's study gives causal evidence in small attention-only models and correlational evidence in larger models with MLPs.^{[10]Reference 10In-context Learning and Induction Heads.https://arxiv.org/abs/2209.11895}
• The indirect-object identification (IOI) circuit in GPT-2 small.^{[11]Reference 11Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallhttps://arxiv.org/abs/2211.00593}
• The greater-than comparison circuit in GPT-2 small.^{[12]Reference 12How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language modelhttps://arxiv.org/abs/2305.00586}
• Sparse feature circuits that explain complex behaviors in terms of human-interpretable feature nodes rather than uninterpretable neuron indices.^{[9]Reference 9Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelshttps://arxiv.org/abs/2403.19647}

Attribution graphs and cross-layer transcoders

Anthropic's 2025 method uses cross-layer transcoders (CLTs). A feature reads at its associated residual-stream layer and has separate decoder vectors contributing to MLP-output reconstructions at that layer and later ones. Each MLP output is reconstructed from features at its own and earlier layers. The model's attention operations remain; CLTs don't replace the entire transformer.^{[13]Reference 13Circuit Tracing: Revealing Computational Graphs in Language Modelshttps://transformer-circuits.pub/2025/attribution-graphs/methods.html}

For a chosen prompt, its local replacement model fixes attention patterns and normalization denominators to the original forward pass and adds back reconstruction errors as constants. Baseline activations and logits then match, without guaranteeing matching responses to edits. Its graph includes feature, token-embedding, unexplained-error, and selected output-logit nodes. Feature edges measure direct contributions to target pre-activations, with intervening MLP outputs stopped in the backward pass; a schematic edge is

$\text{Attr}(u\to v)=z_u\frac{\partial a_v}{\partial z_u}\Big|_{\text{specified frozen paths}},$

where $a_v$ is the target encoder pre-activation. This isn't the full model's unrestricted total derivative through every intermediate feature. Graph pruning selects a manageable hypothesis.^{[13]Reference 13Circuit Tracing: Revealing Computational Graphs in Language Modelshttps://transformer-circuits.pub/2025/attribution-graphs/methods.html}

The graph explains information flow given attention, not how Q/K computations formed that attention. Error nodes and inactive suppressed features can hide essential mechanisms. Check reconstruction fidelity, graph completeness, and interventions in the original model. The companion Biology study investigates particular behaviors of Claude 3.5 Haiku.^{[13]Reference 13Circuit Tracing: Revealing Computational Graphs in Language Modelshttps://transformer-circuits.pub/2025/attribution-graphs/methods.html[14]Reference 14On the Biology of a Large Language Modelhttps://transformer-circuits.pub/2025/attribution-graphs/biology.html}

The negative control is unchanged because we assigned it zero weight in the synthetic function. In a real language model, an ablation experiment requires held-out prompts, matched control directions, and measurements of collateral capabilities (like language modeling perplexity or code execution accuracy). A null ablation could also reflect redundant parallel pathways or an underpowered intervention, so negative results require careful verification.

Activation steering with SAE features

Because each SAE feature corresponds to a specific column $d_f = W_\text{dec}[:, f]$ in the decoder matrix, we can use that direction for activation steering at inference time without modifying model weights:

1. Run the forward pass and cache the residual-stream activation vector $x$ at the target layer.
2. Add a scaled copy of the feature's decoder direction: $x_\text{steered} = x + \alpha \cdot d_f$
3. Continue the forward pass using $x_\text{steered}$.

To set one coefficient in the cached SAE decomposition to $c$ while preserving its reconstruction error, use feature clamping: $\Delta x = \bigl(c - z_f(x)\bigr) d_f$ where $z_f(x)$ is the encoder's computed activation for feature $f$.

This changes a coefficient in the original decomposition. Re-encoding $x+\Delta x$ need not return $z_f=c$, because encoder and decoder aren't necessarily dual, thresholds can switch, and other feature readouts can move. Here a unit decoder direction still misses the requested re-encoded target:

Whether a hypothetical credential-labeled feature increases refusal, and at what scale, must be measured. Anthropic's 2024 Sonnet experiments amplified a Golden Gate Bridge feature to induce bridge-related responses, with effects varying by feature and prompt. Their edits preserved error and ran at every token position; extreme scales could produce nonsense.^{[8]Reference 8Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnethttps://transformer-circuits.pub/2024/scaling-monosemanticity/} Arditi et al. studied difference-of-means refusal directions across 13 open chat models, rather than SAE-derived safety classifiers. Their results don't establish one universal direction or semantic safety boundary for every model.^{[15]Reference 15Refusal in Language Models Is Mediated by a Single Directionhttps://arxiv.org/abs/2406.11717}

Keep reconstruction error separate from feature ablation

Any SAE reconstruction leaves an unexplained residual error $e = x - \hat{x}$. The original activation decomposes as:

$$x = \hat{x} + e = \sum_j z_j d_j + b_\text{dec} + e$$

When testing the causal effect of feature $f$, an error-preserving ablation subtracts only that feature's reconstructed contribution from the clean activation vector:

$$x' = x - z_f d_f$$

A dangerous beginner mistake is naive ablation: setting $z_f = 0$ inside the SAE reconstruction, yielding $x'_\text{naive} = \hat{x} - z_f d_f$. That naive step removes feature $f$ and simultaneously discards the unexplained residual $e$. If downstream attention heads or MLP layers rely on signals inside $e$, the model's performance drops, and researchers falsely attribute that drop to the ablated feature!

A small numerical calculation isolates this trap. Here the SAE has a tiny reconstruction error ($\text{MSE} = 0.005$), but the downstream readout relies entirely on the second coordinate. Predict whether naive ablation produces a false causal signal before running the check.

For an LLM, measure downstream language-model loss as well as activation MSE. Loss Recovered expresses one baseline-relative comparison:

$$\text{Loss Recovered} = 1 - \frac{\mathcal{L}_\text{SAE} - \mathcal{L}_\text{clean}}{\mathcal{L}_\text{ablated} - \mathcal{L}_\text{clean}}$$

Here $\mathcal{L}_\text{clean}$ uses original activations, $\mathcal{L}_\text{SAE}$ substitutes reconstructions, and $\mathcal{L}_\text{ablated}$ uses a specified mean/zero baseline at the same hooks and positions. The ratio is undefined at a zero denominator and unstable near zero. It can be negative or exceed one. A very destructive baseline can make substantial damage look like excellent recovery; Gao explicitly cautions about residual-stream zero-ablation.^{[5]Reference 5Scaling and Evaluating Sparse Autoencodershttps://cdn.openai.com/papers/sparse-autoencoders.pdf} Report all three losses, $\Delta\mathcal{L}=\mathcal{L}_\text{SAE}-\mathcal{L}_\text{clean}$, and task effects too.

Same SAE, same loss damage, different baseline. Which number would you trust without seeing the raw losses?

A toy NumPy SAE you can run today

The lab fits a small SAE on synthetic mixtures of 24 known directions in eight dimensions. Copy the training, summary, and alignment blocks into one file in that order, then run uv run --with numpy python toy_sae.py. This is an uncentered ReLU encoder with a decoder bias, full-batch SGD, and unit-normalized decoder columns. It omits production features such as activation streaming, AuxK, and a learned threshold.

Before running it, keep three separate evaluation criteria in mind:

1. Reconstruction MSE should fall.
2. Count active latents, rather than assuming the sparsity penalty recovered the generator's 2–4 active features.
3. Decoder columns should align with the known generating directions.

Those are three distinct mathematical claims.

First summarize the training set, then evaluate fresh held-out mixtures from the same generating dictionary.

The held-out sample tests generalization to fresh linear mixtures of the 24 ground-truth directions. Its Fraction of Variance Unexplained (FVU) is 0.169, showing solid reconstruction. But does low FVU prove we recovered the true ground-truth feature dictionary?

The alignment diagnostic below measures the maximum absolute cosine similarity between each true feature and all learned decoder columns.

Reconstruction MSE dropped by over $90\%$ ($1.426 \to 0.128$), yet median cosine alignment barely budged ($0.798 \to 0.799$)!

Sparse reconstruction is a proxy, not guaranteed recovery of generating features. These metrics alone don't identify a unique dictionary. Recovery can be identifiable under additional assumptions; this run hasn't demonstrated them. The maximum-match diagnostic also permits multiple true directions to match the same decoder column, and absolute cosine ignores the sign that a nonnegative latent can write. Keep the random-initialization baseline and signed diagnostic. In an LLM, ground-truth directions are usually unavailable, so use semantic counterexamples and causal tests together.

Key implementation choices in the NumPy trainer

• The generator combines 2–4 of 24 non-orthogonal directions in eight dimensions. This known construction lets us test recovery separately from reconstruction.
• Nonzero decoder columns are renormalized after each step. Otherwise, shrinking $z$ and expanding decoder columns reciprocally can preserve reconstruction while reducing this unweighted L1 penalty. A decoder-norm-weighted penalty is another way to address that reparameterization.^{[7]Reference 7Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencodershttps://arxiv.org/abs/2407.14435}
• The subgradient of the L1 term ($\text{sign}(z)$) is added to the backward gradient before the ReLU step.
• About $18.9/64=29.5\%$ of slots fire on training samples, and $19.5$ on held-out samples. That's less dense than using every slot, but much denser than the generator's 2–4 active directions. It doesn't establish production-level sparsity or recovered concepts.

Practical training details and pitfalls

These failure modes can appear in small toys as well as large-model training. Their prevalence depends on data, architecture, and optimization:

Dead latents: tracking and revival

A dead latent is inactive according to a specified monitoring rule. Gao's recipe uses no activation over 10 million tokens; a low but nonzero rate is a different criterion.^{[5]Reference 5Scaling and Evaluating Sparse Autoencodershttps://cdn.openai.com/papers/sparse-autoencoders.pdf} Four observations with no firing only show "not seen in this window," not permanent death. Check rare-domain coverage, observation length, and the rule before resampling a potentially useful rare feature.

Why do latents die? An unlucky initialization, strong sparsity pressure, or competing latents that consistently fire earlier can leave a feature with negative bias and zero gradients.

Two primary mechanisms revive dead latents:

1. Anthropic resampling (2023): At steps 25k/50k/75k/100k, the published recipe checks inactivity over the preceding 12.5k steps. It samples input vectors with probabilities proportional to squared per-input SAE loss. The decoder receives the normalized sampled input, not its reconstruction residual. The encoder uses the same direction at $0.2\times$ the average alive-row norm; its bias and modified Adam state are reset. Resampling can cause loss spikes and isn't mandatory in every recipe.^{[1]Reference 1Towards Monosemanticity: Decomposing Language Models With Dictionary Learninghttps://transformer-circuits.pub/2023/monosemantic-features/index.html}
2. AuxK (Gao, 2024): Select promising dead latents to predict the main error $e=x-\hat{x}$ with an auxiliary decoder pass. The paper typically uses weight $1/32$, alongside tied initialization. Its 16M-latent run reported 7% dead latents; this is a measured recipe result, not a guarantee for any dictionary or dataset.^{[5]Reference 5Scaling and Evaluating Sparse Autoencodershttps://cdn.openai.com/papers/sparse-autoencoders.pdf}

Non-identifiability, absorption, and splitting

Evaluating SAEs requires watching out for three subtle structural failure modes:

Absorption can improve sparsity without changing reconstruction. In a toy hierarchy, a "Paris" instance always includes "capital city." A combined decoder direction can write both while the general detector stays silent:

This is L0 bookkeeping, not a trained SAE. With unit-normalized combined direction $[1,1]/\sqrt2$, its coefficient becomes $\sqrt2$; support remains one. The child direction now writes general information too. Chanin's study investigates absorption in first-letter tasks and shows why sweeping width or sparsity alone needn't fix it.^{[16]Reference 16A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencodershttps://arxiv.org/abs/2409.14507}

Applications to safety and governance

Safety evaluations require the highest standard of evidence. Templeton et al. demonstrated that SAEs can surface safety-relevant features in Claude 3 Sonnet, isolating directions for scam emails, code exploits, bias, and sycophantic agreement.^{[8]Reference 8Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnethttps://transformer-circuits.pub/2024/scaling-monosemanticity/}

Some SAE features generalize across languages or modalities in particular studies.^{[8]Reference 8Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnethttps://transformer-circuits.pub/2024/scaling-monosemanticity/} Others track token strings, formatting, or only part of a concept. A French translation or base64 encoding is a useful test of a proposed semantic label, not a guarantee of invariant activation. Absorption also makes a silent latent weak evidence that a concept is absent.

Return to the imagined password-reset refusal. Compare benign and harmful credential-related prompts, vary urgency separately, and track proposed features at matched hooks. Test whether edits change a chosen refusal metric and whether legitimate task completion improves. A change in refusal wording alone doesn't establish corrected security judgment or preserved safeguards.

What changed in 2026?

Checked September 22, 2026. Heap et al.'s ICLR 2026 study compared residual-stream SAEs on trained, randomized, and initialization checkpoints from the Pythia family. Aggregate automated interpretability scores and reconstruction metrics often looked similar, especially at larger sizes. This challenges those metrics as stand-alone evidence of learned computation; it doesn't show that every SAE feature is meaningless. Add appropriate null-model baselines to metric-driven claims.^{[17]Reference 17Automated Interpretability Metrics Do Not Distinguish Trained and Random Transformershttps://proceedings.iclr.cc/paper_files/paper/2026/hash/7ab63a5314680e2f083cb288abeaeb8e-Abstract-Conference.html}

Anthropic's May 7, 2026 natural language autoencoders instead learn an activation → text → activation round trip. They can help generate auditing hypotheses, but their explanations can hallucinate and their verbalizer can infer beyond the stored activation. They don't supply a feature circuit or a verified transcript of reasoning. Corroborate claims with independent methods; readable explanations inherit the same proxy-versus-mechanism problem.^{[18]Reference 18Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activationshttps://transformer-circuits.pub/2026/nla/index.html}

Limits and caution

SAEs provide deep mechanistic visibility, but have clear operational limits. Four fundamental constraints stop us from treating them as a solved map of model cognition:

1. Reconstruction error matters: Substituting reconstructions can degrade model performance. Cunningham et al.'s layer-2 Pythia-70M experiment raised Pile perplexity from 25 to 40; it isn't a universal SAE penalty.^{[2]Reference 2Sparse Autoencoders Find Highly Interpretable Features in Language Modelshttps://arxiv.org/abs/2309.08600}
2. Coverage is incomplete: The Sonnet study explicitly says its dictionary doesn't capture all features and has no reliable total-feature count.^{[8]Reference 8Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnethttps://transformer-circuits.pub/2024/scaling-monosemanticity/} Reconstruction errors, rare contexts, and features spread across layers can leave relevant computation unexplained. A large dictionary isn't a coverage certificate.
3. Feature splitting can change the units: The Sonnet study observed concepts splitting into narrower sub-features as dictionary size grew.^{[8]Reference 8Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnethttps://transformer-circuits.pub/2024/scaling-monosemanticity/} Individual feature indices shouldn't be treated as immutable atomic units.
4. Labels are hypotheses: Top activations suggest a name, but lower activations and semantic counterexamples test its precision and recall. Causal experiments test the specified direction's role; they don't certify a universal semantic label, a unique dictionary, or complete understanding.

What to check before moving on

• Distinguish finite-point JL distance preservation from a claim about features learned by a transformer; calculate accumulated interference under a coherence bound.
• Explain how mixed coordinates can arise and why polysemanticity alone doesn't establish superposition.
• Compare tied weights, tied initialization, untied detection/reconstruction, and optional learned-reference subtraction.
• Identify the L1 shrinkage problem and explain how TopK, Gated, and JumpReLU SAEs eliminate or mitigate it.
• Report activation MSE, L0, raw downstream losses, and baseline-relative Loss Recovered without treating any one metric as feature recovery.
• Diagnose dead latents and contrast Anthropic resampling with OpenAI's AuxK auxiliary loss.
• Explain the difference between naive ablation and error-preserving ablation, identifying why discarding the residual $e$ creates false causal claims.
• Trace CLT input/output layers and explain local attribution graphs' frozen attention, normalization denominators, error nodes, and omitted mechanisms.

Practice checkpoints

Common failures and fixes

• Symptom: A raw coordinate responds to unrelated contexts. Possible cause: Mixed representations; this alone doesn't prove an overcomplete dictionary. Check: Compare candidate directions, their activation ranges, and controlled effects.
• Symptom: MSE falls but labels stay messy. Possible causes: Dense codes, absorption, splitting, or poor labels. Check: Sweep width/sparsity with matched data, test recall and seeds, and report downstream loss. No universal $M/D\ge16$ rule resolves these problems; tighter sparsity can worsen absorption.
• Symptom: A feature label sounds convincing, but ablation changes nothing. Possible causes: The label is correlational, the edit magnitude is ineffective, or parallel pathways compensate. Fix: Check the hook point, edit magnitude, and alternative pathways before interpreting the null result.
• Symptom: Many slots never fire in the observation window. Possible causes: Competition, initialization, sparsity, or missing rare contexts. Check: State the window and sample coverage before trying a revival recipe; measure its side effects.
• Symptom: Steering works on a demo but damages other prompts. Possible causes: Overlap, scale, wrong hook, or changed attention. Check: Match the checkpoint, hook, and token positions, then sweep edits and controls on held-out tasks. An orthogonal linear probe staying fixed doesn't guarantee downstream capabilities stay fixed.

Practice projects

1. CPU first: Sweep the toy's sparsity coefficient and training seed. Record held-out MSE/FVU, L0, signed alignment, and random-initialization alignment. Try to improve recovery rather than only reconstruction.
2. Inspect an existing dictionary: Use SAELens with its matching model/hook. Sample top, middle, low, and non-activating contexts; test a label's precision and recall. Its current inference interface supports generic PyTorch activation extraction, not only TransformerLens.
3. Train a dictionary if compute allows: Freeze a small model and stream activations from a stated corpus. Plan memory: 200k float32 vectors of width 4,096 alone occupy about 3.05 GiB at one hook, before model, SAE, gradients, and optimizer state. Compare L1/TopK under matched width, support, data, and compute; a few hundred thousand tokens don't guarantee coverage.
4. Test an algorithm: Choose a model-specific comparison task such as greater-than year completion. Find candidate SAE components, preserve error, and test both targeted and control edits. Recovery of a known head-level circuit in SAE coordinates is a research goal, not a promised outcome.^{[12]Reference 12How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language modelhttps://arxiv.org/abs/2305.00586}
5. Trace a prompt: Follow circuit-tracer's tutorial with a supported model and pretrained transcoders, then test one graph prediction. A standard SAE alone isn't a CLT. Record the graph's frozen paths, error nodes, edit positions, and original-model behavior.

Tool compatibility checked September 22, 2026: SAELens documents a v6 training refactor. TransformerLens now recommends TransformerBridge; its README states that the legacy HookedTransformer.from_pretrained API was removed in 4.0. Record installed versions and consult their migration guides before copying older notebooks. Weight folding/centering can change activation coordinates, so match a dictionary's training conventions as well as its model name.

Validate one feature claim

Take one proposed SAE feature through a complete causal audit. Inspect top activations on held-out text, state a behavioral prediction, ablate or steer the decoder direction, and run matched negative controls. Save reconstruction damage and downstream behavior in an audit table beside the intervention result.

If the effect doesn't replicate across held-out prompts or the SAE replacement degrades the model too much, keep the label as an unconfirmed hypothesis.