Mechanistic Interpretability

Layer normalization kept the residual stream numerically stable. Mechanistic interpretability asks the next question: what do the directions inside that stream mean?

A code-assistant LLM keeps refusing legitimate password-reset scripts because they mention credentials. When you inspect the attention heads and MLP neurons that fire on those prompts, dozens of neurons light up in confusing combinations. One neuron activates on "rotate the API key," "credential theft attempt," and "urgent tone." Another neuron fires for "secret manager docs" and also for "shell command syntax." You can't point to a clean "credential-safety feature."

This is the core problem of polysemanticity (one neuron responding to multiple unrelated concepts at once) inside large language models. Raw neurons often fail to map to single human concepts. One influential hypothesis is superposition: useful features can be represented in overlapping directions when clean coordinate axes are scarce. Toy models demonstrate this mechanism; for real language models, it remains a motivating hypothesis and an empirical question.

Why can one neuron fire for credential reset, malicious intent, and urgent tone at the same time?

Answer

Under the superposition hypothesis, useful concepts share overlapping directions instead of separate coordinate axes. A neuron can respond to several of those directions at once, which makes it a weak unit for assigning human labels. The next step is to find more useful candidate units and test them experimentally.

Mechanistic interpretability researchers train sparse autoencoders (SAEs), small models that reconstruct model activations through sparse latent codes, to look for more inspectable feature directions.^{[1]Reference 1Towards Monosemanticity: Decomposing Language Models With Dictionary Learninghttps://transformer-circuits.pub/2023/monosemantic-features/index.html[2]Reference 2Sparse Autoencoders Find Highly Interpretable Features in Language Modelshttps://arxiv.org/abs/2309.08600} In evaluated models, Bricken et al. and Cunningham et al. found SAE features that were more interpretable than raw-neuron or alternative baselines. A candidate feature might fire mostly on credential-reset language, exploit-request language, or urgent instructions. Its label remains provisional until examples and interventions support it.

You'll build that path in layers: how SAEs work, what they reveal beyond raw neurons, how to train a minimal NumPy version, and how feature-level interventions support modern safety research.

The problem: polysemantic neurons and superposition

In a transformer, residual-stream vectors and MLP activations combine many signals. If a model represents more useful features than available orthogonal axes, non-orthogonal directions offer an efficient representation but introduce interference. This is the superposition hypothesis for why the same neuron can participate in unrelated computations.

The 2022 "Toy Models of Superposition" paper demonstrated this mechanism with tiny ReLU networks trained on synthetic sparse data. When the data contained more ground-truth features than hidden units, the networks learned overlapping representations. The experiment doesn't establish the full explanation for an LLM, but it gives researchers a concrete model to test against LLM activations.^{[3]Reference 3Toy Models of Superpositionhttps://transformer-circuits.pub/2022/toy_model/index.html}

Direct inspection of weights or raw neuron activations therefore gives you an entangled view. A raw neuron might mix "credential reset + exploit request + urgency." You can't tell which part of the mixture the model is using at any moment without a cleaner representation and an intervention test.

The calculation below measures that overlap with cosine similarity and then forms the same mixture shown in the figure.

How sparse autoencoders produce candidate features

A sparse autoencoder attacks the entanglement problem with classic dictionary learning. You model activation vectors $x \in \mathbb{R}^{d_\text{model}}$ as approximately sparse linear combinations of directions from a much larger learned dictionary. Some learned directions may correspond to useful model features. That correspondence is a hypothesis to evaluate, not an assumption built into the math.

An SAE consists of two learned matrices plus a non-linearity:

• Encoder: $z = f(W_\text{enc} x + b_\text{enc})$ where $W_\text{enc} \in \mathbb{R}^{n_\text{features} \times d_\text{model}}$ with $n_\text{features} \gg d_\text{model}$ and $f$ is ReLU or a TopK operation. Experiments choose dictionary widths larger than the activation dimension and compare fidelity, sparsity, and feature quality.
• Decoder: $\hat{x} = W_\text{dec} z + b_\text{dec}$

Training minimizes a loss that has two competing terms:

$$L = \underbrace{\|x - \hat{x}\|_2^2}_{\text{reconstruction}} + \lambda \underbrace{\|z\|_1}_{\text{sparsity}}$$

(or a TopK variant that keeps the $k$ largest latent activations and zeros the rest). The reconstruction term rewards preserving information from the original activation. The sparsity term pushes the latent code $z$ toward few active coordinates rather than a dense remapping. Learned columns of $W_\text{dec}$ are candidate feature directions; evaluating them requires top-activating examples and intervention tests in addition to reconstruction metrics.

The original "Towards Monosemanticity" result (2023) made this concrete: an SAE trained on a one-layer transformer decomposed a 512-neuron MLP layer into more than 4,000 features, with inspected examples for patterns such as DNA sequences, legal language, HTTP requests, and Hebrew text. A year later, "Scaling Monosemanticity" (2024) applied the approach to a middle-layer residual stream of Claude 3 Sonnet, extracting up to 34 million features and reporting abstract and multilingual examples.^{[1]Reference 1Towards Monosemanticity: Decomposing Language Models With Dictionary Learninghttps://transformer-circuits.pub/2023/monosemantic-features/index.html[4]Reference 4Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnethttps://transformer-circuits.pub/2024/scaling-monosemanticity/}

Why is an SAE overcomplete for interpretability?

Answer

An overcomplete SAE gives the dictionary more latent slots than the activation space has axes, which lets it search for cleaner candidate directions under the superposition hypothesis. Sparsity prevents the model from turning on all those slots at once.

The SAE learns an overcomplete sparse dictionary. The input activation $x$ is projected into a higher-dimensional sparse code $z$, and the decoder reconstructs $\hat{x}$ from active coordinates. Reconstruction and sparsity are necessary diagnostics; whether a direction is interpretable still has to be measured.

Where to attach SAEs inside a transformer

Researchers attach SAEs at several natural "hook points":

• Residual stream immediately after an attention block or after the MLP block.
• The output of the MLP (the "MLP SAE").
• Attention-layer outputs (more experimental).

Cunningham et al. presented their method for residual-stream, MLP-output, or attention-output activations. They mainly studied residual streams and reported mixed MLP-SAE results, including dead features. Marks et al. evaluated feature circuits with SAEs at several sublayers in Pythia and Gemma models. Hook point changes the question you can ask, but a residual-stream or MLP-output label still needs feature-quality and causal validation.^{[2]Reference 2Sparse Autoencoders Find Highly Interpretable Features in Language Modelshttps://arxiv.org/abs/2309.08600[5]Reference 5Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelshttps://arxiv.org/abs/2403.19647}

Why might a residual-stream SAE and an MLP-output SAE reveal different features?

Answer

The residual stream is the shared communication channel that carries mixed information forward through the model. MLP outputs are the updates written by one sublayer. SAEs trained at either point offer different candidate descriptions, and experiments must establish which candidates carry behaviorally relevant information.

Feature circuits: from features to algorithms

Once you have candidate interpretable features at multiple layers, you can study feature circuits: sparse subgraphs that connect features across layers. Attribution methods can suggest which earlier features feed later features, but those edges are hypotheses rather than causal proof. A circuit becomes credible only when ablation, patching, or steering changes behavior in the predicted direction.^{[5]Reference 5Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelshttps://arxiv.org/abs/2403.19647}

Circuit papers have recovered recognizable behaviors such as:

• Induction heads that copy patterns across context windows.^{[6]Reference 6In-context Learning and Induction Heads.https://arxiv.org/abs/2209.11895}
• An indirect-object identification circuit in GPT-2 small.^{[7]Reference 7Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallhttps://arxiv.org/abs/2211.00593}
• A greater-than circuit for simple year-comparison prompts in GPT-2 small.^{[8]Reference 8How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language modelhttps://arxiv.org/abs/2305.00586}
• Sparse feature circuits that express some behaviors in human-readable feature units rather than raw neuron or head indices.^{[5]Reference 5Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelshttps://arxiv.org/abs/2403.19647}

The promise isn't that every model becomes transparent overnight. The promise is narrower: turn one behavior at a time from "the model produced this and we don't know why" into "these internal components appear to implement it, and interventions support that explanation."

Attribution graphs and circuit tracing

Later work built end-to-end graph hypotheses on top of learned feature decompositions. Anthropic's Circuit Tracing methods article replaces MLPs with cross-layer transcoders, which read from one layer and write to later layers. It then builds attribution graphs whose nodes are active features and whose edges approximate linear contributions to logits and intermediate features. The methods article demonstrates graphs on simple behaviors of a small replacement model; its companion study applies the technique to Claude 3.5 Haiku.^{[9]Reference 9On the Biology of a Large Language Modelhttps://transformer-circuits.pub/2025/attribution-graphs/biology.html}

Attribution graphs compress many candidate dependencies into one object, but the replacement model can diverge from the original model and important graph hypotheses still require perturbation tests. Feature-level circuits therefore explain scoped behaviors under measured approximations, rather than exposing a complete model algorithm.^{[9]Reference 9On the Biology of a Large Language Modelhttps://transformer-circuits.pub/2025/attribution-graphs/biology.html}

How does an attribution graph differ from running activation patching one component at a time?

Answer

Activation patching tests selected components through interventions. An attribution graph uses a replacement model to surface a candidate multi-step chain: its nodes are active transcoder features and its edges approximate contributions among them. You still validate important steps with interventions and measure how faithfully the replacement model preserves behavior.

What extra evidence do you need before trusting an SAE feature as causal?

Answer

Correlation isn't enough. You need an intervention such as activation patching, ablation, or steering that changes the model's behavior in the predicted direction while leaving unrelated behavior mostly intact.

This constructed scoring function illustrates an intervention pattern: the named candidate and a negative control are ablated separately. Applying the pattern to a language model requires measured outputs on held-out prompts, not a label alone.

Activation steering with SAE features

Feature directions selected for interpretation experiments can also become intervention directions. After you have a trained SAE, you can perform activation steering at inference time without touching any weights:

1. Run the prompt and cache the residual-stream activations at the layer where your target SAE lives.
2. Encode those activations with the SAE encoder to obtain the sparse feature vector $z$.
3. Add a multiple of the decoder vector of a chosen feature: $\text{new activation} = x + \alpha \cdot W_\text{dec}[:, f]$.
4. Continue the forward pass with the modified activation.

If feature 17 is the "credential-safety refusal" feature, adding $+2.5 \times W_\text{dec}[:,17]$ might make the model more likely to refuse a borderline credential request. Subtracting the same direction might make it more lenient. You don't trust that from one demo. You run a sweep, measure side effects, and check unrelated prompts.

Anthropic's Claude 3 Sonnet work showed that amplifying or suppressing learned features can change model behavior, including safety-relevant examples such as scam-email and sycophantic-praise features.^{[4]Reference 4Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnethttps://transformer-circuits.pub/2024/scaling-monosemanticity/} Related refusal-direction work shows the same broader lesson from a non-SAE direction: residual-stream edits can strongly affect refusal behavior, so steering needs careful safety handling.^{[10]Reference 10Refusal in Language Models Is Mediated by a Single Directionhttps://arxiv.org/abs/2406.11717}

Why does steering use the decoder vector for a feature rather than the raw latent activation value alone?

Answer

The latent value tells you how strongly the feature is active. The decoder vector tells you which direction in the model's residual stream represents that feature. To edit the model state, you add or subtract that residual-stream direction.

The visual shows the review flow for one candidate credential-safety feature. Repeated top-activating examples suggest a label. Held-out prompts and causal interventions then test whether that label keeps predicting behavior. A plausible name isn't ground truth.

A toy NumPy SAE you can run today

A direct way to internalize the mechanics is to implement a tiny SAE yourself. The lab trains an SAE on synthetic activations that deliberately contain superposition (24 ground-truth features packed into eight-dimensional activations). After a few hundred gradient steps the reconstruction loss drops while the latent activations remain sparse.

Now inspect the learned sparse code after training. This follow-up cell keeps the heavy optimization loop separate from the summary checks you read.

Run the first cell and watch reconstruction loss drop, then use the follow-up cell to confirm that average active features per sample stays far below the full 64-feature dictionary. Reconstruction and sparse use alone still say nothing about whether columns recover known generating directions. Because this dataset has ground truth, the next cell measures that question directly.

Here, reconstruction improves while median direction alignment barely moves (0.798 to 0.799; high matches move from one to two). This tiny trainer therefore demonstrates sparse reconstruction, not recovery of ground-truth features. On synthetic data, a stronger trainer could use rising alignment to support a recovery claim. Real LLM activation datasets lack known generating directions, so feature interpretation instead relies on held-out activation examples and causal tests.

Reading the toy SAE code line by line

• The synthetic data generator deliberately activates only 2 to 4 of the 24 true features per sample and then sums their scaled direction vectors. Because $d_\text{model}=8$, this is textbook superposition.
• The encoder W_enc is deliberately initialized small so that early activation magnitudes stay controlled.
• The manual gradient for the sparsity term adds a sub-gradient of the L1 norm (sign(z)) to the latent gradient before the ReLU backward step. In a real PyTorch implementation you would let torch.autograd handle it.
• Decoder columns are renormalized after every update. Without that constraint, a model could shrink $z$ and inflate decoder vectors, lowering the L1 penalty without changing the reconstruction.
• The learning-rate schedule and $\lambda$ value were hand-tuned for this toy problem; on real LLM activations, optimizer settings are found by sweeping on a held-out set of prompts while monitoring both reconstruction MSE and the fraction of "dead" (never-activated) features.
• After convergence the average $\ell_0$ norm of $z$ (number of non-zero entries) should stay far below the full 64-feature dictionary. It won't exactly match the 2 to 4 ground-truth features because this tiny L1 trainer is deliberately simple; the important result is that reconstruction improves without activating the entire dictionary.

The small SAE lab contains the core ingredients used in large SAE work: an activation dataset, an overcomplete dictionary, a reconstruction objective, a sparsity constraint, and diagnostics for feature quality.^{[4]Reference 4Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnethttps://transformer-circuits.pub/2024/scaling-monosemanticity/[11]Reference 11Scaling and Evaluating Sparse Autoencodershttps://cdn.openai.com/papers/sparse-autoencoders.pdf}

Practical training details and pitfalls

Published SAE experiments use several refinements:

Non-identifiability, absorption, and uniqueness limits

Low reconstruction error can't establish that an SAE recovered a unique ground-truth dictionary. Multiple overcomplete dictionaries can reconstruct the same activations equally well. Two independently trained SAEs on the same residual stream often disagree on which latents fire for a concept, even when both look "good" on MSE and $\ell_0$.

Related failure modes:

Practical posture: treat SAEs as intervention tools, not as unique microscopes. Prefer causal checks (steering, ablation, counterfactual prompts) over dictionary aesthetics. When two SAEs disagree, measure which interventions replicate rather than which feature names look prettier.

Common beginner mistakes are easier to debug if you name the symptom:

L1 Penalty vs TopK SAEs - a practical comparison

These papers evaluate different ways to control sparsity and shrinkage: TopK fixes active count, Gated separates selection from magnitude, and JumpReLU trains a learned threshold with an L0 penalty.^{[11]Reference 11Scaling and Evaluating Sparse Autoencodershttps://cdn.openai.com/papers/sparse-autoencoders.pdf[12]Reference 12Improving Dictionary Learning with Gated Sparse Autoencodershttps://arxiv.org/abs/2404.16014[13]Reference 13Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencodershttps://arxiv.org/abs/2407.14435} Results are architecture- and dataset-specific, so variant choice should be followed by reconstruction, sparsity, feature-quality, and intervention checks.

Why can a low reconstruction loss still produce weak interpretability?

Answer

If sparsity is too weak, the SAE can reconstruct activations using dense mixtures that are accurate but not human-readable. Interpretability needs both fidelity and sparse, stable features that survive causal checks.

Applications to safety and governance

Templeton et al. inspected Claude 3 Sonnet features related to safety-relevant behaviors, including sycophantic praise and scam-related text, and experimented with feature steering.^{[4]Reference 4Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnethttps://transformer-circuits.pub/2024/scaling-monosemanticity/} These results establish useful research probes, not a general safety monitor. A feature that activates on suspicious examples gives researchers an internal signal to test with interventions and behavioral evaluations.

In a code-assistant deployment, the research question becomes more precise. Instead of asking "why did the model refuse this password-reset script?" in the abstract, you can ask which candidate features activated, whether a credential-safety direction was causally involved, and whether changing that direction shifts decisions on a held-out security-prompt set. That remains research evidence rather than a complete compliance story.

Limits and caution

SAEs are a genuine advance, not a solved problem. Keep four limits in mind so you don't oversell them in an interview or a design review.

• Reconstruction error matters. When you splice an SAE reconstruction into the model to evaluate fidelity, $\hat{x}$ replaces the original activation. Cunningham et al. report a notable perplexity increase when substituting an SAE reconstruction in their Pythia-70M experiment, and feature-circuit evaluations include error terms for unexplained computation.^{[2]Reference 2Sparse Autoencoders Find Highly Interpretable Features in Language Modelshttps://arxiv.org/abs/2309.08600[5]Reference 5Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelshttps://arxiv.org/abs/2403.19647}
• Coverage is incomplete. The 34-million-feature SAE for Claude 3 Sonnet didn't capture everything the model knows; for example, only about 60% of London boroughs had a dedicated feature, and rare concepts were far less likely to get one. Exhaustive coverage would likely need orders of magnitude more features.^{[4]Reference 4Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnethttps://transformer-circuits.pub/2024/scaling-monosemanticity/}
• Feature splitting. One human concept can fracture into several near-duplicate features as dictionary width changes, so a feature index shouldn't be treated as a canonical human concept.
• Labels are hypotheses, not ground truth. A name like "credential safety" comes from top-activating examples and survives only if ablation and steering move behavior in the predicted direction. Reconstruction quality alone never proves interpretability.

What to check before moving on

• Explain polysemanticity, superposition, and why raw neurons are often weak interpretability units.
• Write the SAE reconstruction plus sparsity objective and identify encoder, latent code, and decoder shapes.
• Choose a transformer hook point and explain what a residual-stream SAE or MLP-output SAE can reveal.
• Implement a minimal SAE forward pass, loss, and gradient update on toy data.
• Explain how feature circuits connect interpretable features into candidate algorithms, and how attribution graphs extend this with cross-layer transcoders.
• Describe activation steering with a decoder direction and the controls needed before trusting it.
• Diagnose dead features, feature splitting, weak sparsity, and reconstruction-only evaluation.
• Connect SAE research to safety evaluations without claiming the whole model is solved.

Practice checkpoints

Why can't we just inspect weights or individual neuron activations?

Answer

Because of superposition. The model represents more useful concepts than it has clean axes, so it stores concepts in overlapping directions. A single neuron can respond to several unrelated concepts, and a single concept can be distributed across several neurons. SAE features give you a cleaner unit to inspect, but you still need intervention evidence.

What is the difference between an L1 SAE and a TopK SAE?

Answer

An L1 SAE adds $\lambda \|z\|_1$ to the reconstruction loss and encourages sparsity softly. A TopK SAE keeps exactly the $k$ largest latent activations and zeros the rest. TopK gives direct sparsity control and performs well in recent scaling work, but the keep-k operation needs a training recipe that handles the non-smooth selection step.^[11]

How does an SAE enable activation steering?

Answer

After training an SAE at a hook point, each feature has a decoder vector in the model's activation space. You can add or subtract a scaled copy of that decoder vector during a forward pass, then measure whether the model's behavior changes in the predicted direction. The latent activation tells you that the feature is present; the decoder vector gives you the direction to write.

What causes dead features?

Answer

Dead features are latent dimensions that rarely or never activate. They often come from too much sparsity pressure, unlucky initialization, or early training dynamics where some latents never win competition for reconstruction. Large SAE recipes track dead-latent rates and use resampling or auxiliary losses to revive unused capacity.^[11]

How do feature circuits help explain model algorithms?

Answer

A feature circuit connects human-readable features across layers, then tests whether those edges matter for behavior. The output isn't merely a label like "credential safety." The useful artifact is a candidate causal graph: early features activate, later features combine them, and interventions on selected nodes should shift the model's answer in a predictable way before you trust the explanation.

Common failures and fixes

• Symptom: A raw neuron seems to mean three unrelated things at once. Cause: You're reading a polysemantic axis rather than a reliable unit of explanation. Fix: Inspect SAE candidate features and verify them with interventions, not neuron anecdotes alone.
• Symptom: Reconstruction loss is low, but feature labels still look messy. Cause: The SAE may be reconstructing activations with dense mixtures instead of sparse, stable features. Fix: Track active-latent count, dead-feature rate, top examples, and causal effects together.
• Symptom: A feature label sounds convincing, but ablation changes nothing. Cause: The label came from correlational examples instead of causal evidence. Fix: Treat names as hypotheses until patching, ablation, or steering moves behavior in the predicted direction.
• Symptom: Many latent slots never fire. Cause: Sparsity pressure is too strong, initialization is unlucky, or training data coverage is weak. Fix: Sweep sparsity, use dead-feature revival, and inspect domain coverage.
• Symptom: Steering one feature helps on a demo prompt but breaks unrelated prompts. Cause: The direction is entangled or overscaled. Fix: Run held-out sweeps and negative controls before trusting the intervention.

Practice projects

If you want to move from the toy example to larger-model work, the natural next projects are:

1. Train a small SAE on the residual stream of a 1B to 8B open model using SAELens, TransformerLens, or nnsight to capture activations. Use a few hundred thousand tokens from code, security, documentation, or instruction-following data.
2. Implement a TopK SAE trainer in PyTorch and compare reconstruction and interpretability metrics against your L1 version.
3. Build a simple feature browser: for each learned decoder direction, collect the top-20 activating tokens across a large corpus and let a human (or an LLM judge) label the emerging concept.
4. Reproduce a known circuit (e.g., the greater-than circuit) using SAE features instead of raw attention heads, then attempt a causal intervention by zero-ablating the responsible feature.
5. Build an attribution graph for a two-hop factual prompt on a small open model using an open-source circuit-tracing library, then validate one intermediate feature by suppressing it.^{[9]Reference 9On the Biology of a Large Language Modelhttps://transformer-circuits.pub/2025/attribution-graphs/biology.html}

Each project exercises these concepts and produces inspectable experimental evidence.

The stable core is superposition, dictionary learning, reconstruction plus sparsity, and causal interventions on feature directions. Once those four ideas are clear, newer SAE variants (TopK, Gated, JumpReLU) and the transcoder-plus-attribution-graph stack become engineering choices rather than a new conceptual foundation.

Validate one feature claim

Take one proposed SAE feature through a complete causal audit: inspect top activations on held-out text, state a behavioral prediction, ablate or steer the decoder direction, and run matched negative controls. Save reconstruction damage and downstream behavior in an audit table beside the intervention result. If the effect doesn't replicate across held-out prompts or the SAE replacement degrades the model too much, keep the label as an unconfirmed hypothesis.