Mechanistic Interpretability

Imagine a code assistant that refuses a legitimate password-reset script. Was it responding to credentials, urgent wording, or something else? Suppose neuron 412 activates on credential resets, SQL queries, and urgent instructions, while neuron 789 responds to secret-manager docs and shell syntax. These are invented observations to frame the investigation. Neither establishes why the assistant refused.

That mix is polysemanticity: one neuron responds to several unrelated contexts. A raw coordinate needn't map cleanly to a single idea.

Mechanistic interpretability asks how the network computed that refusal. You have the activation numbers, but no coordinate that lines up with one of those reasons. Sparse autoencoders (SAEs) are one way to look for candidate units: reconstruct a dense activation as a sparse combination of learned directions, then test whether those directions participate in the computation. A dictionary is a starting point for an explanation, not a complete algorithm recovered from the model.

The previous chapter showed how layer normalization changes the scale read by transformer branches. It doesn't tell you what individual coordinate directions compute.

To explain why polysemanticity happens, researchers rely on the superposition hypothesis: models represent far more features than they have dimensions by packing them into overlapping, non-orthogonal directions.

Mechanistic interpretability researchers train sparse autoencoders (SAEs), networks that reconstruct language model activations through a wide, sparse bottleneck, to search for these underlying feature directions.^{[1]Reference 1Towards Monosemanticity: Decomposing Language Models With Dictionary Learninghttps://transformer-circuits.pub/2023/monosemantic-features/index.html[2]Reference 2Sparse Autoencoders Find Highly Interpretable Features in Language Modelshttps://arxiv.org/abs/2309.08600}

Bricken et al. and Cunningham et al. found features with clearer activation patterns than comparison directions in their experiments.^{[1]Reference 1Towards Monosemanticity: Decomposing Language Models With Dictionary Learninghttps://transformer-circuits.pub/2023/monosemantic-features/index.html[2]Reference 2Sparse Autoencoders Find Highly Interpretable Features in Language Modelshttps://arxiv.org/abs/2309.08600} A candidate might respond to credential-related instructions or urgent wording. Its label and its computational role are separate hypotheses: an intervention can establish an effect without proving that the label captures every context in which the feature fires.

Reconstruction and causality are distinct claims. A feature can help reconstruct activations while tracking a grammatical correlate of the behavior you care about. Controlled patching, ablation, and steering test the consequences of a specified edit. Held-out prompts, control edits, and collateral-capability measurements determine how far that result supports your explanation.

The problem: polysemantic neurons and superposition

Inside a transformer, residual-stream vectors and MLP hidden activations carry dense mixtures of information. If a model wants to track $M$ useful concepts, but its residual stream dimension is $D$ with $M \gg D$, how can it possibly store them?

An ambient space $\mathbb{R}^D$ holds at most $D$ mutually orthogonal unit vectors. That limits an orthogonal linear dictionary at one activation site. It does not limit a model to $D$ concepts over its lifetime: other layers, nonlinear computations, and context-dependent representations also matter.

The Johnson-Lindenstrauss lemma concerns a finite set of $N$ points: a suitable linear map into $D = O(\epsilon^{-2}\log N)$ dimensions preserves all pairwise squared Euclidean distances within factors $1\pm\epsilon$.^{[3]Reference 3Extensions of Lipschitz mappings into a Hilbert space.https://web.stanford.edu/class/cs114/readings/JL-Johnson.pdf} Related concentration arguments let high-dimensional spaces hold many unit directions with small pairwise inner products. For example, draw each direction independently with coordinates $\pm1/\sqrt D$. A concentration bound and a union bound give

Keeping the right side below a chosen failure probability lets $M$ grow exponentially with $D$ for fixed $\epsilon$. The bound can be loose or exceed one. It describes this random construction, not the exact capacity of a 4,096-dimensional residual stream or proof that a trained transformer uses such a packing.

Feature sparsity reduces interference in toy models of superposition.^{[4]Reference 4Toy Models of Superpositionhttps://transformer-circuits.pub/2022/toy_model/index.html} Residual interference follows the overlap bound: for unit directions with coherence $\mu=\max_{i\ne j}|d_i^T d_j|$ and $x=\sum_{i\in S}a_i d_i$, an inactive direction's readout obeys $|d_j^T x|\le\mu\sum_{i\in S}|a_i|$ for $j\notin S$.

The contributions accumulate with active magnitudes and support size. A threshold must suppress that leakage while retaining weak real signals. Active-latent counts describe a particular SAE recipe; estimate concept prevalence with a defined measurement protocol across models and tokens.

Predict whether a threshold of $1$ rejects the inactive direction below. Each active direction overlaps it by only $0.1$.

import math

def dot(left, right):
    return sum(a * b for a, b in zip(left, right))

inactive = [1.0, 0.0, 0.0]
active_directions = [[0.1, math.sqrt(0.99), 0.0],
                     [0.1, 0.0, math.sqrt(0.99)]]
magnitudes = [10.0, 10.0]
x = [sum(a * d[index] for a, d in zip(magnitudes, active_directions))
     for index in range(3)]
leakage = dot(inactive, x)
bound = 0.1 * sum(magnitudes)
print(f"inactive readout: {leakage:.2f}; coherence bound: {bound:.2f}")
print("threshold 1 falsely activates:", leakage > 1.0)
assert math.isclose(leakage, bound)

inactive readout: 2.00; coherence bound: 2.00
threshold 1 falsely activates: True

Such packing can produce polysemantic coordinates: several directions contribute to one axis. Polysemanticity alone doesn't establish superposition, however; even a rotated orthogonal dictionary can give mixed coordinates without having more features than dimensions. Residual coordinates and MLP neurons also differ: MLP activation functions introduce a coordinate-specific nonlinearity.

View larger

To make that geometric interference measurable, compute the cosine similarity between two candidate directions and construct their combined activation. If the urgency direction isn't orthogonal to the credential direction, predict how much coordinate A increases when both features fire simultaneously.

import math

credential = [1.0, 0.0]
urgency = [0.8, 0.6]

def cosine(left, right):
    dot = sum(a * b for a, b in zip(left, right))
    norm = math.sqrt(sum(a * a for a in left) * sum(b * b for b in right))
    return dot / norm

mixed_activation = [1.5 * credential[0] + urgency[0], 1.5 * credential[1] + urgency[1]]
print(f"credential/urgency cosine: {cosine(credential, urgency):.2f}")
print(f"axis A reading after both fire: {mixed_activation[0]:.2f}")

assert cosine(credential, urgency) > 0
assert mixed_activation[0] > 1.5

credential/urgency cosine: 0.80
axis A reading after both fire: 2.30

How sparse autoencoders produce candidate features

A sparse autoencoder attacks the superposition problem with overcomplete dictionary learning. It treats an activation vector $x \in \mathbb{R}^D$ as a sparse linear combination of directions drawn from a much wider learned dictionary with $M \gg D$. If the model packed features into non-orthogonal directions, the SAE's job is to unroll those directions onto separate latent axes.

An SAE consists of an encoder that projects the activation into a high-dimensional sparse code $z$, and a decoder that reconstructs the activation from that code:

• Encoder: $z = f\bigl(W_\text{enc}(x - b_\text{dec}) + b_\text{enc})\bigr)$ where $W_\text{enc} \in \mathbb{R}^{M \times D}$ with $M \gg D$, and $f$ is an activation function such as ReLU or a TopK operator.
• Decoder: $\hat{x} = W_\text{dec} z + b_\text{dec}$ where $W_\text{dec} \in \mathbb{R}^{D \times M}$. Each column $d_i = W_\text{dec}[:, i]$ represents a candidate feature direction in the model's activation space.

In this formulation, subtracting the learned decoder bias gives encoder and decoder a shared reference point.^{[1]Reference 1Towards Monosemanticity: Decomposing Language Models With Dictionary Learninghttps://transformer-circuits.pub/2023/monosemantic-features/index.html} That bias need not equal the exact dataset mean. Other SAE formulations omit pre-encoder subtraction, including the deliberately simple NumPy lab below.

Tied vs untied decoders

Weight tying ($W_\text{dec}=W_\text{enc}^T$) reduces matrix storage. Untied weights let detection and reconstruction directions differ, as in Bricken's SAE and Gao's TopK recipe. Cunningham's study used tied weights and reported no performance reduction on its residual-stream data, with some reduction on MLP data.^{[1]Reference 1Towards Monosemanticity: Decomposing Language Models With Dictionary Learninghttps://transformer-circuits.pub/2023/monosemantic-features/index.html[2]Reference 2Sparse Autoencoders Find Highly Interpretable Features in Language Modelshttps://arxiv.org/abs/2309.08600[5]Reference 5Scaling and Evaluating Sparse Autoencodershttps://cdn.openai.com/papers/sparse-autoencoders.pdf}

The reason comes down to an asymmetry in what the two layers do:

1. The encoder solves a feature detection problem under noise. Because other features in superposition create non-zero dot-product interference, the encoder needs sharp geometric selectivity and negative bias margins to keep false positives low.
2. The decoder adds its learned candidate direction $d_i$ to the reconstructed activation, weighted by $z_i$. The reconstruction lives at the chosen hook, which may be a branch output rather than the residual stream.

Tying restricts that independence, but its effect depends on the data and recipe. Tied initialization is a different choice: Gao initializes encoder weights from the decoder transpose, then allows the two matrices to train independently.^{[5]Reference 5Scaling and Evaluating Sparse Autoencodershttps://cdn.openai.com/papers/sparse-autoencoders.pdf}

The L1 shrinkage problem

The traditional training loss combines a mean squared error (MSE) reconstruction term with an L1 sparsity penalty:

For a positive latent, this mean-normalized sparsity term contributes $\lambda/M$ to $\partial\mathcal{L}/\partial z_i$. The optimizer updates encoder parameters through the chain rule, so it doesn't simply subtract $\eta\lambda$ from every latent each step. Nevertheless, the penalty favors smaller magnitudes even when support stays unchanged. With a fixed unit decoder in one dimension, minimizing $(a-z)^2+\lambda z$ over $z\ge0$ gives $z^*=\max(0,a-\lambda/2)$. This is L1 shrinkage. Sum/mean conventions change the coefficient. Unconstrained decoder norms introduce a separate scaling loophole discussed below.

Three widely studied designs address magnitude shrinkage in different ways:

1. TopK SAEs: A nonnegative variant keeps at most $k$ nonzero values: $z=\text{TopK}(\text{ReLU}(W_\text{enc}(x-b_\text{dec})+b_\text{enc}),k)$. It removes the L1 term from the main reconstruction objective, not all reasons an activation could decrease. Selection boundaries are non-smooth, and Gao's full recipe adds AuxK for dead latents. That study trained a 16-million-latent dictionary on GPT-4 activations; its main encoder omits $b_\text{enc}$.^{[5]Reference 5Scaling and Evaluating Sparse Autoencodershttps://cdn.openai.com/papers/sparse-autoencoders.pdf}
2. Gated SAEs: A binary gate selects a nonnegative magnitude estimate. The original recipe shares projection directions, with separate row scales and biases. L1 applies to rectified gate pre-activations. An auxiliary reconstruction through those gate pre-activations supplies a learning signal, with decoder parameters detached in that path. Thus the magnitude estimate has no direct L1 penalty, although shared parameters can still change.^{[6]Reference 6Improving Dictionary Learning with Gated Sparse Autoencodershttps://arxiv.org/abs/2404.16014}
3. JumpReLU SAEs: $\text{JumpReLU}(y,\theta)=y\mathbf{1}(y>\theta)$ keeps the full pre-activation above a learned positive threshold. An L0 penalty controls support. Naively differentiating the threshold gives zero almost everywhere; the paper uses custom windowed straight-through estimators, whose bandwidth affects bias and variance.^{[7]Reference 7Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencodershttps://arxiv.org/abs/2407.14435}

At low sparsity penalty $\lambda$, expect dense mixtures to win; at high $\lambda$, expect sparse codes to win. The small calculation below makes that objective switch visible.

def mse(target, reconstructed):
    return sum((a - b) ** 2 for a, b in zip(target, reconstructed)) / len(target)

def objective(target, reconstructed, code, sparsity_weight):
    mean_magnitude = sum(abs(value) for value in code) / len(code)
    return mse(target, reconstructed) + sparsity_weight * mean_magnitude

target = [1.0, 0.0]
sparse_code = [0.75, 0.0]
sparse_reconstruction = [0.75, 0.0]
dense_code = [0.75, 0.50]
dense_reconstruction = [1.0, 0.0]

for weight in [0.02, 0.20]:
    sparse_loss = objective(target, sparse_reconstruction, sparse_code, weight)
    dense_loss = objective(target, dense_reconstruction, dense_code, weight)
    chosen = "sparse" if sparse_loss < dense_loss else "dense"
    print(f"lambda={weight:.2f}: sparse={sparse_loss:.4f}, dense={dense_loss:.4f} -> {chosen}")

assert objective(target, dense_reconstruction, dense_code, 0.02) < objective(target, sparse_reconstruction, sparse_code, 0.02)
assert objective(target, sparse_reconstruction, sparse_code, 0.20) < objective(target, dense_reconstruction, dense_code, 0.20)

lambda=0.02: sparse=0.0387, dense=0.0125 -> dense
lambda=0.20: sparse=0.1063, dense=0.1250 -> sparse

The original "Towards Monosemanticity" study (2023) trained an SAE on a one-layer transformer, decomposing a 512-neuron MLP into 4,096 features in run A/1. Inspected features showed clean specialization for DNA sequences, Hebrew script, base64 text, and legal formatting.^{[1]Reference 1Towards Monosemanticity: Decomposing Language Models With Dictionary Learninghttps://transformer-circuits.pub/2023/monosemantic-features/index.html}

A year later, "Scaling Monosemanticity" applied dictionary learning to a middle-layer residual stream of Claude 3 Sonnet (the March 2024 production model). It scaled dictionary widths to 1 million, 4 million, and 34 million features, isolating abstract concepts like sycophancy, code vulnerabilities, and multilingual entity representations.^{[8]Reference 8Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnethttps://transformer-circuits.pub/2024/scaling-monosemanticity/}

def top_k_sparse(values, k):
    if type(k) is not int or not 0 <= k <= len(values):
        raise ValueError("k must be an integer between zero and the latent width")
    positive = [max(0.0, value) for value in values]
    # Stable sorting makes ties deterministic: earlier indices win.
    ranked = sorted(range(len(values)), key=lambda index: positive[index], reverse=True)
    keep = set(ranked[:k])
    return [value if index in keep else 0.0 for index, value in enumerate(positive)]

latent = [0.1, 2.4, 0.3, 1.7, 0.05]
sparse = top_k_sparse(latent, k=2)

assert sparse == [0.0, 2.4, 0.0, 1.7, 0.0]
assert sum(value != 0.0 for value in sparse) == 2
assert top_k_sparse([-2.0, 0.0, 1.0], 2) == [0.0, 0.0, 1.0]
assert top_k_sparse([2.0, 2.0], 1) == [2.0, 0.0]
assert top_k_sparse(latent, 0) == [0.0] * len(latent)
for invalid in [-1, 6, 1.5]:
    try:
        top_k_sparse(latent, invalid)
    except ValueError:
        pass
    else:
        raise AssertionError("invalid k accepted")

View larger

This pipeline separates two distinct questions:

1. Does the dictionary reconstruct the activation accurately while activating only a small handful of coordinates?
2. Do those learned directions correspond to true computational features that predict and control model behavior?

A low reconstruction error and high sparsity answer the first question. They don't automatically answer the second. That distinction determines how you choose hook points and how you validate features causally.

Where to attach SAEs inside a transformer

Researchers attach SAEs at several hook points across a transformer block:

• Residual stream after attention or MLP: The residual stream is the model's shared trunk communication channel. Hooking here captures information accumulated across all preceding layers.
• MLP output update: Captures the specific non-linear update written by that sublayer into the residual stream.
• Attention layer output: Decomposes the cross-token context routing information before it merges into the residual stream.

These sites can use the same SAE architecture, but a pretrained dictionary is tied to its exact checkpoint and hook. Record the layer, token position, pre/post normalization, attention output projection, activation dtype, and any centering or rescaling. Sites can share a width while differing in distribution. Joint dictionaries are possible too, but require an explicit multi-site training design.

Cunningham et al. reported mixed success on MLP data, with many inactive dictionary features in some middle and later layers. Untied weights retained more active features in those experiments. These results don't establish a universal ranking of activation sites: compare reconstruction, sparsity, and downstream behavior at the particular hook you plan to use.^{[2]Reference 2Sparse Autoencoders Find Highly Interpretable Features in Language Modelshttps://arxiv.org/abs/2309.08600}

Marks et al. trained sublayer SAEs in Pythia-70M and used existing Gemma Scope dictionaries for Gemma-2-2B.^{[9]Reference 9Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelshttps://arxiv.org/abs/2403.19647} The hook conventions differ: Gemma's attention dictionary in that study reads before the output projection, and its MLP dictionary reads after the branch output norm. Choosing a hook defines what your directions describe; matching a layer number alone isn't enough.

Feature circuits: from features to algorithms

A dictionary at one hook still doesn't say how that feature changes the next layer. Feature circuits are sparse subgraphs that try to show the path: earlier features feed later computations, and the last step is the next-token prediction.

Attribution methods suggest candidate dependencies across layers, but correlation along an edge isn't causal proof. A circuit hypothesis becomes credible only when intervention experiments (ablation, feature clamping, or activation patching) shift behavior in the exact direction the circuit predicts.^{[9]Reference 9Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelshttps://arxiv.org/abs/2403.19647}

Circuit studies investigate model- and task-specific mechanisms, with varying completeness:

• Induction heads that complete patterns such as [A][B] … [A] → [B]. Olsson's study gives causal evidence in small attention-only models and correlational evidence in larger models with MLPs.^{[10]Reference 10In-context Learning and Induction Heads.https://arxiv.org/abs/2209.11895}
• The indirect-object identification (IOI) circuit in GPT-2 small.^{[11]Reference 11Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallhttps://arxiv.org/abs/2211.00593}
• The greater-than comparison circuit in GPT-2 small.^{[12]Reference 12How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language modelhttps://arxiv.org/abs/2305.00586}
• Sparse feature circuits that explain complex behaviors in terms of human-interpretable feature nodes rather than uninterpretable neuron indices.^{[9]Reference 9Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelshttps://arxiv.org/abs/2403.19647}

Attribution graphs and cross-layer transcoders

Anthropic's 2025 method uses cross-layer transcoders (CLTs). A feature reads at its associated residual-stream layer and has separate decoder vectors contributing to MLP-output reconstructions at that layer and later ones. Each MLP output is reconstructed from features at its own and earlier layers. The model's attention operations remain; CLTs don't replace the entire transformer.^{[13]Reference 13Circuit Tracing: Revealing Computational Graphs in Language Modelshttps://transformer-circuits.pub/2025/attribution-graphs/methods.html}

For a chosen prompt, its local replacement model fixes attention patterns and normalization denominators to the original forward pass and adds back reconstruction errors as constants. Baseline activations and logits match by construction; evaluate edited responses separately. Its graph includes feature, token-embedding, unexplained-error, and selected output-logit nodes. Feature edges measure direct contributions to target pre-activations, with intervening MLP outputs stopped in the backward pass. A schematic edge is:

where $a_v$ is the target encoder pre-activation. This isn't the full model's unrestricted total derivative through every intermediate feature. Graph pruning selects a manageable hypothesis.^{[13]Reference 13Circuit Tracing: Revealing Computational Graphs in Language Modelshttps://transformer-circuits.pub/2025/attribution-graphs/methods.html}

The graph explains information flow given attention, not how Q/K computations formed that attention. Error nodes and inactive suppressed features can hide essential mechanisms. Check reconstruction fidelity, graph completeness, and interventions in the original model. The companion Biology study investigates particular behaviors of Claude 3.5 Haiku.^{[13]Reference 13Circuit Tracing: Revealing Computational Graphs in Language Modelshttps://transformer-circuits.pub/2025/attribution-graphs/methods.html[14]Reference 14On the Biology of a Large Language Modelhttps://transformer-circuits.pub/2025/attribution-graphs/biology.html}

View larger

def refusal_score(features):
    weights = {"credential_safety": 1.2, "urgency": 0.1, "formatting": 0.0}
    return sum(features[name] * weights[name] for name in weights)

observed = {"credential_safety": 0.9, "urgency": 0.8, "formatting": 0.7}
credential_ablated = {**observed, "credential_safety": 0.0}
formatting_ablated = {**observed, "formatting": 0.0}

baseline = refusal_score(observed)
target_change = baseline - refusal_score(credential_ablated)
control_change = baseline - refusal_score(formatting_ablated)
print(f"target ablation change: {target_change:.2f}")
print(f"negative-control change: {control_change:.2f}")

assert target_change > 1.0
assert control_change == 0.0

target ablation change: 1.08
negative-control change: 0.00

The negative control is unchanged because we assigned it zero weight in the synthetic function. In a real language model, an ablation experiment requires held-out prompts, matched control directions, and measurements of collateral capabilities (like language modeling perplexity or code execution accuracy). A null ablation could also reflect redundant parallel pathways or an underpowered intervention, so negative results require careful verification.

Activation steering with SAE features

Because each SAE feature corresponds to a specific column $d_f = W_\text{dec}[:, f]$ in the decoder matrix, we can use that direction for activation steering at inference time without modifying model weights:

1. Run the forward pass and cache the residual-stream activation vector $x$ at the target layer.
2. Add a scaled copy of the feature's decoder direction: $x_\text{steered} = x + \alpha \cdot d_f$.
3. Continue the forward pass using $x_\text{steered}$.

To set one coefficient in the cached SAE decomposition to $c$ while preserving its reconstruction error, use feature clamping:

where $z_f(x)$ is the encoder's computed activation for feature $f$.

This changes a coefficient in the original decomposition. Re-encoding $x+\Delta x$ need not return $z_f=c$, because encoder and decoder aren't necessarily dual, thresholds can switch, and other feature readouts can move. Here a unit decoder direction still misses the requested re-encoded target:

def encode_feature(x):
    return max(0.0, x[0] + 2.0 * x[1])

x = [1.0, 0.5]
decoder_direction = [0.0, 1.0]
old_coefficient = encode_feature(x)
target = 3.0
edited = [value + (target - old_coefficient) * direction
          for value, direction in zip(x, decoder_direction)]
print(f"cached coefficient: {old_coefficient:.1f}; requested: {target:.1f}")
print(f"re-encoded coefficient: {encode_feature(edited):.1f}")
print(f"overlapping readout: {sum(x):.1f} -> {sum(edited):.1f}")
assert encode_feature(edited) != target

cached coefficient: 2.0; requested: 3.0
re-encoded coefficient: 4.0
overlapping readout: 1.5 -> 2.5

Whether a hypothetical credential-labeled feature increases refusal, and at what scale, must be measured. Anthropic's 2024 Sonnet experiments amplified a Golden Gate Bridge feature to induce bridge-related responses, with effects varying by feature and prompt. Their edits preserved error and ran at every token position; extreme scales could produce nonsense.^{[8]Reference 8Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnethttps://transformer-circuits.pub/2024/scaling-monosemanticity/} Arditi et al. studied difference-of-means refusal directions across 13 open chat models, rather than SAE-derived safety classifiers. Their results don't establish one universal direction or semantic safety boundary for every model.^{[15]Reference 15Refusal in Language Models Is Mediated by a Single Directionhttps://arxiv.org/abs/2406.11717}

View larger

def add_scaled_direction(vector, direction, scale):
    if len(vector) != len(direction):
        raise ValueError("activation and direction must have the same width")
    return [value + scale * delta for value, delta in zip(vector, direction)]

activation = [0.2, -0.1, 0.0, 0.4]
credential_feature_direction = [0.0, 0.5, 0.5, 0.0]
steered = add_scaled_direction(activation, credential_feature_direction, scale=2.0)

assert steered == [0.2, 0.9, 1.0, 0.4]
assert steered[1] > activation[1]
assert steered[2] > activation[2]

def add_scaled(vector, direction, scale):
    return [value + scale * delta for value, delta in zip(vector, direction)]

def project(vector, direction):
    return sum(value * delta for value, delta in zip(vector, direction))

activation = [0.2, 0.1]
candidate_direction = [0.0, 1.0]
unrelated_direction = [1.0, 0.0]
overlapping_direction = [0.6, 0.8]

for alpha in [-1.0, 0.0, 1.0, 2.0]:
    steered = add_scaled(activation, candidate_direction, alpha)
    score = project(steered, candidate_direction)
    control = project(steered, unrelated_direction)
    overlap = project(steered, overlapping_direction)
    print(f"alpha={alpha:+.1f}: candidate={score:+.1f}, orthogonal={control:+.1f}, overlapping={overlap:+.1f}")

assert project(add_scaled(activation, candidate_direction, 2.0), unrelated_direction) == 0.2

alpha=-1.0: candidate=-0.9, orthogonal=+0.2, overlapping=-0.6
alpha=+0.0: candidate=+0.1, orthogonal=+0.2, overlapping=+0.2
alpha=+1.0: candidate=+1.1, orthogonal=+0.2, overlapping=+1.0
alpha=+2.0: candidate=+2.1, orthogonal=+0.2, overlapping=+1.8

Keep reconstruction error separate from feature ablation

Any SAE reconstruction leaves an unexplained residual error $e = x - \hat{x}$. The original activation decomposes as:

When testing the causal effect of feature $f$, an error-preserving ablation subtracts only that feature's reconstructed contribution from the clean activation vector:

A dangerous beginner mistake is naive ablation: setting $z_f = 0$ inside the SAE reconstruction, yielding $x'_\text{naive} = \hat{x} - z_f d_f$. That naive step removes feature $f$ and simultaneously discards the unexplained residual $e$. If downstream attention heads or MLP layers rely on signals inside $e$, the model's performance drops, and researchers falsely attribute that drop to the ablated feature!

A small numerical calculation isolates this trap. Here the SAE has a tiny reconstruction error ($\text{MSE} = 0.005$), but the downstream readout relies entirely on the second coordinate. Predict whether naive ablation produces a false causal signal before running the check.

x = [1.0, 0.1]
x_hat = [1.0, 0.0]
feature_contribution = [1.0, 0.0]  # z_f * decoder_column_f

def readout(vector):
    return 100.0 * vector[1]

preserved = [a - b for a, b in zip(x, feature_contribution)]
naive = [a - b for a, b in zip(x_hat, feature_contribution)]
mse = sum((a - b) ** 2 for a, b in zip(x, x_hat)) / len(x)
print(f"reconstruction MSE: {mse:.3f}")
for name, vector in [("original", x), ("reconstructed", x_hat),
                     ("error-preserving ablation", preserved), ("naive ablation", naive)]:
    print(f"{name}: {readout(vector):.1f}")
assert readout(preserved) == readout(x) == 10.0
assert readout(naive) == readout(x_hat) == 0.0

reconstruction MSE: 0.005
original: 10.0
reconstructed: 0.0
error-preserving ablation: 10.0
naive ablation: 0.0

View larger

For gradient-based attribution, also hold the original error fixed in the backward pass. If you insert decode(z) + (x - decode(z)) with both copies of decode(z) attached to autograd, their derivatives cancel. The forward value is correct, but every latent gradient is zero. Marks et al. stop gradients through the error to compute feature gradients.^{[9]Reference 9Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelshttps://arxiv.org/abs/2403.19647}

Predict which of these numerically identical forward passes reports the feature's effect on the score.

import torch

z = torch.tensor([1.0], dtype=torch.float64, requires_grad=True)
direction = torch.tensor([1.0, 0.0], dtype=torch.float64)
original = torch.tensor([1.0, 0.1], dtype=torch.float64)
readout = torch.tensor([3.0, 100.0], dtype=torch.float64)
decoded = z[0] * direction
error = original - decoded

live_score = (decoded + error) @ readout
fixed_score = (decoded + error.detach()) @ readout
live_gradient = torch.autograd.grad(live_score, z, retain_graph=True)[0]
fixed_gradient = torch.autograd.grad(fixed_score, z)[0]
print(f"forward scores: live={live_score.item():.1f}, fixed={fixed_score.item():.1f}")
print(f"latent gradients: live={live_gradient.item():.1f}, fixed={fixed_gradient.item():.1f}")
torch.testing.assert_close(live_score, fixed_score)
torch.testing.assert_close(live_gradient, torch.zeros_like(z))
torch.testing.assert_close(fixed_gradient, torch.full_like(z, 3.0))

forward scores: live=13.0, fixed=13.0
latent gradients: live=0.0, fixed=3.0

The gradient $3$ measures changing this latent while holding the original error constant. Recomputing the error after editing the latent would undo the intervention. This example covers one hook and one feature. For gradients through multiple upstream SAE hooks, detaching errors alone changes upstream derivatives; Marks et al. also use pass-through gradients to preserve those paths. Follow the attribution method's full backward specification rather than applying detach() everywhere.^{[9]Reference 9Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelshttps://arxiv.org/abs/2403.19647}

For an LLM, measure downstream language-model loss as well as activation MSE. Loss Recovered expresses one baseline-relative comparison:

Here $\mathcal{L}_\text{clean}$ uses original activations, $\mathcal{L}_\text{SAE}$ substitutes reconstructions, and $\mathcal{L}_\text{ablated}$ uses a specified mean/zero baseline at the same hooks and positions. The ratio is undefined at a zero denominator and unstable near zero. It can be negative or exceed one. A very destructive baseline can make substantial damage look like excellent recovery; Gao explicitly cautions about residual-stream zero-ablation.^{[5]Reference 5Scaling and Evaluating Sparse Autoencodershttps://cdn.openai.com/papers/sparse-autoencoders.pdf} Report all three losses, $\Delta\mathcal{L}=\mathcal{L}_\text{SAE}-\mathcal{L}_\text{clean}$, and task effects too.

Same SAE, same loss damage, different baseline. Which number would you trust without seeing the raw losses?

import math

def loss_recovered(clean, reconstructed, ablated):
    losses = (clean, reconstructed, ablated)
    if not all(math.isfinite(value) and value >= 0 for value in losses):
        raise ValueError("losses must be finite and nonnegative")
    denominator = ablated - clean
    if denominator <= 1e-12:
        raise ValueError("baseline must increase loss by more than the chosen tolerance")
    return 1.0 - (reconstructed - clean) / denominator

for baseline in [20.0, 2.4]:
    recovered = loss_recovered(2.0, 2.2, baseline)
    print(f"baseline={baseline:.1f}; delta loss=0.2; recovered={recovered:.3f}")
print(f"worse than baseline: {loss_recovered(2.0, 5.0, 4.0):.2f}")
print(f"better than clean: {loss_recovered(2.0, 1.9, 4.0):.2f}")
try:
    loss_recovered(2.0, 2.2, 2.0)
except ValueError:
    print("zero denominator: rejected")

baseline=20.0; delta loss=0.2; recovered=0.989
baseline=2.4; delta loss=0.2; recovered=0.500
worse than baseline: -0.50
better than clean: 1.05
zero denominator: rejected

A toy NumPy SAE you can run today

The lab fits a small SAE on synthetic mixtures of 24 known directions in eight dimensions. Copy the training, summary, and alignment blocks into one file in that order, then run uv run --with numpy python toy_sae.py. This is an uncentered ReLU encoder with a decoder bias, full-batch SGD, and unit-normalized decoder columns. It omits production features such as activation streaming, AuxK, and a learned threshold.

Before running it, keep three separate evaluation criteria in mind:

1. Reconstruction MSE should fall.
2. Count active latents, rather than assuming the sparsity penalty recovered the generator's 2–4 active features.
3. Decoder columns should align with the known generating directions.

Those are three distinct mathematical claims.

import numpy as np

# --- Synthetic data with known superposition ---
rng = np.random.default_rng(42)
d_model = 8
n_true_features = 24          # more features than dimensions
n_samples = 768

# Ground-truth feature directions (unit vectors)
true_features = rng.normal(size=(n_true_features, d_model))
true_features /= np.linalg.norm(true_features, axis=1, keepdims=True)

# Each sample activates only 2-4 features (sparse)
activations = np.zeros((n_samples, d_model))
for i in range(n_samples):
    active_idx = rng.choice(n_true_features, size=rng.integers(2, 5), replace=False)
    coeffs = rng.normal(loc=1.5, scale=0.4, size=len(active_idx))
    activations[i] = (coeffs[:, None] * true_features[active_idx]).sum(axis=0)

# Add a little noise
activations += 0.03 * rng.normal(size=activations.shape)

# --- Tiny SAE implementation ---
n_features = 64               # 8x expansion
learning_rate = 0.03
lambda_sparsity = 5.0         # L1 coefficient
epochs = 300

W_enc = rng.normal(scale=0.1, size=(n_features, d_model))
b_enc = np.zeros(n_features)
W_dec = rng.normal(scale=0.1, size=(d_model, n_features))
W_dec /= np.linalg.norm(W_dec, axis=0, keepdims=True) + 1e-12
initial_W_dec = W_dec.copy()
b_dec = np.zeros(d_model)

def encode(x: np.ndarray):
    pre_activation = x @ W_enc.T + b_enc
    z = np.maximum(0.0, pre_activation)
    return pre_activation, z

def reconstruct(x: np.ndarray):
    _, z = encode(x)
    return z, z @ W_dec.T + b_dec

def reconstruction_loss(x: np.ndarray) -> float:
    _, x_hat = reconstruct(x)
    return float(np.mean((x - x_hat) ** 2))

def train_sae(x: np.ndarray) -> None:
    global W_enc, b_enc, W_dec, b_dec
    for epoch in range(epochs):
        # Forward
        pre_activation, z = encode(x)
        x_hat = z @ W_dec.T + b_dec

        # Loss
        recon_loss = np.mean((x - x_hat) ** 2)
        sparsity_loss = np.mean(np.abs(z))
        total_loss = recon_loss + lambda_sparsity * sparsity_loss

        # Gradients (manual backprop for clarity)
        d_recon = 2 * (x_hat - x) / x.size
        dW_dec = d_recon.T @ z
        db_dec = d_recon.sum(axis=0)

        dz = d_recon @ W_dec
        dz += (lambda_sparsity / z.size) * np.sign(z)
        dz[pre_activation <= 0] = 0.0                    # ReLU backward
        dW_enc = dz.T @ x
        db_enc = dz.sum(axis=0)

        # SGD step
        W_dec -= learning_rate * dW_dec
        b_dec -= learning_rate * db_dec
        W_enc -= learning_rate * dW_enc
        b_enc -= learning_rate * db_enc

        # Keep decoder columns normalized so scale can't move from z into W_dec.
        W_dec /= np.linalg.norm(W_dec, axis=0, keepdims=True) + 1e-12

        if epoch % 100 == 0:
            print(f"Epoch {epoch:4d} | total={total_loss:.4f} | recon={recon_loss:.4f} | sparsity={sparsity_loss:.4f}")

initial_recon = reconstruction_loss(activations)
train_sae(activations)

Epoch    0 | total=1.9426 | recon=1.4259 | sparsity=0.1033
Epoch  100 | total=0.5301 | recon=0.2370 | sparsity=0.0586
Epoch  200 | total=0.4394 | recon=0.1564 | sparsity=0.0566

First summarize the training set, then evaluate fresh held-out mixtures from the same generating dictionary.

final_recon = reconstruction_loss(activations)
z, _ = reconstruct(activations)
active_per_sample = (z > 0).sum(axis=1).mean()

print(f"Training complete: recon {initial_recon:.3f} -> {final_recon:.3f}, active features {active_per_sample:.1f}")

# Fresh samples from the same known dictionary; no fitting on these samples.
test_rng = np.random.default_rng(123)
heldout = np.zeros((256, d_model))
for i in range(len(heldout)):
    indices = test_rng.choice(n_true_features, size=test_rng.integers(2, 5), replace=False)
    coefficients = test_rng.normal(1.5, 0.4, size=len(indices))
    heldout[i] = (coefficients[:, None] * true_features[indices]).sum(axis=0)
heldout += 0.03 * test_rng.normal(size=heldout.shape)
test_z, test_reconstruction = reconstruct(heldout)
test_mse = np.mean((heldout - test_reconstruction) ** 2)
mean_baseline_mse = np.mean((heldout - activations.mean(axis=0)) ** 2)
fvu = test_mse / mean_baseline_mse
print(f"Held-out MSE: {test_mse:.3f}; mean baseline: {mean_baseline_mse:.3f}; FVU: {fvu:.3f}")
print(f"Held-out active features: {(test_z > 0).sum(axis=1).mean():.1f}")
assert final_recon < initial_recon
assert test_mse < mean_baseline_mse

Training complete: recon 1.426 -> 0.128, active features 18.9
Held-out MSE: 0.128; mean baseline: 0.757; FVU: 0.169
Held-out active features: 19.5

The held-out sample tests generalization to fresh linear mixtures of the 24 ground-truth directions. Its Fraction of Variance Unexplained (FVU) is 0.169, showing solid reconstruction. But does low FVU prove we recovered the true ground-truth feature dictionary?

The alignment diagnostic below measures the maximum absolute cosine similarity between each true feature and all learned decoder columns.

def direction_alignment(decoder: np.ndarray, ignore_sign=True):
    columns = decoder.T.copy()
    columns /= np.linalg.norm(columns, axis=1, keepdims=True) + 1e-12
    similarities = true_features @ columns.T
    if ignore_sign:
        similarities = np.abs(similarities)
    strongest_per_truth = similarities.max(axis=1)
    return float(np.median(strongest_per_truth)), int((strongest_per_truth >= 0.90).sum())

initial_median, initial_high_matches = direction_alignment(initial_W_dec)
trained_median, trained_high_matches = direction_alignment(W_dec)
print(f"median max |cosine|: {initial_median:.3f} -> {trained_median:.3f}")
print(f"true directions at >= 0.90: {initial_high_matches} -> {trained_high_matches}")
initial_signed, _ = direction_alignment(initial_W_dec, ignore_sign=False)
trained_signed, _ = direction_alignment(W_dec, ignore_sign=False)
print(f"median max signed cosine: {initial_signed:.3f} -> {trained_signed:.3f}")

median max |cosine|: 0.798 -> 0.799
true directions at >= 0.90: 1 -> 2
median max signed cosine: 0.695 -> 0.710

Reconstruction MSE dropped by over $90\%$ ($1.426 \to 0.128$), while median cosine alignment barely budged ($0.798 \to 0.799$)!

Sparse reconstruction metrics describe reconstruction and sparsity. Feature identification requires additional assumptions; state which ones an experiment evaluates before assigning meanings. The maximum-match diagnostic also permits multiple true directions to match the same decoder column, and absolute cosine ignores the sign that a nonnegative latent can write. Keep the random-initialization baseline and signed diagnostic. In an LLM, ground-truth directions are usually unavailable, so use semantic counterexamples and causal tests together.

Key implementation choices in the NumPy trainer

• The generator combines 2–4 of 24 non-orthogonal directions in eight dimensions. This known construction lets us test recovery separately from reconstruction.
• Nonzero decoder columns are renormalized after each step. Otherwise, shrinking $z$ and expanding decoder columns reciprocally can preserve reconstruction while reducing this unweighted L1 penalty. A decoder-norm-weighted penalty is another way to address that reparameterization.^{[7]Reference 7Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencodershttps://arxiv.org/abs/2407.14435}
• The subgradient of the L1 term ($\text{sign}(z)$) is added to the backward gradient before the ReLU step.
• About $18.9/64=29.5\%$ of slots fire on training samples, and $19.5$ on held-out samples. That's less dense than using every slot, but much denser than the generator's 2–4 active directions. Production sparsity and concept recovery need separate measurements.

View larger

Practical training details and pitfalls

These failure modes can appear in small toys as well as large-model training. Their prevalence depends on data, architecture, and optimization:

Dead latents: tracking and revival

A dead latent is inactive according to a specified monitoring rule. Gao's recipe uses no activation over 10 million tokens; a low but nonzero rate is a different criterion.^{[5]Reference 5Scaling and Evaluating Sparse Autoencodershttps://cdn.openai.com/papers/sparse-autoencoders.pdf} Four observations with no firing only show "not seen in this window," not permanent death. Check rare-domain coverage, observation length, and the rule before resampling a potentially useful rare feature.

Why do latents die? An unlucky initialization, strong sparsity pressure, or competing latents that consistently fire earlier can leave a feature with negative bias and zero gradients.

Two primary mechanisms revive dead latents:

1. Anthropic resampling (2023): At steps 25k/50k/75k/100k, the published recipe checks inactivity over the preceding 12.5k steps. It samples input vectors with probabilities proportional to squared per-input SAE loss. The decoder receives the normalized sampled input, not its reconstruction residual. The encoder uses the same direction at $0.2\times$ the average alive-row norm; its bias and modified Adam state are reset. Resampling can cause loss spikes and isn't mandatory in every recipe.^{[1]Reference 1Towards Monosemanticity: Decomposing Language Models With Dictionary Learninghttps://transformer-circuits.pub/2023/monosemantic-features/index.html}
2. AuxK (Gao, 2024): Select promising dead latents to predict the main error $e=x-\hat{x}$ with an auxiliary decoder pass. The paper typically uses weight $1/32$, alongside tied initialization. Its 16M-latent run reported 7% dead latents; results depend on dictionary and dataset.^{[5]Reference 5Scaling and Evaluating Sparse Autoencodershttps://cdn.openai.com/papers/sparse-autoencoders.pdf}

latent_batches = [
    [1.2, 0.0, 0.0, 0.0],
    [0.7, 0.3, 0.0, 0.0],
    [0.0, 0.0, 0.0, 0.0],
    [0.0, 0.0, 0.0, 0.5],
]

rates = [
    sum(row[index] > 0 for row in latent_batches) / len(latent_batches)
    for index in range(len(latent_batches[0]))
]
never_observed = [index for index, rate in enumerate(rates) if rate == 0.0]
print("firing rates:", [round(rate, 2) for rate in rates])
print("never observed in this window:", never_observed)

assert never_observed == [2]

firing rates: [0.5, 0.25, 0.0, 0.25]
never observed in this window: [2]

Non-identifiability, absorption, and splitting

Evaluating SAEs requires watching out for three subtle structural failure modes:

Absorption can improve sparsity without changing reconstruction. In a toy hierarchy, a "Paris" instance always includes "capital city." A combined decoder direction can write both while the general detector stays silent:

parent = [1.0, 0.0]
child = [0.0, 1.0]
target = [1.0, 1.0]  # general and specific concepts are both present
interpretable_code = [1.0, 1.0]
absorbed_code = [0.0, 1.0]
combined_direction = [a + b for a, b in zip(parent, child)]
reconstruction = [absorbed_code[0] * parent[i]
                  + absorbed_code[1] * combined_direction[i] for i in range(2)]
assert reconstruction == target
print("active latents:", sum(v != 0 for v in interpretable_code),
      "->", sum(v != 0 for v in absorbed_code))
print(f"general latent: {absorbed_code[0]:.1f}; general coordinate: {reconstruction[0]:.1f}")

active latents: 2 -> 1
general latent: 0.0; general coordinate: 1.0

This hand-computed L0 example isolates support bookkeeping. With unit-normalized combined direction $[1,1]/\sqrt2$, its coefficient becomes $\sqrt2$; support remains one. The child direction now writes general information too. Chanin's study investigates absorption in first-letter tasks and shows why sweeping width or sparsity alone needn't fix it.^{[16]Reference 16A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencodershttps://arxiv.org/abs/2409.14507}

Applications to safety and governance

Safety evaluations require the highest standard of evidence. Templeton et al. demonstrated that SAEs can surface safety-relevant features in Claude 3 Sonnet, isolating directions for scam emails, code exploits, bias, and sycophantic agreement.^{[8]Reference 8Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnethttps://transformer-circuits.pub/2024/scaling-monosemanticity/}

Some SAE features generalize across languages or modalities in particular studies.^{[8]Reference 8Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnethttps://transformer-circuits.pub/2024/scaling-monosemanticity/} Others track token strings, formatting, or only part of a concept. Test a proposed semantic label with French translations, base64 encodings, and other counterexamples; activation can vary with form. Absorption also makes a silent latent weak evidence that a concept is absent.

Return to the imagined password-reset refusal. Compare benign and harmful credential-related prompts, vary urgency separately, and track proposed features at matched hooks. Test whether edits change a chosen refusal metric and whether legitimate task completion improves. A change in refusal wording alone doesn't establish corrected security judgment or preserved safeguards.

What changed in 2026?

Checked September 22, 2026. Heap et al.'s ICLR 2026 study compared residual-stream SAEs on trained, randomized, and initialization checkpoints from the Pythia family. Aggregate automated interpretability scores and reconstruction metrics often looked similar, especially at larger sizes. This challenges those metrics as stand-alone evidence of learned computation; it doesn't show that every SAE feature is meaningless. Add appropriate null-model baselines to metric-driven claims.^{[17]Reference 17Automated Interpretability Metrics Do Not Distinguish Trained and Random Transformershttps://proceedings.iclr.cc/paper_files/paper/2026/hash/7ab63a5314680e2f083cb288abeaeb8e-Abstract-Conference.html}

Anthropic's May 7, 2026 natural language autoencoders instead learn an activation → text → activation round trip. They can help generate auditing hypotheses, but their explanations can hallucinate and their verbalizer can infer beyond the stored activation. They don't supply a feature circuit or a verified transcript of reasoning. Corroborate claims with independent methods; readable explanations inherit the same proxy-versus-mechanism problem.^{[18]Reference 18Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activationshttps://transformer-circuits.pub/2026/nla/index.html}

Limits and caution

An SAE can propose directions and still leave most of the computation unexplained. Four limits keep a dictionary from being a finished account of how the model works:

1. Reconstruction error matters: Substituting reconstructions can degrade model performance. Cunningham et al.'s layer-2 Pythia-70M experiment raised Pile perplexity from 25 to 40; test the effect on the target model and intervention.^{[2]Reference 2Sparse Autoencoders Find Highly Interpretable Features in Language Modelshttps://arxiv.org/abs/2309.08600}
2. Coverage is incomplete: The Sonnet study reports missing features and no reliable total-feature count.^{[8]Reference 8Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnethttps://transformer-circuits.pub/2024/scaling-monosemanticity/} Reconstruction errors, rare contexts, and features spread across layers can leave relevant computation unexplained. Measure coverage with held-out contexts and circuit tests; dictionary width alone cannot establish it.
3. Feature splitting can change the units: The Sonnet study observed concepts splitting into narrower sub-features as dictionary size grew.^{[8]Reference 8Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnethttps://transformer-circuits.pub/2024/scaling-monosemanticity/} Individual feature indices shouldn't be treated as immutable atomic units.
4. Labels are hypotheses: Top activations suggest a name, while lower activations and semantic counterexamples test its precision and recall. Causal experiments test the specified direction's role; universal semantic labels, a unique dictionary, and complete understanding require broader evidence.

What to check before moving on

• Distinguish finite-point JL distance preservation from a claim about features learned by a transformer; calculate accumulated interference under a coherence bound.
• Explain how mixed coordinates can arise and why polysemanticity alone doesn't establish superposition.
• Compare tied weights, tied initialization, untied detection/reconstruction, and optional learned-reference subtraction.
• Identify the L1 shrinkage problem and explain how TopK, Gated, and JumpReLU SAEs eliminate or mitigate it.
• Report activation MSE, L0, raw downstream losses, and baseline-relative Loss Recovered without treating any one metric as feature recovery.
• Diagnose dead latents and contrast Anthropic resampling with OpenAI's AuxK auxiliary loss.
• Explain the difference between naive ablation and error-preserving ablation, identifying why discarding the residual $e$ creates false causal claims.
• Trace CLT input/output layers and explain local attribution graphs' frozen attention, normalization denominators, error nodes, and omitted mechanisms.

Practice checkpoints

Common failures and fixes

• Symptom: A raw coordinate responds to unrelated contexts. Possible cause: Mixed representations. Check: Compare candidate directions, their activation ranges, and controlled effects before attributing the behavior to an overcomplete dictionary.
• Symptom: MSE falls but labels stay messy. Possible causes: Dense codes, absorption, splitting, or poor labels. Check: Sweep width/sparsity with matched data, test recall and seeds, and report downstream loss. No universal $M/D\ge16$ rule resolves these problems; tighter sparsity can worsen absorption.
• Symptom: A feature label sounds convincing, but ablation changes nothing. Possible causes: The label is correlational, the edit magnitude is ineffective, or parallel pathways compensate. Fix: Check the hook point, edit magnitude, and alternative pathways before interpreting the null result.
• Symptom: Many slots never fire in the observation window. Possible causes: Competition, initialization, sparsity, or missing rare contexts. Check: State the window and sample coverage before trying a revival recipe; measure its side effects.
• Symptom: Steering works on a demo but damages other prompts. Possible causes: Overlap, scale, wrong hook, or changed attention. Check: Match the checkpoint, hook, and token positions, then sweep edits and controls on held-out tasks. A fixed orthogonal probe only tracks that probe; held-out tasks test downstream capabilities.

Practice projects

1. CPU first: Sweep the toy's sparsity coefficient and training seed. Record held-out MSE/FVU, L0, signed alignment, and random-initialization alignment. Try to improve recovery rather than only reconstruction.
2. Inspect an existing dictionary: Use SAELens with its matching model/hook. Sample top, middle, low, and non-activating contexts; test a label's precision and recall. Its current inference interface supports generic PyTorch activation extraction, not only TransformerLens.
3. Train a dictionary if compute allows: Freeze a small model and stream activations from a stated corpus. Plan memory: 200k float32 vectors of width 4,096 alone occupy about 3.05 GiB at one hook, before model, SAE, gradients, and optimizer state. Compare L1/TopK under matched width, support, data, and compute; assess coverage on held-out contexts, especially rare ones.
4. Test an algorithm: Choose a model-specific comparison task such as greater-than year completion. Find candidate SAE components, preserve error, and test both targeted and control edits. Recovering a known head-level circuit in SAE coordinates remains an open research goal.^{[12]Reference 12How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language modelhttps://arxiv.org/abs/2305.00586}
5. Trace a prompt: Follow circuit-tracer's tutorial with a supported model and pretrained transcoders, then test one graph prediction. A standard SAE alone isn't a CLT. Record the graph's frozen paths, error nodes, edit positions, and original-model behavior.

Tool compatibility checked October 10, 2026: SAELens documents a v6 training refactor. TransformerLens now recommends TransformerBridge; its README states that the legacy HookedTransformer.from_pretrained API was removed in 4.0. Record installed versions and consult their migration guides before copying older notebooks. Weight folding/centering can change activation coordinates, so match a dictionary's training conventions as well as its model name.

Validate one feature claim

Take one proposed SAE feature through a complete causal audit. Inspect top activations on held-out text, state a behavioral prediction, ablate or steer the decoder direction, and run matched negative controls. Save reconstruction damage and downstream behavior in an audit table beside the intervention result.

If the effect doesn't replicate across held-out prompts or the SAE replacement degrades the model too much, keep the label as an unconfirmed hypothesis.