10. Features

Draft

Neurons that mean several things

A space of three dimensions has three orthogonal directions and no fourth. A small network asked to pass more than three quantities through those dimensions must either drop some or represent them along directions that interfere. Which solution it learns depends on how rarely each quantity is present.

model     x̂ = ReLU(Wᵀ W x + b): 8 features squeezed into 3 numbers and read back. A feature is 0 with
          probability S (its sparsity), else uniform on [0, 1]; feature i weighs 0.9^i in the loss
drawn     the 8 columns of W as arrows from the origin of the 3 hidden dimensions, with the convex hull of
          their tips; the length and the overlap of each arrow as bars; Wᵀ W as a grid; the loss by step
trained   in the page: Adam, 6,000 steps of 1,024 inputs, one random start a run

sparsity   kept     the arrows stand as                     loss ÷ loss of 3 kept, 5 dropped
0.00       3 of 8   three directions at right angles        1.00
0.70       6 of 8   an octahedron (three opposite pairs)    0.67
0.90       6 of 8   an octahedron (three opposite pairs)    0.43
0.95       8 of 8   a square antiprism                      0.24
0.99       8 of 8   a square antiprism                      0.11

(the live figure needs JavaScript; it loads about 0.7 MB of three.js for the box and no data)
A toy model of superposition, trained in the page: x̂ = ReLU(WᵀW x + b) squeezes n sparse features into three numbers and reads them back. Green: the direction of each feature, a column of W. Hairlines: the convex hull of their tips. Bars: each arrow’s length and its overlap with the others. The grid is WᵀW; the curve is the loss over that of keeping three features and dropping the rest. Change the features, their sparsity or importance, or press train for another random start: dense features leave three arrows at right angles, sparse ones pack into a solid.

With eight input quantities, or features, each absent from 95% of inputs, the trained network keeps all eight. Their directions form the corners of a square antiprism, at a quarter of the loss that keeping three and dropping five would cost. With dense features it keeps three orthogonal directions; in between, six form an octahedron of three opposite pairs. Nothing in the training asks for those shapes, and another random start can end in another solid.

The experiment matters for what Section 9 did. Its circuits ran between attention heads and neurons, and Section 5 read the neurons of a block's feed-forward layer, its MLP, as memories with a key and a value. Both take the neuron as the unit of meaning. In vision models that unit had already failed:

This essay may be giving you an overly rosy picture: perhaps every neuron yields a nice, human-understandable concept if one seriously investigates it?

Alas, this is not the case. Neural networks often contain “polysemantic neurons” that respond to multiple unrelated inputs. For example, InceptionV1 contains one neuron that responds to cat faces, fronts of cars, and cat legs.

—C. Olah and others, "Zoom In: An Introduction to Circuits" (2020). Distill, 2020, section "Polysemantic neurons", its first two paragraphs.

gpt-pp, the 12-layer model of this series, has such neurons. Neuron 148 of the MLP in block 10 is one example. Among its 25 strongest triggers in the training text are "the Catholic Social Teachings", "Nightmares" and "Colour perception", the token that excites it in italics: seven of the 25 are the word social, seven night or light, five the end of colour or the word color, three the piece Dec.

The explanation proposed in 2022 is that a trained network packs features together in this way.

Toy Models of SuperpositionTransformer Circuits Thread 2022 · arXiv:2209.10652N. Elhage, T. Hume, C. Olsson and others

The Superposition Hypothesis

Even when there is a privileged basis, it's often the case that neurons are "polysemantic", responding to several unrelated features. One explanation for this is the superposition hypothesis. Roughly, the idea of superposition is that neural networks "want to represent more features than they have neurons", so they exploit a property of high-dimensional spaces to simulate a model with many more neurons.

[…]

Concretely, in the superposition hypothesis, features are represented as almost-orthogonal directions in the vector space of neuron outputs. Since the features are only almost-orthogonal, one feature activating looks like other features slightly activating. Tolerating this "noise" or "interference" comes at a cost. But for neural networks with highly sparse features, this cost may be outweighed by the benefit of being able to represent more features! (Crucially, sparsity greatly reduces the costs since sparse features are rarely active to interfere with each other, and non-linear activation functions create opportunities to filter out small amounts of noise.)

section "The Superposition Hypothesis"
N. Elhage, T. Hume, C. Olsson and others, "Toy Models of Superposition" (Transformer Circuits Thread 2022, arXiv 2209.10652), section "The Superposition Hypothesis".

If so, the neuron is the wrong unit for a reason that no inspection of neurons repairs: features are directions, there are more of them than neurons, and a neuron is active whenever a feature with a component along its axis is. The paper made this exact on a model small enough to solve.

Definition 10.1.1 (the toy model). There are \(n\) features and \(d < n\) hidden dimensions. An input \(x \in \mathbb R^n\) has independent coordinates, each zero with probability \(1 - p\) and uniform on \([0, 1]\) with probability \(p\), the density of a feature (the paper writes the sparsity \(S = 1 - p\)). Feature \(i\) has an importance \(I_i > 0\). The model is a matrix \(W \in \mathbb R^{d \times n}\) with columns \(W_1, \dots, W_n\) and a bias \(b \in \mathbb R^n\),

\[h = Wx, \qquad \hat x = \mathrm{ReLU}\bigl(W^{\top} W x + b\bigr), \qquad L(W, b) = \mathbb E_x \sum_{i=1}^{n} I_i\, (x_i - \hat x_i)^2, \tag{10.1}\]

with \(\mathrm{ReLU}(u) = \max(u, 0)\), the rectified linear unit, on each coordinate. The linear model is (10.1) without the ReLU.N. Elhage, T. Hume, C. Olsson and others, "Toy models of superposition", Transformer Circuits Thread, 2022 (arXiv 2209.10652).

The matrix is used twice: \(Wx = \sum_j x_j W_j\) writes each feature along its column, and \(W^{\top}\) reads it back,

\[\hat x_i = \mathrm{ReLU}\Bigl(x_i\, \lVert W_i \rVert^2 + \sum_{j \ne i} x_j\, W_i \cdot W_j + b_i\Bigr), \tag{10.2}\]

the feature itself plus interference from every column that is not orthogonal to \(W_i\). An earlier essay on this site has a live figure of the same phenomenon, with five features in two dimensions.

Proposition 10.1.2 (a linear model drops features). Let the features be independent with common variance \(\sigma^2\), and \(I_1 \ge \dots \ge I_n\). For the linear model, at the best bias,

\[\frac{L}{\sigma^2} = \sum_i I_i\, \bigl(1 - \lVert W_i \rVert^2\bigr)^2 + \sum_i I_i \sum_{j \ne i} (W_i \cdot W_j)^2 \;\ge\; \sum_{i > d} I_i, \tag{10.3}\]

with equality when \(W^{\top} W\) is the projection onto the first \(d\) coordinates: the \(d\) most important features get orthogonal directions and the rest get none, at every density.

Proof. With \(P = W^{\top} W\) and the bias that cancels the mean, \(x - \hat x = (I - P)(x - \mathbb E x)\), so \(L = \sigma^2 \sum_{i,j} I_i\, (\delta_{ij} - P_{ij})^2 = \sigma^2\, \lVert D^{1/2} - D^{1/2} P \rVert_F^2\) with \(D = \mathrm{diag}(I_1, \dots, I_n)\). As \(D^{1/2} P\) has rank at most \(d\), the Eckart–Young theorem bounds its distance from \(D^{1/2}\) below by the squared singular values of \(D^{1/2}\) after the \(d\) largest, with equality at the truncation. ∎

The first sum of (10.3) rewards representing a feature, and the second charges for every pair of columns that are not orthogonal. In a linear model the charge buys nothing: this is principal component analysis. The ReLU changes the charge. If only feature \(i\) is active, with value \(t\), the output at \(j \ne i\) is \(\mathrm{ReLU}(t\, W_i \cdot W_j + b_j)\), which is zero whenever the inner product and the bias are not positive. Interference of negative sign is free, and the smallest case shows what that is worth.

Proposition 10.1.3 (drop or superpose). Let \(n = 2\), \(d = 1\), \(I_1 = 1\) and \(I_2 = r \le 1\), and compare two settings of (10.1). Dropping: \(W = (1, 0)\) and \(b = (0, p/2)\), so that feature 1 has the dimension and feature 2 is answered by its mean. Pairing: \(W = (1, -1)\) and \(b = 0\), the two features on one axis with opposite signs. Then

\[L_{\text{drop}} = r\, \Bigl(\frac{p}{3} - \frac{p^2}{4}\Bigr), \qquad L_{\text{pair}} = (1 + r)\, \frac{p^2}{6}, \qquad L_{\text{pair}} < L_{\text{drop}} \iff p < \frac{4r}{2 + 5r}. \tag{10.4}\]

Proof. Dropping gives \(\hat x_1 = x_1\) and \(\hat x_2 = \mathbb E x_2\), so the loss is \(r \operatorname{Var}(x_2)\), with \(\mathbb E x_2 = p/2\) and \(\mathbb E x_2^2 = p/3\). Pairing gives \(\hat x_1 = \mathrm{ReLU}(x_1 - x_2)\) and \(\hat x_2 = \mathrm{ReLU}(x_2 - x_1)\). When at most one feature is active both outputs are exact: the active one passes, and the other receives a negative number, which the ReLU removes. When both are active each output falls short by \(\min(x_1, x_2)\), whose square has mean \(\int_0^1 2(1 - t)\, t^2\, dt = 1/6\), and this happens with probability \(p^2\). The inequality rearranges to \(p\,(2 + 5r) < 4r\). ∎

Two equally important features share the dimension once each is present in fewer than \(4/7\) of inputs; a feature a tenth as important is taken in below a density of \(0.16\). The reason is in the powers of \(p\): a dropped feature costs in proportion to how often it occurs, a superposed one in proportion to how often it collides with another, $$p^2$$.Rare features can share space cheaply because most inputs ask for only one of them. The collision cost depends on joint occurrence, so correlations between features matter as well as how sparse each is. That is what sparsity buys. Other settings exist (a pair can be lopsided), so (10.4) is a sufficient condition for dropping to be beaten: in our trained toy model, at a sparsity of 0.4, a density just above \(4/7\), a fourth feature is taken in lopsidedly, and at 0.5 the opposite pairs form. Nor is the change gradual:

As expected, sparsity is necessary for superposition to occur, but we can see that it interacts in an interesting way with relative feature importance. But most interestingly, there appears to be a real phase change, observed in both the empirical and theoretical diagrams! The optimal weight configuration discontinuously changes in magnitude and superposition. (In the theoretical model, we can analytically confirm that there's a first-order phase change: there's crossover between the functions, causing a discontinuity in the derivative of the optimal loss.)

—N. Elhage and others, "Toy Models of Superposition" (2022). Section "Superposition as a phase change".

With more features the pairs become polytopes. Exactness on every input with a single active feature forces \(W_i \cdot W_j \le 0\) for all pairs, and at most \(2d\) nonzero vectors of \(\mathbb R^d\) have pairwise non-positive inner products (project the others onto the complement of one of them and induct on \(d\)). So up to two features a dimension, in opposite pairs, the network pays only for collisions, and the paper's trained models sit on that ceiling over a range of sparsities. Beyond it the columns form other figures, which one number tells apart.

Definition 10.1.4 (feature dimensionality). The dimensionality of feature \(i\) is \(D_i = \lVert W_i \rVert^2 / \sum_j (\hat W_i \cdot W_j)^2\) with \(\hat W_i = W_i / \lVert W_i \rVert\), and zero if \(W_i = 0\): the share of a dimension that the feature has to itself.

Proposition 10.1.5 (the dimensions add up). For every \(W\), \(\sum_i D_i \le \operatorname{rank} W \le d\). If \(k\) unit columns are orthogonal to the others and share a subspace of dimension \(d_0\) evenly, in that \(\sum_j W_j W_j^{\top}\) over them is \(k/d_0\) times the projection onto it, then each has \(D_i = d_0/k\).

Proof. Let \(F = W W^{\top}\), so that \(\sum_j (W_i \cdot W_j)^2 = W_i^{\top} F W_i\) and \(D_i = \lVert W_i \rVert^4 / (W_i^{\top} F W_i)\). As \(W_i\) is in the range of \(F\), the Cauchy–Schwarz inequality for \(F^{1/2} W_i\) and \((F^{+})^{1/2} W_i\), with \(F^{+}\) the pseudo-inverse of \(F\), gives \(D_i \le W_i^{\top} F^{+} W_i\), and \(\sum_i W_i^{\top} F^{+} W_i = \operatorname{tr}(F^{+} F) = \operatorname{rank} F\). In the second case \(W_i^{\top} F W_i = k/d_0\). ∎

The shares are \(1/2\) for an opposite pair, \(2/3\) for a triangle, \(2/5\) for a pentagon, \(3/4\) for a tetrahedron and \(3/8\) for a square antiprism of the right height. These five values are where the dimensionalities of the paper's trained features cluster."Toy models of superposition", section "Uniform superposition", for \(n = 400\) features of equal importance in \(d = 30\) dimensions. Our trained network gives a dimensionality of 0.37 for each corner of its square antiprism, 2.97 in all. Four features never ended as a tetrahedron there, always as one opposite pair and two directions at right angles, which collide in one pair where a tetrahedron collides in six; with equal importances seven features made a pentagonal bipyramid or a capped octahedron, and 12 an icosahedron.

How much room does a real width have? Section 2 proved, by drawing directions at random, that \(\mathbb R^d\) holds at least \(e^{d\varepsilon^2/4}\) unit vectors with pairwise cosines of at most \(\varepsilon\) in absolute value. The exponential is real and its constant is small: at \(d = 768\), the width of gpt-pp, a tolerance of \(\varepsilon = 0.1\) guarantees six directions, \(\varepsilon = 0.2\) about 2,200, \(\varepsilon = 0.3\) about \(3 \times 10^7\). What a width holds depends on the interference a feature can stand, and that on how many features are active at once. This count has a classical theory, compressed sensing: the recovery of a vector with few nonzero coordinates from fewer linear measurements than it has coordinates.

Theorem 10.1.6 (sparse vectors survive a random projection). Let \(A\) be a \(d \times n\) matrix of independent Gaussian entries of mean zero and variance \(1/d\). There are absolute constants \(C\) and \(c\) such that if

\[d \ge C\, s \log\frac{e n}{s}, \tag{10.5}\]

then with probability at least \(1 - 2e^{-cd}\), (i) \(0.9\, \lVert v \rVert \le \lVert A v \rVert \le 1.1\, \lVert v \rVert\) for every \(v\) with at most \(3s\) nonzero coordinates, the restricted isometry property; and (ii) every \(x\) with at most \(s\) nonzero coordinates is the only minimizer of \(\lVert z \rVert_1\) among the solutions of \(Az = Ax\).The property and its use are from E. Candès and T. Tao, "Decoding by linear programming", IEEE Transactions on Information Theory 51, 2005 (arXiv math/0502327). This form is Theorems 9.5.6 and 9.5.7 of R. Vershynin, High-dimensional probability, second edition, Cambridge University Press, 2026.

Property (i) says that any \(3s\) columns are nearly orthonormal: almost-orthogonality for groups and not only for pairs. A fixed set of \(3s\) columns is a tall Gaussian matrix with singular values near one, and a union bound over the \(\binom{n}{3s} \le (en/3s)^{3s}\) sets costs a multiple of \(s \log(en/s)\) in the exponent, the whole price of not knowing which coordinates are nonzero. Read \(n\) as the number of features, \(s\) as the number active on a token and \(d\) as the width: the width must grow with the features in use at once, and only logarithmically with those that exist. The toy-model paper turns the classical lower bounds around: a readout \(\mathrm{ReLU}(W_2 W_1 x - b)\) that recovers every vector with \(s\) nonzero coordinates through a matrix with the restricted isometry property needs \(d = \Omega(s \log(n/s))\), which for \(s\) of the order of \(pn\) allows at most of the order of \(1/(p \log(1/p))\) features a dimension.

Remark 10.1.7 (what the theorem licenses). The recovery in (ii) is a convex program; a network reads a feature with one linear map and one ReLU, which is weaker, and its matrix is learned, not random. What the theorem licenses is the next step: if activations are sparse combinations of more directions than dimensions, the directions can in principle be recovered from the activations alone.

Towards monosemanticity

What is the reader of a network to do, if its features are in superposition? The toy-model paper listed the options:

At a very high level, there seem to be three potential approaches to resolving superposition:

Create models without superposition.

Find an overcomplete basis that describes how features are represented in models with superposition.

Hybrid approaches in which one changes models, not resolving superposition, but making it easier for a second stage of analysis to find an overcomplete basis that describes it.

—"Toy Models of Superposition" (2022). Section "The strategic picture of superposition", under "Three ways out".

The first was tried first. The paper this section is about, published by the same group in October 2023, reports the attempt:

In particular, we made several attempts to induce activation sparsity during training to produce models without superposition, even to the point of training models with 1-hot activations. This indeed eliminates superposition, but it fails to result in cleanly-interpretable neurons! Specifically, we found that individual neurons can be polysemantic even in the absence of superposition. This is because in many cases models achieve lower loss by representing multiple features ambiguously (in a polysemantic neuron) than by representing a single feature unambiguously and ignoring the others.

—T. Bricken and others, "Towards Monosemanticity: Decomposing Language Models With Dictionary Learning" (2023). Transformer Circuits Thread, 4 October 2023, section "Why not use architectural approaches?".

The paper's reason, in its smallest case: a layer passes on one neuron, on or off; four features are equally likely, exactly one is present, and each fixes its own next token. A neuron that fires on one feature leaves an expected loss of \(\tfrac34 \ln 3 \approx 0.82\) nats, one that fires on two leaves \(\ln 2 \approx 0.69\). Cross-entropy pays a neuron for the bits it carries, and a neuron carries most when it fires on half of the inputs, whatever they are.

So the paper took the second way: leave the model alone and learn the basis afterwards, under a squared-error objective that does not pay a unit in this way. The problem is old. A basis larger than the dimension, in which each datum has few nonzero coefficients, is a sparse code, and learning one was proposed for images in 1996:

THE receptive fields of simple cells in mammalian primary visual cortex can be characterized as being spatially localized, oriented1–4 and bandpass (selective to structure at different spatial scales), comparable to the basis functions of wavelet transforms5,6. One approach to understanding such response properties of visual neurons has been to consider their relationship to the statistical structure of natural images in terms of efficient coding7–12. Along these lines, a number of studies have attempted to train unsupervised learning algorithms on natural images in the hope of developing receptive fields with similar properties13–18, but none has succeeded in producing a full set that spans the image space and contains all three of the above properties. Here we investigate the proposal8,12 that a coding strategy that maximizes sparseness is sufficient to account for these properties. We show that a learning algorithm that attempts to find sparse linear codes for natural scenes will develop a complete family of localized, oriented, bandpass receptive fields, similar to those found in the primary visual cortex. The resulting sparse image code provides a more efficient representation for later stages of processing because it possesses a higher degree of statistical independence among its outputs.

—B. A. Olshausen and D. J. Field, "Emergence of simple-cell receptive field properties by learning a sparse code for natural images" (1996). Nature 381, 607–609, abstract; the raised numbers are its references.

Definition 10.2.1 (a dictionary). A dictionary for activations \(x \in \mathbb R^d\) is a set of \(n \ge d\) unit vectors \(u_1, \dots, u_n\), the columns of a matrix \(W_{\text{dec}}\), and a bias \(b_{\text{dec}}\), such that every \(x\) is close to \(b_{\text{dec}} + \sum_i f_i(x)\, u_i\) with coefficients \(f_i(x) \ge 0\) of which few are nonzero. The \(u_i\) are its latents, or features; \(f_i(x)\) is the activation of latent \(i\) on \(x\).

Finding the sparsest coefficients exactly is intractable. A sparse autoencoder replaces the search by an encoder trained to output them in one step.First on synthetic data with known features: L. Sharkey, D. Braun and B. Millidge, "Taking features out of superposition with sparse autoencoders", AI Alignment Forum, December 2022. Then on small language models: H. Cunningham and others, "Sparse autoencoders find highly interpretable features in language models", ICLR 2024 (arXiv 2309.08600). The paper states its experiment in two sentences.

Towards Monosemanticity: Decomposing Language Models With Dictionary LearningTransformer Circuits Thread 2023T. Bricken, A. Templeton, J. Batson and others

The goal of this paper is to provide a detailed demonstration of a sparse autoencoder compellingly succeeding at the goals of extracting interpretable features from superposition and enabling basic circuit analysis. Concretely, we take a one-layer transformer with a 512-neuron MLP layer, and decompose the MLP activations into relatively interpretable features by training sparse autoencoders on MLP activations from 8 billion data points, with expansion factors ranging from 1× (512 features) to 256× (131,072 features). We focus our detailed interpretability analyses on the 4,096 features learned in one run we call A/1.

the introduction
T. Bricken, A. Templeton, J. Batson and others, "Towards Monosemanticity: Decomposing Language Models With Dictionary Learning" (Transformer Circuits Thread 2023), the introduction.

Definition 10.2.2 (the sparse autoencoder of 2023). With an encoder \(W_{\text{enc}} \in \mathbb R^{n \times d}\), a decoder \(W_{\text{dec}} \in \mathbb R^{d \times n}\) and biases \(b_{\text{enc}} \in \mathbb R^n\), \(b_{\text{dec}} \in \mathbb R^d\),

\[f(x) = \mathrm{ReLU}\bigl(W_{\text{enc}}(x - b_{\text{dec}}) + b_{\text{enc}}\bigr), \qquad \hat x = W_{\text{dec}}\, f(x) + b_{\text{dec}}, \tag{10.6}\] \[\mathcal L = \mathbb E_x \bigl[\, \lVert x - \hat x \rVert_2^2 + \lambda\, \lVert f(x) \rVert_1 \bigr], \qquad \lVert u_i \rVert_2 = 1 \ \text{ for every column } u_i \text{ of } W_{\text{dec}} . \tag{10.7}\]

The first term is the squared error of the reconstruction; the second, the sum of the activations, is the \(L_1\) penalty that makes them sparse.

The details are part of the result. The transformer has a residual stream 128 wide and was trained on 100 billion tokens of the Pile. Activations were collected at 250 tokens sampled from each of 40 million contexts, and an autoencoder saw about 8 billion of them in one million steps of Adam at a batch of 8,192, with \(\lambda\) chosen so that fewer than 10 or 20 latents are active on a token. The bias \(b_{\text{dec}}\) starts at the geometric median of the data, and the encoder is not tied to the decoder. The decoder's columns are renormalized after every step, the gradient first made orthogonal to them; without the unit norms the penalty would be empty, since dividing \(f_i\) by \(c\) and multiplying \(u_i\) by \(c\) leaves \(\hat x\) unchanged and divides the penalty by \(c\). A latent whose pre-activation is negative on every input receives no gradient and stays dead. The remedy is stated as a procedure, with the exactness of the whole recipe:

Towards Monosemanticity: Decomposing Language Models With Dictionary LearningTransformer Circuits Thread 2023T. Bricken, A. Templeton, J. Batson and others

Neuron Resampling

Better resampling strategies is an active research area of ours. The approach used in this work is as follows:

At training steps 25,000, 50,000, 75,000 and 100,000, identify which neurons have not fired in any of the previous 12,500 training steps.

Compute the loss for the current model on a random subset of 819,200 inputs.

Assign each input vector a probability of being picked that is proportional to the square of the autoencoder’s loss on that input.

For each dead neuron sample an input according to these probabilities. Renormalize the input vector to have unit L2 norm and set this to be the dictionary vector for the dead autoencoder neuron.

For the corresponding encoder vector, renormalize the input vector to equal the average norm of the encoder weights for alive neurons × 0.2. Set the corresponding encoder bias element to zero.

Reset the Adam optimizer parameters for every modified weight and bias term.

the appendix on neuron resampling
T. Bricken, A. Templeton, J. Batson and others, "Towards Monosemanticity: Decomposing Language Models With Dictionary Learning" (Transformer Circuits Thread 2023), the appendix on neuron resampling. A dead latent is given a new direction: an input that the dictionary reconstructs badly.

And the encoder is one linear layer and a ReLU on purpose:

Despite its difficulty, there are a host of sophisticated methods for dictionary learning (e.g. […]). These methods typically involve optimizing a relaxed problem or doing a greedy search. We tried several of these classic methods, but ultimately decided to focus on a simpler sparse autoencoder approximation of dictionary learning (similar to Sharkey, et al.). This was for two reasons. First, a sparse autoencoder can readily scale to very large datasets, which we believe is necessary to characterize the features present in a model trained on a large and diverse corpus. Secondly, we have a concern that iterative dictionary learning methods might be "too strong", in the sense of being able to recover features from the activations which the model itself cannot access. Exact compressed sensing is NP-hard, which the neural network certainly isn't doing. By contrast, a sparse autoencoder is very similar in architecture to the MLP layers in language models, and so should be similarly powerful in its ability to recover features from superposition.

—"Towards Monosemanticity". Section "Using sparse autoencoders to find good decompositions"; the mark […] stands for references.

This is the caution of Remark 10.1.7 made into a rule of design.

Example 10.2.3 (what monosemantic was measured to mean). The paper does not define the word. It measures it, on the run A/1, in four ways. First, specificity along the whole range of a latent's activation: its examples are sampled from 11 equal intervals between zero and its largest activation, because a unit that looks clean on its strongest examples can be mixed on its weak ones. For the latent labeled Arabic script, Arabic is 0.13% of the tokens of the data, 81% of those on which the latent is active, and 98% of those on which its activation exceeds 5. Second, the output side: an automated test of each unit's explanation against its logit weights scores 74% for latents and 58% for neurons, with chance at 50%. Third, by hand: 412 activation intervals of 162 latents and neurons were scored. Fourth, coverage: with the MLP's activations replaced by their reconstruction the model keeps 79% of the reduction in loss that the MLP provides, and 94.5% with the largest dictionary, against a baseline, an MLP set to zero, that the paper calls weak.

What is measured is mostly one direction of an equivalence: when the latent is strongly active, the text is of the named kind. That such text always activates the latent is a separate claim,These are the two directions of a classifier’s success: when it fires, is the concept present; and when the concept is present, does it fire? Clean strongest examples establish neither complete coverage nor reliable behavior at weak activations. and Section 10.5 shows how it fails. ◁

Two further findings set the agenda of the next three years. The first is that the answer depends on the size of the dictionary:

We conjecture that there is some idealized set of features that dictionary learning would return if we provided it with an unlimited dictionary size. Often, these "true features" are clustered into sets of similar features, which the model puts in very tight superposition. Because the number of features is restricted, dictionary learning instead returns features which cover approximately the same territory as the idealized features, at the cost of being somewhat less specific.

—"Towards Monosemanticity". Section "Feature splitting".

A latent of a small dictionary splits into narrower ones in a larger, and the paper reads the split as a change of resolution on a fixed set of true features.The fixed set of true features is a 2023 conjecture, now under pressure. Paulo and Belrose (2025) find different features across seeds on the same data; Leask et al. (2025) find that learned dictionaries remain incomplete and their features can be decomposed further. An interpretable dictionary need not be a unique inventory of the model’s concepts. The second is universality: for a second transformer trained from another seed, each latent's best match in the other model's dictionary has a median activation correlation of 0.72, against 0.46 for the neurons of the two models.

Remark 10.2.4 (what was shown). An existence proof: in one small language model an overcomplete basis found by a simple learner is more interpretable than the neurons by four measurements, and partly shared between two trainings. Not shown, and not claimed: that the latents are the model's own variables, or that the method survives depth and scale. The title says towards. The two widths of that model, a stream of 128 and an MLP of 512 neurons, happen to be those of a block of gpt-pico, the smallest size of gpt-pp and the one that runs in the page.

Larger dictionaries, sharper sparsity, and steering

In May 2024 the same group reported dictionaries of 1,048,576, 4,194,304 and 33,554,432 latents on the residual stream at the middle of a production model, Claude 3 Sonnet.A. Templeton, T. Conerly, J. Marcus and others, "Scaling monosemanticity: extracting interpretable features from Claude 3 Sonnet", Transformer Circuits Thread, May 2024. On a token fewer than 300 latents are active, and the reconstruction explains at least 65% of the variance. For 82% of latents the best-correlated neuron has a correlation of at most 0.3. Splitting continues with size: one latent for San Francisco, then two, then 11. And the paper estimates what a dictionary of a given size has no feature of its own for, which, as its last sentence says, is not the same as what the dictionary cannot represent:

This finding gives us some handle on the SAE scale at which we should expect a concept-specific feature to appear – if a concept is present in the training data only once in a billion tokens, then we should expect to need a dictionary with on the order of a billion alive features in order to find a feature which uniquely represents that specific concept. Importantly, not having a feature dedicated to a particular concept does not mean that the reconstructed activations do not contain information about that concept, as the model can use multiple related features compositionally to reference a specific concept.

—A. Templeton and others, "Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet" (2024). Section "Feature completeness"; the other numbers are from "Our SAE experiments" and "Features vs. neurons". SAE abbreviates sparse autoencoder.

At this scale the method shows two defects. Of the 33,554,432 latents 65% were dead, active on none of ten million tokens. And every active latent is underestimated, for a reason one line long.

Proposition 10.3.1 (an \(L_1\) penalty shrinks what it keeps). Let \(x = a\, u\) with \(\lVert u \rVert = 1\) and \(a > 0\), reconstructed as \(\hat x = f u\) with \(f \ge 0\). The minimizer of \(\lVert x - \hat x \rVert^2 + \lambda f = (a - f)^2 + \lambda f\) is \(f = \max(a - \lambda/2,\, 0)\).

The penalty that decides whether a latent is active also taxes how active it is. The variants of 2024 separate the two decisions. A gated autoencoder gives the decision its own path of weights, which alone carries the penalty, and reaches comparable fidelity with half as many active latents, for 50% more compute. The two that followed drop the penalty on magnitudes altogether.In that order: S. Rajamanoharan and others, "Improving dictionary learning with gated sparse autoencoders", NeurIPS 2024 (arXiv 2404.16014); L. Gao and others, "Scaling and evaluating sparse autoencoders", ICLR 2025 (arXiv 2406.04093), after A. Makhzani and B. Frey, "k-sparse autoencoders", ICLR 2014 (arXiv 1312.5663); S. Rajamanoharan and others, "Jumping ahead: improving reconstruction fidelity with JumpReLU sparse autoencoders", arXiv 2407.14435, 2024.

Definition 10.3.2 (TopK and JumpReLU). A TopK autoencoder keeps the \(k\) largest activations of (10.6) and sets the others to zero,

\[f(x) = \mathrm{TopK}_k\Bigl(\mathrm{ReLU}\bigl(W_{\text{enc}}(x - b_{\text{dec}}) + b_{\text{enc}}\bigr)\Bigr), \qquad \hat x = W_{\text{dec}}\, f(x) + b_{\text{dec}}, \qquad \mathcal L = \mathbb E_x\, \lVert x - \hat x \rVert_2^2 . \tag{10.8}\]

A JumpReLU autoencoder has a threshold \(\theta_i > 0\) for each latent, \(f_i = z_i\, \mathbb 1[z_i > \theta_i]\) with \(z = W_{\text{enc}} x + b_{\text{enc}}\), and the loss \(\mathbb E_x [\lVert x - \hat x \rVert_2^2 + \lambda\, \lVert f(x) \rVert_0]\), where \(\lVert f \rVert_0\) is the number of active latents.The equation of the TopK paper has no encoder bias and no ReLU; its released code has both, and so have the dictionaries of Example 10.3.4. With them a token has at most \(k\) active latents, and no activation is negative.

In the code. The dictionaries of Example 10.3.4 are objects of one class, and (10.8) is four of its methods: pre is the argument of the ReLU, encode is \(f(x)\), decode is \(\hat x\), and forward returns both. They belong to SparseAutoencoder in gpt-pp's sae.py.

# ---- raw activations <-> the normalized space the autoencoder works in

def normalize(self, x: Tensor) -> Tensor:
    return (x - self.mean) * self.scale

def restore(self, x: Tensor) -> Tensor:
    return x / self.scale + self.mean

# ---- the autoencoder, on normalized activations

def pre(self, x: Tensor) -> Tensor:
    """Pre-activations (B, n): how much of each direction the encoder reads in x."""
    return (x - self.bias) @ self.encoder.T + self.enc_bias

def encode(self, x: Tensor) -> Tensor:
    """The code z (B, n). topk: the k largest positive activations of each token, zeros elsewhere."""
    acts = F.relu(self.pre(x))
    if self.cfg.kind == "relu":
        return acts
    top = acts.topk(self.cfg.k, dim=-1)
    return torch.zeros_like(acts).scatter_(-1, top.indices, top.values)

def decode(self, z: Tensor) -> Tensor:
    return z @ self.dictionary + self.bias

def forward(self, x: Tensor) -> tuple[Tensor, Tensor]:
    """Normalized activations (B, d) -> their reconstruction and their code."""
    z = self.encode(x)
    return self.decode(z), z

encoder is \(W_{\text{enc}}\), of shape \(n \times d\). dictionary is \(W_{\text{dec}}\) transposed, so its rows are the directions \(u_i\) and decode multiplies from the left; bias is \(b_{\text{dec}}\) and enc_bias is \(b_{\text{enc}}\). A batch x is \(B \times d\), one token a row, and the code z is \(B \times n\). The ReLU stands before topk, so a token with fewer than \(k\) positive pre-activations keeps fewer than \(k\) latents; the ReLU autoencoders of the comparison are the same class and return before the topk.

Quirk. The x of these methods is not the stream but normalize of it: the mean is subtracted and one constant brings the mean norm to \(\sqrt d\), both measured on the first 32 batches of training, and bias starts at the geometric median of 32,768 of those vectors. A dictionary reads the stream at every token except the mark that starts a document and the first 16 tokens of a window, and at those tokens a spliced model keeps its own stream. Activations, thresholds and the values a latent is clamped at are in the normalized units; restore leads back.

In both autoencoders of Definition 10.3.2, the loss puts no price on the size of an active value: the active set is chosen by rank or by a threshold, and the gradient on an active value is that of the reconstruction alone. The count \(\lVert f \rVert_0\) has zero derivative in \(\theta\) almost everywhere, but its expectation does not: \(\partial_\theta\, \mathbb P(z > \theta)\) is minus the density of \(z\) at the threshold, and the straight-through estimator that trains a JumpReLU, a stand-in for a gradient that is zero, is a kernel estimate of that density. We test the three rules on made-up features whose true activations are known.

setup    16 features a token, a quarter active; true a (mean 1), read z = a + 0.15 × noise
encoder  L1: f = max(z − λ/2, 0)   JumpReLU: f = z if z > θ   TopK: f = z for the k largest z

encoder    setting    kept a token  error    taken from each kept read
ReLU + L1  λ = 0.60   3.8           6.3 %    0.30
JumpReLU   θ = 0.30   3.8           2.4 %    0.00
TopK       k = 3      3.0           10 %     0.00

true a against recovered f: the L1 cloud lies λ/2 = 0.30 under the diagonal f = a
(the live figure needs JavaScript; it computes in the page and loads no data)
Three ways to make a dictionary sparse, on 16 made-up features a token, a quarter of them active, each read through noise. Left: one token, with the true activation (outline), the read (tick) and what the encoder returns. Green: the recovered activations, there and in the cloud of true against recovered, where the diagonal is a perfect return. Right: the error against the number of features kept, as each knob turns. An L1 penalty takes λ/2 from every activation it keeps; a threshold and TopK return what they read. Computed in the page from 400 tokens, one draw.

Under the penalty the recovered activations fall \(\lambda/2\) below their true values, as Proposition 10.3.1 says; a threshold or TopK avoids this shrinkage. In this example, the penalty keeps 3.8 of 16 features a token and returns each 0.30 short, an error of 6.3% of the signal, where a threshold keeping the same features leaves 2.4%. TopK returns what it reads but keeps the same number on every token: at \(k = 3\) it leaves 10% where a penalty keeping as many leaves 14%, and near \(k = 4\), the mean number active, the two meet. Nor does the toy always favor the newer rules: with the noise doubled the penalty beats the threshold at equal sparsity, 11% against 13%, since taking \(\lambda/2\) from every read also damps the noise that passes. This toy isolates a mechanism; real dictionaries are measured in Example 10.3.4.

Against dead latents the TopK paper starts the encoder as the transpose of the decoder and adds an auxiliary loss, in which the most active dead latents reconstruct the error that the live ones leave: without them up to 90% of a large dictionary died, and with them its dictionary of 16 million latents on GPT-4 had 7% dead.

In the code. gpt-pp takes both remedies: the encoder starts as a copy of the decoder's rows, and the loss carries the auxiliary term at a weight of 1/32. The argument dead marks the latents that count as dead, and k_aux is half the width of the stream. The loss is topk_loss in sae.py.

def topk_loss(sae: SparseAutoencoder, x: Tensor, dead: Optional[Tensor], k_aux: int) -> tuple[Tensor, Tensor, Tensor]:
    ...
    pre = sae.pre(x)
    top = F.relu(pre).topk(sae.cfg.k, dim=-1, sorted=False)
    error = x - decode_sparse(sae, top.values, top.indices)
    fvu = error.pow(2).sum(-1).mean() / sae.variance
    fired = torch.zeros(sae.cfg.n, device=x.device).index_add_(0, top.indices.flatten(), (top.values > 0).float().flatten()) > 0
    loss = fvu
    if dead is not None:
        aux = pre.masked_fill(~dead, -torch.inf).topk(min(k_aux, sae.cfg.n), dim=-1, sorted=False)
        values = torch.where(torch.isfinite(aux.values), aux.values, 0.0)  # fewer dead latents than k_aux: the rest add nothing
        target = error.detach()
        left = target - F.embedding_bag(aux.indices, sae.dictionary, per_sample_weights=values, mode="sum")
        spread = (target - target.mean(0)).pow(2).sum(-1).mean()
        loss = loss + AUX_WEIGHT * (left.pow(2).sum(-1).mean() / spread).nan_to_num(0.0)
    return loss, fvu.detach(), fired

error is \(x - \hat x\), and fvu is the loss of (10.8) divided by sae.variance, the mean squared norm of the normalized activations about their mean: the error as a fraction of the variance. The auxiliary term asks the k_aux dead latents with the largest pre-activations to reconstruct error.detach(), and the detachment keeps it from moving the live latents. fired says which latents were among a token's \(k\) with a positive value.

Known issue. The auxiliary term is not the same in the two files that have one. Here the dead latents' pre-activations go to the decoder as they are, negative ones included (aux.values, with no ReLU); the transcoders of Definition 10.4.1 pass them through a ReLU first. Both divide the term by the spread of the error within the batch (spread), not by the variance that divides the main term, and a test covers the first form only. What the difference does to a dictionary was not measured.

PyTorch. The sums of rows are F.embedding_bag(indices, table, per_sample_weights=values, mode="sum"), here and inside decode_sparse (PyTorch 2.9.1). The call adds each token's weighted rows of the dictionary without building the \(B \times n\) code, gives the numbers of decode on the scattered code, and carries gradients both to the rows and to the weights.

The step around that loss keeps every row of the dictionary a unit vector, as Definition 10.2.1 asks, and counts for each latent the tokens since it last fired. It is Trainer.step in the same file.

def step(self, x: Tensor) -> Tensor:
    """One step on a batch of normalized activations (B, d); returns the fraction of variance unexplained."""
    sae, topk = self.sae, self.sae.cfg.kind == "topk"
    if topk:
        dead = self.since_fired > self.dead_after if self.any_dead else None
        loss, fvu, fired = topk_loss(sae, x, dead, self.k_aux)
    ...
    self.opt.zero_grad(set_to_none=True)
    loss.backward()
    if topk:
        # A unit row can only turn: remove the part of its gradient that would lengthen it (Bricken et al.).
        with torch.no_grad():
            g, rows = sae.dictionary.grad, sae.dictionary
            g -= (g * rows).sum(-1, keepdim=True) * rows
    for group in self.opt.param_groups:
        group["lr"] = self.lr * self.lr_factor()
    self.opt.step()
    if topk:
        with torch.no_grad():
            sae.dictionary /= sae.dictionary.norm(dim=-1, keepdim=True)
    self.since_fired += x.size(0)
    self.since_fired.masked_fill_(fired, 0)
    self.step_no += 1
    if self.step_no % 64 == 0:
        self.any_dead = bool((self.since_fired > self.dead_after).any())
    return fvu

The second and third if topk: are the recipe of 2023 described above: the gradient of each row loses its component along the row, and after the step each row is divided by its norm. since_fired holds for each latent the number of vectors seen since it was last active; past dead_after, 1,000,000 of them, the latent is dead for the auxiliary term, and whether any latent is dead is looked up once in 64 steps. The lines left out are the branch of the ReLU autoencoder, which has another loss.

Quirk. self.lr is not the rate the file's docstring gives. The docstring and the default of Trainer are the rule of the TopK paper, \(2 \times 10^{-4} / \sqrt{n / 2^{14}}\); the command line multiplies it by four unless told otherwise (--lr-scale 4.0), and the dictionaries of Example 10.3.4 were trained at that, 0.0008 for \(n\) = 16,384, in batches of 2,048 on 33,554,432 vectors, a third of the command line's default of \(10^8\). A trial at eight times the rule left dense and dead latents, and at four times the error of gpt-pp's dictionary was still falling when its training ended.

Definition 10.3.3 (the measures of a dictionary). On held-out activations: the sparsity \(L_0 = \mathbb E\, \lVert f(x) \rVert_0\); the normalized error \(\mathbb E \lVert x - \hat x \rVert^2 / \mathbb E \lVert x - \bar x \rVert^2\) with \(\bar x\) the mean activation, which is one minus the fraction of variance explained; the density of a latent, the fraction of tokens on which it is active, a latent being dead if it is active on none of a large sample (the papers take \(10^7\) tokens); and the loss recovered,

\[\frac{H_0 - H^{\ast}}{H_0 - H}, \tag{10.9}\]

where \(H\) is the model's cross-entropy, \(H^{\ast}\) its cross-entropy when \(x\) is replaced by \(\hat x\) in the forward pass, and \(H_0\) when \(x\) is replaced by zero.

In the code. The evaluation runs each dictionary over held-out text and ends by turning its sums into the measures of the definition. clean, zero, mean_abl and spliced are summed losses of the model with its own stream, with zero, with the mean activation and with the reconstruction in the stream's place. It is the end of evaluate in sae.py.

H, H0, Hm = float(clean) / n_targets, float(zero) / n_targets, float(mean_abl) / n_targets
total = square - first_moment.pow(2).sum() / count  # sum |x - mean|^2 about the mean of these held-out activations
out = []
for j, sae in enumerate(saes):
    density = (fires[j] / count).cpu().numpy()
    Hs = float(spliced[j]) / n_targets
    gain = float(dot[j] / rec[j]) if float(rec[j]) > 0 else 1.0  # a dictionary that reconstructs nothing has no gain
    sae.threshold.fill_(float(least[j]) / count)
    logd = np.log10(np.maximum(density[density > 0], 1e-12))
    out.append({
        **asdict(sae.cfg), "name": sae.cfg.name,
        "fvu": float(err[j] / total), "l0": float(active[j]) / count, "gain": gain,
        "fvu_rescaled": float((own[j] - gain * dot[j]) / total),
        "dead": float((density == 0).mean()), "dense": float((density > DENSE).mean()),
        "threshold": float(sae.threshold),
        "loss_clean": H, "loss_spliced": Hs, "loss_zero": H0, "loss_mean": Hm,
        "loss_added": Hs - H, "loss_recovered": (H0 - Hs) / (H0 - H), "loss_recovered_mean": (Hm - Hs) / (Hm - H),
        "density_hist": np.histogram(logd, bins=28, range=(-7.0, 0.0))[0].tolist(),
        "eval_tokens": count, "splice_targets": n_targets,
    })

H, H0 and Hs are \(H\), \(H_0\) and \(H^{\ast}\), and loss_recovered is (10.9); loss_recovered_mean puts the mean activation in the place of zero. fvu is the normalized error about the mean of the held-out activations, l0 is \(L_0\), and dead and dense are the shares of latents that fire on no token read and on more than one in ten. gain is the one factor by which every reconstruction would have to be lengthened to fit best: 1 when nothing shrinks, above 1 under the penalty of Proposition 10.3.1.

Known issue. "Dead" has two meanings in this code. In training a latent is dead when it has not fired for 1,000,000 tokens, looked up once in 64 steps; in the evaluation, dead above, when it fires on none of the held-out tokens that were read: 9.8 million for gpt-pp, 5.1 million for gpt-pico and 262,144 for the transcoders of Example 10.4.5, not the \(10^7\) of the definition. Every count of dead latents quoted here for gpt-pp and gpt-pico is by the second.

Quirk. These tools do not run the model in one precision. A dictionary reads its activations and splices its reconstructions with the model under bfloat16 autocast, the 16-bit precision of training, and computes its own code in float32; the four losses above are all taken so. The transcoders of Definition 10.4.1 pick the \(k\) largest on bfloat16 pre-activations in training and on float32 ones in evaluation, and their clean loss is taken in float32 beside replaced losses under autocast. Patching, the logit lens and the circuit scores of Section 9 run in float32, and what these differences change in the numbers was not measured.

The measures flatter. Zero is a weak baseline for (10.9), and the mean activation is a stricter one. The dictionary of 16 million latents, spliced into GPT-4, leaves the loss of a model trained with about 10% of the compute. And a benchmark of more than 200 dictionaries on eight metrics found that gains in error at fixed sparsity do not reliably carry over to tasks.A. Karvonen and others, "SAEBench: a comprehensive benchmark for sparse autoencoders in language model interpretability", ICML 2025 (arXiv 2503.09532). The GPT-4 figure is quoted from Gao et al. by L. Sharkey and others, "Open problems in mechanistic interpretability", arXiv 2501.16496, 2025, Section 2.1.2.

Example 10.3.4 (dictionaries on gpt-pp and gpt-pico). The setup: TopK autoencoders on the residual stream of gpt-pp after block 8 of 12, where \(d = 768\), with \(n\) of 4,096, 8,192 and 16,384 and \(k\) of 16, 32 and 64, each trained once on 34 million activation vectors of FineWeb-EDU text; four ReLU autoencoders with an \(L_1\) penalty at \(n\) = 8,192 for comparison;These have the form of April 2024, not of (10.7): the penalty is \(\lambda \sum_i f_i \lVert u_i \rVert\) with the decoder's columns left free, and no latent is resampled (T. Conerly and others, "Update on how we train SAEs", Circuits Updates, April 2024; SAE abbreviates sparse autoencoder). Every dictionary works in the normalized units, and reads the tokens, named in the note after Definition 10.3.2. and the same two kinds on the stream of gpt-pico after block 3 of six, 27 dictionaries there (with the 13 of gpt-pp and a second seed of one of gpt-pico's, 41 in all). The figures read the dictionary of gpt-pp with \(n\) = 16,384 and \(k\) = 32, and that of gpt-pico with \(n\) = 4,096 and \(k\) = 16.

a dictionary of 16,384 latents on gpt-pp’s stream after block 8 of 12, 32 active a token, read on
9,833,622 held-out tokens of FineWeb-EDU. One of its features:

#3578  right: at once    fires on 0.008 % of tokens    peak 17.4
most often on   right 74 %   straight 12 %   Que 1 %   -right 1 %
raises          -away  away  -roads  -lash  -bred  -hots   (its direction read out directly)
holds up        -away  away  thing  lesson  ?”↵  lessons   (logits that fall when it is removed)

the text that fires it most (firing tokens in brackets)
17.4  … of alcohol poisoning, you should seek professional medical help [right] away. Treating alcohol
17.1  processes visual images, leading to vision problems.↵See your doctor [right] away if you have or
15.7 .… people with Carpal Tunnel Syndrome do not seek treatment [right] away thinking the condition may
a firing drawn at random at under a quarter of its peak
 2.3  "N". In other segments, Kermit would play [straight] man to the wack

lists: 46 features chosen and named by a reader, a language model; 60 latents drawn at random
(the live figure needs JavaScript and loads about 0.4 MB of data)
The features of a dictionary of 16,384 latents on gpt-pp’s stream after block 8 of 12, read on 9.8 million held-out tokens of FineWeb-EDU. Green: the feature’s activation on a token. First text: what fires it most; second: firings drawn at random at four strengths. Raises, lowers: its direction read out directly; holds up, holds down: the logits that fall and rise most when it is removed where it fires. The first list was chosen and named by a reader, a language model; the second is 60 latents drawn at random, not chosen: judge the dictionary by it.

For each latent we collect its strongest held-out triggers, firings drawn at random at four levels of strength, and its density. A latent at a tenth of its peak does not always mean what it means at its peak. We also compare two readings of its effect on the output: the direct effect of Section 9, the tokens whose logits its direction \(u_i\) raises through the unembedding, and a measured one, the logits that fall when its term is taken out of the stream where it fires most and the later blocks run on. On average the two agree on three tokens of ten on gpt-pp and two of ten on gpt-pico. One sample holds 46 latents chosen and named by a reader of their examples, here a language model; another holds 60 drawn at random, and it is the one to judge the dictionary by. Of those, 48 could be given one label and 12 mix several things or are too weak to read (on gpt-pico, 40 and 20), by one reader, a language model, in one pass.

Quirk. The two readings are not taken at the same place. The direct one sends a latent's direction through the unembedding alone (through_final_norm): without the final norm's division by the length of the stream, which rescales all of a token's effects by one positive number, and without the model's soft cap, the \(\tanh\) that bounds its logits. The measured one, Dashboards.removal, runs the model in float32 and takes a difference of capped logits; the two rankings of tokens need not agree, and the three of ten and two of ten above compare them as they are. The record of every latent's activations on held-out text, from which its strongest examples are drawn, is stored in float16.

Bug, fixed. That record was once reused whenever its file was newer than the dictionary's. Evaluating a dictionary again rewrites its file unchanged, so the dates did not say which weights a record was made with. A record now carries a signature of the weights, and cached_codes reuses it only when the signature matches; no published number is known to have been affected.

the directions of three dictionaries on gpt-pp’s stream after block 8 of 12 (4,096, 8,192, 16,384 latents),
placed in three dimensions by nearness (UMAP on their cosines). A line joins a latent to its nearest latents
in the next larger dictionary, where the cosine is at least 0.5.

relatives in the next dictionary     none     one      two      three or more
4,096 to 8,192                       43       2,378    1,287    388
8,192 to 16,384                      273      4,739    2,384    796

one family: dictionary, latent, cosine with its parent, and the text each member fires on most
4,096 #692   archaeology, and other resources, in [order] to make
  8,192 #589 0.94   tools like CT scans and MRI in [order] to observe
    16,384 #3028 0.91  ?↵What Do We Know↵In [order] to consider
    16,384 #5958 0.71   from huge amounts of Internet data in [order] to learn
    16,384 #5717 0.50  Or[der] for home
  8,192 #2154 0.57  Or[der] [for] home
    16,384 #5717 0.98  Or[der] for home
(the live figure needs JavaScript; it loads about 1.4 MB of data and 0.7 MB of three.js)
Three dictionaries on gpt-pp’s stream after block 8 of 12: 4,096, 8,192 and 16,384 latents, a dot each, the 4,096 shown first. Green: the selected family. Dots are placed by UMAP on the cosine between directions: near dots are near directions, far distances mean nothing (three principal components would keep 2 % of the variance). With all dots, hairlines join a latent to its nearest in the next dictionary, at a cosine of 0.5 or more. Click a dot or a line of the family: the lines show the text each member fires on most. One training run a dictionary.

The directions \(u_i\) are points on a sphere in 768 dimensions, and two at a time they are close to random: their mean absolute cosine is 0.034, where random directions have 0.029. Their neighborhoods are not: a direction's nearest neighbor is at a median cosine of 0.33, where chance among as many directions gives about 0.14 (an estimate, not a measurement: the largest of 16,383 cosines, each close to Gaussian with variance \(1/768\)). Matching nearby directions across dictionary sizes reveals families. One latent of the smallest dictionary fires on the word order. Among 8,192 latents that latent has parted into "in order to" and the other orders, a command and an arrangement; among 16,384, "in order to" parts again, by whether it opens a sentence or stands inside one. Of the 4,096 latents of the smallest dictionary, 41% have two or more such relatives in the next. A larger dictionary mostly cuts the old features up: about two thirds of its latents have a parent in the smaller one at a cosine of 0.5 or more, and about a third (31% of the 8,192, 36% of the 16,384) have none.

gpt-pp’s stream after block 8 of 12, rewritten as k of n directions a token: the share of its variance
left unexplained, and the nats a token the model loses when the rewrite replaces the layer

latents    k = 16        k = 32        k = 64
4,096      27.8 % +0.46  22.2 % +0.29  17.9 % +0.18
8,192      25.1 % +0.38  19.0 % +0.22  14.9 % +0.13
16,384     22.8 % +0.31  16.8 % +0.17  12.6 % +0.10

ReLU with an L1 penalty, 8,192 latents (active latents: variance unexplained, and after lengthening)
14.4: 31.6 %, 29.6 %    26.9: 26.0 %, 24.8 %    47.6: 21.6 %, 20.8 %    86.0: 17.6 %, 17.1 %

used by the other figures: 32 of 16,384 active, 16.8 % unexplained, 97.9 % of the loss recovered
(3.00 nats alone, 3.17 spliced in), 0 latents dead, 15 dense
(the live figure needs JavaScript and loads about 10 kB of data)
What is lost when gpt-pp’s stream after block 8 of 12 is rewritten as a few directions out of thousands. Green: the dictionary the other figures use. Ink curves: TopK dictionaries, one a size. Hairline: ReLU dictionaries with an L1 penalty; hollow marks: the same reconstructions lengthened by their best factor, so the gap is shrinkage. Bars: latents that never fire (solid) and that fire on over a tenth of the tokens (hatched). Point at a dictionary, or use the arrow keys. Measured on 9.8 million held-out tokens (loss on 1,030,000); one run of 34 million training tokens a dictionary.

For the dictionary of 16,384 latents with at most 32 active per token, the normalized reconstruction error on 9.8 million held-out tokens is 0.168, and with it spliced in the cross-entropy, measured on the first 1.03 million of them, goes from 3.003 to 3.173 nats a token: 97.9% of the loss recovered by (10.9), 96.8% against the mean activation. Halving \(k\) multiplies the error by 1.36, halving \(n\) by 1.13. No latent is dead on those 9.8 million tokens, though 719 of the 16,384 fire on fewer than one token in 100,000 and read as noise. At 8,192 latents and equal sparsity the \(L_1\) dictionary, interpolated between two of the four, leaves 24.5% of the variance where TopK leaves 19.0%, and 23.5% once its reconstructions, 11% short, are lengthened by one factor: the shrinkage of Proposition 10.3.1 on real activations. gpt-pico's dictionary leaves 3.9% and recovers 99.1% of the loss. Each number is of one training run. ◁

A latent found by reading activations is a correlate. The 2024 paper tested it as a cause, by holding a latent at a multiple of its largest activation:

For instance, we see that clamping the Golden Gate Bridge feature 34M/31164353 to 10× its maximum activation value induces thematically-related model behavior. In this example, the model starts to self-identify as the Golden Gate Bridge! Similarly, clamping the Transit infrastructure feature 1M/3 to 5× its maximum activation value causes the model to mention a bridge when it otherwise would not. In each case, the downstream influence of the feature appears consistent with our interpretation of the feature, even though these interpretations were made based only on the contexts in which the feature activates and we are intervening in contexts in which the feature is inactive.

—"Scaling Monosemanticity". Section "Influence on behavior".

Definition 10.3.5 (clamping, and adding a direction). Let a dictionary read the stream \(x\) at one layer. To clamp latent \(i\) at the value \(c\) is to replace \(x\) by

\[x + \bigl(c - f_i(x)\bigr)\, u_i \tag{10.10}\]

at that layer, and to let the model run on: the latent's term becomes \(c\, u_i\), and the rest of the stream, with whatever the dictionary fails to reconstruct, stays. To add a direction \(r\) with coefficient \(\alpha\) is to replace \(x\) by \(x + \alpha r\).

In the code. Clamping is one function, and its next-to-last line is (10.10). It is clamp in sae.py.

@torch.no_grad()
def clamp(sae: SparseAutoencoder, x: Tensor, latent: int, value: float) -> tuple[Tensor, Tensor]:
    """Clamp one latent at `value` on raw activations x (..., d): x + (value - f_i(x)) u_i, where f_i(x) is the
    latent's activation as the dictionary reads x and u_i its direction in the units of the stream (Templeton et
    al., 2024). The latent's term in the reconstruction becomes value * u_i; everything else in x, the part the
    dictionary does not reconstruct included, is left as it is. Returns the clamped activations and f_i(x)."""
    flat = x.reshape(-1, x.size(-1)).float()
    f = sae.encode(sae.normalize(flat))[:, latent]
    out = flat + (value - f)[:, None] * sae.directions()[latent]
    return out.reshape(x.shape).to(x.dtype), f.reshape(x.shape[:-1])

value is \(c\) and f is \(f_i(x)\), both in the normalized units of the dictionary. sae.directions() is the dictionary's rows divided by the constant of normalize, so the \(u_i\) of this line is the unit row carried back into the units of the stream, where its length is no longer 1. Only the difference is added, so the stream is never replaced by its reconstruction; the clamp values of Example 10.3.6 are fractions of a latent's largest f on held-out text.

gpt-pico writes from one opening twice, on the same random draws: as trained, and with one feature of its
dictionary held at a value (in brackets: what the other story does not say).
feature: river, pond, lake, held at 0.5 times its largest activation

as trained
  Once upon a time, there was a little boy named Tim. Tim loved to play outside and make new friends.
  One day, he found a big box of glue. He thought it would be fun to make a big snowman with the glue.
  Tim picked up the glue and started to make the snowman with it. He put a big red nose, a small blue
clamped
  Once upon a time, there was a little boy named Tim. Tim loved to play [on the beach]. One day, [Tim
  saw] a big [boat near the river]. He [wanted] to [go fast on] the [river][, so he swam fast]. [Tim
  was very slow], [but he did not give up][. ][As Tim swam], [he saw] a [big rock near the river][. He

over 300 stories: “river”, “pond”, “lake”, “bridge” or “water” in 83 % with the clamp, 2 % without;
loss on windows of held-out stories 1.50 nats a token as trained, 1.95 clamped
(the live figure needs JavaScript; [write] loads about 3.4 MB: gpt-pico’s weights and the dictionary’s encoder)
gpt-pico writing with one feature of its dictionary clamped: at every token the stream after block 3 of 6 has the feature’s term replaced by a fixed multiple of its direction. First text: gpt-pico as trained; second: clamped; same opening, same random draws. Green: the tokens the other story does not have. The trace: the feature’s reading of each text before the clamp. Pick a feature, the value it is held at (multiples of its largest activation) and an opening, and press write. The sentence reports 300 sampled stories a setting and 262,144 held-out tokens; the texts are one draw.

Example 10.3.6 (gpt-pico with one latent clamped). The setup: gpt-pico and its dictionary of Example 10.3.4; latent 3183, which fires most often on the words pond, river, road and lake, clamped at half its largest activation on held-out text; 300 stories sampled from the opening "Once upon a time," with and without the clamp, on the same draws. One of the words river, pond, lake, bridge or water appears in 83% of the clamped stories and 2% of the others, and gpt-pico's loss on windows of held-out text goes from 1.50 to 1.95 nats a token: water in most stories, for about half a nat on text that has nothing to do with water. At the largest activation itself one of them is in all 300 stories and the text breaks: the loss is 3.79, and gpt-pico as trained finds those stories 2.7 nats a token unlikely, against 0.8 for its own. Nor does a label say which way a latent pushes:A direction that responds to worry can participate in suppressing it. The examples that activate a latent name its trigger; intervention tests the effect of making it stronger. Those need not point the same way. the latent that fires on words of worry (worried, scared, sad) makes them rarer when held up, from 22% of the stories as trained to 3% at a quarter of its largest activation and none at the whole of it. ◁

Remark 10.3.7 (what clamping shows). That the direction suffices: pushed far enough along \(u_i\), the later layers produce the behavior that the label names, or its opposite. Not that the model uses the latent so in ordinary operation: the 2024 demonstration held its feature at ten times the largest value it takes on data, and gpt-pico's stories break at one times. Not that the latent is necessary, since other directions may carry the behavior when it is removed. And not that the dictionary's unit is the best handle.

A direction needs no dictionary. In 13 open chat models refusal is mediated by one direction, the difference between the mean stream on harmful and on harmless instructions: erasing it prevents refusal, and adding it makes the model refuse harmless requests.A. Arditi and others, "Refusal in language models is mediated by a single direction", arXiv 2406.11717, 2024. On a benchmark of steering built in 2025, such differences of means and plain prompting did better than dictionary latents.Z. Wu and others, "AxBench: steering LLMs? Even simple baselines outperform sparse autoencoders", ICML 2025 (arXiv 2501.17148). And a 2026 note from the group that introduced clamping finds that latents with the same top examples and promoted tokens can differ in what they cause, and that their connections to later latents predict which ones steer.P. Goel, I. Kauvar and N. L. Turner, "Downstream connections predict which features will steer model behavior", Circuits Updates, May 2026.

From features to circuits

A dictionary says what is represented at one place. It does not say how one representation is computed from another, and the obstacle is the MLP: what a latent before it contributes to a latent after it depends on the MLP's derivative at that input, a matrix that changes with every token. The first circuits between dictionary latents therefore measured every edge prompt by prompt, with gradients, and kept a node for the error of each dictionary.S. Marks and others, "Sparse feature circuits: discovering and editing interpretable causal graphs in language models", ICLR 2025 (arXiv 2403.19647). The alternative changes what the dictionary is asked to do.

Definition 10.4.1 (transcoder). A transcoder for an MLP reads the MLP's input \(x\) and is trained to reconstruct its output:

\[f(x) = \mathrm{ReLU}(W_{\text{enc}}\, x + b_{\text{enc}}), \qquad \mathrm{TC}(x) = W_{\text{dec}}\, f(x) + b_{\text{dec}}, \qquad \mathcal L = \mathbb E_x \bigl[\lVert \mathrm{MLP}(x) - \mathrm{TC}(x) \rVert_2^2 + \lambda\, \lVert f(x) \rVert_1\bigr]. \tag{10.11}\]

Latent \(i\) has an encoder direction \(e_i\), the \(i\)th row of \(W_{\text{enc}}\), and a decoder direction \(u_i\). A rule of Definition 10.3.2 can take the place of the penalty.

In the code. gpt-pp has one transcoder for each MLP and trains them together, so every tensor has a first dimension over the layers. The class is Transcoders in gpt-pp's transcoder.py.

# ---- every layer at once: x and y are (L, B, d)

def pre(self, x: Tensor) -> Tensor:
    """Pre-activations (L, B, n): how much of each latent's input direction the MLP's input holds."""
    return torch.baddbmm(self.enc_bias[:, None, :], x - self.in_mean[:, None, :], self.encoder.transpose(1, 2))

def code(self, pre: Tensor) -> Tensor:
    """The k largest positive pre-activations of each token, zeros elsewhere."""
    top = F.relu(pre).topk(self.cfg.k, dim=-1, sorted=False)
    return torch.zeros_like(pre).scatter_(-1, top.indices, top.values)

def encode(self, x: Tensor) -> Tensor:
    return self.code(self.pre(x))

def normalize_out(self, y: Tensor) -> Tensor:
    return (y - self.out_mean[:, None, :]) / self.out_scale[:, None, None]

def decode_normalized(self, z: Tensor) -> Tensor:
    return torch.baddbmm(self.bias[:, None, :], z, self.dictionary)

def decode(self, z: Tensor) -> Tensor:
    return self.decode_normalized(z) * self.out_scale[:, None, None] + self.out_mean[:, None, :]

def forward(self, x: Tensor) -> tuple[Tensor, Tensor]:
    """MLP inputs (L, B, d) -> the stand-in's outputs (L, B, d) and its code (L, B, n)."""
    z = self.encode(x)
    return self.decode(z), z

pre is the argument of the ReLU of (10.11) for all layers in one batched product, with the mean of the input subtracted first, and forward is \(\mathrm{TC}(x)\). TopK takes the place of the penalty: code keeps the \(k\) largest positive pre-activations. The output is centered and divided by one constant a layer, so the rows of dictionary are unit vectors in those units, and the methods read and write fold the constants back to give the \(e_i\) and \(u_i\) of the definition in the model's own units. The \(x\) is what the MLP itself is given, the stream after the block's second normalization; mlp_io collects it with the MLP's output through PyTorch's register_forward_hook on each MLP, not through the model's own hooks of Section 9.4.

Known issue. One transcoder stands in for one MLP: a latent writes to its own layer only, where the cross-layer transcoders described below write to every later one. The mark that starts a story is trained on but left out of the normalization constants and of the reported error. The learning rate is eight times the rule of the TopK paper, a rate the docstring justifies by a trial on 10 million tokens, a quarter of the training of Example 10.4.5. In gpt-pico's transcoders 6% of the latents of block 1 and 50% to 61% of those of blocks 2 to 6 never fire on held-out text; they die in the last third of training.

Proposition 10.4.2 (an edge is an activation times a weight). Let transcoders stand in for the MLPs of blocks \(\ell < \ell^{\prime}\). Along the residual connection at one position, latent \(s\) of block \(\ell\) contributes \(f_s\, (u_s \cdot e_t)\) to the pre-activation of latent \(t\) of block \(\ell^{\prime}\): that pre-activation is \(e_t\) applied to a stream which contains the term \(f_s u_s\).

The first factor depends on the input; the second is a weight of the replacement, to be read as Section 9 read the weights of small attention-only models.J. Dunefsky, P. Chlenski and N. Nanda, "Transcoders find interpretable LLM feature circuits", NeurIPS 2024 (arXiv 2406.11944), Section 3.2.1. A cross-layer transcoder goes further: a latent reads the stream at one layer and writes, with a decoder for each, to the MLP outputs of that layer and of all later ones, so that a feature which persists across layers is one latent and not a chain of copies.

With every MLP replaced, the network is a replacement model, and it is not the model: the largest one built in 2025 for an 18-layer research model, with ten million latents, predicts the model's top next token on 50% of prompts. So one prompt is studied at a time, and the replacement is repaired on it. The paper builds this local replacement model in three steps, then says what it is and is not.

Circuit Tracing: Revealing Computational Graphs in Language ModelsTransformer Circuits Thread 2025E. Ameisen, J. Lindsey, A. Pearce and others

The Local Replacement Model

While running the replacement model can sometimes reproduce the same outputs as the underlying model, there is still a significant gap, and reconstruction errors can compound across layers. Since we are ultimately interested in understanding the underlying model, we would like to approximate it as closely as possible. To that end, when studying a fixed prompt \(p\), we construct a local replacement model, which

Substitutes the CLT for the MLP layers (as in the replacement model);

Uses the attention patterns and normalization denominators from the underlying model's forward pass on \(p\) (as in […]);

Adds an error adjustment to the CLT output at each (token position, layer) pair equal to the difference between the true MLP output on \(p\) and the CLT output on \(p\) (as in […]).

After this error adjustment and freezing of attention and normalization nonlinearities, we've effectively re-written the underlying model's computation on the prompt \(p\) in terms of different basic units; all of the error-corrected replacement model's activations and logit outputs exactly match those of the underlying model. However, this does not guarantee that the local replacement model and underlying model use the same mechanisms. We can measure differences in mechanism by measuring how differently these models respond to perturbations; we refer to the extent to which perturbation behavior matches as “mechanistic faithfulness”, discussed in § Evaluating Mechanistic Faithfulness.

the section on the local replacement model
E. Ameisen, J. Lindsey, A. Pearce and others, "Circuit Tracing: Revealing Computational Graphs in Language Models" (Transformer Circuits Thread 2025), the section on the local replacement model. CLT is the cross-layer transcoder. The two marks stand for references to earlier work.

Proposition 10.4.3 (exact on the prompt, linear between latents). In the local replacement model of a prompt every activation and every logit equals the model's. With the recorded attention patterns, normalization denominators and error vectors held constant, the stream at each layer and position is a linear function of the token embeddings, the error vectors and the activations of the latents upstream, so that the pre-activation \(h_t\) of a latent \(t\) is, up to the biases of the transcoders,

\[h_t = \sum_{s} A_{s \to t}, \qquad A_{s \to t} = f_s\; e_t^{\top}\, J_{s \to t}\, u_s . \tag{10.12}\]

Here \(s\) runs over the upstream latents, embeddings and errors (for which \(f_s u_s\) is the vector itself), and \(J_{s \to t}\) is the Jacobian of the stream where \(t\) reads with respect to the stream where \(s\) writes, with those quantities constant: an edge is a decoder, a frozen Jacobian and an encoder. The same holds for an output token, with \(e_t\) the gradient of its logit, less the mean logit, in the final stream.

Proof. At each MLP the replacement writes the transcoders' output plus the recorded difference from the true output, that is, the true output; by induction over the layers nothing differs. With its pattern constant, an attention head is a linear map of the stream: a fixed weight for each pair of positions, times the matrix \(W_{OV}\) of Section 9. With its denominator constant, a normalization is a fixed rescaling. Each MLP has become the sum \(\sum_s f_s u_s\) plus a constant. The composition is linear, its matrix between two sites is \(J\), and an encoder is a linear functional. ∎

In the code. The frozen model of the proposition is one function. It is given what enters the first block and what each MLP writes, and it holds the patterns and the denominators of a recorded forward pass, rec. It is frozen_forward in gpt-pp's attribution_graph.py.

def frozen_forward(model: GPT, rec: Record, embed: Tensor, writes: list[Tensor]) -> tuple[list[Tensor], Tensor]:
    """The model with the patterns and denominators of `rec`, given what enters the first block (B, T, d) and what
    each MLP writes (B, T, d): what each MLP reads (B, T, d) and the logits before the cap (B, T, V). Every
    operation is affine in `embed` and `writes`; with the recorded embeddings and MLP outputs it is the model."""
    x, reads = embed, []
    B, T, _ = x.shape
    for l, block in enumerate(model.blocks):
        attn = block.attn
        v = attn.v(frozen_norm(block.norm1, x, rec.denom1[l])).view(B, T, attn.n_kv_head, attn.d_head).transpose(1, 2)
        if attn.n_kv_head != attn.n_head:
            v = v.repeat_interleave(attn.n_head // attn.n_kv_head, dim=1)
        z = rec.patterns[l] @ v  # (H, T, T) @ (B, H, T, dh): each head's recorded average of the values
        x = x + attn.o(z.transpose(1, 2).reshape(B, T, -1))
        reads.append(frozen_norm(block.norm2, x, rec.denom2[l]))
        x = x + writes[l]
    return reads, model.unembed(frozen_norm(model.final_norm, x, rec.denom_final))

rec.patterns[l] holds the attention patterns of block l on the prompt, and rec.denom1, rec.denom2 and rec.denom_final the denominators: in gpt-pp and gpt-pico, whose normalization has no gain and no bias, the root mean square of the stream at each norm. writes[l] is what MLP l adds, its transcoder's output plus the error vector, and reads[l] is what it reads, to which the encoder rows \(e_t\) are applied. No attention score and no division by the stream's own length appears, so the function is linear in embed and writes (affine, in a model with biases), and an edge of (10.12) is a gradient through it times what the source wrote, one backward pass a target. The last line returns logits before the recipe's soft cap, where a logit is still linear in the final stream; an output node is such a logit less the mean over the vocabulary. rec is made through the model's hooks of Section 9.4, and only for a model that normalizes before each sublayer, has MLPs and uses a norm that divides.

Definition 10.4.4 (attribution graph and influence). The attribution graph of a prompt has a node for every token embedding, every active latent at every position, every error vector and each of the most probable next tokens (as many as reach 95% of the probability, at most ten); its edges are the \(A_{s \to t}\) of (10.12). Let \(\tilde A_{ts} = \lvert A_{s \to t} \rvert / \sum_{s^{\prime}} \lvert A_{s^{\prime} \to t} \rvert\), and let \(w\) hold the probabilities of the output tokens and zero elsewhere. The influence of the nodes on the output is

\[\iota = w^{\top} \bigl(\tilde A + \tilde A^2 + \tilde A^3 + \cdots \bigr). \tag{10.13}\]

Error nodes have no inputs: in the paper's words they "pop out of nowhere". Every edge runs from a lower layer to a higher one, so \(\tilde A\) is nilpotent and the sum in (10.13) is finite. It has a plain reading. Start a walk at an output token drawn with the weights \(w\) and step from each node to one of its inputs, to input \(s\) of node \(t\) with probability \(\tilde A_{ts}\): the entry of \(w^{\top} \tilde A^m\) at \(s\) is the probability of being at \(s\) after \(m\) steps, the graph has no cycles, and so \(\iota_s\) is the probability that the walk passes through \(s\). The walk can stop only at a node without inputs, a token embedding or an error node.

In the code. Both objects of the definition are a few lines of NumPy, normalize and influence in attribution_graph.py.

def normalize(A: np.ndarray) -> np.ndarray:
    """A~[t, s] = |A[t, s]| / sum over s' of |A[t, s']|: the share of node t's input that comes from s."""
    absolute = np.abs(A)
    total = absolute.sum(1, keepdims=True)
    return np.divide(absolute, total, out=np.zeros_like(absolute), where=total > 0)


def influence(At: np.ndarray, w: np.ndarray) -> np.ndarray:
    """iota = w (A~ + A~^2 + ...) = w ((I - A~)^-1 - I): the probability that a walk from the outputs to the inputs
    passes through each node. Every edge runs from a lower layer to a higher one, so the sum is finite."""
    n = len(w)
    return w @ (np.linalg.inv(np.eye(n) - At) - np.eye(n))

At is \(\tilde A\), with a row for each target and a column for each source, and w holds the weights of the outputs. The finite sum of (10.13) is taken in closed form, \((I - \tilde A)^{-1} - I\), which nilpotency makes the same number, and normalize leaves the row of a node without inputs at zero.

Quirk. An embedding node is what enters the first block, the token's embedding after the embedding's normalization, not a row of the table. The limit of ten outputs and 95% is in the definition, and on a pruned graph a walk that reaches a pruned edge counts with the errors, which is how the replacement score of Example 10.4.5 goes from 0.912 to 0.488.

The paper keeps the nodes that carry 80% of the influence and then the edges that carry 98%, which removes about nine nodes in ten and all but one edge in 500. What a graph rests on is measured by its replacement score, the share of the walk that ends at token embeddings and not at error nodes: 0.61 for the cross-layer transcoder of ten million latents against 0.37 for per-layer transcoders, whose paths are also longer, 3.7 steps on average against 2.3.

A graph is a statement about the local replacement model. It becomes one about the model by intervention: change a latent's value in the model's own forward pass and compare what follows with what the graph predicts. Over pairs of latents the paper reports a rank correlation of 0.72 between influence and the measured effect of ablating a latent, that is, of removing it; one layer after a perturbation the replacement and the model agree to a cosine of about 0.8, and the agreement decays with every layer crossed.

Example 10.4.5 (an attribution graph of gpt-pico). The setup: a TopK transcoder for each of gpt-pico's six MLPs, of 2,048 latents with at most 16 active on a token, trained once on 40 million tokens of TinyStories. With all six MLPs replaced, the loss on held-out stories, each read from its first token, goes from 1.413 nats a token in float32 to 1.570 under bfloat16 autocast (on windows of held-out text, the basis of Example 10.3.6, gpt-pico is at 1.50), and the top next token is gpt-pico's at 82% of positions. (The width flatters: in blocks 2 to 6 about half or more of the latents never fire on held-out text.) For the prompt "Lily was a little girl. One day," gpt-pico predicts "she" with probability 0.84. The graph has 980 nodes and 214,713 edges, and each node equals the sum of its edges and the transcoders' biases to one part in a million. Of the walks from the output, 91% end at tokens (32% at the mark that starts the text, which every position reads through attention) and 9% at error nodes: a replacement score of 0.912. Pruned at 80% and 98% the graph keeps 191 nodes and 3,378 edges, and its replacement score, with what was pruned counted among the errors, is 0.488; neither number should be set beside the paper's 0.37 and 0.61 without saying which is meant.

The likeliest walk runs from the token "girl" through a latent of block 1 that fires on "girl" and "boy" and a latent of block 3 that fires on "She", "time" and "girl" to the output "she". It reads as an account, and it fails the test. The edge from the second latent to "she" is 3.83, of a logit of 28.9, and its paths together predict a change of +1.23 on its removal (the direct edge alone would predict −3.83; the later latents, answering along their own edges, more than undo it); taken out of gpt-pico's own forward pass, the latent moves that logit by +0.06. With "boy" in place of "girl" gpt-pico writes "he", and "she" falls from 84% to 13%: the token matters, and no single latent on the path does. Over the 50 most influential latents, taken out one at a time, the rank correlation between influence and the measured change of the outputs is 0.27, and between the predicted and the measured change of the top logit 0.54. One cause is in the construction, which holds the normalizations fixed: removing one latent of the last block moves gpt-pico's final denominator by tens of percent. We fixed a rule before the test, both correlations at least 0.5, and built graphs for 12 prompts on which gpt-pico's first choice is the right word. Three pass, two of them barely (0.51 and 0.50 against the 0.5 of the rule). ◁

prompt   “Lily was a little girl. One day,”: gpt-pico predicts “ she” 84 %, “ Lily” 8 %, “ her” 6 %
graph    tokens along the bottom, gpt-pico’s 6 blocks upward, the outputs on top; a node for each active latent of the
         6 transcoders (2,048 latents each, 16 active a token), each token’s embedding, each error vector
unpruned 980 nodes, 214,713 edges; walks from the output end at tokens 91 % (32 % at the start
         mark), at error nodes 9 %; every node equals the sum of its edges to 1 × 10⁻⁶

influence kept   latents   edges    completeness   walks ending at tokens
50 %             17        236      0.36           14 %
60 %             36        506      0.50           24 %
70 %             70        1,272    0.64           36 %
80 % (paper)     130       3,378    0.75           49 %
90 %             246       9,742    0.86           65 %

likeliest walk   “ girl” → block 1 latent 175 → block 3 latent 1139 → “ she”
the test         Taken out of gpt-pico, latent 175 moves the logit of “ she” by −0.25 where the edges predict +0.61, latent 1139 by +0.06 (+1.23). By that test this prompt’s graph fails; 3 of 12 prompts pass.
                 (over 50 latents: rank correlation 0.54, predicted and measured change)
(the live figure needs JavaScript and loads about 0.8 MB of data)
An attribution graph of gpt-pico on one prompt: tokens along the bottom, six blocks above, the output on top. A dot: an active transcoder latent; a diamond: an error node, a transcoder’s miss; hairlines: edges. Green: the selected path and, in the scatter, its latents: what the edges predict of taking each out against what gpt-pico does. Hover a node for what fires it, click for its path; the slider keeps a share of the influence (80 % in Ameisen and others, “Circuit Tracing”, 2025). Attention patterns and normalizations are held as recorded: the graph shows what moved, not why.

In the code. The measured changes of Example 10.4.5 come from ablate, which subtracts a latent's write from the output of its MLP and runs the model on with its own attention, normalizations and MLPs. The predicted ones are read off the graph alone, by linear_effects in attribution_graph.py.

def linear_effects(graph: Graph) -> np.ndarray:
    """E[t, s]: the change of node t's value (a latent's pre-activation, an output's centered logit) that the graph
    predicts for one unit less of latent s, every other latent answering along its edges: with V[t, s] = A[t, s] /
    a_s the direct effect of one unit of s on t, E = -(V + V^2 + ...) restricted to paths through latents. A latent
    in between is taken to move with its pre-activation, which is right while it stays among its token's k
    largest. Columns of nodes that are not latents are zero."""
    is_latent = graph.kind == LATENT
    V = np.zeros_like(graph.A)
    V[:, is_latent] = graph.A[:, is_latent] / graph.activation[is_latent]
    through = V * is_latent[:, None]  # steps that arrive at a latent, which passes them on
    return -V @ np.linalg.inv(np.eye(graph.n) - through)

graph.A holds the \(A_{s \to t}\) of (10.12), target by source, and dividing a column by its latent's activation gives the effect of one unit of it. The inverse sums every path through later latents, which is how the removal of an edge of 3.83 comes to predict +1.23 and not −3.83.

Known issue. The prediction holds fixed what the model does not. The attention patterns and the normalization denominators stay at the prompt's values, no edge passes through an attention score, and a latent in between is taken to move with its pre-activation, which is right only while it stays among its token's \(k\) largest; ablate holds nothing fixed. The correlations of 0.27 and 0.54 in the example, and the three prompts of 12 that pass, are the size of that gap on gpt-pico.

Quirk. The logits of the example are taken before the model's soft cap and less their mean over the vocabulary, the predicted and the measured ones alike. The 28.9 is such a value: the logit of "she" is 13.0 before the cap and 10.5 after it, and the probabilities that weight the outputs come after the cap. Section 9.4 reads its patching metric before the cap as well, and its logit lens after it.

On gpt-pico, then, a graph is exact, nearly complete before pruning and easy to read, and for nine prompts of twelve it fails the test when the model itself is asked. A graph is a hypothesis about one prompt. The intervention is the experiment.Exact accounting is possible because the graph includes errors and freezes parts of the original computation. Its edges can sum perfectly on this prompt while predicting the wrong response to a change, which is why reconstruction and intervention are separate tests.

The companion paper applies the method to a production model, Claude 3.5 Haiku.J. Lindsey, W. Gurnee, E. Ameisen and others, "On the biology of a large language model", Transformer Circuits Thread, March 2025. On a question with two hops (Dallas, Texas, Austin) the graph passes through latents for the intermediate fact, and putting those for California in their place makes the model answer Sacramento. For 36 + 59 a coarse pathway and an exact one run in parallel. And in a rhymed couplet the rhyme is chosen before the line is written:

The attribution graph above, computed by attributing back from the “rabbit” output node, shows an important group of features active on the newline token, before the beginning of the second line. Features active over the “it” token activate rhyming with “eet/it/et” features, which themselves activate features for candidate completions such as “rabbit” and “habit”! The candidate completions in turn have positive edges to say “rabbit” features over the last token, eventually promoting the corresponding output token. This is consistent with the model planning potential completions ahead of time.

—J. Lindsey and others, "On the Biology of a Large Language Model" (2025). Section "Planned words features and their mechanistic role". Injecting a different planned word worked in 70% of 25 poems.

The authors also say how often the method gives an answer.

On the Biology of a Large Language ModelTransformer Circuits Thread 2025J. Lindsey, W. Gurnee, E. Ameisen and others

A note on our approach and its limitations

Like any microscope, our tools are limited in what they can see. Though it’s difficult to quantify precisely, we’ve found that our attribution graphs provide us with satisfying insight for about a quarter of the prompts we’ve tried (see § Limitations for a more detailed discussion of when our methods are likely to succeed or fail). The examples we highlight are success cases where we have managed to learn something interesting; moreover, even in our successful case studies, the discoveries we highlight here only capture a small fraction of the mechanisms of the model. Our methods study the model indirectly using a more interpretable “replacement model,” which incompletely and imperfectly captures the original. Moreover, for the sake of clear communication, we will often present highly distilled and subjectively determined simplifications of the picture uncovered by our methods, losing even more information in the process. To provide a more accurate sense of the rich complexity we have uncovered, we provide readers with an interactive interface for exploring attribution graphs. However, we stress that even these rather complex graphs are simplifications of the underlying model.

the note on the approach and its limitations
J. Lindsey, W. Gurnee, E. Ameisen and others, "On the Biology of a Large Language Model" (Transformer Circuits Thread 2025), the note on the approach and its limitations.

Three graphs of twelve on gpt-pico is the same proportion and another measurement: theirs is a judgment of insight on a production model, ours a rule fixed in advance on a small one. The same paper lists what does not work: prompts beyond about a hundred tokens, questions about why the model does not do something (an inactive latent is not in the graph), and graphs whose weight sits in error nodes. Their own summary:

Our Method as a Stepping Stone. Overall, we see our present method as a stepping stone. It has major limitations, and in particular we expect that cross-layer transcoders are not the best long-term abstraction for understanding models, or at least are very incomplete. We think it's quite likely that we'll share significantly different methods in the future. We believe its value is in establishing a starting point for us to build on, clarifying the remaining problems (§ Limitations), and enabling "biology" work in the interim while better methods are developed.

—"On the Biology of a Large Language Model". Section "What have we learned about our method?", the paragraph headed "Our method as a stepping stone".

One limit is built into the construction. The attention patterns are constants of the graph, not things it explains:

Note that these graphs do not contain information about the influence of nodes on other nodes via their influence on attention patterns, but do contain information about node-to-node influence through the outputs of frozen attention. In other words, we account for the information which flows from one token position to another, but not why the model moved that information. Note also that the outgoing edges from a cross-layer feature aggregate the effect of its decodings at all of the layers that it writes to on downstream features.

—"Circuit Tracing". Section "Constructing an attribution graph for a prompt".

The paper's own example is an induction prompt, the mechanism of Section 9: the graph shows a name copied from its earlier occurrence and not why the head attended there, which the authors say "might even be seen as a kind of regression" from earlier methods. gpt-pico does the same: for "Sara and Ben went to the park. Sara and" the likeliest walk of its graph has one latent, on the earlier Ben, with an edge straight to the output. The repair, four months later, treats a score like an edge. Before the softmax an attention score is bilinear in the streams at the two positions, \(x_q^{\top} W_Q^{\top} W_K\, x_k\), so with each stream written as a sum of latent directions the score is a sum over pairs of latents, one at the reading position and one at the position read.H. Kamath, E. Ameisen, I. Kauvar and others, "Tracing attention computation through feature interactions", Transformer Circuits Thread, 31 July 2025. In gpt-pp one more quantity has to be frozen first: its scores are cosines of rotated queries and keys, bilinear only once the norms of the query and the key are fixed, with a matrix that depends on the distance between the positions.

And a graph is about one prompt by choice: read globally, the weights between latents are dominated by the interference that superposition leaves between unrelated directions.C. Olah, N. L. Turner and others, "A toy model of interference weights", Transformer Circuits Thread, 2025.

Critiques and alternatives

Each step so far rests on an assumption that later work tested.

A latent fires whenever its concept is present. Not when concepts nest:

Absorption is caused by the SAE sparsity penalty in the presence of hierarchical features. When two features form a hierarchy, for instance “starts with S” and “short”, the SAE can merge the “starts with S” feature direction into a latent tracking “short” and then not fire the main “starts with S” latent. This means firing one latent instead of two, increasing sparsity while retaining perfect reconstruction. We demonstrate that hierarchical feature co-occurrence causes absorption in a simple toy setting.

—D. Chanin and others, "A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders" (2024). arXiv 2409.14507, Section 3.

Example 10.5.1 (absorption in two lines). Let \(u\) and \(v\) be orthonormal, and let the inputs be \(u\) (the parent property alone) and \(u + v\) (parent and child). The dictionary \(\{u, v\}\) reconstructs \(u + v\) with two latents, at an \(L_1\) penalty of 2. The dictionary \(\{u, (u + v)/\sqrt 2\}\) reconstructs it exactly with one, at \(\sqrt 2\). Every sparsity objective prefers the second, in which the latent for \(u\) is silent on inputs that have the property. ◁

So the converse left open in Example 10.2.3 fails by construction, and splitting is not a neutral change of resolution.

A large enough dictionary returns the model's features. Two dictionaries of about 131,000 latents, trained from different seeds on the same activations of an MLP of Llama 3 8B, share about 30% of their latents. The latents of a large dictionary decompose into those of a small one (the published example: Einstein into scientist, Germany and famous person), so they are not atoms. And on synthetic data with known features, dictionaries recovered 9% of the features while explaining 71% of the variance.G. Paulo and N. Belrose, "Sparse autoencoders trained on the same data learn different features", arXiv 2501.16615, 2025; P. Leask and others, "Sparse autoencoders do not find canonical units of analysis", ICLR 2025 (arXiv 2502.04878); A. Korznikov and others, "Sanity checks for sparse autoencoders: do SAEs beat random baselines?", arXiv 2602.14111, 2026.

Our own dictionaries agree. Trained again with another seed, gpt-pico's dictionary reaches the same scores to three digits and not the same latents: of its 4,096 directions, 17% have a twin in the second run at a cosine of 0.9 or more, 54% at 0.7 and 87% at 0.5 (the median best cosine is 0.72; as many random directions would give 0.31). The 44 chosen latents return more often, 34%, 73% and 95%: one more reason to judge by a random sample. The displayed latents belong to one fitted dictionary, rather than a uniquely identified set of features of gpt-pico.

What a dictionary leaves out is noise. About half of the error vector, and more than 90% of its norm, can be predicted linearly from the activation itself, and larger dictionaries fail on the same contexts as smaller ones.J. Engels, L. Riggs and M. Tegmark, "Decomposing the dark matter of sparse autoencoders", TMLR 2025 (arXiv 2410.14670).

Features are directions. Dictionaries of language models contain circles: the days of the week and the months of the year each lie on one, and the models use the circles for modular arithmetic, as the small model of Section 8 used its own for addition.J. Engels and others, "Not all language model features are one-dimensionally linear", ICLR 2025 (arXiv 2405.14860). In a production model a count of characters lies on a curved manifold that the latents tile: local coordinates, not independent directions.W. Gurnee, E. Ameisen and others, "When models manipulate manifolds: the geometry of a counting task", Transformer Circuits Thread, 21 October 2025.

The scores measure what was learned. Not by themselves:

Sparse autoencoders (SAEs) are widely used to extract sparse, interpretable latents from transformer activations. We test whether commonly used SAE quality metrics and automatic explanation pipelines can distinguish trained transformers from randomly initialized ones (e.g., where parameters are sampled i.i.d. from a Gaussian). Over a wide range of Pythia model sizes and multiple randomization schemes, we find that, in many settings, SAEs trained on randomly initialized transformers produce auto-interpretability scores and reconstruction metrics that are similar to those from trained models. These results show that high aggregate auto-interpretability scores do not, by themselves, guarantee that learned, computationally relevant features have been recovered. We therefore recommend treating common SAE metrics as useful but insufficient proxies for mechanistic interpretability and argue for routine randomized baselines and targeted measures of feature 'abstractness'.

—T. Heap, T. Lawson, L. Farnik and L. Aitchison, "Automated Interpretability Metrics Do Not Distinguish Trained and Random Transformers" (2025). arXiv 2501.17727, abstract. Pythia is a family of open language models.

With random baselines for the dictionary itself, the sanity checks of 2026 score 0.87 against 0.90 on interpretability, 0.69 against 0.72 on sparse probing and 0.73 against 0.72 on causal editing, the baseline first.

The latents are the right tool. Where the target is known, simpler tools did better: linear probes held up out of distribution better than probes on latents, and differences of means steered better. In March 2025 the mechanistic interpretability team at Google DeepMind, whose members had introduced the gated and JumpReLU autoencoders, changed its priorities:

As a result of this and parallel work, we are deprioritising fundamental SAE research for the moment and exploring other directions, though SAEs will remain a tool in our toolkit

We do not think that SAEs are useless or that no one should work on them, but we also do not think that SAEs will be a game-changer for interpretability, and speculate that the field is over-invested in them.

—L. Smith and others, "Negative Results for SAEs On Downstream Tasks and Deprioritising SAE Research" (2025). Progress update of the team, AI Alignment Forum, 26 March 2025. The probing study is S. Kantamneni and others, "Are sparse autoencoders useful? A case study in sparse probing", ICML 2025 (arXiv 2502.16681).

Other units. Three lines of work change the unit and keep the test. Parameter decomposition writes the weights, not the activations, as a sum of components of low rank, few of which are needed on any one input; its 2026 version, on a four-layer model of 67 million parameters, loses less cross-entropy than per-layer and cross-layer transcoders at equal sparsity, by its own accounting of sparsity.D. Braun and others, "Interpretability in parameter space: minimizing mechanistic description length with attribution-based parameter decomposition", arXiv 2501.14926, 2025; L. Bushnaq and others, "Interpreting language model parameters", arXiv 2610.20748, October 2026. Weight-sparse training makes the network itself sparse, about one weight in a thousand nonzero, and its circuits are about 16 times smaller at equal loss (the circuit that closes a string has 12 nodes and 9 edges), but it has not been scaled past tens of millions of nonzero weights.Gao, Rajaram, Coxon, Govande, Baker and Mossing, "Weight-sparse transformers have interpretable circuits", arXiv 2511.13653, 2025. The third returns to plain MLP neurons, on which traced circuits are reported to be about as sparse.Arora, Wu, Steinhardt and Schwettmann, "Language model circuits are sparse in the neuron basis", ICML 2026 (arXiv 2601.22594).

The state of play in November 2026

claim shown not shown, or disputed
a network can hold more features than dimensions in toy models, by training; in theory, by Theorem 10.1.6 how many a language model holds: no dictionary counts them
dictionary latents are interpretable scores above those of neurons, by hand and by machine aggregate scores nearly as high for random baselines
the latents are the model's features partial agreement between two trainings latents split, absorb and change with size and seed (one in six of gpt-pico's returns under a second seed); 9% recovered where the truth is known
features are directions single directions mediate behaviors (refusal, in 13 chat models) circles and curved manifolds that models compute on
a latent is a cause clamping produces the behavior its label names that it is the mechanism at ordinary activations; for known concepts simpler directions steer better
a graph explains a computation exact for the local replacement model; influence predicts ablation (rank correlation 0.72); interventions in the case studies most prompts (about a quarter give insight, by the authors' count; three of twelve graphs of gpt-pico pass our test); why attention attends; what sits in error nodes
What the evidence of 2022 to 2026 supports, claim by claim.

Our own measurements fall in the same columns. A dictionary of 16,384 latents rewrites a layer of gpt-pp for 0.17 nats a token, and four of five latents drawn from it at random take one label; a second seed on gpt-pico returns one latent in six. Clamping a latent of gpt-pico puts its subject into most stories for about half a nat a token. The graphs of gpt-pico are exact and readable, and an intervention confirms three of twelve.

The question has moved from whether dictionaries work to what they are for, and a census of 138 studies gives the working rule:

For a known predictive target, begin with a well-tuned supervised probe and a matched label budget. For output control, include direct activation directions before incurring dictionary-training and feature-selection costs. For a localized computational hypothesis, neurons, heads, or a functional surrogate may provide the required unit. Direct comparators support these choices (Ye et al., 2025; Arora et al., 2026; Shafran et al., 2026; Gurgurov et al., 2026). An SAE is more compelling when a reusable, fine-grained feature vocabulary enables discovery, comparison across many questions, or a demonstrated specificity/cost advantage.

—S. Lamsiyah and others, "Sparse Autoencoders for Mechanistic Interpretability in NLP: A Critical Survey of Recoverability, Interpretability, and Utility" (2026). arXiv 2610.20139, Section 6.2.

Meanwhile the tools are public, dictionaries, transcoders and cross-layer transcoders for open models,C. McDougall and others, "Announcing Gemma Scope 2", LessWrong, 22 December 2025: sparse autoencoders and transcoders for every layer of ten Gemma 3 models, cross-layer transcoders for two of them. and the newest work, which Section 11 takes up, mixes dictionaries with methods that need none.

A feature is a hypothesis about a network. The one test that every side of this debate accepts is an intervention in the network itself.

← Back to all posts