9. Circuits

Draft

The program: reverse engineering a trained network

A transformer with one layer of attention and no feed-forward layer can be read from its weights. Our example is a model 256 wide with eight heads, trained on the children's stories of TinyStories with the 4,096-token vocabulary of gpt-pico, the smallest size of gpt-pp and the one that runs in the page. A held-out story reaches "there was a big sack. The", and the model's first choice for the next token is "sack". Its logit, 15.8 above the mean over the vocabulary, is a sum of nine terms: 2.3 from the direct path that leads from the present token to the output, and one term from each attention head, of which one supplies 8.7 by putting 0.98 of its attention on the earlier "sack" and copying it. Each term is a lookup in tables over the vocabulary that can be printed without running the model.

... , there was a big sack . The  ->  sack   (probability 0.73)
logit of sack = the direct path + eight heads, attention frozen (from the mean over the vocabulary)
  direct  head 1  head 2  head 3  head 4  head 5  head 6  head 7  head 8  sum
  2.3     8.7     −0.3    −0.4    1.4     0.2     1.9     0.8     1.1     15.8
layer 1, head 1: sack ... The -> sack: attention 0.98, adds 8.7
what follows The:
  the direct path (logit)     a zero-layer model      the stories
  next 10.9                   bird 9 %                bird 9 %
  other 10.8                  cat 7 %                 cat 7 %
  m 10.3                      dog 5 %                 dog 5 %
  big 10.2                    little 4 %              little 4 %
first choice equal to the stories’ most frequent successor, over the vocabulary:
  the direct path 26 % of tokens, the zero-layer model 99 %
(the live figure needs JavaScript and loads about 0.6 MB of data)
One prediction of the one-layer model, split into its nine paths. Green: the chosen path, its segment of the bar. Ink: the other paths; the tick is their sum, the model’s logit, exact to 1 × 10⁻⁵. Below, the table that path is read from: for the direct path, what follows the present token, beside a zero-layer model and the stories’ bigrams; for a head, where it looks and what that look writes. Choose the word the text ends in, the next token asked about, a path, a source. Measured on one trained model and 240 places in held-out stories.

For each head, two tables say where it looks from the present token and what a look at an earlier token writes. The direct path scores next tokens from the present token alone. Section 9.2 proves that a model of this shape can express nothing else.

Reading a model this way is the aim of mechanistic interpretability: to reverse engineer a trained network into an algorithm stated in terms of its own parts, as one would a compiled program. The aim was set out in 2020, for vision models, as three claims.

Claim 1: Features
Features are the fundamental unit of neural networks. They correspond to directions. These features can be rigorously studied and understood.

Claim 2: Circuits
Features are connected by weights, forming circuits. These circuits can also be rigorously studied and understood.

Claim 3: Universality
Analogous features and circuits form across models and tasks.

—Chris Olah and others, "Zoom in: an introduction to circuits" (2020). C. Olah, N. Cammarata, L. Schubert, G. Goh, M. Petrov and S. Carter, Distill, 10 March 2020 (doi 10.23915/distill.00024.001), the box "Three speculative claims about neural networks".

In a convolutional network the candidate features were neurons, and the claims could be tested by looking at what excites one. Transformers broke the method twice. Attention has no neurons: what a head does depends on which positions it joins, which changes with every input. And the vectors that layers pass to one another have no preferred coordinates. The 2021 paper "A mathematical framework for transformer circuits" answered by rewriting the transformer in an equivalent form whose weights can be read again.N. Elhage, N. Nanda, C. Olsson and others (25 authors, C. Olah last), Transformer Circuits Thread, 22 December 2021; no arXiv number. Its expansions into paths are printed as images; the equations here are derived again from the model's definition. Section 9.2 and Section 9.3 rebuild it with its equations. Section 9.4 follows what the field did when real models proved unreadable in this way, which was to intervene on activations; Section 9.5 takes up the feed-forward layers, and Section 9.6 the limit that leads to Section 10.

Definition 9.1.1 (a circuit, and what is asked of it). A circuit for a behavior of a model is a set of its components (token embeddings, attention heads, feed-forward layers or their neurons, the unembedding) and of paths between them through the residual stream, with a statement of what each computes. Three things are asked of it, in rising order. It can be read: the weights or activations of its components exhibit the stated computation. It is causal: changing a component's activation changes the output as the statement predicts. It is accounted for: a number says how much of the behavior remains when everything outside the circuit is replaced by something uninformative.

The tests arrived in that order: the framework reads, activation patching intervenes, faithfulness scores and causal scrubbing account. The published circuits of real models pass the third only in part. Even the best case is understanding of a particular kind, as the framework paper says of its own one-layer model.

There's often skepticism that it's even possible or worth trying to truly reverse engineer neural networks. That being the case, it's tempting to point at one-layer attention-only transformers and say "look, if we take the most simplified, toy version of a transformer, at least that minimal version can be fully understood."

But that claim really depends on what one means by fully understood. It seems to us that we now understand this simplified model in the same sense that one might look at the weights of a giant linear regression and understand it, or look at a large database and understand what it means to query it. That is a kind of understanding. There's no longer any algorithmic mystery. The contextualization problem of neural network parameters has been stripped away. But without further work on summarizing it, there's far too much there for one to hold the model in their head.

—Nelson Elhage and others, "A mathematical framework for transformer circuits" (2021). The section "Do we “fully understand” one-layer models?", its first two paragraphs.

The framework: a transformer as a sum of paths

The framework studies transformers with attention only. Leaving out the feed-forward layers gives up most of a real model's parameters, to which Section 9.5 returns, and buys exactness: with its attention patterns held fixed, such a model is linear in its input tokens.The qualification matters. These paths reconstruct a particular forward pass exactly, but changing the input generally changes the attention patterns too. The conditional linear expansion does not make the transformer a linear function of arbitrary text. The paper's three results are the plan of this section and the next.

To explore the challenge of reverse engineering transformers, we reverse engineer several toy, attention-only models. In doing so we find:

Zero layer transformers model bigram statistics. The bigram table can be accessed directly from the weights.

One layer attention-only transformers are an ensemble of bigram and “skip-trigram” (sequences of the form "A… B C") models. The bigram and skip-trigram tables can be accessed directly from the weights, without running the model. These skip-trigrams can be surprisingly expressive. This includes implementing a kind of very simple in-context learning.

Two layer attention-only transformers can implement much more complex algorithms using compositions of attention heads. These compositional algorithms can also be detected directly from the weights. Notably, two layer models use attention head composition to create “induction heads”, a very general in-context learning algorithm.

—Nelson Elhage and others, "A mathematical framework for transformer circuits" (2021). "Summary of results": the sentence that introduces the reverse engineering results and the first three of its four items.

Notation 9.2.1 (the bare attention-only transformer). The vocabulary has \(V\) tokens, the stream has width \(d\), a head has width \(d_h\), and the context is \(t_1, \dots, t_n\). Write \(e_a \in \{0,1\}^V\) for the one-hot vector of token \(a\), and \(t \in \{0,1\}^{n \times V}\) for the matrix whose row \(i\) is \(e_{t_i}^{\top}\). The embedding is \(W_E \in \mathbb{R}^{d \times V}\), the unembedding \(W_U \in \mathbb{R}^{V \times d}\). The residual stream after layer \(\ell\) is a matrix \(X^\ell \in \mathbb{R}^{n \times d}\) with one row per position, written \(x_i^\ell\) as a column vector when it stands alone, and \(X^0 = t\, W_E^{\top}\). A head \(h\) has matrices \(W_Q^h, W_K^h, W_V^h \in \mathbb{R}^{d_h \times d}\) and \(W_O^h \in \mathbb{R}^{d \times d_h}\). Layer \(\ell\), with its set of heads \(H_\ell\), computes

\[\begin{aligned} x_i^{\ell} &= x_i^{\ell-1} + \sum_{h \in H_\ell} \sum_{j \le i} A^h_{ij}\; W_O^h W_V^h\, x_j^{\ell-1}, \\ A^h_{ij} &= \frac{\exp s^h_{ij}}{\sum_{k \le i} \exp s^h_{ik}}, \qquad s^h_{ij} = \bigl(W_Q^h x_i^{\ell-1}\bigr)^{\top}\, W_K^h\, x_j^{\ell-1}, \end{aligned} \tag{9.1}\]

and the logits at position \(i\), for the token at \(i + 1\), are \(W_U\, x_i^L\). Row \(i\) of the pattern \(A^h\) belongs to the destination, the position that reads; column \(j\) to the source, the position that is read. There are no biases, no normalization and no feed-forward layers. The factor \(1/\sqrt{d_h}\) of Section 3 is absorbed into \(W_Q^h\), and position enters only the scores, through the offset \(i - j\), which the notation omits until Remark 9.2.11. For \(A \in \mathbb{R}^{n \times n}\) and \(W \in \mathbb{R}^{d^{\prime} \times d}\), \(A \otimes W\) denotes the linear map \(X \mapsto A X W^{\top}\): \(A\) mixes positions and \(W\) acts on the vector at each position. Then \((A \otimes W)(A^{\prime} \otimes W^{\prime}) = AA^{\prime} \otimes WW^{\prime}\), because \(A\,(A^{\prime} X W^{\prime\top})\, W^{\top} = (AA^{\prime})\, X\, (WW^{\prime})^{\top}\).

The equations start from the stream.

One of the main features of the high level architecture of a transformer is that each layer adds its results into what we call the “residual stream.” The residual stream is simply the sum of the output of all the previous layers and the original embedding. We generally think of the residual stream as a communication channel, since it doesn't do any processing itself and all layers communicate through it.

—Nelson Elhage and others, "A mathematical framework for transformer circuits" (2021). The section "Virtual weights and the residual stream as a communication channel".

Proposition 9.2.2 (heads add, and a head is one map). (i) If the results \(r_i^1, \dots, r_i^H \in \mathbb{R}^{d_h}\) of a layer's heads are stacked and multiplied by one output matrix \(W_O = [W_O^1 \cdots W_O^H]\), the layer writes \(\sum_h W_O^h r_i^h\): the same as running each head alone and adding. (ii) Given its pattern, a head is the linear map \(A^h \otimes W_{OV}^h\) with \(W_{OV}^h = W_O^h W_V^h\), and its scores are \(s_{ij}^h = x_i^{\top} W_{QK}^h\, x_j\) with \(W_{QK}^h = W_Q^{h\top} W_K^h\). So (9.1) reads

\[X^{\ell} = \Bigl(\mathrm{Id} + \sum_{h \in H_\ell} A^h \otimes W_{OV}^h\Bigr) X^{\ell-1}. \tag{9.2}\]

Proof. (i) is block multiplication. (ii) The head writes \(\sum_j A_{ij} W_O W_V x_j\) at position \(i\), which is row \(i\) of \(A X (W_O W_V)^{\top}\). ∎

The four matrices of a head act only as two products: \(W_{QK}^h\) decides where the head looks and \(W_{OV}^h\) what a look moves, both \(d \times d\) of rank at most \(d_h\); in the paper's words, "which tokens to move information from is completely separable from what information is “read” to be moved and how it is “written” to the destination". And the pattern is the only nonlinearity: with \(A^h\) held fixed, a head is linear.

Proposition 9.2.3 (the stream has no privileged basis). For any invertible \(G \in \mathbb{R}^{d \times d}\), replace every matrix that writes to the stream, \(W_E\) and each \(W_O^h\), by \(G\) times itself, and every matrix that reads from it, \(W_U\) and each \(W_Q^h\), \(W_K^h\), \(W_V^h\), by itself times \(G^{-1}\). All logits are unchanged.

Proof. By induction on the layer every \(x_i^\ell\) becomes \(G x_i^\ell\). A reader sees \(W G^{-1} G x = W x\), so scores, patterns and results are as before, and a writer adds \(G\) times what it added before. The unembedding reads \(W_U G^{-1} G\, x_i^L\). ∎

So a coordinate of the stream means nothing, and neither does a matrix that touches the stream on one side only. What is defined is a writer times a reader, which the paper names a virtual weight.

An especially useful consequence of the residual stream being linear is that one can think of implicit "virtual weights" directly connecting any pair of layers (even those separated by many other layers), by multiplying out their interactions through the residual stream. These virtual weights are the product of the output weights of one layer with the input weights of another (ie. \(W_{I}^2W_{O}^1\)), and describe the extent to which a later layer reads in the information written by a previous layer.

—Nelson Elhage and others, "A mathematical framework for transformer circuits" (2021). The subsection "Virtual weights".

The one-hot basis of tokens, by contrast, is fixed by the task, so every matrix read in this part begins and ends in the vocabulary: \(W_U W_E\), \(W_U W_{OV}^h W_E\) and \(W_E^{\top} W_{QK}^h W_E\) are unchanged by \(G\).

Proposition 9.2.4 (zero layers: a bigram table). With no layers the logits at position \(i\) are column \(t_i\) of the \(V \times V\) matrix \(W_U W_E\). For any table \(L \in \mathbb{R}^{V \times V}\) used in this way, the expected next-token loss is

\[\mathcal{L}(L) = H(c \mid a) + \sum_a P(a)\; \mathrm{KL}\bigl(P(\cdot \mid a)\,\big\Vert\, \mathrm{softmax}(L_{\cdot a})\bigr),\]

where \(P(a)\) is the frequency of token \(a\) in the corpus, \(P(c \mid a)\) the frequency of \(c\) after \(a\), and \(H(c \mid a)\) the conditional entropy of the next token given the present one. The loss is smallest when \(L_{ca} = \log P(c \mid a) + \kappa_a\) for constants \(\kappa_a\).

Proof. The loss is \(-\sum_a P(a) \sum_c P(c \mid a) \log \mathrm{softmax}(L_{\cdot a})_c\), and a cross-entropy is an entropy plus a relative entropy, which vanishes only when the two laws agree (Gibbs' inequality, Section 1). ∎

Since \(W_U W_E\) has rank at most \(d\), the zero-layer model is the best bigram table of rank \(d\) in the sense of the display, and nothing more. With tied matrices, \(W_U = W_E^{\top}\), the table would be symmetric and positive semidefinite, which bigram statistics are not ("of the" is frequent, "the of" is not): one reason the small models of this series keep two matrices (Section 2).

Example 9.2.5 (what bigrams are worth on this corpus). The setup: TinyStories under gpt-pico's tokenizer; n-gram models with interpolated Kneser-Ney smoothing, counted on the first 200,000,000 training tokens and scored on the held-out stories; a zero-layer model of width 256 fitted to the stories' bigram counts; and attention-only models of one and two layers: 256 wide, eight heads of width 32 in a layer, a context of 256 tokens, rotary positions (Section 4), separate embedding and unembedding matrices, and no biases, normalization or cap on the logits, so bare in the sense of Notation 9.2.1, each trained once on 200,015,872 tokens. Costs are in bits per byte of held-out text.

A unigram model costs 2.126, a bigram model 1.282, a trigram model 1.008 and a four-gram model 0.897. By Proposition 9.2.4 a zero-layer model is a bigram model of rank at most 256, and the rank costs next to nothing: the fitted one is within 0.003 nats a token of the entropy of the stories' own bigram table. The one-layer model costs 0.725 and the two-layer model 0.643; gpt-pico, with six layers, costs 0.552.The models are scored on the first 1,048,576 held-out tokens in windows of 256 and their loss is converted at the held-out file's 4.03 bytes a token; the n-gram models are scored on the whole held-out set. gpt-pico was trained on 540,016,640 tokens. One layer of attention is already ahead of the four-gram model, which sees three tokens back and no further. Inside the one-layer model \(W_U W_E\) is one path among nine, and it is no longer the bigram table: with every head's output set to zero the model costs 1.828, and the direct path's first choice is the stories' most frequent successor for 26% of tokens, against 99% for the zero-layer model. Training has moved into the heads what the heads can carry. ◁

Theorem 9.2.6 (one layer: bigrams plus skip-trigrams). The logits \(T \in \mathbb{R}^{n \times V}\) of a one-layer model are

\[T = \underbrace{(\mathrm{Id} \otimes W_U W_E)\, t}_{\text{direct path}} \;+\; \sum_{h \in H_1} \underbrace{\bigl(A^h \otimes W_U W_{OV}^h W_E\bigr)\, t}_{\text{head } h}. \tag{9.3}\]

Entry by entry, with \(C_{OV}^h = W_U W_{OV}^h W_E\) and \(C_{QK}^h = W_E^{\top} W_{QK}^h W_E\),

\[\begin{aligned} T_{ic} &= (W_U W_E)[c, t_i] + \sum_{h \in H_1} \sum_{j \le i} A^h_{ij}\; C_{OV}^h[c, t_j], \\ A^h_{ij} &= \frac{\exp C_{QK}^h[t_i, t_j]}{\sum_{k \le i} \exp C_{QK}^h[t_i, t_k]}. \end{aligned} \tag{9.4}\]

Proof. By (9.2), \(T = (\mathrm{Id} \otimes W_U)\bigl(\mathrm{Id} + \sum_h A^h \otimes W_{OV}^h\bigr)(\mathrm{Id} \otimes W_E)\, t\); distribute and use the product rule of Notation 9.2.1. The scores are \(x_i^{0\top} W_{QK}^h\, x_j^0\) with \(x_i^0 = W_E\, e_{t_i}\). ∎

In the code. With the patterns recorded, (9.3) is a loop over the heads of a layer, and the same loop run again for each further layer gives the expansions of Section 9.3. It is path_expansion in gpt-pp's interp/circuits.py.

@torch.no_grad()
def path_expansion(model: GPT, idx: Tensor, patterns: Optional[Sequence[Tensor]] = None) -> Paths:
    ...
    if not is_exact(model):
        raise ValueError("the path expansion is exact only for an attention-only model without normalization, "
                         "biases, QK normalization or a logit soft-cap (see is_exact)")
    if patterns is None:
        patterns = attention_patterns(model, idx)
    heads: list[tuple[Head, ...]] = [()]
    terms = [embedding(model)[idx]]
    for layer in range(model.cfg.n_layer):
        new_heads, new_terms = [], []
        for head in range(model.cfg.n_head):
            ov, pattern = ov_matrix(model, layer, head), patterns[layer][:, head].double()
            for path, term in zip(heads, terms):
                new_heads.append(path + ((layer, head),))
                new_terms.append(pattern @ (term @ ov.T))
        heads, terms = heads + new_heads, terms + new_terms
    return Paths(heads, torch.stack(terms), unembedding(model))

terms holds one array of shape (batch, positions, \(d\)) for each path and starts with the direct path, the embedded tokens; the line pattern @ (term @ ov.T) applies one \(A^h \otimes W_{OV}^h\) to a path already in the list, and Paths keeps \(W_U\) to turn each term into logits. The code stores a token as a row where the formulas have a column, hence the transposes, and it does the arithmetic on the weights in 64-bit floats. No score is computed: the patterns are the model's own, recorded by the hooks of Section 9.4, so the rotations of Remark 9.2.11 are already inside them.

Known issue. The first lines of the function are the framework's fine print. is_exact is true only for a model with no feed-forward layers, no normalization anywhere, no biases, no normalized queries and keys, no cap on the logits, no dropout, and positions that never enter the stream; for any other model path_expansion raises an error. The tables and scores that follow (ov_circuit, qk_circuit, composition, copying_statistic) make no such check: on gpt-pp or gpt-pico they would return products of the same matrices, which are no longer what the model computes, and none of them is applied to those two here. The attention-only models of this part are separate small runs made to pass the check, with GPT-2's initialization (init=gpt2) and the learning rates of the two tables set by hand; each is one run with one seed, and gpt-pp's review of its results lists them among what it did not check.

The two \(V \times V\) matrices of (9.4) are the objects the paper names.

A Mathematical Framework for Transformer CircuitsTransformer Circuits Thread 2021N. Elhage, N. Nanda, C. Olsson and others

The key thing to notice is that these terms consist of two separable operations, which are at their heart two \([n_\text{vocab},~ n_\text{vocab}]\) matrices:

\(W_E^T W_{QK}^h W_E\) — We call this matrix the "query-key (QK) circuit." It provides the attention score for every query and key token. That is, each entry describes how much a given query token "wants" to attend to a given key token.

\(W_UW_{OV}^hW_E\) — We call this matrix the “Output-Value (OV) circuit.” It describes how a given token will affect the output logits if attended to.

Splitting attention head terms into query-key and output-value circuits
N. Elhage, N. Nanda, C. Olsson and others, "A Mathematical Framework for Transformer Circuits" (Transformer Circuits Thread 2021), Splitting attention head terms into query-key and output-value circuits.

Definition 9.2.7 (the QK and OV circuits of a head). \(C_{QK}^h[b, a]\) is the score with which a destination token \(b\) attends to a source token \(a\). \(C_{OV}^h[c, a]\) is what a unit of attention to a source token \(a\) adds to the logit of \(c\). A source token somewhere earlier, the destination token and an output token, written \(a \dots b \to c\), form a skip-trigram.

With the patterns recorded from one run and then held fixed, (9.3) is linear in \(t\): every dependence on context sits in \(A^h\). Freezing attention in this way is the framework's central device, and it returns in every later section. One caution in reading the tables: a softmax ignores a constant added to all its arguments, so a column of an output table and a row of a score table are centered before they are ranked.

In the code. Each table is a product of one head's slices of the weights, and can be restricted to chosen tokens so that no \(V \times V\) array is built unless it is asked for. They are ov_circuit and qk_circuit in circuits.py.

def ov_circuit(model: GPT, layer: int, head: int, sources: Tokens = None, outputs: Tokens = None) -> Tensor:
    """The OV circuit U W_OV E^T, [output token, source token]: the change in each logit per unit of attention
    paid to a source token (FW, "Splitting attention head terms into query-key and output-value circuits")."""
    w = head_weights(model, layer, head)
    return (_rows(unembedding(model), outputs) @ w.o) @ (w.v @ _rows(embedding(model), sources).T)


def qk_circuit(model: GPT, layer: int, head: int, offset: float = 1, dests: Tokens = None, sources: Tokens = None,
               center: bool = False) -> Tensor:
    """The QK circuit E W_QK E^T, [destination token, source token]: the attention score, before the softmax, of
    a destination token for a source token `offset` positions before it.

    A softmax ignores a constant added to a row, so scores are comparable within a row only: `center=True`
    subtracts each row's mean over the sources given.
    """
    w = head_weights(model, layer, head)
    queries = _rows(embedding(model), dests) @ w.q.T
    keys = _rows(embedding(model), sources) @ w.k.T
    table = queries @ relative_rotation(model, offset) @ keys.T / math.sqrt(model.cfg.d_head)
    return table - table.mean(dim=1, keepdim=True) if center else table

In the docstrings U and E are \(W_U\) and \(W_E^{\top}\), a token to a row, W_OV is w.o @ w.v, and FW is the 2021 paper. The \(1/\sqrt{d_h}\) that Notation 9.2.1 folds into \(W_Q^h\) is written out. Two arguments go beyond the definition: offset chooses the rotation between query and key, the \(R^{-\delta}\) of Remark 9.2.11 below, with the previous token as its default; and center=True is the centering of the paragraph above for a row of scores, while a column of an OV table is left for the caller to center.

Proposition 9.2.8 (skip-trigrams come in rectangles). Fix a head and a source token \(a\). Whatever the destination and the rest of the context, what the head adds to the logits through a source position that carries \(a\) is a positive multiple of the one column \(C_{OV}[\cdot, a]\). Hence if attending to \(a\) raises \(c_1\) after \(b_1\) and \(c_2\) after \(b_2\), it also raises \(c_2\) after \(b_1\) and \(c_1\) after \(b_2\).

Proof. In (9.4) the term of source position \(j\) is \(A_{ij}\, C_{OV}[c, t_j]\). The first factor depends on the destination and not on the output, the second on the output and not on the destination, and \(A_{ij} > 0\). ∎

The paper's instance:

Our one-layer models represent skip-trigrams in a "factored form" split between the OV and QK matrices. It's kind of like representing a function \(f(a,b,c) = f_1(a,b) f_2(a,c)\). They can't really capture the three way interactions flexibly. For example, if a single head increases the probability of both keep… in mind and keep… at bay, it must also increase the probability of keep… in bay and keep… at mind. This is likely a good trade for the model on balance, but is also, in some sense, a bug. We frequently observe these in attention heads.

—Nelson Elhage and others, "A mathematical framework for transformer circuits" (2021). The section "Skip-trigram “bugs”".

A table over the vocabulary has 16,777,216 entries here and about 2.5 billion at the paper's vocabulary of about 50,000, so it needs a summary. Most skip-trigrams the paper found were copies, a token seen earlier raising its own logit, so the summary asks of the OV circuit whether tokens raise themselves.

Definition 9.2.9 (positivity of a circuit). For a square real matrix \(M\) with eigenvalues \(\lambda_1, \dots, \lambda_m\), not all zero,

\[\mathrm{pos}(M) = \frac{\sum_k \lambda_k}{\sum_k \lvert\lambda_k\rvert} = \frac{\operatorname{tr} M}{\sum_k \lvert\lambda_k\rvert} \in [-1, 1].\]

A head is called copying when \(\mathrm{pos}(C_{OV}^h)\) is near one.

The numerator is the trace: the sum over tokens of what a token adds to its own logit when attended to. No \(V \times V\) matrix is needed: the nonzero eigenvalues of \(AB\) are those of \(BA\), so those of \(W_U W_O \cdot W_V W_E\) are the eigenvalues of the \(d_h \times d_h\) matrix \(W_V W_E W_U W_O\). The statistic is evidence and not proof: \(\bigl(\begin{smallmatrix} -1 & -4 \\ 4 & 9 \end{smallmatrix}\bigr)\) has eigenvalues 1 and 7 while its first token lowers its own logit. In the paper's one-layer model, 10 of 12 heads copy by this measure.

Example 9.2.10 (the one-layer model, read from its weights). The setup: the one-layer model of Example 9.2.5. We record every pattern on 256 held-out windows, rebuild the logits from (9.3), and compare with the forward pass; then compute the positivity of every head's OV circuit, at initialization and after training. The rebuilt logits differ from the model's by at most \(4.4 \times 10^{-5}\), where the largest logit is 54. At initialization every head's positivity is within 0.12 of zero; after training four of the eight are above 0.8 (heads 2, 3, 4 and 8) and the others stand at −0.14, 0.06, 0.21 and −0.65. The summary keeps its caveat: the head that copies "sack" in the opening example is head 1, the one at −0.14. ◁

positivity of every head’s OV circuit: the sum of its eigenvalues over the sum of their moduli
(1: every eigenvalue positive, the head raises the tokens it looks at; 0: no tendency)

head                1      2      3      4      5      6      7      8      
the one-layer model
  at initialization 0.03   −0.01  0.02   −0.05  −0.05  0.03   −0.11  −0.01   0 above 0.8
  after training    −0.14  0.88   0.95   0.86   0.06   0.21   −0.65  1.00    4 above 0.8
layer 1 of the two-layer model
  at initialization 0.06   −0.04  0.11   0.01   −0.03  −0.08  −0.03  0.01    0 above 0.8
  after training    −0.56  0.60   −0.56  −0.53  −0.19  0.21   −0.66  0.38    0 above 0.8
layer 2 of the two-layer model
  at initialization −0.02  −0.07  0.00   0.07   −0.05  0.04   0.05   −0.01   0 above 0.8
  after training    0.95   0.98   0.68   0.98   −0.88  0.92   0.93   0.99    6 above 0.8

each head as two tables over 56 tokens:
  names, pronouns, verbs, nouns, adjectives, function words, punctuation
  where it looks   row: the token looking; column: the token looked at, some tokens back
  what it writes   column: the token looked at; row: the token whose logit changes
(the live figure needs JavaScript and loads about 0.5 MB of data)
Heads read from their weights. Green: the heads that copy, drawn as the eigenvalues of every OV circuit with positivity above 0.8. Ink: the other heads’ eigenvalues; in the two tables, positive entries. Blue: negative entries. Upper row of planes: at initialization; lower row: after training. Choose a plane to see that head’s tables over 56 tokens; choose the layer; move the distance of the look. Point at a cell to read it. Everything is computed from the weights of two models, one training run each.

Training moves several heads’ eigenvalues from a cloud around zero toward the positive real axis. Head 8 has a strong copying diagonal over nouns and adjectives in its OV table; its QK table depends on the distance between tokens.

Remark 9.2.11 (what a real recipe changes). Position: the paper adds position vectors to the inputs of \(W_Q\) and \(W_K\) only and rotary embedding rotates queries and keys, so in both position never enters the stream. With rotary embedding the score of (9.1) is \(s_{ij} = x_i^{\top} W_Q^{\top} R^{\,j-i}\, W_K\, x_j\), with \(R\) the rotation of one position, so the QK circuit is one table for each offset, \(C_{QK}^{(\delta)} = W_E^{\top} W_Q^{\top} R^{-\delta} W_K W_E\) for a source \(\delta\) positions back, and the OV circuit is unchanged. Normalization: dividing the stream by its root mean square before each read (Section 5) multiplies every path into that read by one positive scalar for each position; with those scalars recorded beside the patterns the expansion stays exact. With the normalized queries and keys of gpt-pp (Section 3) the table for each offset is still a product, of matrices whose columns are normalized; what is lost is that a score is bilinear in the stream.

Composition and the induction head

A second layer reads a stream that the first has written to.

Proposition 9.3.1 (two layers: paths of three orders). The logits of a two-layer model are

\[\begin{aligned} T ={}& (\mathrm{Id} \otimes W_U W_E)\, t \;+ \sum_{h \in H_1 \cup H_2} \bigl(A^h \otimes W_U W_{OV}^h W_E\bigr)\, t \\ &+ \sum_{h_2 \in H_2} \sum_{h_1 \in H_1} \bigl(A^{h_2} A^{h_1} \otimes W_U W_{OV}^{h_2} W_{OV}^{h_1} W_E\bigr)\, t . \end{aligned} \tag{9.5}\]

Proof. Apply (9.2) twice between \(\mathrm{Id} \otimes W_E\) and \(\mathrm{Id} \otimes W_U\) and multiply out. ∎

With \(H\) heads in a layer there are 1, \(2H\) and \(H^2\) terms: 1, 24 and 144 in the paper's model of 12 heads a layer. A term of the last kind is a virtual head: the product of two heads is again a pattern, \(A^{h_2} A^{h_1}\), with a matrix, \(W_{OV}^{h_2} W_{OV}^{h_1}\). The weight of each order can be measured without multiplying anything out: with the recorded patterns forced, replace every head's output by zero, which leaves the direct path, then by what the head produced in the run before, which leaves the paths through at most one head, and so on. In the paper's two-layer model the direct path is 1.8 nats better than a uniform guess, the 24 single heads gain a further 5.2 nats and the 144 virtual heads a further 0.3. Composition of values does little there; what the second layer does with the first is in its patterns.

Proposition 9.3.2 (the second layer's scores). Let \(h\) be a head of layer 2. Its score from destination \(i\) to source \(j\) is \(x_i^{1\top} W_{QK}^h\, x_j^1\), with \(x_i^1 = x_i^0 + \sum_{g \in H_1} \sum_m A^g_{im} W_{OV}^g\, x_m^0\), and is a sum of four kinds of term:

\[\begin{aligned} s^h_{ij} ={}& x_i^{0\top}\, W_{QK}^h\; x_j^0 && \text{(no composition)} \\ &+ \sum_{g} \sum_{m} A^{g}_{im}\; x_m^{0\top}\, W_{OV}^{g\top} W_{QK}^h\; x_j^0 && \text{(Q-composition)} \\ &+ \sum_{g} \sum_{m} A^{g}_{jm}\; x_i^{0\top}\, W_{QK}^h W_{OV}^{g}\; x_m^0 && \text{(K-composition)} \\ &+ \sum_{g, g^{\prime}} \sum_{m, m^{\prime}} A^{g}_{im} A^{g^{\prime}}_{jm^{\prime}}\; x_m^{0\top}\, W_{OV}^{g\top} W_{QK}^h W_{OV}^{g^{\prime}}\; x_{m^{\prime}}^0 && \text{(both)}. \end{aligned} \tag{9.6}\]

With \(x^0 = W_E\, e_t\) each bilinear form is a table over pairs of tokens: \(W_E^{\top} W_{QK}^h W_E\), \(W_E^{\top} W_{OV}^{g\top} W_{QK}^h W_E\), \(W_E^{\top} W_{QK}^h W_{OV}^g W_E\).

Proof. Expand the product. ∎

The paper names three ways in which a head reads what an earlier head wrote, after the matrix that does the reading. Q-composition and K-composition are the second and third lines of (9.6); V-composition is the virtual head of (9.5).

Q- and K-Composition are quite different from V-Composition. Q- and K-Composition both affect the attention pattern, allowing attention heads to express much more complex patterns. V-Composition, on the other hand, affects what information an attention head moves when it attends to a given position; the result is that V-composed heads really act more like a single unit and can be thought of as creating an additional "virtual attention heads". Composing movement of information with movement of information gives movement of information, whereas attention heads affecting attention patterns is not reducible in this way.

—Nelson Elhage and others, "A mathematical framework for transformer circuits" (2021). The section "Three kinds of composition".

Definition 9.3.3 (composition scores). For a head \(g\) of an earlier layer and a head \(h\) of a later one,

\[\begin{aligned} \mathrm{comp}(g, h) &= \frac{\lVert M\, W_{OV}^{g} \rVert_F}{\lVert M \rVert_F\; \lVert W_{OV}^{g} \rVert_F}, \\ M &= W_{QK}^{h\top} \ \text{(Q)}, \qquad M = W_{QK}^{h} \ \text{(K)}, \qquad M = W_{OV}^{h} \ \text{(V)}, \end{aligned}\]

with \(\lVert\cdot\rVert_F\) the Frobenius norm, the root of the sum of squared entries. Each score lies in \([0, 1]\), because \(\lVert AB \rVert_F \le \lVert A \rVert_F \lVert B \rVert_F\).

The numerators are the matrices of (9.6) and (9.5) without their embeddings: a score asks how much of what \(g\) writes lands where \(h\) reads, weighted by singular values.

Proposition 9.3.4 (the baseline of a composition score). For fixed \(M_1, M_2 \in \mathbb{R}^{d \times d}\) and a uniformly random orthogonal \(Q\), \(\mathbb{E}\,\lVert M_2\, Q\, M_1 \rVert_F^2 = \lVert M_2 \rVert_F^2\, \lVert M_1 \rVert_F^2 / d\): when what one head writes is in general position to what another reads, the mean square of their score is \(1/d\).

Proof. \(\lVert M_2 Q M_1 \rVert_F^2 = \operatorname{tr}\bigl(M_1^{\top} Q^{\top} S\, Q\, M_1\bigr)\) with \(S = M_2^{\top} M_2\), and \(\mathbb{E}[Q^{\top} S Q]\) commutes with every orthogonal matrix, so it is the multiple of the identity with trace \(\operatorname{tr} S\). ∎

The baseline is \(1/\sqrt{d}\), 0.0625 at width 256. The paper subtracts an empirical baseline of this kind, and observes that most pairs of heads compose less than random matrices would and a few far more.

In the code. The three scores are norms of products of two heads' matrices, and the baseline is the closed form just derived. They are composition and baseline in circuits.py; the branch left out of the first, cyclic=True, is the subject of the remark below.

def composition(model: GPT, first: Head, second: Head, offset: float = 0, cyclic: bool = False) -> dict[str, float]:
    ...
    w1, w2 = head_weights(model, *first), head_weights(model, *second)
    rotation = relative_rotation(model, offset)
    ...
    ov = w1.o @ w1.v
    qk = w2.q.T @ rotation @ w2.k
    ov2 = w2.o @ w2.v
    norm_ov, norm_qk, norm_ov2 = _frobenius(ov), _frobenius(qk), _frobenius(ov2)
    return {"q": _ratio(_frobenius(qk.T, ov), norm_qk * norm_ov),
            "k": _ratio(_frobenius(qk, ov), norm_qk * norm_ov),
            "v": _ratio(_frobenius(ov2, ov), norm_ov2 * norm_ov)}
...
def baseline(d_model: int) -> float:
    ...
    return 1 / math.sqrt(d_model)

ov is \(W_{OV}^g\) of the earlier head, and qk and ov2 are \(W_{QK}^h\) and \(W_{OV}^h\) of the later one, each \(d \times d\); _frobenius multiplies its arguments and takes the norm, and _ratio returns 0 when a norm in the denominator is 0, as for an output matrix initialized at zero. The \(1/\sqrt{d_h}\) of the scores is left out of qk and cancels in the ratio.

Quirk. Three choices here are the code's. The baseline subtracted is the closed form and not a measured one; the measurement the paper describes, on Gaussian matrices of the same shapes, is composition_baseline, and it agrees with \(1/\sqrt{d}\) to 4%. \(W_{QK}^h\) is taken at offset=0, without rotation, where qk_circuit defaults to 1; it matters little, the largest K-composition score of Example 9.3.9 being 0.081 over the baseline at offset 0 and 0.081 to 0.083 at offsets of 1, 4, 16 and 64. The permuted product of the remark below has another baseline, about \(1/\sqrt{d_h}\), and that one is measured.

Remark 9.3.5 (two cautions about composition scores). First, a score is not a property of the function the model computes: under the change of basis of Proposition 9.2.3 the product \(W_{QK}^h W_{OV}^g\) becomes \(G^{-\top}\, W_{QK}^h W_{OV}^g\, G^{-1}\), whose Frobenius norm is the original's only when \(G\) is orthogonal. The tables of Proposition 9.3.2 are invariant; the score trusts the coordinates training happened to produce.

Second, the 2021 paper's diagram of composition in its two-layer model showed nearly one thing, a few heads of the second layer K-composing with one head of the first, and a correction added later explains why.This is a correction by the original authors. Their erratum reports a bug in the Frobenius-norm calculation that hid other head compositions. The original diagram and its unusually simple circuit interpretation are superseded; the algebraic path expansion used here is unaffected.

Following the publication of this paper, we became aware of a bug in an underlying library we wrote. This only affected one diagram, but does impact on our interpretation of the "Two-Layer Attention Only Transformers" section in some ways. In particular, there is more attention head composition going on in that model than it seemed.

Our analysis required us to manipulate low-rank matrices efficiently (as discussed in the appendix section "Working with Low-Rank Matrices"). In order to do this, we wrote a library for manipulating "strands" of matrix multiplications which result in low-rank matrices. For several computations, there's an identity where one can transpose a matrix product to compute it more efficiently – for example for the trace (\(Tr(AB) = Tr(BA)\)) or eigenvalues (\(\lambda_i(AB) = \lambda_i(BA)\)). We accidentally applied an analogous identity to accelerate computing the Frobenius norm, possibly because the implementer applied the eigenvalue identity to reasoning about the singular values which govern the Frobenius norm.

—Chris Olah, correction to "A mathematical framework for transformer circuits". "Comments and replications", the note "Correction: attention head composition diagram", its first paragraph and the first under "Technical details". It is undated; the summary of follow-up work beside it is as of February 2023.

Singular values do not survive a cyclic permutation: with \(A = \bigl(\begin{smallmatrix} 1 & 0 \\ 0 & 0 \end{smallmatrix}\bigr)\) and \(B = \bigl(\begin{smallmatrix} 0 & 1 \\ 0 & 0 \end{smallmatrix}\bigr)\), \(AB = B\) has norm one and \(BA = 0\). The code had computed a permuted product such as \(\lVert W_K^h\, W_O^g W_V^g\, W_Q^{h\top} \rVert_F\) in place of \(\lVert W_Q^{h\top} W_K^h\, W_O^g W_V^g \rVert_F\). The permuted product is large only when the key of \(h\) reads what \(g\) writes and the query of \(h\) reads the subspace that the value of \(g\) reads: the signature of the induction circuit described next. The diagram singled that circuit out and hid the rest.

What composition is used for, in the paper's model and in the models examined since, is this.

If you played around with the attention patterns above, you may have already guessed what induction heads do. Induction heads search over the context for previous examples of the present token. If they don't find it, they attend to the first token (in our case, a special token placed at the start), and do nothing. But if they do find it, they then look at the next token and copy it. This allows them to repeat previous sequences of tokens, both exactly and approximately.

—Nelson Elhage and others, "A mathematical framework for transformer circuits" (2021). The section "Function of induction heads".

Proposition 9.3.6 (one layer cannot do this). In a one-layer model whose scores depend on position through offsets only, the ratio of the attention that a destination \(i\) gives to two sources \(j\) and \(j^{\prime}\) is

\[\frac{A^h_{ij}}{A^h_{ij^{\prime}}} = \exp\bigl(C_{QK}^{(i-j)}[t_i, t_j] - C_{QK}^{(i-j^{\prime})}[t_i, t_{j^{\prime}}]\bigr),\]

which does not change when any other token of the context is changed.

Proof. The normalizer of (9.4) cancels. ∎

Induction asks for the opposite: of two candidate sources it must prefer the one whose predecessor is the present token, and exchanging the two predecessors exchanges the right answer while the ratio stays. A one-layer head can copy a token it has seen, and cannot use what stood before it to choose it.

Proposition 9.3.7 (the minimal induction circuit). Let \(p\) be a head of layer 1 that attends from every position to the one before it, \(A^p_{j, j-1} = 1\), and \(h\) a head of layer 2. The K-composition term of \(h\) through \(p\) in (9.6) is

\[\begin{aligned} \sum_m A^p_{jm}\; x_i^{0\top}\, W_{QK}^h W_{OV}^p\; x_m^0 &= M[t_i, t_{j-1}], \\ M &= W_E^{\top} W_{QK}^h W_{OV}^p W_E . \end{aligned}\]

If \(M\) is large on its diagonal and small off it, the term is large exactly at the sources \(j\) whose predecessor carries the present token \(t_i\). If also \(C_{OV}^h\) is copying, the head raises the logit of \(t_j\), the token that followed the earlier occurrence of \(t_i\).

The paper puts the mechanism in two sentences.

The central trick to induction heads is that the key is computed from tokens shifted one token back. The query searches for "similar" key vectors, but because keys are shifted, finds the next token.

—Nelson Elhage and others, "A mathematical framework for transformer circuits" (2021). The section "How induction heads work", first paragraph.

Both conditions can be tested on the weights: an induction head has \(\mathrm{pos}(C_{OV}^h)\) and \(\mathrm{pos}(M)\) both near one, the second computed from the \(d_h \times d_h\) matrix \(W_K^h W_{OV}^p W_E W_E^{\top} W_Q^{h\top}\), and in the paper's two-layer model the heads in that corner of the plane are its induction heads. With rotary positions \(W_{QK}^h\) in (9.6) stands for \(W_Q^{h\top} R^{\,j-i}\, W_K^h\), and \(M\) is one table for each offset.

The induction examples use a second two-layer model: the recipe and stories of Example 9.2.5, with one training window in ten replaced by a block of 8 to 64 random tokens repeated to the end of the window. Trained on the stories alone, the two-layer model grows no usable induction circuit: its best prefix-matching score (Definition 9.3.8) is 0.13, and a row of 25 random tokens costs it 18.3 nats a token the second time against 18.8 the first. The text decides, and not by how often it repeats itself: gpt-pico never learns to copy from stories, and a model of its width with two layers fewer does from web text, which repeats no more often (Section 8).

the model: two layers, attention only, trained on stories with one window in ten of repeated random tokens
a row of 16 random tokens, given twice, the second copy under the first; the reader in brackets:
  first copy    start ...  Sunny   splas ...
  second copy         ... [Sunny]  -> ?
layer 1, head 4   every token reads the one before it (1.00 of its attention):
                    at splas it writes that Sunny came first
layer 2, head 8   the second Sunny finds that mark and gives 0.65 of its attention to splas:
                    the model predicts splas with probability 1.00
loss, nats a token          first copy   repeated half     (256 rows; a guess costs 8.15)
the model                   8.74         0.13
head 4 set to zero          8.95         8.71
head 4 set to its mean      8.68         8.67
K-composition less 0.0625: head of layer 1 (row) with head of layer 2 (column)
        1      2      3      4      5      6      7      8       previous-token score
  1     0.005  −0.007 −0.001 0.000  −0.003 0.001  −0.006 −0.013  0.06
  2     0.027  0.047  0.020  0.030  0.019  0.038  0.039  0.037   0.02
  3     −0.001 −0.021 −0.002 −0.011 −0.007 −0.008 −0.015 −0.017  0.08
  4     −0.013 0.081  −0.028 0.047  −0.016 0.040  0.075  0.068   1.00
  5     −0.002 −0.012 −0.004 −0.009 −0.007 −0.002 −0.014 −0.017  0.05
  6     −0.005 −0.018 −0.003 −0.013 −0.007 −0.011 −0.015 −0.020  0.04
  7     −0.005 −0.017 −0.001 −0.016 −0.006 −0.010 −0.016 −0.020  0.09
  8     0.014  0.032  0.015  0.028  0.017  0.030  0.029  0.029   0.04
  score 0.00   0.91   0.00   0.52   0.00   0.56   0.85   0.94    induction score
(the live figure needs JavaScript and loads about 0.7 MB of data)
The induction circuit of a two-layer model on 16 random tokens given twice; it was trained on stories with one window in ten of repeated random tokens. Green: the induction read, a token’s attention to the token after its earlier copy: that token’s level, the stripe of layer 2, head 8. Ink: layer 1, head 4, which reads the token before; the composition scores less the baseline. Point at a token; choose two heads in the matrix; cut the layer 1 head by zeros or its mean; take another row. One trained model; losses are means over 256 rows.

On a row of 16 random tokens given twice, the model trained with repeated windows exposes the two steps of induction. In layer 1, head 4 each token attends to its predecessor. In layer 2, head 8 a token in the second copy attends to the token that followed its first occurrence, and the model predicts that token. Over 256 rows the first copy costs 8.74 nats a token and the second 0.13. Setting layer 1, head 4 to zero raises the second-copy loss to 8.71 nats again, as Proposition 9.3.7 suggests.

Weights were the paper's evidence in 2021. Its sequel of 2022 looked for the same heads in models with feed-forward layers and many layers, where nothing can be multiplied out, and defined them by behavior.C. Olsson, N. Elhage, N. Nanda and others, "In-context learning and induction heads", Transformer Circuits Thread, 8 March 2022 (arXiv 2209.11895), "Key concepts". The heads form in a short window early in training, the phase change of Section 8, which also takes up how much of learning in context they carry in large models.

Formally, we define an induction head as one which exhibits the following two properties on a repeated random sequence of tokens:

Prefix matching: The head attends back to previous tokens that were followed by the current and/or recent tokens. That is, it attends to the token which induction would suggest comes next.

Copying: The head’s output increases the logit corresponding to the attended-to token.

—Catherine Olsson and others, "In-context learning and induction heads" (2022).

Definition 9.3.8 (scores of a head on activations). Let the input be a start token followed by \(L\) random tokens repeated \(r\) times, and \(A\) the pattern of a head on it. (i) The previous-token score is the mean over positions \(i\) of \(A_{i, i-1}\), on ordinary text. (ii) The prefix-matching score is the mean, over positions \(i\) in the second and later repeats, of \(\sum_{k \ge 1} A_{i,\, i - kL + 1}\): the attention from a token to the tokens that followed its earlier occurrences. (iii) The copying score, for a sample \(S\) of tokens and with \(\tilde C_{OV}\) the OV circuit with each column centered, is

\[\mathrm{copy}(h) = \frac{\sum_{a \in S} \bigl(\tilde C_{OV}[a, a]\bigr)_{+}}{\sum_{a \in S} \sum_{c \in S} \bigl(\tilde C_{OV}[c, a]\bigr)_{+}} \in [0, 1],\]

the share, among the sampled tokens, of the logit increases a token causes that go to itself; \((u)_{+} = \max(u, 0)\).The 2022 paper used \(L = 25\) and \(r = 4\). Its wording of prefix matching can point at either neighbor of the earlier occurrence, and its copying score has no formula: (ii) follows the paper's figure and Proposition 9.3.7, and (iii) is our formula for the same idea, on the weights. On the figure's rows, 16 tokens given twice, score (ii) reads lower: 0.85 for layer 2, head 8, against 0.94 with \(L = 25\) and \(r = 4\).

Quirk. In the code these are prefix_matching_score and previous_token_score and copying_score of circuits.py. The first adds the attention to the token after every earlier occurrence, so it can only grow with the number of repeats: that is the 0.85 against 0.94 of the sidenote, one head on two kinds of row.

Example 9.3.9 (the two-layer model). The setup: the two-layer model trained with repeated windows, one run; the composition scores of all 64 pairs of heads less the baseline 0.0625, as in Definition 9.3.3 and with the permuted product of Remark 9.3.5; the scores of Definition 9.3.8 with \(L = 25\) and \(r = 4\); the positivities of Proposition 9.3.7; the gains of the three orders of (9.5) on held-out stories; and the loss on 256 rows of 25 random tokens given twice.

The first copy of a row costs 8.75 nats a token and the second 0.096, its first token left out (0.43 with it). Five heads of the second layer have a prefix-matching score above 0.5 (heads 2, 4, 6, 7 and 8, from 0.52 to 0.94); the other three score 0.00. All five K-compose with one head of the first layer, head 4, by 0.040 to 0.081 over the baseline. The largest of the 64 scores is its pair with layer 2, head 2 (prefix matching 0.91; copying 0.27, against 0.04 for a head that raised tokens at random). Head 4 is the previous-token head on the rows, where it scores 1.00; on stories it scores 0.26, and the largest score there, 0.29, is head 7's, which K-composes with no head above the baseline.

Under the permuted product the five pairs stand 0.24 to 0.71 above that product's own baseline and every other pair falls below it, including those of two first-layer heads that the correct score puts above the baseline with every head of the second: the paper's first diagram over again. In the plane of Proposition 9.3.7 the five heads have an OV positivity of 0.92 to 0.99 and a positivity of \(M\) of 0.47 to 0.88, against 0.07 or less for the other three: set apart, and short of the corner. On stories the direct path gains 3.01 nats over a uniform guess, the 16 single heads a further 2.84 and the 64 virtual heads a further 0.62, against the paper's 1.8, 5.2 and 0.3. ◁

Proposition 9.3.7 names three paths into an induction head: its value reads the token at the attended position, its query reads the present token, and its key reads the previous-token head. Later work measured how much of the heads' effect those paths carry, by a method of Section 9.4: keep the named paths, give every other path an input drawn at random from the data, and see what share of the heads' effect on the loss remains. On a two-layer attention-only model with eight heads a layer, one previous-token head and two induction heads, the value path alone explained 64.1%, the query path 48.7%, the key path 53.3%, and the three together 28.2%. Admitting further paths at the same position, and matching a window of three tokens in place of one, brought the joint figure to 48.6% and 55.2%. Causal scrubbing of a model of the same kind gave 35% for the minimal hypothesis and 86% for a refined one.N. Goldowsky-Dill, C. MacLeod, L. Sato and A. Arora, "Localizing model behavior with path patching", arXiv 2304.05969 (2023), Tables 1 to 3. L. Chan, A. Garriga-Alonso, N. Goldowsky-Dill and others, "Causal scrubbing: a method for rigorously testing interpretability hypotheses", AI Alignment Forum, 3 December 2022. Neither model is the 2021 paper's, which has 12 heads a layer. The minimal story is right and explains about a third; the rest is longer matches and more paths. In our model it explains 85%: with only the three paths real the loss on the repeated tokens is 2.61 nats, against 0.096 for the whole model and 16.6 with the five induction heads cut off from the row. We read the difference as one of setting, which no run here tests: our model learned induction from rows of random tokens and is measured on them, where one token identifies its earlier place; the published models learned it from web text.

Circuits in a real model

gpt-pp has 12 blocks, each an attention layer of 12 heads followed by a feed-forward layer, and a vocabulary of 32,768. A table over its vocabulary has more than a billion entries, and an attention-only model of its shape would have \(13^{12} \approx 2.3 \times 10^{13}\) paths. The methods that replaced enumeration keep one fact from the framework, that the final stream is the sum of what each component wrote, and add one move: change what a component did on a chosen input, and watch the output.What the heads and the stream of a real model look like, as attention patterns, maps of heads and the geometry of the stream of a 40-layer model, is drawn in LLM internals specimen sheet.

In the code. That one move is a hook. gpt-pp computes its function on two paths: forward, one fused attention call a layer, is the path that was trained, and forward_explicit writes attention out and hands every intermediate tensor, under a name, to an object that may replace it and may keep it. The object is Hooks in gpt-pp's model.py.

class Hooks:
    """What forward_explicit does with each named activation: hand it to a function that may replace it, and
    keep it if it was asked for."""

    def __init__(self, fns: Optional[dict[str, HookFn]] = None, keep: Optional[Callable[[str], bool]] = None):
        self.fns = fns or {}
        self.keep = keep
        self.cache: dict[str, Tensor] = {}

    def wants(self, name: str) -> bool:
        return name in self.fns or (self.keep is not None and self.keep(name))

    def __call__(self, name: str, x: Tensor) -> Tensor:
        fn = self.fns.get(name)
        if fn is not None:
            replacement = fn(x)
            if replacement is not None:
                x = replacement
        if self.keep is not None and self.keep(name):
            self.cache[name] = x
        return x

The names are given where the tensors are made. Attention on the explicit path is CausalSelfAttention.forward_explicit:

def forward_explicit(self, x: Tensor, rot: Rotation, bias: Tensor, hook: Hooks, layer: int,
                     kv: Optional[KVCache] = None) -> Tensor:
    ...
    B, T, _ = x.shape
    q, k, v = self.project(x)
    q = hook(f"attn.{layer}.q", q)
    k = hook(f"attn.{layer}.k", k)
    v = hook(f"attn.{layer}.v", v)
    q, k = self.position(q, k, rot)
    if kv is not None:
        k, v = kv.extend(layer, k, v)
    k, v = self.share(k, v)
    scores = q @ k.transpose(-2, -1) / math.sqrt(self.d_head) + bias  # (B, H, T, T_keys)
    scores = hook(f"attn.{layer}.scores", scores)
    pattern = hook(f"attn.{layer}.pattern", scores.softmax(dim=-1))  # each row sums to one
    z = pattern if self.dropout == 0 else F.dropout(pattern, self.dropout, self.training)
    z = hook(f"attn.{layer}.z", z @ v)  # (B, H, T, dh): each head's weighted average of values
    if hook.wants(f"attn.{layer}.result"):
        # What each head writes into the residual stream, (B, H, T, d): the output matrix applied to z one
        # head at a time. Their sum over heads (plus the bias) is attn_out. Recorded, not replaceable.
        w_o = self.o.weight.view(-1, self.n_head, self.d_head)
        hook(f"attn.{layer}.result", torch.einsum("bhtk,dhk->bhtd", z, w_o.to(z.dtype)))
    return self.o(z.transpose(1, 2).reshape(B, T, -1))

run_with_hooks hands in functions by name and run_with_cache also says which names to keep; both run this path and return the logits. In the listing scores holds the scores with the causal mask added, pattern is \(A^h\) for every head of the layer, of shape (batch, heads, positions, positions), z is each head's weighted average of values before \(W_O\), and result is what each head writes to the stream, an \(o_i^c\) of the definition below. Block.forward_explicit names the stream in the same way, resid_pre.l on entering a block, then attn_out.l, resid_mid.l, mlp_out.l and resid_post.l, and GPT.forward_explicit adds embed, final_norm and logits. Layers and heads are counted from zero in these names and from one in the text.

Quirk. attn.l.q and attn.l.k are the bare projections, taken before position normalizes and rotates them, so \(q k^{\top}/\sqrt{d_h}\) computed from a cache is not gpt-pp's score; attn.l.scores is, with the mask's \(-\infty\) in it and, in the variant that has one, the bias on distance. Where heads share keys and values, k and v have one slice for each key-value head, and in a gated feed-forward unit mlp.l.pre is the gate's input. attn.l.result is computed only when it is asked for and cannot be replaced: a function hooked on it runs and its return value is dropped. Cached tensors keep their gradient history, which is why the tools below run under torch.no_grad().

Quirk. One tool depends on that order. The path patching behind the 85% of Section 9.3, patch_inputs, projects a stream assembled by the caller and swaps the result in at attn.l.q, attn.l.k or attn.l.v of chosen heads; the model then applies its normalization and rotation to what was swapped in, as to its own.

Quirk. Every activation of this part is a number of the explicit path, not of the path that was trained. The two are one function, held equal by a test to \(10^{-5}\) in the logits, in 64-bit floats, on 31 variants of the model (Section 3); under the 16-bit floats of training (bfloat16) they differ by that format's rounding, and a test on a tiny model allows 0.15 between their logits (Section 7). The patches, the lens and the scores of the circuit models run the model in 32-bit floats, and the dictionaries of Section 10 read their activations in 16-bit. What is quoted beside the activations comes from forward: the loss of an untouched model, the probabilities 0.86 and 0.66 of Example 9.4.4, and the logits against which Example 9.2.10 checks its sum of paths.

Definition 9.4.1 (direct logit attribution). Let the stream before the unembedding be \(x_i^L = x_i^0 + \sum_c o_i^c\), the sum running over the heads and the feed-forward layers, or their neurons, and let the logits be \(W_U\, x_i^L / \rho_i\), with \(\rho_i\) the scale of the final normalization on this input (\(\rho_i = 1\) in a bare model). For tokens \(w\) and \(w^{\prime}\), the direct logit attribution of component \(c\) is

\[\mathrm{DLA}_c = \bigl(W_U[w] - W_U[w^{\prime}]\bigr)^{\top} o_i^c \,/\, \rho_i ,\]

with \(W_U[w]\) the row of token \(w\). The attributions of all components and of the embedding sum to the difference between the logits of \(w\) and \(w^{\prime}\).

A difference of two logits is used because one logit is defined only up to a constant. The attribution is the last edge of every path and no more:A component can matter by preparing information that a later component uses. Its own direct contribution to the final logit may be small or zero, even when removing it would break the computation. a head that matters by steering a later head has none.

In the code. The shares of every head and feed-forward layer come from one pass that records what each wrote at the position. It is direct_attribution in gpt-pp's interp/patching.py.

@torch.no_grad()
def direct_attribution(model: GPT, idx: Tensor, a: int, b: int, pos: int = -1) -> dict:
    ...
    idx = idx.view(1, -1)
    L = model.cfg.n_layer
    names = ["embed", "final_norm"] + [f"attn.{l}.result" for l in range(L)] + [f"attn_out.{l}" for l in range(L)]
    names += [f"mlp_out.{l}" for l in range(L)] if model.cfg.use_mlp else []
    names += [f"resid_post.{L - 1}"]
    _, cache = model.run_with_cache(idx, names=names)
    u, constant, rho = frozen_readout(model, cache[f"resid_post.{L - 1}"][0, pos], a, b)
    heads = torch.stack([cache[f"attn.{l}.result"][0, :, pos].float() @ u for l in range(L)])          # (L, H)
    attn = torch.stack([cache[f"attn_out.{l}"][0, pos].float() @ u for l in range(L)])                 # (L,)
    mlp = torch.stack([cache[f"mlp_out.{l}"][0, pos].float() @ u for l in range(L)]) if model.cfg.use_mlp else torch.zeros(L)
    embed = float(cache["embed"][0, pos].float() @ u)
    bias = float((attn - heads.sum(1)).sum()) + constant             # attention's output bias, if the model has one
    total = embed + float(heads.sum()) + float(mlp.sum()) + bias
    return {"heads": heads.cpu(), "mlp": mlp.cpu(), "embed": embed, "bias": bias, "total": total, "rho": rho,
            "ld": float(cache["final_norm"][0, pos].float() @ answer_direction(model, a, b))}

a and b are the tokens \(w\) and \(w^{\prime}\), and u is \((W_U[w] - W_U[w^{\prime}])/\rho_i\): frozen_readout measures \(\rho_i\) as the root mean square of the last resid_post at the position. heads has one share for each layer and head, read from attn.l.result, and mlp one for each layer; no share is computed for a neuron. total is their sum and ld the difference of logits computed directly from final_norm.

Quirk. The scale \(\rho_i\) is frozen at its value on the prompt, which is what makes the difference linear in the stream: a test holds total equal to ld to rounding. The embedding's share is the hook embed, the embedding after its own normalization. A model with post-norm, or with the dynamic tanh in place of a norm, is refused with NotImplementedError.

Definition 9.4.2 (activation patching). Fix a clean input, a corrupted input that differs from it in a controlled way, and a metric; here the difference \(\mathrm{LD}\) between the logits of the clean answer and of the corrupted answer at the last position. A node is the stream entering one block at one position, or the output of a head or of a feed-forward layer. Denoising a node is running the model on the corrupted input with the node's value overwritten by its value on the clean input; noising is the same with the two inputs exchanged. The effect is reported as

\[\frac{\mathrm{LD}_{\text{patched}} - \mathrm{LD}_{\text{corrupted}}}{\mathrm{LD}_{\text{clean}} - \mathrm{LD}_{\text{corrupted}}}, \tag{9.7}\]

which is 0 for a run that behaves as the corrupted one and 1 for a run that behaves as the clean one.The method has three sources: J. Vig, S. Gehrmann, Y. Belinkov and others, "Causal mediation analysis for interpreting neural NLP: the case of gender bias", arXiv 2004.12265 (2020); A. Geiger, H. Lu, T. Icard and C. Potts, "Causal abstractions of neural networks", NeurIPS 2021 (arXiv 2106.02997); and K. Meng, D. Bau, A. Andonian and Y. Belinkov, "Locating and editing factual associations in GPT", NeurIPS 2022 (arXiv 2202.05262).

In the code. The metric, and the map of one site in one direction: denoising is patch_map(model, clean, corrupt, ...), and noising is the same call with the two prompts exchanged. They are logit_difference and patch_map in patching.py.

@torch.no_grad()
def logit_difference(model: GPT, idx: Tensor, a: int, b: int, hooks: Optional[dict] = None, pos: int = -1) -> Tensor:
    """logit(a) - logit(b) at position `pos` of each row of idx (B, T), before the cap on the logits, (B,).
    `hooks` are applied as in GPT.run_with_hooks, so this is also the metric of a patched run."""
    _, cache = model.run_with_cache(idx, names=["final_norm"], hooks=hooks)
    return cache["final_norm"][:, pos].float() @ answer_direction(model, a, b)


@torch.no_grad()
def patch_map(model: GPT, source: Tensor, target: Tensor, a: int, b: int, site: str = "resid_pre",
              batch: int = 64) -> Tensor:
    ...
    source, target = source.view(1, -1), target.view(1, -1)
    if source.shape != target.shape:
        raise ValueError(f"the two prompts must have the same number of tokens: {source.size(1)} and {target.size(1)}")
    L, T = model.cfg.n_layer, source.size(1)
    names = [f"{site}.{l}" for l in range(L)]
    _, cache = model.run_with_cache(source, names=names)
    cells = [(l, t) for l in range(L) for t in range(T)]
    out = torch.empty(L * T)
    for start in range(0, len(cells), batch):
        chunk = cells[start:start + batch]
        hooks = {}
        for l in sorted({l for l, _ in chunk}):
            rows = torch.tensor([r for r, (cl, _) in enumerate(chunk) if cl == l])
            at = torch.tensor([t for cl, t in chunk if cl == l])

            def overwrite(x: Tensor, rows: Tensor = rows, at: Tensor = at, name: str = names[l]) -> Tensor:
                x = x.clone()
                rows, at = rows.to(x.device), at.to(x.device)
                x[rows, at] = cache[name][0, at].to(x.dtype)
                return x

            hooks[names[l]] = overwrite
        out[start:start + len(chunk)] = logit_difference(model, target.expand(len(chunk), -1), a, b, hooks).cpu()
    return out.view(L, T)

a and b are the clean and the corrupted answer, answer_direction returns \(W_U[a] - W_U[b]\), and the value returned by logit_difference is \(\mathrm{LD}\), one for each row of the batch. patch_map records the node on source once and then runs target in batches in which each row is one cell: overwrite writes the recorded value into its rows at their positions and leaves the others alone, so a map costs a forward pass for every 64 cells and not for every cell. The ratio (9.7) is taken afterwards, in patching_effects.

Quirk. logit_difference does not read the model's logits. Those pass through a cap, \(15\tanh(z/15)\) (Section 2), on forward and at the hook logits; the metric takes the hook final_norm and multiplies by \(W_U[a] - W_U[b]\) itself, so it is a difference of logits before the cap, where it is linear in the normalized stream. The +7.90 and −6.31 of Example 9.4.4 and the shares of direct_attribution are in those units. The probabilities of the same example are the model's own, after the cap, and the lens of Definition 9.4.7 applies the cap as well.

Known issue. The sites are three, the stream entering a block (resid_pre), a layer's whole attention output (attn_out) and its feed-forward output (mlp_out), each at one position; no head is patched alone, though the definition names the output of a head, so heads enter Example 9.4.4 through their attributions only. The two prompts must have the same number of tokens, and a map is one pair of prompts, not a mean over a task. gpt-pp has no code for the attribution patching of Proposition 9.4.5, for the three criteria of Definition 9.4.6 or for the edit of Proposition 9.5.1: those are stated here from the papers.

Example 9.4.3 (two gates). Let the output be \(C = A \wedge B\), with \(A = B = 1\) on the clean input and \(A = B = 0\) on the corrupted one. Denoising \(A\) alone gives \(1 \wedge 0 = 0\), no effect; noising \(A\) alone gives \(0 \wedge 1 = 0\), the whole effect. For \(C = A \vee B\) it is the reverse. Denoising finds what is sufficient to restore the behavior and is blind to one part of a conjunction; noising finds what is necessary and is blind to a redundant part. ◁

A guide to the method warns that "these two directions can be very different, and are not just symmetric mirrors of each other".S. Heimersheim and N. Nanda, "How to use and interpret activation patching", arXiv 2404.15255 (2024), Section 2.3. On corruption and metric: F. Zhang and N. Nanda, "Towards best practices of activation patching in language models: metrics and methods", ICLR 2024 (arXiv 2309.16042). Two more choices decide what an experiment finds. The corruption: Gaussian noise on some tokens' embeddings came first; replacing those tokens by others of the same kind keeps every activation on the model's own distribution, and is now preferred. Zeros are the crudest choice: in the two-layer model of Example 9.3.9, on rows of 16 random tokens given twice, setting layer 1, head 2 to zero leaves the second copy at 7.3 nats a token and setting it to its mean at 1.9, against 0.13 for the model untouched. The metric: a probability saturates, while a difference of logits is close to linear in the final stream and registers a component that pushes against the answer.

Quirk. In the code an ablation acts on attn.l.z, a head's output before \(W_O\): ablation_hooks replaces it by zero or by its mean over the given rows, position by position, the two modes behind the 7.3 and 1.9 nats above. Every later layer then recomputes its attention from the changed stream, unless the recorded patterns are passed too and forced.

Example 9.4.4 (where the answer is restored in gpt-pp). The setup: gpt-pp; the clean prompt "In France, the official language is", which the model continues with “ French” (probability 0.86), and the corrupted prompt "In Spain, the official language is", which it continues with “ Spanish” (0.66); the metric (9.7), before the model's cap on its logits; denoising of the stream entering each of the 12 blocks at each of the seven positions; and the direct logit attribution of every head and feed-forward layer on the clean prompt. Activations are counted from zero, so resid_pre.6 is the stream entering block 7. One cell of the map, that stream at “ France”, as a usage example written for this page against the model's run_with_cache and run_with_hooks (it is not gpt-pp's code, which is patch_map above, with overwrite in the place of restore):

_, clean = model.run_with_cache(clean_ids, names=["resid_pre.*"])
def restore(x, name="resid_pre.6", pos=2):   # one cell of the map
    x = x.clone(); x[:, pos] = clean[name][:, pos]; return x
patched = model.run_with_hooks(corrupt_ids, {"resid_pre.6": restore})

The logit difference is +7.90 on the clean prompt and −6.31 on the corrupted one. The clean stream at “ France” restores nearly all of the difference up to block 9, 83% at block 10 and 41% at block 11, where the clean stream at the last position restores 58%: between them attention carries the answer across. The attributions name the heads. Block 10, head 1 adds +2.81 of the +7.90 with 99% of its attention on “ France”, and block 11, head 1 adds +1.60; the heads together add +4.68 and the feed-forward layers +3.21, nearly all of it the last block's. ◁

clean      In France, the official language is → French   (p 0.86)
corrupted  In Spain, the official language is → Spanish   (p 0.66)
logit of “ French” less logit of “ Spanish” at the last position: +7.90 clean, −6.31 corrupted

per cent of that difference restored when the clean stream entering a block is written into the corrupted run at one position
block        In   France        ,      the official language       is
   12         0        5        0        0        0        1       93
   11         0       41        1        0        0        1       58
   10         0       83        0        0        0        0       13
    9         0       98        0        0        0        0        0
    8         0       99        0        0        0        0       -1
    7         0       99        0        0        0       -1        0
    6         0      100        0        0        0       -1       -1
    5         0      100        0        0        0       -1       -1
    4         0      100        0        0        0       -2        0
    3         0      100        0        0        0       -1        0
    2         0      100        0        0        0        0        0
    1         0      100        0        0        0        0        0

largest direct effects on the difference, clean prompt: block 10, head 1 +2.81 (attends 99% to “ France”); block 11, head 1 +1.60 (attends 87% to “ France”)
all heads +4.68, all MLPs +3.21, the embedding +0.00: in all +7.90
(the live figure needs JavaScript and loads about 0.12 MB of data)
Activation patching on gpt-pp. Two prompts differ in one token and in their answer; the metric is the logit of the clean answer less that of the corrupted one at the last position, before the cap on the logits. A cell is one node (the stream entering a block, or its attention or MLP output, at one position) whose clean value is written into the corrupted run. Green: the share of the logit difference the cell restores. Beside it: each head’s and MLP’s direct effect on the clean prompt; pick a head for its attention. One model, 6 pairs, measured.

We repeat the interventions on six pairs of prompts: a fact, agreement in number, a pronoun, a repeated name, an open bracket and a tense. These are single pairs, not averages over a task. In all six the answer sits at the changed token up to block 9, 10 or 11 and at the last position one or two blocks later. Noising tells the same story for the fact and not for agreement in number, where the first block's attention output at the changed token destroys 66% of the difference when noised and restores nothing when denoised: necessary on the clean run, not sufficient on the corrupted one, the conjunction of Example 9.4.3.

Patching a node changes every path through it. Path patching changes paths of one kind: chosen receivers (the queries, keys or values of later heads, or the final stream) get the input they would have had if one sender's output, carried to them through no other head, had come from the other prompt; in a bare model with frozen patterns, patching into the key of one head edits exactly the terms of (9.6) in which the sender stands on the key side. A hypothesis is a set of paths, and its proportion explained is one minus the mean change of the metric when only the paths outside it receive a random other input, over the mean change when the whole model does: the percentages of Section 9.3.

Patching costs one forward pass for each node. Differentiating cuts the cost to a constant.

Proposition 9.4.5 (attribution patching is a first-order expansion). Fix an input and a node, and let \(F(\nu)\) be the metric as a function of the node's value, the rest of the computation being as on that input. If \(F\) is twice continuously differentiable on the segment from the node's own value \(\nu\) to the patched value \(\nu^{\prime}\), then for some \(\xi\) on the segment

\[F(\nu^{\prime}) - F(\nu) = (\nu^{\prime} - \nu)^{\top}\, \nabla F(\nu) + \tfrac12\, (\nu^{\prime} - \nu)^{\top}\, \nabla^2 F(\xi)\, (\nu^{\prime} - \nu). \tag{9.8}\]

The first term is the attribution-patching estimate. It is exact when \(F\) is affine on the segment.

Proof. Taylor's theorem with Lagrange's remainder for \(s \mapsto F(\nu + s(\nu^{\prime} - \nu))\) on \([0, 1]\). ∎

The gradients with respect to all activations come from one backward pass, which is the point of the note that introduced the method.

To compute this, we take a backwards pass on the corrupted prompt with respect to the patching metric, and cache all gradients with respect to the activations. Importantly, we are doing a weird thing, and not just taking the gradients with respect to parameters! Then, for a given activation we can compute ((clean_act - corrupted_act) * corrupted_grad_act).sum(), where we do elementwise multiplication and then sum over the relevant dimensions. Every single patch can be computed after caching activations from two forward passes and one backwards pass.

—Neel Nanda, "Attribution patching: activation patching at industrial scale" (2023). A note on the author's site, 4 February 2023. The repair: J. Kramár, T. Lieberum, R. Shah and N. Nanda, "AtP*: an efficient and scalable method for localizing LLM behaviour to components", arXiv 2403.00745 (2024).

The estimate is exact for a difference of logits as a function of a last-layer head's output in a bare model. It fails where a function saturates. Take an attention weight that is a softmax over two keys, \(\sigma(g)\) in the gap \(g\) between their scores, and a patch that moves the gap from \(+10\) to \(-10\). The weight changes by \(-0.9999\); the first-order estimate is \(-20\,\sigma^{\prime}(10) = -0.0009\), and the node looks a thousand times less important than it is. The repair is to recompute the softmax exactly for a patched query or key and linearize only what follows.

Search can then be automated. One algorithm removes, from the output backward, every edge whose patching moves the output distribution by less than a threshold in relative entropy; on GPT-2 small it kept 68 of 32,000 edges for one task, all found by hand in earlier work. Its successors score every edge at once by (9.8).A. Conmy, A. N. Mavor-Parker, A. Lynch, S. Heimersheim and A. Garriga-Alonso, "Towards automated circuit discovery for mechanistic interpretability", NeurIPS 2023 (arXiv 2304.14997), abstract and Algorithm 1. The successors: A. Syed, C. Rager and A. Conmy, "Attribution patching outperforms automated circuit discovery", NeurIPS 2023 ATTRIB workshop (arXiv 2310.10348); M. Hanna, S. Pezzelle and Y. Belinkov, "Have faith in faithfulness: going beyond circuit overlap when finding model mechanisms", COLM 2024 (arXiv 2403.17806).

The best known circuit found by hand, in 2022, is for indirect object identification in GPT-2 small: in "When Ann and Bob went to the park, Bob gave the ball to", the next word should be Ann. The explanation has 26 attention heads in seven classes, found by causal interventions.K. Wang, A. Variengien, A. Conmy, B. Shlegeris and J. Steinhardt, "Interpretability in the wild: a circuit for indirect object identification in GPT-2 small", arXiv 2211.00593 (2022), abstract and Sections 4 and 4.1. The example sentence is ours. Its algorithm: find every name in the sentence, strike out the one that occurs twice, output the one that is left. Heads that detect the repeated name write to heads at the last position, which change the queries of the name movers so that they attend to the remaining name and copy it: V-composition, then Q-composition, then a copying OV circuit. The study also introduced three criteria for such an explanation.

Definition 9.4.6 (faithfulness, completeness, minimality). Let \(F(C)\) be the mean of the metric over the inputs of the task when every component outside the set \(C\) is replaced by its mean activation on a reference distribution, and let \(M\) be the whole model. A circuit \(C\) is faithful if \(\lvert F(M) - F(C) \rvert\) is small; complete if \(\lvert F(C \setminus K) - F(M \setminus K) \rvert\) is small for every \(K \subseteq C\); and minimal if for every \(v \in C\) there is a \(K \subseteq C \setminus \{v\}\) for which \(\lvert F(C \setminus (K \cup \{v\})) - F(C \setminus K) \rvert\) is large.

Faithfulness alone is satisfied by one of two redundant copies of a mechanism. The study reports \(F(M) = 3.56\) and \(\lvert F(M) - F(C) \rvert = 0.46\), so the circuit carries 87% of the logit difference; and it reports that a greedy search found sets \(K\) with a completeness gap as large as 3.09, itself 87% of \(F(M)\). The circuit is faithful and far from complete:Preserving most of the original answer is one test. Accounting for how the full model responds when parts are removed is a stronger one: a smaller circuit can reproduce ordinary behavior while leaving backup mechanisms unexplained. the model has backup name movers that take over when the name movers are removed, and negative name movers that write against the answer. gpt-pp does not do the task: of three prompts of this form it answered none clearly, the best separating its two answers by +0.65 and −1.04 logits against the two logits asked of the pairs of Example 9.4.4.

The strictest test in use generalizes this. A hypothesis is a correspondence between a small graph that computes the behavior and the model's own graph; the test replaces every activation by its value on another input that the hypothesis calls equivalent, and whatever it calls irrelevant by its value on a random input, and reports the share of performance that survives. Its authors put the idea in one sentence of their summary.

Summary: This post introduces causal scrubbing, a principled approach for evaluating the quality of mechanistic interpretations. The key idea behind causal scrubbing is to test interpretability hypotheses via behavior-preserving resampling ablations. We apply this method to develop a refined understanding of how a small language model implements induction and how an algorithmic model correctly classifies if a sequence of parentheses is balanced.

—Lawrence Chan and others, "Causal scrubbing: a method for rigorously testing interpretability hypotheses" (2022). L. Chan, A. Garriga-Alonso, N. Goldowsky-Dill and others, AI Alignment Forum, 3 December 2022, the summary at the head of the post.

Resampling keeps every activation on the model's own distribution.Each replacement activation comes from a real run. The combination need not: independently patched components may never have occurred together on any input. Resampling addresses one source of artificiality, while the intervention still needs a stated hypothesis. The 35% and 86% of Section 9.3 are scores of this kind.

A cheaper instrument reads the stream directly.

GPT's probabilistic predictions are a linear function of the activations in its final layer. If one applies the same function to the activations of intermediate GPT layers, the resulting distributions make intuitive sense.

This "logit lens" provides a simple (if partial) interpretability lens for GPT's internals.

—nostalgebraist, "interpreting GPT: the logit lens" (2020). LessWrong, 31 August 2020, "overview". The tuned lens: N. Belrose, I. Ostrovsky, L. McKinney and others, "Eliciting latent predictions from transformers with the tuned lens", arXiv 2303.08112 (2023).

Definition 9.4.7 (lenses). The logit lens after block \(\ell\) is the model's own readout applied early: \(\mathrm{LL}(x_i^\ell) = \mathrm{softmax}\bigl(W_U\, N(x_i^\ell)\bigr)\), with \(N\) the final normalization and, in gpt-pp, the cap on the logits before the softmax. The tuned lens puts an affine map before it, one for each block, \(\mathrm{TL}_\ell(x) = \mathrm{LL}(A_\ell\, x + b_\ell)\), which starts at the identity and is fitted, with the model frozen, to minimize the expected relative entropy from the model's final distribution to the lens's.

In the code. The logit lens is two short functions: one records the stream at every depth, and the other is the model's readout, applied to whatever stream it is given. They are streams and readout in gpt-pp's interp/lens.py.

@torch.no_grad()
def streams(model: GPT, idx: Tensor) -> Tensor:
    """The residual stream of idx (B, T) at every depth, (L + 1, B, T, d)."""
    names = depth_names(model)
    _, cache = model.run_with_cache(idx, names=names)
    return torch.stack([cache[name] for name in names])


def readout(model: GPT, x: Tensor) -> Tensor:
    """Log-probabilities (..., V) of the model's own readout applied to a stream x (..., d): the final
    normalization, the unembedding, the soft-cap on the logits when the model has one, the log-softmax."""
    return F.log_softmax(model.unembed_logits(model.final_norm(x)), dim=-1)

model.final_norm is \(N\), and model.unembed_logits multiplies by \(W_U\) and applies the cap, so at the last depth readout returns the model's own distribution. The tuned lens calls the same readout on \(A_\ell\, x + b_\ell\).

Quirk. Depth 0 is the hook embed, the stream after the embedding's own normalization, and depth \(\ell\) is resid_post of block \(\ell\); the relative entropies of Example 9.4.8 run from the model's final distribution to the lens's, in bits. The tuned lens is fitted by fit_tuned_lens, 200 steps of Adam on eight windows a step, its map applied to the stream before the final normalization and each bias learned in units of its depth's root mean square; until commit 702f0af the map acted on the normalized stream. The units are there because the stream grows: its root mean square is 1.0 after the embedding, 99 after block 1, 1,072 after block 11 and 4,568 after block 12, the factor 4.3 of Example 9.4.8.

The logit lens is the model with every later block's contribution set to zero. It is biased: in GPT-Neo-2.7B the marginal distribution of the tokens it predicts is 4 to 5 bits from the model's own at most layers, and in gpt-pp 3.2 bits after block 1 and 1.5 after block 11. The tuned lens removes that bias and follows the drift of the stream's basis from block to block.

Example 9.4.8 (when the answer appears in gpt-pp). The setup: gpt-pp on "The first president of the United States was George", nine tokens, after which it predicts “ Washington” with probability 0.96; the logit lens after the embedding and after every block; and, over 8,192 held-out positions, the relative entropy from the model's final distribution to that of the logit lens and of a tuned lens fitted on other held-out text.The held-out positions are the first 128 tokens of 64 documents. The tuned lens was fitted on 204,800 further positions for 200 steps; at the early blocks its loss was still falling slowly, so its figures there are upper bounds.

The lens has the token before it has the probability. At the last position “ Washington” is its first choice from block 9 on, with probability 0.002 there and 0.005 after block 11; the model's 0.96 appears in the last block. Over the nine positions the median block from which the first choice stays final is 9 of 12. On held-out text the first choice is already the final one at 17% of positions after block 1, 24% after block 6, 46% after block 9 and 64% after block 11, and the lens is 9.51 bits from the final distribution after block 1 and still 4.12 after block 11: the last block multiplies the root mean square of the stream by 4.3, and so writes most of what the readout sees. Through the tuned lens the same two figures are 4.61 and 0.68 bits, and “ Washington” has 0.57 at block 9. In this recipe the logit lens can be read for the token, late, and not for the probability; the tuned lens is what reads the middle of the model. ◁

the logit lens on gpt-pp: its own readout, applied after the embedding (emb) and after each of its 12 blocks
passage  The first president of the United States was George   (9 tokens; the model then predicts “ Washington”, p 0.96)

block  at “ States”     at “ was”        at “ George”     p of the final token at “ George”
 12     ,                born             Washington      0.96
 11     ,                born             Washington      0.005
 10     ,                born             Washington      0.001
  9     ,               [born]           [Washington]     0.002
  8     ,                a                ,               0.001
  7     ,                a                ,               0.001
  6     ,                ,                ,               0.001
  5     ,                ,                ,               0.001
  4     ,                ,                ,               under 0.001
  3     ,                ,                ,               under 0.001
  2    [,]               ,                ,               under 0.001
  1     man              a                ation           0.001
 emb    ,                a                Washington      0.002
       [ ] the first block from which the lens’s top token stays the model’s final prediction

8,192 held-out positions, after block      1      6     11
bits from the final distribution      9.5    8.6    4.1
the same through a tuned lens         4.6    3.1    0.7   (an affine map a block, fitted to the final distribution)
top token already the final one       17%    24%    64%
(the live figure needs JavaScript and loads about 0.09 MB of data)
The logit lens on gpt-pp: the model’s readout applied after the embedding and after each of its 12 blocks. A cell holds the lens’s top token, in ink where it is the model’s final prediction there; the shade is the probability the lens gives that prediction, or its rank. Green: in each column, the first block from which the top token stays the final prediction. Right: bits from the final distribution over 8,192 held-out positions, dashed through a tuned lens, and how often the top token is already final. Pick a passage; point at a cell. Measured on one model.

After the embedding alone the lens already says “ Washington” after “ George”, loses it at block 1 and has it back at block 9. The initial answer comes from the direct path of Section 9.2: the pair is a bigram it holds.

Remark 9.4.9 (confusion: what a lens does not show). A lens answers one question: if the computation stopped here, what would the readout say? It does not show how the answer was computed. It does not show what the stream holds in another basis: by Proposition 9.2.3 nothing obliges an early block to keep information where the unembedding can read it, and an empty lens is not an empty stream. And a trained lens can compute as well as read, which is why the tuned lens is kept affine. A variant of 2026 replaces the trained map by the average Jacobian of the final stream with respect to the stream after block \(\ell\).W. Gurnee and others, "Verbalizable representations form a global workspace in language models", Transformer Circuits Thread, July 2026.

Feed-forward layers in circuits

The framework set the feed-forward layers aside, and they hold two thirds of a transformer's parameters. Section 5 makes them readable on one side: a layer is a sum over its neurons, \(\mathrm{FFN}(x) = \sum_r m_r(x)\, v_r\), with a coefficient \(m_r(x) = \phi(k_r^{\top} x)\) that says how well the stream matches the key \(k_r\), and a value \(v_r\) that is written to the stream. In the terms of this part a neuron's direct logit attribution is a fixed column \(W_U v_r\) over the vocabulary, in the role of an OV circuit, times a coefficient that depends on the context. The activation \(\phi\) is in the role of the attention pattern: hold fixed which neurons fire and with what slope, and the layer is linear, which is how Section 10 carries the path expansion through these layers. And because \(\phi\) acts coordinate by coordinate, the neurons, unlike the stream, are a privileged basis: the first circuits through these layers were stated in neurons or in whole layers.

Quirk. Where Section 5 reads gpt-pp's neurons in this way, the coefficient is the hook mlp.l.post and a value's tokens are ranked by the column \(W_U v_r\) as it stands, without the final normalization and before the cap: the right ranking for a norm without a gain, which is the recipe's, and one that does not cover gated units.

Knowledge neurons are the first kind: the answer to a factual prompt is attributed to single neurons by integrated gradients, 4.13 of them a fact on average; suppressing them lowers the probability of the correct answer by 29.03% on average, against 1.47% for other neurons.D. Dai, L. Dong, Y. Hao and others, "Knowledge neurons in pretrained transformers", ACL 2022 (arXiv 2104.08696), Sections 4.4 and 4.5.

Causal tracing is the second, and it is activation patching under its first name: corrupt the embeddings of the subject's tokens with Gaussian noise of three times their standard deviation, then restore one hidden state at a time from the clean run. In GPT-2 XL, over 1,000 facts, the corruption removes 18.6% of the object's probability on average; restoring a single state recovers most, 8.7%, at layer 15 at the last token of the subject; restoring only the feed-forward outputs there recovers 6.6% and only the attention outputs 1.6%.Meng et al. (2022), Section 2 and Figure 2. The edit that follows is the paper's ROME, rank-one model editing. The reading: a feed-forward layer in the middle of the network retrieves the fact at the subject's last token, and attention carries it to the end of the prompt. The same paper turned the reading into an operation, and it begins from a view of the layer that it sets against Section 5's.

Locating and Editing Factual Associations in GPTNeurIPS 2022 · arXiv:2202.05262K. Meng, D. Bau, A. Andonian and Y. Belinkov

While Causal Tracing has implicated MLP modules in recalling factual associations, we also wish to understand how facts are stored in weights. Geva et al. 2021 observed that MLP layers (Figure 4cde) can act as two-layer key--value memories, where the neurons of the first layer \(\smash{W^{(l)}_{fc}}\) form a key, with which the second layer \(\smash{W^{(l)}_{proj}}\) retrieves an associated value. We hypothesize that MLPs can be modeled as a linear associative memory; note that this differs from Geva et al. 2021’s per-neuron view.

We view \(\smash{W^{(l)}_{proj}}\) as a linear associative memory (Kohonen 1972; Anderson 1972). This perspective observes that any linear operation \(W\) can operate as a key–value store for a set of vector keys \(K=[k_{1}\mid k_{2}\mid\dots]\) and corresponding vector values \(V=[v_{1}\mid v_{2}\mid\dots]\), by solving \(WK\approx V\), whose squared error is minimized using the Moore-Penrose pseudoinverse: \(W=VK^{+}\). Bau et al. 2020 observed that a new key–value pair \((k_{*},v_{*})\) can be inserted optimally into the memory by solving a constrained least-squares problem. In a convolutional network, Bau et al. 2020 solve this using an optimization, but in a fully-connected layer, we can derive a closed form solution:

\[\text{minimize}\;\lVert\hat{W}K-V\rVert\;\text{such that}\;\hat{W}k_{*}=v_{*}\quad\text{by setting}\;\hat{W}=W+\Lambda(C^{-1}k_{*})^{T}. \qquad (2)\]

Section 3.1, its first two paragraphs, to equation 2
K. Meng, D. Bau, A. Andonian and Y. Belinkov, "Locating and Editing Factual Associations in GPT" (NeurIPS 2022, arXiv 2202.05262), Section 3.1, its first two paragraphs, to equation 2. W with the subscript proj is a feed-forward layer's second matrix and W with fc its first; a key is that layer's activations at the last token of a subject. Geva et al. 2021 is the paper on key-value memories read in Section 5.

Proposition 9.5.1 (the rank-one edit). Let \(K \in \mathbb{R}^{m \times N}\) and \(V \in \mathbb{R}^{d \times N}\) hold keys and values in their columns, with \(C = KK^{\top}\) invertible, and let \(W\) minimize \(\lVert WK - V \rVert_F^2\). For a new pair \((k_{\ast}, v_{\ast})\) with \(k_{\ast} \ne 0\), the unique minimizer of \(\lVert \hat W K - V \rVert_F^2\) subject to \(\hat W k_{\ast} = v_{\ast}\) is

\[\hat W = W + \Lambda\, (C^{-1} k_{\ast})^{\top}, \qquad \Lambda = \frac{v_{\ast} - W k_{\ast}}{k_{\ast}^{\top} C^{-1} k_{\ast}} .\]

Proof. The problem is convex with an affine constraint, so a stationary point of the Lagrangian \(\tfrac12 \lVert \hat W K - V \rVert_F^2 - \lambda^{\top}(\hat W k_{\ast} - v_{\ast})\) is the minimizer. Stationarity in \(\hat W\) reads \((\hat W K - V) K^{\top} = \lambda\, k_{\ast}^{\top}\). Subtracting the normal equations of \(W\), \(W C = V K^{\top}\), gives \((\hat W - W)\, C = \lambda\, k_{\ast}^{\top}\), so \(\hat W = W + \lambda\, (C^{-1} k_{\ast})^{\top}\), and the constraint fixes \(\lambda = \Lambda\). ∎

The update has rank one, and multiplying the key by \(C^{-1}\) aims it away from the other keys: \(\hat W k - W k = \Lambda\, (k^{\top} C^{-1} k_{\ast})\) vanishes when \(k\) and \(k_{\ast}\) are orthogonal in the metric of \(C^{-1}\).A later method spreads thousands of edits over several layers by the same least squares: K. Meng, A. Sen Sharma, A. Andonian, Y. Belinkov and D. Bau, "Mass-editing memory in a transformer", arXiv 2210.07229 (2022). What repeated exact writes do to the facts a layer already holds is drawn in another essay's figure.

Two findings qualify the story. Where a fact is found and where it is best edited come apart: across layers the size of the tracing effect does not predict the success of an edit. And the localization depends on the corruption: with tokens replaced by other tokens and a difference of logits as the metric, the peak at the middle feed-forward layers is much weaker; in GPT-2 XL the peaks under noise are two to five times as high.P. Hase, M. Bansal, B. Kim and A. Ghandeharioun, "Does localization inform editing? Surprising differences in causality-based localization vs. knowledge editing in language models", NeurIPS 2023 (arXiv 2301.04213). Zhang and Nanda (2024), Section 3.1.

Limits

A circuit is a claim about which terms of a sum of paths matter. The framework writes the sum out exactly for small attention-only models; patching, attribution and scrubbing estimate its terms where it cannot be written out. Four limits have appeared, and the last decides what comes next.

Weights mislead. A study of 2026 expanded a one-layer model with a feed-forward layer into all of its virtual weights, 2.9 million parameters becoming about 331 million, and measured the effect of each on real data.

Expanding upon this result, we then present three major findings. First, helpful and harmful weights are scattered across the entire range of virtual weight magnitudes. Thus, naive attempts to read functional circuitry from raw weights can miss the ones that implement important circuits while highlighting confusing connections that never matter on real data. Second, ineffective virtual weights are abundant and can be easily removed: pruning the least effective 70% only worsens the model's loss by 0.01 nats, and pruning 85% costs 0.1. The most effective weights are overwhelmingly helpful, and they exceed the effectiveness of any harmful ones by an order of magnitude. Third, the number of helpful virtual weights in this basis still appears large. As a crude benchmark, our virtual weight model contains more helpful weights than parameters in the original transformer, even when constrained to the most effective and helpful. The remaining weights may still be more interpretable than the original model, but we suspect that a similar model within a better basis may still provide further sparsity and interpretability.

—Nicholas Turner, Jeffrey Wu and Joshua Batson, "Characterizing interference weights in a tiny language model" (2026). Transformer Circuits Thread, 21 August 2026, introduction.

Our one-layer model shows it in small: among the largest entries of an OV column are rare word pieces that other paths cancel, so the tables shown here are restricted to tokens seen at least 10,000 times in training.

Interventions depend on their design. The corruption, the kind of ablation and the metric change which circuit is found, and models repair themselves: when a component is removed, later ones make up part of its effect, as the backup name movers do.J. Miller, B. Chughtai and W. Saunders, "Transformer circuit faithfulness metrics are not robust", COLM 2024 (arXiv 2407.08734). T. McGrath, M. Rahtz, J. Kramár, V. Mikulik and S. Legg, "The Hydra effect: emergent self-repair in language model computations", arXiv 2307.15771 (2023).

Explanations are partial: in models trained on text the minimal induction circuit carries about a third of the induction heads' effect, and the indirect-object circuit is faithful without being complete.

And the units are wrong. Every circuit of this part took heads and neurons as its parts. A head named for one behavior on a narrow set of prompts does other things on broad data, and a neuron, which does have a privileged basis, usually answers to several unrelated things. The newest methods keep the framework's device and change the unit.

Freezing attention patterns is a standard approach which divides understanding transformers into two steps: understanding behavior given attention patterns, and understanding why the model attends to those positions. This approach was explored in depth for attention models in A Mathematical Framework, which also discussed a generalization to MLP layers which is essentially the approach used in this paper. Note that factoring out understanding attention patterns in this way leads to the issues with attention noted in § Limitations: Missing Attention Circuits. However, we can then also take the same solution Framework takes of studying QK circuits.

—Emmanuel Ameisen and others, "Circuit tracing: revealing computational graphs in language models" (2025). E. Ameisen, J. Lindsey, A. Pearce and others, Transformer Circuits Thread, 27 March 2025, the second paragraph of a footnote to "Linear attribution between features". On heads over the whole data distribution: R. Luger, H. Kamath, D. Finkbeiner and others, "HeadVis: an interactive tool for investigating attention heads", Transformer Circuits Thread, 4 May 2026.

What the new unit is, a direction in the stream that neither a head nor a neuron owns, and why a network would store more of them than it has dimensions, is where Section 10 begins.

← Back to all posts