11. Frontier, reading and recipe
Draft
What the field settled on
Every open model ships a configuration file: a few dozen lines that name its normalization, its activation, how it encodes position, how many heads share their keys, how many experts a token is routed to. Read side by side, these files are a census of design choices. We compare the transformer of 2017, GPT-2, and 34 open models released between April 2024 and October 2026, and locate the two recipes of gpt-pp among them.
table 36 open model families in order of release, June 2017 to October 2026, then the 2019
recipe and gpt-pp. One mark a row: . the design of 2017 to 2019 # what replaced it
/ a later departure ? not visible in the sources
the families, by release ours of the 24 since 2025
normalization ..#############.##################?# .# 22 RMSNorm
MLP activation ..################################?# ./ 23 gated MLP
position ..#############?########?######?#### .# 21 rotary positions
attention ..#.###########/####/###//###///##/# .. 16 shared keys and values
embeddings ..####..#.##.##..###.######.######.# .# 18 untied embeddings
dense or experts ....#.....#..##..#######.#########.# .. 19 mixture of experts
optimizer .?.?.??.?..?????.##?????.???#?#????# .# 5 Muon
Of the 24 open model families released since 2025, 22 normalize with RMSNorm, 23 gate the MLP,
21 turn queries and keys by position, 8 put recurrent or convolution layers beside attention, 18
untie the embeddings, 19 are mixtures of experts and 5 report Muon. gpt-pp takes the newer
choice in 4 of the 7 columns (normalization, position, embeddings and optimizer) and not in 3:
MLP activation (squared ReLU), attention (full multi-head) and dense or experts (dense).
(the live figure needs JavaScript and loads about 38 kB of data)
Each choice is traced to a configuration file or a report.The cells are read from each model's configuration file on the Hugging Face Hub (for models whose pages require a license, from a mirror), from its model card, or from its technical report, all on 14 November 2026. Dates are those on which the models' pages on the Hub were created, which can precede an announcement by days or weeks. A choice is counted only where a file or a report shows it, so query-key normalization in particular is undercounted, and most cards do not name an optimizer at all. Of the 36 rows, 24 are models released from January 2025 on. Their choices, in counts:
| choice | of the 24 | an early source | what it replaced |
|---|---|---|---|
| RMS normalization | 22 | Zhang and Sennrich, 2019 | layer normalization |
| a gated MLP (SwiGLU, or GeGLU in the Gemma line) | 23 | Shazeer, 2020 | GELU and ReLU |
| rotary positions in at least some layers | 21 | Su and others, 2021 | learned and sinusoidal absolute positions |
| keys and values shared across query heads | 23 | grouped-query attention, Ainslie and others, 2023; latent attention, DeepSeek-V2, May 2024 | a key head and a value head for every query head |
| separate embedding and unembedding matrices | 18 | one tied matrix, which the small models keep | |
| a mixture of experts | 19 | fine-grained experts beside a shared one, DeepSeek-V2, May 2024 | one dense MLP |
| local, chunked or learned-sparse attention in some layers | 14 | Gemma 2, June 2024, one local layer for each global one | full attention in every layer |
| layers that are not attention: linear-recurrent, state-space or convolution | 8 | Qwen3-Next, September 2025; Kimi Linear, October 2025 | attention in every layer |
| query-key normalization, where visible | 6 | Dehghani and others, 2023 | a cap on the attention logits; biases on queries, keys and values |
| Muon, where an optimizer is reported | 5 | Jordan and others, 2024 | AdamW on every parameter |
| no position signal in every fourth layer | 3 | Llama 4, April 2025 | rotary positions in every layer |
| the residual connection replaced | 3 | hyper-connections, September 2024 | the identity skip |
The frame of the block has not moved since 2019, and its norm, its MLP and the way position enters have not moved since 2023. What kept moving is attention, and in 2026 everything around the block: which layers attend at all, what the plus sign of the residual stream is, and which optimizer trains the matrices.
one transformer block, by the year each choice became the default in open models
("default" as the timeline of the census of this part dates it: not a count of models)
2017 the transformer: LayerNorm after each sublayer, ReLU, sinusoidal positions, full
multi-head attention, tied embeddings, one dense MLP
2019 the norm moved in front of each sublayer; GELU replaced ReLU; positions became learned
vectors (GPT-2)
2023 RMSNorm replaced LayerNorm; the MLP became gated (SwiGLU); position became a rotation of
queries and keys (RoPE); the biases went
2024 the base of the rotary frequencies grew fiftyfold and more; query heads began to share
key-value heads; vocabularies passed 100,000 tokens
2025 the routing of experts became sigmoid; a head was added to predict more than the next
token; most layers were confined to a local window; queries and keys were normalized
before their product
in some models: latent attention: one cached vector for all heads (for large mixtures of
experts); a mixture of many small experts beside a shared one (above about 20B
total); a sink or a gate lets a head attend to nothing (in hybrids); Muon trains
the hidden matrices (among the largest open models only)
2026 in some models: an indexer selects the keys a query may see (at 1M context); three
layers of four keep a recurrent state instead of attending (not settled: 6 of the 13
models of 2026); the plus sign of the residual stream is replaced (three flagship
releases)
tokens → embedding → [ norm → attention → + → norm → MLP → + ] × layers → norm → unembedding
The live figure draws this block, morphs it year by year and loads the configuration of any of
36 open models, or of the two recipes of this series, into it.
(the live figure needs JavaScript and loads about 50 kB of data)
“Default” names a historical synthesis here, not a majority vote (Remark 11.1.4). The changes of 2019 and 2023 made the block of Definition 11.1.1; those of 2024 and 2025 changed its attention. The alternatives of 2026 are the subject of Section 11.2.
The block of 2024. The first four rows of the table are one object: normalization by root mean square, before each sub-layer, where GPT-2 put it in 2019 (Section 5.2); a gated MLP (Section 5.3); position as a rotation of queries and keys (Section 4.4); key-value heads shared among query heads; and almost no biases.
Definition 11.1.1 (the settled block). With \(x \in \mathbb R^d\) the residual stream at one position, a layer is
\[x \leftarrow x + \mathrm{Attn}\bigl(\mathrm{RMSNorm}(x)\bigr), \qquad x \leftarrow x + W_2\bigl(\mathrm{SiLU}(W_1 \tilde x) \odot W_3\, \tilde x\bigr), \quad \tilde x = \mathrm{RMSNorm}(x), \tag{11.1}\]where \(\mathrm{RMSNorm}(x) = \gamma \odot x \big/ \sqrt{\tfrac1d \lVert x \rVert^2 + \epsilon}\) with a learned gain \(\gamma\), \(\mathrm{SiLU}(z) = z\, \sigma(z)\), queries and keys are rotated by their positions, \(h\) query heads read \(h_{\mathrm{kv}} \le h\) shared key-value heads, and no linear map has a bias.
The report of one of the models that carry this block says how little of its quality it credits to it.
Llama 3 uses a standard, dense Transformer architecture (Vaswani et al., 2017). It does not deviate significantly from Llama and Llama 2 (Touvron et al., 2023a; Touvron et al., 2023b) in terms of model architecture; our performance gains are primarily driven by improvements in data quality and diversity as well as by increased training scale.
—Grattafiori, Dubey, Jauhri, Pandey and others, "The Llama 3 Herd of Models" (2024). arXiv 2407.21783, the first paragraph of its section on the architecture of the model. The same section raises the base of the rotary frequencies from 10,000 to 500,000 to support longer contexts; several later releases use 1 to 10 million.
Around the block the numbers grew: vocabularies from GPT-2's 50,257 tokens to between 100,000 and 262,144, contexts to 128,000 tokens and, among the largest releases, to a million.
What 2025 added. Three things entered new releases. Queries and keys are normalized before their dot product (Section 3.4), for stability: one report puts this in the place of a cap on the attention logits, another in the place of biases on queries, keys and values. A mixture of experts is used by 19 of the 24, at every scale from about 20 billion parameters up; the dense holdouts are Gemma 3 and four models of at most 7 billion parameters. And most layers stopped attending to the whole context: windows of 128 to 1,024 tokens alternate with full attention, one for one in some models and five for one in others, and a head is given a way to attend to nothing. The model card of the open models released in August 2025 by the laboratory of GPT-2 and GPT-3 has all of it in two paragraphs, and names both ancestors.
2.2 Architecture
Both models have a residual stream dimension of 2880, applying root mean square normalization [6] on the activations before each attention and MoE block. Similar to GPT-2 we use Pre-LN placement [7] [8].
[…]
Attention: Following GPT-3, attention blocks alternate between banded window and fully dense patterns [10] [11], where the bandwidth is 128 tokens. Each layer has 64 query heads of dimension 64, and uses Grouped Query Attention (GQA [12] [13]) with 8 key-value heads. We apply rotary position embeddings [14] and extend the context length of dense layers to 131,072 tokens using YaRN [15]. Each attention head has a learned bias in the denominator of the softmax, similar to off-by-one attention and attention sinks [16] [17], which enables the attention mechanism to pay no attention to any tokens.
The learned bias is one more logit, \(b\), that belongs to no token: the weights of a head become
\[a_{ts} = \frac{e^{\,q_t \cdot k_s}}{e^{\,b} + \sum_{s^{\prime} \le t} e^{\,q_t \cdot k_{s^{\prime}}}},\]and sum to less than one. A plain softmax must spend its whole unit of attention somewhere, and trained models spend the unwanted part on the first token (Section 3.8). The hybrids of 2026 get the same freedom from a gate on the output of each head, \(o_t = \bigl(\sum_s a_{ts} v_s\bigr) \odot \sigma(W_g\, \tilde x_t)\).Qiu, Wang, Zheng, Huang, Wen and others, "Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free", 2025 (arXiv 2505.06708); its abstract reports more than 30 variants of a 15B mixture-of-experts model and a 1.7B dense model trained on 3.5 trillion tokens. The sink is from Xiao and others, 2023, cited in Section 3.
Definition 11.1.2 (mixture of experts, as configured since 2025). The MLP of a layer is replaced by \(E\) MLPs, the experts, each with a centroid \(e_i \in \mathbb R^d\), and one shared MLP. For a token with normalized stream \(\tilde x\), the scores are \(s_i = \sigma(\tilde x^{\top} e_i)\); the selected set \(\mathcal T\) holds the \(K\) experts with the largest \(s_i + b_i\); and the layer's output is
\[\mathrm{MLP}_0(\tilde x) + \sum_{i \in \mathcal T} \pi_i\, \mathrm{MLP}_i(\tilde x), \qquad \pi_i = \frac{s_i}{\sum_{j \in \mathcal T} s_j} .\]The bias \(b_i\) is used for the selection only and is not trained by the gradient: after each step it is lowered if expert \(i\) received more than its share of the batch's tokens and raised if it received less.
A token then pays for \(K + 1\) MLPs while the model stores \(E + 1\): parameters without the matching compute. In most configurations of the table \(E\) is between 128 and 896 and \(K\) between 4 and 16; one model of 671 billion parameters uses 37 billion on a token, and the largest row activates 16 experts of 896.The recipe is that of DeepSeek-AI, "DeepSeek-V3 Technical Report", 2024 (arXiv 2412.19437), Section 2.1.2, where the rule for the bias is called auxiliary-loss-free load balancing; the same recipe recurs in the configurations of Kimi, GLM, MiniMax, Mistral, Hy and ERNIE. The 16 of 896 is from the card of Kimi K3. At 135 million parameters gpt-pp is dense, as the open models of its size are.
Example 11.1.3 (what sharing keys and values saves). The setup: the cache that decoding keeps (Section 3.7), counted from the configurations: with \(h_{\mathrm{kv}}\) key-value heads of width \(d_{\mathrm{head}}\), \(2\, h_{\mathrm{kv}}\, d_{\mathrm{head}}\) numbers for each token in each layer.
Llama 3 8B has 32 layers and 32 query heads of width 128 on 8 key-value heads: \(2 \cdot 8 \cdot 128 = 2{,}048\) numbers a layer, \(2^{16}\) a token. Run on a context of \(2^{20} = 1{,}048{,}576\) tokens it would keep \(2^{36}\) numbers, 128 GiB at two bytes each, and four times that with a key-value head for every query head. gpt-pp keeps 12 key-value heads of width 64, \(2 \cdot 12 \cdot 64 = 1{,}536\) numbers a layer as GPT-2 small does, \(18{,}432\) a token, and 36 MiB for its whole context of 1,024 tokens.
That arithmetic is why attention is the component the releases of 2026 disagree about, and why a model with a context of 1,024 has no stake in the disagreement; Section 11.2 counts the cost layer by layer for nine models. ◁
Remark 11.1.4 (a count is not a comparison). The table records conventions, and a convention can spread by being copied. Few of these choices were published with an ablation at the scale where they are used, and where one was, the result is often modest: of the ratio of local to global layers its authors write "We observe minimal impact on perplexity when changing this ratio."Gemma Team, "Gemma 3 Technical Report", 2025 (arXiv 2503.19786), Section 5.2; the ratio went from one local layer per global layer in Gemma 2 to five in Gemma 3. In the other direction, a measurement on an 8-layer model and 50 million tokens, which is what this series can afford for each choice, says nothing about a trillion parameters. Where the two disagree, neither refutes the other: the disagreement is a question about scale, and most of those are open.
Where gpt-pp sits. The recipe that the series calls the 2026 recipe comes from a second lineage: the public races to train GPT-2 small as fast as possible, in which a change survives only if it wins on the same hardware (Section 5.6). The races and the large releases agree on the frame and differ in several components, and the table sets some rows of the census beside the two recipes of the series.
| model | normalization | MLP | position | attention | embeddings | experts | optimizer |
|---|---|---|---|---|---|---|---|
| Transformer, base (2017) | layer norm, after each sub-layer | ReLU | sinusoidal, absolute | full multi-head, 8 heads | tied | dense | Adam, \(\beta_2 = 0.98\), warmup of 4,000 steps |
| GPT-2 (2019) | layer norm, before each sub-layer and after the last | GELU | learned, absolute | full multi-head | tied | dense | not in the paper |
| Llama 3 8B (April 2024) | RMS norm, before | SwiGLU | rotary, base 500,000 | 32 query heads on 8 key-value heads | untied | dense | AdamW |
| Gemma 3 27B (March 2025) | RMS norm, before and after; query-key norm | GeGLU | rotary, base 1,000,000 in global layers, 10,000 in local | 32 on 16; five local layers of window 1,024 for each global | tied | dense | not reported |
| Qwen3 8B (April 2025) | RMS norm, before; query-key norm | SwiGLU | rotary, base 1,000,000 | 32 on 8 | untied | dense; 128 experts, 8 active, at 30B | not reported |
| SmolLM3 3B (July 2025) | RMS norm, before | SwiGLU | rotary, base 5,000,000, in three layers of four; none in the fourth | 16 on 4 | tied | dense | AdamW (0.9, 0.95), decay 0.1 |
| gpt-oss-120b (August 2025) | RMS norm, before | SwiGLU, clamped | rotary, base 150,000, rescaled for length | 64 on 8; a window of 128 tokens in every other layer; a sink for each head | untied | 128 experts, 4 active | not reported |
| Qwen3.5 27B (February 2026) | RMS norm, before | SwiGLU | rotary on a quarter of the dimensions, base 10,000,000, in attention layers | three recurrent layers for each gated attention layer, 24 on 4 | untied | dense; 512 experts, 10 active, at 397B | not reported |
| DeepSeek-V4-Pro (April 2026) | RMS norm; the skip replaced by constrained hyper-connections | SwiGLU, clamped | rotary on 64 dimensions, rescaled for length | compressed keys and values, one key-value head, a window of 128 tokens | untied | 384 experts, 6 active, one shared | Muon, AdamW for some tensors |
| Kimi K3 (June 2026) | RMS norm; the skip replaced by attention over layers | gated | rotary in the global layers | 69 recurrent layers and 24 gated latent-attention layers | untied | 896 experts, 16 active | not reported |
| the 2019 recipe of the series | layer norm with gain and bias, before each sub-layer and after the last | GELU | learned, absolute | full multi-head, 12 heads | tied | dense | AdamW (0.9, 0.95), decay 0.1 |
| gpt-pp | RMS norm without parameters, before each sub-layer, after the last and on the embedding; query-key norm at a fixed scale of 12 | squared ReLU | rotary, base 10,000 | full multi-head, 12 heads | untied | dense | Muon on hidden matrices, AdamW on the two vocabulary matrices, no decay |
In the seven columns of the census gpt-pp takes the newer choice in four (normalization, position, embeddings, optimizer) and not in three: it squares a ReLU where the field gates, keeps a key-value head for every query head where the field shares them, and has one dense MLP where 19 of the 24 mix experts. What gpt-pp did not adopt, and why:
- The gated MLP. Of the 24, 23 gate; gpt-pp squares a ReLU, the choice of the races since October 2024: as cheap as a ReLU, with a derivative that is continuous at zero (Section 5.3).
- Shared keys and values. By Example 11.1.3 the cache of gpt-pp is 36 MiB; sharing answers a cost that a context of 1,024 tokens does not have.
- A tied embedding at this size. The open models near gpt-pp's size tie. gpt-pp unties and pays \(25{,}165{,}824\) parameters for it, 18.6% of its \(135{,}266{,}304\); the second matrix costs a lookup and no matrix product, and the two are trained at rates almost forty times apart (Section 2.5).
- Gains in the normalization. The RMS norm of the open models carries the gain \(\gamma\) of (11.1); gpt-pp drops it, on the argument that the linear map that follows can absorb it.
- What size and context call for. A large vocabulary, experts, windows, recurrent layers and a longer rotary base each answer a problem that a dense model of 12 layers reading 1,024 tokens does not have.
- Weight decay. The published norm is 0.1; gpt-pp has none, the setting its two references started from (Remark 6.4.5).
- The rest of the races' recipe. Tables indexed by token added to the values of attention, a skip from the embedding into every block, learned scalars on the stream and several refinements of Muon held their place in the races; gpt-pp leaves them out so that its recipe has no component that Sections 2 to 6 do not explain.
The first four can be measured at the scale of this series, and were.
Example 11.1.5 (the field's choices on the 8-layer model). The setup: the 8-layer, 512-wide model of the ladder in the 2026 recipe, trained on 50 million tokens of FineWeb-EDU; then the same run with one component replaced by the choice of the open models; the recipe and five of the replacements in two seeds. Measured: the loss on 1,048,576 held-out tokens, in nats a token.
| the 8-layer model, 2026 recipe, with | where the choice is used | seeds | held-out loss | against the recipe |
|---|---|---|---|---|
| nothing changed | two | 4.044 | ||
| a gated MLP with a SiLU gate (SwiGLU) | 23 of the 24, with the next row | two | 4.035 | −0.009 |
| a gated MLP with a GELU gate (GeGLU) | the Gemma line | one | 4.043 | −0.002 |
| four key-value heads for eight query heads | 23 of the 24 share, at various ratios | one | 4.048 | +0.003 |
| two key-value heads | four query heads to one, the ratio of Llama 3 8B | one | 4.075 | +0.030 |
| one key-value head | eight to one, the ratio of gpt-oss-120b | one | 4.096 | +0.050 |
| one tied vocabulary matrix | the small open models | one | 4.045 | 0.000 |
| RMS norm with a learned gain | 22 of the 24 | two | 4.024 | −0.020 |
| layer norm, with gain and bias | the block of 2019 | two | 4.024 | −0.020 |
| no query-key normalization | 18 of the 24, as far as visible | two | 4.075 | +0.031 |
| AdamW in place of Muon on the hidden matrices, with its own warmup and clipping, the best of four rates | most reports | two | 4.194 | +0.149 |
Sharing costs nothing measurable down to four key-value heads, then 0.030 with two and 0.050 with one, at ratios that open models use with four and eight times as many heads; whether their width buys the sharing back was not measured. Without the normalization of queries and keys the loss is 0.031 higher. AdamW in Muon's place ends about 0.15 higher at its best, 0.136 and 0.163 in its two seeds; the first comparison, which left AdamW the recipe's short warmup and no clipping, had said 0.169. A learned gain and layer norm are each 0.020 better than the recipe's bare norm, in both seeds, so the argument that the next linear map can absorb the gain is not what was measured. The field's gate is a little better than the square, by 0.008 and 0.011. The GELU gate, four key-value heads and the tied table, with 16.8 million fewer parameters, have one seed each and are inside the noise.
These are differences at 59 million parameters and 50 million tokens, to be read with Remark 11.1.4; Section 11.4 says what they would change in the recipe. ◁
Open problems and the directions of 2026
The settled block leaves three questions open, and 2025 and 2026 reopened all of them: how a model spends depth, how it keeps a long context, and how the result is trained and understood. Two words are used strictly below. A result is reported when its authors measured it, in a preprint or on a model card, and checked when a second group did. By that standard almost everything here is reported, and the last column of the table is short.
| direction | reported, by the authors | checked by a second group |
|---|---|---|
| depth as a loop | a model of 3.5B parameters whose core is iterated, trained on 800B tokens, improving up to a compute load equal to that of 50B parameters (arXiv 2502.05171, February 2025); looped models of 1.4B and 2.6B parameters, pretrained on 7.7T tokens, matching models of up to 12B (arXiv 2510.25741, October 2025) | nothing found |
| tiny recursive reasoners | 27M parameters and about 1,000 training examples, about 40% on ARC-AGI-1 (arXiv 2506.21734, 2025); then 7M parameters and two layers, 45% on ARC-AGI-1 and 8% on ARC-AGI-2 (arXiv 2510.04871, October 2025) | the first scores 32% on 100 hidden tasks of ARC-AGI-1 and 2% on ARC-AGI-2, and its hierarchy adds little (ARC Prize, 15 August 2025); in the second, voting over 1,000 samples adds about 11 points and a wrong puzzle identifier gives zero (arXiv 2512.11847) |
| reasoning in hidden states | the last hidden state fed back as the next input (arXiv 2412.06769, December 2024), with a theory of why this searches in parallel (arXiv 2505.12514, May 2025) | nothing found |
| diffusion language models | masked diffusion trained from scratch at 8B parameters (arXiv 2502.09992, February 2025); scaled to 100B by conversion from autoregressive models (arXiv 2512.15745, December 2025) | nothing found |
| recurrent layers in place of most attention | a 7B hybrid that outperforms its transformer twin and scales more efficiently (arXiv 2604.03444, April 2026); a 48B model with a cache smaller by up to 75% and up to six times the decoding throughput at a million tokens (arXiv 2510.26692, October 2025) | nothing found; one maker kept full attention in all 62 layers in October 2025 and April 2026 |
| memory written at test time | fast weights updated in chunks of 2,000 to a million tokens, holding up to 40% of the parameters (arXiv 2505.23884); a lookup memory indexed by n-grams, scaled to 27B (arXiv 2601.07372, January 2026) | nothing found |
| a context of a million tokens | at a million tokens, 27% of the compute for each token and 10% of the cache of the previous version (arXiv 2606.19348, 2026); 75% of the indexer's work removed by reuse across layers (arXiv 2603.12201, 2026) | the configurations are public; the figures are the vendors' |
| the residual connection replaced | several streams mixed by a constrained matrix (arXiv 2512.24880); attention over the outputs of earlier layers (arXiv 2603.15031, March 2026) | nothing found; a 110M imitation of the first reports that it overflows 16-bit floats at inference |
| optimizers after AdamW | Muon in five of the 24 models; gains of 1.4 to 2 times claimed for several methods | 1.4 times at 0.1B parameters, falling to 1.1 times at 1.2B (arXiv 2509.02046, 2025); disputed again in 2026 |
Depth as a loop. A transformer of \(L\) layers spends \(L\) steps of computation on every token, easy or hard. The oldest alternative is to apply the same layers again.
Definition 11.2.1 (recurrent depth). The layers are grouped in three stacks, a prelude \(P\), a core \(R\) and a coda \(C\). On an input \(x\),
\[e = P(x), \qquad s_0 \sim \mathcal N(0, \sigma^2 I), \qquad s_r = R(e, s_{r-1}) \ \ (r = 1, \dots, r_{\max}), \qquad p(\,\cdot \mid x) = C(s_{r_{\max}}) . \tag{11.2}\]The number of iterations \(r_{\max}\) is drawn at random at each training step and chosen freely at test time. A looped transformer is the case without prelude or coda: the same \(k\) layers applied \(L\) times.
We study a novel language model architecture that is capable of scaling test-time computation by implicitly reasoning in latent space. Our model works by iterating a recurrent block, thereby unrolling to arbitrary depth at test-time. This stands in contrast to mainstream reasoning models that scale up compute by producing more tokens. Unlike approaches based on chain-of-thought, our approach does not require any specialized training data, can work with small context windows, and can capture types of reasoning that are not easily represented in words. We scale a proof-of-concept model to 3.5 billion parameters and 800 billion tokens. We show that the resulting model can improve its performance on reasoning benchmarks, sometimes dramatically, up to a computation load equivalent to 50 billion parameters.
—Geiping, McLeish, Jain, Kirchenbauer and others, "Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach" (2025). arXiv 2502.05171, abstract. The looped model pretrained at scale is Zhu, Wang, Hua, Zhang and others, "Scaling Latent Reasoning via Looped Language Models", 2025 (arXiv 2510.25741); the expressive power of loops is in Saunshi, Dikkala, Li, Kumar and Reddi, "Reasoning with Latent Thoughts: On the Power of Looped Transformers", 2025 (arXiv 2502.17416).
Parameters are the wrong unit for a loop: a core applied $$r$$ times costs $$r$$ passes. A looped model of 2.6 billion parameters that matches models of up to 12 billion has saved memory, and whether it has saved compute depends on the passes. What is open is whether iterations compute something that more layers of the same cost would not, and how a model should decide when to stop.
Tiny recursive reasoners. The same idea at the opposite scale: two networks of a few million parameters, set against language models on puzzles. The second fits in a few lines.
Algorithm 11.2.2 TINY-RECURSION
(training on one example; after Jolicoeur-Martineau
2025, Figure 3)
Input a puzzle x with solution y*; one small network net, an output
head and a halting head; n = 6, T = 3; at most N supervision
steps.
Output updated weights; an answer y.
1. y <- y_init, z <- z_init (the answer and a latent state)
2. for step = 1, ..., N:
3. repeat T times, the first T - 1 without gradients:
4. repeat n times: z <- net(x, y, z) (think)
5. y <- net(y, z) (revise the answer)
6. loss <- cross-entropy(output(y), y*)
7. + binary cross-entropy(halt(y), [output(y) = y*])
8. one optimizer step on loss; detach y and z
9. if halt(y) > 0: stop
With \(n = 6\) and \(T = 3\) the network is applied 21 times for each supervision step, and the answer of one step is the starting point of the next. The claim, and two audits of the family:
Hierarchical Reasoning Model (HRM) is a novel approach using two small neural networks recursing at different frequencies. This biologically inspired method beats Large Language models (LLMs) on hard puzzle tasks such as Sudoku, Maze, and ARC-AGI while trained with small models (27M parameters) on small data (around 1000 examples). HRM holds great promise for solving hard problems with small networks, but it is not yet well understood and may be suboptimal. We propose Tiny Recursive Model (TRM), a much simpler recursive reasoning approach that achieves significantly higher generalization than HRM, while using a single tiny network with only 2 layers. With only 7M parameters, TRM obtains 45% test-accuracy on ARC-AGI-1 and 8% on ARC-AGI-2, higher than most LLMs (e.g., Deepseek R1, o3-mini, Gemini 2.5 Pro) with less than 0.01% of the parameters.
—Jolicoeur-Martineau, "Less is More: Recursive Reasoning with Tiny Networks" (2025). arXiv 2510.04871, abstract. TRM is the tiny recursive model; its predecessor, the hierarchical reasoning model or HRM, is Wang, Li, Sun, Chen and others, 2025 (arXiv 2506.21734).
At the same time, by running a series of ablation analyses, we made some surprising findings that call into question the prevailing narrative around HRM:
- The "hierarchical" architecture had minimal performance impact when compared to a similarly sized transformer.
- However, the relatively under-documented "outer loop" refinement process drove substantial performance, especially at training time.
- Cross-task transfer learning has limited benefits; most of the performance comes from memorizing solutions to the specific tasks used at evaluation time.
- Pre-training task augmentation is critical, though only 300 augmentations are needed (not 1K augmentations as reported in the paper). Inference-time task augmentation had limited impact.
—ARC Prize Team, "The Hidden Drivers of HRM's Performance on ARC-AGI" (15 August 2025). arcprize.org/blog/hrm-analysis, the list of findings after the scores. On hidden tasks the team measured 32% on ARC-AGI-1, against the 41% of the paper, and 2% on ARC-AGI-2.
Tiny Recursive Models (TRM) were proposed as a parameter-efficient alternative to large language models for solving Abstraction and Reasoning Corpus (ARC) style tasks. The original work reports strong performance and suggests that recursive latent updates enable non-trivial reasoning, but it remains unclear how much of this performance stems from architecture, test-time compute, or task-specific priors. In this technical note, we empirically analyze the ARC Prize TRM checkpoint on ARC-AGI-1 and report four behavioral findings and an efficiency comparison. First, we show that test-time augmentation and majority-vote ensembling account for a substantial fraction of reported performance: the 1000-sample voting pipeline improves Pass@1 by about 11 percentage points over single-pass canonical inference. Second, a puzzle-identity ablation reveals strict dependence on task identifiers: replacing the correct puzzle ID with a blank or random token yields zero accuracy. Third, a recursion trajectory analysis shows that most of the final accuracy is achieved at the first recursion step and that performance saturates after few latent updates, indicating shallow effective recursion. Fourth, early-stage training experiments under canonical versus heavy augmentation regimes suggest that heavy augmentation broadens the distribution of candidate solutions and improves multi-sample success. Finally, we compare TRM with a naive QLoRA fine-tune of Llama 3 8B on canonical ARC-AGI-1, finding that TRM's non-autoregressive design achieves much higher throughput and substantially lower memory usage in this setting. Overall, TRM's ARC-AGI-1 performance appears to arise from an interaction between efficiency, task-specific conditioning, and aggressive test-time compute rather than deep internal reasoning.
—Roye-Azar, Vargas-Naranjo, Ghai, Balamurugan and Amir, "Tiny Recursive Models on ARC-AGI-1: Inductive Biases, Identity Conditioning, and Test-Time Compute" (2025). arXiv 2512.11847, abstract. A further analysis is Ren and Liu, "Are Your Reasoning Models Reasoning or Guessing? A Mechanistic Analysis of Hierarchical Reasoning Models", 2026 (arXiv 2601.10679).
The dispute is active, and we did not search for the replies of the original authors. What stands either way is the method of the audits: remove one ingredient at a time and see where the score goes.
Reasoning in hidden states. A model that reasons in text squeezes every step through the vocabulary: a token is sampled, embedded and read back. Continuous thought skips the squeeze: the last hidden state is fed back as the next input, \(e_{t+1} = h_t\), as a looped model does along depth. The theory offered is that a hidden state can carry several candidate paths of a search at once, where a token commits to one.Hao, Sukhbaatar, Su, Li and others, "Training Large Language Models to Reason in a Continuous Latent Space", 2024 (arXiv 2412.06769); Zhu, Hao, Hu, Jiao and others, "Reasoning by Superposition: A Theoretical Perspective on Chain of Continuous Thought", 2025 (arXiv 2505.12514). What such states contain is a question for the tools of Sections 9 and 10.
Not left to right. Section 1 rests on the chain rule taken in one order. A masked diffusion language model drops the order.
Definition 11.2.3 (the masked diffusion objective). Draw \(t \sim U(0, 1)\), replace each token of a text \(x_0\) independently with probability \(t\) by a mask token \(\mathrm M\) to get \(x_t\), and train a transformer without a causal mask on
\[\mathcal L(\theta) = -\, \mathbb E_{t,\, x_0,\, x_t} \Bigl[\, \frac1t \sum_{i} \mathbf 1\bigl[x_t^i = \mathrm M\bigr]\, \log p_\theta\bigl(x_0^i \mid x_t\bigr) \Bigr], \tag{11.3}\]an upper bound on the negative log-likelihood of \(x_0\). Generation starts from a fully masked sequence and reveals a fraction of the positions at each step.
The capabilities of large language models (LLMs) are widely regarded as relying on autoregressive models (ARMs). We challenge this notion by introducing LLaDA, a diffusion model trained from scratch under the pre-training and supervised fine-tuning (SFT) paradigm. LLaDA employs a forward data masking process and a reverse generation process, parameterized by a Transformer to predict masked tokens. It provides a principled generative approach for probabilistic inference by optimizing a likelihood lower bound. Across extensive benchmarks on general tasks, math, code, and so on, LLaDA demonstrates strong scalability and performs comparably to our self-constructed ARM baselines. Remarkably, LLaDA 8B is competitive with strong LLMs like LLaMA3 8B in in-context learning and, after SFT, exhibits impressive instruction-following abilities in case studies such as multi-turn dialogue. Moreover, LLaDA addresses the reversal curse, surpassing GPT-4o in a reversal poem completion task. Our findings show the promise of diffusion models for language modeling at scale and challenge the common assumption that core LLM capabilities discussed above inherently depend on ARMs. Project page and codes: https://ml-gsai.github.io/LLaDA-demo/.
—Nie, Zhu, You, Zhang and others, "Large Language Diffusion Models" (2025). arXiv 2502.09992, abstract; ARMs are autoregressive models. The scaled successor is Bie, Cao, Chen, Du and others, "LLaDA2.0: Scaling Up Diffusion Language Models to 100B", 2025 (arXiv 2512.15745); a commercial diffusion model for code is described in Inception Labs, "Mercury: Ultra-Fast Language Models Based on Diffusion", 2025 (arXiv 2506.17298).
The companion series derives this objective and its samplers in its part on discrete diffusion, and states the price in the heading of its third section: one token at a time is exact, many at a time is not. Tokens revealed in the same step are drawn independently given the rest, so what they owe to one another is lost unless more steps are spent; the autoregressive model is the end of that trade where nothing is lost and every token costs a pass.
Layers that are not attention. Attention keeps every key and value, and Example 11.1.3 priced that. A linear-recurrent layer keeps one matrix of fixed size and writes each token into it.
Definition 11.2.4 (the gated delta rule). The state is a matrix \(S_t \in \mathbb R^{d_v \times d_k}\). With a key \(k_t\) of unit norm, a value \(v_t\), a query \(q_t\), and two gates \(\alpha_t, \beta_t \in (0, 1)\), all computed from the stream at position \(t\),
\[S_t = \alpha_t\, S_{t-1} \bigl(I - \beta_t\, k_t k_t^{\top}\bigr) + \beta_t\, v_t k_t^{\top}, \qquad o_t = S_t\, q_t . \tag{11.4}\]Proposition 11.2.5 (the delta rule overwrites). (i) With \(\alpha_t = 1\), (11.4) is one step of gradient descent on \(\tfrac12 \lVert S k_t - v_t \rVert^2\) from \(S_{t-1}\) with step \(\beta_t\). (ii) \(S_t k_t = \alpha_t (1 - \beta_t)\, S_{t-1} k_t + \beta_t v_t\). (iii) If \(q \perp k_t\) then \(S_t q = \alpha_t\, S_{t-1} q\).
Proof. The gradient of \(\tfrac12 \lVert S k - v \rVert^2\) in \(S\) is \((S k - v) k^{\top}\), and \(S - \beta (S k - v) k^{\top} = S (I - \beta k k^{\top}) + \beta v k^{\top}\), which is (i). For (ii) and (iii), \((I - \beta k k^{\top}) k = (1 - \beta) k\) and \(v k^{\top} k = v\) since \(\lVert k \rVert = 1\), while \((I - \beta k k^{\top}) q = q\) and \(v k^{\top} q = 0\) when \(q \perp k\). ∎
For a concrete example, store \(x\) as 3, then \(y\) as 5, then \(x\) as 7. Reading \(x\), linear attention returns 3 + 7, and the delta rule 7, by the one line of its code that first subtracts what the state held under \(x\). With the gate \(\beta\) at one half the read becomes \(0.25 \times 3 + 0.50 \times 7\), which is part (ii); with the key \(y\) leaning on \(x\), overwriting \(x\) disturbs \(y\), which is why part (iii) asks for orthogonal keys.
the same pairs written to two states of 8 × 8 numbers: x ← 3, y ← 5, x ← 7, then x is read
linear attention: add the gated delta rule: remove, then write
S = S + outer(v, k) old = S @ k; S = S - beta * outer(old, k)
S = alpha * S; S = S + beta * outer(v, k)
x y z x y z a column: a key
0 . . . . . . . . 0 . . . . . . . . a row: a value, the digit at its left
1 . . . . . . . . 1 . . . . . . . .
2 . . . . . . . . 2 . . . . . . . .
3 # . . . . . . . 3 . . . . . . . . the first write of x
4 . . . . . . . . 4 . . . . . . . .
5 . # . . . . . . 5 . # . . . . . .
6 . . . . . . . . 6 . . . . . . . .
7 # . . . . . . . 7 # . . . . . . . the second write of x
read x returns 3 + 7 7
Linear attention adds every value written under a key; the delta rule removes what it holds
under the key before it writes. With the gate β at 0.5 instead of 1, the delta rule returns
0.25 × 3 + 0.50 × 7 for x.
(the live figure needs JavaScript; it computes everything in the page and loads no data)
So as \(\beta_t\) approaches one, the value stored under \(k_t\) is replaced by \(v_t\), and what is stored under orthogonal keys is left alone but for the decay \(\alpha_t\); linear attention adds and never erases. Part (i) is the reading of attention as a regression fitted to the context while the model runs (Section 3.3), here with one gradient step for each token, and the layer costs the same at the millionth token as at the first.Yang, Kautz and Hatamizadeh, "Gated Delta Networks: Improving Mamba2 with Delta Rule", ICLR 2025 (arXiv 2412.06464). The state-space line continues in Lahoti, Li, Chen, Wang, Bick, Kolter, Dao and Gu, "Mamba-3: Improved Sequence Modeling using State Space Principles", ICLR 2026 (arXiv 2603.15569), whose abstract reports 0.6 points of average downstream accuracy over the gated delta rule at 1.5B parameters. Eight of the 24 models put recurrent or convolution layers in the place of most of their attention layers, most often in three positions of four:
- Language Model
- Number of Parameters: 27B
- Hidden Dimension: 5120
- Token Embedding: 248320 (Padded)
- Number of Layers: 64
- Hidden Layout: 16 × (3 × (Gated DeltaNet → FFN) → 1 × (Gated Attention → FFN))
- Gated DeltaNet:
- Number of Linear Attention Heads: 48 for V and 16 for QK
- Head Dimension: 128
- Gated Attention:
- Number of Attention Heads: 24 for Q and 4 for KV
- Head Dimension: 256
- Rotary Position Embedding Dimension: 64
- Feed Forward Network:
- Intermediate Dimension: 17408
- LM Output: 248320 (Padded)
- MTP: trained with multi-steps
Until 2026 the case for the hybrid was memory. It then became quality, on the word of the builders:
Recent work has demonstrated the potential of non-transformer language models, especially linear recurrent neural networks (RNNs) and hybrid models that mix recurrence and attention. Yet there is no consensus on whether the potential benefits of these new architectures justify the risk and effort of scaling them up. To address this, we provide evidence for the advantages of hybrid models over pure transformers on several fronts. First, theoretically, we show that hybrid models do not merely inherit the expressivity of transformers and linear RNNs, but can express tasks beyond both, such as code execution. Putting this theory to practice, we train Olmo Hybrid, a 7B-parameter model largely comparable to Olmo 3 7B but with the sliding window layers replaced by Gated DeltaNet layers. We show that Olmo Hybrid outperforms Olmo 3 across standard pretraining and mid-training evaluations, demonstrating the benefit of hybrid models in a controlled, large-scale setting. We find that the hybrid model scales significantly more efficiently than the transformer, explaining its higher performance. However, its unclear why greater expressivity on specific formal problems should result in better scaling or superior performance on downstream tasks unrelated to those problems. To explain this apparent gap, we return to theory and argue why increased expressivity should translate to better scaling efficiency, completing the loop. Overall, our results suggest that hybrid models mixing attention and recurrent layers are a powerful extension to the language modeling paradigm: not merely to reduce memory during inference, but as a fundamental way to obtain more expressive models that scale better during pretraining.
—Merrill, Li, Romero, Svete and others, "Olmo Hybrid: From Theory to Practice and Back" (2026). arXiv 2604.03444, abstract; "its" is the abstract's spelling. The model has 7B parameters and is compared with a transformer of the same family in which windowed attention stands where the recurrent layers are. The earlier claim of a win over full attention under fair comparison is Kimi Team, "Kimi Linear: An Expressive, Efficient Attention Architecture", 2025 (arXiv 2510.26692).
The comparison is controlled and fully open, which is rare, and it is its authors' own. We found no replication of it, and one maker of large models kept full attention in every layer through two releases. The question that the abstract then raises, why expressivity on formal problems should carry over to unrelated tasks, is the open problem; its authors answer it with an argument from theory.
Memory at test time. By Proposition 11.2.5 a recurrent layer already learns while it reads: its state is a small model fitted to the context by gradient steps. A line of work makes that inner model larger, with memory modules trained to memorize as they read and fast weights that hold up to 40% of the parameters; at the other extreme stands a plain table indexed by n-grams, which its authors set beside the experts of Definition 11.1.2 as a second way to add parameters without compute.Behrouz, Zhong and Mirrokni, "Titans: Learning to Memorize at Test Time", 2024 (arXiv 2501.00663); Behrouz, Razaviyayn, Zhong and Mirrokni, "Nested Learning: The Illusion of Deep Learning Architectures", 2025 (arXiv 2512.24695); Zhang, Bi, Hong, Zhang and others, "Test-Time Training Done Right", 2025 (arXiv 2505.23884); Cheng, Tian, Zeng, Dai and others, "Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models", 2026 (arXiv 2601.07372). No second group has measured any of it, that we found.
A million tokens. Eight rows of the census, all from 2026, list a context of 1,048,576 tokens. Using the arithmetic of Example 11.1.3 to price nine models layer by layer shows where the cost concentrates: the layers that keep every token hold 37.4% of the memory of the eight models priced at 1,024 tokens, and 99.84% at 1,048,576.
the cache of keys and values at a context of 1,048,576 tokens, 2 bytes a number one # is 8 GiB kept by layers that keep every token; one - is 8 GiB kept by the others model layers cache Llama 3 8B 32 full ################ 128 GiB DeepSeek-V3 61 full ######### 68.6 GiB gpt-oss-120b 18 window + 18 full ##### 36.0 GiB MiniMax-M2 62 full ############################### 248 GiB Qwen3.5-27B 48 recurrent + 16 full ######## 64.1 GiB Gemma 4 31B 50 window + 10 full ##### 40.8 GiB DeepSeek-V4-Pro 61 compressed not stated: its configuration compresses the cache Kimi K3 69 recurrent + 24 full ### 27.2 GiB gpt-pp 12 full ##### 36.0 GiB a layer keeps: full attention 2·g·d·t numbers for g key-value heads of width d and t tokens (g·d·t when keys and values are one tensor, (512 + 64)·t as a latent); a window of w tokens 2·g·d·min(t, w); a recurrent layer heads × d × d, whatever t. At 1,048,576 tokens the layers that keep every token hold 99.84% of the memory of the 8 models priced; at 1,024 tokens, 37.4%. (the live figure needs JavaScript and loads about 7 kB of data)
At that context Kimi K3 keeps 27.2 GiB, 99.3% of it in the 24 of its 93 layers that attend to everything, and the longest cache, 248 GiB, belongs to the model that kept full attention in all 62 layers. The twelve layers of gpt-pp, run at a thousand times their own context, would keep 36.0 GiB, as much as the 36 layers of gpt-oss-120b: once the context is long, a model can afford few layers of full attention, whatever its size. Only two of the nine models list such a context; the other memory estimates at that length are extrapolations. And a million is \(2^{20}\) tokens here, as the configurations write it: the figure of Section 3.7 stops at \(10^6\), where the same arithmetic gives gpt-pp 34.3 GiB.
Two devices go further. A learned selection puts a small indexer in front of attention, which scores every earlier position and passes on the 512 to 2,048 it ranks highest. A compressed cache stores one entry for a block of tokens, every 4 tokens in 30 layers of one model and every 128 in its other 31; its configuration does not say what an entry holds, so no memory estimate is given for it here.
The plus sign. For a decade nothing stood on the skip (Section 5.1). Three of the 24 models now put something there, two of them flagships of 2026. One states its three changes in a list:
DeepSeek-V4 series incorporate several key upgrades in architecture and optimization:
- Hybrid Attention Architecture: We design a hybrid attention mechanism combining Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA) to dramatically improve long-context efficiency. In the 1M-token context setting, DeepSeek-V4-Pro requires only 27% of single-token inference FLOPs and 10% of KV cache compared with DeepSeek-V3.2.
- Manifold-Constrained Hyper-Connections (mHC): We incorporate mHC to strengthen conventional residual connections, enhancing stability of signal propagation across layers while preserving model expressivity.
- Muon Optimizer: We employ the Muon optimizer for faster convergence and greater training stability.
—DeepSeek-AI, model card of DeepSeek-V4-Pro (2026). huggingface.co/deepseek-ai/DeepSeek-V4-Pro, "Introduction"; FLOPs are floating-point operations and the KV cache is the cache of keys and values. The report is DeepSeek-AI, "DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence" (arXiv 2606.19348). arXiv serves it under that number with a first version dated 26 April 2026, although a number beginning 2606 normally belongs to June; we give both as served.
Hyper-connections keep \(n\) copies of the stream, \(X_\ell \in \mathbb R^{n \times d}\), and let each layer read a mixture of them and write to all:
\[X_{\ell+1} = H^{\mathrm{res}}_\ell X_\ell + \bigl(H^{\mathrm{post}}_\ell\bigr)^{\top} F_\ell\bigl(H^{\mathrm{pre}}_\ell X_\ell\bigr), \qquad H^{\mathrm{res}}_\ell \in \mathbb R^{n \times n}, \quad H^{\mathrm{pre}}_\ell, H^{\mathrm{post}}_\ell \in \mathbb R^{1 \times n} . \tag{11.5}\]With \(n = 1\) and all three equal to one this is the block of Section 5. The constrained version projects each \(H^{\mathrm{res}}_\ell\) onto the doubly stochastic matrices, those with nonnegative entries whose rows and columns each sum to one.Zhu, Huang, Huang, Zeng and others, "Hyper-Connections", 2024 (arXiv 2409.19606); Xie, Wei, Cao, Zhao and others, "mHC: Manifold-Constrained Hyper-Connections", 2025 (arXiv 2512.24880), where the projection is the Sinkhorn-Knopp iteration and the two other maps pass through a sigmoid.
Proposition 11.2.6 (what the constraint buys). Let \(H_1, \dots, H_L\) be doubly stochastic and \(\Pi = H_L \cdots H_1\). Then \(\Pi\) is doubly stochastic, and for every \(X \in \mathbb R^{n \times d}\) the mean over the streams is carried unchanged, \(\tfrac1n \mathbf 1^{\top} \Pi X = \tfrac1n \mathbf 1^{\top} X\), and \(\lVert \Pi X \rVert_F \le \lVert X \rVert_F\).
Proof. \(\Pi \mathbf 1 = \mathbf 1\) and \(\mathbf 1^{\top} \Pi = \mathbf 1^{\top}\) by induction on \(L\), and a product of nonnegative matrices is nonnegative; the mean follows from \(\mathbf 1^{\top} \Pi = \mathbf 1^{\top}\). By Birkhoff's theorem \(\Pi = \sum_j \theta_j P_j\) with permutation matrices \(P_j\) and weights \(\theta_j \ge 0\) that sum to one; each \(P_j\) preserves the Frobenius norm, and the triangle inequality gives the bound. ∎
So the part of the signal that is carried through depth, as opposed to written by a layer, can neither grow nor lose its mean, which is the property the identity has and an unconstrained matrix has not. The other design reaches further: "We propose Attention Residuals (AttnRes), which replaces this fixed accumulation with softmax attention over preceding layer outputs, allowing each layer to selectively aggregate earlier representations with learned, input-dependent weights."Kimi Team, "Attention Residuals", 2026 (arXiv 2603.15031), abstract; the sentences before it are quoted in Section 5.6.
Whether any of these survives is open.
Optimizers. Section 6 ends on a dispute, and it is unresolved. Muon is reported by five of the 24 models (the four whose reports Section 6 cites, and one that gives the name alone); those that say how keep AdamW for embeddings, the output layer and gains. An audit that tuned every optimizer found that "the speedup of matrix-based optimizers is inversely proportional to model scale, decreasing from 1.4x over AdamW for 0.1B parameter models to merely 1.1x for 1.2B parameter models."Wen, Hall, Ma and Liang, "Fantastic pretraining optimizers and where to find them", 2025 (arXiv 2509.02046), abstract. The replies of 2026 are cited in Section 6.5; they are preprints read at the level of their abstracts. Three preprints of 2026 answer: one attributes the shrinking gain to weight decay held constant, one reports that matrix methods keep their lead at several billion parameters, one calls Muon's gains mixed and dependent on the schedule. This series measured the question below the small end of the audit's range, on 59 million parameters and 50 million tokens, each rule at several rates under the warmup and clipping of its own recipe: Muon ends about 0.15 nats below the best AdamW, 0.136 and 0.163 in two seeds; a cosine does not help AdamW; SGD with momentum and Lion end 0.49 and 0.44 behind (Section 6.8).
Theory. The open questions that Sections 1, 6 and 8 met, in one place:
- Which mechanism carries in-context learning in large models. Induction heads are causal in small attention-only models; in larger ones, ablating them changes few-shot accuracy little more than ablating random heads, and the heads that matter there often began as induction heads (Section 8.2).
- What delays generalization, and when a second descent appears. No account of grokking covers both the cases that need weight decay and those without it (Section 8.3), and double descent is shown for repeated data and for datasets that exceed a capacity of about 3.6 bits a parameter, not for one-pass pretraining (Section 8.4, Section 8.6).
- How features are learned. The staircase of exactly solved linear networks has been extended to attention, where heads are added one at a time; whether that describes the pretraining of a language model is open (Section 8.1, Section 8.5).
- Why Adam beats gradient descent on transformers. Class imbalance from Zipf's law, a Hessian that differs by block, smoothness in the max-norm: no study has shown which accounts for most of the gap on a real language model (Section 6.1).
- The form of the scaling law. The additive law of 2022 fixes the mixed derivative of the loss in parameters and tokens at zero, "artificially enforcing a cross-derivative of exactly zero" in the words of the newest fit; coupled forms measure a nonzero one, by extrapolation beyond their own grids and without replication so far (Section 1.7).Videau, Youbi-Idrissi, Lopez-Paz and Ahuja, "Skaling: Chinchilla's exponents meet Kaplan's coupling", 2026 (arXiv 2608.07222), Section 1.
The unit of interpretation. Section 10 ended on a contest that 2026 did not settle. The candidates for the unit in which a network is to be read are the feature of a sparse dictionary, the feature of a transcoder that imitates an MLP, a component of the parameters themselves, the neuron after all, and a network trained to be sparse from the start. The newest entry states its own criterion:
Mechanistic interpretability aims to uncover how neural networks use their parameters to implement their impressive neural algorithms. Although previous work has uncovered substantial structure in the intermediate representations that networks use, little progress has been made to understand how the parameters and nonlinearities of networks perform computations on those representations. In this work, we present a method that brings us closer to this understanding by decomposing a language model's parameters into subcomponents that each implement only a small part of the model's learned algorithm, while simultaneously requiring only a small fraction of those subcomponents to account for the network's behavior on any input. The method, adVersarial Parameter Decomposition (VPD), optimizes for decompositions of neural network parameters into simple subcomponents that preserve the network's input-output behavior even when many subcomponents are ablated, including under ablations that are adversarially selected to destroy behavior. This encourages learning subcomponents that provide short, mechanistically faithful descriptions of the network's behavior that should aggregate appropriately into more global descriptions of the network's learned algorithm. We study how sequences of interactions between these parameter subcomponents produce the network's output on particular inputs, enabling a new kind of 'circuit' analysis. While more work remains to be done to deepen our understanding of how neural networks use their parameters to compute their behavior, our work suggests an approach to identify a small set of simple, mechanistically faithful subcomponents on which further mechanistic analysis can be based.
—Bushnaq, Braun, Clive-Griffin, Bussmann, Hu, Ivanitskiy, Linsefors and Sharkey, "Interpreting Language Model Parameters" (2026). arXiv 2610.20748, 14 October 2026, abstract. Its comparison with transcoders is on a model of four layers and 67M parameters, under its own accounting of sparsity.
A replacement built from features reproduces the model exactly on one prompt once its errors are added back, and yet "this does not guarantee that the local replacement model and underlying model use the same mechanisms." And the graphs show where information moved, not why attention moved it; the method's authors show its failure on the circuit that Section 9.3 derived by hand, in a two-layer attention-only model that reads "I always loved visiting Aunt Sally. Whenever I was feeling sad, Aunt":
These models will have induction heads attend back to "Sally", and then predict that is the correct answer. If we were to apply our present method, the answer isn’t very informative. It would simply tell us that the model predicted "Sally", because there was a token "Sally" earlier in the context.
This misses the entire interesting story! The induction head attends to "Sally" because it was preceded by "Aunt", which matches the present token. Previous methods (e.g. […] were able to elucidate this, and so this case might even be seen as a kind of regression.
—Ameisen, Lindsey, Pearce and others, "Circuit Tracing: Revealing Computational Graphs in Language Models" (2025). Transformer Circuits Thread, 27 March 2025, the limitations, "Example: Induction", two consecutive paragraphs; the mark […] stands for two references, after which the page leaves its parenthesis open; the sentence quoted before the block is from the section on the local replacement model.
What the candidates share is the test: whatever the unit, intervene in the real network and check.
What one GPU can do. A graduate student with one card cannot test a claim about a trillion parameters or a million tokens of context. What one card can do is ablate at ten to a hundred million parameters with seeds, and measure mechanisms where the answer is known.
Example 11.2.7 (what a card buys). The setup: one card of the machine the series was trained on, 32 GB of memory, capped at 165 W; the three sizes of the series in the 2026 recipe; measured: sustained tokens a second and the time of a run.
| model | tokens a second | a run of | takes |
|---|---|---|---|
| gpt-pico, 6 layers, 128 wide, context 256 | 2,218,978 | 540M tokens, one pass over TinyStories | 266 s |
| the 8-layer model, 512 wide, context 512 | 232,000 to 238,000 | 100M tokens of FineWeb-EDU | about 7 min |
| the 12-layer model, 768 wide, context 1,024 | 90,000 to 94,000 | 1B tokens of FineWeb-EDU | about 3 h |
Two seeds of one configuration end 0.001 to 0.008 nats apart under the 2026 recipe, and up to 0.018 apart on the ladder's rungs under AdamW. A design worth 0.02 nats in the 2026 recipe therefore shows in one run each, one worth less than 0.01 needs seeds, and two seeds of each of two designs cost about half a card-hour. ◁
At that price the following are within reach, and each is open:
- The stated reasons for query-key normalization and for the gate. With and without each: the largest attention logit, the largest stable learning rate, the mass on the first token, the loss at two and four times the training length.
- The hybrid at small scale. A gated delta layer in three positions of four against attention in all, at equal tokens: loss, retrieval of a planted token from a long context, bytes of state. It may well lose at this scale; the result is worth reporting either way.
- Depth as a loop, and the tiny recursive models. Four layers applied twice against eight layers, then the loss against the number of loops at test time; and for the models that are small by construction, the audits above as a template.
- A masked diffusion model of gpt-pico's size on TinyStories. The bound (11.3) against the autoregressive loss, and samples against the number of steps.
- The optimizer dispute, in miniature. Muon against AdamW, each at its best rate under its own warmup and clipping, was run here at one size and one budget, in two seeds. What is left: more seeds, longer budgets, larger models, and the weight decay constant, scheduled, or replaced by fixed norms.
- Units of interpretation on a model with a known circuit. The two-layer attention-only model of Section 9 has an induction circuit that can be written down; dictionaries, transcoders, neurons and parameter components can be scored against it, including on why a head attends where it does.
Reading, by section
The sidenotes of the parts are the bibliography: 968 works, from Markov's study of Eugene Onegin in 1913 to papers of 2026, 71% of them from 2023 or later. The accompanying map records 108 lines of descent stated in the research notes, one work refined by another. These are selected intellectual relationships, not a graph of citations.
map 968 works, a mark each, by year (across) and theme (down); the counts:
to 2009 2010-16 2017-19 2020-22 2023 2024 2025 2026 all
language modeling 10 5 5 1 1 1 6 4 33
tokens 2 13 14 14 5 21 12 15 96
attention 0 6 6 22 15 22 34 25 130
position 0 0 3 7 8 14 15 7 54
the block 0 16 18 17 11 12 24 15 113
optimization 0 5 17 11 7 18 32 29 119
training 0 1 6 20 16 42 45 28 158
dynamics and theory 0 2 15 29 29 19 24 12 130
circuits 0 0 0 8 11 11 9 8 47
features 1 1 1 3 4 14 42 22 88
selected Kaplan 2020, Scaling Laws for Neural Language Models
builds on Hestness 2017
built on by Hoffmann 2022, Pearce 2024, Porian 2024; through them Sardana 2023, Besiroglu 2024,
Gadre 2024, Schaeffer 2025, Barkeshli 2026, Videau 2026, Hartenstein 2026
The series read 968 works, from 1913 to 2026; 689 of them (71 %) appeared in 2023 or later. Its
notes state 108 lines of descent among 160 of them.
(the live figure needs JavaScript and loads about 0.2 MB of data)
The table is a shorter route into the same material: for each of Sections 1 to 10, the works to read first, the classical ones beside the latest, each with a sentence on why.
| work | why read it | |
|---|---|---|
| Section 1, predicting the next token | Shannon, "A Mathematical Theory of Communication", Bell System Technical Journal 27 (1948), and "Prediction and Entropy of Printed English", the same journal 30 (1951) | the source model, the series of approximations to English and the guessing game: the problem and its measure, before any network |
| Chen and Goodman, "An empirical study of smoothing techniques for language modeling", Computer Speech and Language 13 (1999) | what counting can do when done well, and the baseline every neural model was first measured against | |
| Bengio, Ducharme, Vincent and Jauvin, "A Neural Probabilistic Language Model", Journal of Machine Learning Research 3 (2003) | names the curse of dimensionality for counts and answers it with learned representations; the embedding table and the softmax are both here | |
| Radford, Wu, Child, Luan, Amodei and Sutskever, "Language Models are Unsupervised Multitask Learners", OpenAI (2019), with Brown and others, "Language Models are Few-Shot Learners", NeurIPS 2020 (arXiv 2005.14165) | GPT-2 and GPT-3: one objective, more scale, and tasks posed as text to be continued | |
| Delétang and others, "Language Modeling Is Compression", ICLR 2024 (arXiv 2309.10668) | the identity between loss and code length put to work: a language model as a general-purpose compressor | |
| Kaplan, McCandlish and others, "Scaling laws for neural language models" (arXiv 2001.08361, 2020), and Hoffmann and others, "Training compute-optimal large language models" (arXiv 2203.15556, 2022) | the two laws for spending a budget, which disagree | |
| Besiroglu, Erdil, Barnett and You, "Chinchilla scaling: a replication attempt" (arXiv 2404.10102, 2024), with Porian, Wortsman, Jitsev, Schmidt and Carmon, "Resolving discrepancies in compute-optimal scaling of language models", NeurIPS 2024 (arXiv 2406.19146) | the audit: why the two laws disagreed, and why an exponent belongs to a protocol | |
| Scheibner, Smith and Bialek, "Large language models and the entropy of English" (arXiv 2512.24969, 2025), and Cagnetta, Raventós, Ganguli and Wyart, "Deriving neural scaling laws from the statistics of natural language" (arXiv 2602.07488, 2026) | Shannon's question asked again with models as the estimator, and a data exponent predicted from two statistics of text with no free parameter | |
| Section 2, tokens and embeddings | Gage, "A New Algorithm for Data Compression", The C Users Journal (February 1994), with Sennrich, Haddow and Birch, "Neural Machine Translation of Rare Words with Subword Units", ACL 2016 (arXiv 1508.07909) | byte-pair encoding as a compressor, then as the answer to the open vocabulary |
| Vershynin, High-Dimensional Probability: An Introduction with Applications in Data Science, second edition (Cambridge University Press, 2026) | concentration of the norm, near-orthogonality and the Johnson-Lindenstrauss lemma, with proofs; the geometry every later Section leans on | |
| Mikolov, Yih and Zweig, "Linguistic Regularities in Continuous Space Word Representations", NAACL 2013, with Levy and Goldberg, "Neural Word Embedding as Implicit Matrix Factorization", NeurIPS 2014 | linear structure found in embeddings, then explained as the factorization of a matrix of co-occurrence statistics | |
| Press and Wolf, "Using the Output Embedding to Improve Language Models", EACL 2017 (arXiv 1608.05859), and Inan, Khosravi and Socher, "Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling", ICLR 2017 (arXiv 1611.01462) | the case for one matrix at both ends, which small models still make and large ones dropped | |
| Yang, Dai, Salakhutdinov and Cohen, "Breaking the Softmax Bottleneck: A High-Rank RNN Language Model", ICLR 2018 (arXiv 1711.03953) | the rank argument: a softmax over a state of width d cannot express every table of conditional distributions | |
| Park, Choe and Veitch, "The Linear Representation Hypothesis and the Geometry of Large Language Models", ICML 2024 (arXiv 2311.03658) | what "a concept is a direction" means precisely, and in which inner product | |
| Pagnoni and others, "Byte Latent Transformer: Patches Scale Better Than Tokens" (arXiv 2412.09871, 2024) | a model that reads bytes and forms its own patches: the alternative to a fixed tokenizer | |
| Godey and Artzi, "Lost in Backpropagation: The LM Head is a Gradient Bottleneck", COLM 2026 (arXiv 2603.10145) | the bottleneck seen from the side of the gradient, most of whose norm is lost at the output layer | |
| Section 3, attention | Bahdanau, Cho and Bengio, "Neural Machine Translation by Jointly Learning to Align and Translate", ICLR 2015 (arXiv 1409.0473) | attention as the removal of a bottleneck: the decoder looks back at every source position |
| Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser and Polosukhin, "Attention Is All You Need", NeurIPS 2017 (arXiv 1706.03762) | the mechanism as still used: scaled dot products, heads, and the block around them | |
| Ramsauer and co-authors, "Hopfield networks is all you need" (arXiv 2008.02217, 2020), with Schlag, Irie and Schmidhuber, "Linear Transformers are secretly fast weight programmers", ICML 2021 (arXiv 2102.11174) | two readings of one formula: retrieval from an associative memory, and a weight matrix written while the model runs | |
| Geshkovski, Letrouit, Polyanskiy and Rigollet, "A mathematical perspective on Transformers", Bulletin of the American Mathematical Society 62 (2025) (arXiv 2312.10794) | tokens as interacting particles on a sphere, and clustering as what a stack of attention does in the long run | |
| Dao, Fu, Ermon, Rudra and Ré, "FlashAttention: fast and memory-efficient exact attention with IO-awareness", NeurIPS 2022 (arXiv 2205.14135), through Shah, Bikshandi, Zhang, Thakkar, Ramani and Dao, "FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision", NeurIPS 2024 (arXiv 2407.08608) | why exact attention won over its approximations: count the traffic between levels of memory, not the multiplications | |
| Ainslie, Lee-Thorp, de Jong, Zemlyanskiy, Lebrón and Sanghai, "GQA: training generalized multi-query Transformer models from multi-head checkpoints", EMNLP 2023 (arXiv 2305.13245) | the cache as the cost of decoding, and the sharing of keys and values that 23 of 24 recent models adopt in some form | |
| Wang, Shi and Fox, "Test-time regression: a unifying framework for designing sequence models with associative memory" (arXiv 2501.12352, 2025) | one frame for attention, linear attention and recurrent layers, each a regression fitted to the context; the bridge to Definition 11.2.4 | |
| Wen, Huang, Meng, He and Li, "Mixture-of-Top-k Attention: Efficient Attention via Scalable Fast Weights", NeurIPS 2026 (arXiv 2602.01219) | the newest sparse attention, built on the reading of attention as fast weights | |
| Section 4, position | Shaw, Uszkoreit and Vaswani, "Self-attention with relative position representations", NAACL 2018 (arXiv 1803.02155) | the first move from where a token is to how far apart two tokens are |
| Dai, Yang, Yang, Carbonell, Le and Salakhutdinov, "Transformer-XL: attentive language models beyond a fixed-length context", ACL 2019 (arXiv 1901.02860) | relative positions obtained by expanding the attention score into its terms | |
| Su, Lu, Pan, Murtadha, Wen and Liu, "RoFormer: Enhanced Transformer with Rotary Position Embedding" (arXiv 2104.09864, 2021) | the rotation that makes the score depend on the offset alone; the scheme of 21 of 24 recent models | |
| Press, Smith and Lewis, "Train short, test long: attention with linear biases enables input length extrapolation" (arXiv 2108.12409, 2021) | a bias with no embedding at all, and length generalization posed as an experiment | |
| Haviv, Ram, Press, Izsak and Levy, "Transformer language models without positional encodings still learn positional information", Findings of EMNLP 2022 (arXiv 2203.16634), with Kazemnejad, Padhi, Natesan Ramamurthy, Das and Reddy, "The impact of positional encoding on length generalization in transformers", NeurIPS 2023 (arXiv 2305.19466) | what the causal mask already gives: a decoder with no position signal can recover position | |
| Peng, Quesnelle, Fan and Shippole, "YaRN: efficient context window extension of large language models" (arXiv 2309.00071, 2023) | how trained frequencies are rescaled to read contexts longer than those of training; in most configurations of 2026 | |
| Barbero, Vitvitskyi, Perivolaropoulos, Pascanu and Veličković, "Round and round we go! What makes rotary positional encodings useful?" (arXiv 2410.06205, 2024) | what a trained model does with the frequencies it is given | |
| Gelberg, Eguchi, Akiba and Cetin, "Extending the context of pretrained LLMs by dropping their positional embeddings" (arXiv 2512.12167, 2025) | the latest turn of the argument: a position signal removed after training | |
| Section 5, the block | He, Zhang, Ren and Sun, "Deep residual learning for image recognition", CVPR 2016 (arXiv 1512.03385), and "Identity mappings in deep residual networks", ECCV 2016 (arXiv 1603.05027) | the skip, and the argument that nothing should stand on it |
| Ba, Kiros and Hinton, "Layer normalization" (arXiv 1607.06450, 2016), with Zhang and Sennrich, "Root mean square layer normalization", NeurIPS 2019 (arXiv 1910.07467) | the normalization of sequence models, and the observation that the re-centering can go | |
| Xiong and others, "On layer normalization in the transformer architecture", ICML 2020 (arXiv 2002.04745) | the gradient argument for normalizing before the sub-layer | |
| Hendrycks and Gimpel, "Gaussian error linear units (GELUs)" (arXiv 1606.08415, 2016), with Shazeer, "GLU variants improve transformer" (arXiv 2002.05202, 2020) | the smooth gate of 2019 and the gated unit of 2023 | |
| Geva, Schuster, Berant and Levy, "Transformer feed-forward layers are key-value memories", EMNLP 2021 (arXiv 2012.14913), and Geva, Caciularu, Wang and Goldberg, "Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space", EMNLP 2022 (arXiv 2203.14680) | what the MLP stores: keys that match patterns of the input, values that promote tokens | |
| Zhang, Dauphin and Ma, "Fixup initialization: residual learning without normalization", ICLR 2019 (arXiv 1901.09321) | initialization as signal propagation: start every block near the identity | |
| Dong, Cordonnier and Loukas, "Attention is not all you need: pure attention loses rank doubly exponentially with depth" (arXiv 2103.03404, 2021) | what skips and MLPs are for: without them a stack of attention collapses to rank one | |
| Zhu, Chen, He, LeCun and Liu, "Transformers without normalization", CVPR 2025 (arXiv 2503.10622), and Kimi Team, "Attention residuals" (arXiv 2603.15031, 2026) | the two oldest components reopened: normalization replaced by a tanh, the skip by attention over depth | |
| Section 6, optimization | Kingma and Ba, "Adam: A Method for Stochastic Optimization", ICLR 2015 (arXiv 1412.6980) | the algorithm and its two promises: a step that does not change when the gradient is rescaled, and that is bounded by the learning rate |
| Loshchilov and Hutter, "Decoupled weight decay regularization", ICLR 2019 (arXiv 1711.05101) | why a penalty inside Adam is not weight decay | |
| Kunstner, Chen, Lavington and Schmidt, "Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Be", ICLR 2023 (arXiv 2304.13960), with Kunstner, Yadav, Milligan, Schmidt and Bietti, "Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Models", NeurIPS 2024 (arXiv 2402.19449) | the noise explanation tested and set aside, and Zipf's law put in its place | |
| Cohen, Kaur, Li, Kolter and Talwalkar, "Gradient descent on neural networks typically occurs at the edge of stability", ICLR 2021 (arXiv 2103.00065) | sharpness rises to two over the learning rate and stays there; the classical condition on the step does not describe training | |
| Kosson, Messmer and Jaggi, "Rotational equilibrium: how weight decay balances learning across neural networks", ICML 2024 (arXiv 2305.17212) | under normalization, weight decay sets an angle of rotation for each step and penalizes nothing | |
| Bernstein and Newhouse, "Old Optimizer, New Norm: An Anthology" (arXiv 2409.20325, 2024), with Jordan and others, "Muon: an optimizer for hidden layers in neural networks" (a write-up of 8 December 2024) | optimizers as steepest descent under a norm; the spectral norm gives Muon | |
| Liu and others, "Muon is scalable for LLM training" (arXiv 2502.16982, 2025) | Muon taken beyond the race in which it was found | |
| Wen, Hall, Ma and Liang, "Fantastic pretraining optimizers and where to find them" (arXiv 2509.02046, 2025) | the audit: every optimizer tuned, and the gain measured against scale | |
| Section 7, training the model | Penedo, Kydlíček, Ben Allal, Lozhkov, Mitchell, Raffel, von Werra and Wolf, "The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale" (arXiv 2406.17557, 2024) | a web corpus with its whole pipeline documented, and the subset filtered by a classifier that gpt-pp is trained on |
| Eldan and Li, "TinyStories: How Small Can Language Models Be and Still Speak Coherent English?" (arXiv 2305.07759, 2023) | a corpus built so that a model of a few million parameters can write; gpt-pico's data | |
| Li, Fang, Smyrnis and 56 others, "DataComp-LM: In search of the next generation of training sets for language models" (arXiv 2406.11794, 2024) | data curation as a controlled benchmark, with rankings of datasets that carry over from small models to large | |
| Sardana, Portes, Doubov and Frankle, "Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws", ICML 2024 (arXiv 2401.00448) | why small models that are deployed are trained far past the compute-optimal point | |
| Ben Allal, Lozhkov, Bakouch and 19 others, "SmolLM2: When Smol Goes Big" (arXiv 2502.02737, 2025) | a published recipe for a model of gpt-pp's size, at thousands of tokens for each parameter | |
| Hägele, Bakouch, Kosson and three others, "Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations", NeurIPS 2024 (arXiv 2405.18392) | the constant rate with a short cooldown, so that the length of a run need not be fixed in advance | |
| Holtzman, Buys, Du, Forbes and Choi, "The Curious Case of Neural Text Degeneration", ICLR 2020 (arXiv 1904.09751) | why the most probable text is not the text to emit, and sampling from the nucleus | |
| nanoGPT and nanochat (A. Karpathy) and modded-nanogpt (K. Jordan and contributors), public code as read on 14 November 2026 | GPT-2 reproduced in the open, then a model of its size trained to a fixed loss in under a minute on eight GPUs; the background the series' recipe starts from | |
| Section 8, what training learns | Saxe, McClelland and Ganguli, "Exact solutions to the nonlinear dynamics of learning in deep linear neural networks", ICLR 2014 (arXiv 1312.6120) | plateaus and sudden drops solved exactly: each mode is learned along its own sigmoid |
| Belkin, Hsu, Ma and Mandal, "Reconciling modern machine learning practice and the bias-variance trade-off", PNAS 2019 (arXiv 1812.11118), with Hastie, Montanari, Rosset and Tibshirani, "Surprises in high-dimensional ridgeless least squares interpolation", Annals of Statistics 2022 (arXiv 1903.08560) | the double descent curve, and its closed form in linear regression | |
| Chizat, Oyallon and Bach, "On lazy training in differentiable programming", NeurIPS 2019 (arXiv 1812.07956), with Yang and Hu, "Feature learning in infinite-width neural networks", ICML 2021 (arXiv 2011.14522) | the kernel regime as a choice of scaling, and the dichotomy between it and the learning of features | |
| Olsson, Elhage, Nanda and others, "In-context Learning and Induction Heads", Transformer Circuits Thread 2022 (arXiv 2209.11895) | the phase change: a bend in the loss curve at which a circuit for in-context learning forms | |
| Power, Burda, Edwards, Babuschkin and Misra, "Grokking: generalization beyond overfitting on small algorithmic datasets" (arXiv 2201.02177, 2022), with Nanda, Chan, Lieberum, Smith and Steinhardt, "Progress measures for grokking via mechanistic interpretability", ICLR 2023 (arXiv 2301.05217) | generalization long after memorization, then the network opened to show the algorithm growing all along | |
| Belrose, Pope, Quirke, Mallen and Fern, "Neural networks learn statistics of increasing complexity", ICML 2024 (arXiv 2402.04362) | language models pass through a unigram stage and a bigram stage; "order" made precise | |
| Morris, Sitawarin, Guo, Kokhlikyan, Suh, Rush, Chaudhuri and Mahloujifar, "How much do language models memorize?" (arXiv 2505.24832, 2025) | capacity measured, about 3.6 bits for each parameter, with generalization beginning once it is full | |
| Zhang, Saxe and Latham, "Saddle-to-saddle dynamics explains a simplicity bias across neural network architectures", ICLR 2026 (arXiv 2512.20607) | the staircase of 2014 extended to attention, where heads are added one at a time | |
| Section 9, circuits | Olah, Cammarata, Schubert, Goh, Petrov and Carter, "Zoom in: an introduction to circuits", Distill (10 March 2020) | the program: features, circuits between them, and the claim that both recur across models |
| Elhage, Nanda, Olsson and others, "A Mathematical Framework for Transformer Circuits", Transformer Circuits Thread (2021) | the residual stream as a linear channel, the expansion into paths, the two circuits of a head, composition | |
| nostalgebraist, "interpreting GPT: the logit lens", LessWrong (31 August 2020), with Belrose, Ostrovsky, McKinney and others, "Eliciting latent predictions from transformers with the tuned lens" (arXiv 2303.08112, 2023) | a prediction read out of every layer | |
| Meng, Bau, Andonian and Belinkov, "Locating and Editing Factual Associations in GPT", NeurIPS 2022 (arXiv 2202.05262) | causal tracing: restore one activation of a corrupted run and see what comes back | |
| Wang, Variengien, Conmy, Shlegeris and Steinhardt, "Interpretability in the wild: a circuit for indirect object identification in GPT-2 small" (arXiv 2211.00593, 2022) | the largest circuit established by hand in a real model: 26 heads in seven classes | |
| Heimersheim and Nanda, "How to use and interpret activation patching" (arXiv 2404.15255, 2024) | the manual of the method: what a patching result shows and what it does not | |
| Conmy, Mavor-Parker, Lynch, Heimersheim and Garriga-Alonso, "Towards automated circuit discovery for mechanistic interpretability", NeurIPS 2023 (arXiv 2304.14997) | the search for a circuit made into an algorithm | |
| Yin and Steinhardt, "Which attention heads matter for in-context learning?", ICML 2025 (arXiv 2502.14010), and Turner, Wu and Batson, "Characterizing interference weights in a tiny language model", Transformer Circuits Thread (21 August 2026) | two corrections from the frontier: induction heads are not the whole of in-context learning in large models, and weights read globally mix signal with interference | |
| Section 10, features | Olshausen and Field, "Emergence of simple-cell receptive field properties by learning a sparse code for natural images", Nature 381 (1996) | sparse coding, from which every dictionary of the Section descends |
| Elhage, Hume, Olsson and others, "Toy Models of Superposition", Transformer Circuits Thread (2022; arXiv 2209.10652) | more features than dimensions, as a phase diagram and as geometry | |
| Bricken, Templeton, Batson and others, "Towards Monosemanticity: Decomposing Language Models With Dictionary Learning", Transformer Circuits Thread (4 October 2023) | a sparse autoencoder on a one-layer model: the existence proof, and the recipe | |
| Templeton and others, "Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet", Transformer Circuits Thread (May 2024), with Gao and others, "Scaling and evaluating sparse autoencoders", ICLR 2025 (arXiv 2406.04093) | dictionaries on a production model, steering by clamping a feature, and how a dictionary is scored | |
| Dunefsky, Chlenski and Nanda, "Transcoders find interpretable LLM feature circuits", NeurIPS 2024 (arXiv 2406.11944) | replace an MLP by a sparse imitation, and attribution factors into an activation times a fixed weight | |
| Ameisen, Lindsey, Pearce and others, "Circuit Tracing: Revealing Computational Graphs in Language Models", and Lindsey, Gurnee, Ameisen and others, "On the Biology of a Large Language Model", Transformer Circuits Thread (27 March 2025) | attribution graphs: the method with its limits stated, and its case studies | |
| Sharkey and others, "Open problems in mechanistic interpretability" (arXiv 2501.16496, 2025), with Smith and others, "Negative Results for SAEs On Downstream Tasks and Deprioritising SAE Research", AI Alignment Forum (26 March 2025) | the agenda and the dissent | |
| Bushnaq and others, "Interpreting language model parameters" (arXiv 2610.20748, October 2026), with Arora, Wu, Steinhardt and Schwettmann, "Language model circuits are sparse in the neuron basis", ICML 2026 (arXiv 2601.22594) | the two newest challengers to the feature as the unit: components of the parameters, and plain neurons |
Fifteen works of 2025 and 2026. These are the works to read for what Section 11.2 describes; each was read on arXiv or on its publisher's site on 14 November 2026, and what each reports is its authors' measurement unless the entry says otherwise.
- Geiping, McLeish, Jain, Kirchenbauer and others, "Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach", arXiv 2502.05171 (February 2025). The clearest statement that depth can be chosen at test time, at 3.5 billion parameters and 800 billion tokens.
- Zhu, Wang, Hua, Zhang and others, "Scaling Latent Reasoning via Looped Language Models", arXiv 2510.25741 (October 2025). Looping carried through pretraining on 7.7 trillion tokens, with experiments that separate what a model knows from how it manipulates it.
- Jolicoeur-Martineau, "Less is More: Recursive Reasoning with Tiny Networks", arXiv 2510.04871 (October 2025), read with the analysis of the ARC Prize team (15 August 2025) and Roye-Azar and others, arXiv 2512.11847. A recursion of 7 million parameters, and a lesson in finding where a result comes from.
- Hao, Sukhbaatar, Su, Li and others, "Training Large Language Models to Reason in a Continuous Latent Space", arXiv 2412.06769 (December 2024), with Zhu, Hao, Hu, Jiao and others, "Reasoning by Superposition: A Theoretical Perspective on Chain of Continuous Thought", arXiv 2505.12514 (May 2025). Reasoning carried in hidden states, and the theory of why that can search in parallel.
- Nie, Zhu, You, Zhang and others, "Large Language Diffusion Models", arXiv 2502.09992 (February 2025). Masked diffusion trained from scratch at 8 billion parameters: the challenge to the assumption that a language model must write left to right.
- Yang, Kautz and Hatamizadeh, "Gated Delta Networks: Improving Mamba2 with Delta Rule", ICLR 2025 (arXiv 2412.06464), with Kimi Team, "Kimi Linear: An Expressive, Efficient Attention Architecture", arXiv 2510.26692 (October 2025). The recurrent layer of Definition 11.2.4, and the first claim that it beats full attention.
- Merrill, Li, Romero, Svete and others, "Olmo Hybrid: From Theory to Practice and Back", arXiv 2604.03444 (April 2026). A fully open, controlled comparison at 7 billion parameters, which names the gap between its theory and its result and then argues its way across it.
- Lahoti, Li, Chen, Wang, Bick, Kolter, Dao and Gu, "Mamba-3: Improved Sequence Modeling using State Space Principles", ICLR 2026 (arXiv 2603.15569). The answer of the state-space line: complex-valued states for tracking state, several inputs and outputs for the hardware.
- Qiu, Wang, Zheng, Huang, Wen and others, "Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free", arXiv 2505.06708 (May 2025). A change of one line, tested on more than 30 variants, that accounts for attention sinks and entered the blocks of several model families.
- DeepSeek-AI, "DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence", arXiv 2606.19348 (2026). The report of a flagship of 2026: compressed and sparse attention, constrained hyper-connections, and Muon at 1.6 trillion parameters.
- Kimi Team, "Attention Residuals", arXiv 2603.15031 (March 2026). The residual stream redesigned as attention over depth, now inside a released model.
- Cheng, Tian, Zeng, Dai and others, "Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models", arXiv 2601.07372 (January 2026). Lookup as a second way, beside experts, to hold parameters that cost no compute.
- Hwang, Wang and Gu, "Dynamic Chunking for End-to-End Hierarchical Sequence Modeling", arXiv 2507.07955 (July 2025). A model that learns its own segmentation of bytes: the most direct challenge to the premise of Section 2.
- Yue, Chen, Lu, Zhao, Wang and others, "Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?", arXiv 2504.13837 (April 2025). The argument, from accuracy over many samples, that training on checkable rewards sharpens what a base model could already produce. It is disputed, and it frames the debate on the stages after pretraining, which this series only touches (Section 7).
- Lindsey, Gurnee, Ameisen, Chen, Pearce and others, "On the Biology of a Large Language Model", Transformer Circuits Thread (27 March 2025). Attribution graphs on a production model: a rhyme planned before its line, reasoning in two steps, and the limits of the method in its authors' words.
The models behind the examples
Every worked example of Sections 1 to 10 is computed on a model trained for this series. This section is their recipe and their model card: the data and the tokenizers, each network, the optimizer, every run with its cost, how the models are scored, and the commands that reproduce all of it. The code is one repository, github.com/d-/gpt-pp: a transformer in which every component can be switched between its historical variants, the optimizers, a trainer for one to four GPUs, the evaluation, and the tools of Sections 9 and 10. It was written for the series; the public reference implementations, nanoGPT, its successor nanochat and the speedrun on GPT-2 small, were read, and neither copied nor run.
Data. Two public corpora, each read in one pass. TinyStories is a corpus of short stories in the vocabulary of a small child, written by a language model; we use its second version, written by GPT-4 alone. FineWeb-EDU is web text kept by a classifier trained to recognize educational pages; we use the publisher's random sample of about ten billion tokens. One seeded permutation of its documents fixes the held-out set, a prefix of it, and the training set, the rest in that order (for TinyStories the held-out set is the publisher's own validation file); the tokenizer learns from training documents only. Pages recur across crawls of the web, so every training document of FineWeb-EDU that has the text or the address of a held-out document is removed, 960 in all; near copies under different addresses cannot be excluded.
| TinyStories | FineWeb-EDU | |
|---|---|---|
| source on the Hugging Face Hub | roneneldan/TinyStories at f54c09fd2331, the two files of version 2 | HuggingFaceFW/fineweb-edu at 87f09149ef47, the sample of 10 billion tokens |
| license | CDLA-Sharing-1.0 | ODC-By 1.0 and the terms of use of Common Crawl |
| vocabulary of the tokenizer | 4,096 | 32,768 |
| training tokens | 542,905,221 | 9,896,904,130 |
| training documents | 2,717,495 | 9,661,492 |
| held-out tokens | 5,481,226 | 10,000,019 |
| held-out documents | 27,630 | 9,649 |
| bytes for each token, held-out | 4.0278 | 4.6283 |
| tokens in a document: mean, median | 198.8, 177 | 1,023.4, 620 |
| documents longer than 1,024 tokens, and their share of all tokens | 0.02%, 0.1% | 26.8%, 64.8% |
| entropy of single tokens, bits per byte | 2.128 | 2.394 |
| 4-gram model with Kneser-Ney smoothing, bits per byte | 0.897 | 1.577 |
| xz at its highest setting, bits per byte | 1.510 | 2.176 |
One property of FineWeb-EDU is worth knowing before asking gpt-pp for anything: the byte { does not occur once in its 9.9 billion training tokens, } occurs 7,395 times and the tab character 42 times. The model has never seen an opening brace.
Tokenizers. Both are byte-level byte-pair encodings (Section 2.1): the 256 bytes, then the merges in the order learned, then five special tokens, of which pretraining uses one, the beginning of a document, and the chat stage the other four. gpt-pico's tokenizer learns 3,835 merges from the training split of TinyStories; gpt-pp's learns 32,507 from two billion characters of training documents. A reference implementation in plain Python learns exactly the same merges from the same text and encodes every held-out document to the same tokens. On FineWeb-EDU the 32,768 tokens write the text in 0.75% fewer tokens than the 50,257 of GPT-2.
Two recipes, three sizes. A model is a recipe at a size. The 2019 recipe is the design published with GPT-2, trained the way nanoGPT reproduces it. The 2026 recipe is the default, the last row of the second table of Section 11.1. Each switch between the two can be flipped alone, which is how the ladder of Sections 3 to 6 measures one change at a time. The sizes are gpt-pico, the one that runs in the page; the 8-layer model of the ladder; and 12 layers, where the series reserves the name gpt-pp for the model in the 2026 recipe.
| model | layers | width | heads | context | vocabulary | embedding | unembedding | attention | MLP | parameters |
|---|---|---|---|---|---|---|---|---|---|---|
| gpt-pico | 6 | 128 | 4 | 256 | 4,096 | 524,288 | 524,288 | 393,216 | 786,432 | 2,228,224 |
| gpt-pico in the 2019 recipe | 6 | 128 | 4 | 256 | 4,096 | 524,288 | tied | 396,288 | 790,272 | 1,746,944 |
| the 8-layer model, 2026 recipe | 8 | 512 | 8 | 512 | 32,768 | 16,777,216 | 16,777,216 | 8,388,608 | 16,777,216 | 58,720,256 |
| the 8-layer model, 2019 recipe | 8 | 512 | 8 | 512 | 32,768 | 16,777,216 | tied | 8,404,992 | 16,797,696 | 42,259,456 |
| gpt-pp | 12 | 768 | 12 | 1,024 | 32,768 | 25,165,824 | 25,165,824 | 28,311,552 | 56,623,104 | 135,266,304 |
| the 12-layer model, 2019 recipe | 12 | 768 | 12 | 1,024 | 32,768 | 25,165,824 | tied | 28,348,416 | 56,669,184 | 111,008,256 |
| attention-only, one layer | 1 | 256 | 8 | 256 | 4,096 | 1,048,576 | 1,048,576 | 262,144 | none | 2,359,296 |
| attention-only, two layers | 2 | 256 | 8 | 256 | 4,096 | 1,048,576 | 1,048,576 | 524,288 | none | 2,621,440 |
The two recipes share a trunk: at 12 layers attention and MLPs hold 84.9 million parameters in both, to within biases, and the model of 2026 is larger by its second vocabulary matrix alone. With GPT-2's vocabulary of 50,257 tokens, the 2019 recipe at 12 layers has exactly the 124,439,808 parameters of GPT-2 small, which a test asserts.
In the code. A preset is the defaults with two dictionaries laid over them, one for the size and one for the recipe, and the 2026 recipe is the empty dictionary. It is preset, with the two tables it reads, in gpt-pp's config.py.
SIZES: dict[str, dict] = {
"pico": dict(n_layer=6, d_model=128, n_head=4, block_size=256, vocab_size=4096),
"ladder": dict(n_layer=8, d_model=512, n_head=8, block_size=512, vocab_size=32768),
"base": dict(n_layer=12, d_model=768, n_head=12, block_size=1024, vocab_size=32768),
}
# The 2026 recipe is the set of defaults of ModelConfig; the 2019 recipe is written as its differences from it.
RECIPES: dict[str, dict] = {
"gptpp": dict(),
"gpt2-2019": dict(pos="learned", norm="layernorm", norm_placement="pre", embed_norm=False, act="gelu",
qk_norm=False, bias=True, tie_embeddings=True, logit_softcap=0.0, init="gpt2"),
}
...
def preset(name: str) -> ModelConfig:
"""The model of a preset, e.g. preset("gptpp-base") or preset("gpt2-2019-ladder")."""
recipe, size = _split(name)
return ModelConfig().replace(**SIZES[size], **RECIPES[recipe])
Its six presets, a recipe joined to a size from gptpp-pico to gpt2-2019-base, are the first six rows of the table above; ladder is the size of the 8-layer model. Nine entries of gpt2-2019 differ from the defaults: they are the switches that the ladder flips, one a rung except for the two of the untied, normalized embedding. The tenth, norm_placement="pre", restates a default, since GPT-2 already normalized before the sub-layer.
The 2026 recipe in one paragraph. Queries and keys are rotated by their positions, with base 10,000, and nothing is added to the stream for position. Normalization is \(x / \sqrt{\mathrm{mean}(x^2) + 10^{-6}}\), without parameters, before attention, before the MLP, after the last block and on the embedding. The MLP is a squared ReLU of hidden width \(4d\). The score of every head is \(12 \cos(q, k)\). There are no biases. The embedding starts at \(\mathcal N(0, 1)\) and the separate unembedding at \(\mathcal N(0, 0.001^2)\); the matrices that write to the stream start at zero, so that the untrained model is embed, then unembed. The logits pass through \(15 \tanh(z / 15)\). Sections 3 to 6 give the source and the reason for each, and where the sources disagree on a value, it was chosen by short runs.
Example 11.4.1 (the runs that chose the defaults). The setup: the 8-layer model on 50 million tokens of FineWeb-EDU, 763 steps of 65,536 tokens, about four minutes of one card; one setting changed at a time from the defaults of the moment; measured: the loss on 1,048,576 held-out tokens, in nats a token.
| the 8-layer model, 50M tokens | held-out loss |
|---|---|
| the defaults of the moment: score scale 8, vocabulary rates 0.245 and 0.0049 | 4.0707 |
| the same with a second seed | 4.0701 |
| score scale 11.5 | 4.0517 |
| score scale 16 | 4.0563 |
| score scale learned for each head, starting at 8 | 4.0477 |
| no normalization of the embedding | 4.0774 |
| Muon at 0.01 in place of 0.02 | 4.0906 |
| Muon at 0.04 | 4.1247 |
| vocabulary rates 0.367 and 0.0098 | 4.0608 |
| the defaults now: score scale 12, vocabulary rates 0.367 and 0.0098 | 4.0452 |
| the 2019 recipe, AdamW at 0.0006 | 5.1250 |
| the 2019 recipe at 0.0012 | 4.8127 |
| the 2019 recipe at 0.0024, its rate on the ladder | 4.6294 |
| the 2019 recipe at 0.0048 | 5.0548 |
The two seeds of these short runs end 0.0006 nats apart; fourteen later pairs under the same recipe differ by 0.001 to 0.008. A scale of 8, which is \(\sqrt{d_{\mathrm{head}}}\) and gives scores of unit variance at initialization, is worse than 11.5 or 16 by 0.014 to 0.019; learned, the scales end at 14.9 on average. The last four rows matter more than the others for every comparison in the series: at the rate of nanoGPT, 0.0006, set for a run of 300 billion tokens, the 2019 recipe would look 0.50 nats worse on a short budget than it is at its best rate. At 12 layers the main rate of each recipe was swept later, in the race and after it (below, and Section 7). ◁
What earned its place, and what did not. The recipe was fixed before it was measured. The ladder goes from the recipe of 2019 to that of 2026 one change at a time, on 100 million tokens, two seeds a rung; its rungs under AdamW share one rate, tuned for the first. The swaps take one choice out of the finished recipe, on 50 million tokens; the recipe and thirteen of them have a second seed. Under the 2026 recipe two seeds differ by 0.001 to 0.008 nats, so 0.009 or more is established; under AdamW, which trains the ladder's first seven rungs, they differ by up to 0.018, so less than about 0.02 between two single runs is not.
The ladder falls from 4.135 to 3.814 on the means of its seeds, by 0.321, and two rungs carry 0.266 of that. Its side of 2019 sits about 0.03 too high: with a warmup of 32 million tokens in place of 8 million, each of its seven rungs under AdamW ends 0.017 to 0.042 lower in the one seed run, and the whole way is 0.286 where that seed had 0.328. Rotary positions are worth 0.110, and no less with the longer warmup; replacing them in the finished recipe costs 0.110 with a learned table and 0.059 with sinusoids. The training recipe of 2026 is worth 0.155, or 0.140 against the rung with the longer warmup, and it is fourteen settings, not one. With AdamW kept on the hidden matrices inside it, Muon in AdamW's place accounts for 0.083 and 0.086 in two seeds, and the rest, which is the vocabulary table's own rate, no decay on it and the schedule, for about 0.07, or 0.06 against the fairer rung.
Three choices read differently in the two settings. The squared ReLU gained 0.033 over GELU on the ladder; in the finished recipe GELU is level with it, ReLU is 0.017 worse and the field's gate 0.009 better, each in two seeds. The zero initialization gained 0.027 on the ladder, only 0.008 with the longer warmup, and nothing as a swap. The normalization of queries and keys is the reverse: 0.011 on the ladder, 0.021 with the longer warmup, and 0.031 as a swap in both seeds. Normalization as such is not in question: with none the loss stopped being a number at the 80th step, and layer norm after the sub-layer is 0.107 above the same norm before it.
Two simplifications cost about 0.02 each. The norm without parameters cost 0.031 on the ladder, and in the finished recipe a learned gain and layer norm are each 0.020 better, in both seeds. Dropping the biases was worth nothing on the ladder, 0.010 down in one seed and 0.010 up in the other; put back into the finished recipe they are 0.021 and 0.013 better.
The rest is small or nothing. The embedding's norm is worth 0.011 in two seeds. Untied tables with it gain 0.011 on the ladder in both seeds, though the tied swap, which changes four things at once, loses nothing in its one seed. The cap on the logits is worth nothing that these runs can see.
So a second version of the recipe would give the normalization its gain back and the linear maps their biases, after measuring whether the two add, would weigh the gate's 0.009 against the simplicity of the square, and would keep the zero initialization and the cap for their simplicity alone. All of it is evidence about 8 layers and at most 100 million tokens. On the 12-layer model two settings were swept at 100 million tokens, one seed each: Muon at 0.01, 0.02 and 0.04 ends at 3.653, 3.654 and 3.679, and the rotary base at 1,000, 10,000 and 100,000 at 3.673, 3.654 and 3.649. No component was swapped there.
Optimizer and schedule.
| model | optimizer | learning rate | warmup | then | weight decay | clipping | tokens a step |
|---|---|---|---|---|---|---|---|
| gpt-pico | Muon on hidden matrices; AdamW (0.8, 0.95) on the two vocabulary matrices | 0.02; 0.735 and 0.0196 | 2M tokens | constant, then linear over the last 65% of the budget to 5% of the peak | none | none | 65,536 |
| gpt-pico in the 2019 recipe | AdamW (0.9, 0.95) on everything | 0.006 | 4M tokens | cosine to 10% | 0.1 on matrices | norm 1 | 65,536 |
| the 8-layer model, 2026 recipe | as gpt-pico | 0.02; 0.367 and 0.0098 | 2M tokens | as gpt-pico | none | none | 65,536 |
| the 8-layer model, 2019 recipe | AdamW (0.9, 0.95) | 0.0024 | 8M tokens; 32M in the seven rungs run again | cosine to 10% | 0.1 on matrices | norm 1 | 65,536 |
| gpt-pp | Muon on hidden matrices; AdamW (0.8, 0.95) on the two vocabulary matrices | 0.02; 0.3 and 0.008 | 2M tokens | constant, then linear over the last 65% of the budget to 5% of the peak | none | none | 147,456 |
| the 12-layer model, 2019 recipe | AdamW (0.9, 0.95) | four rates, 0.0006 to 0.0048 | 32M tokens, and 8M in two runs | cosine to 10% | 0.1 on matrices | norm 1 | 49,152 |
In the code. The table is one function, whose 2026 branch names only the two vocabulary rates: the rest of the recipe is the defaults of TrainConfig. It is train_preset, with the table of batches it reads, in gpt-pp's config.py.
# One card sees 65,536 tokens a step at every size.
_BATCH: dict[str, dict] = {
"pico": dict(device_batch=256, tokens=100e6, eval_tokens=5e6),
"ladder": dict(device_batch=128, tokens=150e6, eval_tokens=10e6),
"base": dict(device_batch=64, tokens=1e9, eval_tokens=20e6),
}
...
def train_preset(name: str, d_model: Optional[int] = None) -> TrainConfig:
...
recipe, size = _split(name)
width = SIZES[size]["d_model"] if d_model is None else d_model
if recipe == "gptpp":
# Muon on the hidden matrices at the usual 0.02. The AdamW rates of the embedding and the unembedding
# are nanochat's values at width 768 (0.3 and 0.008) scaled by sqrt(768 / width), as nanochat scales them.
scale = (768 / width) ** 0.5
recipe_values = dict(lr_embedding=round(0.3 * scale, 4), lr_unembedding=round(0.008 * scale, 5))
else:
...
rate = {"pico": 6e-3, "ladder": 2.4e-3, "base": 6e-4}[size]
warmup = {"pico": 4e6, "ladder": 8e6, "base": 32e6}[size]
recipe_values = dict(optimizer="adamw", lr=rate, lr_embedding=None, lr_unembedding=None, lr_gains=None,
weight_decay=0.1, wd_embedding=0.1, wd_unembedding=0.1, beta1=0.9, beta2=0.95,
eps=1e-8, grad_clip=1.0, schedule="cosine", warmup_tokens=warmup, final_lr_frac=0.1)
return TrainConfig().replace(**_BATCH[size], **recipe_values)
scale is the \(\sqrt{768/d}\) of the caption, and the two rounded products are the table's pairs of rates. Muon's 0.02 has no such rule, and no rate follows the batch: chosen at 65,536 tokens a step, the rates were used unchanged at the main run's 147,456 (Section 6). For the 12-layer model _BATCH asks for 64 sequences on a card, which do not fit in its 32 GB; the main run set 48. The argument d_model is a repair: without it a width set on the command line kept the rates of the preset's own width, the first row of the table that ends this section.
The rates of the groups belong to the recipe as much as the rule does: six runs under Adam and AdamW with one rate for every group, the way of 2019, ended 0.96 to 1.04 nats above the recipe at 50 million tokens.
Every schedule is a function of the tokens seen, so none depends on the batch or on the number of cards. The data are read once, shard by shard, the windows of a shard in an order shuffled by the seed. The seed fixes the initialization and that order. Two runs with one seed are still not the same bit for bit, because some kernels of the cards are not deterministic: two such pairs of full runs ended 0.0023 and 0.0003 nats apart, inside the spread between seeds (Section 7).
Arithmetic and hardware. Matrix products run in bfloat16 (Section 6.7); the residual stream, the normalizations of the blocks, the cap on the logits and the loss stay in float32, and queries and keys are normalized in bfloat16, the type in which the attention kernel reads them. The forward pass is compiled and attention goes through the fused kernel of Section 3.6. The machine has four cards of 32 GB, which the driver names only as "NVIDIA Graphics Device", each capped at 165 W, with 96 processor cores and 360 GB of memory, PyTorch 2.9.1 and CUDA 13.0. One card holds at most 56 sequences of 1,024 tokens of the 12-layer model, in 26.0 GB. The main run used three cards with 48 sequences each, 22.4 GB a card, and took 3.20 hours at 263,957 tokens a second.
Every run. The record below gives the configurations, training histories, costs and final losses of the runs, apart from the 36 short probes that settled the defaults.
chart held-out loss in bits a byte against tokens seen (both axes logarithmic), a curve a run list the runs by family, with tokens seen and final loss family runs tokens card-hours bits a byte, best to worst the main run 2 4.1B 13.0 0.943 to 0.974 the race 11 1.1B 3.3 1.137 to 1.295 the ladder 30 3.0B 5.3 1.188 to 1.292 the swaps 57 2.9B 4.2 1.254 to 2.399 the optimizers 57 3.8B 3.6 0.649 to 2.216 the grid 23 2.0B 1.9 1.137 to 1.516 the earlier grid 19 1.7B 2.0 1.151 to 1.531 the small models 6 1.9B 0.4 0.552 to 0.764 the chat stage 6 174M 1.0 scored in nats a token only 211 runs: together 20.6B tokens in 34.6 card-hours (with 36 short probes of the recipe that are not drawn: 247 runs, 22.3B tokens, 36.0 card-hours). The main run saw 3.0B tokens in 9.6 card-hours and ends at 0.943 bits a byte (3.025 nats a token). (the live figure needs JavaScript and loads about 162 kB of data)
| family | runs | model | data and tokens a run | seeds | cards a run | card-hours |
|---|---|---|---|---|---|---|
| gpt-pp, the main run | 1 | 12 layers, 768 wide | FineWeb-EDU, 2,999,992,320 | one | 3 | 9.60 |
| the same recipe and shape with its schedule laid out for 1.08 billion tokens | 1 | 12 layers, 768 wide | FineWeb-EDU, 1,079,967,744 | one | 3 | 3.44 |
| the chat stage on the main model: two stages at two rates, four probes | 6 | 12 layers, 768 wide | conversations, 67,108,864 a stage and 9,994,240 a probe | one | 1 | 0.99 |
| gpt-pico in both recipes | 2 | 6 layers, 128 wide | TinyStories, 540,016,640, one pass | one | 1 | 0.14 |
| the ladder: ten rungs from the 2019 recipe to the 2026 recipe | 20 | 8 layers, 512 wide | FineWeb-EDU, 100,007,936 | two for each rung | 1 | 4.09 |
| the ladder's seven rungs under AdamW again, with a warmup of 32M tokens in place of 8M | 7 | 8 layers, 512 wide | FineWeb-EDU, 100,007,936 | one | 1 | 0.85 |
| the ladder's step to the 2026 training recipe, split: AdamW in place of Muon, two rates | 3 | 8 layers, 512 wide | FineWeb-EDU, 100,007,936 | two at the better rate | 1 | 0.36 |
| one change at a time from the 2026 recipe | 47 | 8 layers, 512 wide | FineWeb-EDU, 50,003,968 | two for the recipe and thirteen changes, one for the rest | 1 | 3.58 |
| warmup and the place of the norm in the 2019 recipe, at two step sizes | 10 | 8 layers, 512 wide | FineWeb-EDU, 50,003,968 | one | 1 | 0.63 |
| the rule for the hidden matrices changed: Muon, AdamW, Adam, Lion and SGD, several rates each, under the recipe's schedule and, but for Adam, under their own warmup and clipping | 33 | 8 layers, 512 wide | FineWeb-EDU, 50,003,968 | one, and a second for the best AdamW | 1 | 2.14 |
| Adam and AdamW with one rate for every parameter group | 6 | 8 layers, 512 wide | FineWeb-EDU, 50,003,968 | one | 1 | 0.37 |
| the race: the 2019 recipe against the 2026 recipe, and the 2026 recipe at two more rates and two more rotary bases | 11 | 12 layers, 768 wide | FineWeb-EDU, 100M | one | 1 | 3.32 |
| the grid for the scaling law: six sizes, four budgets, the largest size at three of them | 23 | from 4 layers, 128 wide to 12 layers, 768 wide; context 512 | FineWeb-EDU, 25M to 200M | one | 1 | 2.50 |
| the grid again with the recipe's rule for the vocabulary rates, at the five widths other than 512 | 19 | from 4 layers, 128 wide to 12 layers, 768 wide; context 512 | FineWeb-EDU, 25M to 200M | one | 1 | 1.36 |
| gpt-pico under SGD, clipped SGD, sign descent and Adam | 18 | 6 layers, 128 wide | TinyStories, 100,007,936 | one | 1 | 1.06 |
| attention-only models of one and two layers | 4 | 256 wide | TinyStories, plain and with planted repeats, 200,015,872 | one | 1 | 0.22 |
| short runs that settled the defaults and measured throughput | 36 | all three sizes | FineWeb-EDU and TinyStories, 50M to 60M in most | one, and one repeat | 1 or 2 | 1.31 |
| all runs of the trainer | 247 | 36.0 | ||||
| beside them: modular addition | 1 | one layer | 30% of the 12,769 sums modulo 113, 40,000 steps on all of them | one | 1 | 0.17 |
| beside them: 41 dictionaries and six transcoders | 47 | on activations of gpt-pico and of gpt-pp | 33,554,432 tokens a dictionary, 40,000,000 for the transcoders | one, and one repeat | 1 | 1.94 |
What is one seed. The ladder has two seeds for each of its ten rungs; the recipe at the ladder's size, thirteen of its swaps, the best AdamW and the split of the ladder's seventh step have a second; every other run has one. Two seeds differ in the initialization and in the order of the data, and are scored on the same held-out tokens. The two yardsticks above come from these pairs: fourteen under the 2026 recipe, 0.005 nats apart in root mean square, and the ladder's ten, 0.008, with the two seeds of the best AdamW 0.025 apart. At 12 layers there is no such measurement at all.
Evaluation.
Remark 11.4.2 (two numbers with one name). The trainer's monitor is the mean loss over the first 1,048,576 tokens of the held-out stream, cut into windows of the training length, in the arithmetic of training; every curve, every rung and every swap quotes it, in nats a token. The tables that set our models beside released ones quote bits per byte of whole held-out documents under the protocol below, in float32 with TensorFloat-32 products (Section 7). For gpt-pico the first is 1.5398 nats a token, 0.5515 bits per byte, and the second 0.541, on other stories, in windows that begin where a story begins. For gpt-pp the two are 0.9429 and 0.9322. The trainer's bits per byte divide by a constant, the 4.6283 bytes a token of the whole held-out file; the tokens it scores carry 4.6353, with which 0.9429 would read 0.9415.
The protocol has four steps. Each document is encoded by the model's own tokenizer. Its tokens are cut into consecutive windows of at most 1,024 tokens, fewer for a model with a shorter context, each begun by the model's own beginning-of-text token. Every token of every document is scored exactly once, given the earlier tokens of its window. Bits per byte is the sum of the scores, in bits, over the sum of the documents' lengths in bytes: the length of a real code for the text (Section 1.3), independent of the tokenizer, which is what lets a vocabulary of 4,096 tokens be set beside one of 151,936. Windows that do not overlap charge the first tokens of each window for their short context, about 5% for GPT-2 small on long articles, and every model pays it alike.
There are four held-out sets: the 60 test articles of WikiText-103; the first 2,000 of our held-out stories; the first 1,000 of our held-out documents of FineWeb-EDU; and eight essays published on the web in October and November 2026, the one set that no released model can have read. Standard errors take documents as the sampled units, and a difference between two models is computed on the same documents, where its error is several times smaller. Held-out and training documents are disjoint, but web text repeats itself: 4.1% of the held-out blocks of 64 tokens occur word for word in the shards the main run read, and without them its held-out loss of 3.025 nats a token becomes 3.040; without the two scored documents that have a near copy in training, 0.9322 becomes 0.9324.
Nine benchmarks are taken by log-likelihood, without examples in the prompt: an item is answered by scoring each option as a continuation, by log-probability for each byte where the options differ in length and in total where they do not. The summary of a model is the mean over the nine of its accuracy above chance, as a share of the distance from chance to a perfect score.
The same code scores the released models, and it reproduces what was published for GPT-2 small: a perplexity of the last word of 35.11 on LAMBADA against the paper's 35.13, and HellaSwag within 0.55 points of the standard harness. It does not reproduce the published figure of the largest GPT-2 on WikiText-103, 20.24 against 17.48, and we do not know why.
| model | parameters | trained on | FineWeb-EDU | WikiText-103 | TinyStories | fresh essays |
|---|---|---|---|---|---|---|
| GPT-2 small (2019) | 124M | WebText, an amount not disclosed | 1.007 | 1.072 | 0.856 | 1.297 |
| SmolLM2 135M (2025) | 135M | 2 trillion tokens, FineWeb-EDU among them | 0.857 | 0.972 | 0.692 | 1.207 |
| TinyStories 33M (2023) | 69M | TinyStories | 2.875 | 3.380 | 0.436 | 2.716 |
| gpt-pico | 2.2M | 540M tokens of TinyStories | 0.541 | |||
| gpt-pp | 135M | 3.0B tokens of FineWeb-EDU | 0.932 | 1.130 | 0.864 | 1.325 |
gpt-pp is ahead of GPT-2 small on one set of the four, the held-out documents of its own kind of text: 0.932 against 1.007 bits a byte, a difference of 0.075 ± 0.003 on the same documents. On the three others it is behind: by 0.058 ± 0.018 on WikiText-103, by 0.008 ± 0.001 on the stories and by 0.028 ± 0.009 on the fresh essays, the one set that neither model can have read. It is behind SmolLM2 135M on all four, by 0.075 to 0.173; that model has the same size and read about 670 times as many tokens, FineWeb-EDU among them. On the nine benchmarks the summary is 25.8 for gpt-pp, 24.5 for GPT-2 small and 35.3 for SmolLM2 135M.
Against nanochat, the second project the series started from, nothing was run like for like. Its table of January 2026 gives a model of 12 layers, 768 wide and 135 million parameters 0.9825 bits a byte on its own held-out split of FineWeb-EDU after 1.08 billion tokens.That project's write-up of a series of models, 7 January 2026, and its development log, as read on 14 November 2026. Later entries of the log at this depth read 0.965 to 0.977, single runs; all are quoted here as that project reports them. The main run of gpt-pp passed that budget at 1.011 and 1.009, the trainer's figures on its own held-out windows of the same corpus at the evaluations on either side, its rate still within 2% of the peak of a schedule laid out for 3.0 billion tokens; it ended at 0.943. The same recipe and shape with its schedule laid out for 1.08 billion tokens ends at 0.9742, under the 0.9825 and inside the 0.965 to 0.977 of that log's later entries. These figures are not one measurement: the held-out documents, the tokenizers, the contexts (2,048 tokens there, 1,024 here) and the scoring code all differ, and our own two ways of scoring one model are 0.011 apart. That table was made before that project added what Section 11.1 lists as left out of this recipe, and before it retuned its batch for each depth. Nothing measured here sets one recipe against the other under one protocol: at equal tokens the two are level as far as these figures can say, and neither is shown ahead.
The card of gpt-pp.
- What it is. A transformer that reads left to right, of 12 layers, 768 wide, with 12 heads, a context of 1,024 tokens and a vocabulary of 32,768: 135,266,304 parameters, of which 50,331,648 are the two vocabulary matrices.
- Training. One pass over the first 2,999,992,320 tokens of the training stream of FineWeb-EDU, 22 tokens for each parameter, in 20,345 steps and 3.20 hours on three cards; the loss is the cross-entropy of the next token, 3.0249 nats a token on held-out windows at the end; no dropout; one seed.
- What it is for. The worked example of this series: a model to measure, to ablate and to open. It continues English text of the kind found on educational web pages.
- What it is not. It continues text and does not follow instructions; its output is filtered by nothing but the choice of its training data; it has never seen an opening brace; it reads 1,024 tokens.
- The chat stage. A second checkpoint is the same model trained for 32 minutes more on one card, on 67 million tokens of conversations from three subsets of SmolTalk (Apache 2.0), with the loss counted only on what the assistant writes. It answers a question in the form of an answer and stops by itself. On held-out conversations the assistant's tokens went from 2.52 to 1.44 nats each, and plain text paid for it: 0.932 to 1.213 bits a byte on the held-out documents. Of 20 questions fixed beforehand, read once by a language model against criteria written down first, 8 answers were right, 10 wrong and 2 off the subject at temperature 0.7, and none of the sums. Section 7 has the account, and one of the commands below starts a conversation with it.
- Weights. Those of gpt-pico ship with these pages, in 8 bits. Every weight of the three models the examples ran on, in float32 as trained, is on the Hugging Face Hub in the format of transformers: gpt-pp-base, gpt-pp-chat and gpt-pico. The format is transformers' NanoChat with a few lines of code of its own for the gain on the queries, so they load with
trust_remote_code=True; the conversion checks the logits against gpt-pp's own. The commands below retrain them.
gpt-pico in the page. The figures that run a model run gpt-pico. Its weights reach the page once, as 2,288,640 bytes: matrices in 8 bits with one scale for each row. On the trainer's 1,048,576 held-out tokens, in float32, the 8-bit weights score 1.5417 nats a token against 1.5395 for the weights as trained. The page computes the forward pass of Section 5 in plain JavaScript, at 1.3 to 1.6 milliseconds a token on the machine that built these pages; a figure shows the reader's own rate once gpt-pico has run in their browser.
Commands. From a clone of the code, with the versions it pins:
git clone https://github.com/d-/gpt-pp && cd gpt-pp
pip install -r requirements.txt # torch 2.9.1 and the pinned versions
export GPTPP_WORK=$HOME/gpt-work # data, checkpoints and logs go here
PY=python
# data and tokenizers: about a quarter of an hour on 96 cores
$PY scripts/prepare_tinystories.py --download-only
$PY scripts/train_tokenizer.py --name bpe4k --vocab-size 4096 \
--input $GPTPP_WORK/data/raw/tinystories/TinyStoriesV2-GPT4-train.txt \
--separator "<|endoftext|>"
$PY scripts/prepare_tinystories.py --tokenizer bpe4k
$PY scripts/prepare_fineweb.py --tokenizer bpe32k
$PY scripts/corpus_stats.py
# pico in both recipes: four to five minutes of one card each
$PY -m gptpp.train --preset gptpp-pico --data tinystories/bpe4k \
--run pico --tokens 540e6 --eval-tokens 10e6 --save-log 20
$PY -m gptpp.train --preset gpt2-2019-pico --data tinystories/bpe4k \
--run pico-2019 --tokens 540e6 --eval-tokens 10e6 --save-log 20
# gpt-pp, the main run, on three cards
$PY -m torch.distributed.run --standalone --nproc_per_node=3 \
-m gptpp.train --run gptpp-base --preset gptpp-base --tokens 3.0e9 \
--set device_batch=48 --eval-tokens 50e6 --save-log 24 \
--save-steps 2000,4000,6000,8000,10000,12000,14000,16000,18000,20000
# every other run: the job lists in experiments/, one job a card;
# a finished run is skipped when its list is started again
for jobs in experiments/*.jsonl; do
$PY scripts/job_queue.py $jobs --gpus 0,1,2,3
done
# one rung by hand: the block of 2019 changed step by step, then the 2026 training recipe
$PY -m gptpp.train --preset gpt2-2019-ladder --train-preset gptpp-ladder \
--run ladder-7-muon-s0 --tokens 100e6 --seed 0 --eval-tokens 10e6 \
--set bias=false pos=rope norm=rmsnorm_fixed act=relu2 qk_norm=true \
init=zero_proj
# evaluation: bits per byte, then the nine benchmarks, into one file
$PY -m gptpp.evaluate --model $GPTPP_WORK/ckpt/gptpp-base \
--docs fineweb-edu,wikitext103,tinystories \
--out results/eval/gptpp-base.json
$PY -m gptpp.benchmarks --model $GPTPP_WORK/ckpt/gptpp-base \
--tasks all --out results/eval/gptpp-base.json
$PY scripts/eval_released_models.py --table
# write with a trained model
$PY -m gptpp.generate --run pico --prompt "Once upon a time" \
--temperature 0.8
# the chat stage on the trained model: 32 minutes of one card
$PY scripts/prepare_chat.py
$PY -m gptpp.chat train --base gptpp-base --run gptpp-chat --lr-scale 0.8
$PY -m gptpp.chat --run gptpp-chat --prompt "What is a prime number?"
$PY -m pytest -q
Every run writes its configuration, its log and a summary beside its checkpoints, and the small files are kept with the code, so each number of the series can be traced to the run that produced it. The essays of the fourth held-out set are not distributed.
The code, its version, and what is known to be wrong with it
Every listing of the series is the repository's own lines at commit 1890c9a of 14 November 2026, and a script checks each against the file it names. The table below is what a reader of that code should know before trusting a line of it. It was compiled by reading the code at that commit, its tests, the records of the runs and the review of the experiments; nothing was run for it.
The pin. The 211 runs on record all name the same versions, PyTorch 2.9.1 built for CUDA 13.0 under Python 3.13.9; the newest PyTorch in November 2026 is 2.15.0. The release notes of 2.10 to 2.15, versions the series did not test, remove or rename no function that gpt-pp calls, and four of the changes they list would matter to someone who moves the code. The fused attention function gains a kernel in 2.10 and prefers another on some cards in 2.15, and by its documentation the output may differ with the kernel. From 2.11 the fused AdamW uses fused multiply-add instructions, which can change the last bits of the vocabulary matrices' updates. Version 2.15 rewrites the nccl backend through which several cards exchange gradients, and it is built for CUDA 13.2 and 13.4 only: CUDA 13.0 ends with 2.14. PyTorch's own torch.optim.Muon, present at the pin, is not the one used (Section 6).
| what | status | effect | explained in |
|---|---|---|---|
| The rule that scales the two vocabulary rates by \(\sqrt{768/d}\) was not applied when the width was set on the command line, so every size of the first scaling grid trained at the rates of width 512 | fixed in 1890c9a; the 19 runs at other widths were made again |
at width 128 the rule is worth 0.049 nats after 25 million tokens and 0.008 after 200 million, and nothing beyond noise at widths 384 and 768; the fitted exponent for parameters fell from 0.56 to 0.52 | Section 7 |
| Sinusoidal positions in the 2026 recipe were added before the embedding's norm and faded as the token rows grew: their share of the energy entering the norm fell from 33% to 1.4% | fixed in 8bfeabc by a switch, pos_after_norm, that is off by default: pos=sinusoidal alone still fades |
the first run ended at 4.1777, 0.133 behind rotary and behind the learned table; with the switch, at 4.1050 and 4.1019 in two seeds, 0.060 and 0.058 behind rotary and ahead of the table. Every sinusoid number of the series is the second | Section 4 |
| The first comparison of rules for the hidden matrices ran every rule under the schedule chosen with Muon: a warmup of 2 million tokens and no clipping | fixed by new runs in 8bfeabc and 1890c9a: AdamW, Lion and SGD with a warmup of 8 million tokens and clipping at norm one; Adam was not run again |
the best AdamW went from 4.214 to 4.181, so Muon's advantage is 0.136 and 0.163 nats in two seeds, not 0.169; Lion and SGD stay 0.44 and 0.49 behind | Section 6 |
| Several comparisons are not single changes: the ladder's step to the 2026 training recipe changes fourteen settings at once, the next rung also gives the embedding its own rate, initialization and 16.8 million parameters, rungs 1 to 6 keep the rate tuned for the first, and two swaps change two and four switches | open, stated as a limit | with a warmup of 32 million tokens each rung of 2019 is 0.017 to 0.042 nats better and that step is 0.140, not 0.157, in the one seed; Muon in AdamW's place is 0.083 and 0.086 of it in two seeds | Section 5 |
| In the 2019 recipe at the ladder's step of 65,536 tokens, post-norm falls within 40 steps to predicting token frequencies, with the warmup of 8 million tokens and without it | open | 7.690 nats, where the entropy of the frequencies is 7.701; it trains at steps of 8,192 tokens with a warmup (4.537, pre-norm 4.581) and at the ladder's step with a warmup of 32 million tokens and rate 0.0006 (5.255) | Section 5 |
| The trainer's monitor is not the evaluation protocol: it reads the first 1,048,576 held-out tokens in windows cut without regard to documents, in bfloat16, with the beginning-of-document token among the targets | open | 0.9429 against 0.9322 bits per byte for the main model; in windows of 512 tokens the same model reads 3.086 nats, against 3.025 in windows of 1,024 | Section 7 |
| The trainer's bits per byte divide by the bytes per token of the whole held-out file, 4.6283 for FineWeb-EDU and 4.0278 for TinyStories, where the tokens it scores carry 4.6353 and 4.0640 | open | 0.9429 would read 0.9415 for the main run and 0.552 would read 0.547 for gpt-pico; comparisons between runs are unaffected | Section 7 |
| Held-out text is disjoint from training by document, not by content: 4.1% of the held-out blocks of 64 tokens occur verbatim in the shards the main run read, and two of the 1,000 documents scored have a near copy there | open | without the repeated blocks 3.025 nats become 3.040; without the two documents 0.9322 bits per byte become 0.9324 | Section 7 |
| The evaluation code does not reproduce two published results: the perplexity of the largest GPT-2 on WikiText-103, and LAMBADA for the three larger sizes of GPT-2 | open; the cause of the first is not known | 20.24 against the published 17.48; LAMBADA 10% to 17% above the published figures; for GPT-2 small, 35.11 against 35.13 | Section 7 |
| Some scores were not taken under the protocol or on clean text: the run laid out for 1.08 billion tokens has only the trainer's number, the fresh set is eight essays by one author, and 695 of the 2,000 stories scored are in the training file of TinyStories 33M | open | 0.9742 bits per byte is the trainer's figure; TinyStories 33M scores 0.442 on the stories it shares and 0.431 on the others | Section 7 |
| Under bfloat16 the model's two forward paths, the fused one that trains and the explicit one that every analysis and cached decoding use, differ by the rounding of bfloat16 | open | with one seed at temperature 0.8 the two write the same 101 tokens of 1,024 and then part; in float64 their logits agree to \(10^{-5}\) | Section 7 |
| The cache of keys and values grows by concatenation at every token, in every layer, and bought no speed at this size | open; the cause was not profiled | 167 tokens a second with the cache and 176 without it, for the 12-layer model on one card | Section 7 |
| The preset of the 12-layer model asks for 64 sequences of 1,024 tokens on a card, which do not fit in 32 GB | open; the main run set 48 | the main run's step is three cards of 48 sequences, 147,456 tokens | Section 7 |
| A run cannot be resumed: a checkpoint holds the weights and the model's configuration and no optimizer state, and an interrupted run leaves weights without a summary | open | none measured | Section 7 |
A run records the commit it was launched at and nothing about uncommitted changes. Six runs, the main run and gpt-pico's two among them, record no commit; the second scaling grid records 5ecb05b and ran with the fix committed afterwards as 1890c9a; the repaired sinusoid run records 702f0af, its switch committed as 8bfeabc |
open | none on a number: the configuration each run writes, with its rates and switches, is the evidence of what ran | here |
Without the variable GPTPP_WORK the package looks for data and checkpoints under an absolute path of the machine it was built on, and the tests look for a text file under another |
open; the commands above set the variable | none on results | here |
| An attribution graph holds the attention patterns and the denominators of the normalizations at their values on the prompt and has no edge through the scores; it is built only for pre-norm models with MLPs and a norm that divides | open | on the figure's prompt the rank correlations between the graph and measured removals are 0.27 and 0.54; an intervention confirms the graph for three of the 12 prompts | Section 10 |
| The transcoders are one for each MLP, not across layers, and train at eight times the rate of the rule | open | in blocks 2 to 6 of gpt-pico 50% to 61% of the latents never fire on held-out text, against 6% in block 1 | Section 10 |
| "Dead" has two meanings for a latent: in training, no firing for 1,000,000 tokens; in evaluation, no firing on the held-out tokens read, 9.8 million for gpt-pp, 5.1 million for gpt-pico and 262,144 for the transcoders, not the \(10^7\) of the definition | open | "no latent is dead" is by the second meaning | Section 10 |
| The circuit quantities (query-key and output-value matrices, composition, the copying statistic) are computed from the raw matrices of any model, and equal what the model computes only without MLPs, norms, biases, normalized queries and keys, and the cap | open | none on a number: they are read on the attention-only models, never on gpt-pp or gpt-pico | Section 9 |
| Activation patching has three kinds of site at one position, the stream entering a block, a layer's whole attention output and its MLP output: no head is patched alone, and the two prompts must have the same number of tokens | open | heads enter the worked example by attribution only | Section 9 |
| The auxiliary loss that revives dead latents differs between the dictionaries and the transcoders: the first feed the dead latents' pre-activations to the decoder as they are, negative ones included, the second through a ReLU | open | not measured | Section 10 |
The copy test reads its checkpoints from a fixed path of the series' machine whatever GPTPP_WORK says; the review of the experiments did not cover the attention-only models or the rare-token runs, and each of those has one seed |
open | the copy test runs on the series' machine only | Section 8 |
| The unit of the score scale changed during the build: the records of the short runs that chose the defaults count it in units of \(\sqrt{d_{\mathrm{head}}}\) and say 1, 1.44 and 2 where Example 11.4.1 says 8, 11.5 and 16 | open in the records; the checkpoints of those runs were deleted, since today's code would misread them | none on a published number | Section 3 |
| The reference tokenizer and the production tokenizer cut text with two regular-expression engines, whose Unicode tables classify 17,480 code points differently | open | none on this data: none of those characters occurs in either corpus | Section 2 |
The repository's documents were behind its code: the README's command for the main run lacked the batch and the checkpoint steps and its list of files put sign descent under optim.py, the design notes gave the two-card test a tolerance of \(10^{-4}\) throughout and two runs with one seed an agreement that no record supports, and the evaluation notes said that no checkpoint of the series existed |
corrected in the repository with this list, except one comment in bpe.py that calls the four chat tokens "reserved for a later chat stage", which exists |
none on results | here |
The script that runs the lists of jobs was named queue.py; Python puts a script's directory first on its module path, so anything run from that directory imported it in place of the standard library's queue |
fixed in 8bfeabc: it is job_queue.py, and the scripts take their own directory off the path |
none on results | here |
| The stored record of a dictionary's activations was reused whenever its file was newer than the dictionary's, though evaluating a dictionary rewrites its file unchanged, so dates did not say which weights a record was made with | fixed in 702f0af: a record carries a signature of the weights |
none known on a published number | Section 10 |
The full list, 90 items with the evidence for each, quirks and the library's behaviors included, is the file docs/KNOWN-ISSUES.md of the repository.
Quirk. The tests of the model and of the optimizers run in float64 on the processor with every parameter randomized, over 31 variants of the model, none of which sets pos_after_norm, the switch of the table's second row, which has a test file of its own. The test on two cards is skipped where there is one. Two tolerances are wide: two cards against one agree to 0.0001 nats in the loss at initialization and to 0.005 afterwards, and one seed run twice to 0.02.
With more compute
The models of the series were trained in one day on four cards. With more, this is what we would do next, in this order.
- Seeds. Three for every rung and every swap, where there are now two for a rung, for the recipe and for thirteen of its swaps, and one for the rest; and more under AdamW, where two seeds have ended 0.025 nats apart. It is the cheapest item here: a swap is four minutes of a card.
- The recipe's own findings at 12 layers. The learned gain, the biases, the gate and the tied table, each swapped into the 12-layer model on a billion tokens: whether what the 8-layer model said in Section 11.4 holds where the main model lives.
- A longer race and a larger grid. Both recipes on 500 million tokens or more, where each has so far been tuned in its main rate at 100 million only, and a third entrant with the components of the races that gpt-pp left out; and for the scaling grid, since run again with the recipe's rule for the vocabulary rates, two larger budgets, so that the fit can carry a floor (Section 1.7).
- A longer main run. gpt-pp saw 22 tokens for each parameter; its training stream holds enough for 73 in one pass, and the open models of its size are trained on thousands: two trillion tokens for one of 135 million parameters, about 250 days of a card at the rate of Example 11.2.7. Between 73 and 15,000 lies a question that Section 7 cannot answer: where the curve of this recipe bends.
- The inside of gpt-pp. Dictionaries of several sizes and kinds on several layers, attribution graphs on the 12-layer model and not only on gpt-pico, path patching of an induction circuit in it, and its capacity in bits for each parameter.
- After pretraining. The chat stage is half an hour on one card, and plain text paid for it. A stage that keeps more of what pretraining learned comes first; after it, preference learning and rewards from a checker, where the debates of 2025 and 2026 begin.