Keys and values

Draft

The worked example of this series is gpt-pp, a language model with the shape of GPT-2 small. It starts from nanoGPT's recipe for reproducing that model and adds switches to compare newer design and training choices. Its smallest size, gpt-pico, has 2.2 million parameters, was trained on short stories for small children, and is small enough to run in this page.

story    ... a time, there was a little boy named Tim. Tim loved to eat yummy food. One day, he saw a big cake on the table. It was the best cake he ever saw. [Tim]
         the bracketed token is being written: token 42

next     the likeliest tokens here, at temperature 0.8
         Tim         82.6 %  █████████████████████████  drawn
         He           8.5 %  ███
         It           5.1 %  ██
         But          2.7 %  █
         The          0.7 %

depth    first guess   probability it gives the token written
         embedding  The            0.0 %
         block 1    The            0.0 %
         block 2    The            0.0 %
         block 3    The            0.0 %
         block 4    The            0.0 %
         block 5    ↵              0.1 %
         block 6    Tim           82.6 %  █████████████████████████

read     Tim (30 back, weight 0.51), . (1 back, weight 0.43), saw (2 back, weight 0.35); the first by block 6, head 4
(the live figure needs JavaScript; it loads about 0.1 MB with the recorded stories and about 2.3 MB of weights when a control is first used)
gpt-pico, a transformer of 2.2 million parameters trained on children's stories, writing in the page. Green: the token being written, and its probability in both charts. Tinted and tied to it: the earlier tokens attention read; the number is the weight. Tokens appear as they are written. First chart: the eight likeliest tokens, at the chosen temperature. Second chart: each depth's first guess, the stream read out after the embedding and after each block. Pick or type an opening, set the temperature, write, step, pick any token. Before any press the story is a recording, one seed.

The recorded story begins "Once upon a time, there was a little boy named Tim. Tim loved to eat yummy food. One day, he saw a big cake on the table. It was the best cake he ever saw." The next token is Tim again, drawn with a probability of 82.6%. Reading out the model's working state at each depth gives different predictions: after the embedding and after each of the first four blocks the likeliest next token is The, after the fifth a line break, and only after the sixth and last is it Tim. The strongest attention weight is on the earlier name: head 4 of block 6 reads, with weight 0.51, the first Tim, 30 tokens back.

A language model of this kind is a function from a sequence of tokens to a probability distribution over the next token. A token is a piece of text a few bytes long, one of a fixed vocabulary of 4,096 pieces in gpt-pico's case, and the output is one probability for each piece. The parameters are fitted to a large body of text by a single criterion: the token that did come next should have been given a high probability.A spelling pattern and a recurring factual error can both help predict the next token. The loss rewards predictability; it does not separately test whether the text is true. The criterion has a second reading. A predictor can drive a compressor, and the better it predicts a text the shorter the file, so training makes the training text as short as it can.

To write, the model runs in a loop: draw a token from the distribution, append it to the sequence, ask again. Nothing in the training mentions stories, grammar or names. The function is a transformer that reads only leftward, and a model of this kind, trained once on text without labels and then used for whatever that text prepared it for, is a GPT, a generative pre-trained transformer. That prediction alone might be enough was the claim of the paper that introduced GPT-2 in 2019.

While it is a large step from the well-posed setup described above to the messiness of “language in the wild”, Weston (2016) argues, in the context of dialog, for the need to develop systems capable of learning from natural language directly and demonstrated a proof of concept – learning a QA task without a reward signal by using forward prediction of a teacher’s outputs. While dialog is an attractive approach, we worry it is overly restrictive. The internet contains a vast amount of information that is passively available without the need for interactive communication. Our speculation is that a language model with sufficient capacity will begin to learn to infer and perform the tasks demonstrated in natural language sequences in order to better predict them, regardless of their method of procurement. If a language model is able to do this it will be, in effect, performing unsupervised multitask learning. We test whether this is the case by analyzing the performance of language models in a zero-shot setting on a wide variety of tasks.

—A. Radford, J. Wu, R. Child, D. Luan, D. Amodei and I. Sutskever, "Language models are unsupervised multitask learners" (2019).OpenAI, 2019, Section 2. The largest model of the paper has 1.5 billion parameters. Its smallest, GPT-2 small, has 12 layers, a width of 768 and 124 million parameters, and is the yardstick of this series.

A year later the claim was tested at 175 billion parameters, on tasks the model had never been trained for.

Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task. While typically task-agnostic in architecture, this method still requires task-specific fine-tuning datasets of thousands or tens of thousands of examples. By contrast, humans can generally perform a new language task from only a few examples or from simple instructions – something which current NLP systems still largely struggle to do. Here we show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art fine-tuning approaches. Specifically, we train GPT-3, an autoregressive language model with 175 billion parameters, 10x more than any previous non-sparse language model, and test its performance in the few-shot setting. For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model. GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks, as well as several tasks that require on-the-fly reasoning or domain adaptation, such as unscrambling words, using a novel word in a sentence, or performing 3-digit arithmetic. At the same time, we also identify some datasets where GPT-3’s few-shot learning still struggles, as well as some datasets where GPT-3 faces methodological issues related to training on large web corpora. Finally, we find that GPT-3 can generate samples of news articles which human evaluators have difficulty distinguishing from articles written by humans. We discuss broader societal impacts of this finding and of GPT-3 in general.

—T. B. Brown and others, "Language models are few-shot learners" (2020).NeurIPS 2020 (arXiv 2005.14165), abstract. Section 1 follows the line from the letter tables of 1948 to this paper.

Three formulas carry the series. They may look bare here; each is earned in a later Section. Write \(x_1, \dots, x_n\) for the tokens of a text, \(x_{<t}\) for those before the \(t\)-th, and \(q_\theta\) for a model with parameters \(\theta\). Inside the model every token has a vector of \(d\) numbers, and the vectors of a sequence are the rows of a matrix \(X\), the residual stream.

\[\begin{aligned} &\text{the objective:}\quad && \mathcal L(\theta) = -\frac1n \sum_{t=1}^{n} \ln q_\theta(x_t \mid x_{<t}), \\ &\text{the residual stream:}\quad && Y = X + \mathrm{Attn}\bigl(N(X)\bigr), \quad X^{+} = Y + \mathrm{MLP}\bigl(N(Y)\bigr), \\ &\text{attention:}\quad && o_i = W_O \sum_{j \le i} A_{ij}\, v_j, \quad A_{ij} = \frac{e^{s_{ij}}}{\sum_{l \le i} e^{s_{il}}}, \quad s_{ij} = \frac{q_i^{\top} k_j}{\sqrt{d_k}} . \end{aligned}\]

The first line is the training loss: the average surprisal of a text under the model, in nats per token (natural logarithms; divide by \(\ln 2\) for bits). Training minimizes it, and by Theorem 1.3.2 the surprisal of a whole text is, to within two bits, the length of the file to which the model compresses that text. The second line is one block (Definition 5.1.1). Each block updates the stream by addition: attention adds to each row what it gathered from the current and earlier rows, a small network called the MLP (multilayer perceptron) adds what it computes from that row alone, and \(N\) normalizes what each of them reads. A model is the block applied \(L\) times, between an embedding that turns tokens into rows and an unembedding that turns the last rows into the probabilities of the first line (Section 2). The third line is one attention head (Definition 3.2.1), the only part that moves information between positions. With \(x_i\) the normalized row of token \(i\), position \(i\) sends a query \(q_i = W_Q x_i\); every position \(j \le i\) offers a key \(k_j = W_K x_j\) and a value \(v_j = W_V x_j\); the scores of the query against the keys become weights that sum to one; and the head writes the weighted sum of the values.For the quantity in the first line, C. E. Shannon, "A mathematical theory of communication", Bell System Technical Journal 27 (1948), 379–423 and 623–656, where the entropy of a source is defined. For the additions of the second, K. He, X. Zhang, S. Ren and J. Sun, "Deep residual learning for image recognition", CVPR 2016 (arXiv 1512.03385), with the normalization placed as GPT-2 placed it. For the third, A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser and I. Polosukhin, "Attention is all you need", NeurIPS 2017 (arXiv 1706.03762), Section 3.2.1, with the mask of its Section 3.1, which lets a position read only what came before it.

In the code. The three formulas are three short functions of gpt-pp's model.py, shown as they stand. The whole model is GPT.forward.

def forward(self, idx: Tensor, targets: Optional[Tensor] = None) -> Tensor:
    """idx (B, T) token ids -> logits (B, T, V); or, with targets (B, T), the mean cross-entropy in nats
    (a target of -1 is ignored)."""
    T = idx.size(1)
    self.extend_positions(T)
    x = self.embed_tokens(idx)
    rot = self.rotation(0, T)
    bias = self.alibi[:, :T, :T] if self.cfg.pos == "alibi" else None
    for block in self.blocks:
        x = block(x, rot, bias)
    logits = self.unembed_logits(self.final_norm(x))
    if targets is None:
        return logits
    return F.cross_entropy(logits.view(-1, logits.size(-1)), targets.reshape(-1), ignore_index=-1)

One block is Block.forward.

def forward(self, x: Tensor, rot: Rotation, bias: Optional[Tensor]) -> Tensor:
    if self.pre_norm:
        x = x + self.drop(self.attn(self.norm1(x), rot, bias))
        if self.mlp is not None:
            x = x + self.drop(self.mlp(self.norm2(x)))
    ...
    return x

Its attention, all heads at once, is CausalSelfAttention.forward.

def forward(self, x: Tensor, rot: Rotation, bias: Optional[Tensor]) -> Tensor:
    """The fast path: softmax(q k^T / sqrt(dh) + mask) v in one fused kernel (FlashAttention where it applies).
    `bias` is the ALiBi table, which contains the causal mask; without it the kernel masks causally itself."""
    B, T, _ = x.shape
    q, k, v = self.project(x)
    q, k = self.position(q, k, rot)
    k, v = self.share(k, v)
    z = F.scaled_dot_product_attention(
        q, k, v,
        attn_mask=None if bias is None else bias.to(q.dtype),
        is_causal=bias is None,
        dropout_p=self.dropout if self.training else 0.0,
    )
    return self.o(z.transpose(1, 2).reshape(B, T, -1))

The last line of GPT.forward is the first formula, averaged over every position of a batch (\(q_\theta\) is the softmax of logits, taken inside F.cross_entropy), and the loop above it applies the second once for each block. The two additions of Block.forward are that formula's two halves, with norm1 and norm2 for \(N\); the branch left out puts the norm after each addition (Section 5). In CausalSelfAttention.forward, x holds the normalized rows \(x_i\), project applies \(W_Q\), \(W_K\) and \(W_V\) and self.o is \(W_O\); the weights \(A_{ij}\) never appear, because one call to PyTorch's F.scaled_dot_product_attention computes the scores, hides the later positions, takes the softmax and sums the values. What the formulas do not show sits in the helpers: position normalizes queries and keys and turns them by the rotation rot, and unembed_logits caps the logits, three of the newer choices in the table below; self.drop, share and bias do nothing in either of its recipes.

blueprint  the 2026 recipe at 12 layers, 768 wide; * marks what differs from the 2019 recipe

text “Water is made of hydrogen and” → 29 bytes → 7 tokens of 32,768 kinds            §1.1, §2.1
 → embedding (T, 768) · 25,165,824                                                          §2.3
 → *position: no table → *norm: RMS · 0                                               §4.2, §5.2
 → block × 12, 7,077,888 parameters each, *biases: none:                                    §5.1
     stream → *norm → queries, keys, values → *score 12 · cos(q, k) · rotary          §3.2, §3.4
            → softmax → sum of values → *write (starts at zero) → added to the stream       §5.5
     stream → *norm → expand to 3,072 → *ReLU² → *write → added to the stream               §5.3
 → *norm → *unembedding (T, 32,768) · 25,165,824 → *cap: 15 · tanh(z / 15)            §2.5, §5.6
 → softmax → sample: “ oxygen”, p = 0.87 → joins the tokens; the machine runs again         §7.6
training   objective −log p(next token) · *training recipe: Muon + AdamW, own rates   §1.1, §6.5
           the run · what training learns                                               §7.3, §8
reading    circuits · features · the field and the recipe                     §9.2, §10.1, §11.1
The 2026 recipe has 135,266,304 parameters: 25,165,824 in each of two vocabulary matrices and
84,934,656 in the attention and MLPs of 12 blocks. The 2019 recipe ties the two matrices and has
111,008,256, 85,017,600 of them in attention and MLPs. 9 changes separate the two, in the 13
marked parts.
(the live figure needs JavaScript and loads about 10 kB of data)
The whole model as one blueprint, at the shape of gpt-pp (12 layers, 768 wide), reading a real text. Green: the parts that differ between the 2019 recipe and the 2026 recipe; the number on each is its step among the 9 changes. Dashed: a part the recipe on show does not have. Each part gives the shape of what leaves it, its parameters and, as a link, the Section that explains it. Switch the recipe; select a part to read both versions. The parameter counts are exact; the next-token probabilities are gpt-pp’s.

At full size gpt-pp is a small model in the manner of 2026 with the shape of GPT-2 small: 12 layers, a stream 768 wide, a context of 1,024 tokens, 135 million parameters. It read three billion tokens of FineWeb-EDU, a public corpus of educational web pages, once, in 3.20 hours on three graphics cards. It starts from two well-known open reproductions: nanoGPT, which rebuilt GPT-2 small from its paper, its released code and the training recipe printed with GPT-3, and nanochat, its successor, in which nearly every component of 2019 has been replaced.A. Karpathy, nanoGPT (GitHub, 2022 to 2023) and nanochat (GitHub, 2025 to 2026), read on 14 November 2026. Most of the newer components came through a public speedrun on the first: K. Jordan and contributors, modded-nanogpt (GitHub, 2024 to 2026). None of their code is reproduced here: the model, the optimizers and the trainer of this series are written again, as one model with switches. What gpt-pp adds is a switch on every component. The design of 2019 and the design of 2026 are two settings of one model, any step between them can be taken alone, and so each step can be measured on the same tokens with the same code.

component the 2019 recipe the 2026 recipe Section
position a learned vector for each position, added to the token's rotary: queries and keys turned by an angle that grows with their position 4.4
normalization layer norm with gain and bias, before each sublayer RMS norm without parameters, in the same places and after the embedding 5.2
MLP GELU between its two matrices squared ReLU 5.3
attention score \(q^{\top} k / \sqrt{d_k}\) \(12 \cos(q, k)\) 3.4
biases in every linear map none 5.6
embedding and unembedding one shared matrix two matrices 2.5
logits, the scores of the next token unbounded \(15 \tanh(z/15)\) 5.6
initialization \(\mathcal N(0, 0.02^2)\), the matrices that write to the stream scaled by \(1/\sqrt{2L}\) the matrices that write to the stream start at zero 5.5
optimizer AdamW on every parameter, with weight decay and gradient clipping Muon on the matrices of the blocks, AdamW on the two vocabulary matrices; no decay, no clipping 6.5
learning rate linear warmup, cosine decay to a tenth short warmup, constant, linear decay over the last 65% to a twentieth 6.3
The two recipes as two settings of one model. The names are explained in the Sections of the last column.

Where these Sections show gpt-pp's code, it is the code's own lines at one commit, 1890c9a, written against PyTorch 2.9.1, on which every run of the series was made. Beside it, each Section says what is odd, limited or was broken. The list is gathered in Section 11 and kept in full with the code.

The yardstick is GPT-2 small as released. Models with different vocabularies are compared in bits per byte, the length of the compressed text over the length of the text: eight for a model that knows nothing, about one for a good model on English prose. Scored under one protocol on 1,000 documents held out of gpt-pp's own corpus, GPT-2 small pays 1.007 bits per byte and gpt-pp 0.932. That is home ground, and of the four texts the only one on which gpt-pp leads.This comparison tests the resulting models, including the effects of their different training texts and budgets. The recipe experiments below ask a narrower question by holding the model shape and training tokens fixed. It is behind GPT-2 small on WikiText-103 (1.130 against 1.072), on stories (0.864 against 0.856) and on eight essays of this site from October and November 2026, which no released model can have read (1.325 against 1.297, behind on each of the eight). SmolLM2 135M, a released model of its own size that read two trillion tokens, is ahead of it on every set, at 0.857 on the first and 1.207 on the last.L. Ben Allal and others, "SmolLM2: when smol goes big", 2025 (arXiv 2502.02737); FineWeb-EDU is part of its training text. A single model's score on the eight essays has a standard error of 0.03 to 0.04, but on the same documents the difference is sharper: gpt-pp is behind GPT-2 small by 0.028 ± 0.009. On nine benchmarks taken by likelihood, the mean accuracy, rescaled so that chance is 0 and a perfect score 100, is 25.8 for gpt-pp, 24.5 for GPT-2 small and 35.3 for SmolLM2 135M.

chart   bits a byte on held-out text (lower is better) against parameters, a dot a model

  model                        parameters  FineWeb-EDU WikiText-103  TinyStories        fresh
  TinyStories 33M                     69M        2.875        3.380        0.436        2.716
  DistilGPT2                          82M        1.084        1.201        0.964        1.393
  the 2019 recipe, 100M tokens       111M        1.233        1.500        1.232        1.545
  GPT-2 small                        124M        1.007        1.072        0.856        1.297
  SmolLM2 135M                       135M        0.857        0.972        0.692        1.207
> gpt-pp                             135M        0.932        1.130        0.864        1.325
  the 2026 recipe, 100M tokens       135M        1.129        1.383        1.103        1.468
  Pythia 160M                        162M        0.998        1.094        0.882        1.309
  GPT-2 medium                       355M        0.925        0.975        0.759        1.229
  SmolLM2 360M                       362M        0.788        0.869        0.618        1.132
  Qwen3 0.6B base                    596M        0.817        0.888        0.638        1.155
  GPT-2 large                        774M        0.881        0.928        0.707        1.188
  GPT-2 XL                          1.56B        0.854        0.898        0.679        1.163

On FineWeb-EDU gpt-pp scores 0.932 bits a byte, between GPT-2 small (1.007) and GPT-2 medium
(0.925), for 9.6 card-hours of training. SmolLM2 135M, a released model of its size, scores
0.857.
(the live figure needs JavaScript and loads about 7 kB of data)
Bits a byte on held-out text against model size; lower is better. Green: gpt-pp, with its level drawn across and its training cost beside it. Ink dots: released models scored the same way, the sizes of one family joined; whiskers are one standard error over documents; the squares are the 2019 recipe and the 2026 recipe at gpt-pp’s shape after 100M tokens. Choose the held-out text; select a model for its numbers. Measured: one protocol, windows of at most 1,024 tokens.

The recipe itself was compared twice. At the shape of gpt-pp and after 100 million tokens, the 2026 recipe stands at a held-out loss of 3.654 nats a token and the 2019 recipe, at the best of four learning rates and two warmups, at 3.991, a difference of 0.337. Against nanochat nothing was run like for like, and at equal tokens the two recipes are level as far as their own figures can say: its 12-layer model of the same width reports 0.9825 bits per byte on its validation text after 1.08 billion tokens of the same corpus, and the recipe of gpt-pp with its schedule laid out for that budget ends at 0.974 on its own held-out windows.From the series of models that project published on 7 January 2026; later entries of its log at this depth read 0.965 to 0.977. The figures are not one measurement: the validation texts, the tokenizers and the contexts (2,048 tokens there, 1,024 here) differ, and our own two ways of scoring one model are 0.011 apart. The curve of the three-billion-token run reads 1.01 at the same count, its learning rate still near the peak of its longer schedule. That recipe has since gained parts that this one leaves out, among them learned scalars on the stream and token-indexed tables added to the values of attention.

The model, then, is not the result. What the series has to show is the way between the two recipes, with the steps that did not pay left in; a small model taken apart with the tools of 2021 to 2025; and what came out that neither had promised.

Three ideas tie the series together. First, to predict is to compress. The loss of a model on a text is the length of a code for it, so every model here, from a table of letter counts to gpt-pp, is measured in one unit on text it has not seen. On held-out stories of gpt-pico's corpus, gzip pays 2.401 bits per byte, a table of 4-grams 0.897, and gpt-pico 0.541. The identity also says what training asks for. A model is paid for whatever makes text more predictable, be it spelling, grammar, a fact, or the habit of repeating a name it has seen, and for nothing else.

Second, the machine is a stream that is read and added to, and two kinds of component do the reading and the adding. Attention moves information between positions: it is how the second Tim could look at the first. The MLPs work at one position, and Section 5 reads each of their neurons as a memory, with a key that matches a pattern in the stream and a value that is added when it matches.M. Geva, R. Schuster, J. Berant and O. Levy, "Transformer feed-forward layers are key-value memories", EMNLP 2021 (arXiv 2012.14913), abstract: "feed-forward layers in transformer-based language models operate as key-value memories, where each key correlates with textual patterns in the training examples, and each value induces a distribution over the output vocabulary". Queries, keys and values in the one, keys and values in the other: hence the title. Around these two, every component has a history. Positions were first added to the embedding and are now rotations of the queries and keys; normalization moved to the input of each sublayer and then lost its parameters; the optimizer learned to treat a matrix as a matrix. Each history is told as a problem, an answer and the next problem, and then measured on the ladder, a family of 8-layer models: one for each step from the 2019 recipe to the 2026 recipe, on the same 100 million tokens, and one for each component of the 2026 recipe put back alone, on 50 million, with one or two seeds per swap. Not every step pays. A gain on the normalization and biases on the linear maps, when they are put back, each lower the loss by about 0.02 nats a token, in both of two seeds, and GELU in place of the squared ReLU changes nothing in either. What the loss does depend on is a position that attention reads as a difference (the rotation here; a bias on the scores did as well), normalized queries and keys, normalization before the sublayer, and the optimizer.

Across the nine changes of the first family, the held-out loss falls from 4.135 nats a token for the 2019 recipe to 3.814 for the 2026 recipe, each the mean of two seeds.

stair   held-out loss of the ladder model after each change, nats a token; lower to the right

    step              from  4.20                                  3.70   loss     change
    the 2019 recipe   2019       |                                     4.135    
1   no biases         2022       |                                     4.135    (0.000)
2   rotary positions  2021       ---------|                            4.024    −0.110
3   RMS norm          2019             |---                            4.056    +0.031
4   squared ReLU      2021             ---|                            4.023    −0.033
5   cosine scores     2020                -|                           4.011    (−0.011)
6   zero writers      2019                 --|                         3.984    −0.027
7   training recipe   2024                   ------------|             3.828    −0.155
8   untied            2024                               -|            3.817    (−0.011)
9   soft cap          2024                                #            3.814    (−0.003)
    (a change in parentheses is under 0.016 nats: not told from two seeds of one step)
    in full: 3 RMS norm without parameters; 5 cosine scores (QK normalization); 6
    zero-initialized writers; 7 the training recipe of 2026 (Muon, its rates and its schedule);
    8 untied tables, normalized embedding; 9 the logit soft cap

Nine changes take the ladder model (8 layers, 512 wide, 100M tokens) from a held-out loss of
4.135 to 3.814 nats a token; the largest single step is the 2026 training recipe (−0.155); RMS
norm (+0.031) costs loss on this model. Two seeds of one step differ by 0.008 nats (root mean
square); a step under twice that, 0.016, is not established: no biases (0.000), cosine scores
(−0.011), untied (−0.011), soft cap (−0.003). The seven rungs of the 2019 side keep that
recipe’s warmup of 8M tokens and sit about 0.03 high: with 32M each ends 0.017 to 0.042 lower
(seed 0), and against step 6 so trained the 2026 training recipe is worth −0.140, where the same
seed reads −0.157 here.
(the live figure needs JavaScript and loads about 76 kB of data)
From the 2019 recipe to the 2026 recipe in 9 steps, on the ladder model (8 layers, 512 wide, 100M tokens each). Green: the last step, the 2026 recipe. A bar runs from the held-out loss before a change to the loss after; lower is to the right. The year is when the idea was published. Select a step for its numbers; Section 5 has the full figure. Measured, 2 seeds a step (rings; the bar ends at their mean); a difference in parentheses is inside the seeds’ spread.

Two steps carry the stair: the training recipe of 2026 in place of that of 2019 (the optimizer with its rates and its schedule), worth 0.155 nats, and rotary positions, worth 0.110, are 0.27 of the 0.32 between its ends. Removing the biases is worth nothing, normalization without parameters gives back 0.03, and since two seeds of one step differ by 0.001 to 0.018 nats, a step smaller than about 0.02 is inside the noise. The 2019 rungs keep the warmup of their own recipe; with one four times as long each ends about 0.03 lower, which takes the step of the training recipe from 0.16 to 0.14 in the seed where it was tried, and in both seeds 0.085 of that step is the optimizer's rule itself, Muon in place of AdamW.

Third, a trained network can be opened. Because the stream is a sum of what every head and every MLP has written, a prediction can be traced back to the parts that produced it, and the hairline that ties the Tim being written to the first one, tinted for its weight, is the first clue of such a trace. A set of parts with a statement of what each computes is a circuit (Section 9). Where one neuron answers to several unrelated things, a dictionary learned from the activations separates them into features that can be named, measured and switched on (Section 10). Two things came out that the tools did not promise. Whether a model learns to copy from its context depended on what it had read: after 200 million tokens of web pages a model of gpt-pico's width with two layers fewer pays 12.6 bits a token for the second copy of a random sequence against 20.8 for the first; gpt-pico, after 540 million tokens of stories, pays 19.6 against 21.4 (Section 8). And an attribution graph of gpt-pico, the newest of the tools, fails its own test on the path it shows first: taking out the feature at the end of that path moves the answer by 0.06 logits, where the graph gives its edge 3.83 (Section 10).A computed edge proposes a dependency. Removing its source tests how much the original model actually depends on it. Agreement must be measured; drawing the edge does not establish it. The reading is partial, in the literature as here, and the Sections say how partial.

A GPT has one objective and one data structure. Whatever else it can do, it learned because predicting the next token called for it; what it learned sits in keys and values, and part of it can now be read back.

Section 1 is the objective: the chain rule, entropy and code length, the letter tables and the guessing game of Shannon, tables of n-grams and where they stop, and the papers from the first neural language model to GPT-3, ending with the scaling laws that say how to spend a budget. Section 2 is the two ends of the model: byte-pair encoding, which builds a vocabulary by merging frequent pairs, the geometry of the embedding table with the high-dimensional probability behind it, and the softmax that turns scores into probabilities, with the bottleneck that a narrow stream puts on it.

Section 3 builds attention, from alignment in translation to the head of 2017, reads its formula three ways, and follows its cost to the algorithm that let exact attention win. Section 4 is position, which attention cannot see by itself: sinusoids, learned tables, relative schemes and rotations. Section 5 is the block: the stream, normalization, the MLP and its activation, a matrix read as a memory and how much it holds, initialization, and the block of 2019 beside the block of 2026. Section 6 is optimization, from gradient descent through Adam to Muon, with steepest descent as a choice of ball and its update rules as one rule with six switches. Section 7 trains gpt-pp: the data, the pipeline, the run replayed, the evaluation, what a GPT-2 has cost from 2019 to 2026, with the public races beside this run, a chat stage that tunes the model to answer and what that costs it on plain text, and what the model cannot do.

Section 8 asks what training learns and in what order: the statistics of single tokens and of pairs first, then learning from the context, with grokking (generalizing long after the training set is fitted), in a model that adds by rotating, and double descent (a test error that rises and falls again as a model grows) on the way. Section 9 reads circuits from weights and tests them by intervening on activations. Section 10 is features: superposition, in which a layer holds more features than it has dimensions, the dictionaries that undo it and what their three encoders do to an activation, and the graphs that join features into computations. Section 11 gathers what the field has settled, with the block redrawn year by year, and what is open in November 2026, among it the cache at a million tokens and memories that overwrite where attention adds; then further reading and the recipe of every worked model.

By the end, you should be able to read the model file of an open language model and say what each line computes, which older design it replaced and why. You should be able to train a small model and judge it by bits per byte on held-out text, to tell what a claim such as "GPT-2 in two hours" holds fixed and what it does not, and to open a trained model: find the heads behind a behavior, patch an activation, read the stream at each depth, train a dictionary on it. The recipe in Section 11 lets you retrain the worked models and connect each line of their code to the mathematics.

Three models carry the worked examples, gpt-pp, the ladder and gpt-pico, with a few small ones beside them. All were trained for this series on public text, and Section 11 gives the data, the settings and the seeds needed to train them again.FineWeb-EDU: G. Penedo and others, "The FineWeb datasets: decanting the web for the finest text data at scale", 2024 (arXiv 2406.17557), used here in its sample of ten billion tokens (license ODC-By 1.0). TinyStories: R. Eldan and Y. Li, "TinyStories: how small can language models be and still speak coherent English?", 2023 (arXiv 2305.07759), in its second version, written by GPT-4 (license CDLA-Sharing-1.0).

name in the text what it is shape parameters trained on
gpt-pp the main model, in the 2026 recipe 12 layers, 768 wide, 12 heads, context 1,024, vocabulary 32,768 135,266,304 FineWeb-EDU, 3.0 billion tokens, one pass
the 2019 recipe at the same shape the design of GPT-2, trained as its reproductions train it the same 111,008,256 FineWeb-EDU, 100 million tokens, beside the 2026 recipe on the same tokens
the ladder one model for each step from the 2019 recipe to the 2026 recipe, and one for each component of the 2026 recipe put back alone 8 layers, 512 wide, 8 heads, context 512, vocabulary 32,768 41.9 million to 59.0 million FineWeb-EDU, 100 million tokens a step, 50 million a swap
gpt-pico the smallest size, the one that runs in the page 6 layers, 128 wide, 4 heads, context 256, vocabulary 4,096 2,228,224 TinyStories, 540 million tokens, one pass
gpt-pico in the 2019 recipe for the figures that set the two recipes side by side the same 1,746,944 the same
the attention-only models one and two layers without MLPs, for Sections 8 and 9 256 wide, 8 heads, context 256, vocabulary 4,096 2,359,296 and 2,621,440 TinyStories, 200 million tokens, alone and with rows of repeated random tokens mixed in
the modular-addition model one layer, for the grokking of Section 8     pairs of numbers and their sum modulo 113
the dictionaries directions learned from the activations of gpt-pico and of gpt-pp, for Section 10     activations of the two models
The models of the worked examples. At one shape the two recipes differ in size: the 2026 recipe has a second vocabulary matrix, the 2019 recipe has position vectors, biases and gains.

Notation. A text is a string of tokens \(x_{1:n} = x_1 \dots x_n\) from a vocabulary \(\mathcal V\) of \(V\) tokens, and \(x_{<t}\) is its prefix before position \(t\). The source of text is a distribution \(p\); a model is \(q\), or \(q_\theta\) with its parameters \(\theta\), and \(q_\theta(\cdot \mid x_{<t})\) is its distribution over the next token. In Section 1, \(\log\) is to base 2 and counts bits and \(\ln\) counts nats; elsewhere a loss is in nats per token unless bits are named.

The entropy of \(p\) is \(H(p)\), its cross-entropy with \(q\) is \(H(p, q)\), and the relative entropy is \(D(p \,\|\, q)\), written \(\mathrm{KL}\) in Sections 8 and 9. The loss \(\mathcal L(\theta)\) is the first of the three formulas. On a text of \(T\) tokens and \(B\) bytes the perplexity is \(e^{\mathcal L}\) and the bits per byte are \(T \mathcal L / (B \ln 2)\) (Definition 1.1.7). In the scaling laws of Sections 1 and 7, \(N\) counts parameters, \(D\) training tokens and \(C\) floating-point operations.

A model has \(L\) blocks and a stream of width \(d\). A block has \(h\) heads of width \(d_k = d/h\), written \(d_h\) in Sections 4 and 9, where \(h\) names one head, and an MLP of hidden width \(d_m\), which is \(4d\) in every model here. The context has \(n\) tokens at positions \(1, \dots, n\), or \(T\) tokens in Section 5. The embedding is \(E \in \mathbb R^{V \times d}\) with a row \(e_v\) for each token, and the unembedding is \(U \in \mathbb R^{V \times d}\) with rows \(u_v\) (Definition 2.3.1); with the literature on circuits, Section 9 writes \(W_E = E^{\top}\), and Sections 5 and 9 write \(W_U\) for \(U\).

The residual stream is \(X\), one row for each token, and \(X^{\ell}\) after block \(\ell\) where the depth matters. From Section 3 on, \(x_i\) is the vector of the token at position \(i\), where Section 1 writes \(x_t\) for the token itself. \(N(\cdot)\) is the normalization. A head has matrices \(W_Q, W_K, W_V \in \mathbb R^{d_k \times d}\) and \(W_O \in \mathbb R^{d \times d_k}\), queries, keys and values \(q_i = W_Q x_i\), \(k_i = W_K x_i\) and \(v_i = W_V x_i\), scores \(s_{ij}\) and a pattern \(A_{ij}\), written \(a_{ij}\) in Section 4: row \(i\) is the position that reads and column \(j \le i\) the position that is read. The products \(W_{QK} = W_Q^{\top} W_K\) and \(W_{OV} = W_O W_V\) say where a head looks and what it moves (Section 9). The MLP is \(W_{\mathrm{out}}\, \phi(W_{\mathrm{in}} x)\) with an activation \(\phi\); the rows of \(W_{\mathrm{in}}\) are its keys and the columns of \(W_{\mathrm{out}}\) its values (Definition 5.4.1).

In Section 6 the loss is written \(L(\theta)\), \(g_t\) is its gradient on the batch of step \(t\) and \(\eta\) is the learning rate. Positions, blocks and heads are counted from 1. Numbered statements are Definition, Theorem, Proposition, Example and Remark \(N.S.K\), with \(N\) the part and \(S\) the section, and a worked Example opens with its setup.

Five things are worth having open beside the series. The first two are the model files of the reproductions named above, to be read against the table of the two recipes. The third is the Transformer Circuits Thread, a run of papers published on the web from 2021 on, whose vocabulary of streams, heads and features Sections 9 and 10 use. The fourth is Vershynin's textbook of high-dimensional probability, for the concentration results behind Sections 2 and 3. The fifth is Shannon's paper of 1948, whose first sections already hold the language model, its samples and its entropy.N. Elhage, N. Nanda, C. Olsson and others, "A mathematical framework for transformer circuits", Transformer Circuits Thread, 22 December 2021, transformer-circuits.pub/2021/framework, followed in the same thread by papers on induction heads (2022), superposition (2022), dictionaries (2023) and attribution graphs (2025). R. Vershynin, High-dimensional probability: an introduction with applications in data science, second edition, Cambridge University Press, 2026, with a free copy at the author's page. Shannon's paper and the two reproductions are cited above.

(General disclaimers apply.) Nothing here is investment, legal or tax advice. The results of papers and the numbers of other projects are as published at the dates given, and the sidenotes say where each was read. The measurements on the models of this series are from single training runs unless a count of seeds is given, and the released models were scored on 14 November 2026.

← Back to all posts