8. What training learns
Draft
The loss curve hides structure
Train gpt-pico, the smallest size of gpt-pp and the one that runs in the page (six layers, 128 wide, 2.2 million parameters, on TinyStories, a public corpus of short synthetic stories), and plot its loss against the tokens it has seen. The curve starts at \(\log 4096 \approx 8.32\) nats, a uniform guess over the vocabulary, and bends smoothly toward its final value. On the way it crosses two landmarks: the loss of the best predictor that guesses by token frequency alone, and the loss of the best predictor that looks at the last token only. The sharper question is: when the model's loss equals that of a frequency table, is the model a frequency table?
gpt-pico on TinyStories: loss on held-out stories against the tokens seen, in bits per byte (bits per token / 4.028 bytes per token), beside the n-gram levels of the corpus tokens seen 0 0.72 M 1.8 M 2.9 M 7.5 M 540 M step 0 11 28 45 115 8,240 loss, bits per byte 2.979 1.849 1.370 1.229 0.882 0.554 from unigram, nats 2.38 0.54 2.60 3.10 4.64 8.76 from bigram, nats 4.75 1.59 0.32 0.64 1.93 4.28 levels uniform 2.979 | unigram 2.126 | bigram 1.282 | trigram 1.008 | 4-gram 0.897 passed after 0.61 M (unigram), 2.5 M (bigram), 5.0 M (trigram), 7.2 M (4-gram) tokens nearest the unigram predictor at step 11 (0.54 nats), the bigram predictor at step 28 (0.32 nats) after "Once upon a time" gpt-pico writes step 11 , it outside was alert and takeed and rabbit on my proud sat sad step 28 , a nail was a smallweed and good on a big leaf to have all day, step 8,240 , there was a big boat. The boat was on a blue lake. The boat wa (the live figure needs JavaScript; it loads 16 kB of data)
The divergence from each predictor to the model falls to a trough and climbs again, first for the unigram predictor, then for the bigram predictor: the model comes close to each table in turn and leaves it. Matched samples show how close. From the same random draws, the checkpoint at step 17 and the bigram predictor both write "on the puppy sat down". gpt-pico trained with the recipe of 2019 (Section 5) passes through the same two stages later.
Section 8 asks what happens inside such a run: what is learned, in what order, why some of it arrives suddenly, and what to measure when the loss says nothing.
Definition 8.1.1 (the n-gram levels). For a stationary source of tokens \(X_1, X_2, \dots\) over a vocabulary of size \(V\), the \(k\)-gram level is the conditional entropy \(H_k = H(X_k \mid X_1, \dots, X_{k-1})\), the \(F_k\) of Definition 1.2.1: \(H_1\) is the unigram level, the entropy of the token frequencies, and \(H_2\) the bigram level.
Proposition 8.1.2 (the levels descend). The level \(H_k\) is the smallest expected loss of a predictor that reads only the \(k - 1\) tokens before the one it predicts, and \(\log V \ge H_1 \ge H_2 \ge \dots\), with limit the entropy rate of the source.
Proof. A predictor \(q\) has expected loss \(H_k + \mathbb E\, \mathrm{KL}(p \,\|\, q) \ge H_k\), the divergence being between the true and the predicted conditional (Gibbs' inequality, Section 1). By stationarity \(H_k = H(X_{k+1} \mid X_2, \dots, X_k)\), and conditioning on \(X_1\) as well cannot increase it. By the chain rule the entropy per token is \(\tfrac1n \sum_{k \le n} H_k\), which has the same limit. ∎
A model that ends below the bigram level must cross both levels. What is not forced is that it resembles the unigram predictor while its loss is near \(H_1\): many predictors have that loss. That it does is an instance of a regularity with a name.
The distributional simplicity bias (DSB) posits that neural networks learn low-order moments of the data distribution first, before moving on to higher-order correlations. In this work, we present compelling new evidence for the DSB by showing that networks automatically learn to perform well on maximum-entropy distributions whose low-order statistics match those of the training set early in training, then lose this ability later. We also extend the DSB to discrete domains by proving an equivalence between token \(n\)-gram frequencies and the moments of embedding vectors, and by finding empirical evidence for the bias in LLMs. Finally we use optimal transport methods to surgically edit the low-order statistics of one class to match those of another, and show that early-training networks treat the edited samples as if they were drawn from the target class. Code is available at https://github.com/EleutherAI/features-across-time.
—Belrose and others, "Neural networks learn statistics of increasing complexity" (2024). Belrose, Pope, Quirke, Mallen and Fern, ICML 2024 (arXiv 2402.04362), abstract; Theorem 2.1 and Figures 6 and 7 are used below.
For sequences of tokens the word moment has an exact meaning. Encode a sequence by the indicators \(f_{t,v}(x) = \mathbb 1[x_t = v]\). Since \(f_{t,v}^2 = f_{t,v}\) and \(f_{t,v} f_{t,v^{\prime}} = 0\) for \(v \ne v^{\prime}\), a product of \(k\) of them is zero or the indicator that at most \(k\) distinct positions hold given tokens: every moment of order \(k\) is zero or the probability of an n-gram on at most \(k\) positions, gaps allowed, and conversely. Low-order moments first then reads: unigrams, then bigrams, then longer n-grams. Their test scores each checkpoint on synthetic text that matches the corpus up to order \(k\) and is otherwise random: independent tokens with the corpus frequencies for \(k = 1\), a Markov chain with the corpus's bigram table for \(k = 2\). On the checkpoints of the Pythia language models the loss on such samples reaches a trough between steps \(2^6\) and \(2^8\), climbs until step \(2^{10}\) as the model stops being an n-gram predictor, and falls again once the model can infer the sampling distribution from its context.
Example 8.1.3 (gpt-pico against its n-gram levels). The setup: gpt-pico, trained in one pass over TinyStories, at 21 checkpoints spaced evenly in the logarithm of the step; at each, the loss on 1,024 held-out windows and the mean divergence from the unigram and from the bigram table of the training text to the model's next-token distribution. The levels drawn are the held-out losses of smoothed n-gram models (Section 1), upper estimates of \(H_k\). One run.
The unigram level is 2.126 bits per byte and the bigram level 1.282. gpt-pico's loss passes the first after 0.61 million tokens and the second after 2.5 million, 0.5% of its run. Its divergence from the unigram predictor falls from 2.38 nats to 0.54 at step 11, and its divergence from the bigram predictor from 4.75 nats to 0.32 at step 28; at the end they are 8.76 and 4.28. For scale, the two levels are 2.36 nats apart. So the answer to the opening question is: nearly, for a few steps, and twice. Equal loss is still not the same predictor: when the loss crosses the bigram level, near step 38, the model is already about 0.5 nats away from the bigram table. On text drawn from the tables themselves, the test of Belrose et al., the troughs are at the same two checkpoints, and afterwards the loss climbs and stays up: gpt-pico shows no second fall of the kind the Pythia models show. ◁
In the code. The loss and the two divergences of this example come from one pass over the held-out windows: at every position, how far each table's prediction is from the model's next-token distribution. It is measure in gpt-pp's interp/ngram_order.py.
@torch.no_grad()
def measure(model, windows: Tensor, predictors: Predictors, batch: int = 32) -> dict[str, float]:
"""One checkpoint on held-out windows (N, T + 1): its loss and its divergence from each predictor, in nats per
token, averaged over all N * T positions.
At a position whose last token is x the bigram predictor is row x of the table; the divergence from a
predictor q to the model m is sum_v q(v) (log q(v) - log m(v)) = -H(q) - sum_v q(v) log m(v).
"""
uni, bi = predictors.unigram.float(), predictors.bigram.float()
minus_h_uni = -entropy(predictors.unigram)
minus_h_row = -entropy(predictors.bigram) # (V,), float64
total = {"loss": 0.0, "kl_unigram": 0.0, "kl_bigram": 0.0}
for start in range(0, windows.size(0), batch):
chunk = windows[start:start + batch]
idx, targets = chunk[:, :-1], chunk[:, 1:]
logp = log_probs(model, idx)
total["loss"] += -logp.gather(-1, targets.unsqueeze(-1)).double().sum().item()
total["kl_unigram"] += (minus_h_uni - (logp @ uni).double()).sum().item()
total["kl_bigram"] += (minus_h_row[idx] - (bi[idx] * logp).sum(dim=-1).double()).sum().item()
positions = windows.size(0) * (windows.size(1) - 1)
return {name: value / positions for name, value in total.items()}
logp holds the model's log-probabilities \(\log m(v)\) at every position, of shape (windows, positions, \(V\)) with \(V = 4096\) for gpt-pico; uni is the unigram table, and bi[idx] picks at each position the row of the bigram table for the token just read. Each divergence is \(\mathrm{KL}(q \,\|\, m) = -H(q) - \sum_v q(v) \log m(v)\) with \(q\) the table's distribution: logp @ uni and (bi[idx] * logp).sum(dim=-1) are the two sums over the vocabulary, and the entropies are taken once from the tables. The first of the three totals is the ordinary loss, \(-\log m\) of the token that came. The log-probabilities are float32 and the sums float64.
Quirk. Two kinds of n-gram model stand side by side in this example. The predictors that the divergences start from are tables of raw frequencies with no smoothing, counted by count_tables over all 542,905,221 training tokens as one stream, so the pairs that cross from one story into the next are counted like any other; the two levels are held-out losses of other models, with interpolated Kneser-Ney smoothing, which keeps some probability for what was never counted, built on the first 200 million training tokens. The divergence runs from the table to the model because that is the finite direction: the model gives every token some probability and the tables are full of zeros. The pair table is dense, \(V \times V\), which suits gpt-pico's 4,096 tokens and would mean a billion entries at gpt-pp's 32,768.
Why would gradient descent learn in this order? The one network in which the order can be computed has no nonlinearities.
Theorem 8.1.4 (Saxe, McClelland and Ganguli). Train \(y = W_2 W_1 x\) by gradient flow with time constant \(\tau\) on the squared error, on inputs with identity correlation, and let \(\Sigma = \sum_\alpha s_\alpha\, \phi_\alpha \psi_\alpha^{\top}\) be the singular value decomposition of the input-output correlation \(\sum_\mu y^\mu x^{\mu\top}\). If at time zero \(W_1 = \sum_\alpha a_\alpha\, e_\alpha \psi_\alpha^{\top}\) and \(W_2 = \sum_\alpha b_\alpha\, \phi_\alpha e_\alpha^{\top}\) with orthonormal \(e_\alpha\), the weights keep this form and each mode evolves alone:
\[\tau \dot a_\alpha = b_\alpha (s_\alpha - a_\alpha b_\alpha), \qquad \tau \dot b_\alpha = a_\alpha (s_\alpha - a_\alpha b_\alpha). \tag{8.1}\]The difference \(c = a^2 - b^2\) of a mode is conserved, and its strength \(u = ab\) obeys
\[\tau \dot u = \sqrt{c^2 + 4u^2}\; (s - u). \tag{8.2}\]For a balanced start, \(c = 0\) and \(0 < u_0 < s\),
\[u(t) = \frac{s\, e^{2st/\tau}}{e^{2st/\tau} - 1 + s/u_0}, \qquad t_{1/2} = \frac{\tau}{2s} \ln \frac{s - u_0}{u_0}, \tag{8.3}\]where \(t_{1/2}\) is the time at which \(u = s/2\).
Proof. The flow is \(\tau \dot W_1 = W_2^{\top} (\Sigma - W_2 W_1)\), \(\tau \dot W_2 = (\Sigma - W_2 W_1) W_1^{\top}\). For weights of the stated form \(\Sigma - W_2 W_1 = \sum_\alpha (s_\alpha - a_\alpha b_\alpha)\, \phi_\alpha \psi_\alpha^{\top}\), so both right-hand sides have the stated form with the coefficients of (8.1). Then \(\tfrac{d}{dt}(a^2 - b^2) = 0\), \(\tau \dot u = (a^2 + b^2)(s - u)\), and \((a^2 + b^2)^2 = c^2 + 4u^2\). For \(c = 0\) this is the logistic equation \(\tau \dot u = 2u(s - u)\), whose solution is (8.3). ∎
The loss is \(\tfrac12 \sum_\alpha (s_\alpha - u_\alpha)^2\) up to a constant, and (8.3) is a sigmoid. From a small balanced start a mode stays near zero for a time \(t_{1/2} \approx \tfrac{\tau}{2s} \ln \tfrac{s}{u_0}\) and then climbs from a tenth of \(s\) to nine tenths in \(\tfrac{\tau}{2s} \ln 81\), whatever \(u_0\) was. A smaller initialization lengthens every wait and leaves the climbs as they are: the loss becomes a staircase, one step for each mode, the strongest first.
Here \(t\) is the time it takes for \(u\) to travel from \(u_{0}\) to \(u_{f}\). If we assume a small initial condition \(u_{0}=\epsilon\), and ask when \(u_{f}\) is within \(\epsilon\) of the fixed point \(s\), i.e. \(u_{f}=s-\epsilon\), then the learning timescale in the limit \(\epsilon\rightarrow 0\) is \(t=\tau/s\ln\left(s/\epsilon\right)=O(\tau/s)\) (with a weak logarithmic dependence on the cutoff). This yields a key result: the timescale of learning of each input-output mode \(\alpha\) of the correlation matrix \({\Sigma^{31}}\) is inversely proportional to the correlation strength \(s_{\alpha}\) of the mode. Thus the stronger an input-output relationship, the quicker it is learned.
—Saxe, McClelland and Ganguli, "Exact solutions to the nonlinear dynamics of learning in deep linear neural networks" (2014). ICLR 2014 (arXiv 1312.6120), the paragraph after equation 11; their \(\Sigma^{31}\) is our \(\Sigma\), and (8.2) is their calculation without the assumption \(a = b\).
The point at which the first \(m\) modes are learned and the others are at zero is a stationary point of (8.1), where \(W_2 W_1\) is the best approximation of rank \(m\) to \(\Sigma\). It is a saddle: moving mode \(m + 1\) to \(a = b = \epsilon\) lowers the loss by about \(s_{m+1} \epsilon^2\). From a small initialization the network leaves the saddle at zero along the strongest mode and lands next to the best map of rank one, a saddle again; then the second mode, and so on. This is saddle-to-saddle dynamics, and its general form is about fixed points: a fixed point of a network with \(H - 1\) units (neurons, convolution kernels or attention heads) gives fixed points of the network with \(H\) units, so the minimum of each narrower network is a saddle of the wider one and learning adds a unit at a time.Zhang, Saxe and Latham, "Saddle-to-saddle dynamics explains a simplicity bias across neural network architectures", ICLR 2026 (arXiv 2512.20607), Theorem 1. In a transformer the narrowest network has no blocks, only embedding and unembedding, and its best fit is a table of bigrams (Section 9).
Whether a run waits at a landing depends on where it starts. By (8.2) a balanced small start waits, and a start with \(u_0 = 0\) and \(c \ne 0\) does not: \(\tau \dot u = \lvert c \rvert\, s\) at time zero. In gpt-pp the matrices through which a block writes into the residual stream, the running sum that every block reads and adds to, start at zero, and those through which it reads start at their usual scale (Section 5): a start of the second kind. gpt-pico's curve has no landing. At the unigram level it does not slow at all, and where it is nearest the bigram predictor its descent per unit of the logarithm of the step halves, from 1.6 nats to 0.76 over steps 30 to 40, and then recovers: a shoulder, not a plateau.
If a loss curve is a sum of steps, why is its envelope regular enough for Section 7 to fit a power law to it? One answer puts the regularity in the distribution of the steps: what a model knows comes in discrete pieces, quanta, each learned or not, used with frequencies that follow a power law.Michaud, Liu, Girit and Tegmark, "The quantization model of neural scaling", NeurIPS 2023 (arXiv 2303.13506), Section 2. If the \(k\)-th quantum is needed on a fraction \(p_k \propto k^{-(\alpha + 1)}\) of the tokens, the loss still to be gained once the first \(n\) are learned is proportional to \(\sum_{k > n} p_k \propto n^{-\alpha}\). It falls as \(N^{-\alpha}\) if \(N\) parameters hold \(n \propto N\) quanta, and as \(D^{-\alpha/(\alpha + 1)}\) if a quantum is learned once \(D\) tokens hold enough examples of it, that is, for \(k^{\alpha + 1} \lesssim D\). Each quantum arrives as a step and the power law is their sum. The newest account derives the data exponent from the text itself: a model trained on \(D\) tokens can detect a dependency between distant tokens only if it stands out of the sampling noise \(D^{-1/2}\), so its usable context grows as a power of \(D\), and its loss is the level of Definition 8.1.1 at that horizon. For TinyStories the predicted data exponent is 0.185, with no free parameter.Cagnetta, Raventós, Ganguli and Wyart, "Deriving neural scaling laws from the statistics of natural language", 2026 (arXiv 2602.07488), Sections 3 and 4. More data buy a longer usable context: the order of Example 8.1.3 read along the axis of data.
Remark 8.1.5 (what it changes). Draw the n-gram levels on every loss plot: they are cheap to count and they say what an early loss means. Do not read a flat stretch as convergence: by (8.3) a wait before a drop is the normal course of a small balanced start, and by (8.2) the wait is a choice of initialization. Save checkpoints evenly in the logarithm of the step: gpt-pico's first two stages are over within 0.5% of its run.
In-context learning
Take a sequence of tokens drawn at random and write it twice. Nothing in any corpus makes the second copy predictable; the first copy makes it certain. A model that predicts the second copy has taken something from its context that its weights could not hold. The name for this ability comes from the paper on GPT-3, which showed that a frozen model given a few examples of a task in its prompt will often perform the task.
Figure 1.1: Language model meta-learning. During unsupervised pre-training, a language model develops a broad set of skills and pattern recognition abilities. It then uses these abilities at inference time to rapidly adapt to or recognize the desired task. We use the term “in-context learning” to describe the inner loop of this process, which occurs within the forward-pass upon each sequence. The sequences in this diagram are not intended to be representative of the data a model would see during pre-training, but are intended to show that there are sometimes repeated sub-tasks embedded within a single sequence.
For a language model the simplest measurement needs no task: compare the loss late in a context with the loss early in it.
Definition 8.2.1 (in-context learning score). Let \(\ell_t\) be the mean over held-out windows of the loss at position \(t\). For positions \(t_0 < t_1\) the in-context learning score is \(\ell_{t_0} - \ell_{t_1}\), positive when the later token is the easier one.Olsson et al. (2022), cited below, take positions 50 and 500 in contexts of 512 tokens and state the difference with both signs; we keep the positive one. Our measurements average each loss over the positions within a quarter of \(t\) (38 to 62 and 375 to 625); one position over a few hundred windows is too noisy.
By Proposition 8.1.2 an ideal predictor has \(\ell_t = H_t\), so its score, \(H_{t_0} - H_{t_1} \ge 0\), is a property of the text, and a table of bigrams already scores positive between the first position and the second. Using context is not yet learning in context. The test that isolates adaptation is the one this section opened with: on \((r_1, \dots, r_L, r_1, \dots, r_L)\) with the \(r_i\) uniform over the vocabulary, the first half costs at least \(\log V\) a token for any predictor, corpus statistics are of no use in the second half, and a predictor that copies pays almost nothing there.
gpt-pp: loss in bits per token by position in the context, through training step 0 32 994 8,586 20,345 text, near position 50 15.00 8.66 5.60 4.95 4.69 text, near position 500 15.00 8.71 5.26 4.60 4.34 random tokens, first copy 15.00 19.44 20.79 19.69 19.01 the same tokens, second copy 15.00 19.47 10.36 6.49 3.95 a uniform guess costs 15.00 bits; At step 20,345 gpt-pp predicts the second copy of 50 random tokens at 4.0 bits a token against 19.0 for the first: it has learned to look back. The second copy has cost less than half the first since step 994, 147 million tokens. gpt-pico does not learn to copy from its context: at step 8,240 the second copy of 50 random tokens still costs 19.6 bits a token against 21.4 for the first. (the live figure needs JavaScript; it loads about 91 kB of data)
We run both measurements on the checkpoints of gpt-pp, the 12-layer model, with 50 random tokens written twice. On text its score between positions 50 and 500 goes from zero to 0.31 bits between 17 and 62 million tokens and stays near 0.35 bits to the end of the run. Before training both copies of the random sequence cost 15.0 bits a token, a uniform guess, and soon about 20, because a model that has learned text is confidently wrong about random tokens. From 17 million tokens the loss on the second copy separates from that on the first; by 147 million it costs less than half as much, and at the end 3.95 bits a token against 19.01. The 12-layer model has learned to look back. gpt-pico has the other outcome: its second copy ends at 19.6 bits against 21.4 for the first, while on text it gains 0.61 bits between positions 12 and 125. gpt-pico uses its context and does not copy from it.
Copying has a known mechanism. A head whose current token is \(r_m\) looks back for the earlier \(r_m\), attends to the token \(r_{m+1}\) that followed it, and promotes \(r_{m+1}\) as the next token.
Definition 8.2.2 (induction head, induction score). On the input \((\text{BOS}, r_1, \dots, r_L, r_1, \dots, r_L)\), with positions counted from one after the beginning-of-sequence token, let \(A^h\) be the attention pattern of head \(h\). Its induction score is the attention that the tokens of the second copy pay to the token that followed their first occurrence,
\[\mathrm{IS}(h) = \frac1L \sum_{m=1}^{L} A^h_{L + m,\; m + 1}, \tag{8.4}\]averaged over random sequences. An induction head is a head with a high induction score whose output increases the logit of the token it attends to.The prefix-matching score of Olsson et al. (2022), who measure it on 25 random tokens repeated four times. Ours are for 16 tokens given twice, which reads lower: the code adds up the attention to the token after every earlier occurrence of the current one, so the score grows with the number of repeats (Section 9.3), and the strongest head of Example 8.2.3 scores 0.85 this way and 0.94 by their protocol.
In the matrix \(A^h\) the entries of (8.4) form a line parallel to the diagonal, \(L - 1\) places below it: the induction stripe. No head of a one-layer model can draw it, because an attention score is computed from the token at the reading position and the token at the position read, and the stripe needs the token before the position read; an earlier head has to write it there first. Section 9 takes the circuit apart. Here only its shape matters: a composition of at least four learned pieces, the pattern of a previous-token head, what that head writes, the match that a later head makes against it, and the copy that the later head outputs. Where it was first studied it arrives all at once.
Our first observation is that if we measure in-context learning for transformer models over the course of training (defined as the 50th token loss minus the 500th token loss as described in Key Concepts), it develops abruptly in a narrow window early in training (roughly 2.5 to 5 billion tokens) and then is constant for the rest of training (see figure below). Before this window there is less than 0.15 nats of in-context learning, after it there is roughly 0.4 nats, an amount that remains constant for the rest of training and is also constant across many different model sizes (except for the one layer model where not much in-context learning ever forms). This seems surprising – naively, one might expect in-context learning to improve gradually over training, and improve with larger model sizes, as most things in machine learning do.
[…]
This already strongly suggests some connection between induction heads and in-context learning, but beyond just that, it appears this window is a pivotal point for the training process in general: whatever's occurring is visible as a bump on the training curve (figure below). It is in fact the only place in training where the loss is not convex (monotonically decreasing in slope).
We replayed the experiment on the model that Section 9 takes apart: two layers of attention only, 256 wide, eight heads a layer, trained for 200 million tokens. Its training text decides the outcome: TinyStories in which one window in ten is replaced by a block of 8 to 64 random tokens repeated to the end of the window.
tokens seen loss, stories loss, repeated half highest induction score one layer stories alone 0 8.32 8.32 0.04 0.04 0.04 983k 4.62 8.96 0.04 0.04 0.04 9.4M 2.78 9.09 0.04 0.04 0.05 30M 2.37 8.48 0.11 0.03 0.06 31M 2.38 7.49 0.26 0.03 0.06 106M 2.11 0.80 0.73 0.02 0.12 200M 1.85 0.13 0.85 0.02 0.14 losses in nats per token; repeated half: the second copy of 16 random tokens given twice induction score: the attention a token of the second copy pays to the token that followed its first occurrence, from 0 to 1; one layer and stories alone: the highest score of each control the window of the change (the middle eight tenths of the fall): 31M to 106M tokens; half of the fall is made by 34M, a factor of 1.07 after the start pattern of layer 2, head 8 after the window: each token of the second copy reads the token after its first occurrence, a stripe parallel to the diagonal (the live figure needs JavaScript and loads about 1.5 MB of data)
For 29 million tokens the loss on the second copy of 16 random tokens does not come down. Then it falls abruptly as a few heads of the second layer acquire high induction scores. Neither a one-layer model on the same text nor a two-layer model on the stories alone makes this transition.
Example 8.2.3 (the phase change in the two-layer model). The setup: the two-layer model above at 80 checkpoints of its run; at each, the loss on the second copy of 16 random tokens given twice and the induction score (8.4) of each of its 16 heads on the same rows. The window is the stretch of training over which that loss makes the middle eight tenths of its fall, from 8.32 nats before training to 0.13 at the end. One seed.
The window runs from 31.5 to 105.8 million tokens, 37% of the run: by this definition it is not narrow. The start is a cliff: the loss goes from 8.9 nats at 28.8 million tokens to 3.0 at 33.6 million, 2.4% of the run. The rest is a tail: the loss stays between 1.2 and 1.7 nats from 40 to 95 million tokens and comes down as the learning rate decays. Across the window the highest induction score of any head goes from 0.26 to 0.73 (0.85 at the end of the run). The in-context learning score on stories does not jump at the cliff; it climbs from 5 million tokens on, to 0.28 nats. The one-layer model on the same text ends at 8.33 nats on the second copy, a uniform guess, and none of its heads passes 0.05. ◁
None of this happens without the repeated windows. Trained on the stories alone, the same two-layer model ends with a best induction score of 0.14 and a loss of 18.1 nats on the second copy, more than twice the loss of a guess: it grows no usable induction circuit. These two runs are the controlled experiment, and in it the data decide. Stories and web text differ too, at gpt-pico's size: gpt-pico, after 540 million tokens of stories, does not copy; a model of its width with two layers fewer has begun to after 200 million tokens of web text, with 12.6 bits a token on the second copy of 50 random tokens against 20.8 on the first. The reason is not how often text repeats itself: in windows of 256 tokens, 51% of the positions of held-out stories hold a token seen earlier and 17% of those continue as before, against 39% and 15% for web text. But the two-layer model trained on stories alone, which does not copy, already gets 81% of the continuing tokens right. That share was not measured on web text, so it is an inference that what differs is what a repeat is worth to a model that could have guessed it anyway.
Bug, fixed. The four-layer model of this comparison belongs to the first scaling grid, which was trained while a width set on the command line left the learning rates of the two tables at their values for width 512 (Section 7.5). The grid was trained again after the fix and the copy test was not repeated: the 12.6 and 20.8 bits are of the model trained before it. In held-out loss the rule is worth 0.008 nats at width 128 after 200 million tokens.
Known issue. Two tools behind this section are tied to the place the series was made in. copy_test.py, which scores finished models on the two copies and counts how often held-out text repeats itself, reads its checkpoints from a path written into the file, whatever the environment variable GPTPP_WORK says, so as it stands it runs on that machine only. measure_circuits.py takes the numbers of Example 8.2.3 from the data file exported for the figure of the phase change, which is kept with the site and not with the code, and leaves them out without a message when the file is missing. The attention-only models of this section were each trained once, and they were not among the runs checked again when the series' numbers were reviewed.
Why a cliff? The smallest model of the answer is the mode of Theorem 8.1.4 with more factors.
Proposition 8.2.4 (a product of three or more factors waits, then jumps). Train \(u = a_1 a_2 \cdots a_k\), \(k \ge 3\), by gradient flow on \(\tfrac12 (s - u)^2\) with time constant \(\tau\), from \(a_i = a_0 > 0\) with \(a_0^k \ll s\). As long as \(u \ll s\),
\[a(t)^{-(k-2)} = a_0^{-(k-2)} - (k - 2)\, \frac{s\, t}{\tau}, \tag{8.5}\]so the product stays small until shortly before \(t^\star = \tau / \bigl((k-2)\, s\, a_0^{k-2}\bigr)\): the wait grows as a power of \(1/a_0\), where for two factors it grows as a logarithm, by (8.3).
Proof. The factors start equal and obey the same equation, so they stay equal, and \(\tau \dot a = (s - u)\, a^{k-1} \approx s\, a^{k-1}\), which integrates to (8.5). ∎
During the wait every factor grows and the loss, which sees only the product, hardly moves: it has fallen by about \(s\, u(t)\). The progress is real and it is hidden. An induction circuit is such a product: it does next to nothing for the loss unless all of its pieces exist, and the gradient of each piece is proportional to the product of the others. Analyses of minimal models offer this explanation of the abruptness.Reddy, "The mechanistic basis of data dependence and abrupt learning in an in-context classification task", ICLR 2024 (arXiv 2312.03002); Nichani, Damian and Lee, "How transformers learn causal structure with gradient descent", ICML 2024 (arXiv 2402.14735). The run of Example 8.2.3 has this shape, and the product accounts for its cliff and not for its tail.
What does the forward pass compute when it learns in context? Three accounts compete.
Inference. A model that fits its training distribution exactly computes a posterior over whatever latent variable generated the document, and predicts by averaging over it:
\[p(\text{output} \mid \text{prompt}) = \int p(\text{output} \mid \text{concept}, \text{prompt})\; p(\text{concept} \mid \text{prompt})\; d(\text{concept}). \tag{8.6}\]The paper that proposed this account in 2021 draws its conclusion from the same integral, its equation 1.
If \(p(\text{concept }\vert \text{prompt})\) concentrates on the prompt concept with more examples, then the LM learns via marginalization by “selecting” the prompt concept. Thus, in-context learning can be viewed as the LM implicitly performing Bayesian inference.
—Xie, Raghunathan, Liang and Ma, "An explanation of in-context learning as implicit Bayesian inference" (2022). ICLR 2022 (arXiv 2111.02080), Section 1, the paragraph after equation 1; LM is the language model.
In one family of tasks the integral has a closed form.
Proposition 8.2.5 (the Bayes predictor of an in-context Markov chain). Let a sequence be generated by a Markov chain on \(k\) states whose transition matrix \(P\) has rows drawn independently and uniformly from the simplex, from an initial state whose law does not depend on \(P\). Given a context in which the transition from \(i\) to \(j\) occurred \(c_{ij}\) times, with \(N_i = \sum_j c_{ij}\), the posterior predictive probability that the state after \(i\) is \(j\) is \((c_{ij} + 1)/(N_i + k)\).
Proof. The likelihood of the context is the probability of the initial state times \(\prod_{i,j} P_{ij}^{c_{ij}}\), a product over rows, so under the posterior row \(i\) is Dirichlet with parameters \(1 + c_{ij}\), and the predictive probability is its mean. ∎
This is a bigram predictor whose table is counted in the context instead of stored in the weights. A two-layer transformer trained on fresh chains reaches it in the order of Example 8.1.3, one level up: uniform predictions, a plateau on unigram statistics counted in the context, then a rapid transition to the bigram solution.Edelman, Edelman, Goel, Malach and Tsilivis, "The evolution of statistical induction heads: in-context learning Markov chains", 2024 (arXiv 2402.11004), abstract and Section 2.
Optimization. The second account looks for an algorithm. A transformer trained on prompts \((x_1, f(x_1), \dots, x_N, f(x_N), x_q)\), with a fresh linear \(f\) in every prompt, predicts at \(x_q\) what least squares predicts.Garg, Tsipras, Liang and Valiant, "What can transformers learn in-context? A case study of simple function classes", NeurIPS 2022 (arXiv 2208.01066). A construction shows how attention can do such a thing. Let the context hold tokens \((x_i, y_i)\), \(i \le N\), and a query token \((x_q, 0)\). One step of gradient descent with rate \(\eta\) from zero on \(\tfrac{1}{2N} \sum_i (w^{\top} x_i - y_i)^2\) predicts \(\tfrac{\eta}{N} \sum_i y_i \langle x_i, x_q \rangle\) at the query, and one attention head without softmax writes that number into the query's empty slot: its query-key product reads the \(x\) parts, so that its score from the query to token \(i\) is \(\langle x_q, x_i \rangle\), and its output matrix moves \(\tfrac{\eta}{N} y_i\). Trained linear transformers find this construction, and with separate key and query matrices they learn it one principal component at a time, the staircase of Theorem 8.1.4 again.von Oswald and others, "Transformers learn in-context by gradient descent", ICML 2023 (arXiv 2212.07677), Proposition 1; Zhang, Singh, Latham and Saxe, "Training dynamics of in-context learning in linear attention", ICML 2025 (arXiv 2501.16265). The evidence is for linear and small models trained on regression.
Memory. The third account describes the weights. A sum of outer products, \(W = \sum_i v_i u_i^{\top}\) with unit keys \(u_i\), is a lookup table up to crosstalk,
\[W u_j = v_j + \sum_{i \ne j} \langle u_i, u_j \rangle\, v_i, \tag{8.7}\]and random keys in dimension \(d\) have inner products of order \(1/\sqrt d\) (Section 2). Gradient descent writes such sums unasked, since the gradient of the cross-entropy in \(W\) is an error times the input that caused it. On this reading each piece of the induction circuit is an associative memory, filled in an order that the data decide.Bietti, Cabannes, Bouchacourt, Jegou and Bottou, "Birth of a transformer: a memory viewpoint", NeurIPS 2023 (arXiv 2306.00802).
In the literature the dependence on data has a general form: in-context learning emerges when the training data are bursty and long-tailed.Chan and others, "Data distributional properties drive emergent in-context learning in transformers", NeurIPS 2022 (arXiv 2205.05055), abstract. It need not last: with more training in the same setting, "ICL first emerges, then disappears and gives way to IWL, all while the training loss decreases" (ICL and IWL are in-context and in-weights learning).Singh and others, "The transient nature of emergent in-context learning in transformers", NeurIPS 2023 (arXiv 2311.08360), abstract. The sharpest form is a threshold. A transformer trained on regression prompts from a short list of tasks matches the posterior mean over the list, which amounts to memorizing it. Above a threshold of diversity, between \(2^{14}\) and \(2^{15}\) tasks in the paper's setting, it matches ridge regression, which is not optimal for the distribution it was trained on and is the predictor that works on a new task.Raventós, Paul, Chen and Ganguli, "Pretraining task diversity and the emergence of non-Bayesian in-context learning for regression", NeurIPS 2023 (arXiv 2306.15063).
The work of 2025 and 2026 reads these results as one. A trained network does not run one in-context algorithm. It is a mixture of predictors, one that memorizes and one that generalizes, and what training time and data diversity change is the weight of each, under a trade-off between the loss a predictor reaches and the complexity of implementing it.Park, Lubana, Pres and Tanaka, "Competition dynamics shape algorithmic phases of in-context learning", ICLR 2025 (arXiv 2412.01003); Wurgaft and others, "In-context learning strategies emerge rationally", 2025 (arXiv 2506.17859). The memorizing predictor has the lower training loss on a finite list and a cost that grows with the list; the generalizing one costs the same at any diversity. So generalization wins when tasks are many, and when training is short, because a small advantage in loss needs many examples to pay for its complexity; transience is the moment it has paid. Section 8.7 gives the trade-off a formula. At scale the heads change as well: in an ablation study of 12 models, few-shot performance depends mainly, and more so in larger models, on another family, function-vector heads, whose mean outputs, added together, carry the task that the examples demonstrate; many of those heads begin training as induction heads.Yin and Steinhardt, "Which attention heads matter for in-context learning?", ICML 2025 (arXiv 2502.14010).
Remark 8.2.6 (what it changes). Plot the loss by position and not only its mean: the mean mixes what the weights hold with what the context adds. Look early: in gpt-pp the score on text formed between 0.6% and 2.1% of the way through the run, and a run sampled coarsely would not show it. Treat in-context ability as a property of the data as much as of the architecture: without repeated windows our two-layer model grew no induction circuit.
Grokking
The cleanest case of hidden progress needs a dataset of a few thousand examples and a great deal of patience.
In this paper we propose to study generalization of neural networks on small algorithmically generated datasets. In this setting, questions about data efficiency, memorization, generalization, and speed of learning can be studied in great detail. In some situations we show that neural networks learn through a process of "grokking" a pattern in the data, improving generalization performance from random chance level to perfect generalization, and that this improvement in generalization can happen well past the point of overfitting. We also study generalization as a function of dataset size and find that smaller datasets require increasing amounts of optimization for generalization. We argue that these datasets provide a fertile ground for studying a poorly understood aspect of deep learning: generalization of overparametrized neural networks beyond memorization of the finite training dataset.
—Power, Burda, Edwards, Babuschkin and Misra, "Grokking: generalization beyond overfitting on small algorithmic datasets" (2022). arXiv 2201.02177, abstract.
The version that has since been taken apart weight by weight is addition modulo the prime 113: a one-layer transformer reads the tokens \(a\), \(b\) and an equals sign and predicts \((a + b) \bmod 113\), trained with weight decay on 30% of the \(113^2\) pairs and tested on the rest.Nanda, Chan, Lieberum, Smith and Steinhardt, "Progress measures for grokking via mechanistic interpretability", ICLR 2023 (arXiv 2301.05217), Section 3: width 128, four heads, full-batch AdamW at weight decay 1 for 40,000 epochs.
a + b mod 113 on a model of one layer, trained on 3,830 of the 12,769 pairs
(AdamW, weight decay 1, 40,000 steps of the whole training set)
step 136 1,372 6,931 8,129 40,000
training pairs right 99.7 % 100 % 100 % 100 % 100 %
held-out pairs right 3.1 % 7.3 % 50.6 % 99.3 % 100 %
loss, held-out pairs 12.65 25.78 4.20 0.025 3.2 × 10⁻⁷
restricted loss 4.84 3.14 0.039 8.9 × 10⁻⁷ 7.0 × 10⁻⁸
power in 5 key frequencies 11.8 % 15.8 % 42.5 % 62.9 % 99.2 %
norm of the weights 58.2 57.8 44.9 38.4 30.7
fitted understood
└──── 60 times as many steps ────┘
at the end the frequencies 49, 10, 47, 33 and 1, of 56, hold 99.2 % of the embedding's power;
in the plane of frequency 10 the 113 numbers stand on a circle, the number a at 10·a/113 of a turn
(within 1.4° on average): adding 1 is a rotation
(the live figure needs JavaScript; it loads about 0.8 MB of data and 0.7 MB of three.js)
Our run separates three events: fitting the training pairs, concentrating the embedding in five frequencies, and answering the held-out pairs correctly. The frequencies begin to form while test accuracy still gives little sign of the generalizing solution.
Example 8.3.1 (modular addition, replayed). The setup: a one-layer transformer on \((a + b) \bmod 113\) after the recipe above, trained on 3,830 of the \(12{,}769\) pairs, 30%, by full-batch AdamW with weight decay 1 for 40,000 steps, and tested on the other 8,939. One seed.
All training pairs are right from step 144, where 3.3% of the held-out pairs are; 99% of the held-out pairs are right from step 8,129, 56 times later. (Measured from 99% training accuracy at step 136, the delay is 60-fold.) In between the held-out loss first rises, from 12.65 nats at step 136 to 25.78 at step 1,372, and five frequencies of the embedding, 49, 10, 47, 33 and 1, go from 12% of its power to 63%; at the end they hold 99.2%. They are not the paper's 14, 35, 41, 42 and 52: the frequencies depend on the seed. ◁
In the code. The model of this example is not a second implementation: it is gpt-pp's GPT class with its switches set to the paper's architecture, trained by a loop of four statements with an optimizer of its own. They are build_model, answer_logits and the loop of run in gpt-pp's interp/grokking.py.
def build_model(p: int, seed: int, d_model: int = 128, n_head: int = 4, init: str = "recipe") -> GPT:
"""The one-layer model of the paper as a configuration of the lab's GPT, initialized at the paper's scale."""
cfg = ModelConfig(vocab_size=p + 1, n_layer=1, n_head=n_head, d_model=d_model, block_size=3, pos="learned",
norm="none", embed_norm=False, act="relu", mlp_mult=4.0, qk_norm=False, bias=True,
tie_embeddings=False, logit_softcap=0.0, init="gpt2")
torch.manual_seed(seed)
model = GPT(cfg)
if init == "recipe":
with torch.no_grad():
for parameter in model.parameters():
if parameter.dim() >= 2:
nn.init.normal_(parameter, std=1 / math.sqrt(d_model))
return model
def answer_logits(model: GPT, tokens: Tensor, p: int) -> Tensor:
"""The logits of the p residues at the `=` position, in float64: (B, p)."""
return model(tokens)[:, -1, :p].double()
...
def run(p: int = 113, fraction: float = 0.3, steps: int = 40000, lr: float = 1e-3, weight_decay: float = 1.0,
seed: int = 0, data_seed: int = 0, init: str = "recipe", device: str = "cuda", every: int = 2000,
budget: float | None = None, resume: dict | None = None) -> dict:
...
tokens, answers = all_pairs(p)
train = train_mask(p, fraction, data_seed)
model = build_model(p, seed, init=init).to(device)
tokens, answers = tokens.to(device), answers.to(device)
train_idx, test_idx = torch.where(train)[0].to(device), torch.where(~train)[0].to(device)
x_train, y_train = tokens[train_idx], answers[train_idx]
optimizer = torch.optim.AdamW(model.parameters(), lr=lr, weight_decay=weight_decay, betas=(0.9, 0.98),
fused=str(device).startswith("cuda"))
...
for step in range(first, steps + 1):
...
loss = F.cross_entropy(answer_logits(model, x_train, p), y_train)
optimizer.zero_grad(set_to_none=True)
loss.backward()
optimizer.step()
Eight of the switches differ from the recipe's defaults: learned positions in place of rotary ones, no normalization of the stream, of the embedding or of queries and keys, a ReLU in place of its square, biases, no cap on the logits (the bound that Remark 8.3.4 speaks of is off here), and GPT-2's initialization, which the loop after GPT(cfg) then replaces. Every parameter with two or more dimensions, both tables and the position table included, is redrawn from \(\mathcal N(0, 1/d)\), a standard deviation of 0.088 at \(d = 128\) against the 0.02 of GPT-2; the biases keep their zeros. answer_logits reads the last of the three positions, the equals sign, and keeps the logits of the 113 residues: the vocabulary has 114 tokens and the loss is a softmax over 113. x_train holds all 3,830 training pairs, \(\lfloor 0.3 \cdot 12{,}769 \rfloor\) of them drawn by train_mask, and each pass of the loop takes one gradient of their mean loss, so a step is an epoch. The cast .double() is there because, by the module's own note, a float32 softmax cannot represent a loss below about \(10^{-7}\): the saturation from which the numerical account later in this section starts.
Quirk. The run shares the model's code with the rest of the series and nothing of its training. It uses neither the trainer of Section 7 nor the optimizers of Section 6: one torch.optim.AdamW holds every parameter, biases and both tables included, at a constant rate of \(10^{-3}\) with weight decay 1 and \((\beta_1, \beta_2) = (0.9, 0.98)\), on the whole training set at every step. The model is measured at 301 steps spaced evenly in the logarithm, so a step quoted above is known to that spacing: the logged step before 8,129 is 7,916.
PyTorch. On a graphics card fused=True selects the fused implementation of torch.optim.AdamW, here that of version 2.9.1. The release notes of 2.11 say that it now uses fused multiply-add instructions, which changes the last bits of an update; this is from the notes and was not tested, and the example was run on 2.9.1 only.
What the network has found is an algorithm, and it can be read off the weights.
We claim that the learned networks use the following algorithm (Figure 1):
Given two one-hot encoded tokens \(a,b\) map these to \(\sin({w_{k}a})\), \(\cos(w_{k}a)\), \(\sin(w_{k}b)\), and \(\cos({w_{k}b})\) using the embedding matrix, for various frequencies \(w_{k}=\frac{2k\pi}{P},k\in\mathbb{N}\).
Compute \(\cos\left(w_{k}(a+b)\right)\) and \(\sin\left(w_{k}(a+b)\right)\) using the trigonometric identities:
\[\cos\left(w_{k}(a+b)\right)=\cos\left(w_{k}a\right)\cos\left(w_{k}a\right)-\sin\left(w_{k}a\right)\sin\left(w_{k}b\right)\]
\[\sin\left(w_{k}(a+b)\right)=\sin(w_{k}a)\cos\left(w_{k}b\right)+\cos\left(w_{k}a\right)\sin\left(w_{k}b\right)\]
In our networks, this is computed in the attention and MLP layers.
For each output logit \(c\), compute \(\cos\left(w_{k}(a+b-c)\right)\) using the trigonometric identity:
\[\cos\left(w_{k}(a+b-c)\right)=\cos\left(w_{k}(a+b)\right)\cos\left(w_{k}c\right)+\sin\left(w_{k}(a+b)\right)\sin\left(w_{k}c\right).\]
This is a linear function of the already-computed values \(\cos(w_{k}(a+b))\), \(\sin(w_{k}(a+b))\) and is implemented in the product of the output and unembedding matrices \(W_{L}\).
The unembedding matrix also adds together \(\cos\left(w_{k}({a+b-c})\right)\) for the various \(k\)s. This causes the cosine waves to constructively interfere at \(c^{*}=a+b\mod p\) (giving \(c^{*}\) a large logit), and destructively interfere everywhere else (thus giving small logits to other \(c\)s).
With \(w_k = 2\pi k/P\), the two identities are \(\cos w_k(a + b) = \cos w_k a \cos w_k b - \sin w_k a \sin w_k b\) and its companion for the sine: products of a feature of \(a\) and a feature of \(b\), which the attention (by bringing both numbers to one position) and the MLP, the block's feed-forward layer (by multiplying, approximately), supply. The readout is then linear.
Proposition 8.3.2 (interference picks the right residue). Let \(P\) be prime, \(K \subset \{1, \dots, P - 1\}\) nonempty and \(\alpha_k > 0\). The logits
\[\ell(c) = \sum_{k \in K} \alpha_k \cos\bigl(w_k (a + b - c)\bigr), \qquad c \in \{0, \dots, P - 1\}, \tag{8.8}\]have their unique maximum \(\sum_k \alpha_k\) at \(c \equiv a + b \pmod P\).
Proof. At that \(c\) every cosine is one. For any other \(c\) put \(\delta = a + b - c\): the prime \(P\) divides neither \(k\) nor \(\delta\), so \(k\delta/P\) is not an integer and every cosine is below one. ∎
One frequency already gives the right answer and is nearly useless as a classifier: its runner-up is at \(\cos(2\pi/113) = 0.9985\) of the winner. More frequencies do not move the winner and scatter the runners-up: with the five frequencies \(K = \{14, 35, 41, 42, 52\}\) of the model of Nanda et al. and equal weights, the largest wrong logit is 2.59 against 5.Their Sections 4.1 and 4.2, where (8.8) explains 95% of the variance of the logits. The two ratios are evaluations of (8.8) over the 112 wrong residues. In our run the five waves can be read off the logits: frequencies 10, 49, 47, 33 and 1 with weights 12.6, 11.5, 7.7, 4.9 and 3.7 hold 96.6% of their variance, and the model answers all 12,769 sums.
a = 71, b = 60: 71 + 60 = 131 = 113 + 18, the answer modulo 113 is 18 a circle a frequency k: residue c stands at the angle 2πkc/113, and adding is turning frequency k 10 49 47 33 1 weight α 12.6 11.5 7.74 4.87 3.74 turn by a 102° 284° 191° 264° 226° turn by b 112° 6° 344° 188° 191° lands on tick 67 = 18 91 = 18 55 = 18 29 = 18 18 = 18 frequencies used 1 2 3 4 5 logit of 18 12.6 24.0 31.8 36.7 40.4 best wrong answers 12.6 23.6 27.4 25.2 21.5 as a share 99.85 % 98 % 86 % 69 % 53 % (the live figure needs JavaScript and loads 1 kB, the weights of our trained model)
Represent the 113 residues on a circle at multiples of one frequency, so adding \(b\) to \(a\) is one more turn, and the two turns end on the residue \(a + b\) whatever the frequency. A candidate's logit is the weighted sum, over the circles, of the cosine of its distance from that place. With the strongest circle alone the right answer wins by 0.15%; including further frequencies makes the best wrong answer fall to 98%, 86%, 69% and 53% of it, because the neighbors of the answer on one circle are far from it on the others. The embedding learns the first step of this algorithm: in the plane of frequency 10 the number \(a\) ends within 1.4° of \(10a/113\) of a turn on average.
Definition 8.3.3 (progress measures; Nanda et al.). Write the logits as a function of \((a, b)\) and take their discrete Fourier transform in both arguments. The restricted loss is the loss when only the constant and the key-frequency components are kept. The excluded loss is the loss on the training set when exactly the key-frequency components are removed.
In the code. Both measures are one projection of the logits of all 12,769 pairs and two subtractions; the function before them decides which frequencies count as key. They are key_frequencies and progress_measures in gpt-pp's interp/grokking.py.
def key_frequencies(power: Tensor, share: float = 0.1) -> list[int]:
"""The frequencies the trained embedding is made of: those holding at least `share` of the strongest one's
power, strongest first. In a grokked model these are a handful out of (p - 1) / 2; which ones depends on the seed."""
waves = power[1:]
order = torch.argsort(waves, descending=True)
return [int(k) + 1 for k in order if waves[k] >= share * waves.max()]
...
def progress_measures(logits: Tensor, answers: Tensor, train: Tensor, test: Tensor, directions: Tensor) -> Tensor:
...
along = directions.T @ (directions @ logits)
restricted = along + (logits - along).mean(dim=0, keepdim=True)
excluded = logits - along
return torch.stack((F.cross_entropy(restricted[test], answers[test]), F.cross_entropy(restricted, answers),
F.cross_entropy(excluded[train], answers[train])))
power is the squared norm of the embedding's component at the constant and at each of the 56 frequencies. logits has one row for each of the 12,769 pairs and one column for each candidate \(c\), so a column is one logit as a function of \((a, b)\); directions holds 20 orthonormal functions of the pair, built by key_directions: for each of the five key frequencies the four products of \(\cos w_k a\) or \(\sin w_k a\) with \(\cos w_k b\) or \(\sin w_k b\), the terms that the two identities above are made of. along is the projection of every column onto them, the key-frequency components of the definition, and the constant is the mean of the remainder over the pairs, one number for each candidate. The function returns the restricted loss on the held-out pairs and on all pairs, then the excluded loss on the training pairs; the restricted loss quoted below is the first.
Quirk. The key frequencies are chosen with hindsight. key_frequencies is called once, on the embedding of the last step, and both measures are then taken with its five frequencies at every saved step, as the paper does. As computed here the restricted loss is therefore not something to watch during a run: at step 6,931 it uses frequencies read off the model of step 40,000.
The restricted loss measures how good the Fourier circuit already is, the excluded loss how much the fit of the training set depends on anything else. They divide the run in three (the Gini coefficient measures how concentrated the spectrum is; our run records the power of the embedding at each frequency and no Gini coefficient).
Memorization (Epochs 0k–1.4k). We first observe a decline of both excluded and train loss, with test and restricted loss both remaining high and the Gini coefficient staying relatively flat. In other words, the model memorizes the data, and the frequencies \(w_{k}\) used by the final model are unused.
Circuit formation (Epochs 1.4k–9.4k). In this phase, excluded loss rises, sum of squared weights falls, restricted loss starts to fall, and test and train loss stay flat. This suggests that the model’s behavior on the train set transitions smoothly from the memorizing solution to the Fourier multiplication algorithm. The fall in the sum of squared weights suggests that circuit formation likely happens due to weight decay. Notably, the circuit is formed well before grokking occurs.
Cleanup (Epochs 9.4k–14k). In this phase, excluded loss plateaus, restricted loss continues to drop, test loss suddenly drops, and sum of squared weights sharply drops. As the completed Fourier multiplication circuit both solves the task well and has lower weight than the memorization circuit, weight decay encourages the network to shed the memorized solution in favor of focusing on the Fourier multiplication circuit. This is most cleanly shown in the sharp increase in the Gini coefficient for the matices \(W_{E}\) and \(W_{L}\), which shows that the network is becoming sparser in the Fourier basis.
—Nanda, Chan, Lieberum, Smith and Steinhardt, "Progress measures for grokking via mechanistic interpretability" (2023). Section 5.2, the three paragraphs on the phases; "matices" is the paper's spelling.
Our run shows the same sequence. The restricted loss, taken on the held-out pairs, is already 0.04 nats at step 6,931, when the model still loses 4.2 nats on them; the excluded loss goes from \(1.7 \times 10^{-4}\) at step 1,372 to 11.8 at step 8,129; and the norm of the weights falls from 60.1 at its top to 30.7 at the end. So the delay is not a delay in learning. The circuit grows all through the flat stretch of the test curve, hidden as in Proposition 8.2.4; what happens late is the removal of the memorized solution. Why is it removed?
We analyse the interplay between the internal mechanisms that the neural network uses to calculate the outputs, which we loosely call “circuits” (Olah et al., 2020). We hypothesise that there are two families of circuits that both achieve good training performance: one which generalises well (\(C_{\text{gen}}\,\)) and one which memorises the training dataset (\(C_{\text{mem}}\,\)). The key insight is that when there are multiple circuits that achieve strong training performance, weight decay prefers circuits with high “efficiency”, that is, circuits that require less parameter norm to produce a given logit value.
Efficiency answers our question above: if \(C_{\text{gen}}\,\)is more efficient than \(C_{\text{mem}}\,\), gradient descent can reduce nearly perfect training loss even further by strengthening \(C_{\text{gen}}\,\)while weakening \(C_{\text{mem}}\,\), which then leads to a transition in test performance. With this understanding, we demonstrate in Section 3 that three key properties are sufficient for grokking: (1) \(C_{\text{gen}}\,\)generalises well while \(C_{\text{mem}}\,\)does not, (2) \(C_{\text{gen}}\,\)is more efficient than \(C_{\text{mem}}\,\), and (3) \(C_{\text{gen}}\,\)is learned more slowly than \(C_{\text{mem}}\,\).
—Varma, Shah, Kenton, Kramár and Kumar, "Explaining grokking through circuit efficiency" (2023). arXiv 2309.02390, Section 1, two consecutive paragraphs.
Three properties then suffice for grokking: the generalizing circuit generalizes and the memorizing one does not, it is the more efficient of the two, and it is learned more slowly. A two-line model shows why the more efficient circuit ends with everything. If a circuit of depth \(\kappa \ge 2\) turns weights of norm \(\rho\) into a margin \(\epsilon \rho^\kappa\), with \(\epsilon\) its efficiency, the squared norm it needs for a margin \(o\) is \((o/\epsilon)^{2/\kappa}\), a concave function of \(o\) that vanishes at zero. Such a cost is subadditive, so a margin split between two circuits costs more norm than the same margin produced by the more efficient one alone, and a penalty on the squared norm switches the other off. A lookup table for \(D\) arbitrary pairs needs weights that grow with \(D\), while the algorithm costs the same for any \(D\), so the two cross at a critical size of training set: below it a network that has grokked un-groks when trained further, and near it test accuracy settles in between, at 83% in their experiment.Varma et al. (2023), Section 3 and Figure 1(c); the two-line model is a caricature of their argument.
Efficiency is one of several currencies in which the delay has been priced. In the lazy-to-rich account the fast solution is a kernel fit on the features of the initialization, the slow one is learned features, and the dial is the scale of the output or of the initialization. For homogeneous networks started at scale \(\alpha\) with weight decay \(\lambda\) this is a theorem: the flow stays in the kernel regime for a time just short of \(\tfrac1\lambda \log \alpha\). A numerical account adds that once every training point is fitted, growing the logits lowers the loss without changing a prediction, until the softmax saturates in floating point and the gradient is exactly zero.Kumar, Bordelon, Gershman and Pehlevan, "Grokking as the transition from lazy to rich training dynamics", ICLR 2024 (arXiv 2310.06110); Lyu and others, "Dichotomy of early and late phase implicit biases can provably induce grokking", ICLR 2024 (arXiv 2311.18817), Section 3; Prieto, Barsbey, Mediano and Birdal, "Grokking at the edge of numerical stability", 2025 (arXiv 2501.04697).
The accounts are not exclusive, and none covers both the cases that need weight decay and the cases that grok without it. For the pretraining of language models there is one report so far, on a model of seven billion parameters trained in one pass, which describes grokking as local and asynchronous; we know of no replication.Li, Fan and Zhou, "Grokking in LLM pretraining? Monitor memorization-to-generalization without test", ICLR 2026 (arXiv 2506.21551), abstract.
Remark 8.3.4 (what it changes). On a small or repeated dataset a flat validation curve is not a verdict: before stopping early, track something that sees the circuit, such as the weight norm, the spectrum of the embedding or a restricted loss. Bounding the logits closes the direction in which a fitted model lowers its loss without learning anything; gpt-pp caps its logits at \(15 \tanh(z/15)\). In one-pass pretraining every batch is new and the gap between training and test loss that defines grokking does not open. What carries over is that a circuit can grow for a long time before the loss shows it.
Double descent
Fit \(n\) noisy points with \(p\) random features by least squares, taking the solution of smallest norm when there are many, and let \(p\) grow.
least squares on p random ReLU features of 40 noisy points of 8 inputs
(noise 0.5 on a signal of 1; test error 1 = predicting zero; median of 24 draws)
features p 2 8 20 36 40 42 48 80 400
test error 1.10 0.93 0.58 4.41 25.0 32.4 4.39 0.78 0.40
▁▁▁▁▁▁▁▁▁▁▁▁▁▁▂▂▂▂▂▆▆▆▆▆█████▂▂▂▂▂▂▁▁▁▁▁▁▁▁▁▁▁
too few to fit the points p = n ten times n: every point
(best 0.58 at p = 20) (worst) fitted, the lowest error
with a ridge penalty of 0.01: no peak; 0.35 at p = 42, 0.22 at p = 400
the solvable case, p isotropic features, no ridge, in closed form
below n σ² p / (n − p − 1) p = 20: 0.26 (simulated 0.33)
above n 1 − n/p + σ² n / (p − n − 1) p = 80: 0.76 (simulated 0.79)
at p = n infinite
size of the fitted coefficients: 345 at p = 40, 24 at p = 400
(the live figure needs JavaScript; it computes everything in the page and loads no data)
We test this on 40 points of eight inputs with a linear target and label noise and random features \(\max(0, w \cdot x + c)\). In units of the error of predicting zero, the error on fresh points is 0.58 with 20 features, 32 with 42 and 0.40 with 400: the worst model is the one that can just fit its data, and the largest is better than the best small one. Left of the peak is the textbook trade-off between bias and variance; right of it, where the model fits its training data exactly and improves as it grows, is where networks large enough to fit their training set are used.
Breakthroughs in machine learning are rapidly changing science and society, yet our fundamental understanding of this technology has lagged far behind. Indeed, one of the central tenets of the field, the bias-variance trade-off, appears to be at odds with the observed behavior of methods used in the modern machine learning practice. The bias-variance trade-off implies that a model should balance under-fitting and over-fitting: rich enough to express underlying structure in data, simple enough to avoid fitting spurious patterns. However, in the modern practice, very rich models such as neural networks are trained to exactly fit (i.e., interpolate) the data. Classically, such models would be considered over-fit, and yet they often obtain high accuracy on test data. This apparent contradiction has raised questions about the mathematical foundations of machine learning and their relevance to practitioners. In this paper, we reconcile the classical understanding and the modern practice within a unified performance curve. This "double descent" curve subsumes the textbook U-shaped bias-variance trade-off curve by showing how increasing model capacity beyond the point of interpolation results in improved performance. We provide evidence for the existence and ubiquity of double descent for a wide spectrum of models and datasets, and we posit a mechanism for its emergence. This connection between the performance and the structure of machine learning models delineates the limits of classical analyses, and has implications for both the theory and practice of machine learning.
—Belkin, Hsu, Ma and Mandal, "Reconciling modern machine learning practice and the bias-variance trade-off" (2019). PNAS 2019 (arXiv 1812.11118), abstract.
Definition 8.4.1 (minimum-norm least squares). For \(X \in \mathbb R^{n \times p}\) and \(y \in \mathbb R^n\) the estimator is \(\hat\beta = X^{+} y\), with \(X^{+}\) the pseudo-inverse: the least-squares solution when \(X\) has rank \(p \le n\), and when it has rank \(n < p\) the interpolating solution of smallest Euclidean norm, \(X^{\top} (X X^{\top})^{-1} y\).
It is where gradient descent from zero ends: every gradient \(X^{\top}(X\beta - y)\) lies in the row space of \(X\), and \(X^{+} y\) is the only least-squares solution in that space. The preference for a small norm belongs to the algorithm, not to the class of models.
Theorem 8.4.2 (the risk of the minimum-norm solution). Let \(y_i = x_i^{\top} \beta + \varepsilon_i\) with \(x_i \sim \mathcal N(0, I_p)\) independent, noise independent of the features with mean zero and variance \(\sigma^2 > 0\), and \(\lVert \beta \rVert^2 = r^2\). The risk at a fresh point, \(R = \mathbb E\, (x^{\top} \hat\beta - x^{\top} \beta)^2 = \mathbb E\, \lVert \hat\beta - \beta \rVert^2\), is
\[R = \sigma^2\, \frac{p}{n - p - 1} \quad (p \le n - 2), \qquad R = r^2 \Bigl(1 - \frac np\Bigr) + \sigma^2\, \frac{n}{p - n - 1} \quad (p \ge n + 2), \tag{8.9}\]and it is infinite in between. As \(n, p \to \infty\) with \(p/n \to \gamma\), \(R \to \sigma^2 \gamma/(1 - \gamma)\) for \(\gamma < 1\) and \(R \to r^2 (1 - 1/\gamma) + \sigma^2/(\gamma - 1)\) for \(\gamma > 1\).
Proof. For \(p < n\), \(\hat\beta - \beta = (X^{\top} X)^{-1} X^{\top} \varepsilon\), so \(R = \sigma^2\, \mathbb E\, \mathrm{tr}\, (X^{\top} X)^{-1}\), and the inverse of a Wishart matrix of dimension \(p\) with \(n\) degrees of freedom has mean \(I/(n - p - 1)\). For \(p > n\), \(\hat\beta = \Pi \beta + X^{\top} (X X^{\top})^{-1} \varepsilon\) with \(\Pi\) the orthogonal projection on the row space of \(X\), and the error has two orthogonal parts. The first, \((I - \Pi)\beta\), is the part of \(\beta\) outside a uniformly distributed subspace of dimension \(n\), of mean squared norm \(r^2 (1 - n/p)\). The second has squared norm \(\varepsilon^{\top} (X X^{\top})^{-1} \varepsilon\), of mean \(\sigma^2 n/(p - n - 1)\) by the same fact with \(n\) and \(p\) exchanged. ∎Hastie, Montanari, Rosset and Tibshirani, "Surprises in high-dimensional ridgeless least squares interpolation", Annals of Statistics 2022 (arXiv 1903.08560), Theorem 1, prove the limit for features with independent entries, Gaussian or not.
The peak is an effect of conditioning, not of capacity: at \(p = n\) the matrix \(X\) is square and nearly singular, and the noise is divided by its smallest singular value; the fitted coefficients have norm 345 at \(p = n\) and 24 at ten times \(n\). Beyond the peak the noise is still fitted exactly, but it is spread over \(p - n\) spare directions which a fresh point hardly excites, and its cost \(\sigma^2 n/(p - n - 1)\) falls to zero: overfitting that does no harm, or benign overfitting.The term is the title of Bartlett, Long, Lugosi and Tsigler, "Benign overfitting in linear regression", PNAS 2020 (arXiv 1906.11300). In this isotropic model the bias grows meanwhile and the second descent ends at \(r^2\), the risk of predicting zero. It goes below the first when the added features also carry signal. If the response depends on \(d > n\) such features of which the model sees \(p\), the unseen ones act as noise, Theorem 8.4.2 applies to the seen part, and the risk falls all the way from the threshold to \(p = d\), where the model interpolates its noisy training set and, if the noise is small enough, is the best of all. Our random features are of a milder kind: the largest model wins with 40 points (0.40 against 0.58) and loses with 80 (0.36 against 0.27 with 24 features).
For networks the horizontal axis needs care, because the number of parameters is not the capacity of a training procedure. The effective model complexity of a procedure (architecture, optimizer and number of steps) is the largest number of samples it fits to a small training error, and the conjecture is that test error is non-monotone where it is close to the number of samples. Training longer adds complexity, so the peak also appears along the axis of epochs, and near the threshold more data can hurt.Nakkiran and others, "Deep double descent: where bigger models and more data hurt", ICLR 2020 (arXiv 1912.02292), Definition 1 and Hypothesis 1.
How much of the classical picture is left? More than the first papers suggested. The peak disappears under a ridge penalty of the right size (in our experiment a penalty of 0.01 leaves an error of 0.35 where the peak was), and the second descent vanishes when an effective number of parameters is plotted in place of the raw count.Nakkiran, Venkat, Kakade and Ma, "Optimal regularization can mitigate double descent", ICLR 2021 (arXiv 2003.01897); Curth, Jeffares and van der Schaar, "A U-turn on double descent: rethinking parameter counting in statistical learning", NeurIPS 2023 (arXiv 2310.18988). Theorem 8.4.2 is linear regression, and it has the whole phenomenon. What changed is where the control of complexity sits: no longer in the size of the class but in a preference inside a large class, here the minimum norm that gradient descent supplies. One statement of the lesson, from 2025, gives the preference a name.
Deep neural networks are often seen as different from other model classes by defying conventional notions of generalization. Popular examples of anomalous generalization behaviour include benign overfitting, double descent, and the success of overparametrization. We argue that these phenomena are not distinct to neural networks, or particularly mysterious. Moreover, this generalization behaviour can be intuitively understood, and rigorously characterized, using long-standing generalization frameworks such as PAC-Bayes and countable hypothesis bounds. We present soft inductive biases as a key unifying principle in explaining these phenomena: rather than restricting the hypothesis space to avoid overfitting, embrace a flexible hypothesis space, with a soft preference for simpler solutions that are consistent with the data. This principle can be encoded in many model classes, and thus deep learning is not as mysterious or different from other model classes as it might seem. However, we also highlight how deep learning is relatively distinct in other ways, such as its ability for representation learning, phenomena such as mode connectivity, and its relative universality.
—Wilson, "Deep learning is not so mysterious or different" (2025). ICML 2025 (arXiv 2503.02113), abstract.
A companion exposition, Bayesian deep learning, follows this argument through soft inductive biases and compression.
Where is a language model on this curve? A model trained in one pass on more text than it can store (Section 8.6 does the count) has no training set that it could interpolate. It sits on the classical side, far from the threshold, where whatever adds effective complexity helps: a larger model, a longer run. The peak returns where data repeat: repeating a tenth of a percent of the data a hundred times degraded a model of 800 million parameters to the level of one of 400 million.Hernandez and others, "Scaling laws and interpretability of learning from repeated data", 2022 (arXiv 2205.10487), abstract. We know of no study that shows the peak along a one-pass run of a large model.
Remark 8.4.3 (what it changes). Count parameters against data in units of what the procedure can fit, and stay away from the point where it can only just fit the training set: with a small or repeated dataset, a larger model and a tuned weight decay are both ways out. The preference for simple solutions comes from the optimizer and its starting point, which Section 8.5 varies.
Lazy and rich regimes
Theorem 8.4.2 is about a model whose features are fixed. A network becomes such a model if its weights hardly move, and for one scaling of a wide network this is exactly what happens.
Proposition 8.5.1 (training in function space). Let \(\Theta_\theta(x, x^{\prime}) = \nabla_\theta f_\theta(x)^{\top} \nabla_\theta f_\theta(x^{\prime})\) be the tangent kernel of a model \(f_\theta\). Under gradient flow \(\dot\theta = -\nabla_\theta \sum_i \ell(f_\theta(x_i), y_i)\),
\[\partial_t f_t(x) = -\sum_i \Theta_{\theta_t}(x, x_i)\; \partial_f \ell(f_t(x_i), y_i). \tag{8.10}\]If the kernel is constant in time and \(\ell\) is the squared loss, the outputs on the training set are \(f_t = y + e^{-\Theta t} (f_0 - y)\): each eigenvector of the kernel matrix is fitted at a rate equal to its eigenvalue.
Proof. By the chain rule \(\partial_t f_t(x) = \nabla_\theta f(x)^{\top} \dot\theta\), which is (8.10); with \(\ell = \tfrac12 (f - y)^2\) and a constant kernel matrix it reads \(\dot f = -\Theta (f - y)\). ∎
In the limit of infinite width, under the scaling that has taken the kernel's name, \(\Theta\) is deterministic at initialization and constant during training, and (8.10) describes the network exactly: the first exact account of the training of a deep network, and a simplicity bias fixed by a kernel before any data are seen.Jacot, Gabriel and Hongler, "Neural tangent kernel: convergence and generalization in neural networks", NeurIPS 2018 (arXiv 1806.07572), Theorems 1 and 2. A year later the regime was shown to be a choice.
In a series of recent theoretical works, it was shown that strongly over-parameterized neural networks trained with gradient-based methods could converge exponentially fast to zero training loss, with their parameters hardly varying. In this work, we show that this "lazy training" phenomenon is not specific to over-parameterized neural networks, and is due to a choice of scaling, often implicit, that makes the model behave as its linearization around the initialization, thus yielding a model equivalent to learning with positive-definite kernels. Through a theoretical analysis, we exhibit various situations where this phenomenon arises in non-convex optimization and we provide bounds on the distance between the lazy and linearized optimization paths. Our numerical experiments bring a critical note, as we observe that the performance of commonly used non-linear deep convolutional neural networks in computer vision degrades when trained in the lazy regime. This makes it unlikely that "lazy training" is behind the many successes of neural networks in difficult high dimensional tasks.
—Chizat, Oyallon and Bach, "On lazy training in differentiable programming" (2019). NeurIPS 2019 (arXiv 1812.07956), abstract.
Example 8.5.2 (both regimes in one mode). The setup: the mode of Theorem 8.1.4, \(f = ab\) with target \(s\), started at \(a = a_0\) and \(b = 0\). Its tangent kernel is the number \(\Theta = a^2 + b^2 = \sqrt{c^2 + 4u^2}\) with \(c = a_0^2\), and (8.2) is the kernel flow (8.10). If \(a_0^2 \gg s\) the kernel changes by a fraction of about \(2s^2/a_0^4\) over the whole of training, and every mode is learned at the same rate whatever its strength: the lazy regime. From a small balanced start the kernel grows from \(2u_0\) to \(2s\), and its growth is the learning: the sigmoid (8.3), strong modes first. What separates the regimes is how far the weights have to travel compared with where they start. ◁
For wide networks the alternative is a theorem, the dynamical dichotomy: every stable way of scaling the weights, their initialization and the learning rate with the width has an infinite-width limit that either lets the representations evolve or is kernel gradient descent in function space, never both. The standard and the tangent-kernel scalings give kernel limits; the maximal-update scaling, in which the update of every layer changes the input of the next by order one, gives feature learning.Yang and Hu, "Feature learning in infinite-width neural networks", ICML 2021 (arXiv 2011.14522), Section 1 and Corollary 3.9.
Language models live in the rich regime, and Section 8 has already collected the evidence. An induction head (Section 8.2) is a product of changes in several matrices, and a model that is linear in its parameters around their initial values contains no such product. The order in which a model learns follows the statistics of its data (Example 8.1.3, Theorem 8.1.4), not the eigenvalues of a kernel fixed at initialization. And the circuits of Section 9 and the features of Section 10 are, by definition, representations that were not there at the start.
Remark 8.5.3 (what it changes). The regime is set by how the initialization and the learning rates scale with width, which is why a recipe carries rules such as gpt-pp's factor \(\sqrt{768/d}\) on the learning rates of its embedding and unembedding at width \(d\): found by tuning, and the theory says that some such rule must exist. The scale of the initialization, or of the output, is a dial between the regimes, and Section 8.3 showed its price: turned toward the lazy side it buys a fast fit and a late generalization.
Memorization
A language model can be made to recite. GPT-2 produced hundreds of passages that stand word for word in its training data: 604 of the 1,800 candidates that were checked, some of them from a single document.
Definition 8.6.1 (extractable, k-eidetic; Carlini et al.). A string \(s\) is extractable from a model if there is a prefix after which \(s\) is the model's most likely continuation. It is \(k\)-eidetic memorized if it is extractable and occurs in at most \(k\) documents of the training data.Carlini and others, "Extracting training data from large language models", USENIX Security 2021 (arXiv 2012.07805), Definitions 1 and 2 and the sentence after Table 1.
How much can a model store? The measurement trains models on strings of uniformly random bits, where there is nothing to generalize, and counts the information that the weights end up holding. It levels off at a capacity proportional to the number of parameters, about 3.6 bits per parameter for GPT-style models from 500 thousand to 1.5 billion parameters.
We propose a new method for estimating how much a model knows about a datapoint and use it to measure the capacity of modern language models. Prior studies of language model memorization have struggled to disentangle memorization from generalization. We formally separate memorization into two components: unintended memorization, the information a model contains about a specific dataset, and generalization, the information a model contains about the true data-generation process. When we completely eliminate generalization, we can compute the total memorization, which provides an estimate of model capacity: our measurements estimate that GPT-style models have a capacity of approximately 3.6 bits per parameter. We train language models on datasets of increasing size and observe that models memorize until their capacity fills, at which point "grokking" begins, and unintended memorization decreases as models begin to generalize. We train hundreds of transformer language models ranging from \(500K\) to \(1.5B\) parameters and produce a series of scaling laws relating model capacity and data size to membership inference.
—Morris and others, "How much do language models memorize?" (2025). Morris, Sitawarin, Guo, Kokhlikyan, Suh, Rush, Chaudhuri and Mahloujifar, arXiv 2505.24832, abstract. Where the data exceed the capacity they find the double descent of Section 8.4.
Example 8.6.2 (capacity against data). The setup: the estimate of 3.6 bits per parameter, applied to the parameter counts of our models and set against the text they are trained on. gpt-pico has 2,228,224 parameters, hence room for about 8.0 million bits, one megabyte. TinyStories is 542,905,221 tokens, about 2.2 gigabytes of text, and at the rate gpt-pico itself reaches, 0.5515 bits per byte, it still holds 150.8 megabytes of information, about 150 times the capacity. The 12-layer gpt-pp has 135,266,304 parameters: 487 million bits, 61 megabytes, against 2,999,992,320 training tokens that hold 1,636 megabytes at its own rate of 0.9429 bits per byte, about 27 times the capacity. Both rates are the trainer's own readings; under the evaluation protocol they are 0.541 and 0.932 (Section 7), which moves neither ratio by more than 2%. Neither model has room for its corpus. ◁
What is repeated is the first candidate for storage.
Known issue. The held-out text of gpt-pp is disjoint from its training text by document, not by content. The sample of FineWeb-EDU, educational web pages, that gpt-pp is trained on contains 403,945 documents whose text repeats that of another document, and they are left in; the 960 training documents that shared their text or their URL with a held-out document were removed, excluding exact document duplicates from the benchmark of Section 7. Shared passages remain: 4.1% of held-out blocks of 64 tokens also occur in the text the main run read, and without those blocks the main model's held-out loss of 3.025 nats a token reads 3.040 (Section 7.1). Whether gpt-pp recites any of what it saw more than once was not tested: its code has no extraction test in the sense of Definition 8.6.1.
Remark 8.6.3 (what it changes). Deduplicate, or at least count the duplicates, and hold out by content and by source, not by position in a file. Report how many times a model has seen its data: one pass and forty epochs are different regimes of the same curve.
The developmental view
Every stage so far was found by someone who knew what to look for: a table of bigrams, a stripe in an attention pattern, a spectrum with five peaks. The developmental view asks for an observable that needs no such knowledge and still marks the stages. Its candidate measures how degenerate the loss is around the current weights.
Definition 8.7.1 (local learning coefficient; Lau et al.). Let \(w^\star\) be a local minimum of the population loss \(L\) and \(V(\epsilon)\) the volume of the set of weights \(w\) near \(w^\star\) with \(L(w) - L(w^\star) < \epsilon\). Under the regularity assumptions that Lau et al. make on the loss, there are a rational \(\lambda(w^\star) > 0\), an integer \(m(w^\star) \ge 1\) and a constant \(c > 0\) with
\[V(\epsilon) = c\, \epsilon^{\lambda(w^\star)}\, (-\log \epsilon)^{m(w^\star) - 1}\, (1 + o(1)) \qquad (\epsilon \to 0), \tag{8.11}\]and \(\lambda(w^\star)\) is the local learning coefficient.Lau, Furman, Wang, Murfet and Wei, "The local learning coefficient: a singularity-aware complexity measure", 2023 (arXiv 2308.12108), Definitions 1 and 2 and equations 2 to 12.
Example 8.7.2 (two minima). At a regular minimum, \(L = \sum_{i \le d} w_i^2\), the set is a ball of radius \(\sqrt\epsilon\) and \(\lambda = d/2\), half the number of parameters. For the mode of Theorem 8.1.4, \(L = \tfrac12 (s - ab)^2\), the minima form the curve \(ab = s\), the set is a strip around it of width proportional to \(\sqrt\epsilon\), and \(\lambda = \tfrac12\), half of what two parameters give at a regular minimum: the loss cannot see one of them. A smaller \(\lambda\) means more volume at a given loss: a more degenerate solution, and a simpler one. ◁
The coefficient enters the Bayesian weight of a solution. Write \(L_n\) for the empirical loss, the mean over \(n\) samples, of which \(L\) is the expectation. The posterior then gives a neighborhood of \(w^\star\) a weight \(e^{-F_n}\) with, to leading orders,
\[F_n \approx n\, L_n(w^\star) + \lambda(w^\star) \log n, \tag{8.12}\]the Bayesian information criterion with \(\lambda\) in the place of half the number of parameters. This is the trade-off that Sections 8.2 and 8.3 stated in words. Of two solutions, the one with the higher loss and the lower \(\lambda\) is preferred as long as \(n\, \Delta L < \Delta\lambda\, \log n\) and loses afterwards: the simple solution with few samples, the accurate one with many. The accounts of stage-wise development use it with training time in the place of $$n$$, as a heuristic: a sequence of stages, each of which pays in complexity for a lower loss.
To estimate \(\lambda\) at a checkpoint, perturb the weights thermally and see how much the loss goes up:
\[\hat\lambda(w^\star) = n \beta\, \Bigl( \mathbb E_{w}\, L_n(w) - L_n(w^\star) \Bigr), \qquad w \sim p(w) \propto \exp \Bigl( -n \beta L_n(w) - \tfrac{\gamma}{2} \lVert w - w^\star \rVert^2 \Bigr), \tag{8.13}\]with \(\beta = 1/\log n\) and the expectation taken by stochastic gradient Langevin dynamics (gradient steps with injected Gaussian noise) started at \(w^\star\). Estimated along the training of a small language model, \(\hat\lambda\) shows plateaus, and the plateaus separate stages that can then be identified by other means.
Plateaus in LLC estimates (Figures 1(a) and 1(a)) reveal five developmental stages for our language model. In order to validate that this stage division is meaningful, we search for concomitant changes in the model’s input/output behavior and its internal computational structure. In this section, we report a range of setting-specific metrics that reveal the following significant, interpretable changes coinciding with each stage: in LM1 the model learns to predict according to bigram statistics; in LM2 the model learns to predict frequent \(n\)-grams and use the positional embedding; in LM3 and LM4 the model respectively forms “previous-token heads” and “induction heads” as part of the same induction circuit studied by Olsson et al. (2022). Note that we did not discover significant changes in LM5, and we do not claim that these are the only interesting developmental changes occurring throughout training. There may be other interesting developmental changes that are not captured by our metrics, or are not significant enough to not show up in the LLC curve.
—Hoogland and others, "Loss landscape degeneracy and stagewise development in transformers" (2025). Hoogland, Wang, Farrugia-Roberts, Carroll, Wei and Murfet, TMLR 2025 (arXiv 2402.02364), Section 6, first paragraph, where the arXiv page prints both figure references as 1(a); LLC is the local learning coefficient. The model is a two-layer attention-only transformer of about three million parameters.
These are the stages of Sections 8.1 and 8.2 in the same order, marked off by a measure that knows nothing of them and then confirmed by looking for a change of behavior at each boundary (the authors found none at the last, and do not claim that these are the only changes): bigrams, longer n-grams, the previous-token head, then the induction head that needs it. Two cautions apply. Definition 8.7.1 assumes a minimum of the population loss, and a checkpoint is not one; and the value of (8.13) depends on the settings of the sampler. The authors present the link between degeneracy and development as suggestive evidence, and the plateaus are best taken as an empirical signal. We did not estimate the coefficient on our own checkpoints: their stages were found the other way, by knowing what to look for.
Remark 8.7.3 (what it changes). Keep the checkpoints: every measurement in Section 8 was made after the fact on saved weights, and the early ones matter most. Prefer observables that need no hypothesis about the circuit when searching for a stage, and observables that test one when explaining it.
A loss curve is a sum of events. Each one is a circuit that forms when its parts exist and the data pay for it, and each can be seen by measuring something more specific than the loss. Section 9 opens the circuits themselves, beginning with the bigram table and the induction head whose formation was followed here; Section 10 asks what the units of such circuits are.