7. Training gpt-pp
Draft
The data
gpt-pp is the product of one run. A model of 12 layers and width 768, with 135 million parameters, read three billion tokens of educational web text once, on three graphics cards, in 3.20 hours. The loss on text held out of training falls from 3.24 bits per byte, a uniform guess over the vocabulary, to 0.943.By the trainer's constant: the trainer turns its nats into bits per byte with the 4.628 bytes per token of the whole held-out file, where the tokens it scores have 4.635, so that its 0.9429 would read 0.9415; on the stories, gpt-pico's 0.552 would read 0.547. The scores of Section 7.4 are computed document by document and are not affected. Its log and 34 saved checkpoints record both the falling loss and the text the model could write. Two minutes into the run it continues "Photosynthesis is the process by which" with "a physical or material is formed, and it is a form of a…"; at the last checkpoint, with "a plant uses sunlight and energy to produce plants. However, the…".
tokens seen held-out loss written after "Photosynthesis is the process by which" 0 (step 0) 3.24 ██████████ Hend-sidedodend attacked regular geology… 1.9M (step 13) 2.07 ██████ have an ancient material and residents (see cell… 17M (step 115) 1.55 █████ a physical or material is formed, and it is a… 225M (step 1,529) 1.11 ███ a light bulb draws and converts light to form the… 3.00B (step 20,345) 0.94 ███ a plant uses sunlight and energy to produce… loss bits per byte of text held out of training, against tokens seen levels uniform guess 3.24 · token frequencies 2.40 · 4-gram table 1.58 scored by document: the released GPT-2 small 1.007, gpt-pp at step 20,345 0.932 run 3.00 billion tokens in 3.20 hours on three cards 264,000 tokens a second; held-out loss 3.24 at the start, 0.943 at the end charts the loss, the learning rate and the tokens a second against tokens seen; a scrubber over the 34 saved checkpoints; the three texts the model wrote at each (the live figure needs JavaScript and loads about 0.1 MB of data)
This part is the account of that run and of what it is worth: the text (this section), the pipeline (Section 7.2), the recipe and the run (Section 7.3), the evaluation (Section 7.4), a scaling law fitted to smaller runs (Section 7.5), sampling (Section 7.6), what the model cannot do (Section 7.7), the stage that turns continuing into answering (Section 7.8), and the cost (Section 7.9).
From a scrape to a graded corpus. GPT-2 read 40 GB of text from web pages linked on Reddit, a corpus that was never released. The public corpora of 2024 are filtered from public crawls of the web, and they differ in the filter. FineWeb is 15 trillion tokens from 96 snapshots of the Common Crawl, a public archive of the web, after text extraction, a language filter, the removal of near-copies and a few rules of thumb. FineWeb-EDU is the 1.3 trillion tokens of it that a classifier judged educational, and the classifier was taught by a large language model.
4 FineWeb-Edu
To build the synthetic annotations, we use Llama-3-70B-Instruct to score 460,000 randomly sampled webpages from the FineWeb CC-MAIN-2024-10 snapshot for their educational quality on a scale from 0 to 5. We explored several prompt formats to automatically extract an educational score using an LLM and found that the additive scale used in previous work Yuan et al. [63] worked best. It allows the LLM to evaluate each criterion and build the score step-by-step, unlike the single-rating scale [64] which assigns a fixed score based on predefined categories. To avoid having the LLM favor highly technical pages like arXiv abstracts and submissions, we prompted it to focus on grade-school and middle-school level knowledge. The prompt used for synthetic annotations is in Section F.1.
To scale our filtering to the entirety of FineWeb, we trained a linear regression model on top of the Snowflake-arctic-embed-m embedding model [65]. We fine-tuned this linear regressor on 410,000 of our Llama 3 synthetic annotations for 20 epochs with a learning rate of 3e-4 (while keeping the embedding and encoder layers frozen). We selected the checkpoint with the highest F1 score on the held-out validation set containing the remaining 50,000 samples, treating Llama 3 annotations as ground-truth. After training, we rounded the model’s output scores to integers from 0 to 5. We then used fixed thresholds to classify whether a given document from FineWeb was educational. We investigated the impact of using different thresholds for the filtering and ultimately chose a minimum threshold of 3 for FineWeb-Edu, which ultimately gave the best trade-off between performance on knowledge and reasoning intensive benchmarks and the performance on other benchmarks like HellaSwag [66]. With a threshold of 3, the model achieved an F1 score of 82% on the validation set, indicating strong performance in distinguishing high-quality educational content.
Applying the classifier to the 15 trillion tokens of FineWeb required 6,000 H100 GPU hours.
The paper measured what the filter buys. A model of 1.71 billion parameters trained on 350 billion tokens of each corpus scored 37% in place of 33% on MMLU, a test of academic knowledge, and 57% in place of 46% on ARC, a test of school science. A second group found the same lever in the same year with a cheaper tool, and said so where it summed up its first experiments:
As a starting point for DCLM, we conduct 416 baseline experiments with different training sets and compute scales. Our experiments identify model-based filtering as a key component for effective data curation. We also show that details of the filtering model can have a large impact on performance, ranging from 35% to 44% accuracy on MMLU 5-shot [77] at the 7B parameter scale (280B training tokens). Interestingly, a simple bigram classifier, combined with a carefully selected set of positive and negative examples, performs best among the classifiers we experimented with. In addition, we find that human quality judgments have only limited value in identifying high-quality training data.
—J. Li, A. Fang, G. Smyrnis and others, "DataComp-LM: in search of the next generation of training sets for language models" (2024). arXiv 2406.11794, Introduction, its seventh paragraph. In its Table 4 a fastText classifier scores 30.2 on the benchmark's core tasks, filtering by perplexity 29.0 and a classifier on the features of an embedding model 27.2. The MMLU and ARC figures are in Section 4 of the FineWeb paper.
By 2026 the single score of quality is giving way to the annotation of many properties of a document and to mixtures of sources found by search.Idahl, Droste, Plüster and Harries, "propella-1: multi-property document annotation for LLM data curation at scale", 2026 (arXiv 2602.12414); S. Diao and others, "Nemotron-CLIMB: clustering-based iterative data mixture bootstrapping for language model pre-training", NeurIPS 2025 (arXiv 2504.13161). One public recipe that left FineWeb-EDU for the second in March 2026 reports that its time to a fixed benchmark score fell from 2.76 to 2.02 hours; it is the recipe of the last table of Section 7.4. Its log also records what the educational filter is worth at equal settings: a benchmark score of 0.2602 on FineWeb-EDU and of 0.2241 on FineWeb.
Removing near-copies. A crawl holds many copies of a page. FineWeb finds them with MinHash, a sketch that makes the comparison of two documents cheap: a document is the set of its 5-grams, a hash function sends each 5-gram to a number and the document to the smallest of them, and two documents are compared by the minima they share.
Proposition 7.1.1 (MinHash in bands). Let \(A\) and \(B\) be finite nonempty sets with Jaccard similarity \(s = \lvert A \cap B\rvert / \lvert A \cup B\rvert\), and let \(h\) rank the elements of \(A \cup B\) in a uniformly random order. Then \(\Pr[\min h(A) = \min h(B)] = s\). With \(b\) bands of \(r\) independent rankings each, the two sets agree on every minimum of at least one band with probability
\[P(s) \;=\; 1 - (1 - s^{r})^{b} .\]Proof. The element of \(A \cup B\) ranked first is uniform on \(A \cup B\), and the two minima coincide exactly when it lies in \(A \cap B\). A band agrees with probability \(s^r\) by independence, and \((1 - s^r)^b\) is the probability that no band does. ∎
FineWeb takes \(r = 8\) and \(b = 14\), 112 numbers a document. Then \(P(0.7) = 0.56\), \(P(0.75) = 0.77\), \(P(0.8) = 0.92\) and \(P(0.85) = 0.988\): a switch that turns on between similarities of 0.6 and 0.9, with no pair of texts ever compared in full.The same four values, 56%, 77%, 92% and 98.8%, are in Appendix E.1 of the FineWeb paper. The filter is applied inside each snapshot of the crawl and not across snapshots, which the next example meets.
Example 7.1.2 (the corpus, cut and held out). The setup: the publisher's own random sample of FineWeb-EDU, 9,672,101 documents in 28.5 GB, read at a pinned revision under a license that asks for attribution; one random permutation of the documents fixes everything that follows.
The held-out set is the shortest prefix of the permutation with at least ten million tokens: 9,649 documents, 10,000,019 tokens, 46.3 MB. The training set is the rest, without the 960 documents that share their text (815) or their web address (702) with a held-out document: 9,661,492 documents and 9,896,904,130 tokens, written as 99 files of about 100 million tokens, every document led by a beginning-of-text token. The tokenizer of Section 2 was trained on the last 423,491 documents of the permutation, two billion characters, all of them training documents.
The sample holds 403,945 documents whose text repeats that of another, 4.2% of it, and 351,489 that repeat an address: a page fetched in several snapshots was never compared with itself. These repeats stay in the training set. After the removal no training document has exactly the tokens of a held-out one, and that was checked over every file. A second run from the downloaded files wrote the same bytes, file for file. The main run reads three billion tokens, 30% of the training set, once: the first 30 files, each in shuffled windows, and 720 windows of the 31st. ◁
In the code. How a run reads those files, seen from one card of several. It is TrainLoader in gpt-pp's data.py, with the function that cuts its windows above it.
def _windows(tokens: np.ndarray, starts: np.ndarray, seq_len: int) -> np.ndarray:
...
return tokens[starts[:, None] + np.arange(seq_len + 1)[None, :]]
...
def _next_shard(self) -> None:
...
for _ in range(len(self.data.train_files) + 1):
self.shard_index += 1
if self.shard_index == len(self.data.train_files):
self.shard_index, self.epoch = 0, self.epoch + 1
self.tokens = self.data.shard(self.shard_index)
n_windows = (len(self.tokens) - 1) // self.seq_len
if n_windows >= self.world * self.device_batch:
# The same permutation on every rank: it depends on the seed, the pass and the shard only.
rng = np.random.default_rng([self.seed, self.epoch, self.shard_index])
self.order = rng.permutation(n_windows)
self.cursor = 0
return
...
def next_batch(self) -> tuple[Tensor, Tensor]:
...
if self.cursor + self.world * self.device_batch > len(self.order):
self._next_shard() # the few windows left over at the end of a shard are dropped
first = self.cursor + self.rank * self.device_batch
starts = self.order[first:first + self.device_batch] * self.seq_len
self.cursor += self.world * self.device_batch
self.staging_view[:] = _windows(self.tokens, starts, self.seq_len)
# A blocking copy: when it returns the card has the batch and the staging buffer may be refilled.
block = self.staging.to(self.device)
return block[:, :-1].contiguous(), block[:, 1:].contiguous()
A shard is one of the 99 files and a rank is one card's process. A window is seq_len + 1 consecutive tokens, the first seq_len the inputs and the last seq_len the targets, and order is a permutation of a shard's windows that depends on the seed, the pass and the shard only, so every rank computes the same one. A step takes the next world * device_batch windows of it and rank rank reads its own slice, so the text of a step does not depend on the number of cards. A window is cut without regard to documents, so the beginning-of-text token is also a target wherever a document begins inside one, and the model learns to end a document by writing that token (Section 2).
Quirk. The data is read once, shard by shard, without replacement, where nanoGPT draws every window at random and with replacement. Consecutive windows share one token, the last target of one being the first input of the next, so that every token is a target once. The windows left at the end of a shard, fewer than the 144 of a step in the main run, are dropped.
Remark 7.1.3 (held out of what). The words have three meanings in this part. The 9,649 documents are held out of our training set by exact text and by address. Near-copies are another matter: by Proposition 7.1.1 a copy at similarity 0.7 inside one snapshot passes the publisher's filter almost half the time, and copies in different snapshots are not compared at all. Measured on the text that the main run read, 4.1% of the held-out blocks of 64 tokens also stand there word for word, boilerplate for the most part, and without them the final loss of 3.025 nats per token would read 3.040 (Section 8.6 asks what a model does with text it has seen more than once). Second, the documents are held out of nobody else's training set: two models of Section 7.4 were trained on FineWeb-EDU itself, and any model trained on a crawl may have read the same pages. Third, only text written after every model's data was collected is held out of all of them. Section 7.4 has one such set, and it is small.
Known issue. The held-out text is disjoint from the training text by document and not by content. Beyond the repeated blocks of the remark, two of the 1,000 held-out documents that Section 7.4 scores have a near copy in the training set; without those two, the 0.9322 bits per byte of gpt-pp become 0.9324.
Example 7.1.4 (what the tokenizer makes of it). The setup: the tokenized corpus, counted. A token is 4.63 bytes on the held-out documents. A document has 1,023 tokens on average and 620 at the median; 26.8% of the documents are longer than the model's context of 1,024 tokens, and they hold 64.8% of all tokens, so most of what the model reads is the inside of a long page seen through a window. Token frequencies alone cost 2.400 bits per byte on the held-out text, and a table of 4-grams counted on 200 million training tokens costs 1.577, with 65.77% of the held-out 4-grams absent from those 200 million (Example 1.3.3, Example 1.4.5). ◁
Remark 7.1.5 (what the corpus lacks). The byte { does not occur once in the 9.9 billion training tokens. The byte } occurs 7,395 times and the tab 42 times; in the 46 million characters of held-out text there is no blank line, and two spaces in a row occur 310 times. Why the dataset is so was not verified, and the consequence does not depend on the reason: a model trained on this corpus has never seen an opening brace, a JSON object, an indented line of code or a table, and the token for { is one of 50 in its vocabulary that the training set does not contain, the four chat tokens of Section 7.8 among them (Section 2.2). The corpus is also filtered to English. Section 7.7 collects the bill.
gpt-pico, the smallest size of gpt-pp and the one that runs in the page, reads a corpus made for models of its size.
Language models[4, 5, 21] (LMs) are powerful tools for natural language processing, but they often struggle to produce coherent and fluent text when they are small. Models with around 125M parameters such as GPT-Neo (small) [3] or GPT-2 (small) [23] can rarely generate coherent and consistent English text beyond a few words even after extensive training. This raises the question of whether the emergence of the ability to produce coherent English text only occurs at larger scales (with hundreds of millions of parameters or more) and complex architectures (with many layers of global attention). In this work, we introduce TinyStories, a synthetic dataset of short stories that only contain words that a typical 3 to 4-year-olds usually understand, generated by GPT-3.5 and GPT-4. We show that TinyStories can be used to train and evaluate LMs that are much smaller than the state-of-the-art models (below 10 million total parameters), or have much simpler architectures (with only one transformer block), yet still produce fluent and consistent stories with several paragraphs that are diverse and have almost perfect grammar, and demonstrate reasoning capabilities. We also introduce a new paradigm for the evaluation of language models: We suggest a framework which uses GPT-4 to grade the content generated by these models as if those were stories written by students and graded by a (human) teacher. This new paradigm overcomes the flaws of standard benchmarks which often require the model’s output to be very structured, and moreover it provides a multidimensional score for the model, providing scores for different capabilities such as grammar, creativity and instruction-following. We hope that TinyStories can facilitate the development, analysis and research of LMs, especially for low-resource or specialized domains, and shed light on the emergence of language capabilities in LMs.
—R. Eldan and Y. Li, "TinyStories: how small can language models be and still speak coherent English?" (2023). arXiv 2305.07759, abstract. gpt-pico reads the second version of the dataset, written by GPT-4 alone: 2,717,495 stories and 542,905,221 tokens under a vocabulary of 4,096, with 27,630 stories held out.
The pipeline, and where the time goes
The run reads 263,957 tokens per second on its three cards, and the same recipe 91,551 on one. A training loop written in the plainest way, on that one card, reads 15,548. Nothing in the difference changes what is computed, beyond rounding.
bars tokens per second of the 12-layer model, 48 sequences of 1,024 tokens a card, as each
piece is switched on
float32, attention written out, not compiled 15,500 ██
+ TensorFloat-32 products 25,000 ×1.61 ██
+ bfloat16 products 27,500 ×1.10 ███
+ the fused attention kernel 46,600 ×1.70 █████
+ compilation: the recipe on one card 91,600 ×1.96 █████████
the same on one card, a whole run 89,600 █████████
two cards 177,100 ×1.98 █████████████████
three cards: the main run 264,000 ×1.49 ██████████████████████████
step one step of the recipe on one card, 32 sequences: 328 ms
the blocks, forward and backward 236 ms 72 %
unembedding, soft cap, loss 57 ms 17 %
optimizer: Muon and AdamW 35 ms 11 %
loading the batch 0.2 ms 0.06 %
On one card the pieces multiply the rate by 5.9, from 15,500 to 91,600 tokens a
second; the largest of them is compilation (×1.96). Three cards read 264,000, 98 % of
three times a whole run on one card.
(the live figure needs JavaScript)
In the code. The model, its optimizers and one step of the loop, with the bookkeeping between them left out. They are lines of train in gpt-pp's train.py.
torch.backends.cuda.matmul.allow_tf32 = True # TF32: float32 matrix products with a 10-bit mantissa
torch.backends.cudnn.allow_tf32 = True
torch.manual_seed(train_cfg.seed) # the same initialization on every rank
model = GPT(model_cfg).to(device)
torch.manual_seed(train_cfg.seed + 1 + rank) # different dropout masks on every rank
optimizers = build_optimizers(model, train_cfg, tokens_per_step)
compiled = torch.compile(model) if train_cfg.compile else model
net = DistributedDataParallel(compiled, device_ids=[device.index]) if world > 1 else compiled
autocast = torch.autocast("cuda", dtype=torch.bfloat16, enabled=on_gpu)
...
try:
for step in range(n_steps + 1):
...
# The rate of this step is the schedule read at the middle of the step.
multiplier = lr_multiplier((step + 0.5) * tokens_per_step, total_tokens, train_cfg)
set_lr(optimizers, multiplier)
for micro_step in range(train_cfg.grad_accum):
x, y = train_loader.next_batch()
if world > 1: # ranks exchange gradients only on the last backward pass of the step
net.require_backward_grad_sync = micro_step == train_cfg.grad_accum - 1
with autocast:
loss = net(x, y)
(loss / train_cfg.grad_accum).backward()
loss_sum += loss.detach() / train_cfg.grad_accum
if train_cfg.grad_clip > 0:
torch.nn.utils.clip_grad_norm_(model.parameters(), train_cfg.grad_clip)
for optimizer in optimizers:
optimizer.step()
for optimizer in optimizers:
optimizer.zero_grad(set_to_none=True)
model is the module, compiled its compiled form and net that form wrapped by PyTorch's DistributedDataParallel when a run has several cards; net(x, y) returns the mean loss in nats and nothing else, since with targets the model never returns its logits (Section 1). Under autocast the products run in bfloat16 and the loss in float32 (Section 6.7). The rate is read in tokens, at the middle of the step, on a warmup that begins at zero, so the first step's rate is half a step's worth of warmup (Section 6.3). optimizers is the list [adamw, muon]: AdamW on the two vocabulary matrices steps first and Muon on the hidden matrices second (Section 6.5), an order that changes nothing, since the two hold disjoint parameters. The recipe makes one pass a step and does not clip; where a recipe clips, as that of 2019 does at norm one, the norm is taken once over all parameters before any optimizer steps (Section 6.4).
PyTorch. The first two lines switch TensorFloat-32 on for the whole process and never switch it back. In PyTorch 2.9.1 torch.backends.cuda.matmul.allow_tf32 is off by default; torch.backends.cudnn.allow_tf32 is on by default, and the model has no convolution for it to act on. The evaluator and the chat stage set the same two flags, and a test that ran one of them loosened every later float32 comparison on a card until the fixture exact_float32_products was added to the tests. The documentation of 2.9 calls the flags "going to be deprecated" in favor of fp32_precision. The release notes of 2.10 to 2.15 say that the compiler has moved off the flags and that no release removes them; the series tested none of those versions.
Quirk. Every rank builds the same model, because torch.manual_seed(train_cfg.seed) precedes its construction on each; the seed then becomes seed + 1 + rank, which only dropout would see, and the recipe has no dropout. The same seed fixes the order of the data. Two runs of one seed and one configuration still do not repeat bit for bit, because some kernels of the card are not deterministic: two such pairs of full runs ended 0.0023 and 0.0003 nats apart, inside the spread between seeds (Section 7.9).
Quirk. The model is compiled for training and for the held-out loss, not for sampling or saving. The loss and the trainer's monitor call compiled; the sample written at each reading calls the uncompiled model through its other path, the one that takes the cache (Section 7.6); checkpoints save model, so their keys carry no prefix. All 211 recorded runs were compiled.
Quirk. The budget is rounded to whole steps, n_steps = max(1, round(tokens / tokens_per_step)), and the schedule is laid over n_steps * tokens_per_step, so the tokens used are not the tokens asked for: 3.0 billion asked and 2,999,992,320 used in the main run, 100 million asked and 100,007,936 used in a run of 1,526 steps.
Quirk. The vocabulary size comes from the data: where a preset's vocab_size differs from that of the tokenized corpus, the trainer replaces it and says so. An override given with --set is parsed as the declared type of its field, and an unknown key is an error.
Quirk. A checkpoint holds the weights, the model's configuration, the step and the tokens, and nothing else. save_checkpoint writes no optimizer state, and Muon keeps its momentum outside the state that PyTorch would save in any case (Section 6.5). Loading refuses a configuration with an unknown field and fills a missing one with today's default, with a warning.
Known issue. So a run cannot be resumed. An interruption leaves interrupted_<step>.pt with the weights only and no summary, a crash leaves an error record in the log, and a loss that is not finite stops the run at the next log line.
Example 7.2.1 (one step, timed). The setup: the forward and backward pass and the optimizer's step, timed apart over a few seconds on one card, for the 8-layer model of the ladder and for the 12-layer model, both in the 2026 recipe.
| 8 layers, 512 wide, 128 × 512 tokens | 12 layers, 768 wide, 32 × 1,024 tokens | |
|---|---|---|
| forward and backward pass | 250 ms | 293 ms |
| of which the unembedding, the soft cap and the loss | 84 ms | 57 ms |
| optimizer step, Muon and AdamW | 11 ms | 35 ms |
| optimizer step if every parameter used AdamW | 3 ms | 6 ms |
| loading the batch | 0.3 ms | 0.2 ms |
The time is in the model: loading a batch costs about a thousandth of a step. The last layer is the largest single item: 32,768 logits for every token are a third of the 8-layer model's pass and a fifth of the 12-layer model's, where the unembedding holds 40% and 23% of the parameters that cost arithmetic. Muon's loop is most of the optimizer's step, 4% of the whole at the smaller size and 11% at the larger with this batch; its cost does not depend on the batch (Example 6.5.10). ◁
The changes, measured in this order:
- Arithmetic. The matrix products run in bfloat16 and the weights stay in float32 (Section 6.7); the products that remain in float32 use TensorFloat-32, a mode of the card that keeps ten bits of their fraction.
- Attention. Scores, mask, softmax and average are one fused kernel that never writes the square table of scores to the card's memory (Section 3.6).
- Compilation. The forward pass is traced once into a single graph, so that the interpreter leaves the loop and adjacent operations are fused.
- Cards. Each card holds the whole model and reads its own slice of every step.
| switched on | tokens per second | factor | passes a step |
|---|---|---|---|
| nothing: float32, attention written out, no compilation | 15,548 | 4 | |
| TensorFloat-32 products | 25,003 | 1.61 | 4 |
| bfloat16 products | 27,452 | 1.10 | 4 |
| the fused attention kernel | 46,637 | 1.70 | 2 |
| compilation: the recipe on one card | 91,551 | 1.96 | 1 |
| the recipe over a whole run on one card | 89,590 | 1 | |
| two cards | 177,069 | 1 | |
| three cards: the main run | 263,957 | 1 |
On one card the pieces multiply the rate by 5.9. The largest single step is compilation, a factor of 1.96, and not the fused attention kernel, at 1.70. Left to the interpreter, each small operation of the recipe is a pass of its own over the card's memory, and the logits alone are 6 GB in float32 for 48 sequences. The fused kernel is worth memory as well as time: with attention written out, a sequence of 1,024 tokens takes 2.2 GB of the card, and with the kernel 1.1, which is why a step then fits in two passes instead of four. The kernel is reached through F.scaled_dot_product_attention, a function that PyTorch 2.9.1 documents as beta (Section 3.6); handed the bias of ALiBi as a mask, it read 6% fewer tokens per second in a short timing with 32 sequences of 512 tokens (Section 4). bfloat16 adds only 10% at its place in the order, because the square tables of scores stay in float32 while attention is written out. The factor of a piece depends on the pieces that are already on.The lecture of 2024 that rebuilt GPT-2 on one A100 card, with 16 sequences of 1,024 tokens, reports 1,000 ms a step in float32, 333 with TensorFloat-32, 300 in bfloat16, 130 compiled and 96 with the fused kernel: there too compilation is the larger of the last two.
The frontier has gone further down in precision: matrix products in eight bits were reported at scale in 2024, and in four bits in 2025.DeepSeek-AI, "DeepSeek-V3 technical report", 2024 (arXiv 2412.19437), Section 3.3, which keeps the embedding, the output head, normalization and attention at the higher precision; NVIDIA, "Pretraining large language models with NVFP4", 2025 (arXiv 2509.25149). Neither was tried on our cards. Mixed precision and data loaders on another machine: Deep learning at scale.
What the three cards do together has a precise statement.
Proposition 7.2.2 (one step on the averaged gradient). Let the loss of a batch be the mean of its per-token losses, \(L_B(\theta) = \lvert B\rvert^{-1}\sum_{x \in B}\ell(x; \theta)\), and let \(B\) be split into \(k\) parts \(B_1, \dots, B_k\) of equal size. Then
\[\nabla L_B(\theta) \;=\; \frac{1}{k}\sum_{j=1}^{k} \nabla L_{B_j}(\theta) .\]Hence (i) \(k\) cards that start from the same parameters and the same optimizer state, compute \(\nabla L_{B_j}\) each, average the results and take the optimizer's step on the average, end with the same parameters and the same state as one another and as one card that had read \(B\); and (ii) if the parts are independent draws, the covariance of the averaged gradient is \(1/k\) of one part's.
Proof. The gradient is linear and the parts have equal weight in the mean. For (i), the step is a function of the parameters, the state and the gradient, which are equal on every card, and the claim follows by induction on the steps. (ii) is the variance of a mean. ∎
The conditions are that no layer's output depends on the rest of its batch, which holds for every layer of the model, and that sums of floating-point numbers taken in another order are counted as equal.
Quirk. In the trainer the cards exchange gradients and nothing else. Each holds the whole model and its own optimizer state, reads its slice of the step and takes the same step on the averaged gradient, so Muon's orthogonalization is repeated on every card. A test trains with two cards of eight sequences and with one card of 16: the loss before the first step agrees to \(10^{-4}\) and every later record to \(5 \times 10^{-3}\), which is all that the test asks (Section 11). Every run records at its end the largest difference between the cards' sums of all parameters, rank_spread, which was zero in the runs on two cards and in the main run on three.The averaging is that of PyTorch's DistributedDataParallel. Other ways of sharing work among cards, and their costs: GPU programming for exascale. When a step is accumulated over several passes, the cards exchange gradients on the last pass only: the loop sets require_backward_grad_sync, the attribute that DistributedDataParallel.no_sync() toggles. Accumulation on several cards is exercised by no test and by no run.
Remark 7.2.3 (cards buy batch, not steps). With three cards a step reads 147,456 tokens, and the run takes 20,345 steps where one card would have taken 61,035. By part (ii) of the proposition each step is taken on a gradient with a third of the variance; it is not three steps. Muon's rate of 0.02 was chosen at 65,536 tokens a step and is used unchanged at 147,456 (Section 6.8): the trainer can scale every rate by the square root of the batch, through lr_ref_batch, and the recipe leaves that off. The run was not repeated at another batch.
Example 7.2.4 (throughput). The setup: the 12-layer model in the 2026 recipe at context 1,024, with the rates that the trainer reports over a whole run, the first interval, which pays for compilation, left out. One card reads 89,590 and 94,213 tokens per second in two runs of the same configuration, on different cards: a rate is good to about 3%. A step of 48 sequences uses 22.5 GB of the card's 32, and 64 do not fit. Two cards read 177,069 tokens per second and the three cards of the main run 263,957. Against the slower of the two runs on one card, that is 99% of twice and 98% of three times one card; against the faster, 94% and 93%. The rest is the exchange of gradients. The 2019 recipe at the same shape reads 100,416 on one card: AdamW on every parameter is cheaper than Muon's loop, by 6% to 11% of the rate. In the count of Definition 1.7.1 the main run is \(6ND = 1.98 \times 10^{18}\) floating-point operations, with \(N = 110{,}100{,}480\) parameters in the blocks and the output layer and \(D = 2{,}999{,}992{,}320\) tokens. ◁
Known issue. The preset of the main model asks for a batch that these cards cannot hold: gptpp-base has device_batch=64 at a context of 1,024, the 64 sequences that do not fit in 32 GB. The main run was launched with --set device_batch=48, hence its step of 3 × 48 × 1,024 = 147,456 tokens, and with --eval-tokens 50e6 --save-log 24 --save-steps 2000,4000,...,20000; of its 34 checkpoints, --save-log 24 alone would have written 24. The preset run as it stands stops for lack of memory on the cards that trained the model (Section 11 has the command).
The recipe and the run, replayed
The recipe is the sum of the choices that Section 2 to Section 6 argued one at a time. The table sets it beside the recipe of 2019 at the same shape, which is the other runner of Section 7.4.
| the 2019 recipe | the 2026 recipe, gpt-pp | argued in | |
|---|---|---|---|
| layers, width, heads, context | 12, 768, 12, 1,024 | the same | |
| vocabulary | 32,768 | the same | Section 2.1 |
| parameters | 111,008,256 | 135,266,304 | |
| of them in attention and MLPs | 85,017,600 | 84,934,656 | |
| position | a learned vector for each position | rotary | Section 4.4 |
| normalization | layer norm with gain and bias | RMS norm without parameters, also after the embedding | Section 5.2 |
| MLP | GELU | squared ReLU | Section 5.3 |
| attention score | \(q \cdot k / \sqrt{d_{\mathrm{head}}}\) | \(12 \cos(q, k)\) | Section 3.4 |
| biases | in every linear layer | none | Section 5.6 |
| embedding and unembedding | one matrix | two | Section 2.5 |
| logits | unbounded | \(15 \tanh(z/15)\) | Section 5.6 |
| writers at initialization | \(\mathcal N(0, 0.02^2)\), scaled by \(1/\sqrt{2L}\) | zero | Section 5.5 |
| optimizer | AdamW on every parameter, \(\beta = (0.9, 0.95)\) | Muon on the hidden matrices; AdamW, \(\beta = (0.8, 0.95)\), on the two vocabulary matrices | Section 6.5 |
| learning rate | \(6 \times 10^{-4}\) | 0.02 for Muon; 0.3 for the embedding, 0.008 for the unembedding | Section 6.8 |
| schedule, in tokens | warmup over 32M, cosine to a tenth | warmup over 2M, flat, linear over the last 65% to a twentieth | Section 6.3 |
| weight decay, clipping | 0.1 on matrices; gradient norm clipped at 1 | none | Section 6.4 |
| arithmetic | bfloat16 products, float32 weights | the same | Section 6.7 |
One stage. The run has one corpus, one sequence length, one batch size and one pass, where the published recipes of 2025 and 2026 change their mixture of web text, code and mathematics along the way and bring long sequences last.Bakouch and others, "SmolLM3: smol, multilingual, long-context reasoner", Hugging Face blog, 8 July 2025: web, code and mathematics at 85%, 12% and 3%, then 75%, 15% and 10%, then 63%, 24% and 13%, with "the AdamW optimizer (beta1: 0.9, beta2: 0.95) with weight decay of 0.1 and gradient clipping of 1", on 384 cards for 24 days. The paper on Llama 3 gives the reason for the order:
In the final stages of pre-training, we train on long sequences to support context windows of up to 128K tokens. We do not train on long sequences earlier because the compute in self-attention layers grows quadratically in the sequence length. We increase the supported context length in increments, pre-training until the model has successfully adapted to the increased context length. We assess successful adaptation by measuring whether (1) model performance on short-context evaluations has recovered completely and (2) the model perfectly solves “needle in a haystack” tasks up to that length. In Llama 3 405B pre-training, we increased context length gradually in six stages, starting from the original 8K context window and ending in the final 128K context window. This long-context pre-training stage was performed using approximately 800B training tokens.
—Grattafiori and others, "The Llama 3 herd of models" (2024). arXiv 2407.21783, Section 3.4.2, whole.
Their optimizer was that of the left column through 2025; in 2026 the newest open recipe of the frontier has the split of the right column.
DeepSeek-V4-Flash. We employ the Muon optimizer (Jordan et al., 2024; Liu et al., 2025) for the majority of parameters, but use the AdamW optimizer (Loshchilov and Hutter, 2017) for the embedding module, the prediction head module, and the weights of all RMSNorm modules. For AdamW, we set its hyper-parameters to \(\beta_{1}=0.9\), \(\beta_{2}=0.95\), \(\varepsilon=10^{-20}\), and \(\mathrm{weight\_decay}=0.1\). For Muon, we set the momentum to 0.95 and the weight decay to 0.1, and rescale the RMS of each update matrix to 0.18 for reutilization of the AdamW learning rate. We train DeepSeek-V4-Flash on 32T tokens, and as in DeepSeek-V3, we also employ a batch size scheduling strategy that increases the batch size (in tokens) from a small size to 75.5M and then keeps it at 75.5M during most of the training. The learning rate is linearly warmed up in the first 2000 steps, maintained at \(2.7\times 10^{-4}\) for most of the training. Near the end of the training, we finally decay the learning rate to \(2.7\times 10^{-5}\) following a cosine schedule. The training starts with a sequence length of 4K, and we gradually extend the training sequence length to 16K, 64K, and 1M. As for the setups of sparse attention, we first warmup the model with dense attention for the first 1T tokens, and introduce sparse attention at the sequence length of 64K and keep sparse attention during the rest of the training. When introducing attention sparsity, we first set a short stage to warm up the lightning indexer in CSA, and then train the model with sparse attention for most of the training. For auxiliary-loss-free load balancing, we set the bias update speed to 0.001. For the balance loss, we set its loss weight to 0.0001 to avoid extreme imbalance within single sequences. The MTP loss weight is set to 0.3 for most of the training, and to 0.1 upon the start of learning rate decay.
—DeepSeek-AI, "DeepSeek-V4: towards highly efficient million-token context intelligence" (2026). arXiv 2606.19348, Section 4.2.2, the paragraph on the training of DeepSeek-V4-Flash, a model of 284 billion parameters.
Tokens per parameter. gpt-pp reads 22 tokens for each of its parameters, 27 for each parameter that costs arithmetic and 35 for each parameter of its blocks. The first figure is near the 20 that the study of Section 1.7 found best for the cost of training. Small models that are meant to be used are trained far past it.
Recent trends in language models pretraining show a clear shift towards significantly longer training durations, especially for smaller models (Yang et al., 2024a; Yang et al., 2024b; AI@Meta, 2024b). While this strategy deviates from the Chinchilla-optimal guidelines (Hoffmann et al., 2022), the resulting performance gains and reduced inference costs make extended training a worthwhile trade-off (de Vries, 2023). For example, Qwen2-1.5B was trained on 7 trillion tokens, Qwen2.5-1.5B on 18 trillion tokens, and Llama3.2-1B, derived from a pruned 8B model, was trained using distillation on 9 trillion tokens (Yang et al., 2024a; Yang et al., 2024b; AI@Meta, 2024b).
—L. Ben Allal and others, "SmolLM2: when smol goes big" (2025). arXiv 2502.02737, Section 4, its first paragraph. The SmolLM2 of 135 million parameters read two trillion tokens, about 15,000 for each parameter. N. Sardana, J. Portes, S. Doubov and J. Frankle, "Beyond Chinchilla-optimal: accounting for inference in language model scaling laws", ICML 2024 (arXiv 2401.00448), Section 4: "loss does not plateau as we scale to 10,000 tokens per parameter for our 150M model".
Remark 7.3.1 (optimal for which cost). Training a model of \(N\) parameters on \(D\) tokens costs about \(6ND\) operations, and using it costs about \(2N\) for each token read or written, so a model that will be used much is best trained at a smaller \(N\) and a larger \(D\) than the cost of training alone would choose. gpt-pp will answer few queries, and at the budget of three cards for a few hours a ratio near 20 is the classical choice; Section 7.5 asks our own runs whether it was a good one, and they do not settle it.
The first loss and the checkpoints show what this recipe produced.
Example 7.3.2 (the first loss is \(\ln V\)). The setup: the held-out loss before the first step, on 1,048,576 tokens. gpt-pp reads 10.397 nats per token, and \(\ln 32{,}768 = 10.397\). gpt-pico reads 8.318, and \(\ln 4{,}096 = 8.318\). A model whose writers start at zero is its embedding followed by its unembedding, and the unembedding starts a thousand times smaller than the embedding, so every logit is near zero and every token has probability \(1/V\) (Section 5.5). ◁
Example 7.3.3 (the run against its levels). The setup: the held-out loss of the main run, read every 50 million tokens on 1,024 fixed windows of 1,024 tokens, in bits per byte; and the 64 tokens that the trainer samples at each reading, from the beginning-of-text token alone, at temperature 0.8 and always with the same random draws.
At its first reading, after 50 million tokens, the model stands at 1.282 bits per byte, under the 1.577 of the 4-gram table: a model that has read 0.37 tokens for each of its parameters predicts better than counts of four tokens made on four times as many. The checkpoints saved before that reading show when the tables were passed: the token frequencies after 1.1 million tokens, 28 seconds into the run, and the 4-grams after 15 million, two minutes in. The learning rate is flat until 1.05 billion tokens and then falls in a straight line. The curve ends at 0.943 bits per byte, 3.025 nats per token, and it was still falling when the budget ran out, by 0.015 bits per byte over the last 500 million tokens.
When did the run pass the released GPT-2 of its shape? GPT-2 small scores 1.007 under the protocol of Section 7.4, document by document, and the trainer's readings first fall under it at 1.15 billion tokens. That sets one protocol against another. Like for like there are two answers. Read the trainer's way on the trainer's own held-out text, GPT-2 small scores 1.021, and the curve passes that after 861 million tokens, 56 minutes into the run. Scored document by document, the saved checkpoints pass 1.007 after about 943 million tokens, 61 minutes. Either way the text is of the kind that gpt-pp was trained on and GPT-2 was not, and Section 7.4 asks again on other text.
What the model writes changes in kind.
tokens the first words of the sample
0 laptop underneath pearandra femin juven merchand writer Mt
aggBPedarOk coron governedchelagan Natalges Don twelve translate
50M As the Sloan is a writer and writer you will get familiar with
RSS and PW. Rames have a wonderful time out of their own words.
100M In the past few decades we have been labeled as a "reason". But
in 1907 we learned that the desire to cut back on the "jaws" of
solving a problem remains largely the best way to improve the
outcomes of the children.
400M Assuratore, B. 1977. "Littoral Races". Potholes Rems. 12 (3):
775–767.
500M As the number of people who have been labeled as genetically
"non-malignant" has grown, the prevalence of invasive breast
cancer (AIMD) has become increasingly important.
1,100M In the past few decades, British writer James Joyce (1842–1911)
has been considered one of the most important writers of the
British literary canon. The novel is a compilation of poems by
the best-known and most productive English literary figures of
the time
3,000M As the climate changes, the British writer James Joyce (1866–1957)
was born Don Juan R. Losch. He was a leader in the literary world
of the twentieth century. He was best known for his novels Rudyard
Kipling's The R Milky WayAt zero tokens the draws fall on a uniform distribution. After 50 million the words are English and the sentences are not. After 100 million a sentence parses and says nothing. By 400 million the model knows what a reference list looks like, down to a volume, an issue and a page range, which runs backwards; by 500 million it knows the opening of a medical explainer, with an abbreviation that does not abbreviate its phrase. At 1.1 billion tokens, where the trainer reads 1.009 bits per byte, the sentence about Joyce is fluent, assured and wrong in each particular: Joyce was Irish and was born in 1882, and a novel is not a compilation of poems. At the end of the run the same draws make him British again, with new dates and another name at birth. The form of each kind of text arrives long before its content, and for this content three billion tokens were not enough. Several samples open with the same words because the random draws are the same; what changed is the distribution they are applied to. ◁
The order in which the loss passes the tables of counts, frequencies first, then pairs, then longer contexts, is the subject of Section 8.1.
Evaluation
A training curve is read on the trainer's monitor: fixed windows of the held-out stream, cut without regard to where documents begin, in the arithmetic of training. It serves to watch one run. As an instrument for comparing models it has two defects, and the author of a later public recipe named both when he declined to rank his models by validation loss.
Comparison to GPT-2/GPT-3 miniseries. I did not want to use the validation loss to compare models because while it is simple, it can be subtle and deceiving. For example, modded nanogpt (which I otherwise love) merged a few changes that I thought were mildly gaming the metric (e.g. using very long sequences and batch size 1). It's a bit subtle but stretching out your validation batches into one long row (i.e. batch size B=1) just means there are fewer tokens with cropped contexts at the first few columns of your (B, T) batches when B»1. This basically changes the validation loss by increasing the amount of context for many of these tokens, so it's not apples to apples, and the resulting "improvement" is not real. In addition, it's a bad form of comparison to GPT-2 and GPT-3 models because they were pretrained on a very different and unknown data mix distribution, so comparing FineWeb loss is not fair or informative. Only actual metrics are real and comparable. Earlier in the year I stumbled by the DCLM paper where they presented a nice ensemble metrics over a lot of different datasets. The metric is called CORE metric and it incorporates performance across 22 nice and high quality datasets. DCLM code had a complicated and bloated way of calculating it, so I stripped it all the way to a single, simple, dependency-free file that evaluates the CORE metric given a model. Then we can chart a nice, valid comparison of our miniseries v1 models to GPT-2 and GPT-3 (more on how I calculated/estimated their CORE scores below) where the x axis is resource spend (FLOPs, time) and y axis is CORE score. To get $ as the x-axis, simply take the time (hours) and multiply by $24 (as the cost of 8XH100 is $3/GPU/hour X 8 GPUs = $24/hour).
—A. Karpathy, notes on the first series of nanochat models (2026). "[Jan 7 2026] nanochat miniseries v1", the 420th discussion of that project on GitHub, the whole paragraph that compares its series with GPT-2 and GPT-3.
The two objections call for different answers: for the first, a protocol that cuts the text in the same way for every model and scores every token once; for the second, several held-out sets, with what each model may have read of each stated beside its score, and tasks beside losses.
Definition 7.4.1 (the scoring protocol). Fix a set of documents. Let a model have a tokenizer, a beginning-of-text token \(\mathtt b\) and a context of \(n_{\mathrm{ctx}}\) tokens, and let \(w = \min(n_{\mathrm{ctx}}, 1024)\). Each document is encoded by the model's tokenizer with nothing added, and its tokens \(x_1, \dots, x_n\) are cut into consecutive windows of \(w - 1\). The score of the document is
\[\ell(x) \;=\; \sum_{t=1}^{n} -\log_2 q\bigl(x_t \mid \mathtt b,\, x_{s(t)}, \dots, x_{t-1}\bigr) \quad \text{bits},\]where \(s(t)\) is the first token of the window that holds \(x_t\). The bits per byte of the model on the set are \(\sum_d \ell(x^{(d)}) / \sum_d B_d\), with \(B_d\) the number of UTF-8 bytes of document \(d\).
The window is 1,024 tokens because that is the context of GPT-2, the oldest model at the table; a model with a shorter context is scored in its own. The beginning-of-text token is the model's own, and it is given, not scored. No end-of-text token is appended, since models differ in whether they were trained to predict one, and the text is not normalized. Every model is scored in float32 with TensorFloat-32 products, which moves the fourth decimal: bfloat16, the arithmetic of our own training, is safe for the models trained in it and costs one of the released models 16%.
Proposition 7.4.2 (what the number is). (i) For every document, \(\ell(x) + 2\) bits suffice to write its tokens down for a reader who holds the model and knows the document's length. (ii) Let the tokens be a stretch of a stationary source with the conditional entropies \(F_k\) of Definition 1.2.1, and let the predictor be ideal inside each window. Then a token that stands \(k\)-th in its window costs \(F_k\) on average, and a long document costs
\[G_{w-1} \;=\; \frac{1}{w-1}\sum_{k=1}^{w-1} F_k \;\ge\; F_{w-1}\]per token, where a predictor that always saw the \(w - 2\) tokens before would pay \(F_{w-1}\).
Proof. (i) The conditionals of Definition 7.4.1 are a language model in the sense of Definition 1.1.2, one whose prediction ignores the tokens before \(s(t)\), and Theorem 1.3.2 applies to it. (ii) is Proposition 1.2.2. ∎
So bits per byte are the rate of a real code for the documents, and the gap in (ii) is the price of scoring every token exactly once with a bounded context. Every model pays it. A real model pays more than (ii) at the head of a window, because the beginning-of-text token announces a document where, in most windows of a long document, none begins. GPT-2 small pays 4.87 bits per token over whole windows of WikiText-103 and 4.62 on tokens with at least 511 tokens before them, 5% less; Example 1.6.2 has the whole curve for gpt-pp.
Proposition 7.4.3 (the error of a ratio, and of a difference). Let documents \(d = 1, \dots, m\) be drawn independently from a population, with \(\ell_d\) bits and \(B_d\) bytes, and let \(\hat R = \sum_d \ell_d / \sum_d B_d\) estimate \(R = \mathbb E\,\ell / \mathbb E\,B\). With \(\bar B\) the mean of the \(B_d\),
\[\hat R - R \;=\; \frac{1}{m \bar B}\sum_{d}\,(\ell_d - R\,B_d), \qquad \widehat{\mathrm{se}}(\hat R)^2 \;=\; \frac{m}{m-1}\,\frac{\sum_d (\ell_d - \hat R\,B_d)^2}{\bigl(\sum_d B_d\bigr)^2},\]the first an identity, the second the estimate of the variance of \(\hat R\) to first order in \(1/m\). For two models scored on the same documents, the second formula applied to the differences \(\ell_d - \ell^{\prime}_d\) gives the standard error of the difference of their bits per byte.
Proof. \(\hat R - R = (\bar\ell - R\bar B)/\bar B\), and the numerator is the mean of the \(m\) independent terms \(\ell_d - R B_d\), which have mean zero. As \(\bar B\) converges to \(\mathbb E\,B\), the variance of \(\hat R\) is that of one term divided by \(m\,(\mathbb E\,B)^2\) to first order; the estimate puts \(\hat R\) for \(R\), \(\bar B\) for \(\mathbb E\,B\) and the sample variance of the terms for their variance. ∎
The documents are the sampled units, not the tokens: the tokens of a document share its subject and its difficulty. The error says how far a figure would move with other documents of the same kind. It does not say how well two models can be told apart, because the same documents are hard for every model. GPT-2 medium scores 0.0823 ± 0.0008 bits per byte below GPT-2 small on FineWeb-EDU, where the standard errors of the two models alone are 0.0068 and 0.0065. Independent scores would give 0.0094 for their difference: the correlation between the two models' document residuals is 0.99.The linearization is that of the ratio estimator in W. G. Cochran, Sampling Techniques (1977). Scoring rules for forecasts that state probabilities, and what a score rewards, are the subject of Tuning the piano.
In the code. The protocol and its error, in the order of the file: the estimate of Proposition 7.4.3, then the scoring of Definition 7.4.1, which calls it. They are ratio_standard_error and bits_per_byte in gpt-pp's evaluate.py.
def ratio_standard_error(numerator: np.ndarray, denominator: np.ndarray) -> float | None:
...
n = len(numerator)
if n < 2:
return None
ratio = numerator.sum() / denominator.sum()
residual = numerator - ratio * denominator
return float(math.sqrt(n / (n - 1) * float((residual ** 2).sum())) / denominator.sum())
def bits_per_byte(model: LanguageModel, docs: Sequence[str], batch_tokens: int | None = None,
progress: str | None = None) -> dict:
...
doc_tokens = model.encode(docs)
step = model.window - 1 # tokens of the document in a window; the first place holds the beginning-of-text token
windows: list[list[int]] = []
owner: list[int] = []
for d, ids in enumerate(doc_tokens):
for i in range(0, len(ids), step):
windows.append([model.bos_id] + ids[i:i + step])
owner.append(d)
scores = score_sequences(model, windows, starts=1, batch_tokens=batch_tokens, progress=progress)
n_docs = len(docs)
doc_bits = np.bincount(owner, weights=scores.nll, minlength=n_docs) / LN2
doc_bytes = np.array([len(t.encode("utf-8")) for t in docs], dtype=np.float64)
n_tokens = int(scores.n_scored.sum())
n_bytes = int(doc_bytes.sum())
assert n_tokens == sum(len(ids) for ids in doc_tokens), "every token of every document is scored exactly once"
total_bits = float(doc_bits.sum())
loss = total_bits * LN2 / n_tokens
...
return {
"bits_per_byte": total_bits / n_bytes,
"bits_per_byte_se": ratio_standard_error(doc_bits, doc_bytes),
...
}
step is the \(w - 1\) of Definition 7.4.1, and each row of windows is \(\mathtt b\) followed by at most \(w - 1\) tokens of one document; starts=1 tells the scorer that the first place is given and not scored. doc_bits and doc_bytes are the \(\ell_d\) and \(B_d\) of Proposition 7.4.3, one entry a document, and ratio_standard_error is its second formula, with n for \(m\) and residual for the terms \(\ell_d - \hat R\,B_d\). The assertion is the claim that every token of every document is scored exactly once.
Quirk. The windows do not overlap: range(0, len(ids), step) starts each where the last one ended, so the first tokens of every window are predicted from little context, the gap of Proposition 7.4.2. And every model, gpt-pp included, is scored in float32 with TensorFloat-32 products, the default precision="tf32", not in the bfloat16 that gpt-pp trained in.
Quirk. A released model enters through its own tokenizer, its own beginning-of-text token and its own context up to 1,024. Qwen3 defines no such token, and its end-of-text token is used in its place; TinyStories 33M is scored in windows of 512 tokens, the length it was trained at, not the 2,048 of its configuration.
Known issue. The trainer's monitor is a different instrument. evaluate in train.py averages the loss over the first 1,048,576 tokens of the held-out stream, in windows cut without regard to documents, under bfloat16, and its targets include the beginning-of-text token wherever a document begins inside a window; bits_per_byte scores whole documents from their first byte and never scores that token. For gpt-pp the two read 0.9429 and 0.9322 bits per byte.
Known issue. The monitor also turns its nats into bits per byte with a constant, val_loss / math.log(2) / bytes_per_token, the bytes per token of the whole held-out file, where bits_per_byte divides by the bytes of the documents it scored. The note at the head of this part gives the size of the difference. One constant serves every run, so comparisons between runs of the trainer are not affected.
| held-out set | what it is | documents scored | bytes | who may have read it |
|---|---|---|---|---|
| WikiText-103 | the test articles of the benchmark, text as distributed | 60 | 1,287,656 | not GPT-2, whose corpus had Wikipedia removed; possible for the later models trained on the web |
| TinyStories | validation stories of the dataset's second version | 2,000 | 1,615,901 | held out of gpt-pico's training set; 695 of them stand word for word in the first version, on which TinyStories 33M was trained |
| FineWeb-EDU | documents held out of our training set (Example 7.1.2) | 1,000 | 4,579,235 | the training distribution of gpt-pp; training data of SmolLM2; possible for every model trained on the web |
| fresh essays | plain prose of eight essays of this site, October and November 2026 | 8 | 430,307 | no model here |
The two recipes at the same shape. The first comparison holds everything fixed but the recipe: shape, tokenizer, tokens, batch, card and held-out text. We compare held-out loss against both tokens and elapsed time on one card, with each recipe at the best of its runs.
loss held-out bits per byte against tokens; 12 layers, 768 wide, same batch, one card each
tokens 2019 recipe, rate 0.0024 2026 recipe
20.0M 1.781 1.417
40.0M 1.521 1.293
60.0M 1.368 1.222
80.0M 1.284 1.173
100M 1.244 1.139
needed to reach the 2019 recipe the 2026 recipe fewer tokens less time
1.781 20.0M, 3.4 min 5.2M, 57 s 3.88 times 3.57 times
1.368 60.0M, 10 min 25.9M, 4.7 min 2.32 times 2.14 times
1.244 100.0M, 17 min 53.8M, 9.8 min 1.86 times 1.71 times
2019 warmup 32M, rate, final loss: 0.0006 1.295 · 0.0012 1.257 · 0.0024 1.244 · 0.0048 1.256
warmup 8M, rate, final loss: 0.0012 1.269 · 0.0024 1.274
(the live figure needs JavaScript and loads about 39 kB of data)
Example 7.4.4 (two recipes, two sizes). The setup, at gpt-pico's shape: 540 million tokens of TinyStories, one pass, one seed each, 65,536 tokens a step, the 2019 recipe at a learning rate chosen by a sweep of short runs, ten times that of the larger size. The 2019 recipe ends at 0.5849 bits per byte on the held-out stories and the 2026 recipe at 0.5515, lower by 0.093 nats per token. The 2026 recipe is under the other's final loss from its reading at 271 million tokens on, half of the budget, with its learning rate still at 78% of the peak.
The setup, at the shape of gpt-pp: 100 million tokens of FineWeb-EDU on one card, 2,035 steps of 49,152 tokens, one seed each. The 2019 recipe is run at \(6 \times 10^{-4}\), the rate at which GPT-2 small is usually rebuilt, and at two, four and eight times that, with its warmup of 32 million tokens, and at two and four times again with a warmup of 8 million. The 2026 recipe has the rates of the main run, and is run again with Muon's rate halved and doubled.
| recipe, 12 layers and 768 wide | learning rate | warmup, tokens | held-out loss, nats per token | bits per byte |
|---|---|---|---|---|
| 2019 | \(6 \times 10^{-4}\) | 32M | 4.155 | 1.295 |
| 2019 | \(1.2 \times 10^{-3}\) | 32M | 4.032 | 1.257 |
| 2019 | \(2.4 \times 10^{-3}\) | 32M | 3.991 | 1.244 |
| 2019 | \(4.8 \times 10^{-3}\) | 32M | 4.030 | 1.256 |
| 2019 | \(1.2 \times 10^{-3}\) | 8M | 4.073 | 1.269 |
| 2019 | \(2.4 \times 10^{-3}\) | 8M | 4.088 | 1.274 |
| 2026 | 0.01, 0.3, 0.008 | 2M | 3.653 | 1.139 |
| 2026, the recipe of gpt-pp | 0.02, 0.3, 0.008 | 2M | 3.654 | 1.139 |
| 2026 | 0.04, 0.3, 0.008 | 2M | 3.679 | 1.147 |
The 2026 recipe ends at 3.654 nats per token and the 2019 recipe, at the best of its six runs, at 3.991: a gap of 0.337 nats, or 0.105 bits per byte, where a difference under about 0.02 nats between two single runs is not established (Section 7.9). The 2026 recipe had passed the other's final loss after 54 million tokens, 1.9 times fewer, in 1.7 times less time on one card. The rate of the 2019 recipe is bracketed: four times the usual rate is its best, and eight times is worse. The shorter warmup did not help it: at equal rates its runs with a warmup of 8 million tokens end 0.041 and 0.097 nats above those with 32 million, and no longer warmup was tried.
Three things bear on the reading. The 2019 recipe had six tries and the 2026 recipe three: half of Muon's rate is level and twice is worse, and the rates of its two vocabulary matrices are as chosen on smaller models. The 2026 model also has 24 million more parameters, its second vocabulary matrix. A hundred million tokens are 0.74 tokens for each parameter and 3.3% of the main run: the race measures the start of training, and at gpt-pico's shape, with 242 tokens for each parameter, the 2026 recipe needed half the tokens of the other. And the race sets two whole recipes against each other. On the 8-layer model of Section 6.8 the step from the training recipe of 2019 to that of 2026, which changes fourteen settings at once, is worth 0.157 nats, 0.140 if the 2019 recipe is given a warmup of 32 million tokens; of it, Muon in place of AdamW on the hidden matrices is 0.083 and 0.086 in two seeds, and the rest of that recipe 0.06 to 0.07. ◁
Bits per byte. Every model is scored under the same protocol on four held-out sets. Beside each score are its uncertainty and what is known about whether the model trained on that source; the budgets and benchmarks give further context.
chart bits per byte on 1,000 held-out FineWeb-EDU documents (lower is better): the models
scored here, best first, with parameters and training tokens where stated;
* a model on its home ground; > gpt-pp
SmolLM2 360M* 362M 4,000B 0.788 ·o
Qwen3 0.6B base 596M - 0.817 ··o
GPT-2 XL 1,558M - 0.854 ····o
SmolLM2 135M* 135M 2,000B 0.857 ·····o
GPT-2 large 774M - 0.881 ······o
GPT-2 medium 355M - 0.925 ·········o
> gpt-pp* 135M 3.0B 0.932 ··········o
Pythia 160M 162M 300B 0.998 ··············o
GPT-2 small 124M - 1.007 ···············o
DistilGPT2 82M - 1.084 ····················o
2026 recipe, 100M tokens* 135M 0.1B 1.129 ·······················o
2019 recipe, 100M tokens* 111M 0.1B 1.233 ······························o
TinyStories 33M 69M - 2.875 (off the scale)
On FineWeb-EDU gpt-pp scores 0.932 bits per byte, seventh of the 13 models scored
here; against GPT-2 small, the released model of its shape, it is lower by 0.075 ±
0.003 on the same documents. Counting differences beyond two standard errors, it is
ahead of GPT-2 small on one of the four held-out sets and behind on three, ahead on
four of the nine benchmarks and behind on two.
(the live figure needs JavaScript; it holds four held-out sets, nine benchmarks and 19 models)
| model | year | parameters | training tokens | WikiText-103 | TinyStories | FineWeb-EDU | fresh essays |
|---|---|---|---|---|---|---|---|
| gpt-pp | 2026 | 135M | 3.0B | 1.130 ± 0.017 | 0.864 ± 0.002 | 0.932 ± 0.007 | 1.325 ± 0.035 |
| same shape, 2026 recipe | 2026 | 135M | 0.1B | 1.383 ± 0.018 | 1.103 ± 0.002 | 1.129 ± 0.008 | 1.468 ± 0.025 |
| same shape, 2019 recipe | 2026 | 111M | 0.1B | 1.500 ± 0.019 | 1.232 ± 0.002 | 1.233 ± 0.008 | 1.545 ± 0.021 |
| GPT-2 small | 2019 | 124M | not disclosed | 1.072 ± 0.010 | 0.856 ± 0.002 | 1.007 ± 0.007 | 1.297 ± 0.030 |
| GPT-2 medium | 2019 | 355M | not disclosed | 0.975 ± 0.011 | 0.759 ± 0.002 | 0.925 ± 0.007 | 1.229 ± 0.034 |
| GPT-2 large | 2019 | 774M | not disclosed | 0.928 ± 0.010 | 0.707 ± 0.002 | 0.881 ± 0.006 | 1.188 ± 0.036 |
| GPT-2 XL | 2019 | 1,558M | not disclosed | 0.898 ± 0.010 | 0.679 ± 0.002 | 0.854 ± 0.006 | 1.163 ± 0.037 |
| DistilGPT2 | 2019 | 82M | distilled from GPT-2 small | 1.201 ± 0.011 | 0.964 ± 0.002 | 1.084 ± 0.007 | 1.393 ± 0.028 |
| Pythia 160M | 2023 | 162M | about 300B | 1.094 ± 0.010 | 0.882 ± 0.002 | 0.998 ± 0.007 | 1.309 ± 0.034 |
| SmolLM2 135M | 2025 | 135M | 2,000B | 0.972 ± 0.012 | 0.692 ± 0.002 | 0.857 ± 0.006 | 1.207 ± 0.039 |
| SmolLM2 360M | 2025 | 362M | 4,000B | 0.869 ± 0.010 | 0.618 ± 0.002 | 0.788 ± 0.006 | 1.132 ± 0.040 |
| Qwen3 0.6B base | 2025 | 596M | not stated for this size | 0.888 ± 0.009 | 0.638 ± 0.002 | 0.817 ± 0.007 | 1.155 ± 0.039 |
| TinyStories 33M | 2023 | 69M | stories only | 3.380 ± 0.032 | 0.436 ± 0.002 | 2.875 ± 0.025 | 2.716 ± 0.052 |
Two of the released models have the trunk of gpt-pp, 12 layers of width 768: GPT-2 small of 2019 and Pythia 160M of 2023. The lowest figure among the released models on three of the four sets is that of SmolLM2 360M; on stories it is that of the model trained on stories, which pays 2.7 to 3.4 bits per byte everywhere else.
On held-out FineWeb-EDU gpt-pp scores 0.932 bits per byte, behind six of the ten released models and ahead of four. On the same documents it is ahead of GPT-2 small by 0.075 ± 0.003 and of Pythia 160M by 0.066 ± 0.002, and it is 0.007 ± 0.003 behind GPT-2 medium, a model of 2.6 times its parameters. FineWeb-EDU is its home ground, and it is the only ground on which it leads GPT-2 small. On WikiText-103 it is behind by 0.058 ± 0.018, on the stories by 0.008 ± 0.001, and on the fresh essays, the one set that no model can have read, by 0.028 ± 0.009, behind on each of the eight. Against SmolLM2 135M, the released model of its own size, it is behind on every set, by 0.075 on FineWeb-EDU and by 0.12 to 0.17 on the others. Three billion tokens of one kind of text, under the recipe of 2026, make a model that reads that kind of text nearly as well as a 2019 model of more than twice its size, and reads the other three sets a little worse than the 2019 model of its own size.
The budgets are the context. gpt-pp read 3.0 billion tokens in 3.2 hours on three cards. What GPT-2 read was never disclosed; Pythia 160M read about 300 billion tokens and SmolLM2 135M 2,000 billion, FineWeb-EDU among them. The budget explains the row and does not change it. Part of the deficit off the home ground can be located. On the stories gpt-pp and GPT-2 small pay the same for a token, 3.474 and 3.473 bits, and gpt-pp's tokenizer, fitted to FineWeb-EDU with a vocabulary two thirds the size, writes 0.9% more tokens. On WikiText-103, whose text is distributed with its punctuation set apart by spaces, that tokenizer writes 6.1% more tokens than GPT-2's; gpt-pp pays a little less for each, 4.84 bits against 4.87, and more for the whole. A tokenizer is part of a model, and it too is made for a source.
The two rows under gpt-pp are the models of the race, the 2019 recipe at its best rate. After 100 million tokens the 2026 recipe is ahead of it on every document of every set, by 0.104 bits per byte on FineWeb-EDU and by 0.08 to 0.13 on the others. Thirty times the tokens then take the 2026 recipe from 1.129 to 0.932.
Known issue. What was not scored. Gemma 3 270M is gated, and it is absent from the table. The fresh essays are eight essays by one author. TinyStories 33M, trained on the first version of its dataset, scores 0.442 on the 695 held-out stories that also stand in that version and 0.431 on the others. And the run laid out for 1.08 billion tokens, which is set beside another project's figure below, was never scored under this protocol: its 0.9742 is the trainer's number.
Benchmarks. A model that only predicts the next token answers a question by scoring each candidate answer as a continuation.
Definition 7.4.5 (answering by likelihood). An item is a given text \(c\) and options \(o_1, \dots, o_K\), one of them right. The model scores option \(k\) by \(s_k = \log q(o_k \mid \mathtt b, c)\), the sum of the log-probabilities of the option's tokens after the given text. Three rules choose an option: the largest \(s_k\), the largest \(s_k\) per byte of the option, the largest per token. Options that tie share the credit, so that a model whose scores do not depend on \(k\) scores exactly the chance level \(1/K\). The centered mean of a model is the mean over tasks of \((a - 1/K)/(1 - 1/K)\), with \(a\) its accuracy on the task.
The rescaling is that of a benchmark of 2024 for data curation: "linearly rescaling the accuracy per task so that 0 corresponds to random guessing and 1 corresponds to perfect accuracy".Li and others (2024), cited in Section 7.1, on its Core score over 22 tasks. Our nine: HellaSwag (Zellers et al., ACL 2019, arXiv 1905.07830), BLiMP (Warstadt et al., TACL 2020, arXiv 1912.00582), ARC-Easy and ARC-Challenge (Clark et al., 2018, arXiv 1803.05457), LAMBADA (Paperno et al., ACL 2016, arXiv 1606.06031), PIQA (Bisk et al., AAAI 2020, arXiv 1911.11641), WinoGrande (Sakaguchi et al., 2019, arXiv 1907.10641), SciQ (Welbl et al., 2017, arXiv 1707.06209) and CommonsenseQA (Talmor et al., NAACL 2019, arXiv 1811.00937). On LAMBADA an item is right when every token of the passage's last word is the model's first choice, and chance is zero. The rule is part of the result. The total prefers short options, since every token costs something, so the headline is the score per byte where the options are free text of unequal length, and the total where the candidates have the same length, as in the pairs of sentences of BLiMP. For GPT-2 small, SciQ is 77.6% by the total and 68.2% per byte, and ARC-Challenge is 19.2%, below chance, and 22.2%. The standard error of an accuracy \(a\) on \(n\) items is \(\sqrt{a(1-a)/n}\).
The code was checked against published figures: for GPT-2 small it gives a perplexity of the last word of LAMBADA of 35.11, where the GPT-2 paper has 35.13.
Known issue. The check holds at that size and not at every size. For GPT-2 small the code also reproduces the published accuracy on HellaSwag within 0.55 points. For GPT-2 XL it gives a perplexity of 20.24 on WikiText-103, on de-tokenized text, where 17.48 is published, with the three smaller sizes within 2.2%; on LAMBADA it reads 10% to 17% above the published figures of the three larger sizes. The cause of the difference for GPT-2 XL is not known. Unlike the common harness, run_task begins every sequence with the beginning-of-text token, and a context too long for the window loses its beginning.
| model | HellaSwag | BLiMP | ARC-Easy | ARC-Challenge | LAMBADA | PIQA | WinoGrande | SciQ | CommonsenseQA | centered mean |
|---|---|---|---|---|---|---|---|---|---|---|
| chance | 25.0 | 50.0 | 25.0 | 25.0 | 0.0 | 50.0 | 50.0 | 25.0 | 20.0 | 0.0 |
| gpt-pp | 33.1 | 79.6 | 47.7 | 26.2 | 26.8 | 63.1 | 50.9 | 71.8 | 30.4 | 25.8 |
| same shape, 2026 recipe, 100M tokens | 27.1 | 72.6 | 40.1 | 21.5 | 12.2 | 57.2 | 51.4 | 60.1 | 27.3 | 16.5 |
| same shape, 2019 recipe, 100M tokens | 26.8 | 69.5 | 35.6 | 21.5 | 9.5 | 54.8 | 50.5 | 53.0 | 25.4 | 12.8 |
| GPT-2 small | 31.3 | 82.5 | 40.5 | 22.2 | 33.7 | 62.6 | 50.5 | 68.2 | 30.1 | 24.5 |
| GPT-2 medium | 39.9 | 82.6 | 43.7 | 25.4 | 43.9 | 65.9 | 53.0 | 72.3 | 33.5 | 30.2 |
| GPT-2 large | 45.6 | 83.5 | 46.8 | 25.5 | 48.5 | 69.4 | 54.5 | 73.6 | 34.8 | 33.8 |
| GPT-2 XL | 51.4 | 83.3 | 50.2 | 27.1 | 51.4 | 70.3 | 57.6 | 79.4 | 38.0 | 37.8 |
| DistilGPT2 | 27.6 | 79.5 | 38.4 | 23.1 | 25.1 | 59.7 | 52.1 | 65.6 | 29.7 | 21.4 |
| Pythia 160M | 30.7 | 78.5 | 40.3 | 24.5 | 35.7 | 62.0 | 52.3 | 71.5 | 31.5 | 25.0 |
| SmolLM2 135M | 43.7 | 80.4 | 58.2 | 29.0 | 43.6 | 68.7 | 53.4 | 81.5 | 35.6 | 35.3 |
| SmolLM2 360M | 56.5 | 81.4 | 67.3 | 36.9 | 54.5 | 72.5 | 59.6 | 87.6 | 43.1 | 45.4 |
| Qwen3 0.6B base | 53.4 | 80.5 | 67.5 | 40.0 | 55.0 | 70.6 | 58.2 | 92.0 | 47.5 | 45.8 |
| TinyStories 33M | 27.5 | 63.6 | 27.6 | 23.0 | 11.4 | 54.0 | 52.1 | 24.9 | 27.4 | 7.1 |
One column of the table does not sort the released models by size: BLiMP, a test of grammar, does not separate the four sizes of GPT-2, 82.5 to 83.5, and the models of 2025 stand under them, 80.4 to 81.4. Grammar is learned early and does not grow with the rest.
The centered mean of gpt-pp is 25.8, where GPT-2 small has 24.5, Pythia 160M 25.0 and SmolLM2 135M 35.3. Item by item against GPT-2 small, and counting only differences beyond two paired standard errors, it is ahead on four tasks, HellaSwag (by 1.7 ± 0.3 points), ARC-Easy (7.2 ± 0.9), ARC-Challenge (4.0 ± 1.1, with GPT-2 small below chance) and SciQ (3.5 ± 1.2), behind on two, BLiMP (3.0 ± 0.2) and LAMBADA (7.0 ± 0.6), and level on PIQA, WinoGrande and CommonsenseQA. The gains are in school science, which is what an educational corpus is about. The losses are in grammar and in LAMBADA, which asks for the last word of a passage of narrative, a kind of text that an educational filter does not favor: there the last word has a perplexity of 73 under gpt-pp and of 35 under GPT-2 small. Against SmolLM2 135M it is behind on eight of the nine tasks, by 5 to 17 points on six of them, and level on WinoGrande.
After 100 million tokens the two models of the race have centered means of 16.5 and 12.8, and the 2026 recipe is ahead by more than two paired standard errors on five tasks, BLiMP, ARC-Easy, LAMBADA, PIQA and SciQ, and level on the other four.
Example 7.4.6 (hosted models on the same items). The setup: a hosted chat model returns no log-probabilities, so it is asked in words. The items are a fixed subset of three tasks, 200 of HellaSwag, 200 of ARC-Easy and 300 pairs of BLiMP shown as a forced choice between two sentences. The prompt is the item and "Answer with the letter only.", with no system prompt, the least hidden reasoning that each model allows, and each item asked once; a reply with no readable letter counts as wrong. The six models were chosen on 14 November 2026 from a public list by its own descriptions. gpt-pp is scored on the same items by likelihood.
| model | how it answers | HellaSwag | ARC-Easy | BLiMP |
|---|---|---|---|---|
| GPT-6 Luna, OpenAI's small model | in words | 90.5 ± 2.1 | 98.5 ± 0.9 | 88.7 ± 1.8 |
| GPT-6.1 Terra, its mid-tier model | in words | 94.5 ± 1.6 | 99.0 ± 0.7 | 95.0 ± 1.3 |
| GPT-6 Astra, its flagship | in words | 97.5 ± 1.1 | 99.5 ± 0.5 | 96.7 ± 1.0 |
| Claude Fable 5.2 | in words | 98.4 ± 0.9 | 99.5 ± 0.5 | 98.3 ± 0.8 |
| Qwen3.8 27B, open weights | in words | 96.0 ± 1.4 | 99.0 ± 0.7 | 87.7 ± 1.9 |
| LFM3 1.7B, open weights, incomplete | in words | 63.9 ± 8.0 | 100.0 ± 0.0 | 68.8 ± 6.7 |
| gpt-pp | by likelihood | 30.0 ± 3.2 | 51.0 ± 3.5 | 77.3 ± 2.4 |
The distance depends on the task. On the endings of HellaSwag and the science of ARC-Easy the hosted models are near the ceiling and gpt-pp is far below. On grammar, at 77%, it is within reach of two of them, at 88%. What a few billion tokens buy is the form of English; what they do not buy is the world that the endings and the questions are about. The three benchmarks were published between 2018 and 2020, so any model trained since may have read their items. ◁
What two public projects report. The two projects that this series started from publish results for their own models, and the table sets them beside gpt-pp without converting anything.
| model, as its project reports it | shape | parameters | training text | cards and time | result reported |
|---|---|---|---|---|---|
| nanoGPT, 2023: GPT-2 small rebuilt | 12 × 768 | 124M | OpenWebText, about 300B tokens | 8 A100, about 4 days | 2.85 nats per token on its validation text, where released GPT-2 small scores 3.12 |
| its successor of 2024 | 12 × 768 | 124M | FineWeb-EDU, 10B tokens | about an hour, about 10 dollars | HellaSwag above released GPT-2 small (no figure given); 33.24% after 40B tokens |
| nanochat, first release, October 2025 | 20 × 1,280 | 561M | FineWeb-EDU, 11.2B tokens | 8 H100, about 3 hours | CORE 0.2219; about 0.81 bits per byte |
| the same, 12-layer model of its series of January 2026 | 12 × 768 | 135M | FineWeb-EDU, 1.08B tokens | 8 H100, 7.8 minutes | CORE 0.1059; 0.9825 bits per byte |
| the same, record of 14 March 2026 | 24 × 1,536 | 1,384M | a searched mixture, about 5.8B tokens | 8 H100, 1.65 hours | CORE 0.2626, where released GPT-2 XL scores 0.2565; 0.718 bits per byte |
| gpt-pp | 12 × 768 | 135M | FineWeb-EDU, 3.0B tokens | 3 cards of 32 GB, 3.20 hours | 0.932 bits per byte on held-out FineWeb-EDU; centered mean 25.8 |
The row nearest to gpt-pp is the 12-layer model of the second project: the same depth, width and vocabulary size, the same corpus, a third of the tokens. At 1.08 billion tokens the main run, with its learning rate at 99% of the peak in a schedule laid out for three billion, reads 1.01 bits per byte on the trainer's held-out windows. The same recipe with its schedule laid out for 1.08 billion tokens ends at 0.9742 bits per byte on those windows, where that model reports 0.9825 on its own validation text, under another tokenizer and with a context of 2,048 tokens: the two figures are not one measurement, and the 0.008 between them is less than the 0.011 by which our own two ways of scoring one model differ (0.943 by the trainer's windows, 0.932 by document). At equal tokens the two recipes are level as far as this can say, and neither is shown ahead; that recipe has since gained parts that gpt-pp lacks, among them learned scalars on the stream at every layer, tables of embeddings added to the values of attention, and optimizer settings from a sweep of some 320 runs.
What does cross from one protocol to another is the released GPT-2, which every project scores with its own instrument. The successor of the first project puts GPT-2 small at 3.2924 nats per token on its slice of FineWeb-EDU, where our 1,000 documents give 3.238, and at 28.92% and 31.14% on HellaSwag by the two rules of the common harness, where we measure 29.03% and 31.35%. What each protocol can state is a distance from the released GPT-2 of the same size on a named text. Ours states it on four texts, and on three of the four the distance is against gpt-pp.nanoGPT and its successor, build-nanogpt (GitHub, 2022 to 2024), and nanochat (GitHub, 2025 to 2026): the files that introduce each project, the first, the 420th and the 481st discussion of the last, and its experiment log, whose entries of 11, 17 and 19 to 22 January 2026 record the three additions. GPT-2's own paper says of its training only that the learning rate "was manually tuned for the best perplexity on a 5% held-out sample of WebText" and that "All models still underfit WebText". The cost of the largest GPT-2 that these projects cite, 32 chips for a week, is second-hand.
Remark 7.4.7 (a fair reading). A benchmark figure is a model, a protocol and a set of documents, and it changes when any one of them does. The comparison of the two recipes holds the most fixed and says the least about use: it ranks two ways of training on the first hundred million tokens. The comparison with released models on FineWeb-EDU is the least fair, since a code is made for its source (Remark 1.3.4) and this source is gpt-pp's own; the other three sets say what that lead is worth elsewhere, and against the 2019 model of the same size it is worth nothing. The fresh essays are fair to every model and too few to rank neighbors without pairing. None of the tables is a ranking of models.
Scaling laws on our own grid
Section 1.7 stated the law \(L(N, D) = E + A N^{-\alpha} + B D^{-\beta}\) and its frontier (Proposition 1.7.3), and ended on a warning: an exponent belongs to a protocol (Remark 1.7.6). Here the law is fitted to runs of our own, under three ways of counting parameters, and tested against the much larger budget of the main run.
box held-out loss, nats per token, over parameters and training tokens: the grid's runs
as points, the fitted law L = E + A / N^alpha + B / D^beta as a surface through them
grid layers x width 25M tokens 50M tokens 100M tokens 200M tokens
4 x 128 4.864 4.584 4.375 4.256
6 x 192 4.686 4.342 4.144 4.017
8 x 256 4.608 4.237 4.013 3.868
8 x 384 4.488 4.112 3.885 3.726
8 x 512 4.415 4.045 3.813 3.647
12 x 768 4.329 3.936 3.687 -
fit N counts alpha beta E a main run
the blocks 0.173 0.734 2.80 0.81 3.378
the blocks and the output layer 0.524 0.707 3.19 0.57 3.380
every parameter 0.622 0.705 3.20 0.53 3.381
a = beta / (alpha + beta): the best size for a compute C grows as C^a. Last column: the
law at the main run, measured 3.086 in windows of 512 tokens
(the live figure needs JavaScript and WebGL; it loads about 0.7 MB of three.js)
Example 7.5.1 (the grid). The setup: six models in the 2026 recipe, each trained on 25, 50, 100 and 200 million tokens of FineWeb-EDU at a context of 512 tokens, with 65,536 tokens a step and one seed; the largest model was not run at the largest budget, which leaves 23 runs. Each is a run of its own, with its own schedule. No rate was tuned for a size: Muon's rate is 0.02 at every size, and the rates of the two vocabulary matrices follow the recipe's rule, 0.3 and 0.008 times \(\sqrt{768/d}\). The grid was first run with the rates of the 512-wide model at every width, and the 19 runs at the other widths were then made again: at width 128 the rule is worth 0.049 nats after 25 million tokens and 0.008 after 200 million, and nothing beyond the noise at widths 384 and 768.
| layers × width | in the blocks | with the output layer | all | share of the output layer |
|---|---|---|---|---|
| 4 × 128 | 786,432 | 4,980,736 | 9,175,040 | 84% |
| 6 × 192 | 2,654,208 | 8,945,664 | 15,237,120 | 70% |
| 8 × 256 | 6,291,456 | 14,680,064 | 23,068,672 | 57% |
| 8 × 384 | 14,155,776 | 26,738,688 | 39,321,600 | 47% |
| 8 × 512 | 25,165,824 | 41,943,040 | 58,720,256 | 40% |
| 12 × 768 | 84,934,656 | 110,100,480 | 135,266,304 | 23% |
The grid spans a factor of 108 in the first count, of 22 in the second and of 15 in the third, where the two vocabulary matrices are 91% of the smallest model and 37% of the largest, and a factor of eight in tokens. ◁
In the code. The rule of the example is applied where a run's two configurations are built, and the order of the two matters. It is build_configs in gpt-pp's train.py.
def build_configs(args: argparse.Namespace) -> tuple[ModelConfig, TrainConfig]:
...
flags = dict(data=args.data, tokens=args.tokens, seed=args.seed, eval_tokens=args.eval_tokens)
def configs(width: Optional[int] = None) -> tuple[ModelConfig, TrainConfig]:
train_cfg = train_preset(args.train_preset or args.preset, d_model=width)
train_cfg = train_cfg.replace(**{k: v for k, v in flags.items() if v is not None})
if args.no_compile:
train_cfg = train_cfg.replace(compile=False)
model_cfg, train_cfg = apply_overrides([preset(args.preset), train_cfg], args.overrides)
return model_cfg, train_cfg
model_cfg, _ = configs() # the model does not depend on the training recipe: built first, for its width
return configs(model_cfg.d_model)
configs is called twice: once for the model alone, whose width \(d\) may come from --set d_model=128 on the command line, and once more at that width, so that train_preset computes the rates of the two vocabulary matrices, 0.3 and 0.008 times \(\sqrt{768/d}\), from the width that will be trained. Muon's rate has no such rule (Section 6.8), and a rate given with --set wins, like every override.
Bug, fixed. Before the fix train_preset went by the preset's own width, so a width set on the command line kept the preset's rates, and every size of the first grid trained at the rates of width 512, 0.3674 and 0.0098. The 19 runs at the other widths were made again, and every figure of this section is from the second grid; on the first, \(\alpha\) under the second count was 0.56, where Example 7.5.3 has 0.52. The runs of the second grid record a commit older than the fix, because a run stores the commit it was launched at and nothing of uncommitted changes: their recorded rates are the evidence of what ran (Section 11).
Proposition 7.5.2 (one loss, two exponents). Let a family of models be indexed by a count \(N\), let \(h(N)\) be a second group of parameters with elasticity \(\varepsilon = d\ln h/d\ln N\), and put \(N^{\prime} = N + h\) and \(s = h/N^{\prime}\). If the reducible loss is a power law \(A N^{-\alpha}\) in \(N\), its local exponent in \(N^{\prime}\) is
\[\alpha^{\prime} \;=\; -\frac{d\ln(L - E)}{d\ln N^{\prime}} \;=\; \frac{\alpha}{1 - (1 - \varepsilon)\,s} .\]Proof. \(dN^{\prime}/d\ln N = N + \varepsilon h\), so \(d\ln N^{\prime}/d\ln N = 1 - (1 - \varepsilon)s\). ∎
In a transformer whose depth grows with its width the blocks hold \(12Ld^2 \propto d^3\) parameters and the unembedding \(Vd\), so \(\varepsilon = 1/3\); at a fixed depth \(\varepsilon = 1/2\). A law that is a straight line of slope \(\alpha\) against the count of the blocks is therefore, against the count with the output layer, a curve whose slope runs from \(2.1\alpha\) between the two smallest sizes of the grid to \(1.3\alpha\) between the two largest. At most one of the two counts gives a power law, and the frontier exponent \(a = \beta/(\alpha + \beta)\) of Proposition 1.7.3 is smaller under the count that makes \(\alpha\) larger. It is one mechanism behind the first step in the table of Section 1.7, where counting the output layer took the exponent from 0.86 to 0.70, and the reason for fitting under more than one count.
The fit. The law is fitted as the study of 2022 fitted it and as its audit corrected it, by minimizing over the runs \(i\) the sum of a robust loss on the residuals of \(\ln L\),
\[\min_{a,\,b,\,c,\,\alpha,\,\beta}\; \sum_i \mathrm{Huber}_{\delta}\Bigl(\ln\bigl(e^{a - \alpha\ln N_i} + e^{b - \beta\ln D_i} + e^{c}\bigr) - \ln L_i\Bigr), \qquad (A, B, E) = (e^{a}, e^{b}, e^{c}),\]from many starting points, with \(\delta = 10^{-3}\); the Huber loss is quadratic for small residuals and linear for large ones. A sum and not a mean, and predictions made from the constants before they are rounded, are the two corrections of Section 1.7. The floor \(E\) is fitted, not assumed: "It is crucial to include the irreducible loss, without which one can easily underestimate scaling exponents by over \(2\times\) in practice (see Fig. 9)."M. Barkeshli, A. Alfarano and A. Gromov, "On the origin of neural scaling laws: from random graphs to natural language", 2026 (arXiv 2601.10684), Section 2.1. The form of the objective is that of Hoffmann et al. (2022), Appendix D.2, with the corrections of Besiroglu et al. (2024). Intervals come from fitting again on resampled sets of runs: they say how much the constants depend on which runs are in the grid, not what a second seed would change.
Example 7.5.3 (the fit, and what it says of the main run). The setup: the 23 runs of Example 7.5.1, each with its final held-out loss in nats per token on the same 1,048,576 tokens in windows of 512, fitted under three counts of \(N\).
| \(N\) counts | \(\alpha\) | \(\beta\) | \(E\) | \(a = \beta/(\alpha + \beta)\) | largest residual | predicted loss of the main run |
|---|---|---|---|---|---|---|
| the blocks | 0.17 (0.06 to 0.29) | 0.73 (0.44 to 0.86) | 2.80 (0.84 to 3.20) | 0.81 (0.71 to 0.92) | 2.3% | 3.378 |
| the blocks and the output layer | 0.52 (0.34 to 0.73) | 0.71 (0.47 to 0.86) | 3.19 (2.85 to 3.37) | 0.57 (0.47 to 0.66) | 2.0% | 3.380 |
| every parameter | 0.62 (0.40 to 0.85) | 0.71 (0.48 to 0.86) | 3.20 (2.87 to 3.37) | 0.53 (0.43 to 0.64) | 2.0% | 3.381 |
Counting the blocks and the output layer, the 23 runs give \(\alpha = 0.52\) and \(\beta = 0.71\), so the best size for a budget of compute \(C\) grows as \(C^{0.57}\). Counting the blocks alone moves \(\alpha\) to 0.17 and that exponent to 0.81, the direction that Proposition 7.5.2 predicts. All three counts fit the grid to within 2.3%, so the grid does not choose among them, and under the last two the interval of \(a\) includes one half: a constant ratio of tokens to parameters is not excluded.
At the main run the law fails. The run stands far outside the grid, at 15 times the tokens of its longest budget and 30 times the compute of its largest run. It also reads windows of 1,024 tokens where the grid reads 512, so the law is set against the main model scored in windows of 512: 3.086 nats per token, where its own windows give 3.025. The law says 3.38 under every count, 9.5% too much, and under two of the three counts its floor \(E\) lies above what the run reached. For the budget of the main run, \(1.98 \times 10^{18}\) operations, the second fit asks for 316 million parameters and 1.0 billion tokens, about three for each parameter, where gpt-pp has 110 million and reads 27, and by Proposition 1.7.5 it prices the difference at 20.0% of the reducible loss. A law that misses the run by 9.5% cannot judge its split.
Five things limit what the fit can bear. One seed a cell. No rate was tuned for a size, and the rule for the vocabulary matrices alone (Example 7.5.1) moved \(\alpha\) from 0.56 to 0.52 under the second count, which is why the exponents are given to two digits. The budgets span a factor of eight, over which \(E\) is weakly determined and is not to be read as the entropy of the text. The shortest budget is 381 steps, of which the warmup takes 31. And the additive form is itself in question (Section 1.7). ◁
The constants are local, and other fits of the same law show how local. In January 2026 the second project of Section 7.4 fitted its own family of models and reported: "In Chinchilla, they empirically measure k₂/k₁ to be 20. In nanochat, when you do the fits you actually get something much lower: 8." Ten days later, after tables of embeddings had been added that hold parameters and cost no arithmetic, "the optimal ratio has halved from 8 to 4".The same discussion and the project's experiment log, entries of 7, 17 and 27 January 2026. The last reports 10.3 to 11.2 tokens per parameter with the block matrices and the output layer counted, 3.0 to 4.0 with every parameter and 8.5 to 17.2 with the block matrices alone, and frontier exponents of 0.54, 0.37 and 0.70 under the three counts. The project's default is now 12, on the first of them. A ratio of tokens to parameters is a statement about a count, a recipe and a range of budgets.
Sampling
A trained model gives a distribution over the next token (Definition 2.5.1), and a text has to be drawn from it one token at a time. Temperature and truncation change that distribution before the draw. We compare seven sampling rules on gpt-pico using the same opening and random numbers.
text The little dog [was] running around. He was running around in circles with a paste ... the bracketed token is the first one gpt-pico wrote after the opening next after "The little dog", temperature 1, top-p 0.9: the 14 likeliest of 4,096 are kept 1 was 71.5 % █████████████████████ kept, drawn 2 and 4.8 % █ kept 3 had 2.8 % █ kept 4 went 2.0 % █ kept ... 14 with 0.7 % kept 15 lived 0.5 % cut the kept hold 90.5 % of the probability; the other 4,082 tokens are cut rule kept here tokens mean probability bits a token repeated greedy 1 112 0.48 1.25 15 % temperature 1 4,096 120 0.27 3.57 0 % 1 · top-k 40 40 120 0.33 2.96 2 % 1 · top-p 0.9 14 120 0.43 2.06 0 % temperature 1.5 4,096 120 0.15 8.10 0 % 1.5 · top-p 0.9 289 120 0.20 6.71 0 % 1.5 · min-p 0.1 3 120 0.27 3.16 0 % (the live figure needs JavaScript; it loads 2.3 MB of weights when a button is first pressed)
Definition 7.6.1 (decoding rules). Let \(z\) be the logits at a position and \(p = \mathrm{softmax}(z/T)\) the distribution at temperature \(T\). A truncation rule keeps a set \(S\) of tokens and samples from \(p\) restricted to \(S\) and renormalized. Top-\(k\) keeps the \(k\) most probable tokens; the nucleus, or top-\(p\), the smallest set of most probable tokens whose probability reaches a given mass; min-\(p\) the tokens whose probability is at least a given fraction \(\rho\) of the largest. Greedy decoding takes the most probable token, the limit \(T \to 0\).
In the code. The temperature, the three cuts and the draw. They are truncate and sample in gpt-pp's generate.py.
def truncate(probs: Tensor, top_k: Optional[int] = None, top_p: Optional[float] = None,
min_p: Optional[float] = None) -> Tensor:
...
keep = torch.ones_like(probs, dtype=torch.bool)
if top_k is not None and top_k < probs.size(-1):
kth_largest = probs.topk(top_k, dim=-1).values[:, -1:]
keep &= probs >= kth_largest
if top_p is not None and top_p < 1.0:
ordered, order = probs.sort(dim=-1, descending=True)
mass_before = ordered.cumsum(dim=-1) - ordered # probability of the tokens more likely than each one
keep &= torch.zeros_like(keep).scatter(-1, order, mass_before < top_p)
if min_p is not None and min_p > 0.0:
keep &= probs >= min_p * probs.max(dim=-1, keepdim=True).values
probs = probs * keep
return probs / probs.sum(dim=-1, keepdim=True)
def sample(logits: Tensor, temperature: float = 1.0, top_k: Optional[int] = None, top_p: Optional[float] = None,
min_p: Optional[float] = None, generator: Optional[torch.Generator] = None) -> Tensor:
...
if temperature == 0.0:
return logits.argmax(dim=-1, keepdim=True)
probs = torch.softmax(logits.float() / temperature, dim=-1)
return torch.multinomial(truncate(probs, top_k, top_p, min_p), num_samples=1, generator=generator)
probs is the \(p = \mathrm{softmax}(z/T)\) of Definition 7.6.1, one row a sequence, and keep is the set \(S\) as a mask over the vocabulary. mass_before is, for each token, the probability of the tokens more probable than it, so the test mass_before < top_p keeps the token that carries the sum across the mass: the smallest set that reaches it. The line of min_p is the rule that Proposition 7.6.5 turns into a window below the top logit. Temperature reshapes the whole distribution, and lowering it can only lower the entropy (Proposition 2.5.2).
Quirk. One distribution, every cut at once: top-k, top-p and min-p are each computed on the same temperature-scaled probs, not one on what another left; a token must pass all that are set, and what is kept is renormalized once. Top-k keeps every token tied with the \(k\)-th, so it can keep more than \(k\). Temperature zero is the argmax, with no draw. The seed feeds a torch.Generator created on the prompt's device, from which all rows of a batch draw, so one seed need not give the same text on a processor and on a card, or in a batch of another size.
The first rule, of 2018, fixed the number of tokens kept. The second, of 2019, fixed their mass, since a fixed number is too many where the model is sure and too few where it is not. The third, of 2024, fixes a ratio to the top.Fan, Lewis and Dauphin, "Hierarchical neural story generation", ACL 2018 (arXiv 1805.04833). Holtzman, Buys, Du, Forbes and Choi, "The curious case of neural text degeneration", ICLR 2020 (arXiv 1904.09751). Nguyen, Baker, Neo, Roush, Kirsch and Shwartz-Ziv, "Turning up the heat: min-p sampling for creative and coherent LLM outputs", ICLR 2025 (arXiv 2407.01082).
Example 7.6.2 (seven rules, one opening). The setup: gpt-pico, the opening "The little dog", seven rules with the same random numbers under each, and 120 tokens or the end of the story; everything is measured by the model that runs in the page, on one seed. After the opening gpt-pico puts 71.5% on "was" and 4.8% on "and", with an entropy of 2.44 bits. At temperature 1 a nucleus of 0.9 keeps the 14 likeliest of the 4,096 tokens, which hold 90.5% of the probability, and leaves 1.45 bits. At temperature 1.5 it keeps 289, where min-p at 0.1 keeps three.
Greedy decoding wrote 112 tokens to which gpt-pico gave a mean probability of 0.48, at 1.25 bits a token, and 15% of them complete a run of four tokens already written: it repeats whole sentences and then ends the story. Sampling at temperature 1 with nothing cut cost 3.57 bits a token, and 2.06 with the nucleus. At temperature 1.5 with nothing cut the mean probability fell to 0.15 and the cost rose to 8.10 bits a token; the nucleus brought it to 6.71 and min-p to 3.16. ◁
Why not the most probable text. A model trained to give real text a high probability might be expected to write well when asked for its most probable text. It writes worst then.
One might wonder if the issue with maximization is a search error, i.e., there are higher quality sentences to which the model assigns higher probability than to the decoded ones, beam search has just failed to find them. Yet Figures 2 & 6 show that the per-token probability of natural text is, on average, much lower than text generated by beam search. Natural language rarely remains in a high probability zone for multiple consecutive time steps, instead veering into lower-probability but more informative tokens. Nor does natural language tend to fall into repetition loops, even though the model tends to assign high probability to this, as seen in Figure 4.
—A. Holtzman and others, "The curious case of neural text degeneration" (2020). Section 4.3 of the paper, its first paragraph. The caption of its Figure 4: "The probability of a repeated phrase increases with each repetition, creating a positive feedback loop."
| method | perplexity | self-BLEU4 | Zipf coefficient | repetition % | HUSE |
|---|---|---|---|---|---|
| Human | 12.38 | 0.31 | 0.93 | 0.28 | |
| Greedy | 1.50 | 0.50 | 1.00 | 73.66 | |
| Beam, b=16 | 1.48 | 0.44 | 0.94 | 28.94 | |
| Stochastic Beam, b=16 | 19.20 | 0.28 | 0.91 | 0.32 | |
| Pure Sampling | 22.73 | 0.28 | 0.93 | 0.22 | 0.67 |
| Sampling, t=0.9 | 10.25 | 0.35 | 0.96 | 0.66 | 0.79 |
| Top-k=40 | 6.88 | 0.39 | 0.96 | 0.78 | 0.19 |
| Top-k=640 | 13.82 | 0.32 | 0.96 | 0.28 | 0.94 |
| Top-k=40, t=0.7 | 3.48 | 0.44 | 1.00 | 8.86 | 0.08 |
| Nucleus p=0.95 | 13.13 | 0.32 | 0.95 | 0.36 | 0.97 |
Human text costs the model \(\log_2 12.38 = 3.63\) bits a token. Its own greedy output costs it 0.58, and three passages in four end in a repeated phrase. gpt-pico's greedy text in Example 7.6.2 is the same thing at a small scale: probabilities near a half, and sentences that return. The reason is older than language models.
Proposition 7.6.3 (the most probable sequence is not typical). Let \(X_1, X_2, \dots\) be independent draws from a distribution \(p\) on a finite set, with entropy \(H\) bits, not uniform on its support. Then (i) for every \(\epsilon > 0\), \(\Pr\bigl[\,\lvert -\tfrac1n \log_2 p(X_{1:n}) - H \rvert > \epsilon\,\bigr] \to 0\) as \(n \to \infty\); and (ii) a most probable sequence of length \(n\) consists of most probable symbols, has probability \(p_{\max}^{\,n}\) with \(-\log_2 p_{\max} < H\), and lies outside the set of (i) whenever \(\epsilon < H + \log_2 p_{\max}\).
Proof. (i) \(-\log_2 p(X_{1:n})\) is a sum of \(n\) independent terms of mean \(H\), and the weak law of large numbers applies. (ii) \(H\) is the mean of \(-\log_2 p(x)\) under \(p\), which exceeds its smallest value unless \(-\log_2 p\) is constant on the support. ∎
Example 3.3. Consider an i.i.d. stochastic process \(\boldsymbol{Y}=\{Y_{t}\}_{t=1}^{\infty}\) where \(Y_{t}\) is defined as the outcome of flipping a biased coin: we have \(p(\text{heads})=.6\) and \(p(\text{tails})=.4\). If we flip 100 coins, the most likely outcome is the sequence of 100 heads. However, this would be a surprising outcome to most people, who would intuitively expect the sequence to consist of roughly \(60\%\) heads and \(40\%\) tails. Indeed, even for relatively large \(\varepsilon\), the sequence of 100 heads is not in the \(\mathcal{T}_{\varepsilon}^{(T)}\) typical set; its average symbol probability is \(.6\gg 2^{-\mathrm{H}(Y_{t})}\approx 0.51\).
The above example demonstrates that the typical set often does not contain the most likely sequence. Additionally, the typical set is interesting because, as \(T\rightarrow\infty\), it contains nearly all the probability mass; we formalize this property in a proposition.
—Meister, Pimentel, Wiher and Cotterell, "Locally typical sampling" (2022). TACL 2022 (arXiv 2202.00666), Example 3.3 and the paragraph after it. For sources with memory the statement (i) is the Shannon-McMillan-Breiman theorem.
The sequence of a hundred heads has probability \(6.5 \times 10^{-23}\), eleven million times that of any one sequence with 60 heads; but there are \(1.4 \times 10^{28}\) of those, and the sequences with 55 to 65 heads have probability 0.74 between them. A model of language is a coin with many faces whose bias depends on the past, and the past is what greedy decoding corrupts: each repetition makes the next more probable.
The opposite rule, sampling from the whole distribution, fails for a reason that the loss hides.
Proposition 7.6.4 (a tail compounds). Let the model's distribution at every step be \(q = (1 - \varepsilon)\,p + \varepsilon\,r\), where \(p\) is the distribution of real text and \(r\) is carried by tokens from which a text does not recover. Then (i) on real text the loss of \(q\) exceeds that of \(p\) by at most \(-\ln(1 - \varepsilon)\) nats per token; and (ii) a sample of \(n\) tokens from \(q\) contains no draw from \(r\) with probability \((1 - \varepsilon)^n\).
Proof. (i) \(q \ge (1 - \varepsilon)p\) at every token, so \(-\ln q(x) \le -\ln p(x) - \ln(1 - \varepsilon)\). (ii) The steps choose between the two parts independently. ∎
With \(\varepsilon = 0.01\), (i) is a hundredth of a nat, a third of a percent of a loss of three nats, and (ii) is 0.13 at \(n = 200\). The loss charges a hundredth of a nat for a tail of one percent, and 87 of 100 samples of 200 tokens meet it. Training cannot see what sampling cannot avoid (Remark 1.1.6). Truncation is the remedy on the side of sampling: it cuts the tail and renormalizes, at the price of a bias toward the head, and greedy decoding is the limit in which the bias is everything.
Proposition 7.6.5 (min-\(p\) is a window below the top logit). At temperature \(T\), token \(i\) passes the rule of min-\(p\) with ratio \(\rho\) if and only if \(z_{\max} - z_i \le T\ln(1/\rho)\).
Proof. \(p_i/p_{\max} = e^{(z_i - z_{\max})/T}\). ∎
At \(\rho = 0.1\) the window is \(2.3\,T\) nats wide, so the kept set still moves with the temperature. A set of fixed mass moves more: in Example 7.6.2 the nucleus held 14 tokens at temperature 1 and 289 at 1.5, where the window of min-p held three, and 1,239 at one later position of its text.
The cache in use. Writing uses the cache of Section 3.7: the prompt is run once and leaves its keys and values, and each new token then costs one row in every head. For gpt-pp that is 18,432 numbers a token, 36 MiB at the full context of 1,024 tokens in 16-bit floats, and writing 1,024 tokens without the cache would compute 342 times as many scores. On one card gpt-pp writes 167 tokens per second with the cache and 176 without it (1,024 tokens after a short prompt, one sequence, the better of two timings each): at this size the cache saves arithmetic, not time.
In the code. The loop that writes with the cache. It is the body of generate in gpt-pp's generate.py.
generator = None
if seed is not None:
generator = torch.Generator(device=idx.device).manual_seed(seed)
window = model.cfg.block_size
kv = KVCache(model.cfg.n_layer)
logits = model.forward_explicit(idx[:, -window:], kv=kv)[:, -1]
for step in range(max_new_tokens):
token = sample(logits, temperature, top_k, top_p, min_p, generator)
idx = torch.cat((idx, token), dim=1)
if step == max_new_tokens - 1:
break
if kv.length < window:
logits = model.forward_explicit(token, kv=kv)[:, -1]
else:
kv = KVCache(model.cfg.n_layer)
logits = model.forward_explicit(idx[:, -max(1, window // 2):], kv=kv)[:, -1]
kv is the cache of Section 3.7, and forward_explicit is the path of the model that takes it, the one that writes attention out where training calls the fused kernel (Section 3.6).
Quirk. Writing never exceeds the trained context, and it slides by halves. A prompt is cut to its last block_size tokens; when the cache is full it is thrown away and rebuilt from the most recent half of the window, so that past a full window a token is predicted from between half a context and a whole one, where nanoGPT crops to the last block_size tokens at every step. The loop of the chat stage does not slide: it stops when the context is full.
Known issue. The cache grows by concatenation: KVCache.extend joins the new keys and values onto the stored ones with torch.cat, at every token and in every layer, and nothing is preallocated. The cache bought no speed in the timing above, and the cause was not profiled.
Known issue. The two timings did not write the same text. Under bfloat16 the explicit path adds a float32 mask to its scores and takes the softmax in float32, where the fused kernel does scores, softmax and average on bfloat16 inputs, so the two paths differ by the rounding of bfloat16: a test allows 0.15 between their logits on a tiny model. With the same seed at temperature 0.8, the text written with the cache and the text written through the fused path are the same for 101 of the 1,024 tokens and then part. In float64 the two paths agree to \(10^{-5}\) in the logits, and greedy decoding is identical (Section 3.6).
What the model cannot do
The model that the run produced completes text. What it does not do makes a longer list than what it does, and each item has a cause that this series has met.
It knows the form of a fact before the fact. The two sentences about Joyce in Example 7.3.3 are the case. After "The chemical symbol for gold is", the finished model writes "gold. It is the most common element in the Earth’s": the grammar of an answer, with the word in the place of the symbol and a false sentence begun after it.
It has never seen a brace. After a line of Python that ends in return, the model gives the token { a probability of \(3.9 \times 10^{-12}\), where a uniform guess would give \(3.1 \times 10^{-5}\): the token ranks 32,686th of 32,768. It writes "0 ⏎ You can perform the functions shown in the following code: ⏎ If you like", with ⏎ for a line break. The row of { in its embedding is where initialization left it, and the row in its unembedding has only ever been pushed away from the states the model produces (Proposition 2.2.5). More training on this corpus would not change that.
It does not follow instructions. It was trained to continue documents, and a request is the beginning of a document. After "Write a poem about the sea." it writes "Learn the names of the creatures you like and love you. ⏎ Okay, I like your Pictionary. I'm not a huge fan of dictionaries, but I think you may be thinking. You might". Turning a model that continues into one that answers is a further stage of training, on conversations, and Section 7.8 runs it.
It does not calculate. Numbers reach it in chunks of at most three digits cut from the left (Example 2.1.5). After "Question: What is 17 + 26?" and "Answer:" on the next line it writes "17 + 26 = 17".
It forgets. Its context is 1,024 tokens, about 4,700 bytes, and what it wrote before them is gone.
It guesses where a question needs more than recall. On ARC-Challenge it scores 26.2%, where chance is 25%, and on WinoGrande 50.9%, where chance is 50%. Every released model under 300 million parameters is within four points of chance on both.
None of these is a fault of the run. Two belong to the corpus and the objective, one to the tokenizer and three to the scale.
From completing to answering
One item of that list has a remedy, and it is a stage of training. The weights stay and the text changes. A conversation is written as one sequence of tokens, in which four tokens that the tokenizer reserved and pretraining never used mark where the user's turn and the assistant's begin and end, and the same loss is counted only on what the assistant writes, its end token included: that is how a model learns to stop.
tokens <|bos|> <|user_start|> Hi there <|user_end|> <|assistant_start|> [Hello! How can I help you today? <|assistant_end|>] <|user_start|> I'm learning about ancient civilizations. What continent was ancient Egypt on? <|user_end|> <|assistant_start|> [Ancient Egypt was located on the continent of Africa. <|assistant_end|>] [ ] carries the loss: 21 of 44 tokens, the assistant's text and its end token stage 67.1M tokens of 147,036 conversations in 32 minutes on one card cost of a held-out token, nats, before → after: the assistant's text 2.40 → 1.45 · its end token 26.36 → 0.24 · the user's text 3.03 → 2.88 · the three markers it is given 24.85 → 18.55 all tokens that carry the loss 2.52 → 1.44; plain text by document 0.932 → 1.213 bits per byte a second full run at 0.2 of the rates instead of 0.8: 1.53 on those tokens, by document 1.087 asked What is the capital of France? (texts at temperature 0.7) before The capital of France is the British colony of France. It is located in the Atlanti… after The capital of France is Paris. Paris is located in the northern part of France and… tally 20 answers at temperature 0.7, judged in one reading by a language model, not a person, against criteria fixed beforehand: 8 right, 10 wrong, 2 off the subject (the live figure needs JavaScript and loads about 78 kB of recorded text; the 12-layer model does not run in the page)
In the code. The format of a conversation and the mask of its loss, then the mask turned into targets. They are assemble and inputs_and_targets in gpt-pp's chat.py.
def assemble(roles: Sequence[str], pieces: Sequence[Sequence[int]],
special: dict[str, int]) -> tuple[list[int], list[int]]:
...
ids, mask = [special[BOS]], [0]
for i, (role, piece) in enumerate(zip(roles, pieces)):
if role != ("user", "assistant")[i % 2]:
raise ValueError(f"message {i} is by {role!r}: turns alternate, the user first")
if role == "user":
ids += [special[USER_START], *piece, special[USER_END]]
mask += [0] * (len(piece) + 2)
else:
ids += [special[ASSISTANT_START], *piece, special[ASSISTANT_END]]
mask += [0] + [1] * (len(piece) + 1)
return ids, mask
...
def inputs_and_targets(ids: np.ndarray, carries: np.ndarray, device: Union[str, torch.device]) -> tuple[Tensor, Tensor]:
...
block = torch.from_numpy(ids.astype(np.int64))
counted = torch.from_numpy(carries[:, 1:].astype(bool))
x, y = block[:, :-1], torch.where(counted, block[:, 1:], -1)
return x.contiguous().to(device), y.contiguous().to(device)
pieces holds the messages already encoded, one a turn. mask is 1 on the tokens that carry the loss, the assistant's text and its end token, and 0 on the beginning-of-text token, on the user's turn and on the token that opens the assistant's turn, which the model is given and never asked to write. In inputs_and_targets a place that does not carry the loss gets the target -1, the value that the model's loss ignores, where PyTorch's own default is -100 (Section 1).
Quirk. The format has no system role: a leading system message is joined to the first user message, a blank line between them. Whole conversations are packed into rows of the context length, and the rest of a row is filled with the beginning-of-text token, which carries no loss; a row's later conversations can see its earlier ones. At use, an answer stops at any special token, not only at the assistant's end token.
Example 7.8.1 (the stage, measured). The setup: 147,036 conversations from three subsets of SmolTalk, the corpus on which the SmolLM2 models were taught to answer, 112,979 of them questions and answers written by Llama 3.1 405B; gpt-pp reads them once, in 2,048 steps of 32,768 tokens, 32 minutes on one card, with the optimizers of the main run at 0.8 of their peak rates; 3,000 conversations held out; one seed.
On the held-out conversations the tokens that carry the loss cost 2.52 nats each before the stage and 1.44 after, and the end token alone goes from 26.4 nats to 0.24. Asked "What is the capital of France?", the pretrained model continues the page that the question might stand on, "The capital of France is the British colony of France. It is located in the Atlantic Ocean. The capital of France is Paris.", and goes on until it is cut off. The tuned model answers "The capital of France is Paris. Paris is located in the northern part of France and is a major European city.", adds two sentences and stops, as it does in 39 of the 40 recorded answers. The 20 answers at temperature 0.7 were judged in one reading by a language model, against criteria fixed beforehand, and not by a person: eight are right, ten wrong and two off the subject. None of the three sums is right under either decoding: "7 + 5 = 15" in one, "7 + 5 = 5" in the other. ◁
Quirk. The stage has no recipe of its own. stage_config takes the main run's training configuration and multiplies every peak rate by 0.8, with a warmup of one million tokens and a linear decay, and the optimizers start from fresh state, since a checkpoint holds none. It runs on one card, with no clipping and no distributed mode; its loss is computed by the compiled model and its evaluations by the uncompiled one. A base checkpoint without its run's config.json is tuned from today's default configuration instead of its own.
The stage teaches the form of an answer and when to stop; what the answer says is still the knowledge of Section 7.7. And it is paid for in plain text, which by document goes from 0.932 to 1.213 bits per byte: read as the main run read itself, the tuned model stands where that run stood between its readings at 50 and 100 million tokens. Run in full at a quarter of the rates, the same stage ends at 1.53 nats on the conversations in place of 1.44, and at 1.087 bits per byte of plain text, by document; the rate of 0.8 was chosen among four, in short trials, by a rule fixed beforehand that looked only at the conversations. The scores of Section 7.4, and the model that Section 8 to Section 10 open, are those of the model before this stage.
The accounting
Noise between seeds. The ten rungs of the ladder of Section 6.8, 8-layer models on 100 million tokens, were each trained twice. The two seeds of a rung, which read the same text in another order, differ by 0.001 to 0.018 nats, by 0.006 or less on eight of the ten and by more on two rungs trained with AdamW, with a root mean square of 0.008. Fourteen runs of the same model under the 2026 recipe, on 50 million tokens, have a second seed too, and differ by 0.001 to 0.008 nats, with a root mean square of 0.005. Between two single runs of ours, a difference under about 0.02 nats is not established, or under about 0.01 where both were trained with the 2026 recipe.
Aggregates of benchmarks are louder. The second project of Section 7.4 launched one configuration seven times: "Mean is 0.25714 (higher than the GPT-2 threshold needed), max-min is 0.01646."The notes on the fourth entry of its list of records, March 2026. The seven scores run from 0.25119 to 0.26765. A range of 0.016 on a score of 0.257 is 6%. This is why the claims of this part are made in bits per byte with paired errors, and why a difference of a point or two between two rows of the benchmark table is not offered as a finding unless it stands beyond two paired standard errors over the items. Those errors are over documents and items; what another seed of training would change was not measured for the main run.
What it cost.
| what | cards and time | card-hours |
|---|---|---|
| both corpora and tokenizers, after the downloads | no card; 96 processor cores, about five minutes | |
| gpt-pico, each recipe | 1 card, 4 minutes | 0.07 |
| the main run | 3 cards, 3.20 hours | 9.6 |
| the same recipe laid out for 1.08 billion tokens | 3 cards, 1.15 hours | 3.4 |
| the stage of Section 7.8: two runs and four short trials | 1 card each | 1.0 |
| the race at 100M tokens: nine runs, and two at other rotary bases | 1 card each | 3.3 |
| the grid: 23 runs as fitted, and the 19 that they replaced | 1 card each | 3.9 |
| 37 further runs on the 8-layer model: seeds, warmups, other rules | 1 card each | 2.9 |
| scoring ten released models | 1 card, 40 minutes | 0.7 |
| six hosted models | 2.79 dollars, pilots included |
Two public races. Two public races give the scale of such a run in 2026, each with its own target. One has asked since May 2024 for "the fastest algorithm to use 8 NVIDIA H100 GPUs to train a language model that attains 3.28 cross-entropy loss" on the validation text of FineWeb, at the size of GPT-2 small: its first entry took 45 minutes, and its 92nd record, of 30 August 2026, takes 40 seconds. The other asks for the benchmark score of the largest released GPT-2 and counts hours, from 3.04 in January 2026 to 1.65 in March. Our run is an entry in neither race.
two races on eight H100 cards, on a logarithmic time axis; each compares only with itself the 124M race: validation loss 3.28 on FineWeb, 92 records record 1 3 5 12 20 53 92 date 2024-05 2024-10 2024-10 2024-11 2025-01 2025-12 2026-08 time 45 min 25 min 15 min 5.0 min 3.0 min 2.0 min 40 s time to GPT-2: the benchmark score of the released 1.5B model, 6 records date 2026-01 2026-02 2026-02 2026-03 2026-03 2026-03 time 3.04 h 2.91 h 2.76 h 2.02 h 1.80 h 1.65 h references: GPT-2 itself, 2019, about a week on 32 TPU v3 (second-hand); nanoGPT, 2023, 4 days on eight A100; llm.c, 2024, 90 minutes on eight A100 gpt-pp: 3.0B tokens on three cards in 3.2 hours; no target, on neither line (the live figure needs JavaScript and loads 16 kB of data)
Most of the 2026 column of the recipe table was adopted or tested in the first race (Section 5 and Section 6). Not all that wins there is expected to hold elsewhere: its latest record carries a hashed table of n-grams with 65 billion parameters for a run of under 330 million tokens, and the keeper of the records grants that "some of the methods used in the speedrun are unlikely to scale, particularly those which impose additional structure on the network, such as logit softcapping".K. Jordan and contributors, modded-nanogpt (GitHub, 2024 to 2026), the opening of its description, its answers to objections and the notes of the record of 30 August 2026. gpt-pp keeps the soft cap and was not timed against this target. In card-hours, the first race fell from 6.0 to 0.089 and the second from 24.3 to 13.2, on cards of another kind; the main run took 9.6, for a smaller model than the second race trains and for no target but its budget of tokens.
The commands that reproduce the run and the model's card are in Section 11. Section 8 asks what three billion tokens taught the model and in what order, and Section 9 and Section 10 open it.