1. Predicting the next token

Draft

One distribution, three uses

A language model is one object that does three jobs. It is a probability distribution over strings of text, stored as a rule for guessing the next symbol from the symbols before it. Score the guesses against what was actually written, and it is a predictor. Hand the guesses to a coder, and it is a compressor: the text shrinks to about as many bits as the model was surprised by it. Draw from the guesses, and it is a generator. To predict is to compress is to generate. The three were one idea in 1948, when the rule was a table of letter counts, and they are one idea in a network of 175 billion parameters. What changed in between is the function that makes the guess, and how much of the past it can use.

The oldest form of the idea can be tried on a book of 366 short children's stories. At orders 0 and 1, the next symbol is drawn at random, uniformly or by its frequency in the book. At higher orders it is drawn from what followed a matching window: one preceding symbol at order 2, two at order 3, and so on, with the same construction for whole words. gpt-pico, the smallest size of gpt-pp (Section 11), keeps the sampling loop and replaces the lookup with a learned predictor of about two million parameters.

line           window              bits   what it writes after the same opening
characters 0   none: all alike     5.98   Once upon a time,c↵VyyEbiNzPS1MC5S'LlF?!NanF?e
characters 1   none: by frequency  4.40   Once upon a time,e"  aneen  am ibece sthsd  We
characters 2   1 character         3.29   Once upon a time, There hed phe asame lind an 
characters 3   2 characters        2.40   Once upon a time, ant hey tre putte de withe w
characters 4   3 characters        1.89   Once upon a time, that there ribbon a like com
characters 5   4 characters        1.68   Once upon a time, the good with me chair mom h
characters 6   5 characters        1.59   Once upon a time, in a small house. The girl n
words 1        none: by frequency  2.11   Once upon a time, lose upon into and Finny
words 2        1 word              1.64   Once upon a time, there was still to help
words 3        2 words             1.59   Once upon a time, there was a ball from a
gpt-pico           all so far          0.51   Once upon a time, there was a little boy named

window  what the next symbol may depend on: it is taken from a random place of a book of 366
        stories (300,445 characters) where the window occurs; gpt-pico draws from its own guess
bits    bits per character on 1,400 other stories, each rule taken as a predictor
(the live figure needs JavaScript; it loads 0.3 MB of text, and 2.3 MB of weights for gpt-pico)
Shannon's series of approximations, on a book of 366 children's stories. Every line continues the same opening: its next symbol is whatever followed the window at a random place of the book, and in the last line a draw from gpt-pico's distribution. Green: the window, the symbols the next one may depend on, in the line and where the book has it. The numbers are each rule's loss as a predictor, in bits per character of 1,400 other stories, measured with back-off to shorter windows. Pick a line, write, step, or pick a symbol. Samples are one seed each.

Every predictor uses the same loop: look at a window of what has been written, obtain a distribution over the next symbol, draw from it, append, repeat. The predictors differ in where the distribution comes from and how wide the window is; their samples become more like English as the window widens. They also become cheaper to predict. Scored as predictors on stories outside the book, the rules pay 5.98 bits per character at order 0 and 1.59 with a window of five letters; gpt-pico, which reads everything written so far, pays 0.51. (These 1,400 stories are the plain ones, 87% of the held-out stories; the rest, with stray blanks or other characters, cost gpt-pico 0.73, which is why its figure over all kinds, further on, is 0.54.) The procedure dates from 1948, and Section 1.2 sets out the page it comes from. First the loop in formulas, and the scoring of its guesses.

Throughout, text is a string \(x_{1:n} = x_1 x_2 \dots x_n\) over a finite vocabulary \(\mathcal V\): 27 symbols in the paper of 1948, 256 for the bytes of a file, some tens of thousands of tokens for a modern model (Section 2 builds them). The prefix \(x_1 \dots x_{t-1}\) is written \(x_{<t}\). The source of the text is a distribution \(p\); a model is \(q\), or \(q_\theta\) when its parameters matter. Logarithms are to base 2 and count bits; \(\ln\) counts nats.

Proposition 1.1.1 (the chain rule). Every distribution \(p\) on \(\mathcal V^n\) is the product of its next-symbol conditionals,

\[p(x_{1:n}) \;=\; \prod_{t=1}^{n} p(x_t \mid x_{<t}), \tag{1.1}\]

and conversely any choice of a distribution \(q(\cdot \mid x_{<t})\) on \(\mathcal V\) for every prefix defines, through the same product, a distribution on \(\mathcal V^n\) for every \(n\).

Proof. The definition of conditional probability gives \(p(x_{1:t}) = p(x_{<t})\, p(x_t \mid x_{<t})\) whenever \(p(x_{<t}) > 0\); apply it \(n\) times. If some prefix has probability zero, so has the string, and the product vanishes at its first zero factor whatever value the undefined factors after it are given. For the converse, summing the product over \(x_n\) removes its last factor, so by induction on \(n\) the products are non-negative and sum to one. ∎

The chain rule is an identity, not an assumption: it holds for every source.Writing a joint distribution as next-token conditionals does not assert that tokens are independent. A model makes its approximation when it restricts which parts of the prefix those conditionals can use. What a model assumes is in how it computes each factor. The table at order 3 forgets everything but the last two symbols. A transformer forgets everything beyond its context length, and uses the same parameters at every position.

Definition 1.1.2 (language model). A language model is a map that sends every prefix \(x_{<t}\) to a distribution \(q_\theta(\cdot \mid x_{<t})\) on \(\mathcal V\). Its probability of a string is the product (1.1), and it generates a string by drawing \(x_1, x_2, \dots\) in turn, each from its conditional. Such a model is called autoregressive: every symbol is predicted from the earlier symbols of the same sequence.

The number \(-\log q_\theta(x_{1:n}) = \sum_t -\log q_\theta(x_t \mid x_{<t})\) is the surprisal of the text under the model, in bits. It is the predictor's score. By Section 1.3 it is also, to within two bits, the size of the file to which the model compresses the text. Training chooses the parameters that make the training text least surprising, which is maximum likelihood, and the next three statements say what that optimizes.

Definition 1.1.3 (entropy, cross-entropy, relative entropy). For distributions \(p\) and \(q\) on a finite set,

\[\begin{aligned} H(p) &= -\sum_x p(x) \log p(x), \qquad H(p, q) = -\sum_x p(x) \log q(x), \\ D(p \,\|\, q) &= \sum_x p(x) \log \frac{p(x)}{q(x)}, \end{aligned}\]

with \(0 \log 0 = 0\), and \(D(p \,\|\, q) = \infty\) if \(q(x) = 0 < p(x)\) for some \(x\). They are tied by \(H(p, q) = H(p) + D(p \,\|\, q)\).

The cross-entropy \(H(p, q)\) is the model's average surprisal when the source does the writing, the entropy \(H(p)\) is the source's average surprisal at itself, and the relative entropy, also called the Kullback–Leibler divergence, is the excess of the first over the second.

Proposition 1.1.4 (Gibbs' inequality). \(D(p \,\|\, q) \ge 0\), with equality if and only if \(p = q\). Hence \(H(p, q) \ge H(p)\): on average, no model is less surprised by a source than the source itself.

Proof. If \(q(x) = 0 < p(x)\) for some \(x\) the divergence is infinite. Otherwise use \(\ln u \le u - 1\), with equality only at \(u = 1\):

\[\begin{aligned} -D(p \,\|\, q)\, \ln 2 &= \sum_{x:\, p(x) > 0} p(x) \ln \frac{q(x)}{p(x)} \;\le\; \sum_{x:\, p(x) > 0} \bigl(q(x) - p(x)\bigr) \\ &= \sum_{x:\, p(x) > 0} q(x) \;-\; 1 \;\le\; 0 . \end{aligned}\]

Equality in the first step forces \(q(x) = p(x)\) wherever \(p(x) > 0\), and then \(q\) has no mass left for any other \(x\). ∎

With \(q\) uniform on \(k\) symbols the inequality reads \(H(p) \le \log k\): no source over 27 symbols carries more than \(\log 27 = 4.755\) bits per symbol. Its second consequence is the one training rests on.

Proposition 1.1.5 (maximum likelihood is minimum relative entropy). For a family of models \(q_\theta\) and a source \(p\),

\[\arg\min_\theta\; \mathbb E_{x \sim p}\bigl[-\log q_\theta(x)\bigr] \;=\; \arg\min_\theta\; D(p \,\|\, q_\theta),\]

and if the family contains \(p\) the minimum is attained exactly where \(q_\theta = p\). The same holds with the empirical distribution \(\hat p\) of a training set in place of \(p\): maximizing the likelihood of the training set minimizes \(D(\hat p \,\|\, q_\theta)\).

Proof. \(\mathbb E_p[-\log q_\theta] = H(p, q_\theta) = H(p) + D(p \,\|\, q_\theta)\), and \(H(p)\) does not depend on \(\theta\). The rest is Proposition 1.1.4. ∎

By the chain rule, what is minimized on a training text of \(n\) tokens is the average surprisal per token,

\[\mathcal L(\theta) \;=\; -\frac{1}{n} \sum_{t=1}^{n} \ln q_\theta(x_t \mid x_{<t}), \tag{1.2}\]

in nats by convention. This is the training loss of every model in this series.

In the code. In gpt-pp the model computes (1.2) itself, in the last line of the function that carries a batch, \(B\) sequences of \(T\) tokens each, to its scores for the next token. It is the end of GPT.forward in gpt-pp's model.py.

def forward(self, idx: Tensor, targets: Optional[Tensor] = None) -> Tensor:
    """idx (B, T) token ids -> logits (B, T, V); or, with targets (B, T), the mean cross-entropy in nats
    (a target of -1 is ignored)."""
    ...
    logits = self.unembed_logits(self.final_norm(x))
    if targets is None:
        return logits
    return F.cross_entropy(logits.view(-1, logits.size(-1)), targets.reshape(-1), ignore_index=-1)

At each position logits holds one score for each of the \(V\) tokens of the vocabulary; their softmax, the exponentials of the scores divided by their sum, is the model's conditional for the token after that position, and targets holds the token that came, the input moved one place. F.cross_entropy takes the softmax, the natural logarithm and the mean, so the value is (1.2) in nats with \(n = B \times T\): every position of every sequence is one term, and the mean is over tokens, not over sequences. What the formula leaves open is the prefix. In the code \(x_{<t}\) is the part of the text that lies in the same sequence, so the terms run over contexts of one token up to \(T\) tokens in equal numbers; Section 1.6 measures what the longer contexts are worth.

Quirk. Given targets, GPT.forward returns this one number and not the pair of logits and loss returned by nanoGPT, the open reproduction of GPT-2 small that gpt-pp starts from;nanoGPT (GitHub, 2022 to 2023), model.py, and nanochat (GitHub, 2025 to 2026), gpt.py, as read on 14 November 2026: the first returns the pair, the second the loss alone, and both leave out a target of -1. without targets it returns the logits, so a caller gets one or the other and never both. A target of -1 is left out of the mean, where the default of F.cross_entropy in PyTorch 2.9.1 is -100. In pretraining every target of a training batch counts; the loader of held-out batches pads its last batch with targets of -1, the chat training of Section 7 gives them to the tokens that do not carry the loss, and the evaluation behind Definition 1.1.7 below masks with -100 in a call of its own.

Remark 1.1.6 (what the objective punishes and what it forgives). The divergence is infinite as soon as the model gives probability zero to something the source writes, so a trained model must spread some probability over everything. In the other direction it is lenient: if \(q = \tfrac12 p + \tfrac12 r\) for any distribution \(r\) whatever, then \(q \ge \tfrac12 p\) and \(D(p \,\|\, q) \le 1\) bit, on strings of any length. A model may spend half of its probability on text nobody would write and lose one bit in all.The mixture here is over whole strings. A fresh half-and-half mixture at every token could instead pay one extra bit per token. The distinction is why the bound stays one bit as the string grows. A low loss says that real text is likely under the model, not that the model's samples are all like real text. The same inequality returns as good news in Proposition 1.5.4.

Definition 1.1.7 (perplexity, bits per token, bits per byte). For a held-out text of \(T\) tokens and \(B\) bytes on which a model has loss \(\mathcal L\) nats per token,

\[\mathrm{perplexity} = e^{\mathcal L}, \qquad \text{bits per token} = \frac{\mathcal L}{\ln 2}, \qquad \text{bits per byte} = \frac{T\, \mathcal L}{B \ln 2} .\]

Perplexity was introduced in 1977 by the speech recognition group at IBM, who needed a number for how hard a recognition task is. Their publication is the abstract of a talk, given here whole.

Using counterexamples, we show that vocabulary size and static and dynamic branching factors are all inadequate as measures of speech recognition complexity of finite state grammars. Information theoretic arguments show that perplexity (the logarithm of which is the familiar entropy) is a more appropriate measure of equivalent choice. It too has certain weaknesses which we discuss. We show that perplexity can also be applied to languages having no obvious statistical description, since an entropy-maximizing probability assignment can be found for any finite-state grammar. Table I shows perplexity values for some well-known speech recognition tasks.

—F. Jelinek, R. L. Mercer, L. R. Bahl and J. K. Baker, "Perplexity: a measure of the difficulty of speech recognition tasks" (1977).Journal of the Acoustical Society of America 62 (S1), S63, 1977.

A model with perplexity \(k\) is as uncertain, on average, as one that chooses among \(k\) equally likely symbols.

Example 1.1.8 (one model, three units). The setup: gpt-pp, a transformer of 12 layers and 135 million parameters trained for this series on three billion tokens of FineWeb-EDU, a corpus of educational web pages, and scored on 1,000 documents held out of its training, 4,579,235 bytes; each document is cut into windows of at most 1,024 tokens and every token is scored once.FineWeb-EDU: Penedo, Kydlíček, Ben allal, Lozhkov, Mitchell, Raffel, Von Werra and Wolf, "The FineWeb datasets: decanting the web for the finest text data at scale", 2024 (arXiv 2406.17557). TinyStories, the corpus of gpt-pico: R. Eldan and Y. Li, "TinyStories: how small can language models be and still speak coherent English?", 2023 (arXiv 2305.07759), in its second version, written by GPT-4. Section 7 describes the data, the run and the scoring.

The model's tokenizer makes 984,160 tokens of these documents, 4.653 bytes each on average, and its loss (1.2) on them is 3.007 nats per token. So its perplexity is \(e^{3.007} = 20.2\): out of a vocabulary of 32,768 it is as uncertain as a fair die with 20 faces. In bits, \(3.007 / \ln 2 = 4.338\) per token, and \(4.338 / 4.653 = 0.932\) bits per byte. For scale, a predictor that knows nothing pays eight bits per byte, and the entropy of the token frequencies of this corpus, the cost for a model that knows which tokens are common and nothing about their order, is 2.394 bits per byte (a table of those frequencies, scored on held-out text in Example 1.3.3, pays 2.400). ◁

In the code. The three numbers come from one function, which settles what the definition leaves open: which tokens are scored, from what context, and which bytes are counted. It is bits_per_byte in gpt-pp's evaluate.py.

def bits_per_byte(model: LanguageModel, docs: Sequence[str], batch_tokens: int | None = None,
                  progress: str | None = None) -> dict:
    ...
    doc_tokens = model.encode(docs)
    step = model.window - 1  # tokens of the document in a window; the first place holds the beginning-of-text token
    windows: list[list[int]] = []
    owner: list[int] = []
    for d, ids in enumerate(doc_tokens):
        for i in range(0, len(ids), step):
            windows.append([model.bos_id] + ids[i:i + step])
            owner.append(d)

    scores = score_sequences(model, windows, starts=1, batch_tokens=batch_tokens, progress=progress)

    n_docs = len(docs)
    doc_bits = np.bincount(owner, weights=scores.nll, minlength=n_docs) / LN2
    doc_bytes = np.array([len(t.encode("utf-8")) for t in docs], dtype=np.float64)
    n_tokens = int(scores.n_scored.sum())
    n_bytes = int(doc_bytes.sum())
    assert n_tokens == sum(len(ids) for ids in doc_tokens), "every token of every document is scored exactly once"

    total_bits = float(doc_bits.sum())
    loss = total_bits * LN2 / n_tokens
    ...
    return {
        "bits_per_byte": total_bits / n_bytes,
        "bits_per_byte_se": ratio_standard_error(doc_bits, doc_bytes),
        "loss": loss,  # nats per token
        "perplexity": math.exp(loss),
        "tokens_per_byte": n_tokens / n_bytes,
        "bytes_per_token": n_bytes / n_tokens,
        ...
    }

n_tokens is the definition's \(T\), n_bytes its \(B\) and loss its \(\mathcal L\); scores.nll holds, for each window, the summed surprisal in nats of every token after the first place, and total_bits / n_bytes is the third formula. A window is the model's beginning-of-text token, the mark that opens a document, followed by at most 1,023 tokens of one document (model.window is the model's context length or 1,024, whichever is smaller). The mark is given and never scored, no end-of-text token is appended, and the assert holds the function to its count: every token of every document, once. The bytes are each document's own in UTF-8, the encoding of the text, and not a mean number of bytes per token; the bits are kept by document (doc_bits) because the standard error takes the documents as the sample.

Quirk. The windows do not overlap: the 1,000 documents make 1,510 of them, and the first tokens of every window after a document's first are predicted from less context than the document offers. The model is run in float32 (32-bit floating point) with TensorFloat-32 products, which round each factor of a matrix product to ten bits of mantissa, and not in bfloat16, the 16-bit format it trained in. Section 7 sets out the protocol.

Known issue. The bits per byte that the trainer prints during a run are another measurement: a mean over windows cut from the held-out text as one stream, without regard to documents, in bfloat16, with the beginning-of-text tokens among the targets. For this model it reads 0.9429 where the function above reads 0.9322; Section 7 sets the two side by side.

Remark 1.1.9 (why bits per byte). Perplexity belongs to a protocol as much as to a model. It is per token, so models with different vocabularies cannot be compared by it. Per word, it depends on how the text file was prepared: GPT-2 small on the test set of WikiText-103 has a perplexity of 49.22 per word on the text as distributed and 36.94 once the benchmark's own tokenization is undone (Example 1.5.3). Bits per byte has neither problem. It is the length of a code for the same bytes whatever the vocabulary (Section 1.3), and it is the measure used throughout this series.Bits per byte fixes the unit of comparison. The evaluated bytes, available context and treatment of document boundaries must still be the same; changing those changes the question being measured.

Shannon's source and the guessing game

A Mathematical Theory of CommunicationBell System Technical Journal 27 1948C. E. Shannon

3. THE SERIES OF APPROXIMATIONS TO ENGLISH

To give a visual idea of how this series of processes approaches a language, typical sequences in the approximations to English have been constructed and are given below. In all cases we have assumed a 27-symbol "alphabet," the 26 letters and a space.

1. Zero-order approximation (symbols independent and equiprobable).

XFOML RXKHRJFFJUJ ZLPWCFWKCYJ FFJEYVKCQSGHYD QPAAMKBZAACIBZLHJQD.

2. First-order approximation (symbols independent but with frequencies of English text).

OCRO HLI RGWR NMIELWIS EU LL NBNESEBYA TH EEI ALHENHTTPA OOBTTVA NAH BRL.

3. Second-order approximation (digram structure as in English).

ON IE ANTSOUTINYS ARE T INCTORE ST BE S DEAMY ACHIN D ILONASIVE TUCOOWE AT TEASONARE FUSO TIZIN ANDY TOBE SEACE CTISBE.

4. Third-order approximation (trigram structure as in English).

IN NO IST LAT WHEY CRATICT FROURE BIRS GROCID PONDENOME OF DEMONSTURES OF THE REPTAGIN IS REGOACTIONA OF CRE.

5. First-order word approximation. Rather than continue with tetragram, … , n-gram structure it is easier and better to jump at this point to word units. Here words are chosen independently but with their appropriate frequencies.

REPRESENTING AND SPEEDILY IS AN GOOD APT OR COME CAN DIFFERENT NATURAL HERE HE THE A IN CAME THE TO OF TO EXPERT GRAY COME TO FURNISHES THE LINE MESSAGE HAD BE THESE.

6. Second-order word approximation. The word transition probabilities are correct but no further structure is included.

THE HEAD AND IN FRONTAL ATTACK ON AN ENGLISH WRITER THAT THE CHARACTER OF THIS POINT IS THEREFORE ANOTHER METHOD FOR THE LETTERS THAT THE TIME OF WHO EVER TOLD THE PROBLEM FOR AN UNEXPECTED.

The resemblance to ordinary English text increases quite noticeably at each of the above steps. Note that these samples have reasonably good structure out to about twice the range that is taken into account in their construction. Thus in (3) the statistical process insures reasonable text for two-letter sequences, but four-letter sequences from the sample can usually be fitted into good sentences. In (6) sequences of four or more words can easily be placed in sentences without unusual or strained constructions. The particular sequence of ten words "attack on an English writer that the character of this" is not at all unreasonable. It appears then that a sufficiently complex stochastic process will give a satisfactory representation of a discrete source.

Section 3, page 7 of the reprint with corrections
C. E. Shannon, "A Mathematical Theory of Communication" (Bell System Technical Journal 27 1948), Section 3, page 7 of the reprint with corrections. The six orders of the figure in Section 1.1, as first printed. The line-break hyphens of the reprint are removed from the samples: the alphabet has no hyphen.

The page is from 1948. To say how much a channel must carry, its paper needed a model of whatever produces messages, and the second paragraph of its introduction sets aside what messages mean.

The fundamental problem of communication is that of reproducing at one point either exactly or approximately a message selected at another point. Frequently the messages have meaning; that is they refer to or are correlated according to some system with certain physical or conceptual entities. These semantic aspects of communication are irrelevant to the engineering problem. The significant aspect is that the actual message is one selected from a set of possible messages. The system must be designed to operate for each possible selection, not just the one which will actually be chosen since this is unknown at the time of design.

—C. E. Shannon, "A Mathematical Theory of Communication" (1948).Bell System Technical Journal 27; the introduction, page 1 of the reprint with corrections.

A source, then, is a stochastic process that emits symbols, and the six samples are six processes fitted to English with more and more context. The last sentence of the fragment is the hypothesis the field has worked under ever since: make the process complex enough and it represents the source. The samples were made by hand, and the paragraph after them says how:

The first two samples were constructed by the use of a book of random numbers in conjunction with (for example 2) a table of letter frequencies. This method might have been continued for (3), (4) and (5), since digram, trigram and word frequency tables are available, but a simpler equivalent method was used. To construct (3) for example, one opens a book at random and selects a letter at random on the page. This letter is recorded. The book is then opened to another page and one reads until this letter is encountered. The succeeding letter is then recorded. Turning to another page this second letter is searched for and the succeeding letter recorded, etc. A similar process was used for (4), (5) and (6). It would be interesting if further approximations could be constructed, but the labor involved becomes enormous at the next stage.

—C. E. Shannon, "A Mathematical Theory of Communication" (1948). Bell System Technical Journal 27; Section 3, pages 7 and 8 of the reprint with corrections, whose page numbers are used here.

To open a book at random and read on until a letter turns up is to draw from the distribution of the next letter given the last one, with the book as the sample. It is the lookup of the figure in Section 1.1, done by hand.

The paper also gave the source a number.

Definition 1.2.1 (N-gram entropies and the entropy rate). For a stationary source \(X_1, X_2, \dots\) let

\[F_N \;=\; H(X_N \mid X_{1:N-1}), \qquad G_N \;=\; \tfrac{1}{N}\, H(X_{1:N}),\]

where the conditional entropy \(H(Y \mid X)\) is the average over \(x\) of the entropy of \(p(\cdot \mid x)\). \(F_N\) is the entropy of the next symbol when the \(N - 1\) before it are known, \(G_N\) the entropy per symbol of a block of \(N\), and the entropy rate of the source is \(H = \lim_{N \to \infty} F_N\).

Proposition 1.2.2 (the entropy rate exists). \(F_N\) is non-increasing in \(N\); \(G_N\) is the average of \(F_1, \dots, F_N\), so that \(G_N \ge F_N\); and both converge to the same limit.

Proof. Two facts follow from Section 1.1. Taking \(-\mathbb E \log\) of (1.1) gives \(H(X_{1:N}) = \sum_{t \le N} H(X_t \mid X_{<t})\). And conditioning cannot raise entropy on average, \(H(Y \mid X, Z) \le H(Y \mid Z)\): for each value of \(Z\) the difference is the relative entropy of the joint law of \((X, Y)\) from the product of its marginals, which Proposition 1.1.4 makes non-negative. Hence \(F_{N+1} = H(X_{N+1} \mid X_{1:N}) \le H(X_{N+1} \mid X_{2:N}) = F_N\), the equality by stationarity. A non-increasing sequence bounded below by zero converges. By the first fact and stationarity \(G_N\) is the running average of the \(F_t\), so it has the same limit and is at least its last and smallest term. ∎Theorems 5 and 6 of the paper of 1948.

The number \(F_N\) is the loss of the best possible predictor that sees \(N - 1\) symbols, and the curve \(N \mapsto F_N\) says what context is worth. From tables of the frequencies of letters, pairs and triples, on the 27-symbol alphabet, the values are \(F_1 = 4.03\), \(F_2 = 3.32\) and \(F_3 = 3.1\) bits per letter, against 4.76 for symbols drawn at random: the first four samples of the fragment, as numbers. With structure out to about eight letters the estimate was 2.3 bits, half of the maximum, so that "when we write English half of what we write is determined by the structure of the language and half is chosen freely".Page 14 of the reprint. The values of \(F_1\) to \(F_3\) and the 2.3 bits at eight letters are in the paper of 1951 cited below, pages 50 to 54.

There the tables stop. A table of N-gram frequencies has \(27^N\) rows: 19,683 at \(N = 3\), 14 million at \(N = 5\), \(2.8 \times 10^{11}\) at \(N = 8\). No table could reach the statistics that span a sentence. Three years later the same author found a predictor that already had them: a person.C. E. Shannon, "Prediction and Entropy of Printed English", Bell System Technical Journal 30, pages 50 to 64, 1951.

Prediction and Entropy of Printed EnglishBell System Technical Journal 30 1951C. E. Shannon

The new method of estimating entropy exploits the fact that anyone speaking a language possesses, implicitly, an enormous knowledge of the statistics of the language. Familiarity with the words, idioms, cliches and grammar enables him to fill in missing or incorrect letters in proof-reading, or to complete an unfinished phrase in conversation. An experimental demonstration of the extent to which English is predictable can be given as follows: Select a short passage unfamiliar to the person who is to do the predicting. He is then asked to guess the first letter in the passage. If the guess is correct he is so informed, and proceeds to guess the second letter. If not, he is told the correct first letter and proceeds to his next guess. This is continued through the text. As the experiment progresses, the subject writes down the correct text up to the current point for use in predicting future letters. The result of a typical experiment of this type is given below. Spaces were included as an additional letter, making a 27 letter alphabet. The first line is the original text; the second line contains a dash for each letter correctly guessed. In the case of incorrect guesses the correct letter is copied in the second line.

[…]

An extension of the above experiment yields further information concerning the predictability of English. As before, the subject knows the text up to the current point and is asked to guess the next letter. If he is wrong, he is told so and asked to guess again. This is continued until he finds the correct letter. A typical result with this experiment is shown below. The first line is the original text and the numbers in the second line indicate the guess at which the correct letter was obtained.

(1) T H E R E   I S   N O   R  E V  E R S E   O N   A   M O T O R C Y C L E   A
(2) 1 1 1 5 1 1 2 1 1 2 1 1 15 1 17 1 1 1 2 1 3 2 1 2 2 7 1 1 1 1 4 1 1 1 1 1 3 1
(1) F R I E N D   O F   M I N E   F O U N D   T H I S   O U T
(2) 8 6 1 3 1 1 1 1 1 1 1 1 1 1 1 6 2 1 1 1 1 1 1 2 1 1 1 1 1 1
(1) R A T H E R   D  R A M A T I C A L L Y   T H E   O T H E R   D A Y
(2) 4 1 1 1 1 1 1 11 5 1 1 1 1 1 1 1 1 1 1 1 6 1 1 1 1 1 1 1 1 1 1 1 1 1

Out of 102 symbols the subject guessed right on the first guess 79 times, on the second guess 8 times, on the third guess 3 times, the fourth and fifth guesses 2 each and only eight times required more than five guesses. Results of this order are typical of prediction by a good subject with ordinary literary English. Newspaper writing, scientific work and poetry generally lead to somewhat poorer scores.

The reduced text in this case also contains the same information as the original. Again utilizing the identical twin we ask him at each stage to guess as many times as the number given in the reduced text and recover in this way the original. To eliminate the human element here we must ask our subject, for each possible N-gram of text, to guess the most probable next letter, the second most probable next letter, etc. This set of data can then serve both for prediction and recovery.

Section 3, pages 54 to 56
C. E. Shannon, "Prediction and Entropy of Printed English" (Bell System Technical Journal 30 1951), Section 3, pages 54 to 56. The placing of the numbers under the letters is reconstructed from a scan in which they run together, one number for each letter and each space. The counts agree with the sentence under the table; the positions of the three two-digit guesses are inferred.

In a version of Shannon's guessing game with tokens for letters, a table of token pairs sees one token of context and gpt-pico sees everything so far. A guess number \(r\) is charged \(\log r(r + 1)\) bits, the length of a code that a twin could decode. On the sentence of 1951, the two machines, which have read nothing but children's stories, spend 3.08 and 2.45 bits per character; Shannon's subject, by the same charge, spends 1.77.

The game: a sentence is hidden; guess its next token until you have it. Under each token: the
number of the guess that found it, by a table of token pairs (it sees one token) and by gpt-pico
(it sees the sentence so far).

text       Once    upon       a    time       ,   there     was       a  little    girl   named
pairs         1       1       1       2       1       3       1       1       3       1       1
gpt-pico          1       1       1       1       1       1       1       1       1       1       1
text       Lucy       .     She     had       a   black     cat   named      Sh    adow       .
pairs         6       1       4       3       1     198       2       1     126       4       4
gpt-pico          3       1       1       1       1      33       1       1      27       1       1

bits    a guess number r is charged log2 r(r + 1) bits: the table 74.5 (0.86 per character),
        gpt-pico 42.3 (0.49)
first   on 1,000 held-out stories the first guess is right: gpt-pico 61.5 %, the table 25.1 %
(the live figure needs JavaScript; it loads less than 0.1 MB)
Shannon's guessing game of 1951, with tokens for letters. Type the next token of the hidden sentence, or pick one of the chips, until you have it; a table of token pairs and gpt-pico guess beside you. Under each token stand the three guess numbers, the reduced text of the paper. Green: the bits spent, one bar a player, a guess number r being charged log₂ r(r + 1) bits. The machines' guesses are computed beforehand on the page's own table and weights; the figures at large are measured on 1,000 held-out stories.

Two things in the fragment carry the rest of this section. The first is the twin. The row of numbers, the reduced text, is the sentence in another alphabet, and anyone with the same habits of guessing can turn it back. A predictor is therefore an invertible map from a text to a sequence that is mostly ones, which is cheap to store; Section 1.3 completes that thought. The second is that the frequencies of the numbers bound the entropy from both sides.

Proposition 1.2.3 (bounds from the guesses, 1951). Let a stationary source have \(K\) symbols, and let an ideal predictor, one that knows the conditional distribution of the next symbol given the \(N - 1\) before it, guess in order of decreasing probability. Let \(r_i\) be the frequency with which the true symbol is its \(i\)-th guess, and \(r_{K+1} = 0\). Then

\[\sum_{i=1}^{K} i\, (r_i - r_{i+1}) \log i \;\;\le\;\; F_N \;\;\le\;\; -\sum_{i=1}^{K} r_i \log r_i . \tag{1.3}\]

The upper bound holds for any predictor whose guesses depend only on those \(N - 1\) symbols.

Proof. Fix a context \(c\) and write \(p_{(1)}, p_{(2)}, \dots\) for the conditional probabilities of the symbols in the order they are guessed. Given \(c\), the symbol and the number \(R\) of the guess that finds it determine each other, so \(F_N = H(R \mid C) \le H(R)\), which is the right side. For the left side the guesses are in decreasing order of probability, and a decreasing distribution is a mixture of uniform ones: with \(U_i\) uniform on the first \(i\) guesses and weights \(w_i = i\,(p_{(i)} - p_{(i+1)}) \ge 0\), the mixture \(\sum_i w_i U_i\) gives the \(j\)-th guess the probability \(\sum_{i \ge j} (p_{(i)} - p_{(i+1)}) = p_{(j)}\). Entropy is concave, so \(H(p(\cdot \mid c)) \ge \sum_i w_i \log i\), and the average over \(c\) turns each \(p_{(i)}\) into \(r_i\). ∎Equation 17 of the paper of 1951.

Example 1.2.4 (the bounds of 1951). The setup: one subject, one book, 100 passages of 15 letters and a further test with 100 letters of context; for each \(N\) the frequencies of the guess numbers are put into the two sides of (1.3). For a human subject the lower bound is a heuristic, since its proof needs the ideal predictor.

\(N\) upper bound lower bound from frequency tables
1 4.03 3.19 4.03
2 3.42 2.50 3.32
3 3.0 2.1 3.1
4 2.6 1.7  
5 2.7 1.7  
8 1.8 1.0  
10 2.1 1.0  
15 2.1 1.2  
100 1.3 0.6  
Bounds on \(F_N\), the entropy of the next letter given the \(N - 1\) before it, in bits per letter on a 27-symbol alphabet, from one subject's guesses on passages of one book (Shannon 1951, page 64; nine of his sixteen columns; for \(N = 1\) and 2 he calculated the guess frequencies from the tables of letters and pairs), beside the values computed from tables of letter frequencies.

With 100 letters of context the entropy lies between 0.6 and 1.3 bits per letter, which the paper sums up as follows.

Since then a new method has been found for estimating these quantities, which is more sensitive and takes account of long range statistics, influences extending over phrases, sentences, etc. This method is based on a study of the predictability of English; how well can the next letter of a text be predicted when the preceding N letters are known. The results of some experiments in prediction will be given, and a theoretical analysis of some of the properties of ideal prediction. By combining the experimental and theoretical results it is possible to estimate upper and lower bounds for the entropy and redundancy. From this analysis it appears that, in ordinary literary English, the long range statistical effects (up to 100 letters) reduce the entropy to something of the order of one bit per letter, with a corresponding redundancy of roughly 75%. The redundancy may be still higher when structure extending over paragraphs, chapters, etc. is included. However, as the lengths involved are increased, the parameters in question become more erratic and uncertain, and they depend more critically on the type of text involved.

—Shannon (1951), pages 50 and 51.

The rows wobble, 1.8 at \(N = 8\) and 2.1 at \(N = 15\): this is one subject and a hundred samples, smoothed by hand.

A machine can be put through the same test, and for a machine the loss is known. On a thousand held-out stories gpt-pico's first guess is right 61.5% of the time and the pair table's 25.1%. Put into (1.3), gpt-pico's guess numbers give 0.34 and 0.63 bits per character, and its own loss on those stories, 0.54, lies between them; the table's give 0.96 and 1.39, around its loss of 1.31. ◁

The experiment has been repeated for seventy years with better subjects. In 1978 twelve people bet on the next letter instead of guessing it; the growth of a gambler's capital gives an estimate that converges where the bounds of (1.3) do not, and it came to between 1.25 and 1.34 bits per letter.T. M. Cover and R. C. King, "A Convergent Gambling Estimate of the Entropy of English", IEEE Transactions on Information Theory 24(4), pages 413 to 421, 1978. Each subject spent about five hours on 75 characters of the same book. In 1992 the subject became a model: a table of word triples counted on 583 million words and scored on a corpus it had never seen gave 1.75 bits per character, over 95 printable characters and so not comparable with the figures for 27 letters.P. F. Brown, V. J. Della Pietra, R. L. Mercer, S. A. Della Pietra and J. C. Lai, "An Estimate of an Upper Bound for the Entropy of English", Computational Linguistics 18(1), pages 31 to 40, 1992. In 2019 the subject was a transformer of 1.5 billion parameters, the GPT-2 of Section 1.5, which reached 0.98 bits per character on Wikipedia text reduced to the same 27 symbols. At the end of 2025 four language models played, on web text and out to ten thousand characters of context, for authors who put the whole history in one sentence.

Entropy helps us to describe phenomena ranging from steam engines to black holes. Entropy also quantifies our intuitive notion of information, and this measure is unique in satisfying plausible constraints [30]. Shannon used language as an accessible example of these ideas, and played a “guessing game” with human subjects to estimate the entropy of written English [31]. The results of this experiment gave bounds on the conditional entropy for a single character in the text given knowledge of the previous \(N\) characters; tighter bounds can be obtained by asking subjects not just to guess but to bet [6]. Decades later, performance at the guessing game is essentially the objective function for training large language models.

—C. Scheibner, L. M. Smith and W. Bialek, "Large language models and the entropy of English" (2025). arXiv 2512.24969, the opening paragraph.

The entropy of English is therefore not a constant that anyone knows. Every figure that a model gives is an upper bound, for one corpus, one alphabet and one length of context, and the bounds are still coming down.

Prediction is compression

The subject of 1951 turned a sentence into a row of numbers, most of them ones, and a twin could turn it back. That is half of a compressor: a good predictor makes the text boring, and a boring sequence is short to write down. Arithmetic coding is the other half. It turns the predictor's probabilities directly into bits, with no detour through guess numbers, and it wastes fewer than two bits on the whole text.

One sentence can be coded under four predictors: one that holds every token equally likely, one that knows how frequent each token is, a table of pairs, and gpt-pico.The table of pairs in the page keeps only the pairs seen at least 100 times and scores 1.311 bits per byte on the held-out stories; the full table of Example 1.3.3 scores 1.282. Think of the interval from 0 to 1 as holding every possible text. Each token narrows the current interval to the slice that the predictor had reserved for it, wide if the token was expected and thin if not; that slice becomes the interval subdivided for the next token. The code is a binary fraction that names the final interval, so each token costs about as many bits as its surprisal. Coded with gpt-pico, the example sentence takes 28 bits, 1.4 more than its 26.6 bits of surprisal, against 696 as text, and two tokens, the word "black" and the first half of the cat's name, cost 16 of them.

An arithmetic coder on "Once upon a time, there was a little...", with gpt-pico as the predictor.
Each row: the interval so far, and in it the slice (#) gpt-pico kept for the token that came.

token    0                                      1    bits    sum  settled
Once     ........############################....    0.51    0.5        0
upon     .#######################################    0.05    0.6        0
a        ########################################    0.00    0.6        0
time     ########################################    0.00    0.6        0
,        #####################################...    0.12    0.7        0
there    ........################################    0.37    1.0        0
was      #######################################.    0.03    1.1        0
a        #######################################.    0.03    1.1        0
...      
adow     .....##################################.    0.24   26.6       23
.        #######################################.    0.07   26.6       23

code    28 bits, 1.4 more than the 26.6 bits of surprisal; as text the sentence is 696 bits
others  the same sentence: uniform 264.0 bits, frequencies 178.9, a table of pairs 75.0
corpus  bits per byte on 22.1 MB of held-out text: gzip 2.40, xz 1.51, pairs 1.31, gpt-pico 0.55
(the live figure needs JavaScript; a typed sentence loads 2.8 MB: a table, gpt-pico's weights)
An arithmetic coder on one sentence. Each strip is the interval so far, cut into the slices a predictor gives the tokens that could come next. Green: the slice of the token that came, deeper the more bits it cost; stretched, it is the next strip. The columns give each token's surprisal, the running sum and the code bits it settles; the last strip shows the stretch that the code word names. Switch the predictor, pick or type a sentence, scrub or play the tokens. The arithmetic is exact; the corpus figures are measured on 22.1 MB of held-out stories.
Algorithm 1.3.1  ARITHMETIC-CODE
                 (in exact arithmetic)

Input   a model q over an ordered vocabulary;
        a string x_1 ... x_n with q(x_1 ... x_n) > 0.
Output  a binary word of fewer than -log2 q(x_1 ... x_n) + 2 bits.

 1. low <- 0,  width <- 1
 2. for t = 1, ..., n:
 3.    c <- sum of q(v | x_<t) over the symbols v before x_t
 4.    low <- low + width * c
 5.    width <- width * q(x_t | x_<t)
 6. L <- ceil(-log2 width) + 1
 7. z <- ceil(low * 2^L)
 8. return z, written with L binary digits

Theorem 1.3.2 (the code-length identity). Let \(q\) be a language model and \(n\) a length. (i) Algorithm 1.3.1 sends the strings of \(\mathcal V^n\) that \(q\) allows to binary words, none a prefix of another, of length

\[\ell_q(x_{1:n}) \;<\; -\log q(x_{1:n}) + 2 \;=\; \sum_{t=1}^{n} -\log q(x_t \mid x_{<t}) \;+\; 2 \quad \text{bits}, \tag{1.4}\]

and a decoder needs nothing but the same \(q\). (ii) Conversely, if \(\ell\) gives the lengths of any prefix code for \(\mathcal V^n\), then \(q(x) = 2^{-\ell(x)} / \sum_y 2^{-\ell(y)}\) is a distribution with \(-\log q(x) \le \ell(x)\) for every \(x\). (iii) Hence, for a source \(p\) on \(\mathcal V^n\), every prefix code has expected length at least \(H(p)\), and the code of a model \(q\) has expected length less than \(H(p) + D(p \,\|\, q) + 2\).

Proof. (i) Lines 1 to 5 give each string an interval \([\mathrm{low}, \mathrm{low} + w)\) whose width is \(w = q(x_{1:n})\), by (1.1). The intervals of two different strings are disjoint: at the first position where the strings differ they are sent into disjoint slices of a common interval. With \(L = \lceil -\log w \rceil + 1\) we have \(2 \cdot 2^{-L} \le w\), so the dyadic interval \([z\, 2^{-L}, (z + 1)\, 2^{-L})\) of line 7, which begins less than \(2^{-L}\) above \(\mathrm{low}\), lies inside the string's interval, and the \(L\) digits of \(z\) are its name. Dyadic intervals inside disjoint intervals are disjoint, so no name is a prefix of another. The decoder runs the same subdivision and takes, at each step, the one slice that contains the named interval. Finally \(L < -\log w + 2\). (ii) A code word of length \(\ell\) names a dyadic interval of width \(2^{-\ell}\), and the intervals of a prefix code are disjoint parts of \([0, 1)\), so \(\sum_y 2^{-\ell(y)} \le 1\) and \(-\log q(x) = \ell(x) + \log \sum_y 2^{-\ell(y)} \le \ell(x)\). (iii) By (ii) and Proposition 1.1.4, \(\mathbb E_p\, \ell \ge H(p, q) \ge H(p)\); by (i), \(\mathbb E_p\, \ell_q < H(p, q) + 2 = H(p) + D(p \,\|\, q) + 2\). ∎The inequality used in (ii) is Kraft's, and it holds for every uniquely decodable code (the Kraft–McMillan inequality), so (ii) and (iii) hold at that generality. The decoder must also know where to stop: the length is sent first, or the vocabulary has an end symbol.

Read (1.4) from left to right and a model is a compressor. The size of the compressed text is the sum of the surprisals of its tokens, so the loss (1.2), times the number of tokens, divided by \(\ln 2\), is the size in bits of the compressed training set. Maximum likelihood is a search for the smallest file, or in the words of a paper of 2023, "current language model training protocols use a maximum-compression objective".G. Delétang and others, "Language Modeling Is Compression", ICLR 2024 (arXiv 2309.10668), under "Likelihood Maximization".

Read (ii) and a compressor is a model: gzip gives every file a probability and every prefix a guess about what follows. It is a language model, a poor one. And (iii) says there is nothing else to win: the expected length cannot go below the entropy of the source, and what a model pays above it is its relative entropy from the source, the very quantity that training minimizes.

The decoder holds the same model as the encoder, so the model must exist before the text does. The authors of the bound of 1992 put the rule that every evaluation since has depended on:

We emphasize that for this paradigm to be reasonable, the language model M must be constructed without knowledge of the test sample. Without this proscription, one might, for example, construct a model that assigns probability one to the test sample and zero to any other character string of the same length. Even quite subtle use of knowledge of the test sample can have a profound effect on the cross-entropy. For example, the cross-entropy would be noticeably lower had we restricted ourselves to characters that appear in the test sample rather than to all printable ASCII characters, and would be lower still had we used the actual vocabulary of the test sample. But these values could not be trumpeted as upper bounds to the entropy of English since Equation (13) would no longer be valid.

—P. F. Brown and others, "An Estimate of an Upper Bound for the Entropy of English" (1992). Section 2.2, page 33.

Example 1.3.3 (compressors, tables and networks on the same text). The setup: the held-out text of the two corpora of this series, 46.3 MB of FineWeb-EDU and 22.1 MB of TinyStories, coded in three ways: by general-purpose compressors, which start knowing nothing and learn from the file as they go; by tables of token n-gram counts made beforehand on 200 million training tokens of the same corpus (Section 1.4); and by networks, charged their surprisal as (1.4) allows. Everything is in bits per byte of the original text.

predictor what it has seen FineWeb-EDU TinyStories
gzip -9 the file itself 3.020 2.401
bzip2 -9 the file itself 2.395 1.609
zstd -22 the file itself 2.204 1.519
xz -9e the file itself 2.176 1.510
token frequencies 200M tokens of the corpus 2.400 2.126
2-gram table the same 1.764 1.282
3-gram table the same 1.607 1.008
4-gram table the same 1.577 0.897
GPT-2 small, 124M parameters other web pages 1.007 0.856
GPT-2 medium, 355M other web pages 0.925 0.759
GPT-2 large, 774M other web pages 0.881 0.707
GPT-2 XL, 1,558M other web pages 0.854 0.679
TinyStories 33M TinyStories 2.875 0.436
gpt-pico, 2.2M TinyStories 3.032 0.541
gpt-pp, 135M FineWeb-EDU 0.932 0.864
Bits per byte on held-out text; lower is better. Compressors: the held-out documents joined into one stream. Tables: interpolated Kneser–Ney models over tokens, scored on the same streams (Example 1.4.5). Networks: the first 1,000 held-out documents of FineWeb-EDU (4.58 MB) and the first 2,000 held-out stories (1.62 MB), in windows of at most 1,024 tokens (512 for TinyStories 33M, 256 for gpt-pico). The standard error over documents is at most 0.007 on web pages and 0.004 on stories, except about 0.03 for the two models trained on stories alone when they read web pages. TinyStories 33M was trained on an earlier version of the stories, which holds 695 of these 2,000; it scores no better on them. The row of gpt-pp is the one in bold.

There are three bands. The compressors, which must learn English from the file they are compressing, stay above two bits per byte on web pages. A table made beforehand does better as soon as it knows pairs: the 2-gram table at 1.764 is ahead of xz at 2.176, though its 15 million entries are a dictionary that the decoder must already own. The networks are a band below, and inside the band the loss falls with size: 1.007, 0.925, 0.881 and 0.854 for the four sizes of GPT-2, a family trained on other web pages and not on this corpus.GPT-2's training text is 40 GB of pages linked from Reddit, with Wikipedia removed. Pages of FineWeb-EDU that were on the web by 2017 may be among them; TinyStories, written in 2023, cannot be. Trained on this corpus, gpt-pp, which has the shape of the smallest of them, stands at 0.932, between GPT-2 small and GPT-2 medium. Off its own kind of text the order turns: on stories it is a little behind GPT-2 small, 0.864 against 0.856, and on the Wikipedia articles of WikiText-103 it is at 1.130 against 1.072. It is ahead where the text is like its training text and behind where it is not. gpt-pico, with two million parameters, pays 0.541 on stories (by document; the figure above prints the trainer's reading on its own windows, 0.55): less than the 4-gram table, and less than GPT-2 XL, which has 700 times as many. ◁

Remark 1.3.4 (a code is made for a source). The two models trained on stories alone are part (iii) of the theorem at work. On stories they are the best compressors in the table. On web pages gpt-pico pays 3.032 bits per byte, as much as gzip, which has never seen a web page, and TinyStories 33M pays 2.875. On a source \(p\) a model \(q\) pays \(H(p) + D(p \,\|\, q)\), and a model fitted to stories is far, in relative entropy, from the web.

Remark 1.3.5 (who pays for the weights). The decoder needs the model, and a model has a size. The two-part code sends the weights and then the text coded with them. The prequential code sends only the training program: both ends train the same model on what has been sent so far, and each new batch is coded by the model as it then stands, so that in a single pass over the data the code length is the area under the training-loss curve. Either way a model compresses only if what it saves on the text exceeds the cost of describing it.

The difference is not small. On \(10^9\) bytes of English Wikipedia a network of 70 billion parameters reaches 8.3% of the original size, 0.66 bits per byte, where gzip reaches 32.3%; charged two bytes per parameter it comes to 14,008%, and a transformer of 3.2 million parameters, at 17.0% without its weights and 17.7% with them, is the better compressor.Delétang et al., Table 1 of the second version: Chinchilla 70B and a transformer trained on enwik8, both on enwik9 in chunks of 2,048 bytes; gzip on the unchunked file. The public benchmark for that file ranks compressors together with their programs. On 13 November 2026 its leading entry stood at 89,346,392 bytes, 0.715 bits per byte, or 93,388,956 with the compressor counted, 0.747, and the benchmark's table classes its first ten entries as transformer-based.M. Mahoney, Large Text Compression Benchmark, as updated on 13 November 2026: the entry cmix-neif 2.0. The page lists both totals without saying which of them its ranking uses.

Counting

Counting came first. In a lecture of January 1913, Markov took the opening of Pushkin's Eugene Onegin as "20,000 connected trials, which are either a vowel or a consonant", and counted. A letter is a vowel 43.2% of the time; after a vowel, 12.8%; after a consonant, 66.3%. "As we can see, the probability of a letter being a vowel changes considerably depending upon which letter – vowel or consonant – precedes it."A. A. Markov, "An Example of Statistical Investigation of the Text Eugene Onegin Concerning the Connection of Samples in Chains", lecture of 23 January 1913, translated by G. Custance and D. Link, Science in Context 19(4), pages 591 to 600, 2006. Of the 20,000 letters 8,638 are vowels; 1,104 vowels follow a vowel, and 7,534 follow one of the 11,362 consonants. His aim was a theorem, that the law of large numbers survives dependence, and the poem was his example of a chain.

Definition 1.4.1 (n-gram model). An n-gram model of order \(N\) lets the next symbol depend on the \(N - 1\) before it and on nothing else, \(q(x_t \mid x_{<t}) = q(x_t \mid x_{t-N+1:t-1})\). Below, \(h\) is a history of \(N - 1\) symbols, \(w\) the symbol after it, and \(c(\cdot)\) a count in the training text.

Proposition 1.4.2 (maximum likelihood is relative frequency). Among n-gram models of order \(N\), the likelihood of the training text is largest for \(\hat q(w \mid h) = c(hw) / c(h)\).

Proof. The log-likelihood is \(\sum_h \sum_w c(hw) \log q(w \mid h) = -\sum_h c(h)\, H(\hat q_h, q_h)\), where \(\hat q_h\) and \(q_h\) are the two distributions of the symbol after \(h\). By Proposition 1.1.4 each cross-entropy is smallest at \(q_h = \hat q_h\). ∎

This is the table of the opening figure and the book of 1948, and its defect is as plain as its merit. A sequence \(hw\) that never occurred in the training text gets probability zero, and one such sequence in a test text makes the loss infinite (Remark 1.1.6). Such sequences are the rule, because there are \(\lvert \mathcal V \rvert^{N}\) possible n-grams and a corpus of fixed size. Ten consecutive words from a vocabulary of 100,000 make a table of \(10^{50}\) entries.

From the 1970s the speech recognition group at IBM made the n-gram model a working part, the prior over word sequences against which a recognizer weighs what it hears, and gave it its measure, the perplexity of Definition 1.1.7. The obstacle they met was sparse data. The answers, collectively called smoothing, are ways of moving probability from what was seen to what was not. Interpolation mixes the orders,

\[q(w \mid h) \;=\; \lambda_h\, \frac{c(hw)}{c(h)} \;+\; (1 - \lambda_h)\; q(w \mid h^{\prime}),\]

where \(h^{\prime}\) is \(h\) without its oldest symbol and the weights \(\lambda_h\) are fitted on held-out text. Back-off trusts a count where there is one, after discounting it, and falls back on the shorter history where there is none: "The model offers, via a nonlinear recursive procedure, a computation and space efficient solution to the problem of estimating probabilities from sparse data."S. M. Katz, "Estimation of Probabilities from Sparse Data for the Language Model Component of a Speech Recognizer", IEEE Transactions on Acoustics, Speech, and Signal Processing 35(3), pages 400 to 401, 1987; the abstract. The third answer changed the question put to the shorter history.

Usually the normal probability distribution of the coarser model is taken for backing-off. In that case the information that the event is not covered by the detailed model is lost. All events are taken into consideration for the estimation of the backing-off distribution, also those already covered by the M-gram models. This results in a bias towards words heavily conditioned on the immediate predecessor words. Consider for example the word dollars which is a very frequent word in the Wall-Street-Journal corpus but occurs almost exclusively after numbers and some country names. The latter fact makes it very unlikely that dollars will occur after some word x if the bigram (x, dollars) has not been observed. On the other hand, the smoothed probability estimate will be relatively high, if the unigram probability p(dollars) is taken for backing-off. This suggests to use some backing-off distribution different from the normal probability distribution.

—R. Kneser and H. Ney, "Improved backing-off for M-gram language modeling" (1995). ICASSP 1995, pages 181 to 184; Section 1. The text is read from a scan.

Definition 1.4.3 (interpolated Kneser–Ney, for pairs). Fix a discount \(0 < \delta \le 1\). Let \(N_{1+}(v\, \cdot)\) be the number of distinct symbols seen after \(v\), and \(N_{1+}(\cdot\, w)\) the number of distinct symbols seen before \(w\). The model is

\[\begin{aligned} q_{\mathrm{KN}}(w \mid v) &= \frac{\max\{c(vw) - \delta,\, 0\}}{c(v)} \;+\; \frac{\delta\, N_{1+}(v\, \cdot)}{c(v)}\; \beta(w), \\ \beta(w) &= \frac{N_{1+}(\cdot\, w)}{\sum_{u} N_{1+}(\cdot\, u)} . \end{aligned} \tag{1.5}\]

The first term gives up exactly the mass \(\delta N_{1+}(v\, \cdot) / c(v)\) that the second hands out. Higher orders repeat the construction, each passing its discounted mass to the order below.

Proposition 1.4.4 (the lower order is forced). Let the second term of (1.5) carry an arbitrary distribution \(\beta\). The model reproduces the symbol frequencies of the training text, \(\sum_v c(v)\, q_{\mathrm{KN}}(w \mid v) = c(w)\) for every \(w\), if and only if \(\beta\) is the one in (1.5).

Proof. Count pairs around the end of the text, so that \(\sum_v c(vw) = c(w)\). Every pair that was seen has \(c(vw) \ge 1 \ge \delta\), hence

\[\sum_v c(v)\, q_{\mathrm{KN}}(w \mid v) \;=\; c(w) \;-\; \delta\, N_{1+}(\cdot\, w) \;+\; \delta\, \beta(w) \sum_v N_{1+}(v\, \cdot),\]

and \(\sum_v N_{1+}(v\, \cdot) = \sum_u N_{1+}(\cdot\, u)\), both being the number of distinct pairs. The right side equals \(c(w)\) exactly when \(\beta(w) = N_{1+}(\cdot\, w) / \sum_u N_{1+}(\cdot\, u)\). ∎

So the right thing to ask of a word, when its history fails, is not how often it occurs but in how many different contexts it has been seen. The word of the example is frequent and has few contexts, and its probability after an unfamiliar word is small, as it should be. A systematic comparison in 1999, by cross-entropy on held-out text, put a variant of this method ahead of all the others.

We survey the most widely-used algorithms for smoothing models for language n-gram modeling. We then present an extensive empirical comparison of several of these smoothing techniques, including those described by Jelinek and Mercer (1980); Katz (1987); Bell, Cleary and Witten (1990); Ney, Essen and Kneser (1994), and Kneser and Ney (1995). We investigate how factors such as training data size, training corpus (e.g. Brown vs. Wall Street Journal), count cutoffs, and n-gram order (bigram vs. trigram) affect the relative performance of these methods, which is measured through the cross-entropy of test data. We find that these factors can significantly affect the relative performance of models, with the most significant factor being training data size. Since no previous comparisons have examined these factors systematically, this is the first thorough characterization of the relative performance of various algorithms. In addition, we introduce methodologies for analyzing smoothing algorithm efficacy in detail, and using these techniques we motivate a novel variation of Kneser–Ney smoothing that consistently outperforms all other algorithms evaluated. Finally, results showing that improved language model smoothing leads to improved speech recognition performance are presented.

—S. F. Chen and J. Goodman, "An empirical study of smoothing techniques for language modeling" (1999).Computer Speech and Language 13, pages 359 to 394, 1999; the abstract. The variant has three discounts, for counts of one, of two, and of three or more.

Example 1.4.5 (counting on two corpora). The setup: interpolated Kneser–Ney models of orders 1 to 4 over tokens, counted on the first 200 million training tokens of FineWeb-EDU (vocabulary 32,768) and of TinyStories (vocabulary 4,096), with one discount for each order estimated from the counts and nothing tuned on held-out text; scored on the held-out text of each corpus, 10.0 and 5.5 million tokens.

corpus order n-grams in the table seen once held-out n-grams not in the table bits per byte
FineWeb-EDU 1 32,707 12 0.00% 2.400
  2 15,097,774 8,407,269 4.98% 1.764
  3 76,558,188 58,045,771 31.78% 1.607
  4 141,394,600 124,542,435 65.77% 1.577
TinyStories 1 3,996 19 0.00% 2.126
  2 1,127,841 368,819 0.22% 1.282
  3 9,832,757 5,150,883 2.72% 1.008
  4 32,591,218 21,614,465 11.06% 0.897
Tables of token n-grams on the two corpora of this series: the number of distinct n-grams counted in 200 million tokens, how many of them occurred once, the share of the n-grams of held-out text that the table does not hold, and the loss of the smoothed model on held-out text.

On web pages counting stalls at the fourth token. The second token of context was worth 0.16 bits per byte and the third is worth 0.03, for which the table nearly doubles, to 141 million entries, 88% of them seen once. Two thirds of the four-token sequences of new text had never occurred in 200 million tokens, and for those the model can only back off. The cause is arithmetic: there are \(32{,}768^4 \approx 1.2 \times 10^{18}\) possible entries. Inside its own sample the table looks far better than it is, since there the fourth token costs 2.34 bits and on held-out text 7.30: a table memorizes.

Stories are a narrower world, and there counting goes further: 0.897 bits per byte at order 4, with one four-token sequence in nine unseen. It is still far from a network. The table has 32,591,218 entries of four tokens; gpt-pico has 2,228,224 parameters, and after one pass over 540 million tokens of stories (the table was counted on 200 million of them) it scores 0.541 on held-out stories (Example 1.3.3). On web pages the same tables are set against gpt-pp, token for token, in Section 1.6. ◁

In the code. A model of the table is a chain of levels, one for each length of context, and every level is the same few lines: the counts of its sequences and of their contexts, and (1.5) with the estimate of the level below in the place of \(\beta\). It is Level in gpt-pp's corpus_stats.py; a second copy with the same recursion, in counted_models.py, serves the figures of this page.

class Level:
    """One level of an interpolated n-gram model: counts of k-grams and of their contexts of k - 1 tokens."""

    def __init__(self, grams: np.ndarray, counts: np.ndarray, bits: int) -> None:
        self.grams, self.counts, self.bits = grams, counts, bits
        ones, twos = int((counts == 1).sum()), int((counts == 2).sum())
        self.discount = ones / (ones + 2 * twos) if ones and twos else 0.5  # Ney, Essen and Kneser (1994)
        contexts = grams >> np.uint64(bits)  # sorted, because the k-grams are
        first = np.flatnonzero(np.concatenate(([True], contexts[1:] != contexts[:-1])))
        self.contexts = contexts[first]
        self.totals = np.add.reduceat(counts, first)  # how often each context was continued
        self.types = np.diff(first, append=len(grams))  # by how many different tokens

    def probability(self, grams: np.ndarray, lower: np.ndarray) -> np.ndarray:
        ...
        count = lookup(self.grams, self.counts, grams)
        contexts = grams >> np.uint64(self.bits)
        total = lookup(self.contexts, self.totals, contexts)
        types = lookup(self.contexts, self.types, contexts)
        seen = total > 0
        p = lower.copy()
        p[seen] = (np.maximum(count[seen] - self.discount, 0) + self.discount * types[seen] * lower[seen]) / total[seen]
        return p

Sequences are single integers, the tokens side by side at bits bits each, so grams >> bits is a context and lookup returns a stored count or zero; count, total and types are then \(c(vw)\), \(c(v)\) and \(N_{1+}(v\, \cdot)\), self.discount is \(\delta\), lower is \(\beta(w)\), and the assignment to p[seen] is (1.5), for a context of any length. The top level of a model holds the counts of its n-grams and each level under it continuation counts, the number of distinct tokens seen before a sequence, which for single tokens is the \(N_{1+}(\cdot\, w)\) of the definition. Where the definition fixes \(\delta\), the code estimates one for each level from that level's counts, \(n_1 / (n_1 + 2 n_2)\) with \(n_1\) and \(n_2\) the numbers of entries counted once and twice (0.5 if either is zero), and where a context was never seen, so that (1.5) would divide by zero, it returns lower unchanged. Under the single tokens stands one level more, the uniform distribution over the vocabulary: the tokens that the sample never shows (on web pages the table of order 1 holds 32,707 of the 32,768) keep a probability above zero, and the code's \(\beta\) is the one of (1.5) after a discount of its own.

Quirk. These tables are not scored as the networks beside them are. A table is scored by stream: every held-out token from the fourth on, the beginning-of-text tokens among them and with contexts that run across the boundary between two documents, and its bits per token are divided by the bytes per token of the whole held-out text. The networks of the same comparisons are scored document by document, never on that token, against each document's own bytes (Section 7). The size of the difference was not measured.

Quirk. Section 8 puts two kinds of table in one figure. The levels that gpt-pico crosses as it trains, 2.126 and 1.282 bits per byte, are the held-out losses of the first two models of stories above; the predictors from which its distance is measured are tables of plain relative frequencies of the whole training text, without smoothing.

Every method of this section generalizes by the same device, a shorter context. None of them can use the fact that two different words are alike.

Learned representations, from the neural language model to GPT-3

Three changes carried the next-symbol predictor from tables of counts to GPT-3: in 2003, the table became a network over learned vectors; in 2014, attention let a network read back over its input; and in 2018, the writing half of the transformer was trained as a language model.

year  paper          kind of predictor      its idea, in a few words
1913  Markov         table of counts        chains counted on a poem
1948  Shannon        table of counts        entropy; approximations to English
1951  Shannon        table of counts        the guessing game: 0.6 to 1.3 bits a letter
1977  Jelinek        table of counts        perplexity
1978  Cover          table of counts        betting on the next letter: 1.25 to 1.34
1987  Katz           table of counts        back-off to a shorter history
1992  Brown          table of counts        a model as the subject: 1.75 bits
1995  Kneser         table of counts        back off to counts of contexts
1999  Chen           table of counts        smoothing methods compared
2003  Bengio         learned vectors        a learned vector for every word
2010  Mikolov        learned vectors        a recurrent state, no window
2013  word2vec       learned vectors        word vectors, cheaply
2014  Sutskever      learned vectors        sequence to sequence
2014  Bahdanau       attention              attention
2017  transformer    attention              attention alone, in parallel
2018  GPT            pre-training           pre-train, then fine-tune
2018  BERT           attention              masked tokens, both sides
2019  GPT-2          pre-training           a task is more conditioning: 0.98 bits
2020  scaling laws   pre-training           loss as a power law
2020  GPT-3          pre-training           learning in the context
GPT-3 builds on 13 of the others: GPT-2, GPT, the transformer, attention, and so back to 1913
(the live figure needs JavaScript; it loads no data)
The papers behind this section in order of time, from a count of vowels in 1913 to GPT-3; the row is the kind of predictor. Hairlines join a paper to what it built on. Green: the lineage of the picked paper, everything it built on. Under the marks, the estimates of the entropy of English that some of the papers give, in bits per character; their alphabets and texts differ, so the strip is one unit and not one benchmark. Point at a mark, or walk them with the arrow keys. Numbers are the papers' own.

The first step off the table was taken in 2003, by authors who named the obstacle before they removed it.

A goal of statistical language modeling is to learn the joint probability function of sequences of words in a language. This is intrinsically difficult because of the curse of dimensionality: a word sequence on which the model will be tested is likely to be different from all the word sequences seen during training. Traditional but very successful approaches based on n-grams obtain generalization by concatenating very short overlapping sequences seen in the training set. We propose to fight the curse of dimensionality by learning a distributed representation for words which allows each training sentence to inform the model about an exponential number of semantically neighboring sentences. The model learns simultaneously (1) a distributed representation for each word along with (2) the probability function for word sequences, expressed in terms of these representations. Generalization is obtained because a sequence of words that has never been seen before gets high probability if it is made of words that are similar (in the sense of having a nearby representation) to words forming an already seen sentence. Training such large models (with millions of parameters) within a reasonable time is itself a significant challenge. We report on experiments using neural networks for the probability function, showing on two text corpora that the proposed approach significantly improves on state-of-the-art n-gram models, and that the proposed approach allows to take advantage of longer contexts.

When building statistical models of natural language, one considerably reduces the difficulty of this modeling problem by taking advantage of word order, and the fact that temporally closer words in the word sequence are statistically more dependent. Thus, n-gram models construct tables of conditional probabilities for the next word, for each one of a large number of contexts, i.e. combinations of the last n − 1 words:

\[\hat P(w_t \mid w_1^{t-1}) \approx \hat P(w_t \mid w_{t-n+1}^{t-1}).\]

We only consider those combinations of successive words that actually occur in the training corpus, or that occur frequently enough. What happens when a new combination of n words appears that was not seen in the training corpus? We do not want to assign zero probability to such cases, because such new combinations are likely to occur, and they will occur even more frequently for larger context sizes. A simple answer is to look at the probability predicted using a smaller context size, as done in back-off trigram models (Katz, 1987) or in smoothed (or interpolated) trigram models (Jelinek and Mercer, 1980). So, in such models, how is generalization basically obtained from sequences of words seen in the training corpus to new sequences of words? A way to understand how this happens is to think about a generative model corresponding to these interpolated or back-off n-gram models. Essentially, a new sequence of words is generated by “gluing” very short and overlapping pieces of length 1, 2 … or up to n words that have been seen frequently in the training data. The rules for obtaining the probability of the next piece are implicit in the particulars of the back-off or interpolated n-gram algorithm. Typically researchers have used n = 3, i.e. trigrams, and obtained state-of-the-art results, but see Goodman (2001) for how combining many tricks can yield to substantial improvements. Obviously there is much more information in the sequence that immediately precedes the word to predict than just the identity of the previous couple of words. There are at least two characteristics in this approach which beg to be improved upon, and that we will focus on in this paper. First, it is not taking into account contexts farther than 1 or 2 words, second it is not taking into account the “similarity” between words. For example, having seen the sentence “The cat is walking in the bedroom” in the training corpus should help us generalize to make the sentence “A dog was running in a room” almost as likely, simply because “dog” and “cat” (resp. “the” and “a”, “room” and “bedroom”, etc...) have similar semantic and grammatical roles.

—Y. Bengio, R. Ducharme, P. Vincent and C. Jauvin, "A Neural Probabilistic Language Model" (2003). Journal of Machine Learning Research 3, pages 1137 to 1155; the abstract, and from Section 1 the paragraph on tables of word sequences, pages 1138 and 1139, a footnote mark omitted. The example of ten words from a vocabulary of 100,000 in Section 1.4 is theirs.

Definition 1.5.1 (the neural probabilistic language model). Give each word \(w\) of the vocabulary a vector \(C(w) \in \mathbb R^m\), a row of a matrix \(C\) that is learned. With \(x\) the concatenation of the vectors of the \(N - 1\) previous words,

\[q_\theta(w_t = i \mid w_{t-N+1:t-1}) \;=\; \frac{e^{y_i}}{\sum_j e^{y_j}}, \qquad y \;=\; b + W x + U \tanh(d + H x), \tag{1.6}\]

and all of \(\theta = (C, b, W, U, d, H)\) is trained by gradient ascent on the log-likelihood.

The number of parameters now grows in proportion to the vocabulary and to the length of the window, not as a power of the vocabulary. Generalization comes from the smoothness of (1.6) in the vectors: a sentence never seen is likely if its words lie close to the words of sentences that were seen. On a corpus of 1.2 million words the perplexity fell from 312 for the best table to 252 for the network mixed with a trigram, and the price was compute: on its second corpus, of news text, the paper reports a run of three weeks on 40 processors for five passes over the data.Tables 1 and 2 of the paper: the first corpus is the Brown corpus of 1,181,041 words, the second is news text from the Associated Press. Formula (1.6) is the model in the form in which it is usually stated. Every model since keeps the two ends of (1.6), a learned vector for each token going in and a softmax over the vocabulary coming out, and replaces what lies between.

The fixed window went next. A recurrent network, in 2010, read the text one word at a time and kept a state vector that could in principle summarize everything so far, and mixtures of such models halved the perplexity of the best tables.T. Mikolov, M. Karafiát, L. Burget, J. Černocký and S. Khudanpur, "Recurrent neural network based language model", Interspeech 2010; the abstract. In 2014 a pair of such networks translated English into French end to end, the second writing the translation as a language model conditioned on one vector produced by the first. The one vector was the weak point, "we conjecture that the use of a fixed-length vector is a bottleneck in improving the performance of this basic encoder–decoder architecture", and the remedy was to let the second network look back, with learned weights, at every position of the input. That is attention.I. Sutskever, O. Vinyals and Q. V. Le, "Sequence to Sequence Learning with Neural Networks", NeurIPS 2014 (arXiv 1409.3215): 34.8 BLEU, a score of overlap with reference translations, from English into French, against 33.3 for a system built on phrase tables. D. Bahdanau, K. Cho and Y. Bengio, "Neural Machine Translation by Jointly Learning to Align and Translate", ICLR 2015 (arXiv 1409.0473); the abstract.

In 2017 the transformer kept attention and dropped the recurrence, a network "based solely on attention mechanisms, dispensing with recurrence and convolutions entirely". Every position reads every earlier position in one step, and all positions are computed at once. One sentence of that paper is what makes it a language model. The attention of the half that writes is masked, and with the outputs shifted by one position the mask "ensures that the predictions for position i can depend only on the known outputs at positions less than i". One pass over a sequence of \(n\) tokens then yields all \(n\) conditionals of (1.1): \(n\) training signals for the price of one. Section 3 is about this mechanism.A. Vaswani and others, "Attention Is All You Need", NeurIPS 2017 (arXiv 1706.03762); the abstract and Section 3.1.

In 2018 that writing half alone, 12 layers of width 768 with a context of 512 tokens, was trained as a left-to-right language model on more than 7,000 unpublished books and then fine-tuned, with labels, on each of 12 tasks; it set the best published result on nine. This was GPT, from generative pre-training. Its objective is (1.2) with the context cut to a window, and near its end the paper ventures the guess that its successor made into a method (LSTMs are the recurrent networks then in use).

We’d like to better understand why language model pre-training of transformers is effective. A hypothesis is that the underlying generative model learns to perform many of the tasks we evaluate on in order to improve its language modeling capability and that the more structured attentional memory of the transformer assists in transfer compared to LSTMs. We designed a series of heuristic solutions that use the underlying generative model to perform tasks without supervised finetuning. We visualize the effectiveness of these heuristic solutions over the course of generative pre-training in Fig 2(right). We observe the performance of these heuristics is stable and steadily increases over training suggesting that generative pretraining supports the learning of a wide variety of task relevant functionality. We also observe the LSTM exhibits higher variance in its zero-shot performance suggesting that the inductive bias of the Transformer architecture assists in transfer.

—A. Radford, K. Narasimhan, T. Salimans and I. Sutskever, "Improving Language Understanding by Generative Pre-Training" (2018).OpenAI, 2018; Section 5, the paragraph headed "Zero-shot Behaviors". The figures given before the passage are from its Sections 3.1 and 4.1.

The same year BERT gave the factorization up: its model "randomly masks some of the tokens from the input, and the objective is to predict the original vocabulary id of the masked word based only on its context". Reading both sides helps a model that will be fine-tuned, but what results is no longer a distribution over strings, with no chain rule to sample from and no code length.J. Devlin, M.-W. Chang, K. Lee and K. Toutanova, "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding", NAACL 2019 (arXiv 1810.04805), Section 1. Masked models came back as generators through diffusion on discrete tokens, where one token at a time is exact and many at a time is not: Noises and flows, Section 9. For generation the left-to-right line prevailed, and its next two papers are the reason.

GPT-2: a task is more conditioning. The paper of 2019 is called "Language Models are Unsupervised Multitask Learners", and its abstract is careful about the verb.

Natural language processing tasks, such as question answering, machine translation, reading comprehension, and summarization, are typically approached with supervised learning on task-specific datasets. We demonstrate that language models begin to learn these tasks without any explicit supervision when trained on a new dataset of millions of webpages called WebText. When conditioned on a document plus questions, the answers generated by the language model reach 55 F1 on the CoQA dataset - matching or exceeding the performance of 3 out of 4 baseline systems without using the 127,000+ training examples. The capacity of the language model is essential to the success of zero-shot task transfer and increasing it improves performance in a log-linear fashion across tasks. Our largest model, GPT-2, is a 1.5B parameter Transformer that achieves state of the art results on 7 out of 8 tested language modeling datasets in a zero-shot setting but still underfits WebText. Samples from the model reflect these improvements and contain coherent paragraphs of text. These findings suggest a promising path towards building language processing systems which learn to perform tasks from their naturally occurring demonstrations.

—A. Radford, J. Wu, R. Child, D. Luan, D. Amodei and I. Sutskever, "Language Models are Unsupervised Multitask Learners" (2019).OpenAI, 2019; the abstract.

The argument stands on the paper's second page.

Language Models are Unsupervised Multitask LearnersOpenAI 2019A. Radford, J. Wu, R. Child, D. Luan, D. Amodei and I. Sutskever

2. Approach

Learning to perform a single task can be expressed in a probabilistic framework as estimating a conditional distribution p(output|input). Since a general system should be able to perform many different tasks, even for the same input, it should condition not only on the input but also on the task to be performed. That is, it should model p(output|input, task). This has been variously formalized in multitask and meta-learning settings. Task conditioning is often implemented at an architectural level, such as the task specific encoders and decoders in (Kaiser et al., 2017) or at an algorithmic level such as the inner and outer loop optimization framework of MAML (Finn et al., 2017). But as exemplified in McCann et al. (2018), language provides a flexible way to specify tasks, inputs, and outputs all as a sequence of symbols. For example, a translation training example can be written as the sequence (translate to french, english text, french text). Likewise, a reading comprehension training example can be written as (answer the question, document, question, answer). McCann et al. (2018) demonstrated it was possible to train a single model, the MQAN, to infer and perform many different tasks on examples with this type of format.

Language modeling is also able to, in principle, learn the tasks of McCann et al. (2018) without the need for explicit supervision of which symbols are the outputs to be predicted. Since the supervised objective is the the same as the unsupervised objective but only evaluated on a subset of the sequence, the global minimum of the unsupervised objective is also the global minimum of the supervised objective. In this slightly toy setting, the concerns with density estimation as a principled training objective discussed in (Sutskever et al., 2015) are side stepped. The problem instead becomes whether we are able to, in practice, optimize the unsupervised objective to convergence. Preliminary experiments confirmed that sufficiently large language models are able to perform multitask learning in this toy-ish setup but learning is much slower than in explicitly supervised approaches.

While it is a large step from the well-posed setup described above to the messiness of “language in the wild”, Weston (2016) argues, in the context of dialog, for the need to develop systems capable of learning from natural language directly and demonstrated a proof of concept – learning a QA task without a reward signal by using forward prediction of a teacher’s outputs. While dialog is an attractive approach, we worry it is overly restrictive. The internet contains a vast amount of information that is passively available without the need for interactive communication. Our speculation is that a language model with sufficient capacity will begin to learn to infer and perform the tasks demonstrated in natural language sequences in order to better predict them, regardless of their method of procurement. If a language model is able to do this it will be, in effect, performing unsupervised multitask learning. We test whether this is the case by analyzing the performance of language models in a zero-shot setting on a wide variety of tasks.

Section 2, its last three paragraphs, pages 2 and 3
A. Radford, J. Wu, R. Child, D. Luan, D. Amodei and I. Sutskever, "Language Models are Unsupervised Multitask Learners" (OpenAI 2019), Section 2, its last three paragraphs, pages 2 and 3. The doubled word in the second paragraph is the paper's.

Three things are said here, of three kinds. The first is a definition: a task is a conditional distribution, and task, input and output can all be written as text. The second is a theorem, and it is true. The third is a speculation, and is called one.

Proposition 1.5.2 (tasks as conditioning). Let the strings of a source \(p\) have the form (task, input, output), and let \(q^\star\) minimize the expected surprisal \(\mathbb E_p[-\log q(X)]\) over all distributions on strings. Then \(q^\star = p\), so that \(q^\star(\text{output} \mid \text{task}, \text{input}) = p(\text{output} \mid \text{task}, \text{input})\) for every task and input of positive probability. The supervised loss of a task is the part of the sum (1.2) that runs over the output positions of that task's strings, and \(q^\star\) minimizes every such part at once.

Proof. \(q^\star = p\) by Proposition 1.1.4, and equal distributions have equal conditionals. Each part of the sum is an expected cross-entropy between a conditional of \(p\) and the same conditional of \(q\), smallest when the two agree. ∎

The proposition is silent about a model of finite size trained on finite text, and about prompts unlike anything in that text; of practice the paper reports only preliminary experiments in a toy setting, where large models did learn several tasks from text alone, far more slowly than with supervision. What it showed is accordingly narrower than what it argued. Four models, of 117 million to 1,542 million parameters by the paper's count, were trained on 40 GB of text from eight million web pages and then tested with no further training of any kind.Table 3 and Sections 3.5 to 3.7 of the paper. The note released with the weights corrects the counts: "Note that our original parameter counts were wrong due to an error (in our previous blog posts and paper). Thus you may have seen small referred to as 117M and medium referred to as 345M." The released weights hold 124, 355, 774 and 1,558 million parameters. As language models they were the best yet on seven of eight benchmarks they had never been trained on: on LAMBADA, which asks for the last word of a passage, the perplexity fell from a best published 99.8 to 8.63. As performers of tasks they were beginners. Asked questions about a passage, the largest model scored 55 F1 (a measure of word overlap with the reference answers), level with three of four trained baselines. Asked to summarize by appending TL;DR: to an article, it scored 21.40 on the average of the ROUGE measures of overlap with reference summaries, where three sentences picked at random from the article score 20.98. Asked to translate English into French, it scored 5 BLEU, where the recurrent networks of 2014 had reached 34.8. The abstract's verb was begin.

Example 1.5.3 (two of GPT-2's numbers, measured again). The setup: the four released GPT-2 models, scored as in Example 1.1.8 on the test set of WikiText-103 (60 Wikipedia articles) and on the 5,153 passages of LAMBADA; perplexity per word for the first and perplexity of the last word for the second, beside the figures of the paper's Table 3.

model WikiText-103, paper here, tokenization undone here, text as distributed LAMBADA, paper here
GPT-2 small 37.50 36.94 49.22 35.13 35.11
GPT-2 medium 26.37 26.65 34.56 15.60 17.15
GPT-2 large 22.05 22.52 29.17 10.87 12.01
GPT-2 XL 17.48 20.24 26.18 8.63 10.13
Perplexities of the released GPT-2 models: as published in 2019, and as measured for this series on the same benchmarks.

The smallest model reproduces both published figures, to 1.5% and to 0.1%. The two middle sizes reproduce WikiText-103 to about 2% and come out 10% high on LAMBADA. The largest does not reproduce, 20.24 against 17.48 and 10.13 against 8.63, and we do not know why. And how the text file is prepared moves the perplexity by about 30%, more than the step from GPT-2 medium to GPT-2 large. ◁

One clause of the abstract mattered more for what came next than any benchmark: the largest model "still underfits WebText". The loss on held-out text was like the loss on training text, and both went on improving: "performance on both the training and test sets of WebText are similar and improve together as model size is increased". Language modeling had spent decades fighting overfitting on corpora of a million words. Here the corpus was no longer the limit; the model and the compute were. The paper also drew a design rule from the wish to score any text at all, "A general language model (LM) should be able to compute the probability of (and also generate) any string", and built its vocabulary from bytes (Section 2).

GPT-3: learning inside the context. A year later "Language Models are Few-Shot Learners" answered the open question with scale: 175 billion parameters, more than a hundred times the largest GPT-2, a context of 2,048 tokens, 300 billion training tokens.

Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task. While typically task-agnostic in architecture, this method still requires task-specific fine-tuning datasets of thousands or tens of thousands of examples. By contrast, humans can generally perform a new language task from only a few examples or from simple instructions – something which current NLP systems still largely struggle to do. Here we show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art fine-tuning approaches. Specifically, we train GPT-3, an autoregressive language model with 175 billion parameters, 10x more than any previous non-sparse language model, and test its performance in the few-shot setting. For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model. GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks, as well as several tasks that require on-the-fly reasoning or domain adaptation, such as unscrambling words, using a novel word in a sentence, or performing 3-digit arithmetic. At the same time, we also identify some datasets where GPT-3’s few-shot learning still struggles, as well as some datasets where GPT-3 faces methodological issues related to training on large web corpora. Finally, we find that GPT-3 can generate samples of news articles which human evaluators have difficulty distinguishing from articles written by humans. We discuss broader societal impacts of this finding and of GPT-3 in general.

—T. B. Brown and others, "Language Models are Few-Shot Learners" (2020). NeurIPS 2020 (arXiv 2005.14165); the abstract.

The paper calls this language-model meta-learning. Its outer loop is training by stochastic gradient descent; its inner loop is adaptation within the forward pass, from examples supplied in the context.

Language Models are Few-Shot LearnersNeurIPS 2020 · arXiv:2005.14165T. B. Brown and others
sequence #1      sequence #2        sequence #3
5 + 8 = 13       gaot => goat       thanks => merci
7 + 2 = 9        sakne => snake     hello => bonjour
1 + 0 = 1        brid => bird       mint => menthe
3 + 4 = 7        fsih => fish       wall => mur
5 + 9 = 14       dcuk => duck       otter => loutre
9 + 8 = 17       cmihp => chimp     bread => pain

Figure 1.1: Language model meta-learning. During unsupervised pre-training, a language model develops a broad set of skills and pattern recognition abilities. It then uses these abilities at inference time to rapidly adapt to or recognize the desired task. We use the term "in-context learning" to describe the inner loop of this process, which occurs within the forward-pass upon each sequence. The sequences in this diagram are not intended to be representative of the data a model would see during pre-training, but are intended to show that there are sometimes repeated sub-tasks embedded within a single sequence.

Section 1, Figure 1.1
T. B. Brown and others, "Language Models are Few-Shot Learners" (NeurIPS 2020, arXiv 2005.14165), Section 1, Figure 1.1. The contents of the paper's three example sequences and the accompanying caption.

The paper then fixes three settings, none of which changes a weight. Zero-shot: the model is given an instruction in words and nothing else. One-shot: an instruction and one demonstration. Few-shot: "a few demonstrations of the task at inference time as conditioning", typically 10 to 100 of them, as many as fit in the context. What the name leaves open is set down in a footnote: the terms "are intended to remain agnostic on the question of whether the model learns new tasks from scratch at inference time or simply recognizes patterns seen during training".

What should a model do with demonstrations, if it does the best that the objective allows? That can be calculated.

Proposition 1.5.4 (the inner loop at the optimum). Suppose every training sequence is produced by first drawing a task \(\tau\) from a prior \(\pi\) and then the text from \(p_\tau\), so that the source is the mixture \(p = \sum_\tau \pi(\tau)\, p_\tau\). Then the minimizer of the loss predicts with

\[\begin{aligned} p(x_t \mid x_{<t}) &= \sum_\tau \pi(\tau \mid x_{<t})\; p_\tau(x_t \mid x_{<t}), \\ \pi(\tau \mid x_{<t}) &\propto \pi(\tau)\, p_\tau(x_{<t}), \end{aligned} \tag{1.7}\]

and for every task \(\tau\) and every string

\[-\log p(x_{1:n}) \;\le\; -\log p_\tau(x_{1:n}) \;+\; \log \frac{1}{\pi(\tau)} . \tag{1.8}\]

Proof. The minimizer is \(p\) by Proposition 1.1.5, and (1.7) is its conditional, by Bayes' rule. For (1.8), \(p(x_{1:n}) \ge \pi(\tau)\, p_\tau(x_{1:n})\). ∎

By (1.7) the best predictor infers the task from the prefix and then predicts as that task's own model would. No weight moves; the forward pass is the learner. By (1.8) the whole price of not being told the task is at most \(\log 1/\pi(\tau)\) bits however long the sequence, so the loss per token must sink toward that of a model that knows the task. In this picture the footnote's two possibilities are a broad prior and a narrow one. The proposition describes the optimum of the objective, not a trained network; whether a transformer gets there, and how, is measured in Section 8 and opened in Section 9.

What the paper showed was the size of the effect. On the questions about passages where GPT-2 had scored 55 F1, GPT-3 scored 81.5 with an instruction alone, 84.0 with one demonstration and 85.0 with a few; on a set of trivia questions, 64.3%, 68.0% and 71.2%. Shown a few sums, it added two-digit numbers without error and three-digit numbers 80.2% of the time. All of it came from one set of weights, trained on nothing but (1.2).Sections 1 and 3.9.1 of the paper; the two benchmarks are CoQA and TriviaQA. The paper is as plain about the limits.

A more fundamental limitation of the general approach described in this paper – scaling up any LM-like model, whether autoregressive or bidirectional – is that it may eventually run into (or could already be running into) the limits of the pretraining objective. Our current objective weights every token equally and lacks a notion of what is most important to predict and what is less important. [RRS20] demonstrate benefits of customizing prediction to entities of interest. Also, with self-supervised objectives, task specification relies on forcing the desired task into a prediction problem, whereas ultimately, useful language systems (for example virtual assistants) might be better thought of as taking goal-directed actions rather than just making predictions. Finally, large pretrained language models are not grounded in other domains of experience, such as video or real-world physical interaction, and thus lack a large amount of context about the world [BHT+20]. For all these reasons, scaling pure self-supervised prediction is likely to hit limits, and augmentation with a different approach is likely to be necessary. Promising future directions in this vein might include learning the objective function from humans [ZSW+19a], fine-tuning with reinforcement learning, or adding additional modalities such as images to provide grounding and a better model of the world [CLY+19].

—Brown et al. (2020).Section 5, pages 33 and 34 of the arXiv version.

Both papers put their whole method into the prefix. The tasks of GPT-2 are prefixes, and the demonstrations of GPT-3 are prefixes. The next question is what a prefix is worth.

What context buys

A model that reads 1,024 tokens costs more to train and to run than one that reads three. What does the context buy? The curve \(F_N\) of Section 1.2 is the answer for an ideal predictor, and any model estimates it from above.

Proposition 1.6.1 (a model with \(k\) symbols of context bounds \(F_{k+1}\)). Let the source be stationary and let \(q\) be any model whose prediction depends on at most the last \(k\) symbols. Its expected loss is at least \(F_{k+1}\), with equality only if \(q(\cdot \mid c) = p(\cdot \mid c)\) for every context \(c\) of positive probability.

Proof. For each context \(c\) of \(k\) symbols the expected loss is \(H(p(\cdot \mid c), q(\cdot \mid c)) \ge H(p(\cdot \mid c))\), by Proposition 1.1.4, and the average over \(c\) of the right side is \(F_{k+1}\). ∎

So a table of order \(N\) is a point above \(F_N\), a transformer whose context is cut to \(k\) tokens is a point above \(F_{k+1}\), and loss against \(k\) is the curve of 1951 drawn with tokens for letters and a network for the subject.

To measure the value of context, the same held-out tokens of FineWeb-EDU are scored repeatedly by gpt-pp with only the last \(k\) tokens in view, and by the count tables on those same tokens. gpt-pico is tested in the same way on stories.

bits per byte on held-out text, by the number of tokens in view
tokens   gpt-pp   its tables   gpt-pico     its tables
0                 2.403                 2.196
1        1.837    1.766        1.468    1.374
2        1.628    1.616        1.231    1.091
3        1.507    1.583        1.111    0.977
4        1.431                 1.036    
8        1.276                 0.875    
16       1.159                 0.765    
32       1.073                 0.689    
64       1.015                 0.630    
128      0.976                 0.598    
256      0.952                 0.585    
512      0.935                          
1,024    0.927                          

gpt-pp  FineWeb-EDU: 12,288 held-out tokens, each with 1,024 tokens of its document before it
gpt-pico    TinyStories: 16,384 held-out tokens, each with 256 tokens of its story before it
tables  n-grams counted on 200 million tokens, scored on the same tokens; order n sees n - 1
last    the doubling from 512 to 1,024 tokens still buys gpt-pp 0.008 bits per byte
(the live figure needs JavaScript; it loads a few kilobytes)
Bits per byte on held-out text when the model may see only the last k tokens. Green: gpt-pp on FineWeb-EDU; every point is the same 12,288 held-out tokens, each with 1,024 tokens of its own document before it. Squares: tables of n-grams counted on 200 million tokens and scored on the same tokens; the table of order n sees n − 1 tokens. The second chart: gpt-pico on TinyStories, with its tables. Point at a context to read it. Measured once; the standard errors are in the readout.

Example 1.6.2 (what context buys gpt-pp). The setup: 12,288 held-out tokens of FineWeb-EDU, drawn at random among those with at least 1,024 tokens of their own document before them; gpt-pp scores each of them twelve times, given the last \(k\) tokens and nothing else; the tables of orders 1 to 4 are scored on the same tokens, the table of order \(n\) seeing \(n - 1\) of them. It is one draw of tokens and one model; the standard errors are over the scored tokens, and differences are paired on them.

At equal context the network is ahead of a table only from three tokens. With one token in view gpt-pp pays 1.837 bits per byte, and the table of pairs, which sees the same token, pays 1.766: the table is ahead by 0.071 ± 0.003. With two tokens it is 1.628 against the table of triples' 1.616, a difference of 0.011 ± 0.004, which is 2.6 standard errors and was still shrinking as training went on (two thirds of the way through it was 0.051). With three it is 1.507 against the 4-gram table's 1.583, and the network is ahead by 0.076 ± 0.005. So the tables of pairs and of triples are not beaten at their own range, and from three tokens on gpt-pp is under every table. A table cannot use more context, because almost every longer sequence is new to it (Example 1.4.5). The network can: 1.431 with four tokens, 0.976 with 128, 0.927 with all 1,024.

Most of what context buys, it buys early: three quarters of the fall from one token is over by 16 tokens, and of the fall from two tokens only 7% comes after 128. But the curve has not stopped. The last doubling of the context, from 512 tokens to 1,024, still buys 0.008 ± 0.001 bits per byte. gpt-pico, on stories, says the same from further back: with one and two tokens in view it is behind its tables by 0.094 and 0.140 bits per byte, and it is under all of them from eight tokens on. ◁

This is not the comparison of 2003, when a network beat every table on a million words. Counted on 200 million tokens, the tables of pairs and of triples hold their own range against a network trained on three billion. What the learned representation buys, at this size, is the use of contexts too long to have been seen before.

Whether the curve ever flattens is not settled. The data of 1951 were read as levelling off near 100 letters. The study of 2025 quoted in Section 1.2, with language models as its subjects, found otherwise.

Figure 1 shows the code length \(L(N)\) for the C4 corpus in the OLMo 2 1B [33], Llama 3.2 1B [14], Qwen3 8B [37], and DCLM 1.7B [21] models. To make comparisons across models meaningful, we use characters rather than tokens as the unit of length, converting via the mean \(N/K\) for each model. We note that these results are within Shannon’s bounds at \(N=100\) [31], but the code length continues to fall slowly out to the largest \(N\sim 10^{4}\).

—Scheibner, Smith and Bialek (2025). The paragraph on their Figure 1; [31] is the paper of 1951. Four models of one to eight billion parameters, on text from the C4 corpus; the code lengths are upper bounds, and the paper is a short report that announces a fuller account.

In 2026 the decay of this curve was given work to do. It is one of two measurable properties of text from which a theory derives how fast the loss of a model falls with the amount of its training data.F. Cagnetta, A. Raventós, S. Ganguli and M. Wyart, "Deriving neural scaling laws from the statistics of natural language", 2026 (arXiv 2602.07488). The two properties are "(i) the decay of pairwise token correlations with time separation between token pairs, and (ii) the decay of the next-token conditional entropy with the length of the conditioning context". Section 8 takes the theory up. How the loss depends on what is spent is the last subject of this section.

Scaling laws

GPT-2 underfit its data, and GPT-3 was a hundred times larger and did what GPT-2 had begun to do. Two questions follow. How does the loss fall as the model, the data and the compute grow? And given a budget of compute, how should it be divided between a larger model and more training tokens? An essay of 2019 had put the general observation bluntly, "general methods that leverage computation are ultimately the most effective, and by a large margin", without saying how the computation should be spent.R. Sutton, "The bitter lesson", 13 March 2019. The scaling laws say it.

Definition 1.7.1 (parameters, tokens, compute). \(N\) is the number of parameters of a model, \(D\) the number of tokens it is trained on, and \(C\) the compute spent in training, counted in floating-point operations (FLOPs). For a transformer,

\[C \;\approx\; 6\, N D . \tag{1.9}\]

The six comes from counting. A matrix of \(m\) weights applied to one token costs \(m\) multiplications and \(m\) additions, so a forward pass costs about \(2N\) operations per token, and the backward pass about twice the forward. The papers differ in what they count as \(N\), and it will matter: the first leaves out the embedding tables, the second counts everything.

The first law came in January 2020, from models of 768 to 1.5 billion parameters trained on 22 million to 23 billion tokens.

Scaling laws for neural language modelsarXiv 2020 · arXiv:2001.08361J. Kaplan, S. McCandlish and others

1.2 Summary of Scaling Laws

The test loss of a Transformer trained to autoregressively model language can be predicted using a power-law when performance is limited by only either the number of non-embedding parameters \(N\), the dataset size \(D\), or the optimally allocated compute budget \(C_{\rm min}\) (see Figure 1):

1. For models with a limited number of parameters, trained to convergence on sufficiently large datasets:

\[L(N)=\left(N_{\mathrm{c}}/N\right)^{\alpha_{N}};\ \ \alpha_{N}\sim 0.076,\quad N_{\mathrm{c}}\sim 8.8\times 10^{13}\ \text{(non-embedding parameters)} \qquad (1.1)\]

2. For large models trained with a limited dataset with early stopping:

\[L(D)=\left(D_{\mathrm{c}}/D\right)^{\alpha_{D}};\ \ \alpha_{D}\sim 0.095,\quad D_{\mathrm{c}}\sim 5.4\times 10^{13}\ \text{(tokens)} \qquad (1.2)\]

3. When training with a limited amount of compute, a sufficiently large dataset, an optimally-sized model, and a sufficiently small batch size (making optimal use of compute):

\[L(C_{\rm min})=\left(C_{\mathrm{c}}^{\rm min}/C_{\rm min}\right)^{\alpha_{C}^{\rm min}};\ \ \alpha_{C}^{\rm min}\sim 0.050,\quad C_{\mathrm{c}}^{\rm min}\sim 3.1\times 10^{8}\ \text{(PF-days)} \qquad (1.3)\]

These relations hold across eight orders of magnitude in \(C_{\rm min}\), six orders of magnitude in \(N\), and over two orders of magnitude in \(D\). They depend very weakly on model shape and other Transformer hyperparameters (depth, width, number of self-attention heads), with specific numerical values associated with the Webtext2 training set [RWC+19]. The power laws \(\alpha_{\rm N},\alpha_{\rm D},\alpha_{C}^{\rm min}\) specify the degree of performance improvement expected as we scale up \(N\), \(D\), or \(C_{\rm min}\); for example, doubling the number of parameters yields a loss that is smaller by a factor \(2^{-\alpha_{N}}=0.95\). The precise numerical values of \(N_{\mathrm{c}},C_{\rm c}^{\rm min},\) and \(D_{\mathrm{c}}\) depend on the vocabulary size and tokenization and hence do not have a fundamental meaning.

The critical batch size, which determines the speed/efficiency tradeoff for data parallelism ([MKAT18]), also roughly obeys a power law in \(L\):

\[B_{\rm crit}\left(L\right)=\frac{B_{\ast}}{L^{1/\alpha_{B}}},\qquad B_{\ast}\sim 2\cdot 10^{8}\text{ tokens},\ \ \alpha_{B}\sim 0.21 \qquad (1.4)\]

Equation (1.1) and (1.2) together suggest that as we increase the model size, we should increase the dataset size sublinearly according to \(D\propto N^{\frac{\alpha_{N}}{\alpha_{D}}}\sim N^{0.74}\). In fact, we find that there is a single equation combining (1.1) and (1.2) that governs the simultaneous dependence on \(N\) and \(D\) and governs the degree of overfitting:

\[L(N,D)=\left[\left(\frac{N_{c}}{N}\right)^{\frac{\alpha_{N}}{\alpha_{D}}}+\frac{D_{c}}{D}\right]^{\alpha_{D}} \qquad (1.5)\]

with fits pictured on the left in figure 4. We conjecture that this functional form may also parameterize the trained log-likelihood for other generative modeling tasks.

Section 1.2, its first four paragraphs, equations 1.1 to 1.5
J. Kaplan, S. McCandlish and others, "Scaling laws for neural language models" (arXiv 2020, arXiv 2001.08361), Section 1.2, its first four paragraphs, equations 1.1 to 1.5. The equation numbers inside the fragment are the paper's. The loss is in nats per token, and a PF-day, a petaflop-day, is \(8.64 \times 10^{19}\) operations.

The exponents are small: doubling the parameters multiplies the loss by \(2^{-0.076} = 0.95\). But no floor is in sight in any of the three laws, the shape of the network hardly matters, and the fits hold over eight orders of magnitude of compute and six of model size, though only two of data. From the laws its authors derived an allocation. The best model grows as \(C^{0.73}\) and its data only as \(C^{0.27}\), so that ten times the compute should buy a model about five times larger and less than twice the tokens: "optimally compute-efficient training involves training very large models on a relatively modest amount of data and stopping significantly before convergence". The largest models of the next two years had that shape: 175 billion parameters on 300 billion tokens, then 280 billion on 300 billion, then 530 billion on 270 billion.The allocation of 2020 was superseded as a training prescription by Hoffmann et al. (2022): their experiments favor growing parameters and tokens roughly equally with compute. The power-law pattern survived; the recommended division of the budget changed. The audit below explains the disagreement.

In 2022 a second study asked the same question and got another answer.

We investigate the optimal model size and number of tokens for training a transformer language model under a given compute budget. We find that current large language models are significantly undertrained, a consequence of the recent focus on scaling language models whilst keeping the amount of training data constant. By training over 400 language models ranging from 70 million to over 16 billion parameters on 5 to 500 billion tokens, we find that for compute-optimal training, the model size and the number of training tokens should be scaled equally: for every doubling of model size the number of training tokens should also be doubled. We test this hypothesis by training a predicted compute-optimal model, Chinchilla, that uses the same compute budget as Gopher but with 70B parameters and 4× more more data. Chinchilla uniformly and significantly outperforms Gopher (280B), GPT-3 (175B), Jurassic-1 (178B), and Megatron-Turing NLG (530B) on a large range of downstream evaluation tasks. This also means that Chinchilla uses substantially less compute for fine-tuning and inference, greatly facilitating downstream usage. As a highlight, Chinchilla reaches a state-of-the-art average accuracy of 67.5% on the MMLU benchmark, greater than a 7% improvement over Gopher.

—J. Hoffmann and others, "Training compute-optimal large language models" (2022). arXiv 2203.15556; the abstract, whose doubled word is the paper's. The study is usually called by the name of the model trained to test it, Chinchilla.

It reached the answer in three ways: from the lowest point of each training curve, from sweeps of the model size at a fixed budget, and from a formula fitted to all of its runs.The third fit needs its later correction. Besiroglu et al. (2024) found problems with its optimization, rounded coefficients and reported uncertainty. Their refit supports the first two methods. The original constants are retained below to show what went wrong; they should not be treated as a current training recipe.

Training compute-optimal large language modelsarXiv 2022 · arXiv:2203.15556J. Hoffmann, S. Borgeaud, A. Mensch and others

Lastly, we model all final losses from experiments in Approach 1 & 2 as a parametric function of model parameter count and the number of seen tokens. Following a classical risk decomposition (see Section D.2), we propose the following functional form

\[\hat{L}(N,D)\triangleq E+\frac{A}{N^{\alpha}}+\frac{B}{D^{\beta}}. \qquad (2)\]

The first term captures the loss for an ideal generative process on the data distribution, and should correspond to the entropy of natural text. The second term captures the fact that a perfectly trained transformer with \(N\) parameters underperforms the ideal generative process. The final term captures the fact that the transformer is not trained to convergence, as we only make a finite number of optimisation steps, on a sample of the dataset distribution.

Section 3.3, its first paragraph, with equation 2
J. Hoffmann, S. Borgeaud, A. Mensch and others, "Training compute-optimal large language models" (arXiv 2022, arXiv 2203.15556), Section 3.3, its first paragraph, with equation 2. The first term is the entropy rate of Section 1.2, the floor that the laws of 2020 did not have.
approach \(a\), where \(N_{\mathrm{opt}} \propto C^{a}\) \(b\), where \(D_{\mathrm{opt}} \propto C^{b}\)
1. Minimum over training curves 0.50 (0.488, 0.502) 0.50 (0.501, 0.512)
2. IsoFLOP profiles 0.49 (0.462, 0.534) 0.51 (0.483, 0.529)
3. Parametric modelling of the loss 0.46 (0.454, 0.455) 0.54 (0.542, 0.543)
Kaplan et al. (2020) 0.73 0.27
Table 2 of Hoffmann et al. (2022): how the best model size and the best number of tokens grow with compute, by three methods, with the 10th and 90th percentiles of a bootstrap in parentheses, and the allocation of 2020 for comparison. The interval of the third row is the one that the audit of 2024 found implausibly narrow.

Definition 1.7.2 (compute-optimal). Given a law \(L(N, D)\) and a budget \(C\), the compute-optimal model is the pair \((N_{\mathrm{opt}}, D_{\mathrm{opt}})\) that minimizes \(L\) on the curve \(6ND = C\), which is called an isoFLOP curve.

Proposition 1.7.3 (the frontier of the additive law). Let \(L(N, D) = E + A N^{-\alpha} + B D^{-\beta}\) with positive constants, and \(C = 6ND\). Then

\[\begin{aligned} N_{\mathrm{opt}}(C) &= G \Bigl(\frac{C}{6}\Bigr)^{a}, \qquad D_{\mathrm{opt}}(C) = \frac{1}{G} \Bigl(\frac{C}{6}\Bigr)^{b}, \\ G &= \Bigl(\frac{\alpha A}{\beta B}\Bigr)^{\frac{1}{\alpha + \beta}}, \qquad a = \frac{\beta}{\alpha + \beta}, \qquad b = \frac{\alpha}{\alpha + \beta} . \end{aligned} \tag{1.10}\]

At the optimum the two reducible terms stand in the ratio \(A N_{\mathrm{opt}}^{-\alpha} : B D_{\mathrm{opt}}^{-\beta} = \beta : \alpha\), the number of tokens per parameter is \(D_{\mathrm{opt}} / N_{\mathrm{opt}} = G^{-2}\, (C/6)^{(\alpha - \beta)/(\alpha + \beta)}\), and \(L - E\) is proportional to \(C^{-\alpha\beta/(\alpha + \beta)}\).

Proof. Put \(D = C/(6N)\) and \(n = \ln N\). The reducible loss is \(R(n) = A e^{-\alpha n} + B\, (6/C)^{\beta} e^{\beta n}\), a strictly convex function of \(n\), so its one stationary point is its minimum. There \(\alpha A N^{-\alpha} = \beta B D^{-\beta}\), which is the stated ratio. With \(D = C/(6N)\) it reads \(N^{\alpha + \beta} = (\alpha A / \beta B)\, (C/6)^{\beta}\), and the rest is substitution. ∎

Three readings. The number of tokens per parameter is constant along the frontier if and only if \(\alpha = \beta\); otherwise it drifts as a power of \(C\), and a difference of 0.06 between two exponents near 0.3 tilts it sixfold over eight decades of compute. At the optimum the remaining loss is shared between a model that is too small and a run that is too short in the proportion \(\beta : \alpha\), half and half when the exponents are equal. And the frontier depends neither on \(E\) nor on any increasing function applied to the reducible part.

loss  L(N, D) = E + A / N^alpha + B / D^beta, with the constants as printed in 2022:
      E = 1.69, A = 406.4, B = 410.7, alpha = 0.34, beta = 0.28
      compute C = 6 N D = 5.76e23 FLOPs; along that line, by the size of the model:

3.2 billion parameters  1.997                                             .
8.0 billion parameters  1.954                 .
16 billion parameters   1.936      .
32 billion parameters   1.931  o  lowest: 93 tokens per parameter
64 billion parameters   1.936      .
129 billion parameters  1.953                .
322 billion parameters  1.991                                         .

best   32 billion parameters on 3.0 trillion tokens; twice too large gives up 2.3 % of the
       reducible loss, twice too small 2.3 %
       before rounding: 40 billion at 59 tokens each; the refit of 2024: 72 billion at 18
(the live figure needs JavaScript; it loads no data)
The law fitted for Chinchilla in 2022, L = E + A / N^α + B / D^β, as a calculator. First chart: the loss along the line of constant compute against model size, the sum of two terms: a model too small, a run too short. Green: the compute-optimal point. Hollow marks: twice too small, twice too large. Second chart: the same point among parameters and tokens, with contours of the loss above its floor E, the budget's line, the line of best models, and four models of 2020 to 2022. Move the budget, switch the constants.

Example 1.7.4 (one budget, three sets of constants). The setup: the budget of Gopher, a model of 280 billion parameters trained on 300 billion tokens, \(5.76 \times 10^{23}\) FLOPs as the study of 2022 gives it, put into (1.10) with the three sets of constants of the figure: the paper's as printed (\(A = 406.4\), \(B = 410.7\), \(\alpha = 0.34\), \(\beta = 0.28\)), the paper's before rounding (\(\alpha = 0.3392\), \(\beta = 0.2849\)), and the refit of 2024 described below (\(A = 482.01\), \(B = 2085.43\), \(\alpha = 0.3478\), \(\beta = 0.3658\)).

The printed constants give 32 billion parameters at 93 tokens per parameter; the unrounded ones, 40 billion at 59; the refit, 72 billion at 18. The refit recovers Chinchilla itself, the model that the paper trained at this budget: 70 billion parameters and 1.4 trillion tokens, 20 for each parameter, and it outscored the model four times its size that had spent the same compute. The allocation of 2020 asks, at this budget, for 800 billion parameters and 0.15 tokens for each. At \(10^{18}\) FLOPs, about the scale of the models of this series, the four rules ask for 26, 19, 26 and 66 tokens per parameter. The disagreement is one of exponents, and it grows with the distance extrapolated. ◁

Proposition 1.7.5 (the valley is flat). Under the hypotheses of Proposition 1.7.3, write \(R(N) = L(N, C/6N) - E\) for the reducible loss along the isoFLOP curve of a budget \(C\). Then

\[\frac{R(r N_{\mathrm{opt}})}{R(N_{\mathrm{opt}})} \;=\; 1 + \frac{\alpha \beta}{2}\, (\ln r)^2 + O\bigl((\ln r)^3\bigr) .\]

Proof. In the notation of the last proof, let \(K = \alpha A e^{-\alpha n} = \beta B\, (6/C)^{\beta} e^{\beta n}\) at the minimum. Then \(R = K/\alpha + K/\beta\) and \(R^{\prime\prime} = \alpha K + \beta K\), so \(R^{\prime\prime}/R = \alpha \beta\), and the claim is Taylor's formula in \(\ln r\). ∎

With the constants as printed, a model twice too large or twice too small gives up 2.3% of the reducible loss, one four times too large 9.0%, and one ten times too large a quarter. The optimum is cheap to miss and, for the same reason, hard to find:Near the bottom, substantially different model sizes can have almost the same loss. A precise-looking optimal size may therefore be much less certain than the predicted loss at that size. a study has to locate the bottom of a shallow bowl through the noise of its runs.

The audit of 2024. The third row of the table did not survive a second look. A group without access to the runs recovered the data from one of the paper's figures and fitted the same formula to them. Its abstract gives what it found and why.

Hoffmann et al. (2022) propose three methods for estimating a compute-optimal scaling law. We attempt to replicate their third estimation procedure, which involves fitting a parametric loss function to a reconstruction of data from their plots. However, we find that their reported estimates are inconsistent with their first two estimation methods, fail at fitting the extracted data, and report implausibly narrow confidence intervals—intervals this narrow would require over 600,000 experiments, while they likely only ran fewer than 500. Two factors explain these findings: firstly, the optimizer used by Hoffmann et al. stopped before convergence due to a poor choice of loss scale, and secondly, the parameter estimates reported in the body of the paper (as opposed to the TeX source) are rounded in a way that results in substantial bias in the predictions of the scaling law. In contrast, our re-derivation of the scaling law using the third approach yields results that are compatible with the findings from the first two estimation procedures described by Hoffmann et al.

—T. Besiroglu, E. Erdil, M. Barnett and J. You, "Chinchilla scaling: a replication attempt" (2024).arXiv 2404.10102; the abstract, its two footnotes omitted. The cause of the early stop is in its Section 3.1. The refit is \(L = 1.8172 + 482.01/N^{0.3478} + 2085.43/D^{0.3658}\).

Two small things had gone wrong. The fit had stopped early, because "they averaged the Huber loss values over different data points instead of summing them" (the Huber loss is the robust measure of error that the fit minimizes), which the original authors confirmed. And the constants printed in the paper were rounded, which at \(10^{11}\) tokens changes the data term by about 13%. Fitted again, the formula agrees with the other two methods: \(a = 0.51\), and about 20 tokens per parameter.

That left the disagreement with the law of 2020, 0.73 against 0.50. The second study had blamed the learning-rate schedule, which the first had not matched to the length of each run. A reproduction of the first study on two public corpora then removed its differences from the second one at a time, and found the causes elsewhere: "last layer computational cost, warmup duration, and scale-dependent optimizer tuning".

what is corrected \(a\), first corpus \(a\), second corpus tokens per parameter over the budgets, first corpus
nothing: the setup of 2020, reproduced 0.864 0.835 5 to 2,617
the compute of the output layer is counted 0.699 0.706 8 to 262
warmup is shortened for small models 0.603 0.602 7 to 55
the learning rate decays over each run's own length 0.574 0.571 7 to 42
learning rate and batch size are tuned for each size, without decay 0.518 0.497 11 to 22
The exponent \(a\) of \(N_{\mathrm{opt}} \propto C^{a}\) as the differences between the two studies are removed, on OpenWebText2 and on RefinedWeb, with the range of the optimal number of tokens per parameter over the budgets studied (Porian et al. 2024, Table 1).

The exponent walks from 0.86 to 0.50, and the spread of the best ratio of tokens to parameters closes from a factor of 500 to a factor of two. Most of the walk is bookkeeping and the tuning of small models: at small sizes the output layer is a large share of the compute, and a warmup of fixed length, the opening stretch of training in which the learning rate is raised from zero, takes up most of a short run.T. Porian, M. Wortsman, J. Jitsev, L. Schmidt and Y. Carmon, "Resolving discrepancies in compute-optimal scaling of language models", NeurIPS 2024 (arXiv 2406.19146); the abstract, which also finds that "careful learning rate decay is not essential" to the result. T. Pearce and J. Song, "Reconciling Kaplan and Chinchilla scaling laws", TMLR 2024 (arXiv 2406.12907), obtain the 0.73 from the second study's own law by leaving out the embedding parameters at small scale.

Remark 1.7.6 (an exponent belongs to a protocol). What is \(N\)? For gpt-pp there are three answers: 84,934,656 parameters in the attention and feed-forward matrices of its 12 layers, 110,100,480 with the output layer, and 135,266,304 with the input embedding as well. By these counts its own run of three billion tokens comes to 35, 27 or 22 tokens per parameter. For a model of 8 layers and width 512 with the same vocabulary, the output layer is 40% of the parameters that cost compute. Which count is called \(N\), how warmup is scaled and whether each size gets its own learning rate decide the exponent that a small study reports. Section 7 measures the law on a grid of our own runs.

After the audit. Three further corrections belong to the picture of 2026. The floor matters: the law of 2020 has no \(E\) and exponents below 0.1, the law of 2022 has one and exponents near 0.3, and part of that gap is the floor itself, since a power law fitted without its floor comes out with an exponent too small, by more than a factor of two in the cases examined.M. Barkeshli, A. Alfarano and A. Gromov, "On the origin of neural scaling laws: from random graphs to natural language", 2026 (arXiv 2601.10684), Section 2.1. The additive form is an assumption too, namely that what more data gains does not depend on the size of the model. A form proposed in August 2026 raises the reducible part to a fitted power, \(L = E + (A N^{-\alpha} + B D^{-\beta})^{k}\), with \(k\) between 0.31 and 0.45 on its authors' two main grids; the condition of the optimum keeps its form, and what changes is the prediction far from the measured grid.M. Videau, B. Youbi-Idrissi, D. Lopez-Paz and K. Ahuja, "Skaling: Chinchilla's exponents meet Kaplan's coupling", 7 August 2026 (arXiv 2608.07222): the additive law is "artificially enforcing a cross-derivative of exactly zero". It is a preprint, and no replication was found. And compute-optimal means optimal for the cost of training alone: a model that will answer many queries is better made smaller and trained longer than (1.10) says.N. Sardana, J. Portes, S. Doubov and J. Frankle, "Beyond Chinchilla-optimal: accounting for inference in language model scaling laws", 2023 (arXiv 2401.00448).

Why the loss should follow a power law at all is a separate question, and the authors of the first law were plain about it, in the first of the caveats that an appendix of their paper lists.

In this section we list some potential caveats to our analysis.

At present we do not have a solid theoretical understanding for any of our proposed scaling laws. The scaling relations with model size and compute are especially mysterious. It may be possible to understand scaling at very large \(D\) holding model size fixed [AS17], and also the shape of learning curves late in training, by modeling the loss with a noisy quadratic. But the scaling with \(D\) at very large model size still remains mysterious. Without a theory or a systematic understanding of the corrections to our scaling laws, it’s difficult to determine in what circumstances they can be trusted.

—J. Kaplan, S. McCandlish and others, "Scaling laws for neural language models" (2020).Appendix C, "Caveats": the sentence that opens the list and the first of its six items.

Section 8 sets out the candidate answers, the decaying curve of Section 1.6 among them.

The objective has not changed since the page of 1948: give a probability to the next symbol, and be charged its logarithm. Shannon's subject, the table of counts, the network of 2003 and GPT-3 are all scored by that number, which is also the length of a file. What changed is the function that stands between the context and the distribution, and how much context it can hold. The rest of the series opens that function: its tokens (Section 2), its attention (Section 3), its sense of position (Section 4), its block (Section 5), its training (Section 6 and Section 7), what training makes of it (Section 8), and what can be read inside it (Section 9 and Section 10).

← Back to all posts