2. Tokens and embeddings

Draft

From bytes to tokens

A language model never sees text. It sees integers, and two decisions precede its first matrix product: how text is cut into units, the tokens, and where each token is put in a vector space, its embedding. The last layer mirrors the second: it compares the final state with one vector per token, the unembedding, and turns the comparisons into probabilities. A sentence of 64 bytes is cut differently by the tokenizer of gpt-pp, the 12-layer model that is the worked example of this series, and by that of gpt-pico, the smallest size of gpt-pp, the one that runs in the page.

gpt-pp's tokenizer, a vocabulary of 32,768: 17 tokens
The| transformer|'s| residual| stream| has| |7|68| dimensions|;| it|
's| |202|6|.

gpt-pico's tokenizer, a vocabulary of 4,096: 30 tokens
The| tr|an|s|f|orm|er|'s| res|id|ual| stream| has| |7|6|8| d|im|ens|
ions|;| it|'s| |2|0|2|6|.

A word that the tokenizer's corpus knew well is one token, with the space before it: transformer, for the vocabulary learned from educational web pages. A word it did not know falls into pieces, down to single bytes if need be, so no text is ever unknown. One short algorithm, byte-pair encoding, built both vocabularies: at each step the most frequent pair of neighboring tokens becomes one new token everywhere. Applied to the opening of A Tale of Two Cities, it first absorbs the common words. Ten merges are enough for was, the, of and it to become single tokens, each with the space before it; the rare words are still spelled in bytes.

byte-pair encoding learning on the opening of A Tale of Two Cities, 286 bytes (· a space)

merge  pair fused       stood together   tokens   bytes a token
0      (the bytes)                          286            1.00
1      · + w                        13      273            1.05
2      · + t                        12      261            1.10
3      a + s                        12      249            1.15
7      ·w + as                      10      209            1.37
8      ·t + he                      10      199            1.44
9      ·o + f                       10      189            1.51
10     ·i + t                        9      180            1.59
92     ·hop + e                      1       70            4.09
after merge 92 every piece is one token and nothing is left to merge

the text after 10 merges (| a cut):
I|t|·was|·the|·|b|e|s|t|·of|·t|i|m|e|s|,|·it|·was|·the|·w|o|r|s|t|·of|·t|i|m|e|s|,|
·it|·was|·the|·|a|g|e|·of|·w| ...
next: e + s, 6 times

without the pre-split, merge 7 is already t·w, merge 14 is ,·it·was·the·,
and after 121 merges the whole text is one token
(the live figure needs JavaScript; it computes everything in the page and loads no data)
Byte-pair encoding learning on a text. The text starts as its bytes; at each merge the most frequent pair of neighboring tokens becomes one new token, everywhere. Green: the pair being merged, in the text and at the head of the counts. The curve is the number of tokens the text needs, in grey for the other setting of the pre-split, the pattern that first cuts the text into words; without it, tokens reach across words (underlined). The program is the one that learned the series' vocabularies; the lit line is where the figure stands. Computed in the page.

The obvious unit is the word. But a list of words is never complete: names, numbers, misspellings and inflected forms keep arriving, and a model that maps them all to one token, unknown, can neither read nor write them. One token per byte has no unknowns and makes every sequence several times longer (a token of gpt-pp covers 4.6 bytes on average), while attention costs the square of the length (Section 3); in 2019 byte-level models were not yet competitive either. The report on GPT-2 took a middle:

Language Models are Unsupervised Multitask LearnersOpenAI report 2019A. Radford and others

2.2 Input Representation

Byte Pair Encoding (BPE) (Sennrich et al., 2015) is a practical middle ground between character and word level language modeling which effectively interpolates between word level inputs for frequent symbol sequences and character level inputs for infrequent symbol sequences. Despite its name, reference BPE implementations often operate on Unicode code points and not byte sequences. These implementations would require including the full space of Unicode symbols in order to model all Unicode strings. This would result in a base vocabulary of over 130,000 before any multi-symbol tokens are added. This is prohibitively large compared to the 32,000 to 64,000 token vocabularies often used with BPE. In contrast, a byte-level version of BPE only requires a base vocabulary of size 256. However, directly applying BPE to the byte sequence results in sub-optimal merges due to BPE using a greedy frequency based heuristic for building the token vocabulary. We observed BPE including many versions of common words like dog since they occur in many variations such as dog. dog! dog? . This results in a sub-optimal allocation of limited vocabulary slots and model capacity. To avoid this, we prevent BPE from merging across character categories for any byte sequence. We add an exception for spaces which significantly improves the compression efficiency while adding only minimal fragmentation of words across multiple vocab tokens.

Section 2.2, page 4
A. Radford and others, "Language Models are Unsupervised Multitask Learners" (OpenAI report 2019), Section 2.2, page 4.

Two things outlived the model. Merging the bytes of UTF-8, the encoding of Unicode text as one to four bytes a character, gives a base vocabulary of 256 that covers every string: GPT-2's 50,257 tokens are those bytes, 50,000 merges and one token for the end of a text. And the ban on merging across character categories made the regular expression that first cuts the text a part of the design. Without that preliminary split, the fourteenth merge on that opening is ,·it·was·the· (a dot for a space), comma and spaces included, a token that no other text will ever need. The tokenizers of this series use the pattern of GPT-4's tokenizer: a word takes the space before it, punctuation and whitespace form pieces of their own, and a run of digits is cut three at a time.The pattern of cl100k_base in OpenAI's tiktoken library (GitHub, 2023 to 2024), less two constructs, one that only guards the speed of matching and one that matters only in the whitespace that ends a text. nanochat (GitHub, 2024 to 2026) cuts digits two at a time.

The algorithm is older than language models. It was published in 1994 as a compressor of files.

The algorithm compresses data by finding the most frequently occurring pairs of adjacent bytes in the data and replacing all instances of the pair with a byte that was not in the original data. The algorithm repeats this process until no further compression is possible, either because there are no more frequently occurring pairs or there are no more unused bytes to represent pairs. The algorithm writes out the table of pair substitutions before the packed data.

—P. Gage, "A New Algorithm for Data Compression" (1994).The C Users Journal, February 1994; under "Theory", the second paragraph.

Twenty-two years later machine translation had the problem of the open vocabulary, and the compressor's loop was reused to cut words:

Byte Pair Encoding (BPE) (Gage, 1994) is a simple data compression technique that iteratively replaces the most frequent pair of bytes in a sequence with a single, unused byte. We adapt this algorithm for word segmentation. Instead of merging frequent pairs of bytes, we merge characters or character sequences.

Firstly, we initialize the symbol vocabulary with the character vocabulary, and represent each word as a sequence of characters, plus a special end-of-word symbol ‘·’, which allows us to restore the original tokenization after translation. We iteratively count all symbol pairs and replace each occurrence of the most frequent pair (‘A’, ‘B’) with a new symbol ‘AB’. Each merge operation produces a new symbol which represents a character \(n\)-gram. Frequent character \(n\)-grams (or whole words) are eventually merged into a single symbol, thus BPE requires no shortlist. The final symbol vocabulary size is equal to the size of the initial vocabulary, plus the number of merge operations – the latter is the only hyperparameter of the algorithm.

—R. Sennrich, B. Haddow and A. Birch, "Neural Machine Translation of Rare Words with Subword Units" (2016).ACL 2016 (arXiv 1508.07909), Section 3.2, its first two paragraphs.

The ordered list of merges is the whole tokenizer, and their number is, in the paper's words, "the only hyperparameter of the algorithm".

Definition 2.1.1 (byte-pair encoding, training). Cut the corpus into pieces with a fixed regular expression and write each piece as its bytes; the corpus is then a table of distinct pieces with their counts. Tokens 0 to 255 are the values of a byte. For \(k = 0, 1, \dots, K - 1\): count every pair of adjacent tokens inside the pieces; let \((a_k, b_k)\) be the most frequent pair, ties going to the smaller ids; give it the id \(256 + k\); replace each of its occurrences, scanning every piece from the left. The tokenizer is the ordered list of merges, and its vocabulary has \(256 + K\) tokens, plus special tokens that stand for structure and are never produced from text.

In the code. The definition is three short functions. words is its table of pieces, from the tuple of a piece's tokens to its count (a helper, count_words, builds it with the pattern), and the line to look for is the one that chooses pair. They are count_pairs, merge and train in gpt-pp's bpe.py.

def count_pairs(words: dict[Word, int]) -> Counter[Pair]:
    """How often each pair of adjacent tokens occurs in the corpus."""
    counts: Counter[Pair] = Counter()
    for word, n in words.items():
        for pair in zip(word, word[1:]):
            counts[pair] += n
    return counts


def merge(ids: Sequence[int], pair: Pair, new_id: int) -> list[int]:
    """Replace each occurrence of `pair` in `ids` by `new_id`, scanning from the left."""
    out, i = [], 0
    while i < len(ids):
        if i + 1 < len(ids) and (ids[i], ids[i + 1]) == pair:
            out.append(new_id)
            i += 2
        else:
            out.append(ids[i])
            i += 1
    return out


def train(
    texts: str | Iterable[str],
    vocab_size: int,
    pattern: str = GPT4_PATTERN,
    special_tokens: Sequence[str] = SPECIAL_TOKENS,
) -> list[Pair]:
    ...
    words = count_words(texts, pattern)
    merges: list[Pair] = []
    for new_id in range(256, vocab_size - len(special_tokens)):
        counts = count_pairs(words)
        if not counts:
            break  # every word is a single token: there is nothing left to merge
        pair = min(counts, key=lambda p: (-counts[p], p))  # the most frequent; ties: the smallest ids
        merges.append(pair)
        words = {tuple(merge(word, pair, new_id)): n for word, n in words.items()}
    return merges

Word is a tuple of token ids and Pair a tuple of two. The loop variable new_id is the definition's \(256 + k\) and pair its \((a_k, b_k)\): the key (-counts[p], p) puts the highest count first and, among equal counts, the smaller ids. vocab_size counts the special tokens, so \(K\) is vocab_size less the 256 bytes and the five special tokens, 32,507 for a vocabulary of 32,768. The loop also stops early once every piece is a single token, a case the definition leaves out.

Quirk. These lines did not train the tokenizers in use: their merges were learned by a compiled library, Hugging Face's tokenizers, called from train_merges in tokenizer.py, which cuts the text with the same pattern in Python and hands each piece over as its bytes written as the characters U+0000 to U+00FF, so that the library's ids and its rule for ties are those of train. train recounts every pair at every round and is quadratic, the reference for small texts; train_fast, under it in bpe.py, keeps the counts up to date and learns all 32,507 merges of gpt-pp's tokenizer from two billion characters in 368 seconds of plain Python on one processor core, and the 3,835 of gpt-pico's in 157. Tests hold the three trainers to the same merges on small texts, ties included, and on the two training texts every merge of train_fast is the library's. Those two billion characters are the last 423,491 documents of the corpus in its shuffled order, not the whole training set.

Definition 2.1.2 (encoding and decoding). To encode a text, cut it with the same regular expression and, inside each piece, apply again and again the merge learned earliest among the pairs present, until no pair of the piece is a merge. To decode a sequence of ids, concatenate the bytes of its tokens, where the bytes of token \(256 + k\) are those of \(a_k\) followed by those of \(b_k\).

In the code. Encoding and decoding are four functions. ranks is the list of merges as a table, from a pair to the id of its merge, and vocabulary builds the table that decoding reads. They are vocabulary, encode_piece, encode and decode in bpe.py.

def vocabulary(merges: Sequence[Pair], special_tokens: Sequence[str] = SPECIAL_TOKENS) -> list[bytes]:
    """The bytes of every token, indexed by id: the 256 bytes, the merges, the special tokens."""
    vocab = [bytes([b]) for b in range(256)]
    for left, right in merges:
        vocab.append(vocab[left] + vocab[right])
    return vocab + [name.encode("utf-8") for name in special_tokens]


def encode_piece(piece: bytes, ranks: dict[Pair, int]) -> list[int]:
    """Encode one piece of text: apply the merges it contains, earliest learned first.

    `ranks` maps a pair to the id of its merge; a smaller id was learned earlier.
    """
    ids = list(piece)
    while len(ids) >= 2:
        pair = min(zip(ids, ids[1:]), key=lambda p: ranks.get(p, float("inf")))
        if pair not in ranks:
            break  # no pair of this piece was ever merged
        ids = merge(ids, pair, ranks[pair])
    return ids


def encode(text: str, merges: Sequence[Pair], pattern: str = GPT4_PATTERN) -> list[int]:
    """Text to token ids. Special tokens are never produced: text is always encoded as text."""
    ranks = {pair: 256 + k for k, pair in enumerate(merges)}
    ids: list[int] = []
    for piece in regex.findall(pattern, text):
        ids.extend(encode_piece(piece.encode("utf-8"), ranks))
    return ids


def decode(ids: Iterable[int], merges: Sequence[Pair], special_tokens: Sequence[str] = SPECIAL_TOKENS) -> str:
    """Token ids to text. Bytes that are not valid UTF-8 (a character cut in two) become U+FFFD."""
    vocab = vocabulary(merges, special_tokens)
    return b"".join(vocab[i] for i in ids).decode("utf-8", errors="replace")

In vocabulary the bytes of token \(256 + k\) are vocab[left] + vocab[right], those of \(a_k\) followed by those of \(b_k\), and the special tokens take the last ids. In encode_piece the merge learned earliest among the pairs present is the pair of smallest rank, and float("inf") puts the pairs that were never merged last. decode has one argument that the definition does not: with errors="replace", ids that no text encodes to, such as a sample that stops inside a character of several bytes, decode to U+FFFD, the replacement character, instead of raising an error, so the round trip of Proposition 2.1.3 below runs from text to ids and back and not from ids to text and back. As in training, the encoder that wrote the training data is the library's, built from the same merges; these functions encode every held-out document to the same ids.

Proposition 2.1.3 (nothing is lost, nothing is unknown). Every Unicode text \(x\) has an encoding, and \(\operatorname{decode}(\operatorname{encode}(x)) = x\).

Proof. Every character is a letter, a digit, whitespace or none of these, and the pattern has an alternative that begins with each, so the pieces are a partition of \(x\). A piece starts as its UTF-8 bytes, each of which is a token, and a merge replaces two adjacent tokens by one whose bytes are theirs concatenated: the bytes of a piece are the same before and after every merge. ∎

In the code. The proof rests on the pattern's alternatives, which the code writes one a line, with the five special tokens under them. They are GPT4_PATTERN and SPECIAL_TOKENS in bpe.py, the defaults of the functions above.

GPT4_PATTERN = "|".join(
    [
        r"'(?i:[sdmt]|ll|ve|re)",  # contractions: 's 'd 'm 't 'll 've 're
        r"[^\r\n\p{L}\p{N}]?\p{L}+",  # a word, with the space (or other single character) before it
        r"\p{N}{1,3}",  # digits, at most three at a time
        r" ?[^\s\p{L}\p{N}]+[\r\n]*",  # punctuation and symbols, with the line breaks after them
        r"\s*[\r\n]",  # whitespace up to and including a line break
        r"\s+(?!\S)",  # whitespace, leaving the last space to the word that follows
        r"\s",  # a single whitespace character that nothing above took
    ]
)

# The special tokens of gpt-pp. `<|bos|>` begins every document; the other four delimit the turns of
# a conversation and are reserved for a later chat stage.
SPECIAL_TOKENS = ["<|bos|>", "<|user_start|>", "<|user_end|>", "<|assistant_start|>", "<|assistant_end|>"]

\p{L} is any letter and \p{N} any digit, two classes that Python's standard re module does not have, which is why the code imports regex. At each position the alternatives are tried in order, and a letter, a digit, whitespace or any other character begins one of them. The special tokens take the last five ids, 32,763 to 32,767 in gpt-pp's vocabulary, the one that begins a document first.

Quirk. The pattern is that of GPT-4's tokenizer with two constructs changed: its possessive quantifiers (?+, ++, *+) are written as greedy ones, which never alters the cut, and its alternative \s++$ is left out, which alters it only inside the last run of whitespace of a text, when that run holds a line break followed by more whitespace. Compared once with the original, the pattern cuts all 37,279 held-out documents of the two datasets identically. Text is never read as a special token: the seven characters <|bos|> in a web page are encoded as seven ordinary characters, and the token enters a sequence only when the caller asks for it, with bos=True in the encode of the tokenizer's classes, where the default is False.

Known issue. Two regular-expression engines cut text with this pattern, each with its own copy of the Unicode tables: these functions ask Python's regex module, and the compiled tokenizer asks Oniguruma, the engine inside the library. They classify 17,480 code points differently, letters and digits that regex knows and Oniguruma does not, and around such a character the two encoders cut differently, though both still decode to the original text. None of these characters occurs in either corpus (a scan made once), so on this data the two agree on every document.

Quirk. Of the five special tokens, training uses one. Every document begins with <|bos|> and none ends with a token of its own; the training windows are cut without regard to documents, so that token is also a target at every boundary between two documents, and the model learns to end a document by writing it. Counting it among the targets, as the trainer's held-out loss does and the evaluation protocol does not, accounts for 0.0007 bits per byte of the difference between the two (Section 7). Token ids are stored as unsigned 16-bit integers, so a vocabulary cannot exceed 65,536 tokens.

Example 2.1.4 (the two tokenizers of this series). The setup: two byte-level tokenizers trained by Definition 2.1.1 with GPT-4's pattern and five special tokens. gpt-pico's has 4,096 tokens and is trained on TinyStories, 2.18 billion characters; gpt-pp's has 32,768 and is trained on 2.00 billion characters of FineWeb-EDU. Bytes per token are measured on held-out documents (27,630 stories, 9,649 web pages), the token that begins each document counted; the entropy is that of the token frequencies in the training set.

tokenizer vocabulary merges time to train, compiled bytes per token entropy, bits per token bits per byte never used
gpt-pico's 4,096 3,835 25 s 4.028 8.569 2.128 86
gpt-pp's 32,768 32,507 52 s 4.628 11.110 2.394 50
The two tokenizers, measured. Bytes per token on held-out text; entropy of the token frequencies on the training text, per token and per byte of that text (4.026 and 4.640 bytes per token there); the last column counts the tokens that never occur in the training text.

The first merges show what each corpus is made of (a dot stands for a space):

gpt-pico      ·t  he  ·a  ·s  ·w  nd  ·the  ed  ·b  ·to  ·and  ·h  ·f  in
gpt-pp    ·t  ·a  he  in  re  ·the  on  er  ·o  at  ·s  en  ·c  es

Each tokenizer is good at its own text: on the 9,672,101 documents of the FineWeb-EDU sample, GPT-2's tokenizer writes 9,972,918,177 tokens and gpt-pp's 9,898,298,520, 0.75% fewer with a vocabulary 35% smaller. The entropy column says how much the tokenizer alone compresses: a model that knew only how often each token occurs would score 2.400 bits per byte on the held-out web text, where bzip2 at its strongest setting reaches 2.395. What a language model adds is measured below that line and in the same unit: gpt-pp, after 3.0 billion tokens of training, scores 0.941 on the first million of the same held-out tokens (the trainer prints 0.943, because it divides by the 4.628 bytes per token of the whole held-out file, where the million tokens it scores have 4.635, a known issue that Section 7 explains; under the evaluation protocol of that section, by document, it is 0.932). For a text of \(B\) bytes cut into \(T\) tokens on which the mean loss is \(\ell\) nats per token,

\[\text{bits per byte} \;=\; \frac{T}{B}\cdot\frac{\ell}{\ln 2}. \tag{2.1}\]

Losses per token cannot be compared across tokenizers and bits per byte can, which is why Section 1 defines them and Section 7 reports them. ◁

Quirk. The byte { does not occur once in the 9.9 billion training tokens of FineWeb-EDU, where } occurs 7,395 times and the tab 42 times, and after a line of Python the finished model gives the opening brace a probability of \(3.9 \times 10^{-12}\). gpt-pp's vocabulary has no token for two spaces, so four spaces of indentation are four tokens, and the pattern gives a number no space of its own: ·2026 (a dot for a space) is three tokens, the space, 202 and 6. The tokens of the table's last column, 50 and 86, include the four reserved for conversations; their rows are never the input and never the target, and Section 2.2 follows what training does to them.

one sentence, 64 bytes, cut three ways (| a cut, · a space)

bytes: 64 tokens, one a byte
  T|h|e|·|t|r|a|n|s|f|o|r|m|e|r|'|s|·|r|e|s|i|d|u|a|l|·|s|t|r|e|a|m|·|h|a|s|·|7|
  6|8|·|d|i|m|e|n|s|i|o|n|s|;|·|i|t|'|s|·|2|0|2|6|.
gpt-pico's tokenizer, a vocabulary of 4,096: 30 tokens, 2.13 bytes a token
  The|·tr|an|s|f|orm|er|'s|·res|id|ual|·stream|·has|·|7|6|8|·d|im|ens|ions|;|
  ·it|'s|·|2|0|2|6|.
gpt-pp's tokenizer, a vocabulary of 32,768: 17 tokens, 3.76 bytes a token
  The|·transformer|'s|·residual|·stream|·has|·|7|68|·dimensions|;|·it|'s|·|202|
  6|.

the two cut at the same 16 places; gpt-pico's cuts at 13 more, gpt-pp's at no more
(the live figure needs JavaScript and loads about 0.17 MB: the merges of the two tokenizers)
One sentence cut three ways over the same bytes. Every column is a byte; a token is a run of columns over its own hairline, and the gaps are the cuts. Green: a cut that one learned tokenizer makes and the other does not. The counts at the left are the tokens each row needs. Type a sentence or pick a preset; point at a byte to read the token that holds it in each row. gpt-pico's tokenizer learned 3,835 merges on children's stories, gpt-pp's 32,507 on educational web pages; both run in the page.

Over the same 64 bytes, gpt-pico's tokenizer makes every cut that gpt-pp's makes and 13 more: ten inside the words its stories never used and three inside the two numbers. The model receives the row of integers and nothing else, and two kinds of text show what that costs.A companion post, When a word opens, puts the 24 labels of a cell, in English and in Chinese, through 12 tokenizers.

Example 2.1.5 (digits and letters). The setup: strings are looked up in the vocabularies of GPT-2, of GPT-4 and of gpt-pp. Digits. GPT-2's vocabulary holds all 100 two-digit strings, 777 of the 1,000 three-digit strings and 94 of the 10,000 four-digit ones, so a long number is cut wherever frequency happened to leave a piece. GPT-4's pattern cuts digits three at a time from the left, and its vocabulary holds all 1,000 three-digit strings. With the same pattern and a smaller vocabulary, gpt-pp's tokenizer holds 319 of them: in the sentence at the top, 768 became 7 and 68, and the longer number 299792458, becomes 299, 79, 2, 45 and 8. Chunks cut from the left do not line up with the carries of addition, which run from the right. When a hosted model was given 90 additions of seven- to nine-digit numbers, with eight solved examples before each, it was right 75.6% of the time with chunks cut from the left and 97.8% when commas forced them from the right.A. K. Singh and D. J. Strouse, "Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs", arXiv 2402.14903 (2024), Section 3.1.

Letters. The word strawberry is one token of GPT-2's vocabulary when a space precedes it and three, st, raw, berry, when none does. Which letters a token contains is not in its id, and a model can spell only what it has learned token by token.The tokenizer can recover the bytes of a token exactly. The network receives an integer and then a learned vector; it is not handed that decoding table as a list of letters. ◁

A tokenizer is fitted once, by a greedy rule, apart from the model, and these costs follow from that. The alternative is a model that reads bytes, and the designs since 2023 shorten the byte sequence inside the model: by patches of fixed size (MegaByte, 2023), by patches that end where a small byte model's uncertainty about the next byte jumps (the Byte Latent Transformer, 2024), by cuts that are themselves learned (H-Net, 2025).L. Yu et al., "MEGABYTE", NeurIPS 2023 (arXiv 2305.07185). A. Pagnoni et al., "Byte Latent Transformer: Patches Scale Better Than Tokens", arXiv 2412.09871 (2024), Sections 2.3 and 2.4. S. Hwang, B. Wang and A. Gu, "Dynamic Chunking for End-to-End Hierarchical Sequence Modeling", arXiv 2507.07955 (2025). The authors of the Byte Latent Transformer also name what a vocabulary costs.

A crucial improvement of BLT over tokenization-based models is that redefines the trade off between the vocabulary size and compute. In standard llms, increasing the size of the vocabulary means larger tokens on average and therefore fewer steps for the model but also larger output dimension for the final projection layer of the model. This trade off effectively leaves little room for tokenization based approaches to achieve significant variations in token size and inference cost. For example, Llama 3 increases the average token size from 3.7 to 4.4 bytes at the cost of increasing the size of its embedding table 4x compared to Llama 2.

—A. Pagnoni and others, "Byte Latent Transformer: Patches Scale Better Than Tokens" (2024).arXiv 2412.09871, Section 2.4, the second paragraph; the wording and the lower case are those of the page read.

In the 8-layer model of this series the final projection layer, 32,768 logits for each of 65,536 tokens, takes with the loss 84 of the 250 milliseconds of a step's forward and backward pass. In 2026 the evidence is split, in three preprints read here from their abstracts: byte models that out-scale subword ones under special training, a subword advantage traced mostly to throughput and to the prior that token boundaries carry, and a byte penalty that is larger for diffusion language models.J. Wang et al., "Byte Language Models: Scaling, Emergent Abstractions, and Information Allocation", arXiv 2610.05978. T. Gigant et al., "Decoupling the Benefits of Subword Tokenization for Language Model Training via Byte-level Simulation", arXiv 2604.27263. C. Lee et al., "The Efficiency Gap in Byte Modeling", arXiv 2605.12928. All 2026. The models of this series keep a tokenizer.

Zipf's law and what it does to training

In the 9.9 billion training tokens of FineWeb-EDU, cut by gpt-pp's tokenizer, one token in 25 is the with the space before it (3.99%). The comma follows (3.89%), then the period (2.60%), of (2.48%) and and (2.19%), each with its space. The token of rank 10,000 occurs 101,664 times, once in about 97,000 tokens, and 50 of the 32,768 tokens never occur. The shape of this list is the oldest quantitative fact about language:

One of the most puzzling facts about human language is also one of the most basic: Words occur according to a famously systematic frequency distribution such that there are few very high-frequency words that account for most of the tokens in text (e.g., “a,” “the,” “I,” etc.) and many low-frequency words (e.g., “accordion,” “catamaran,” “ravioli”). What is striking is that the distribution is mathematically simple, roughly obeying a power law known as Zipf’s law: The \(r\)th most frequent word has a frequency \(f(r)\) that scales according to

\[f(r) \propto \frac{1}{r^{\alpha}} \qquad (1)\]

for \(\alpha \approx 1\) (Zipf, 1936, 1949). In this equation, \(r\) is called the frequency rank of a word, and \(f(r)\) is its frequency in a natural corpus. Since the actual observed frequency will depend on the size of the corpus examined, this law states frequencies proportionally: The most frequent word (\(r = 1\)) has a frequency proportional to 1, the second most frequent word (\(r = 2\)) has a frequency proportional to \(\frac{1}{2^{\alpha}}\), the third most frequent word has a frequency proportional to \(\frac{1}{3^{\alpha}}\), and so forth.

—Steven T. Piantadosi, "Zipf's word frequency law in natural language: a critical review and future directions" (2014).Psychonomic Bulletin & Review 21, 2014, the first paragraph of the Introduction, a footnote mark omitted. In the passage a token is an occurrence of a word, the linguist's sense; elsewhere in this series a token is a unit of the vocabulary.

Definition 2.2.1 (Zipf's law). Order the types of a text by decreasing frequency. Zipf's law says that the frequency of the type of rank \(r\) is

\[f(r) \propto r^{-\alpha}, \qquad \alpha \approx 1, \tag{2.2}\]

and the Zipf-Mandelbrot form is \(f(r) \propto (r + \beta)^{-\alpha}\), with \(\alpha \approx 1\) and \(\beta \approx 2.7\) for words.

Example 2.2.2 (the tokens of two corpora). The setup: every token of the two training sets is counted, 9,896,904,130 tokens of FineWeb-EDU under gpt-pp's tokenizer and 542,905,221 tokens of TinyStories under gpt-pico's. The exponent is the least-squares slope of log count against log rank over ranks 10 to 1,000; the entropy is that of the frequencies.

The web text gives \(\alpha = 0.883\) and the stories \(\alpha = 1.169\), on either side of the law. With \(\alpha = 1\) exactly, 32,768 types would have an entropy of 10.55 bits and half of the text would lie within the 136 most frequent; gpt-pp's tokens measure 11.110 bits and 278. Stories are more repetitive than the law: 8.569 bits against 8.75 for 4,096 ideal types, and 1,000 tokens cover 91.0% of everything written. The exponent depends on the window, which belongs with the number: for GPT-2's tokens on WikiText-103 the same fit gives 0.924, and 1.32 when it is made over ranks 1,000 to 30,000. ◁

occurrences of each of gpt-pp's 32,768 tokens in 9.9 billion tokens of FineWeb-EDU

rank     token        occurrences    the top k cover
1        ·the         394,914,231              4.0 %
10       .↵           102,160,557             22.2 %
100      7              8,730,989             41.1 %
1,000    ·outside       1,142,555             63.3 %
10,000   ·wheels          101,664             89.9 %
32,718   byte f1                1            100.0 %
occurrences fall as rank^−0.883 over ranks 10 to 1,000;
half of the text is 278 tokens; 50 tokens never occur

how far a token's row has moved from its initialization: the median row,
by the token's occurrences in the 9.9 billion (gpt-pp read 3.0 billion of them)
occurrences      tokens   input embedding   unembedding
never                50                 0          8.01
1 to 99              16               105          8.48
100 to 9,999      1,057               631          13.3
10⁴ to 10⁶       30,504               821          12.2
over 10⁶          1,141               870          13.2
52 rows of the input embedding are exactly where they started:
those 50 and 2 whose tokens did not come up in the 3.0 billion read
(the live figure needs JavaScript and loads about 0.5 MB of data)
Left: every token of gpt-pp's vocabulary, its rank against its occurrences in the 9.9 billion training tokens of FineWeb-EDU; dashed, the power law fitted on ranks 10 to 1,000; the line that climbs is the share of all text that the k most frequent tokens cover. Middle and right: how far each token's row of the input embedding and of the unembedding has moved from its initialization, against its occurrences. Green: the median row at each frequency. Type a word, point at a token, or walk the ranks with arrow keys. Counts are exact; the rows are one training run's.

The hundred most frequent of gpt-pp's 32,768 tokens are 41% of the training set. After 3.0 billion tokens of training, input rows have moved farther from initialization when their tokens are frequent; output rows have moved by roughly similar amounts across frequencies. The reference counts come from the whole 9.9-billion-token corpus, of which the model read the first 3.0 billion. The difference between the tables follows from one computation. Write \(E\) for the table of embeddings, one row \(e_v\) per token, and \(U\) for the unembedding, with rows \(u_v\) (Definition 2.3.1).

Proposition 2.2.3 (sparse input gradient, dense output gradient). Let a sequence \(x_1, \dots, x_n\) enter a network through the rows \(a_s = e_{x_s}\) and leave it through the logits \(z_t = U h_t\), with \(p_t = \operatorname{softmax}(z_t)\) and the loss \(\mathcal L = -\sum_t \ln p_t(x_{t+1})\). Then for every token \(v\)

\[\frac{\partial \mathcal L}{\partial u_v} = \sum_{t}\bigl(p_t(v) - [x_{t+1} = v]\bigr)\, h_t, \qquad \frac{\partial \mathcal L}{\partial e_v} = \sum_{s \,:\, x_s = v} \frac{\partial \mathcal L}{\partial a_s}. \tag{2.3}\]

(the bracket is 1 when the statement inside it holds and 0 otherwise). Every row of \(U\) receives a term from every position; a row of \(E\) receives one only from the positions where its token stands.

Proof. The derivative of \(-\ln \operatorname{softmax}(z)_o\) in \(z\) is \(p - \mathbf 1_o\), with \(\mathbf 1_o\) the one-hot vector of the target \(o\), and \(z_t = U h_t\) is linear in \(U\). The lookup \(a_s = e_{x_s}\) is linear in \(E\) and involves only the row \(x_s\). ∎

The observation dates from 2017 and the recurrent models of the time:

The update for row \(k\) of the input embedding is:

\[\frac{\partial{\cal L}_{t}}{\partial U_{k}}=\begin{cases}({\sum_{x=1}^{C}p_{t}(x\mid i_{1:t})\cdot V_{x}^{\top}}-V_{o_{t}}^{\top})\frac{\partial h_{2}^{(t)}}{\partial U_{i_{t}}}&k=i_{t}\\ 0&k\neq i_{t}\end{cases}\]

For the output embedding, row \(k\)’s update is:

\[\frac{\partial{\cal L}_{t}}{\partial V_{k}}=\begin{cases}(p_{t}(o_{t}\mid i_{1:t})-1)h_{2}^{(t)}&k=o_{t}\\ p_{t}(k\mid i_{1:t})\cdot h_{2}^{(t)}&k\neq o_{t}\end{cases}\]

Therefore, in the untied model, at every timestep, the only row that is updated in the input embedding is the row \(U_{i_{t}}\) representing the current input word. This means that vectors representing rare words are updated only a small number of times. The output embedding updates every row at each timestep.

—O. Press and L. Wolf, "Using the Output Embedding to Improve Language Models" (2017).EACL 2017 (arXiv 1608.05859), Section 3. In their notation \(U\) is the input embedding, the matrix written \(E\) here, \(V\) the output embedding, \(h_2^{(t)}\) the state that the output layer reads at step \(t\), \(i_t\) and \(o_t\) the input word and the target word, and \(C\) the size of the vocabulary.

At the 65,536 tokens a step of the 8-layer models of this series, the token of rank 10,000 is expected 0.67 times a step, so at least a third of the steps give its row no gradient, and 22,768 rows are rarer still. The embedding is a table whose rows receive a gradient only when their token occurs; the unembedding is a dense layer that every position updates. One shape, V by d, holds two different learning problems.

In the finished model 52 rows of the input embedding stand exactly where initialization put them: those of the 50 tokens that never occur and of two bytes that did not come up in the 3.0 billion tokens the model read. From a start 27.7 long, the median row of the tokens that occur fewer than a hundred times in the corpus has moved 105 and that of the tokens that occur more than a million times 870; over all tokens that occur, the rank correlation between distance moved and occurrences is 0.90.

Remark 2.2.4 (one occurrence under Adam). Under plain gradient descent a row moves in proportion to the gradients it receives, so a token a thousand times rarer learns a thousand times more slowly; Section 6 measures it, with Nesterov momentum on the gradient and the vocabulary cut into ten groups of about a tenth of the text each. Adam divides, coordinate by coordinate, a running mean of the gradients (decay \(\beta_1\)) by the root of a running mean of their squares (decay \(\beta_2\)). A coordinate long at rest that receives one gradient, and none after, moves at once by \(\eta\,(1 - \beta_1)/\sqrt{1 - \beta_2}\) against the sign of the gradient, whatever its size, and at each later step by \(\beta_1/\sqrt{\beta_2}\) times its previous move. With the \((\beta_1, \beta_2) = (0.8, 0.95)\) that gpt-pp uses for its two vocabulary matrices this is \(0.89\,\eta\) at once and, summing the geometric series, \(4.99\,\eta\) in all: one occurrence of a rare token moves every coordinate of its row by five learning rates. That makes a Zipfian table trainable, and makes its rows change scale quickly (Section 2.3). The most frequent tokens of all, fewer than a hundred, are another case: their rows receive a gradient at every step, and they have moved less than the middle of the vocabulary. We have not traced why.

The unembedding has the opposite problem. It is never left alone.

Proposition 2.2.5 (the push on a token that is never the target). Suppose token \(v\) is never a target, \(x_{t+1} \ne v\) for all \(t\), and the final states lie in an open half-space: there are a unit vector \(w\) and \(c > 0\) with \(\langle w, h_t\rangle \ge c\) for all \(t\). Then, the other parameters fixed, \(s \mapsto \mathcal L(u_v - s w)\) is strictly decreasing on \([0, \infty)\), and a gradient step of rate \(\eta\) moves \(u_v\) by a vector whose inner product with \(w\) is at most \(-\eta\, c \sum_t p_t(v) < 0\).

Proof. The term of position \(t\) depends on \(u_v\) only through \(\ln\bigl(C_t + e^{\langle u_v, h_t\rangle}\bigr)\), where \(C_t > 0\) sums the exponentials of the other logits. It increases with \(\langle u_v, h_t\rangle\), which falls by at least \(cs\) along \(u_v - sw\). By (2.3) the gradient is \(\sum_t p_t(v)\, h_t\), whose inner product with \(w\) is at least \(c \sum_t p_t(v)\). ∎

The push has no end and the same direction for every token that is rarely a target. The paper that found the effect gives the same argument in words.

We provide some intuitive explanation and theoretical justification for the problem. Intuitively speaking, during the training process of a model with likelihood loss, for any given hidden state, the embedding of the corresponding ground-truth word will be pushed towards the direction of the hidden state in order to get a larger likelihood, while the embeddings of all other words will be pushed towards the negative direction of the hidden state to get a smaller likelihood. As in natural language, word frequency is usually very low comparing to the size of a large corpus, the embedding of the word will be pushed towards the negative directions of most hidden states which drastically vary. As a result, the embeddings of most words in the vocabulary will be pushed towards similar directions negatively correlated with most hidden states and thus are clustered together in a local region of the embedding space.

—J. Gao and others, "Representation Degeneration Problem in Training Natural Language Generation Models" (2019).ICLR 2019 (arXiv 1907.12009), Section 1. The embeddings meant are those of the output, our \(U\).

No row of gpt-pp's unembedding stands still:An output token receives a gradient even when it is never the correct answer: the model still has to learn to assign it little probability. An input row is read only when its token occurs. the rows of the 50 tokens that never occur (46 single bytes and four tokens reserved for conversations) have moved 8.01 from a start of 0.028, driven only by positions at which they were not the answer, where the median row of the bulk of the vocabulary has moved 12.2. Above the rare tail the line is level, and the rank correlation of distance moved with occurrences is −0.013: at the output a row is updated at every position, whether its token is there or not. gpt-pp bounds the push by capping its logits at \(\pm 15\) with a hyperbolic tangent.

Tokens that are in the vocabulary and almost never in the data are the extreme case. GPT-2's vocabulary holds some, such as SolidGoldMagikarp with a space before it, a single token with id 43453, which were found in 2023 to derail completions when typed.J. Rumbelow and M. Watkins, "SolidGoldMagikarp (plus, prompt generation)", LessWrong, February 2023. gpt-pp's vocabulary has its own kind: fragments such as earthqu, a step on the way to earthquake that the tokenizer almost always merges further, which occurs 163 times in 9.9 billion tokens.

The embedding table and its geometry

Definition 2.3.1 (embedding and unembedding). Let \(\mathcal V\) be the vocabulary, \(V = \lvert\mathcal V\rvert\), and \(d\) the width of the model. The embedding is a matrix \(E \in \mathbb R^{V \times d}\): token \(v\) enters the model as its row \(e_v\). The unembedding is a matrix \(U \in \mathbb R^{V \times d}\) with rows \(u_v\): the final state \(h \in \mathbb R^d\) at a position leaves the model as the logits \(Uh\), one inner product \(\langle u_v, h\rangle\) per token.The literature on circuits, and Section 9 with it, writes \(W_E\) and \(W_U\).

In the code. The two matrices are made in the model's constructor, and tying them is one assignment. It is GPT.__init__ in gpt-pp's model.py.

def __init__(self, cfg: ModelConfig):
    ...
    self.embed = nn.Embedding(cfg.vocab_size, cfg.d_model)
    self.pos_embed = nn.Embedding(cfg.block_size, cfg.d_model) if cfg.pos == "learned" else None
    self.embed_norm = make_norm(cfg) if cfg.embed_norm else None
    self.blocks = nn.ModuleList(Block(cfg) for _ in range(cfg.n_layer))
    # With post-norm every block ends in a norm; with pre-norm the stream needs one before it is read out.
    self.final_norm = make_norm(cfg) if cfg.norm_placement == "pre" else nn.Identity()
    self.unembed = nn.Linear(cfg.d_model, cfg.vocab_size, bias=False)
    if cfg.tie_embeddings:
        # Weight tying (Press and Wolf, 2017, arXiv:1608.05859; Inan et al., 2017, arXiv:1611.01462): one
        # matrix reads tokens in and writes them out. It saves V * d parameters and was standard through
        # GPT-2. Untied, the two roles get their own geometry and their own learning rates.
        self.unembed.weight = self.embed.weight
    self.init_weights()
    ...

The matrix \(E\) is self.embed.weight and \(U\) is self.unembed.weight; neither end has a bias. PyTorch stores the weight of a Linear from \(d\) numbers to \(V\) as \(V\) rows of \(d\), the shape of the lookup table, so the assignment under cfg.tie_embeddings makes the two modules share one tensor (Definition 2.5.5). make_norm builds the same normalization for the embedding, for the blocks and for the end. The lookup and the readout are the methods embed_tokens and unembed_logits.

def embed_tokens(self, idx: Tensor, t0: int = 0) -> Tensor:
    ...
    x = self.embed(idx)
    after = self.cfg.pos == "sinusoidal" and self.cfg.pos_after_norm
    if self.cfg.pos == "learned":
        x = x + self.pos_embed.weight[t0:t0 + idx.size(1)]
    elif self.cfg.pos == "sinusoidal" and not after:
        x = x + self.pos_table[t0:t0 + idx.size(1)]
    if self.embed_norm is not None:
        x = self.embed_norm(x)
    if after:
        x = x + self.pos_table[t0:t0 + idx.size(1)] / self.embed_std
    return F.dropout(x, self.cfg.dropout, self.training) if self.cfg.dropout > 0 else x

def unembed_logits(self, x: Tensor) -> Tensor:
    """Logits in float32. The soft-cap cap * tanh(logits / cap) bounds them smoothly (the clipping of Bello
    et al., 2016, arXiv:1611.09940, used on output logits by Gemma 2, 2024, arXiv:2408.00118): the model
    cannot spend its capacity on ever more extreme logits for tokens it is already sure of."""
    logits = self.unembed(x).float()
    cap = self.cfg.logit_softcap
    return cap * torch.tanh(logits / cap) if cap > 0 else logits

idx holds token ids, one row a sequence, and self.embed(idx) selects their rows \(e_v\). self.embed_norm is the normalization written out in the display below; the recipe's positions are rotary and act inside attention (Section 4), so neither branch for positions runs. Where a variant adds fixed sinusoids, the default adds them before the norm, and there their share of the energy that enters the norm fell from 33% to 1.4% as the rows grew (Proposition 2.3.2 below): a bug, fixed by the switch pos_after_norm, that Section 4 tells. In unembed_logits, x is the final state \(h\) of the definition, after the last normalization, and self.unembed(x) is \(Uh\) at every position.

Quirk. The logits that leave gpt-pp are not \(Uh\) but \(15\tanh(Uh/15)\), no longer linear in the state: the cap is inside the model (logit_softcap, 15 in the recipe and 0, which is off, in the 2019 one), so the loss, sampling and every reading of the model's output receive capped logits, computed in float32, the 32-bit format, from a product that training makes in bfloat16, its 16-bit format (Section 6). An analysis that multiplies a state by model.unembed.weight itself reads the logits before the cap, which are the logits of Definition 2.5.1 and of Theorem 2.5.3; the analysis tools of the later sections do one or the other, and the exact identities of Section 9 require a model without the cap. On the ladder of Section 5, and when it alone is changed in the finished recipe, the cap moves the held-out loss by 0.003 and 0.002 nats, which is nothing. After PyTorch 2.9.1, the version of every run here, the release notes of 2.13 list nn.LinearCrossEntropyLoss, which fuses the last linear map with the loss; gpt-pp has its cap between the two (from the notes, not tested).

For gpt-pp, \(V = 32{,}768\) and \(d = 768\): each matrix has 25,165,824 parameters, and the two are 37% of the model's 135,266,304. For gpt-pico, 4,096 by 128, they are 47% of 2,228,224. The embedding does no arithmetic, since multiplying the one-hot vector of a token by \(E\) selects a row; all of its content is in where training puts the rows. In gpt-pp the row is normalized on its way into the stream (the running sum of vectors, one a position, that every layer reads and adds to: Section 5), by the parameter-free normalization that Section 5 calls RMSNorm,

\[\hat e_v \;=\; \frac{e_v}{\sqrt{\tfrac1d \lVert e_v\rVert^2 + 10^{-6}}} \;\approx\; \sqrt d\,\frac{e_v}{\lVert e_v\rVert},\]

so every token starts on the sphere of radius \(\sqrt d\) and only the direction of its row matters. That decides how the row moves.

Proposition 2.3.2 (a normalized row only grows). Let \(\mathcal L\) be a differentiable function of a nonzero row \(e \in \mathbb R^d\) that depends only on its direction: \(\mathcal L(ce) = \mathcal L(e)\) for all \(c > 0\). Then \(\langle \nabla \mathcal L(e), e\rangle = 0\) and \(\nabla\mathcal L(ce) = \nabla\mathcal L(e)/c\), and the gradient step \(e^{+} = e - \eta\,\nabla\mathcal L(e)\) has \(\lVert e^{+}\rVert^2 = \lVert e\rVert^2 + \eta^2\,\lVert\nabla\mathcal L(e)\rVert^2\).

Proof. Differentiate \(\mathcal L(ce) = \mathcal L(e)\) in \(c\) at \(c = 1\), then in \(e\); the step is orthogonal to \(e\). ∎

Each gradient step lengthens the row, and a longer row turns less for a step of the same size: the learning rate of each row decays at the row's own pace. The table is trained by Adam, whose step is not the gradient but the gradient rescaled coordinate by coordinate and averaged with earlier ones: it is not exactly orthogonal to the row, and a single step can shorten it. But the loss still has no pull on the row's length, by Remark 2.2.4 the steps are large, and the rows of the finished model show the growth. They are 811 long on average, 29 times their length at the start, and the more often a token occurs the longer its row: the rank correlation of length with occurrences is +0.903, and the mean length falls from 868 in the most frequent tenth of the tokens to 698 in the least frequent. Without the normalization those lengths would be the scale at which tokens enter the stream. With it removed, the 8-layer model ended 50 million tokens 0.014 nats worse (4.059 against 4.045), and 0.008 worse in a second seed of both runs: a small cost, and a repeated one.

gpt-pp's 32,768 tokens as points: a token's row of a table, scaled to unit length, along the
three directions in which the 4,000 most frequent rows vary most (5.2 % of their variance
in the input embedding, 2.5 % in the unembedding)

the ten tokens nearest ·March by cosine in all 768 dimensions, and their place in the picture
    input embedding   cosine   place    unembedding     cosine   place
 1  March               0.71   6,319    ·April *          0.75     416
 2  ·April *            0.54     136    ·February *       0.73     405
 3  ·February *         0.52   2,243    ·January *        0.69     523
 4  ·march              0.47   6,109    ·December         0.66     372
 5  ·January *          0.46     307    ·July *           0.64     207
 6  ·June *             0.45   2,437    ·June *           0.64      78
 7  ·July *             0.43     556    ·October *        0.63     477
 8  ·August *           0.41   1,063    ·November *       0.63     635
 9  ·October *          0.40   7,124    ·September        0.62     715
10  ·November *         0.40     453    ·August *         0.59     173
* in both lists: 8 of ten. Place 1 is the point nearest ·March among all 32,768.
(the live figure needs JavaScript; it loads about 1.5 MB of data and 0.7 MB of three.js)
The tokens of gpt-pp as points: each token's row of a vocabulary table, scaled to unit length, along the three directions in which the 4,000 most frequent tokens vary most. Green: the chosen token and its ten nearest neighbors by cosine, found in all 768 dimensions. The lists give each neighbor's cosine and its place in the picture, which keeps only a few percent of the variance; underlined neighbors are in both tables' lists. Type a word, click a point or a neighbor, switch between input embedding and unembedding, drag to turn. One training run.

The nearest tokens to March, with its space, are other months: in the unembedding its ten nearest tokens are ten months, in the input embedding eight months and its own other spellings, March without its space and march. The neighbors are found in all 768 dimensions.A close pair in a three-dimensional projection can be far apart in the original space. The listed neighbors come from the full vectors, so their order is not an inference from the picture. The picture, three directions that hold 5.2% of the variance of the rows, is a hint and no more: a token's true neighbors stand hundreds or thousands of places away among its 32,768 points.

The two tables mostly agree on who is near whom. The five nearest tokens of king are King, kings, King without its space, queen and emperor, in that order in both, with cosines from 0.62 down to 0.42; those of the digit 7, which takes no space in this vocabulary, are 6, 9, 8, 5 and 4 in the one and the same five in the other. Where the two differ, as for March, the input table ranks first a token's other written forms and the output table the tokens that could stand in its place.

The rarest tokens show Proposition 2.2.5 as a direction. In the unembedding the ten nearest rows of a fragment such as earthqu are bytes that hardly occur, the unused special tokens and other fragments of its kind, with cosines of 0.65 for that token and up to 0.99 for others like it. In the input embedding the same tokens have no neighbors to speak of, at cosines of 0.16 to 0.26. The rows of the tokens that are almost never the answer have all been pushed the same way.

As wholes, the two tables share almost no common direction: two rows chosen at random have cosine 0.034 on average, in the input embedding and in the unembedding alike, and 0.000 once the mean row is subtracted. A released model of 2019 looks different.

Example 2.3.3 (the table of GPT-2). The setup: the embedding matrix of the released GPT-2 small, 50,257 rows of 768 numbers, which that model uses as both \(E\) and \(U\); token frequencies are counted on 59.0 million tokens of WikiText-103.

The rows have norm 3.96 on average, between 2.45 and 6.32. Frequency is written into the lengths: the mean norm rises from 3.42 in the most frequent tenth of the tokens to 4.15 in the least frequent, and is 4.21 for the 4,284 tokens that never occur in that text (rank correlation of norm with log frequency: −0.46). Direction is not uniform either. Two rows chosen at random have cosine 0.268 on average, and 0.000 once the mean row, whose norm is 2.05, is subtracted from all of them. The table is a cloud displaced from the origin: its tokens share one direction and spread over many around it. ◁

GPT-2's numbers differ from gpt-pp's on both counts: its frequent tokens have the shorter rows (−0.46, against +0.903 for gpt-pp's input rows and −0.013 for its output rows), and its rows share a direction far more strongly (a mean cosine of 0.268 against 0.034). Neither sign is a fact about language. In GPT-2's one table the length of a row reaches the model at both ends. In gpt-pp the length of an input row is normalized away before the model sees it, nothing shrinks it, since the recipe has no weight decay, and the updates add to it (Proposition 2.3.2): it is a record of how much the row has been moved, and the rows of frequent tokens have been moved most, all but the few at the very top (Remark 2.2.4). The output rows are moved at every position, frequent or not.

To say how far a trained table is from one without structure, we need to know what vectors without structure look like in 768 dimensions. They look nothing like vectors in three.

High-dimensional probability

High-dimensional spaces can be counterintuitive. Pick two random points, and they will probably be almost orthogonal!

To be more concrete, let \(X, Y\) be independent random vectors uniformly distributed on the unit sphere \(S^{n-1}\). Then \(\sqrt{n} X\) and \(\sqrt{n} Y\) are independent, identically distributed, and isotropic by Proposition 3.3.8. By Proposition 3.2.1(c), we have \(\mathbb E \langle \sqrt{n} X, \sqrt{n} Y \rangle^2 = \operatorname{tr}(I_n) = n\). Dividing by \(n^2\), we obtain

\[\mathbb E \langle X, Y \rangle^2 = \frac{1}{n}.\]

Then Markov inequality yields

\[\lvert \langle X, Y \rangle \rvert = O(1/\sqrt{n}) \quad \text{with high probability,} \qquad (3.14)\]

showing that \(X\) and \(Y\) typically are almost orthogonal. This contrasts with low dimensions, where a pair of random directions in the plane have the (smaller) angle averaging \(\pi/4\). (Check!) In higher dimensions, there is more room, so vectors spread out like lone stars in the sky.

—Roman Vershynin, High-Dimensional Probability (second edition, 2026).R. Vershynin, High-Dimensional Probability: An Introduction with Applications in Data Science, second edition, Cambridge University Press, 2026, Section 3.3, under the heading "Isotropic random vectors are almost orthogonal". The statements below correspond to his Theorem 3.1.1, Section 3.3, Theorem 5.3.1 and Exercise 3.41; the first is proved there for coordinates more general than Gaussian.

cosine t of two random directions in d dimensions: exact density
proportional to (1 − t²)^((d − 3)/2), each drawn to its own peak

d          1/√d     −1               0               +1
3          0.577    ███████████████████████████████████  flat
16         0.250             ▁▂▄▆█████████▆▄▂▁
128        0.088                  ▂▅█████▅▂
768        0.036                     ▐▌
sampled at d = 768: 0 of 4,950 pairs beyond ±0.3, the largest 0.13

length of a Gaussian vector at d = 768: 27.7 ± 0.71 (√d = 27.7)

directions that fit with every pairwise |cosine| ≤ 0.3
d          guaranteed     kept by a search of 1,000 random tries
16         1.4            15
64         4.2            173
128        17.8           806
256        317            1,000
768        3.2 × 10⁷      1,000
guaranteed: exp(d ε²/4), which the random argument proves
(the live figure needs JavaScript; it loads about 0.7 MB of three.js for the box and no data)
Random directions in d dimensions. Green: the histogram of the cosines of the 4,950 pairs among 100 sampled directions. Ink: the exact densities; the count that the random argument guarantees; what the search kept. The box draws three of the d coordinates of each direction: height is the cosine with one fixed direction, the band is the tolerance. Move d, change the tolerance, zoom the axes, resample. The densities, the tail and the guarantee are exact; the sample and the search are one seed each, computed in the page.

As the dimension grows the cosines between them gather around zero, the lengths of Gaussian vectors gather around one value, and the number of directions that can be kept nearly orthogonal leaves the dimension far behind. This section proves the three, and a fourth, the lemma of Johnson and Lindenstrauss, which follows from the second. The second and the fourth rest on one computation.

Lemma 2.4.1 (the tail of a sum of squares). Let \(g \sim \mathcal N(0, I_m)\). For \(0 \le \varepsilon \le 1\),

\[\mathbb P\Bigl(\Bigl\lvert \frac{\lVert g\rVert^2}{m} - 1 \Bigr\rvert \ge \varepsilon\Bigr) \;\le\; 2\, e^{-m\varepsilon^2/8}. \tag{2.4}\]

Proof. For \(\lambda < \tfrac12\), \(\mathbb E\, e^{\lambda g_i^2} = (1 - 2\lambda)^{-1/2}\). Markov's inequality for \(e^{\lambda \lVert g\rVert^2}\) with \(\lambda = \varepsilon/(2(1 + \varepsilon))\) gives

\[\mathbb P\bigl(\lVert g\rVert^2 \ge m(1 + \varepsilon)\bigr) \;\le\; e^{-\lambda m(1+\varepsilon)}\,(1 - 2\lambda)^{-m/2} \;=\; e^{-\frac m2(\varepsilon - \ln(1+\varepsilon))} \;\le\; e^{-m\varepsilon^2/8},\]

because \(\varepsilon - \ln(1 + \varepsilon) - \varepsilon^2/4\) vanishes at 0 and has derivative \(\varepsilon(1 - \varepsilon)/(2(1 + \varepsilon)) \ge 0\) on \([0, 1]\). For \(\varepsilon < 1\) the same argument with \(e^{-\lambda\lVert g\rVert^2}\) and \(\lambda = \varepsilon/(2(1 - \varepsilon))\) gives \(\mathbb P(\lVert g\rVert^2 \le m(1 - \varepsilon)) \le e^{-\frac m2(-\varepsilon - \ln(1 - \varepsilon))} \le e^{-m\varepsilon^2/4}\), since \(-\ln(1 - \varepsilon) \ge \varepsilon + \varepsilon^2/2\); at \(\varepsilon = 1\) that event has probability zero. ∎

Theorem 2.4.2 (the norm concentrates). Let \(g \sim \mathcal N(0, I_d)\). Then \(\mathbb E\lVert g\rVert^2 = d\), \(\operatorname{Var}\lVert g\rVert^2 = 2d\), and for \(0 \le t \le \sqrt d\)

\[\mathbb P\bigl(\bigl\lvert\, \lVert g\rVert - \sqrt d \,\bigr\rvert \ge t\bigr) \;\le\; 2\, e^{-t^2/8}. \tag{2.5}\]

Proof. \(\lVert g\rVert^2\) is a sum of \(d\) independent squares of mean 1 and variance 2. Put \(z = \lVert g\rVert/\sqrt d\) and \(\delta = t/\sqrt d \le 1\). Since \(\lvert z^2 - 1\rvert = \lvert z - 1\rvert (z + 1) \ge \lvert z - 1\rvert\), the event \(\lvert z - 1\rvert \ge \delta\) lies inside \(\lvert z^2 - 1\rvert \ge \delta\), whose probability is at most \(2e^{-d\delta^2/8} = 2e^{-t^2/8}\) by (2.4). ∎

The bound does not contain \(d\): the radius of the shell grows as \(\sqrt d\) and its thickness stays of order one, because \(\sqrt{d \pm O(\sqrt d)} = \sqrt d \pm O(1)\). The standard deviation of \(\lVert g\rVert\) tends to \(1/\sqrt2\) as \(d\) grows: at \(d = 768\) the length is \(27.70 \pm 0.71\), a relative spread of 2.55%.The constant 8 is the price of a short proof. The Gaussian concentration inequality gives \(2e^{-t^2/2}\) around the mean for every 1-Lipschitz function of \(g\), the norm among them.

Proposition 2.4.3 (random directions are nearly orthogonal). Let \(u\) be uniform on the unit sphere of \(\mathbb R^d\) and \(v\) a unit vector, fixed or independent of \(u\). Then \(t = \langle u, v\rangle\) has density proportional to \((1 - t^2)^{(d-3)/2}\) on \([-1, 1]\), mean 0 and variance \(1/d\), and for \(0 \le \varepsilon \le 1/\sqrt2\)

\[\mathbb P\bigl(\lvert\langle u, v\rangle\rvert \ge \varepsilon\bigr) \;\le\; 2\, e^{-d\varepsilon^2/2}. \tag{2.6}\]

Proof. By rotation invariance take \(v\) to be the first basis vector \(b_1\), so that \(t = u_1\). The coordinates of \(u\) have one law and their squares add to 1, so \(\mathbb E u_1^2 = 1/d\); the mean is 0 by symmetry. The points of the sphere with first coordinate between \(t\) and \(t + dt\) form a band around a sphere of dimension \(d - 2\) and radius \(\sqrt{1 - t^2}\), of width \(dt/\sqrt{1 - t^2}\), hence of area proportional to \((1 - t^2)^{(d-2)/2}(1 - t^2)^{-1/2}\,dt\). For the tail, let \(C\) be the cap of points with \(u_1 \ge \varepsilon\) and \(K\) the cone of the points \(su\) with \(u \in C\) and \(0 \le s \le 1\); the probability of \(C\) is the share of the unit ball's volume that \(K\) takes. For such a point,

\[\lVert su - \varepsilon b_1\rVert^2 \;=\; s^2 - 2 s \varepsilon u_1 + \varepsilon^2 \;\le\; s^2 - 2 s\varepsilon^2 + \varepsilon^2 \;\le\; 1 - \varepsilon^2,\]

the last step because \(s \mapsto s^2 - 2s\varepsilon^2 + 2\varepsilon^2 - 1\) is convex, at most 0 at \(s = 0\) (as \(\varepsilon \le 1/\sqrt2\)) and equal to 0 at \(s = 1\). So \(K\) lies in a ball of radius \(\sqrt{1 - \varepsilon^2}\) and \(\mathbb P(C) \le (1 - \varepsilon^2)^{d/2} \le e^{-d\varepsilon^2/2}\). The other tail is its mirror image. ∎

The cosine of two random directions is \(0 \pm 1/\sqrt d\). At \(d = 768\) the spread is 0.0361, and the exact probabilities of a cosine beyond 0.1, 0.2 and 0.3 in absolute value are \(5.5 \times 10^{-3}\), \(2.2 \times 10^{-8}\) and \(1.9 \times 10^{-17}\). At \(d = 3\) the density is flat.

Theorem 2.4.4 (Johnson and Lindenstrauss). Let \(x_1, \dots, x_N\) be points of \(\mathbb R^D\), \(0 < \varepsilon \le 1\) and \(m \ge 16\,\varepsilon^{-2}\ln N\). There is a linear map \(A : \mathbb R^D \to \mathbb R^m\) with

\[(1 - \varepsilon)\,\lVert x_i - x_j\rVert^2 \;\le\; \lVert A x_i - A x_j\rVert^2 \;\le\; (1 + \varepsilon)\,\lVert x_i - x_j\rVert^2 \qquad\text{for all } i, j, \tag{2.7}\]

and if \(m \ge 24\,\varepsilon^{-2}\ln N\), the matrix \(A = G/\sqrt m\) with independent standard Gaussian entries in \(G\) satisfies (2.7) with probability at least \(1 - 1/N\).

Proof. Fix \(z \ne 0\) in \(\mathbb R^D\). The \(m\) coordinates of \(Gz\) are independent \(\mathcal N(0, \lVert z\rVert^2)\) variables, so \(\lVert Az\rVert^2/\lVert z\rVert^2\) has the law of \(\lVert g\rVert^2/m\) in Lemma 2.4.1 and leaves \([1 - \varepsilon, 1 + \varepsilon]\) with probability at most \(2e^{-m\varepsilon^2/8}\). Apply this to the \(N(N-1)/2\) differences \(z = x_i - x_j\) and add the probabilities: (2.7) fails with probability at most \(N(N - 1)\,e^{-m\varepsilon^2/8}\), which is at most \(1 - 1/N\) in the first case, so that some matrix succeeds, and less than \(1/N\) in the second. ∎

The dimension \(D\) of the points does not appear, and the map is drawn without looking at them. The proof used nothing but the thin shell: each difference of two points, once projected, lands in a shell around its own length. The dependence of \(m\) on \(\varepsilon^{-2}\ln N\) cannot be improved in general.The lemma is from W. B. Johnson and J. Lindenstrauss (1984); for its optimality, K. G. Larsen and J. Nelson, "Optimality of the Johnson-Lindenstrauss Lemma", FOCS 2017 (arXiv 1609.02094).

The last statement is the one most quoted about language models, usually without its constant.

Proposition 2.4.5 (exponentially many almost-orthogonal directions). For every \(d\) and \(0 < \varepsilon \le 1/\sqrt2\) there are \(\lfloor e^{d\varepsilon^2/4}\rfloor\) unit vectors in \(\mathbb R^d\) whose inner products are all at most \(\varepsilon\) in absolute value.

Proof. Draw \(N = \lfloor e^{d\varepsilon^2/4}\rfloor\) independent uniform points of the sphere. By (2.6) each of the \(N(N-1)/2\) pairs has an inner product beyond \(\varepsilon\) with probability at most \(2e^{-d\varepsilon^2/2}\), so some pair does with probability at most \(N(N - 1)\,e^{-d\varepsilon^2/2} < 1\). With positive probability none does. ∎

There is a ceiling as well. Unit vectors of \(\mathbb R^d\) whose inner products are all at most \(1/(2\sqrt d)\) in absolute value, which is 0.018 at \(d = 768\), number fewer than \(2d\); at a few times \(1/\sqrt d\) their number is at most polynomial in \(d\), and only at a fixed tolerance is it exponential.T. Tao, "A cheap version of the Kabatjanskii-Levenstein bound for almost orthogonal vectors", 18 July 2013: Lemma 2, by a rank argument on the matrix of inner products, and Theorem 1. What Proposition 2.4.5 guarantees is this:

width \(d\) tolerance 0.1 tolerance 0.2 tolerance 0.3
128 (gpt-pico) 1.4 3.6 17.8
768 (gpt-pp) 6.8 2.2 × 103 3.2 × 107
4,096 2.8 × 104 6.1 × 1017 1.1 × 1040
The number of directions with all pairwise cosines within the tolerance that the random argument guarantees, \(e^{d\varepsilon^2/4}\). It is a lower bound and can fall below the \(d\) directions of a basis.

At \(d = 768\) and a tolerance of 0.3 the guarantee is 32 million directions, a thousand times gpt-pp's vocabulary. At a tolerance of 0.1 it is six, in a space that holds 768 exactly orthogonal directions: there the bound says nothing. It comes from a union bound over random draws, and its exponent only bites when \(d\varepsilon^2\) is large. A greedy search keeps 351 random directions at that tolerance, still fewer than a basis; neither number is the maximum. The sentence usually quoted should be read with the table beside it:

Even when there is a privileged basis, it's often the case that neurons are "polysemantic", responding to several unrelated features. One explanation for this is the superposition hypothesis. Roughly, the idea of superposition is that neural networks "want to represent more features than they have neurons", so they exploit a property of high-dimensional spaces to simulate a model with many more neurons.

Several results from mathematics suggest that something like this might be plausible:

Almost Orthogonal Vectors. Although it's only possible to have \(n\) orthogonal vectors in an \(n\)-dimensional space, it's possible to have \(\exp(n)\) many "almost orthogonal" (\(<\epsilon\) cosine similarity) vectors in high-dimensional spaces. See the Johnson–Lindenstrauss lemma.

—Nelson Elhage and others, "Toy Models of Superposition" (2022).Transformer Circuits Thread, 2022 (arXiv 2209.10652), section "The Superposition Hypothesis": its first paragraph, and the first of the two results it then lists. The logical piano trains such a packing in the page: 256 features in 4 to 32 dimensions.

For a stream of width 768 the four statements say the following.

The shell. The rows of gpt-pp's embedding start as independent draws of \(\mathcal N(0, I_{768})\), so at the first step all 32,768 tokens have length \(27.7 \pm 0.7\). Normalizing such a vector changes its length by a few percent and its direction not at all: normalization is cheap because the vectors were almost on the sphere already.

Orthogonality. In 768 dimensions two random directions have cosine 0 ± 0.036 and a Gaussian vector has length 27.7 ± 0.7, so a random vector is, to within a few percent, a point of one sphere and orthogonal to every other. Whatever departs from that in a trained model, the model built. Against this null the mean cosine of 0.268 in GPT-2's table (Example 2.3.3) is more than seven spreads from zero, and the 0.034 of gpt-pp's tables just under one. The source of both is a shared mean: if rows are a common vector \(\mu\) plus independent parts, the parts are nearly orthogonal to each other and to \(\mu\), and the cosine of two rows is close to \(\lVert\mu\rVert^2\) over the mean squared length of a row. That ratio is \(2.05^2/3.96^2 = 0.27\) for GPT-2, and \(151^2/811^2 = 0.035\) and \(2.36^2/12.8^2 = 0.034\) for the two tables of gpt-pp. The same fact sets the scale of every inner product in the model: for vectors with unit-variance coordinates it has standard deviation \(\sqrt d\), which is why attention divides its scores (Section 3).

Projection. With the crude constant of Theorem 2.4.4, any 32,768 points of any dimension can be mapped linearly into 666 dimensions with all squared distances kept within 50%. A vocabulary can exceed the width by a factor of 43, as gpt-pp's does, and the tokens still be told apart.

Packing. A direction in the stream need not be a token. Section 10 looks for the directions that a trained model uses for its features, with dictionaries of more directions than the stream has dimensions; Proposition 2.4.5 and the ceiling are the two ends of what is possible, according to the interference tolerated.

The unembedding and the softmax

At the last layer the question is reversed: given a state, which token? The state \(h\), normalized onto the sphere of radius \(\sqrt d\), is compared with every row of \(U\), and a logit is the length of the row times \(\sqrt d\) times a cosine, \(z_v = \sqrt d\,\lVert u_v\rVert \cos\angle(u_v, h)\). To put probability on a token, the model must bring the state's cosine with the token's row far above the \(1/\sqrt d\) of Proposition 2.4.3.

gpt-pico after "Once upon a time, there was a little girl named":
the next token, of 4,096

token      logit   at T = 1   at T = 0.8
·Lily       9.41      0.319        0.413  █████████████████████████
·Sue        8.64      0.148        0.158  █████████
·Lucy       8.52      0.130        0.135  ████████
·Amy        8.22      0.097        0.093  ██████
·Mia        8.20      0.095        0.091  █████
·Sally      7.03      0.029        0.021  █
·Jane       6.77      0.023        0.015  █
·Emma       6.55      0.018        0.012  █
the rest              0.141        0.064  ████

temperature    entropy    effective choices
0.1            0.01 bits  1.0
0.5            1.77 bits  3.4
0.8            2.80 bits  7.0
1.0            3.47 bits  11
2.0            6.67 bits  102
10.0           11.88 bits 3,771

the embedding read straight out, with no block between: gpt-pico (two tables) answers ·Tim,
0.16; gpt-pico-2019 (one tied table) answers ·named, 1.00
(the live figure needs JavaScript: 2.3 MB of gpt-pico's weights, 1.8 MB more for gpt-pico-2019)
The last step of gpt-pico, the model that runs in this page, after the framed token of a sentence. Left: the logits of the thirty likeliest next tokens; the hairlines stand one temperature apart, and each is a factor e in probability. Green: the probability mass at the chosen temperature; the ticks are temperature 1, where the model was trained. The curve is the entropy against the temperature. Move the temperature, click a token, type a sentence, change the model to the 2019 recipe with its one tied table, or read the embedding straight out with no block between.

Consider gpt-pico after the words Once upon a time, there was a little girl named. At temperature 1 gpt-pico spreads the name over about eleven choices, Lily first with probability 0.32; at 0.8 the same logits give her 0.41 and seven choices, at 0.1 a single one, and at 10 nearly all 4,096. A logit gap of one temperature is a factor \(e\) in the ratio of probabilities: the temperature is the unit in which a gap between logits counts.

Definition 2.5.1 (logits, softmax, temperature). For logits \(z \in \mathbb R^n\) and an inverse temperature \(\beta > 0\), the softmax is the Gibbs measure

\[p_\beta(i) \;=\; \frac{e^{\beta z_i}}{Z(\beta)}, \qquad Z(\beta) = \sum_{j} e^{\beta z_j}, \qquad H(\beta) = -\sum_i p_\beta(i)\ln p_\beta(i). \tag{2.8}\]

A language model is trained at \(\beta = 1\) on \(z = Uh\), and sampled at temperature \(T\) with \(\beta = 1/T\).

The name is from statistical mechanics: the tokens are states of energy \(-z_i\), and \(Z\) is the partition function.

In the code. Sampling at temperature \(T\) is one line of a short function. It is sample in gpt-pp's generate.py.

def sample(logits: Tensor, temperature: float = 1.0, top_k: Optional[int] = None, top_p: Optional[float] = None,
           min_p: Optional[float] = None, generator: Optional[torch.Generator] = None) -> Tensor:
    """One token id per row of logits (B, V), returned as (B, 1)."""
    if temperature == 0.0:
        return logits.argmax(dim=-1, keepdim=True)
    probs = torch.softmax(logits.float() / temperature, dim=-1)
    return torch.multinomial(truncate(probs, top_k, top_p, min_p), num_samples=1, generator=generator)

temperature is \(T\), and torch.softmax(logits.float() / temperature, dim=-1) is the \(p_\beta\) of (2.8) at \(\beta = 1/T\), one distribution for each row of logits. Temperature zero, which the definition excludes, is handled separately, as the most likely token, the limit \(\beta \to \infty\). truncate zeroes the tail of \(p_\beta\) by the rules of Section 7 before the draw. The logits come from unembed_logits (Section 2.3), so gpt-pp's \(z\) is the capped one, in training and in sampling alike.

Proposition 2.5.2 (the Gibbs measure). (a) \(\ln Z(\beta) = \max_p\,\{\beta\langle p, z\rangle + H(p)\}\) over probability vectors \(p\), attained only at \(p_\beta\). (b) \(\tfrac{d}{d\beta}\ln Z = \mathbb E_{p_\beta} z\), \(\tfrac{d^2}{d\beta^2}\ln Z = \operatorname{Var}_{p_\beta}(z)\) and \(\tfrac{dH}{d\beta} = -\beta\operatorname{Var}_{p_\beta}(z) \le 0\). (c) \(\max_i z_i \le \tfrac1\beta\ln Z(\beta) \le \max_i z_i + \tfrac{\ln n}{\beta}\).

Proof. (a) For every \(p\), \(\beta\langle p, z\rangle + H(p) = \ln Z - D(p \,\Vert\, p_\beta)\), with \(D\) the relative entropy of Definition 1.1.3 in nats, which is zero only at \(p = p_\beta\) (Proposition 1.1.4). (b) Differentiate (2.8) and use \(H = \ln Z - \beta\,\mathbb E_{p_\beta} z\). (c) \(e^{\beta\max_i z_i} \le Z \le n\, e^{\beta\max_i z_i}\). ∎

By (a) the softmax trades score against entropy at the rate \(\beta\): it is the least committed distribution for its expected score. By (b) lowering the temperature can only lower the entropy. By (c) it is a smooth maximum. At \(\beta = 1\) the derivative of \(\ln Z\) in the logits is \(p\) itself, which is where the \(p - \mathbf 1_o\) of (2.3) came from. Section 7 has the ways of cutting the distribution before sampling from it.

The softmax is the only nonlinear step between the state and the probabilities (gpt-pp's cap on the logits aside), and before it stands a matrix of rank at most \(d\). In 2018 that was shown to be a limit of its own:

Breaking the Softmax Bottleneck: A High-Rank RNN Language ModelICLR 2018 · arXiv:1711.03953Z. Yang, Z. Dai, R. Salakhutdinov and W. W. Cohen

Now the expressiveness question becomes: does there exist a parameter \(\theta\) and \(\mathbf{A}^{\prime}\in F(\mathbf{A})\) such that

\[\mathbf{H}_{\theta}\mathbf{W}^{\top}_{\theta}=\mathbf{A}^{\prime}.\]

This is essentially a matrix factorization problem. We want the model to learn matrices \(\mathbf{H}_{\theta}\) and \(\mathbf{W}_{\theta}\) that are able to factorize some matrix \(\mathbf{A}^{\prime}\in F(\mathbf{A})\). First, note that for a valid factorization to exist, the rank of \(\mathbf{H}_{\theta}\mathbf{W}_{\theta}^{\top}\) has to be at least as large as the rank of \(\mathbf{A}^{\prime}\). Further, since \(\mathbf{H}_{\theta}\in\mathbb{R}^{N\times d}\) and \(\mathbf{W}_{\theta}\in\mathbb{R}^{M\times d}\), the rank of \(\mathbf{H}_{\theta}\mathbf{W}_{\theta}^{\top}\) is strictly upper bounded by the embedding size \(d\). As a result, if \(d\geq\text{rank}(\mathbf{A}^{\prime})\), a universal approximator can theoretically recover \(\mathbf{A}^{\prime}\). However, if \(d<\text{rank}(\mathbf{A}^{\prime})\), no matter how expressive the function family \(\mathcal{U}\) is, no \((\mathbf{H}_{\theta},\mathbf{W}_{\theta})\) can even theoretically recover \(\mathbf{A}^{\prime}\). We summarize the reasoning above as follows (see Appendix A for the proof).

Proposition 1. Given that the function family \(\mathcal{U}\) is a universal approximator, there exists a parameter \(\theta\) such that \(P_{\theta}(X|c)=P^{*}(X|c)\) for all \(c\) in \(\mathcal{L}\) if and only if \(d\geq\min_{\mathbf{A}^{\prime}\in F(\mathbf{A})}\text{rank}(\mathbf{A}^{\prime})\).

Combining Proposition 1 with the Property 2 of \(F(\mathbf{A})\), we are now able to state the Softmax Bottleneck problem formally.

Corollary 1. (Softmax Bottleneck) If \(d<\text{rank}(\mathbf{A})-1\), for any function family \(\mathcal{U}\) and any model parameter \(\theta\), there exists a context \(c\) in \(\mathcal{L}\) such that \(P_{\theta}(X|c)\not=P^{*}(X|c)\).

Section 2.2, from the factorization question to Corollary 1
Z. Yang, Z. Dai, R. Salakhutdinov and W. W. Cohen, "Breaking the Softmax Bottleneck: A High-Rank RNN Language Model" (ICLR 2018, arXiv 1711.03953), Section 2.2, from the factorization question to Corollary 1. \(\mathbf{A}\) is the matrix of true log-probabilities, one row a context and one column a token; \(\mathbf{A}^{\prime}\) is that matrix with a constant added to each row, and \(F(\mathbf{A})\) the set of all such matrices. "Strictly upper bounded" is to be read as "at most".

Theorem 2.5.3 (the softmax bottleneck). Let \(c_1, \dots, c_N\) be contexts and \(x_1, \dots, x_M\) tokens, with true conditional probabilities \(P^{\ast}(x_j \mid c_i) > 0\) and \(A_{ij} = \ln P^{\ast}(x_j \mid c_i)\). Let a model give the logits \(Z = HW^{\top}\), with one row of \(H \in \mathbb R^{N \times d}\) per context and one row of \(W \in \mathbb R^{M \times d}\) per token. If \(d < \operatorname{rank}(A) - 1\), the model's distribution differs from \(P^{\ast}\) in at least one context, whatever \(H\) and \(W\) are.

Proof. Two vectors of logits have the same softmax exactly when they differ by a constant. So the model is right in every context only if \(Z = A + \lambda \mathbf 1^{\top}\) for some \(\lambda \in \mathbb R^N\), and then \(\operatorname{rank}(A) \le \operatorname{rank}(Z) + 1 \le d + 1\). ∎

What the theorem costs can be counted on a table small enough to know. Take the 300 most frequent tokens of TinyStories and, after each of them, the probabilities of the same 300 as the next token, estimated from 200 million tokens of text: a matrix \(A\) of rank 300, which by the theorem no width below 299 reproduces. Fitting logits \(HW^{\top}\) of each width \(d\) to that table by cross-entropy measures how close a narrower model can come.

tables  P*(next | last) for the 300 most frequent tokens of TinyStories: 300 by 300, rank 300
fit     logits H Wᵀ of width d, trained to the smallest divergence from the true table

width d         1           4           16          64          256         298         299
bits a token    2.42        0.56        0.053       0.0011      1.3 × 10⁻⁶  2.3 × 10⁻¹¹ 0
least squares   4.03        3.00        1.73        0.42        6.8 × 10⁻⁴  2.3 × 10⁻¹¹ 0
the entropy of the true table is 4.02 bits a token; a uniform guess is 4.21 bits away

after “was”     a        very     so       happy    not      sad
true            0.29     0.16     0.089    0.064    0.058    0.049
width 16        0.29     0.16     0.089    0.063    0.058    0.047
(the live figure needs JavaScript and loads about 1.4 MB of data)
The softmax bottleneck on real statistics. Left: the true table, the probability of each of 300 next tokens after each of 300 last tokens of TinyStories (ink on a logarithmic scale). Beside it: the closest table found for logits \(HW^{\top}\) of width d. Green: the divergence between the two, in bits a token, at each width. Grey: the same for the least-squares fit of the logits. Bars: one row of both tables. Move d, point at a row. The table has rank 300, so only width 299 is exact. Fits trained ahead of time; divergences computed in the page.

The distance falls fast. Logits of width 16 come within 0.053 bits a token of the table, which is 1.3% of its entropy, and width 64 within 0.0011. It is zero only at width 299; one width short it is \(2.3 \times 10^{-11}\) bits, which is the theorem. The best rank-\(d\) approximation to the logarithms themselves is 1.73 bits away at width 16: cross-entropy spends the width on the tokens that carry probability, least squares on the logarithms of events that almost never occur.

Remark 2.5.4 (rank against approximation). Theorem 2.5.3 is about exactness, and a model needs approximation:A full-rank table can be extremely close to a low-rank one. Here width 64 misses by only 0.0011 bits per token; the rank theorem alone cannot tell us whether increasing width is worth its cost. a distribution is nearly right as soon as the logits of the few plausible tokens stand in the right order a few units above the rest. That is also why the theorem does not contradict Theorem 2.4.4, by which 32,768 tokens fit in 768 dimensions.

The network that produces \(H\) can be as expressive as one likes; the theorem is about the last layer alone. Three things followed from it. An attack: since the logits of a model lie in a subspace of dimension \(d\) of \(\mathbb R^V\), the width of a hosted model can be read off its outputs, and was, for under 20 dollars.N. Carlini et al., "Stealing Part of a Production Language Model", ICML 2024 (arXiv 2403.06634). An account of the plateau of small models: "models based on less than 1000 hidden dimensions tend to adopt degenerate latent representations in late pretraining, which leads to reduced evaluation performance." And, in 2026, the same matrix read backward: the gradient that reaches the trunk is \(\partial\mathcal L/\partial h = U^{\top}(p - \mathbf 1_o)\), so of the \(V\)-dimensional error only the part in the column space of \(U\) is passed on, and its authors "measure empirically that 95-99% of the gradient norm is suppressed by the output layer, resulting in vastly suboptimal update directions."N. Godey, É. de la Clergerie and B. Sagot, "Why do small language models underperform? Studying Language Model Saturation via the Softmax Bottleneck", COLM 2024 (arXiv 2404.07647). N. Godey and Y. Artzi, "Lost in Backpropagation: The LM Head is a Gradient Bottleneck", COLM 2026 (arXiv 2603.10145). Both from the abstracts. Whether a width of 768 binds for a model of gpt-pp's size is not settled by any of this: that the true matrix of a language has high rank is the paper's hypothesis, and its rank cannot be measured, only guessed from larger models.

The other question about the last layer is whether it needs a matrix of its own. Through 2016 it usually had one. Then two papers proposed \(U = E\), and the saving is large: GPT-2 small has \(50{,}257 \times 768 = 38{,}597{,}376\) parameters in its one table, 31% of the model. The first argued from the updates of Proposition 2.2.3.

To conclude, in the tied NNLM, every row of \(S\) is updated during each iteration, and for all rows except one, this update is similar to the update of the output embedding of the untied model. This implies a greater degree of similarity of the tied embedding to the untied model’s output embedding than to its input embedding.

—Press and Wolf (2017).Section 3; an NNLM is a neural network language model and \(S\) the shared matrix. Their Table 5 reports that tying lowers the validation and test perplexity of a large recurrent model and not its training perplexity.

The second argued from the structure of the problem:

Despite the overwhelming success achieved by recurrent neural networks in modeling long range dependencies between words, current recurrent neural network language models (RNNLM) are based on the conventional classification framework, which has two major drawbacks: First, there is no assumed metric on the output classes, whereas there is evidence suggesting that learning is improved when one can define a natural metric on the output space (Frogner et al., 2015). In language modeling, there is a well established metric space for the outputs (words in the language) based on word embeddings, with meaningful distances between words (Mikolov et al., 2013; Pennington et al., 2014). Second, in the classical framework, inputs and outputs are considered as isolated entities with no semantic link between them. This is clearly not the case for language modeling, where inputs and outputs in fact live in identical spaces. Therefore, even for models with moderately sized vocabularies, the classical framework could be a vast source of inefficiency in terms of the number of variables in the model, and in terms of utilizing the information gathered by different parts of the model (e.g. inputs and outputs).

—H. Inan, K. Khosravi and R. Socher, "Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling" (2017).ICLR 2017 (arXiv 1611.01462), Section 1, its second paragraph. The paper's derivation holds under idealizations it states: it motivates tying and does not prove it optimal.

Definition 2.5.5 (weight tying). A model ties its embeddings when \(U = E\): one matrix is read by rows at the input and applied as a linear map at the output, and its gradient is the sum of the two gradients of (2.3).

GPT-1 and GPT-2 tie. What tying forbids is easiest to see on the shortest path through the model, which Section 9 calls the direct path.

Proposition 2.5.6 (the direct path of a tied model). Let token \(v\) be embedded, normalized to \(\hat e_v = \sqrt d\, e_v/\lVert e_v\rVert\) and unembedded with no block between. With tied matrices the logits are \(z_w = \langle e_w, \hat e_v\rangle\), and \(z_v \ge z_w\) for every token \(w\) with \(\lVert e_w\rVert \le \lVert e_v\rVert\). The table \((\langle e_w, e_v\rangle)_{w, v}\) is symmetric.

Proof. \(z_v = \sqrt d\,\lVert e_v\rVert\), and \(z_w \le \sqrt d\,\lVert e_w\rVert\) by the Cauchy-Schwarz inequality. ∎

Among tokens of equal length, the direct path of a tied model predicts that the next token repeats the current one, and it weighs "A then B" as it weighs "B then A"; language does neither. The two versions of gpt-pico behave as the proposition predicts. Read straight out of its embedding, gpt-pico in the 2019 recipe, the same shape with one tied table, answers named after named with probability 1.00, and the current token is its first answer at every position of the sentence; gpt-pico, with a table of its own for the output, answers Tim, with probability 0.16. The blocks of a tied model must undo a prediction that the parameterization built in. With two matrices the direct path is a free table of rank \(d\), and gpt-pp starts as exactly that: its blocks are initialized to write nothing, so at the first step the model is embed, then unembed, a model of pairs of adjacent tokens.

Two matrices came back with small models trained fast. The public speedrun of GPT-2 small's training untied its matrices in November 2024, tied them again in December 2025 ("I found that re-tying the Embed and LM Head could significantly decrease step count. […] I do not know exactly why"), and three days later untied them at two thirds of training.K. Jordan et al., modded-nanogpt (GitHub), records 8 (3 November 2024), 51 (19 December 2025) and 53 (22 December 2025). Each record changes other things too, so the history holds no clean comparison. nanochat unties, with a learning rate for each matrix. The 2017 paper had already found the tied matrix closer to an untied model's output embedding than to its input embedding; a mechanism was proposed in 2026: the shared matrix is shaped by its output role, because "output gradients dominate early in training", and the first layers are left with inputs formed for another purpose.A. Lopardo, A. Harish, C. Arnett and A. Gupta, "Weight Tying Biases Token Embeddings Towards the Output Space", arXiv 2603.26663 (2026), abstract.

Example 2.5.7 (two matrices, two rates). The setup: gpt-pp's recipe. The embedding starts as \(\mathcal N(0, 1)\) entries, is normalized after the lookup, and is trained by Adam at rate 0.3 (in the code PyTorch's AdamW with a weight decay of zero, which is the same update); the unembedding starts as \(\mathcal N(0, 0.001^2)\) entries, so that the first loss is \(\ln V\), and is trained at rate 0.008. Both rates are for width 768 and are multiplied by \(\sqrt{768/d}\) at other widths. There is no weight decay.

The rates differ by a factor of almost forty, and one matrix cannot take both: when the 2019 model, which ties, was trained with this recipe at the embedding's rate, the loss rose in the first 14 steps instead of falling, from 10.7 to 12.5 in one configuration and from 10.5 to 145 in another. A step of Adam moves each coordinate of a row by about the rate \(\eta\). At the input the row is normalized, so a large step can only turn it (Proposition 2.3.2). At the output a step along the signs of \(h\) moves the logit \(\langle u_v, h\rangle\) by up to \(\eta\,\lVert h\rVert_1 \le \eta\, d\): about 6 at rate 0.008 and width 768, and 230 at rate 0.3, against logits that the recipe keeps within \(\pm 15\). The second matrix costs \(Vd\) parameters, 25.2 million for gpt-pp, and they are cheap ones: an embedding costs a lookup and no matrix product. ◁

In the code. The two starting scales of the example are the first lines of the model's initialization. It is GPT.init_weights in model.py.

@torch.no_grad()
def init_weights(self) -> None:
    ...
    cfg = self.cfg
    gpt2 = cfg.init == "gpt2"
    self.embed_std = 0.02 if gpt2 or cfg.tie_embeddings else 1.0
    nn.init.normal_(self.embed.weight, std=self.embed_std)
    if self.pos_embed is not None:
        nn.init.normal_(self.pos_embed.weight, std=self.embed_std)
    if not cfg.tie_embeddings:
        nn.init.normal_(self.unembed.weight, std=0.02 if gpt2 else 0.001)
    ...

embed_std is the standard deviation of the embedding's entries: 1 only for an untied table under the recipe's init, which is zero_proj, and 0.02 otherwise, so a tied table starts at 0.02 whatever init says. The unembedding is drawn only when it is a matrix of its own, at 0.001 in the recipe. The same switch sets every matrix of the blocks, in the lines left out, and Section 5 explains it. The two rates are a rule in the width, in train_preset in config.py.

def train_preset(name: str, d_model: Optional[int] = None) -> TrainConfig:
    ...
    recipe, size = _split(name)
    width = SIZES[size]["d_model"] if d_model is None else d_model
    if recipe == "gptpp":
        # Muon on the hidden matrices at the usual 0.02. The AdamW rates of the embedding and the unembedding
        # are nanochat's values at width 768 (0.3 and 0.008) scaled by sqrt(768 / width), as nanochat scales them.
        scale = (768 / width) ** 0.5
        recipe_values = dict(lr_embedding=round(0.3 * scale, 4), lr_unembedding=round(0.008 * scale, 5))
    ...

width is \(d\) and scale is \(\sqrt{768/d}\); the rates are rounded to four and to five decimals, 0.3674 and 0.0098 for the 8-layer models of width 512. A model that ties its tables and trains with this recipe has one table for two rates: build_optimizers in optim.py gives it the unembedding's rate and weight decay, the remedy for the divergence told above, which Section 6 explains.

What the second matrix buys has been measured on the 8-layer model, in short runs. On the ladder of such models that goes from the 2019 recipe to the 2026 one a change at a time (Section 5), the rung that unties the tables and normalizes the embedding (the embedding then also trains at its own, larger rate) ends 100 million tokens at a held-out loss of 3.817 nats a token, against 3.828 for the rung before it, each the mean of two seeds: about a hundredth of a nat, 0.0124 on one seed and 0.0103 on the other, where the two seeds of these two rungs differ by 0.0014 and 0.0035 (elsewhere on the ladder two seeds have differed by as much as 0.018). It costs 16.8 million parameters. Going the other way, the 2026 recipe with its tables tied again, and no normalization after the lookup, ends 50 million tokens at 4.0447 against 4.0451 for the recipe itself, one seed each: no difference at that budget. A third run separates the two changes of the rung: with two tables and no normalization after the lookup the model ends at 4.059, which is 0.014 worse than the recipe and 0.015 worse than the tied model: at this budget untying by itself costs a little and the normalization wins it back (the cost repeats in a second seed, 0.008 against that seed's recipe; Section 2.3). The recipe keeps two matrices; at this size and these budgets the measured case for them is thin.

Linear structure

A table of vectors learned only to predict words turned out, in 2013, to hold relations between words as directions:

Continuous space language models have recently demonstrated outstanding results across a variety of tasks. In this paper, we examine the vector-space word representations that are implicitly learned by the input-layer weights. We find that these representations are surprisingly good at capturing syntactic and semantic regularities in language, and that each relationship is characterized by a relation-specific vector offset. This allows vector-oriented reasoning based on the offsets between words. For example, the male/female relationship is automatically learned, and with the induced vector representations, “King - Man + Woman” results in a vector very close to “Queen.” We demonstrate that the word vectors capture syntactic regularities by means of syntactic analogy questions (provided with this paper), and are able to correctly answer almost 40% of the questions. We demonstrate that the word vectors capture semantic regularities by using the vector offset method to answer SemEval-2012 Task 2 questions. Remarkably, this method outperforms the best previous systems.

—T. Mikolov, W. Yih and G. Zweig, "Linguistic Regularities in Continuous Space Word Representations" (2013).NAACL 2013, abstract.

Definition 2.6.1 (analogy by offset). For unit vectors and the question "\(a\) is to \(a^{\ast}\) as \(b\) is to what", the offset rule answers

\[\arg\max_{x \notin \{a, a^{\ast}, b\}} \cos\bigl(x,\; a^{\ast} - a + b\bigr) \;=\; \arg\max_{x \notin \{a, a^{\ast}, b\}} \bigl(\cos(x, a^{\ast}) - \cos(x, a) + \cos(x, b)\bigr), \tag{2.9}\]

the two sides agreeing because the length of \(a^{\ast} - a + b\) does not depend on \(x\).

The right-hand side shows what the rule is: two attractions and one repulsion, added.O. Levy and Y. Goldberg, "Linguistic Regularities in Sparse and Explicit Word Representations", CoNLL 2014, Section 6, where the sum is called a "balancing act" and replaced by a product. Where the structure comes from was explained a year later. Skip-gram with negative sampling, the method of word2vec, gives each word \(w\) a vector \(e_w\) and each context word \(c\) a vector \(\gamma_c\), and it "is implicitly factorizing a word-context matrix, whose cells are the pointwise mutual information (PMI) of the respective word and context pairs, shifted by a global constant": at its optimum, in a dimension large enough, \(\langle e_w, \gamma_c\rangle = \operatorname{PMI}(w, c) - \ln k\), with \(k\) the number of negative samples.O. Levy and Y. Goldberg, "Neural Word Embedding as Implicit Matrix Factorization", NeurIPS 2014, abstract. Offsets follow. If king differs from queen, in its affinity to every context, as man differs from woman, the vector \(e_{\text{king}} - e_{\text{queen}} - e_{\text{man}} + e_{\text{woman}}\) is orthogonal to every \(\gamma_c\), and it is zero if the context vectors span the space. A relation is a direction because the logarithm turns ratios of co-occurrence into differences and the model is linear in its vectors.

Example 2.6.2 (analogies in tables that were never trained for them). The setup: rule (2.9) on the rows of the tied matrix of the released GPT-2 small (Example 2.3.3) and of the two tables of gpt-pp, all trained only to predict the next token; the tokens carry their leading spaces.

In GPT-2's table, with the three inputs excluded, king − man + woman gives queen, Paris − France + Germany gives Berlin, walked − walk + run gives ran, better − good + bad gives worse, and bigger − big + small gives smaller. Without the exclusion the first answer to king − man + woman is king itself, and queen is second: the offset is small beside the vectors it joins.

gpt-pp's tables answer fewer of the five. Both give queen and smaller. For Paris − France + Germany the input embedding gives Berlin and the unembedding German, with Berlin second by 0.0003 of cosine. For walked − walk + run both give runs, with ran second; for better − good + bad both give Better, with worse second in the input embedding and third in the unembedding. The misses are of one kind, another form of one of the question's own words, which the exclusion of the three inputs does not cover and which these tables keep among a token's nearest neighbors (Section 2.3). ◁

For language models the modern form of the claim is about the output side, and it is stated with more care.

Definition 2.6.3 (linear representations, after Park, Choe and Veitch). A concept, such as male to female, is a variable that pairs of words differ in: (king, queen), (man, woman). It has an unembedding representation if the differences \(u_{\text{queen}} - u_{\text{king}}\), \(u_{\text{woman}} - u_{\text{man}}\) of its pairs all point along one direction. It has an embedding representation if there is a direction of the stream which, added to the final state, raises the probability of the concept and leaves unchanged the concepts that can vary independently of it. A causal inner product is one under which the directions of such independent concepts are orthogonal.

Nothing in the loss fixes an inner product on the rows of \(U\): the logits \(\langle u_v, h\rangle\) are unchanged if the rows are transformed by an invertible matrix and the states by its inverse transpose, and the probabilities are unchanged if one vector is added to every row. Euclidean cosines between rows are a property of the coordinates that training happened to choose, and in GPT-2's coordinates a shared mean dominates them. The paper shows that a causal inner product, where one exists, is tied to the covariance of the unembedding vectors, and in practice takes its inverse: whiten the rows of \(U\), then use the Euclidean inner product.K. Park, Y. J. Choe and V. Veitch, "The Linear Representation Hypothesis and the Geometry of Large Language Models", ICML 2024 (arXiv 2311.03658), Theorem 3.4.

Remark 2.6.4 (what 2026 keeps of the hypothesis). Two recalibrations. First, a direction is not the only shape: the sequels extend the geometry to categorical and hierarchical concepts, and one title states its finding, "Not All Language Model Features Are One-Dimensionally Linear".K. Park et al., "The Geometry of Categorical and Hierarchical Concepts in Large Language Models", ICLR 2025 (arXiv 2406.01506). J. Engels et al., ICLR 2025 (arXiv 2405.14860). Both from their abstracts. Second, how much of the geometry is shared between models. The strong claim of 2024 was that networks "are converging to a shared statistical model of reality in their representation spaces".M. Huh, B. Cheung, T. Wang and P. Isola, "The Platonic Representation Hypothesis", ICML 2024 (arXiv 2405.07987), Section 1. The work of 2026 keeps less. After calibration against a permutation null, the trend in global measures of similarity largely disappears and what remains is local: "representations in neural networks are converging to shared local neighborhood relationships."F. Gröger, S. Wen and M. Brbić, "Revisiting the Platonic Representation Hypothesis: An Aristotelian View", ICML 2026 (arXiv 2602.14486).

Two later sections start from the objects of this one. Section 9 treats the product of the two tables as the model's shortest circuit, the direct path of Proposition 2.5.6, and uses \(U\) as a dictionary for reading the stream at every layer. Section 10 looks for the directions that are not rows of either table, the ones a trained model computes with in its stream, and asks the question that Proposition 2.4.5 leaves open: how many almost-orthogonal directions a trained model packs into 768 dimensions, and what it pays in interference for each.

← Back to all posts