When a word opens
The first figure in Gray's Anatomy is a generalized cell, opened like a building in an architect's drawing. Twenty-four fine lines lead from its structures to their names. I can carry the picture away with me: microvilli crowding the roof like the pile of a carpet, mitochondria scattered like beans, the nucleus in its envelope with the nucleolus tucked inside like a yolk. The Golgi lies in a stack of flattened sacs. Everything has a shape. Everything stays where I put it.
The words will not stay. Desmosome, peroxisome, lysosome, centriole, reticulum. I recognize them the way I recognize street signs in a language I cannot speak: by their silhouettes, by a remembered sound. After the tenth reading, I still cannot find my way by them.
Then I read the same chart in Chinese. 线粒体: thread-grain-body. 溶酶体: dissolve-enzyme-body. 核仁: the kernel inside the kernel. 桥粒: bridge-grain, the little rivet joining two cells. The picture has not changed, but the names have begun to point. Each word reaches into something I already know, and the strange thing acquires a place to live.
the generalized cell, redrawn as a flat section: cilia and microvilli on the apical surface with a pit between them, junctions and a desmosome on the lateral membranes, folds at the base, and inside it the 24 structures of the chart, each labeled in English and in Chinese 1 cilia 纤毛 13 peroxisomes 过氧化物酶体 2 microvilli 微绒毛 14 cytosol 胞质溶胶 3 surface invagination 表面内陷 15 nuclear pore 核孔 4 actin filaments 肌动蛋白丝 16 nuclear envelope 核膜 5 vesicle 囊泡 17 nucleus 细胞核 6 cell junctions 细胞连接 18 nucleolus 核仁 7 desmosome 桥粒 19 ribosome 核糖体 8 intermediate filaments 中间丝 20 microtubules 微管 9 smooth endoplasmic reticulum 滑面内质网 21 centriole 中心粒 10 rough endoplasmic reticulum 粗面内质网 22 lysosomes 溶酶体 11 Golgi apparatus 高尔基体 23 cell surface folds 细胞表面褶皱 12 mitochondrion 线粒体 (the live figure needs JavaScript)
It is tempting to give the credit to the characters, to imagine a script of little pictures through which meaning shines. But 桥 gives me a bridge because I have crossed it in 天桥 and 过桥; 粒 gives me a grain because I have met it in 米粒. The familiarity was earned elsewhere. The chart is spending it.
The Greek roots hold those images too. Mitochondrion is mitos, thread, and chondrion, a little grain. 线粒体 rebuilds that compound out of characters a 12-year-old knows, adding 体, body, as the organelle's classifier. Nucleus is Latin for the kernel of a nut: 核. Nucleolus is the little kernel: 核仁, the seed inside the pit. Lysosome joins lysis and soma, loosening and body; 溶酶体 adds 酶, enzyme, the translator's knowledge of what the body does.
These are calques: meanings carried across a language boundary, piece by piece. Across the 24 labels, 61 ties join a Chinese character to the Greek or Latin morpheme it translates. Three carry only sound: 高尔基 for Golgi, the chart's one surname. Taiwan writes 高基氏體, where 氏 quietly warns, "a name, do not decode." Another 16 characters are additions, mostly 体 and the candid 酶.
The information was waiting in the English word all along, behind a language I had never learned. Chinese brought it into the language of ordinary life. An English reader who knows soma, lysis, desmos and mitos can feel the same release: desmosome opens into bond-body as readily as 桥粒 opens into bridge-grain.
the 24 labels of the generalized cell, each as its English morphemes (Greek and Latin, mostly) and as its Chinese characters, with a tie wherever a character translates a morpheme: 61 such calques, 3 ties by sound (高尔基, Golgi), 16 characters added (体 body, 酶 enzyme) mito chondrion desmo some lyso some nucle olus thread little grain bond body loosen body kernel little 线 粒 体 桥 粒 溶 酶 体 核 仁 thread grain body bridge grain dissolve enzyme body kernel seed kernel two vocabularies: English words plus 0 to 17 Greek and Latin roots (learned in the order of how many labels use them), and the N commonest Chinese characters (Jun Da's frequency list, 500 to 4,000). A label decodes when every meaning-bearing unit of it is known: English alone, 6 of 24; with the first 2 roots, 9; with 5, 12; with 10, 16; with all 17, 21 (the last 3 are names: Golgi, ribo-, act-). Chinese with the 1,000 commonest characters, 2 of 24; 1,500, 10; 2,000, 17; 2,500, 20; 3,000, 21; all 24 at 3,800 (绒 2672, 酶 3052, 褶 3760 are the characters beyond 2,500) (the live figure needs JavaScript)
The figure gives that feeling a count. Call each unit of meaning a token: 42 English morphemes and 48 Chinese characters cover the 24 labels. With ordinary English and no specialist roots, a reader can decode 6, built from words such as cell, surface, fold, pore, envelope and nucleus. Learn the chart's 17 roots in order of how many labels they serve, and the count rises: 9 labels with the first 2 roots, the diminutives and soma; 12 with 5; 16 with 10; 21 with all 17. The remaining 3 lead back to acts of naming: Golgi; ribo-, from ribose, which Fischer coined by rearranging letters from arabinose; and act- in actin, a coinage of 1942.
On the Chinese side, the vocabulary comes from ordinary reading, ordered by frequency. Ranks and coverage from Jun Da's Modern Chinese Character Frequency List, compiled from a corpus of some 190 million characters; the 2,500 characters of the primary-school 常用字 roughly coincide with the 2,500 commonest. The 1,000 commonest characters cover 89% of running text but unlock only 2 labels. At 1,500, the count is 10; at 2,000, 17. The 2,500 characters taught in primary school cover 98.5% of running text and decode 20 labels. The last label opens at 3,800, with 褶 in 褶皱, the pleat in "cell surface folds." Only 3 of the chart's 48 characters lie beyond 2,500: 绒, down; 酶, enzyme; and that final pleat.
The two readers can arrive at much the same place, 20 or 21 labels out of 24. They have paid for the journey at different times. The Chinese reader paid in childhood, in characters used for everything; 体 and 核 return five times each in this single chart. The English reader must learn a second language concealed inside the first, perhaps years later in a terminology course. For many readers, that course never comes. The labels remain street signs.
A name can give the mind a foothold without telling it how far there is to climb. 溶酶体 says dissolve-enzyme-body. It says nothing of acid or membrane, nothing of the cell consuming its own worn parts. In a simulated consultation in Hong Kong, Chinese medical terms, with or without a lay gloss, did not improve what patients understood. The effects of Chinese medical terms on patient comprehension of information in a simulated clinical consultation, a University of Hong Kong thesis. Terminographers find the Chinese equivalents of English medical terms more explicit in their reference, often resolving an indeterminacy in the English compound: Peters, Qian and Ding, Translating medical terminology and bilingual terminography, Lexicography 3 (2016). A word that seems to explain itself can bring inquiry to a premature rest. We feel we have arrived because we can read the sign.
Still, I want the foothold. Thread, grain, body: three familiar things with which to begin asking about an unfamiliar one.
名無固宜，約之以命，約定俗成謂之宜，異於約則謂之不宜。名無固實，約之以命實，約定俗成，謂之實名。名有固善，徑易而不拂，謂之善名。
A name has no fixed fitness. It is fixed by agreement: once the agreement settles and custom forms, the name is called fitting, and what departs from the agreement is called unfitting. A name has no fixed referent. The referent is fixed by agreement: once the agreement settles and custom forms, it is called the name of that thing. But a name has a fixed goodness. One that is direct and easy and does not rub against its thing is called a good name.
—《荀子·正名》, chapter 22. Xunzi, "Rectifying Names". 約定俗成, "settled by agreement and formed by custom", entered the language from this passage and is still the ordinary word for a convention. 拂 is to brush against, to go against the grain. The 3 sentences run from convention to referent to goodness, and only the last is not conventional.
In three sentences, Xunzi separates what custom can settle from what it cannot. We can agree which sound belongs to which thing. Agreement alone cannot make the name good. Its goodness is fixed, 固, by another measure: whether it is direct and easy, whether it passes without rubbing against what it names.
线粒体 passes easily for me. Mitochondrion catches at every syllable until someone shows me the thread and the grain. Lavoisier understood how much could depend on that passage. In the preface to his Traité élémentaire de chimie, he made the word, the idea and the fact three impressions of a single seal:
Toute science physique est nécessairement formée de trois choses : la série des faits qui constituent la science ; les idées qui les rappellent ; les mots qui les expriment. Le mot doit faire naître l'idée ; l'idée doit peindre le fait : ce sont trois empreintes d'un même cachet ; et comme ce sont les mots qui conservent les idées et qui les transmettent, il en résulte qu'on ne peut perfectionner le langage sans perfectionner la science, ni la science sans le langage.
Every physical science is necessarily made of three things: the series of facts that make up the science; the ideas that recall them; the words that express them. The word must give birth to the idea; the idea must paint the fact: three impressions of one seal. And since it is the words that keep the ideas and pass them on, it follows that one cannot perfect the language without perfecting the science, nor the science without the language.
—Lavoisier, Discours préliminaire, Traité élémentaire de chimie (1789). Robert Kerr's 1790 translation renders the last clause "we cannot improve the language of any science without at the same time improving the science itself; neither can we, on the other hand, improve a science, without improving the language or nomenclature which belongs to it." The Discours defends the nomenclature Lavoisier, Guyton de Morveau, Berthollet and Fourcroy had published in 1787, in which a compound is named by what it is made of.
Lavoisier changed the pieces from which chemistry made its names. Oil of vitriol became sulfuric acid; its salts became sulfates. The gases were named for what they made: oxygène, the acid-former; hydrogène, the water-former. A compound's name could be taken apart to disclose its composition. In the language of the machines, this was a change of tokenizer.
The Chinese chemical vocabulary developed in the 1870s carried classification into the written character. Gases carry 气, the radical for air: 氢, 氧, 氮, 氯. Metals carry 金: 钠, 钾, 钙, 铁. Solid non-metals carry 石, stone: 碳, 硅, 硫, 磷. Even an unfamiliar element offers a clue to its state and class. 氧 itself descends from 养气, the nourishing gas, the one that keeps a flame and a lung alive.
English biology kept its inheritance in Greek and Latin, down to the plurals: mitochondria, cilia, villi. For its classically educated makers, the compounds came apart into sense. As that education receded, the names remained, their familiar pieces sealed inside them. What one generation could read, another had to memorize.
Children meet a version of this difficulty before they meet a cell. Chinese counts eleven as ten-one, 十一, and twelve as ten-two. English asks for eleven and twelve, then carries its irregularities into the teens. In a comparison of Chinese and American preschoolers, the Chinese children counted further, and the gap opened where the English names became opaque. Miller, Smith, Zhu and Zhang, "Preschool origins of cross-national differences in mathematical competence: the role of number-naming systems", Psychological Science 6 (1995), 99 Chinese and 98 American 3- to 6-year-olds; a review of the counting-system literature in Dowker and Li, English and Chinese children's performance on numerical tasks, Frontiers in Psychology 9 (2019). A name can show the next step or require the child to remember it.
We know how to help a reader find the pieces. Meta-analyses of morphological instruction in schools report small to moderate gains in reading and spelling, and larger gains in vocabulary, about half a standard deviation, with the greatest benefit for the weakest readers. Bowers, Kirby and Deacon, The effects of morphological instruction on literacy skills, Review of Educational Research 80 (2010), 22 studies; Rueda-Sánchez and López-Bastida, meta-analysis of 31 studies, Anales de Psicología 32 (2016), vocabulary g = 0.50. Medical schools teach the same practice in their first weeks and call it terminology. The student is learning where to divide a word so that the pieces can speak. A tokenizer is being installed.
Before a language model reads a sentence, a tokenizer has already decided where to cut it. The resulting units are what the model receives, each represented by a vector of the same size. English words do not get "bigger embeddings" than Chinese characters. What matters is how many tokens a concept costs, and what those tokens have learned from their lives in other words.
Early tokenizers, built around English text, often broke Chinese into bytes. Across languages, the same text could cost up to 15 times as many tokens; in GPT-2, more than half the kanji required 3 tokens each. Petrov, La Malfa, Torr and Bibi, Language model tokenizers introduce unfairness between languages, NeurIPS 2023, across 17 tokenizers. The vocabulary imposed a toll on the reader before any learning could begin. Newer vocabularies have greatly reduced it.
The placement of the cuts matters as much as their number. A pretrained model can remember a complex word whole or work out its meaning from the pieces. BERT cut superbizarre so that part of the stem clung to the prefix: superb emerged from bizarre, turning strangeness into praise. Give the model the derivational cut, super and bizarre, and it generalizes better to words it has never seen. Hofmann, Pierrehumbert and Schütze, Superbizarre is not superb: derivational morphology improves BERT's interpretation of complex words, ACL 2021. I recognize the predicament. Desmosome falls into des, mos and ome in a mind that has never met desmos. The word breaks, but nothing opens.
In a study of Chinese representations, models reading characters consistently outperformed models reading whole words. The word vocabulary was scattered across entries seen once or never; the character vocabulary was smaller, denser, continually revisited. Li, Meng, Sun, Han, Yuan and Li, Is word segmentation necessary for deep learning of Chinese representations?, ACL 2019: character models win on language modeling, translation, matching and classification, and the word models lose to sparsity and out-of-vocabulary words. Across 69 languages, the size of the word inventory predicted modeling difficulty better than morphological complexity. A tokenizer's compression of text also predicts downstream performance, especially in smaller models. Mielke, Cotterell, Gorman, Roark and Eisner, What kind of language is hard to language-model?, ACL 2019; Goldman, Caciularu, Eyal, Cao, Szpektor and Tsarfaty, Unpacking tokenization: evaluating text compression and its correlation with model performance, Findings of ACL 2024. Reusable pieces let one encounter serve the next.
Arithmetic makes the same point with unusual clarity. GPT-3 divided numbers into arbitrary chunks of 1, 2 or 3 digits; Llama and PaLM used single digits. Merely aligning the chunks with place values, by inserting commas at inference, improved a frontier model's arithmetic by as much as 20%. Singh and Strouse, Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs, 2024. The old difficulty of 十一 and eleven returns inside the machine. The arrangement of the units changes what can be learned from them.
The smaller the model, the more those units matter. In the number of encounters available to us, we are the small models. And the cell chart is small enough to take this analogy apart and look inside.
For that I use Qwen2.5-0.5B, a model Alibaba trained in 2024 on English and Chinese among other languages: half a billion parameters, small enough for a laptop, open enough to examine. Qwen Team, Qwen2.5 Technical Report, 2024; the base model Qwen2.5-0.5B has 494 million parameters, 24 layers and a vocabulary of 151,936 tokens. The measurements in this section and the next are mine, made on the published weights, and the token counts use each tokenizer's published vocabulary. Of its 494 million parameters, 136 million, or 28%, belong to the embedding table. Each of the 151,936 tokens has a row of 896 numbers. Before a sentence supplies any context, this row is all the model has to go on. It is the token's accumulated familiarity.
The table gives that familiarity a visible structure. There are 9,015 rows for single Chinese characters, 16,374 for strings of 2 to 4 characters, and 68,923 for words in the Latin alphabet. Common combinations have rooms of their own: 细胞, 表面, 蛋白, 身体, 路线 and 在线 each occupy a single token. None of the chart's 24 Chinese labels does. The model assembles 线粒体 from 线, 粒 and 体; 细胞核 from 细胞 and 核. It reads 染色体, chromosome, as 染, 色, 体. This is how a Chinese reader approaches a new term: bringing familiar pieces to an unfamiliar whole.
On the English side, nucleus is the only label stored whole. Mitochondrion becomes mitochond, r, ion; desmosome becomes des, mos, ome. GPT-2 and BERT make that same cut in desmosome. In the vocabulary of bert-base-uncased, desmosome is des, ##mos, ##ome, and superbizarre is superb, ##iza, ##rre, as Hofmann, Pierrehumbert and Schütze reported. In GPT-2 and in Qwen2.5 it is des, mos, ome. The division I imagined in the reader's mind is literal in the machine.
A larger vocabulary can store more labels intact. Gemma 3 270M, Google's smallest model of 2025, spends 63% of its parameters on a table of 262,144 rows. It buys 7 of the 24 English labels whole: cilia, vesicle, cytosol, nucleus, ribosome, microtubules and lysosomes. It buys no complete Chinese label. Google's announcement of August 2025 gives 170 million embedding parameters and 100 million in the transformer blocks; the published weights have 167.8 million of 268.1 million in the table. But storing a name is its own kind of enclosure. In the single token for lysosomes, neither lysis nor soma has a separate place. The model must learn the word through the company it keeps, as an English reader without Greek must do.
| tokenizer | vocabulary | tokens, 24 labels | labels in 1 token | morpheme cuts found | cuts inside a character | ||
|---|---|---|---|---|---|---|---|
| English | Chinese | English | Chinese | English | Chinese | ||
| BERT-base-chinese2018 | 21,128 | 122 | 80 | 0 | 0 | 16/30 | 0 |
| BERT-base-uncased2018 | 30,522 | 86 | 80 | 1 | 0 | 8/30 | 0 |
| GPT-22019 | 50,257 | 81 | 195 | 1 | 0 | 8/30 | 115 |
| XLM-R2019 | 250,002 | 84 | 70 | 0 | 0 | 14/30 | 0 |
| BLOOM2022 | 250,680 | 67 | 68 | 1 | 0 | 4/30 | 0 |
| GPT-4 (cl100k)2023 | 100,277 | 76 | 128 | 1 | 0 | 9/30 | 49 |
| Llama 22023 | 32,000 | 93 | 168 | 0 | 0 | 8/30 | 88 |
| DeepSeek-V32024 | 128,815 | 56 | 66 | 1 | 0 | 3/30 | 0 |
| GPT-4o (o200k)2024 | 200,019 | 70 | 87 | 1 | 0 | 8/30 | 9 |
| Llama 32024 | 128,256 | 76 | 82 | 1 | 0 | 9/30 | 7 |
| Qwen2.52024 | 151,665 | 76 | 69 | 1 | 0 | 9/30 | 0 |
| Gemma 32025 | 262,145 | 61 | 70 | 1 | 0 | 5/30 | 0 |
The table passes the chart through 12 tokenizers, from BERT's of 2018 to Gemma's of 2025. You can watch the cost of reading Chinese fall. GPT-2 takes 195 tokens for the Chinese labels against 81 for the English, a ratio of 2.4, and makes 115 cuts inside characters. Llama 2 takes 168 against 93. GPT-4's tokenizer narrows the gap to 128 against 76, but still makes 49 cuts inside characters. Llama 3 and GPT-4o bring the Chinese count to 82 and 87, with only a handful of such cuts.
Qwen2.5, DeepSeek-V3, Gemma 3 and BLOOM leave every character intact. They read the Chinese chart in 69, 66, 70 and 68 tokens: fewer than the 80 a reader would need at one token per character, because some words are stored whole. English ranges from 122 tokens under BERT's Chinese vocabulary, which fragments it heavily, to 56 under DeepSeek-V3. In the newer vocabularies, the two scripts can finish within a few tokens of each other. The toll has largely gone.
The cuts still leave their marks. Inside the 24 English labels are 30 morpheme boundaries: mito|chondrion, desmo|some, lyso|some, nucle|olus. The byte-pair tokenizers find only 8 or 9. Their successes include micro|villi, act|in, junction|s, per|oxisomes, nucle|olus, the -ic of endoplasm|ic and the -ules of microtubules. None finds mito|chondrion, desmo|some, lyso|some or cyto|sol. Three quarters of the cuts they make inside English words pass through a morpheme: mitochond + r + ion, cy + tos + ol, ves + icle. XLM-R, whose vocabulary was built by a different algorithm (a unigram model rather than byte-pair merges), finds 14 of the 30 boundaries, among them mito|chondrion, cyto|sol and ribo|some, and reads mitochondrion as mito, cho, ndri, on. BERT's Chinese vocabulary finds 16 by shredding every English word into pieces of 2 or 3 letters. A word can become cheaper to read without becoming easier to take apart.
A byte-pair vocabulary grows by merging the commonest adjacent pairs until its budget runs out. Frequency sometimes rediscovers a meaningful joint. Qwen has osome, shared by ribosome and lysosome, and ase, shared by lipase and protease. The Greek soma and the chemists' suffix surface in the statistics; ase even sits among the nearest neighbors of 酶. Yet chromosome, analysis, paralysis, catalyst and mitochondrial each occupy a single token. Those encounters contribute nothing to a separate input row for soma, lysis or mito. Each word keeps its earnings to itself.
Chinese sometimes does the same. Qwen stores 路线, 在线 and 身体 whole. But enough other encounters remain for 线 and 体 to acquire meanings they can carry into the next word.
What a row has learned is visible in its neighbors. Closest to 体 is 體, its traditional form, followed by body, Body and bodies. Closest to 核 is nuclear, with nucleus fifth. 酶 finds enzyme and enzymes, then the suffix ase in fifth place. Across scripts, the table has brought meanings together.
The neighborhood of mitochond is lonelier. First comes mitochondrial; then a Tai Viet letter, a stray byte, a Hangul syllable, noise at a cosine of 0.37. The row has kept too little company. 褶, the rare pleat in the chart's last label, finds 皱 and folds as its two nearest neighbors before it too runs into noise. The other pieces of mitochondrion have wandered elsewhere: ion lives with ions and IONS, r with R, mos with mosaic. The thread and the grain have no address here.
Does the model still hear a name's parts inside the name? I give it each label in a short frame, term: mitochondrion or 术语：线粒体, and average the hidden states of the label's tokens at a middle layer. I do the same for the chart's 39 Chinese units, characters and groups such as 线, 粒, 体 and 细胞, and their 38 English glosses, such as thread, grain, body and cell. Then I rank the units by their cosine similarity to each label and look for its own pieces in the list. Layer 12 of 24, every vector centered on the mean of the whole set before the cosines are taken. The ranks at each of the other layers tell the same story, and at the input layer, before any attention, a Chinese label's characters already rank 2nd and an English label's morphemes 4th. Gemma 3 270M is too weak for this test: its hidden states find the right translation less than half the time.
First, a check that the model knows the terms. At its best layer, the nearest Chinese label to an English one is the correct translation for 22 of the 24 labels. In the other direction, it finds 19. The two languages reach much the same objects. The question is what they bring with them.
A Chinese label's own characters rank at a median of second among the 39 units; their English meanings rank third among the 38 glosses. 线粒体 is closer to 线 than to any other character in the chart, and closer to thread than to any other gloss. An English label's components rank sixth by both their English meanings and their Chinese forms. Mitochondrion finds thread ninth, grain sixteenth and body twenty-sixth. Desmosome finds bridge twenty-third and grain thirty-third.
The separation follows the old language boundary. Where the English component is an everyday English word, its meaning ranks second, as well as on the Chinese side: plasma membrane finds membrane first, centriole finds center first, cell junctions finds junction first. Where the component is Greek or Latin, its meaning falls to a median rank of ninth and its Chinese equivalent to eighteenth. Through the Chinese label, those same units rank second. The meaning is available; the English name has a harder time calling it back.
Word families leave a similar trace. The 5 labels ending in 体 are closer to one another than to the rest of the chart by 0.15 in cosine; the 5 containing 核, by 0.39. The 4 English labels built on nucle- also gather together, by 0.31. Their stem remains visible in nuclear, nucleus and nucleolus, and the tokenizer keeps it in one piece.
The 4 labels ending in -some scarcely gather at all: desmosome, peroxisome, ribosome and lysosome are closer by only 0.03. Their common ending has been divided four ways, into ome, omes, osome and osomes. Little in their measured resemblance preserves the family of bodies. Here, in the geometry of a small model, I find the difficulty I felt at the chart: names I can recognize, and names whose parts I can still hear.
An embedding row carries a history of encounters. Each appearance of a token supplies another occasion to learn; Adam, the optimizer used to train these models, normalizes the gradient, making the learning rate central to the size of the step. Kingma and Ba, Adam: a method for stochastic optimization, ICLR 2015: the magnitude of each update is approximately bounded by the step size and invariant to the scale of the gradient. The measure below, a row's distance from the table's mean, follows Land and Bartolo; in Gemma 3 270M, whose table is tied to its output layer like Qwen2.5-0.5B's, the same correlation is -0.13, present and faint. A row's distance from where it began can therefore bear a trace of how often it was called upon, and how consistently those encounters pulled it in one direction.
That trace survives in the finished table. Center Qwen's table on its mean and measure each character's distance from the center. Among the 7,325 single-character rows with a rank in Jun Da's frequency list, distance falls as rank rises: a Spearman correlation of -0.46. The average distance is 0.50 for the 100 commonest characters, 0.48 for ranks 1,001 to 2,500, and 0.44 beyond 5,000. Of all 7,325 characters, 体 has traveled farthest. 面 and 线 are in the top 2%, 核 in the top 5%. 粒 is at the 69th percentile, 酶 at the 42nd, 褶 at the 39th. The rarer pieces remain closer to the center, where their neighbors are harder to recognize.
The signature is stronger in Qwen3-14B, the 14-billion-parameter member of the family. Its input table is separate from its output layer, so a token's appearances supply the direct learning signal to its input row. The correlation is -0.92 for Chinese characters and 0.55 for English words. Character distance falls band by band, from 1.83 for the commonest 100 to 1.04 beyond rank 5,000. Frequency has left something close to a contour map in the table. Land and Bartolo used a related signature to find under-trained tokens in a dozen open models: rows still near where initialization had left them. Land and Bartolo, Fishing for Magikarp: automatically detecting under-trained tokens in large language models, EMNLP 2024.
Back in the smaller Qwen model, the English evidence is less clean. My frequency list blends several corpora rather than reproducing the model's training text. Even so, the three highest bands fall in order: 0.49 for words occurring at least once in 1,000 words, 0.46 for once in 10,000, and 0.43 for once in 100,000.
With a model saved at intervals during training, we can watch familiarity accumulate. EleutherAI's Pythia-160M read 300 billion tokens from the Pile over 143,000 steps of roughly 2 million tokens each. Checkpoints preserve steps 1, 2, 4 and so on to 512, then every thousandth step. Biderman et al., Pythia: a suite for analyzing large language models across training and scaling, ICML 2023; the checkpoints are public. The sample is the pile-10k set of 10,000 documents, tokenized with the model's own tokenizer; the bits are the model's loss on the label's tokens inside the frames "Inside the cell, the …" and "细胞内有…", summed over the label. I count its 50,000 tokens in a sample of 10,000 Pile documents, 15 million tokens altogether, and follow the rows in each frequency band toward their final positions.
By step 32,000, after 67 billion tokens, the rows seen at least 10,000 times in the sample have reached a cosine similarity of 0.90 to their final positions. Those seen 1,000 to 9,999 times are at 0.87; those seen 100 to 999 times, at 0.78; 10 to 99 times, at 0.70; fewer than 10 times, at 0.64. At step 64,000, nearly all of the two commonest bands have reached a cosine of at least 0.9. So have 75% of the next band, 9% of the next, and 1% of the rarest. The frequent tokens settle first. The rare ones are still finding their way when training draws toward its end.
The chart's loss records the same uneven education. In a sentence, the loss on an English label falls from 38 bits at initialization to 17 at the end; most of that descent is over by step 8,000. On a Chinese label, it falls from 92 bits to 33. The steepest drop, between steps 128 and 512, mostly reflects learning which UTF-8 bytes can follow which. There is very little Chinese in the Pile from which to learn the words themselves. In the sample's 15 million tokens, 体 appears 35 times, mitochond 4 times, cell 2,790 times. One of the three bytes of 酶 never appears at all. No later checkpoint can supply an encounter the corpus does not contain.
A reader also learns by meeting a word again, though an encounter is no guarantee of possession. A meta-analysis of incidental learning during ordinary reading found that readers learned something of about 15% of the unknown words they encountered. Swanborn and de Glopper, Incidental word learning while reading: a meta-analysis, Review of Educational Research 69 (1999), 20 experiments. Much depends on which pieces are available to be met again. The tokenizer determines how far one meeting can reach.
In a million characters of modern Chinese, 体 occurs 1,415 times; 线, 508; 核, 217; 粒, 53; 酶, 9. A reader of 10 million characters a year meets 体 some 14,000 times and 粒 some 500, in 身体 and 体育, 米粒 and 颗粒. Years before the cell chart arrives, ordinary reading has been preparing its vocabulary. The body and the grain are already familiar.
In 10 million English words, mitochondrial occurs about 19 times and chromosome about 30. If the reader stores them whole, the encounters stay with those words. Analysis may arrive 800 times a year while the loosening inside lysis remains unheard. English frequencies from Speer's wordfreq, a blend of corpora (books, news, subtitles, Wikipedia, the web), on the Zipf scale: mitochondrial 3.3, chromosome 3.5, analysis 4.9, perfect 5.2; a Zipf of 4 is 1 occurrence in 100,000 words. The Chinese counts are per million characters of Jun Da's 193.5 million. The yearly counts take the 10 million words a year of the arithmetic below, and 10 million characters, which is fewer words, for the Chinese reader. Six of the chart's 17 roots have a host in everyday English occurring at least once in 100,000 words: the diminutive in article, particle, muscle and molecule; plasm in plastic; -in in protein; per in perfect; sol in solution and aerosol; lysis in analysis, paralysis and catalyst. The other 11 inhabit rarer words, or have no familiar host: soma in chromosome, chondros in hypochondriac, cilium in supercilious, villus in velvet, vagina in vanilla, desmos nowhere at all. On the Chinese side, 46 of 48 characters clear the same frequency threshold. The exceptions, 酶 and 褶, are among the last the figure's slider reaches.
Teaching morphology gives the reader a new way to divide a familiar past. Analysis becomes a loosening; the hypochondriac's complaint lies beneath the cartilage of the ribs; velvet carries its shaggy hair. The reader receives lysis, chondros and villus with a lifetime of encounters already behind them, suddenly available for use. It feels like back pay. Hofmann's DelBERT makes the same move in a machine: the same corpus, divided into more useful units, improves its handling of words it has never seen.
The machines make it tempting to prescribe more reading. Read everything, in every form: books and pictures and sound, an ever wider world pouring in. If intelligence grew from that abundance, perhaps we could grow it in ourselves by opening the same floodgates.
A lifetime has narrower banks. Speech carries about 39 bits a second across the languages measured, faster languages giving less information per syllable and slower ones more. Coupé, Oh, Dediu and Pellegrino, Different languages, similar encoding efficiency, Science Advances 5 (2019), 17 languages. Silent reading of non-fiction runs at about 240 words a minute. Keep that pace for 2 hours a day, every day for 50 years, and you have read roughly 500 million words. A training run of 10 trillion tokens is still about four orders of magnitude away. Brysbaert, How many words do we read per minute?, Journal of Memory and Language 109 (2019), a meta-analysis of 190 studies: 238 words a minute for non-fiction, 260 for fiction. There is no spare lifetime in which to close the gap. A mixture of experts can economize on a model's computation; it cannot give a person more hours to read.
What we can borrow is the attention paid to each encounter. A tokenizer determines the units in which experience arrives. Training asks the model to predict what comes next, then measures its prediction against what actually follows. That measure of error is the loss. The text becomes a succession of attempts and corrections. Each error has somewhere to go.
This changes how I understand my own ease with the Chinese chart. At first I thought the words had simply soaked in. I had looked, and they had stayed. But familiarity with an open answer sheet can be deceptive. Students who try to answer a question before being taught the answer remember it better than students who spend the same time studying: about 10% better on tests a day or two later. The benefit persists even when the comparison includes only questions they initially answered wrong. Kornell, Hays and Bjork, "Unsuccessful retrieval attempts enhance subsequent learning", Journal of Experimental Psychology: Learning, Memory, and Cognition 35 (2009); Richland, Kornell and Kao, The pretesting effect, Journal of Experimental Psychology: Applied 15 (2009), 5 experiments on a text about vision. The failed attempt makes a place for the correction.
Perhaps that is what the Chinese labels gave me. Each word began a small question: thread, grain, what? The picture completed it. I can make that exchange deliberate. Cover the label and try to name the shape. Cover the picture and try to draw what the word describes. Let the answer arrive after the attempt, while there is still a question for it to answer.
Space the attempts; shuffle the labels. In the analogy with training, these are choices about the schedule, and sleep offers its own replay. Choose the reading with care, too. Small models trained on text of textbook quality have matched much larger models on the tasks studied. Gunasekar et al., Textbooks are all you need, 2023, the first of the phi models: 1.3 billion parameters, trained on a few billion tokens of curated and synthetic textbook text. A person's reading life is small enough for each book to claim a meaningful share of it. The explanation written afterwards is another occasion to work: to discover which connections can be made without looking, and where the thread breaks.
These practices offer something more specific than a general enlargement of intelligence. The brain-training industry promised that larger transformation. A review of the evidence its companies cited found extensive gains on the trained tasks, smaller gains on closely related ones, and little transfer to distant tasks or everyday life. Simons, Boot, Charness, Gathercole, Chabris, Hambrick and Stine-Morrow, Do "brain-training" programs work?, Psychological Science in the Public Interest 17 (2016). Practice takes the shape of what is practiced. A vocabulary can become more articulate; a prediction can become more exact. Neither gives us a dial for turning up the whole mind.
The cell brings the ambition back to a human scale. Twenty-four structures, one picture, two ways of naming what is there. The useful lesson from the machines lies in how we divide an encounter, what we try to predict, and what we allow the mistake to teach us. The tokenizer and the loss: pieces we can carry forward, and an attempt that gives the next encounter somewhere to begin.
A good name, Xunzi wrote, is direct and easy and does not rub: 徑易而不拂. I look again at 线粒体. Thread, grain, body. The fine line on the page still joins a name to a shape. Now it reaches something I know. The picture gives the word a body; the word gives memory a way back.