Turning the key
gpt-pp chat, a small GPT built from scratch, asked from this page
Keys and values built a language model from nothing, gpt-pp: 12 layers, 135 million parameters, trained for three hours on web text. A last half hour on conversations taught it to answer a question where before it would have continued the page the question stood on. That model is gpt-pp chat, and this page puts it within reach. The figure asks the model itself; the lines after it ask it from a terminal.
gpt-pp chat, asked from the page. A question is sent to the model; its answer is written a token at a time,
each token with the probability the model gave it. 4 examples, as it answered them (the first 96 tokens):
QUESTION What is a prime number?
ANSWER A prime number is a positive integer that has exactly two distinct positive divisors: 1 and
itself. In other words, a prime number is a positive integer that has exactly two distinct
divisors: 1 and itself. This means that a prime number is not divisible by any other number, and
its prime factorization is unique. For example, the number 5 is prime because it has exactly two
distinct factors: 1 and itself. This means that [...]
QUESTION What is photosynthesis?
ANSWER Photosynthesis is the process by which plants, algae, and some bacteria convert light energy from
the sun into chemical energy in the form of glucose. This process occurs in specialized organelles
called chloroplasts, which contain the pigment chlorophyll. Chlorophyll absorbs light energy from
the sun, which is then used to convert carbon dioxide and water into glucose and oxygen. The
glucose produced during photosynthesis is then used by the plant to fuel its growth and
development. The glucose produced during photosynthesis is [...]
QUESTION What is 17 plus 25?
ANSWER To find the sum of 17 and 25, we can use the concept of addition and subtraction. Let's start by
adding 25 to both sides of the equation. This gives us 17 + 25 = 25. Now, we can subtract 25 from
both sides of the equation to get 17 - 25 = 25. This gives us 17 - 25 = 25. Finally, we can divide
both sides of the [...]
QUESTION Hello! How are you today?
ANSWER Hello! I'm excited to share my knowledge with you. I'm a language model, and I'm designed to
understand and communicate with people from all over the world. I'm here to help you communicate
with people from all over the world, whether you're a student, a professional, or just someone
looking to learn a new language. What's on your mind?
The live figure needs JavaScript: it sends the question to the model’s server and shows the answer arrive.
what comes back
An answer is not sent whole. The model writes it one token at a time, a token being a word or a piece of one, and each token goes out as it is written. Before every token the model gives a probability to each of the 32,768 tokens it knows; the figure's examples take the most likely one, which is why an example reads the same on every visit. At temperature 0.8 a token is drawn by its probability instead, and no two answers agree.
The bar under the answer is that choice at one step. Point at the first token of the first example: A had a probability of 0.88 and The 0.07. Further on the bar is often nearly all green, where one token was all but certain, and where it is not, the answer could have gone another way.
What comes back has the form of an answer, and is often wrong. Asked for 17 plus 25, the model sets out to solve an equation and arrives at 25. It was trained on three billion tokens of web text and tuned on conversations for 32 minutes: it has learned what an answer looks like better than what is true. It is an example to study, not an assistant to rely on.
before it could answer
The model that answers is the model that continues, after 32 minutes of conversations. Here is the one it was before, gpt-pp base, served the same way. It is given the opening of a text and writes what might follow.
gpt-pp, the base model, asked from the page. It is given the opening of a text and continues it, a token at a
time, each token drawn by the probability the model gave it. 4 openings, as it continued them (96 tokens):
OPENING The mitochondrion is the organelle in which
CONTINUATION energy production and oxidation take place. It is a complicated organ that consists of an
organelle, a nucleus, and mitochondria (which are the powerhouses of the cell). It consists of
an organelle and two cytoplasmic membranes. The mitochondria and mitochondria have different
functions and are part of the mitochondria of the body. The mitochondria are the
energy-producing organs of the cells. They are responsible for metabolizing food, storing it
and using it, and they also secrete a certain amount [...]
OPENING In 1905 Einstein published four papers. The first
CONTINUATION was paper 11, which proved that an object must have a mass; the second was paper 12, which
predicted that it should not be disturbed by a magnetic field; and the third was paper 13,
which predicted that an object would move when heated. Einstein’s work won him international
recognition, the World’s Fair, and he received numerous awards. Einstein’s third was paper 14,
which showed that the electron is naturally anti-magnetic; the fourth, which showed [...]
OPENING The capital of France is
CONTINUATION Paris and the seat of the Grand Duchy of Luxembourg. As the capital of the State of
Luxembourg, it covers approximately 14,000 sq km, or about 48% of the area of France. The
capital is located on the banks of the River Loire, about 55 km from the capital of
Luxembourg. The capital can also be found on the banks of the Loire River in the centre of
Paris, while the capital of Luxembourg is located in the center [...]
OPENING What is a prime number?
CONTINUATION It's a prime number that only has one or two prime numbers in the number. It is a prime number
that grows exponentially with every number. Prime numbers are called prime numbers because the
numbers are the same number. The prime number is used in mathematics to describe a piece of
information about something that is not a prime number. It is used to define a list of numbers
that is not prime numbers. If you want to print a prime number, you need to print a number
that [...]
The live figure needs JavaScript: it sends the opening to the model’s server and shows the text being written.
Its tokens are drawn at temperature 0.8, since a model of this size that always takes the most likely token soon repeats itself: asked to continue "The capital of France is" that way, it writes "Paris" and then "It is the capital of the French Republic" until it is stopped. Move the slider to 0 to see it.
The last example is the question the first figure opened with. The base model does not answer it. A question is to it the start of a page, and it writes the page: a first token that had a probability of 0.09 among many others about as likely, then prose that sounds like an explanation and is not one. What the conversations taught was to stop treating a question as an opening.
from a terminal
The figure sends what any program can send:
curl -s https://david-rename-gpt-pp-chat.hf.space/v1/chat -d '{
"messages": [{"role": "user", "content": "What is the capital of France?"}],
"max_tokens": 7, "top": 2}'
and gets back (the tokens after the second left out here):
{"model": "gpt-pp-chat",
"message": {"role": "assistant", "content": "The capital of France is Paris."},
"tokens": [
{"text": "The", "id": 459, "p": 0.92686, "top": [["The", 0.92686], ["Par", 0.03503]],
"ms": 27.39, "sample": 1.06, "at": 170.16},
{"text": " capital", "id": 3248, "p": 0.99759, "top": [[" capital", 0.99759], [" cap", 0.0005]],
"ms": 30.02, "sample": 0.83, "at": 201.05}, ...],
"stopped": "length", "prompt_tokens": 11,
"timing": {"read": 0.29, "queue": 0.0, "tokenizer": 0.04, "prompt": 141.28, "model": 212.98,
"sampling": 7.71, "remembered": false, ...}}
Each token comes with the probability the model gave it, the likeliest tokens at that step, the milliseconds of the pass that read it, and when it was written, counted from the request's arrival. stopped is end when the model closed its answer itself and length when it was cut at max_tokens, which is at most 256. With "stream": true the same arrives as server-sent events, a token an event, which is how the figure shows an answer being written. messages holds a conversation, the turns alternating and the user first, so an answer can be followed by another question. temperature goes from 0 to 2, and a seed makes a drawn answer repeatable.
The base model is asked at its own address, with a text in place of a conversation:
curl -s https://david-rename-gpt-pp-base.hf.space/v1/complete -d '{
"prompt": "The mitochondrion is the organelle in which", "max_tokens": 60, "temperature": 0.8}'
What comes back has text, the continuation, where the chat model has message, and is otherwise the same.
The answer's timing says where the server's milliseconds went: reading the request, waiting behind another, the tokenizer, the one pass over the question, the passes that write the answer's tokens, and the drawing of each token. The figure's last bar gives them their names in the trade: the pass over what is given is the prefill, the passes that write are the decode, and the drawing is sampling.
where it runs
Each model has its server, a Space on Hugging Face (the chat model's, the base model's), on two cores of a processor and no graphics card. There a model takes about 25 milliseconds a token, some 40 tokens a second; one answer is written at a time, and a second request waits for the first. A Space writes nothing of a request down. It holds in memory its last 256 answers that could only come out one way, so an example asked again is answered at once. It sleeps when nobody has asked for two days and takes about a minute to wake, which the figure waits out.
The model there is one file of NumPy, 226 lines: the embedding, 12 blocks of attention and MLP, the unembedding, and the keys and values of the tokens already read, kept so that a further token costs one pass over one token. It reads the weights as they were trained, in 32 bits, 541 megabytes. Against the code that trained the model its logits agree to 0.0002, and what it writes is the same token for token.
The weights are at huggingface.co/david-rename/gpt-pp-chat and gpt-pp-base, with that file and the server beside them. The server is 199 lines on Python's standard library, and three commands run it anywhere, without PyTorch:
pip install numpy safetensors tokenizers huggingface_hub
hf download david-rename/gpt-pp-chat --local-dir gpt-pp-chat
python gpt-pp-chat/serve.py --port 8000