Tuning the piano

How a decision model is trained, from a released language model, on four graphics cards in an afternoon Nov 13, 2026

1. what is known, and what can be rebuilt
2. the request
3. cases whose chances are known
4. pretraining is already the right loss
5. what a checker's reward does
6. three ways to read a decision
7. adapters
8. the loss, and what Clef's adds
9. the mask
10. doubt with reasoning
11. on a larger backbone
12. what the name contains

The forecaster often finds himself in the position of choosing to ignore the verification system or to let it do the forecasting for him by "hedging" or "playing the system." This may lead the forecaster to forecast something other than what he thinks will occur.

—Glenn W. Brier, "Verification of Forecasts Expressed in Terms of Probability" (1950). Monthly Weather Review 78, p. 1. Brier was at the United States Weather Bureau; the paper is 3 pages long, and the score that carries his name is its equation 2.

what is known, and what can be rebuilt

The logical piano argued about the models that decide without writing, Jev and the open ones made after it, and left their training in prose. What follows is the same matter with the cover off: the request such a model is given, the three ways of reading an answer out of a transformer, the adapters that are trained and the weights that are not, the loss and what each term of it does, the mask that lets many questions be read in one pass, and the one place where a lottery comes back in, which is reasoning. Each step is derived, then shown as the code that ran, then measured. The measurements were made for the purpose, on four graphics cards of 32 gigabytes in an afternoon, from a released language model of 0.6 billion parameters and from public data, and a reader with one such card can repeat every one of them.

The sources are uneven, and it is as well to say at the start how uneven. Of Jev itself nothing technical is published: its makers say that it is a transformer, that it is trained on synthetic cases only, and that the training is called RLCD, a name the company itself has called a north star and not an algorithm. Clef is another matter. Cloudflare released the weights of both sizes, the file that defines the model, and a description of the recipe; the data and the training code it kept. Cloudflare, 1 October 2026: Clef on a Qwen backbone of 27 billion parameters, Clef-flash on one of 9 billion, under the Apache license. The model file is joint_schema_model.py in the repository of the weights. An outside reimplementation of the finetuning exists and says of itself that it has never been run on a graphics card. Open weights, then, more than open source. Laya, the encoder, has a model card that states its training in four lines. And OpenJev-RLCD is a paper with its code, the one attempt in the open to say what RLCD could be for a model that reasons before it answers. Zhimin Gao and Pichao Wang, "OpenJev-RLCD: A Working RLCD Implementation", arXiv:2609.38850 (September 2026). A later manuscript by the same authors renames its first stage and extends it to four small models; where the two differ I follow the paper and say so. So Clef gives the architecture, Laya and Clef give two losses, OpenJev gives the join with reasoning, and Jev's own recipe stays behind its door. What can be rebuilt is the family's common design, and whether the rebuilt thing behaves as the theory of the essay said it must.

A few words are used throughout and had better be fixed now. A token is the piece of a word in which a language model counts text. A model's weights, or parameters, are the numbers that training adjusts. Pretraining is the long first training in which a network learns to continue text, and post-training is whatever is done to it afterwards to give it a use. The backbone is the pretrained network a decision model is built on; a head is a small network set on top of it to read answers out; an adapter is a small trainable correction added to a weight that is itself left frozen. The hidden state at a position is the list of numbers the backbone's last layer leaves there. Logits are the raw numbers a network gives to its options, and the softmax turns them into probabilities: each is exponentiated and the results are divided by their sum. A loss is the number that training makes small, and a score, here, is a loss with its sign changed, so that more is better. A model is calibrated when its probabilities come true at the rate they state.

Three acronyms mark the lineage. RLHF, reinforcement learning from human feedback, trains a model on a reward that predicts which of two answers a person would prefer; it is what made the chat assistants. Reinforcement learning from verifiable rewards puts a checker in the person's place, a program that marks an answer right or wrong; it is what made the systems that write code and proofs. RLCD, reinforcement learning for calibrated decisions, is the name Jev's makers give to their own method, and what follows is an attempt to say what it can be.

It is rebuilt here on a small scale and on real material. The backbone is Qwen3-0.6B, a released chat model of 596,049,920 weights, the smallest of the line that Clef's backbones belong to. The cases are of three public kinds, chosen because in two of them the chance of an answer is a known number and not a manner of speaking. The model that comes out is called jev-pp below, Jevissimo if it has to be said aloud: pp for pianissimo, a Jev of the smallest size. All the numbers below come from these runs. The code, with the results of every run, is at github.com/d-/jev-pp. Where a run disagrees with the essay, or with the paper it follows, the disagreement is reported as it came.

the request

A decision model is asked with two things: a state, which is any text or record, and a set of typed questions about it. The types of the Jev interface are three, and the open models kept them. A noul is a proposition, answered true or false. A choice is one of a list of options that the caller supplies with the request, each with a description. A score is a level on an ordered scale whose levels the caller describes. The answer to each question is a probability for each of its options, and the request costs no output tokens, because nothing is written: Clef's interface returns, beside the probabilities, a usage record in which the count of output tokens is 0. A request to Clef, in the form of the Jev interface that it copies, is a record with the fields state and questions, each question having a type, instructions and criteria. For a score the answer returned is the mean level, \(\sum_i i\,p_i\), and the confidence is the largest \(p_i\), which is a statement about the spread of the probabilities and not, by itself, about how often they are right.

Two properties of this request decide the rest of the design. The first is that the options arrive with the request. A classifier of 2019 had a fixed set of classes and a row of weights for each; here the classes are text that the model has never seen as classes, so they must be read, by the same network that reads the state, and an answer must be found among them by something other than an index. The second is that the questions are many and the state is one. A ticket of 2,000 tokens with eight questions about it should be read once, not eight times, and then the questions sit in one sequence and can see each other unless something stops them.

The three kinds of case used below have one type each. A moderation queue: the state is a comment from the Civil Comments set, and the seven questions are nouls, one for each thing its raters were asked, that a rater would call the comment toxic, severely toxic, obscene, a threat, an insult, an attack on someone's identity, or sexually explicit. Inference: the state is a premise and a hypothesis from ChaosNLI, and the question is a choice of three, the relation a reader would see between them. Reviews: the state is a review from Yelp, and the question is a score of five levels, the stars its customer gave. Written out for the model, a request is the state, then each question with its options and a slot where the answer belongs:

<|im_start|>system
Read the state, then answer every question about it. Each answer must be exactly one of that question's options.<|im_end|>
<|im_start|>user
STATE:
Funny how our leaders are ignorant of history.  Sad, actually.

QUESTIONS:

QUESTION: A rater would say this comment threatens someone.
OPTIONS: yes, no
ANSWER:

QUESTION: A rater would say this comment insults someone.
OPTIONS: yes, no
ANSWER:

QUESTION: A rater would call this comment sexually explicit.
OPTIONS: yes, no
ANSWER:

QUESTION: A rater would call this comment severely toxic: very hateful or aggressive.
OPTIONS: yes, no
ANSWER:

QUESTION: A rater would call this comment obscene.
OPTIONS: yes, no
ANSWER:

QUESTION: A rater would call this comment toxic: rude or disrespectful enough to make people leave a discussion.
OPTIONS: yes, no
ANSWER:

QUESTION: A rater would say this comment attacks someone for their identity.
OPTIONS: yes, no
ANSWER:

The function that writes it keeps three things for each question as it goes: where the answer is read, where each option's own token stands in the text, and what that token is.

def encode(tok, record, max_state_tokens=256):
    """A record as token ids, the state first and the questions after it. Returns the ids, for each token the question
    it belongs to (-1: the state), and for each question where its answer is read, where each option's own token
    stands, and what that token is."""
    ids = lambda text: tok(text, add_special_tokens=False).input_ids
    out = ids(f"<|im_start|>system\n{SYSTEM}<|im_end|>\n<|im_start|>user\nSTATE:\n") + ids(record["state"])[:max_state_tokens] + ids("\n\nQUESTIONS:")
    owner, fields = [-1] * len(out), []
    for q, question in enumerate(record["questions"]):
        start = len(out)
        out += ids("\n\nQUESTION: " + question["instructions"] + "\nOPTIONS:")
        labels = [ids(" " + label) for label, _ in question["options"]]        # " yes" is 1 token; " 3" is a space, then 3
        listed, at = any(text for _, text in question["options"]), []
        for k, ((_, text), label) in enumerate(zip(question["options"], labels)):
            out += ids("\n-") if listed else (ids(",") if k else [])
            out += label
            at.append(len(out) - 1)
            out += ids(": " + text) if listed else []
        out += ids("\nANSWER:") + labels[0][:-1]                                # what every answer begins with is given
        owner += [q] * (len(out) - start)
        fields.append({"read": len(out) - 1, "options": at, "option_ids": [label[-1] for label in labels]})
        assert len(set(fields[-1]["option_ids"])) == len(labels) and all(label[:-1] == labels[0][:-1] for label in labels)
    return out, owner, fields

cases whose chances are known

The essay's argument turned on a distinction that ordinary data cannot show. A model can mean what it says, so that of the cases to which it gives 0.7 about 70 in 100 come, and still say very little, giving 0.7 to everything; calibration measured against outcomes sees the first property and is blind to the second. To see both one needs cases in which the chance itself is known, and there is a kind of public data in which it nearly is: data that many people judged. Each comment of Civil Comments was shown to several raters, most often five, six or ten to judge by the fractions, and the set records for each of the seven attributes the share of them who said yes. Civil Comments: 1,804,874 comments from news sites, released by Jigsaw in 2019. A share over five raters is a coarse estimate of a chance, and I use it as the chance all the same; the error this puts into the comparisons below is the same for every training compared. Each of the 1,599 items of ChaosNLI was labeled by 100 readers, because its authors wished to show that disagreement about inference is a property of the item and not noise. Yixin Nie, Xiang Zhou and Mohit Bansal, "What Can We Learn from Collective Human Opinions on Natural Language Inference Data?" (EMNLP 2020). The part used here is the one drawn from MultiNLI. Its items were chosen for being disputed: the largest of the three shares is 0.65 on average. In both, the question put to the model is about a rater drawn at random, and the chance that this rater says yes is the share.

Training never sees a share. Each time a case is visited, one rater is drawn by lot from the shares, and that verdict is the label: one outcome a case, as in the world, where a ticket is escalated or it is not and nobody records the chance. The shares are kept for measuring. A report is compared with what came, which gives the usual calibration, and with the shares, which gives a second number that I shall call the truth gap: the root of the mean squared distance between the probabilities reported and the shares. For a question with two options this is the root-mean-square difference between the probability of yes and the share who said yes; with more options the squared differences are summed over options and halved. My first runs used the mean absolute difference, and by that yardstick a model that answered no to everything with certainty came out nearer the truth than a model that reported the average share, most shares being 0. A distance from the truth has to be proper too: the squared one is least, on average, only for the reporter who states the chances. A reporter that never read the comment and gave every question its average share would have a calibration gap near 0 and a large truth gap, and the tables below carry that reporter as their first row, because on these comments it is hard to beat. The reviews are the control in the other direction. A review has one customer, there are no shares, and the model can be measured against outcomes only, as it would be in any deployment.

The comments were taken so that half had been called toxic by at least one rater, which is the composition of a queue of flagged comments and not of the site. The test sets are 1,500 comments (10,500 questions), 400 inference items and 1,500 reviews, held apart from everything trained on; ChaosNLI being small, 1,000 of its items are trained on, with a fresh reader drawn at every visit, and 199 are kept for tuning.

pretraining is already the right loss

Begin where the essay began, with the claim that a language model is a calibrated decision model before anyone trains it to decide. Pretraining shows the network a text and, at every position, charges it the negative logarithm of the probability it gave to the token that came. Suppose the next token at some position really comes with chances \(q_1,\dots,q_V\) and the model reports \(u\). Its expected score is

\[\sum_k q_k \log u_k \;=\; -H(q)\;-\;\mathrm{KL}(q\,\|\,u),\qquad H(q)=-\sum_k q_k\log q_k,\quad \mathrm{KL}(q\,\|\,u)=\sum_k q_k\log\frac{q_k}{u_k}\;\ge\;0,\]

where \(H\) is the entropy of the truth, which the report cannot change, and KL is the divergence named for Kullback and Leibler, a distance of the report from the truth that is never negative. It vanishes only at \(u=q\): the report that scores best is the truth, whatever the truth is. That is what statisticians mean by calling the logarithm a proper score. The slope the network climbs says the same thing locally. With \(u=\mathrm{softmax}(z)\) and \(e\) the vector that is 1 at the token \(y\) that came and 0 elsewhere,

\[\frac{\partial}{\partial z_k}\log u_y \;=\; e_k-u_k,\qquad\text{and on average over what comes}\qquad q_k-u_k .\]

Each logit is pushed up by exactly the amount by which its probability understates the chance, and the push is 0 when there is nothing to correct. Nothing else about the method matters to the argument: a network trained this way on enough text has had every one of its probabilities, at every position of every document, pressed toward a frequency.

Whether the released models bear this out can be measured with no training at all. Take a model as pretrained, before its maker turned it into an assistant; Qwen publishes these beside the chat models, with "Base" in the name. Run it over text, and at each position record the probability of its first choice and whether that choice came. Two texts were used: the test part of WikiText-103, which a model may well have met in training, and the text of "The logical piano", which was published after every one of these models and which none of them can have read.

model	weights (billions)	WikiText: right	says	gap	the essay: right	says	gap
Qwen3-0.6B, as pretrained	0.60	0.475	0.481	0.006	0.330	0.336	0.017
Qwen3-0.6B, chat model	0.60	0.424	0.523	0.098	0.299	0.399	0.100
Qwen3-1.7B, as pretrained	1.72	0.511	0.518	0.007	0.362	0.364	0.017
Qwen3-1.7B, chat model	1.72	0.481	0.656	0.175	0.336	0.464	0.128
Qwen3-4B, as pretrained	4.02	0.539	0.552	0.013	0.386	0.389	0.011
Qwen3-4B, chat model	4.02	0.511	0.693	0.182	0.360	0.498	0.138

"Right" is how often the model's first choice was the token that came, "says" the probability it gave that choice on average, and "gap" the expected calibration error: the positions are sorted into 15 bands by what the model said, and the gap is the average distance, band by band, between what was said and how often it came. The pretrained model of 0.6 billion weights gives its first choice 0.481 on WikiText and is right 0.475 of the time, over 134,400 positions; the gap is 0.006. On the essay, which is harder for it (5.21 bits of surprise a token against 3.84), it says 0.336 and finds 0.330, a gap of 0.017 on text it has never seen. It knows less there and says less, by the same amount. The larger pretrained models are sharper and no less honest. Now read down to the chat models, which are these same networks after their maker's post-training: on plain text the 0.6-billion chat model says 0.523 and finds 0.424, a gap of 0.098, some 17 times the pretrained model's. Whatever the post-training was for, it was paid for in this coin.

The second measurement is the one OpenAI printed for GPT-4, repeated on models three orders of magnitude smaller: questions with four options from the MMLU examination, written as plain text with five worked examples before each, the chances read at the word "Answer:" for the four letters.

model	as text: right	says	gap	temperature	as a chat turn: right	says	gap
Qwen3-0.6B, as pretrained	0.521	0.532	0.024	1.00
Qwen3-0.6B, chat model	0.471	0.692	0.221	2.71	0.394	0.768	0.375
Qwen3-1.7B, as pretrained	0.630	0.637	0.018	1.00
Qwen3-1.7B, chat model	0.595	0.869	0.274	3.66	0.482	0.949	0.467
Qwen3-4B, as pretrained	0.736	0.743	0.025	1.11
Qwen3-4B, chat model	0.701	0.873	0.172	2.46	0.639	0.954	0.315

The pretrained model of 0.6 billion weights answers 0.521 of 3,000 questions and says 0.532: a gap of 0.024, and the single temperature that would best rescale its logits is 1.00, which is to say that there is nothing to rescale. (A temperature divides every logit before the softmax: above 1 it flattens the probabilities, below 1 it sharpens them, and fitting one number in this way on held-out cases is the standard repair for a model that is too sure.) The chat model made from it, given the same text, answers 0.471 and says 0.692, a gap of 0.221; asked in its own format, as an assistant, it answers 0.394 and says 0.768, a gap of 0.375. OpenAI's figures for GPT-4 were 0.007 before and 0.074 after. On a model three orders of magnitude smaller the same thing happens and more of it: the calibrated decision is not something to be added to a language model, and what has to be explained about decision models is less how they come to mean what they say than how the others ceased to.

Reading an answer this way, from the model's own next-token logits at the place where the answer would be written, kept for the tokens of the options and renormalized, is the second of the three ways the essay listed, and it needs five lines:

def read_logits(model, hidden, b):
    """Way 1, reading the logits: at each question's ANS token, the model's own next-token logits, kept only for
    that question's option tokens. Returns [n, Q, MAX_OPTIONS], -inf where there is no option."""
    at = hidden[torch.arange(hidden.shape[0], device=hidden.device)[:, None], b["read"]]    # [n, Q, width]
    z = torch.gather(model.logits(at), 2, b["option_ids"])                     # [n, Q, MAX_OPTIONS]
    return z.masked_fill(b["options"] < 0, float("-inf"))

Here hidden is the backbone's last layer at every position of the request, b["read"] holds the position of each question's answer slot, and b["option_ids"] the tokens of its options: " yes" and " no" for a noul, " A", " B", " C" for a choice, the digits for a score. Applied to the chat model and to the requests of the three kinds, with no training:

The result is a warning against taking the previous paragraph too far. On the comments the released model's first answer is the one that came for 10.2% of the questions, where answering no every time without reading would be right on 94.1%: it says yes to 95% of them, with a mean confidence of 0.69, and no rescaling of its logits mends it. On inference it is right on 44.5% where the most that can be had, by naming the readers' favorite every time, is 63.5%, and it says 0.82; on the reviews it names the stars 33.3% of the time and is off by 1.12 stars on average. The model has knowledge that bears on all three, as the trainings below show within a few hundred steps, and a pretrained network has calibration on text. Neither helps on a format it has never met, a list of questions with empty answer slots, read at a position where no document of its training ever asked it anything. Reading the logits works on large models with familiar prompts. A small model has to be taught the request, and the question is by what loss.

what a checker's reward does

The systems that write code and proofs are trained by a checker: an answer is drawn by lot from what the model says, a program marks it right or wrong, and the answer is made more probable if its mark beat the marks of the other answers drawn with it. Put a decision model under that reward. It says \(u\), an answer \(a\) is drawn from \(u\), and it is paid 1 if \(a\) is what came. With the truth \(q\), its expected pay is

\[\mathbb{E}\,[\,\mathbf 1(a=y)\,]\;=\;\sum_k q_k\,u_k ,\]

which is linear in \(u\), and a linear function on the set of all probability vectors, which is a simplex, is greatest at a corner: all of \(u\) on the likeliest option. A model that knows a comment would be called toxic by six raters in ten is paid most for saying ten in ten. The slope shows how it gets there. On average over what comes,

\[\frac{\partial}{\partial z_k}\sum_j q_j u_j\;=\;u_k\Big(q_k-\sum_j q_j u_j\Big):\]

an option gains in proportion to the probability it already has and to how far its chance exceeds the model's present expected pay. Biologists know this equation as the replicator dynamics, under which the fittest type takes the whole population, and it has no rest point inside the simplex unless all the chances are equal. In code, the reward is this, with a baseline for each drawn answer made of the others drawn with it:

def checker_surrogate(z, y, draws=4):
    """Reinforcement from a checker: draw answers from u, pay 1 for each that is the one that came, and raise the
    log-chance of the answers paid more than the others of their group. Its mean gradient is that of linear_score."""
    u = chances(z)
    a = torch.multinomial(u.reshape(-1, u.shape[-1]), draws, replacement=True).view(*u.shape[:-1], draws)
    paid = (a == y[..., None]).float()
    advantage = paid - (paid.sum(-1, keepdim=True) - paid) / (draws - 1)       # against the mean of the others
    return -(advantage * z.log_softmax(-1).gather(-1, a)).mean(-1)             # a loss: minimize

A copy of the released model was trained this way on the three kinds of case, with the adapters described below, and another copy by the logarithm, for the same 1,000 steps on the same cases. The checker's model ends by answering no to every question about every comment: 100.0% of its first answers are no, and the probability it gives them rounds to 1.000. This is right 94.1% of the time, because 94.1% of the verdicts are no, and it is what the algebra said the reward was asking for: the likeliest answer, with all the probability on it. The model trained by the logarithm is right on 94.3% and says 0.954. An examination would not tell the two apart. The rule of the documentation does, act alone when the model says at least 0.9. On the question raters most often dispute, whether a comment is toxic, the checker's model acts on 100% of the cases and is wrong on 17.4% of them, under a rule that promised 10% at most; the logarithm's acts on 54% and is wrong on 9.9%.

Rescaling repairs part of this, as the essay said it would. A single temperature fitted on held-out comments, which divides every logit by 9.9 before the softmax, brings the calibration gap of the checker's model from 0.059 to 0.011. What it cannot bring back is in the truth gap, the distance from the raters' shares: 0.141 after rescaling, where the reporter who never read the comment has 0.127 and the model trained by the logarithm 0.102. A temperature can make a model's confidence right on average. It cannot restore the difference between a comment that three raters in ten object to and one that none does, once training has pressed both into the same corner.

And on the other two kinds the damage is not to the confidence alone. On inference the checker's model gives the same letter to 100% of the items, whatever stands behind it, and is right on 34.3%, which with three options lettered in a random order is chance, and below the 44.5% of the model it started from; the logarithm's reaches 55.8%. On the reviews, 39.9% against 62.5%. The slope explains it. Under the checker an option is taught in proportion to the probability it already has, so a model that begins by leaning toward one letter is confirmed in the letter before it has learned to read the options; the rich get richer, and the first riches were an accident. The logarithm teaches most where the model was most surprised.

three ways to read a decision

Reading the logits is the least one can do, and it has a ceiling that the banking example will show: it needs every option to be one token, and a caller's options are sentences. The published models all go further in, and they agree on the direction: score each option where it stands in the text, from what the network made of it there.

Laya, the encoder, does this in the simplest way. Its input is the question, then each option preceded by a marker token of its own, then the state; every token reads every other, in both directions; the hidden state at each marker is turned into one logit by a small head, and the softmax is taken over the markers of a question. Convai Innovations, model card of Laya (September 2026): ModernBERT-large, 395 million parameters, trained whole, with a head of two transformer layers trained from nothing, 421 million in all; a context of 512 tokens. The card says plainly that about 20 options is the most a choice will bear, and that a noul sometimes follows the wording of its options more than the state. An option is a place in the text, and its score is read at that place. Nothing in the head knows how many options there are or what they are called.

Clef does the same on a backbone that reads in one direction, and its released file can be followed line by line. The prompt is the state first, then the schema: each field with its instruction and its options, the spans of the instruction and of every option being kept as the prompt is built. The state stands first so that one pass serves any schema, and so that every option, coming after it, has read it. The backbone runs once over the whole. Then the head, which Cloudflare calls the joint schema head, works on the last layer \(h\):

1. Every token's hidden state, normalized and projected to the head's width, is kept as a memory that the later steps read by attention.
2. A question is summed up as the mean of \(h\) over its instruction. An option gets two summaries: the mean of \(h\) over its span, which is what the network made of the option in context, and the mean of the rows of the output embedding for its tokens, which is what the network would have to say to write the option. The second is the lexical prior of Cloudflare's description.
3. Each option becomes a query, the sum of projections of those two summaries and of its question, and the queries pass through layers that attend to the memory: evidence is routed from the state to each option.
4. Each field becomes a vector too, made of its question, an attention-weighted sum of its routed options, the last token's state and an embedding of its type; the fields then pass through decoder layers in which they attend to one another and to the memory. This is the "joint" of the name: a field's answer may depend on the other fields.
5. An option's logit is a prior plus a gated correction, and a softmax is taken over the options of each field.

In symbols, with the three scalars \(s_0\), \(s_1\) and \(g\) all starting at 0,

\[\begin{aligned} z &= e^{s_0}\cos\big(\text{lexical},\ \text{question}+\text{last}\big)+\sigma(g)\,c,\\ c &= e^{s_1}\cos(\text{field},\text{option})+\mathrm{MLP}\big[\text{field},\,\text{option},\,\text{field}\odot\text{option},\,\lvert\text{field}-\text{option}\rvert\big]. \end{aligned}\]

The first term is the prior and \(c\) the correction; \(\sigma\) is the logistic function, so the gate \(\sigma(g)\) starts at one half. From joint_schema_model.py as released with the weights. The widths and depths of the head are in a configuration file that I have not read, so no count of its parameters is given here. The identifiers of a choice's options are sorted before the prompt is built, which matters to anyone who finetunes: renaming an option moves it.

The design has a logic that is easy to miss in the detail. The prior is what reading the logits would give, generalized to options of any length: how well does the text of this option fit what the network expects at the end of the prompt. The correction is everything the prior cannot see, and the gate lets training decide how much of it to use. The head used in the measurements below keeps that logic and drops the rest. Its prior is the model's own logit for the option's token, its correction is computed from the hidden state at the option and the hidden state at the answer slot, and its gate starts shut, so that before training it says exactly what reading the logits says:

class TypedHead(nn.Module):
    """Way 3, a head of its own, after the shape of Clef's: every option is scored where it stands. The score is a
    prior, the model's own logit for the option, plus a gated correction computed from the hidden state at the
    option, the hidden state at the ANS token, and their product (the 2 hidden states normalized first). At the
    start the gate is shut and the head says exactly what reading the logits says."""
    def __init__(self, width):
        super().__init__()
        self.norm = nn.LayerNorm(width)
        self.mlp = nn.Sequential(nn.Linear(3 * width, width), nn.GELU(), nn.Linear(width, 1))
        self.prior_scale, self.gate = nn.Parameter(torch.zeros(())), nn.Parameter(torch.zeros(()))

    def forward(self, model, hidden, b):
        rows = torch.arange(hidden.shape[0], device=hidden.device)[:, None, None]
        at_read = hidden[rows[:, :, 0], b["read"]]                             # [n, Q, width]
        prior = torch.gather(model.logits(at_read), 2, b["option_ids"])
        at_option = self.norm(hidden[rows, b["options"].clamp_min(0)])         # [n, Q, MAX_OPTIONS, width]
        at_read = self.norm(at_read)[:, :, None].expand_as(at_option)
        correction = self.mlp(torch.cat([at_option, at_read, at_option * at_read], -1)).squeeze(-1)
        z = self.prior_scale.exp() * prior + self.gate * correction
        return z.masked_fill(b["options"] < 0, float("-inf"))

It adds 3,149,827 numbers to the 10,092,544 of the adapters. Trained with the logarithm beside a copy that reads the logits, it changes nothing that the measurements can establish: on the toxicity question 83.2% right with a truth gap of 0.169 against 83.5% and 0.169, and on inference 51.5% against 55.8%, with truth gaps of 0.247 and 0.250, a difference in first answers that 400 items cannot settle. That is the expected result and worth having on record. Where every option is one token and the adapters are free to move the whole network, the logits already are a head; a separate one earns its place when options are sentences, which is the case the last section takes up, and there it is not an improvement but the only way in.

adapters

None of the open decision models built on a writer trains the writer's weights. Clef keeps its backbone frozen and learns, for each weight matrix \(W\) of the backbone, a correction of low rank: \(W x\) becomes \(W x + \tfrac{\alpha}{r}\,B A x\), where \(A\) has \(r\) rows and \(B\) has \(r\) columns, and only \(A\) and \(B\) are trained. Edward Hu and others, "LoRA: Low-Rank Adaptation of Large Language Models" (2021). Cloudflare's rank is 256; the outside reimplementation uses 64 and notes the difference; the runs here use 16, on a backbone 15 to 45 times smaller. \(B\) starts at 0, so the adapted model starts as the released one. There is a library for this, and it hides nothing worth hiding:

class LoRA(nn.Module):
    """A linear layer whose weight W stays frozen, plus a correction of rank r that is trained: W x + (alpha/r) B A x.
    B starts at 0, so the layer starts as it was."""
    def __init__(self, layer, rank=8, alpha=16):
        super().__init__()
        self.layer, self.scale = layer, alpha / rank
        for p in layer.parameters():
            p.requires_grad_(False)
        self.A = nn.Parameter(torch.randn(rank, layer.in_features) / math.sqrt(rank))
        self.B = nn.Parameter(torch.zeros(layer.out_features, rank))

    def forward(self, x):
        return self.layer(x) + self.scale * (x @ self.A.T) @ self.B.T

def adapt(model, rank=16, alpha=32):
    """Freeze the backbone and put the lab's adapter on every linear layer of every block, 7 to a block. Returns the
    adapters' parameters, the only ones left to train."""
    for p in model.parameters():
        p.requires_grad_(False)
    params = []
    for block in model.lm.model.layers:
        for part, names in ((block.self_attn, ("q_proj", "k_proj", "v_proj", "o_proj")), (block.mlp, ("gate_proj", "up_proj", "down_proj"))):
            for name in names:
                lora = M.LoRA(getattr(part, name), rank, alpha)
                setattr(part, name, lora)
                params += [lora.A, lora.B]
    return params

The backbone here has 28 blocks with 7 linear layers each, 4 in attention and 3 in the small network after it, so 196 adapters, and at rank 16 they hold 10,092,544 trained numbers beside 596,049,920 frozen ones, 1.7%. Two details of practice are worth knowing. The frozen weights are kept in 16-bit floating point and the adapters in 32, because the updates to \(A\) and \(B\) are small enough to vanish in 16 bits; and since nothing is generated, a training step is one forward and one backward pass over a request, with none of the machinery of sampling.

the loss, and what Clef's adds

With the answer a smooth function of the weights, training is the classifier's: a score of the reported probabilities against what came, differentiated. The proper scores are these.

def log_score(z, y):
    """log u_y. Pretraining's own score: cross-entropy is this with the sign changed."""
    return at(z.log_softmax(-1), y)

def brier_score(z, y):
    """2 u_y - |u|^2 = 1 - |u - e_y|^2. Brier (1950), with the sign and origin that make more better."""
    u = chances(z)
    return 2 * at(u, y) - (u * u).sum(-1)

def ranked_score(z, y):
    """For options in an order (the levels of a score): minus the squared distance between the running totals of
    the report and of what came (Epstein, 1969). A near miss costs less than a far one, and the score is proper."""
    u = chances(z)
    came = (torch.arange(u.shape[-1], device=z.device) >= y[..., None]).float()
    return -((u.cumsum(-1) - came) ** 2).sum(-1)

The last is for the levels of a score. The logarithm and Brier's square treat a report of four stars for a review of five exactly as they treat a report of one star: wrong. Epstein's ranked score compares running totals, the chance of at most one star, at most two, and so on, with those of what came, so that probability placed near the outcome costs less than probability placed far from it; and it is still proper. Edward Epstein, "A Scoring System for Probability Forecasts of Ranked Categories", Journal of Applied Meteorology 8 (1969), 985–987. Laya's card names this score for its ordinal questions.

Clef's published loss is a sum: a cross-entropy with label smoothing, plus Brier's loss, plus, for scores, a term on the running totals. Label smoothing replaces the target \(e_y\) by \((1-\varepsilon)\,e_y+\varepsilon/K\), keeping \(\varepsilon\) back and spreading it over the \(K\) options. What report does that reward? The loss is \(-\sum_k t_k\log u_k\) with that target \(t\), and averaged over outcomes drawn from \(q\) the target becomes \(\bar t=(1-\varepsilon)\,q+\varepsilon/K\). By the argument that made the logarithm proper, the best \(u\) is \(\bar t\):

\[u^\star \;=\;(1-\varepsilon)\,q\;+\;\frac{\varepsilon}{K}.\]

The truth, pulled toward the toss-up. With \(\varepsilon=0.05\) and two options, a model that fits this target exactly never says more than 0.975, and of a comment that nine raters in ten call toxic it says 0.88. Brier's term has its optimum at \(q\), and the sum of the two has its optimum between. The recipe as the outside reimplementation writes it:

def smoothed_log_score(z, y, eps=0.05):
    """The log score against a target that keeps eps back and spreads it over the options: label smoothing.
    Greatest at u = (1 - eps) q + eps / K."""
    valid = torch.isfinite(z)
    target = F.one_hot(y, z.shape[-1]) * (1 - eps) + valid * eps / valid.sum(-1, keepdim=True)
    return (target * z.log_softmax(-1).masked_fill(~valid, 0)).sum(-1)

def clef_loss(z, y, is_score, eps=0.05, brier_weight=1.0, ordinal_weight=0.5):
    """The published recipe of Clef as an open reimplementation writes it: cross-entropy with label smoothing, plus
    Brier's loss, plus (for score questions) the squared distance between running totals, a loss and not a score."""
    loss = -smoothed_log_score(z, y, eps) + brier_weight * (1 - brier_score(z, y))
    levels = torch.isfinite(z).sum(-1).clamp_min(2) - 1
    return loss + ordinal_weight * is_score * (-ranked_score(z, y)) / levels

Laya's card describes something that sounds different: reinforcement. The policy reports a distribution; it is explored by adding normal noise to the logits; each noisy report is paid a proper score; and the logits are moved toward the noisy reports that were paid more than their group's mean. Work out the mean of that update. A noisy report is \(z'=z+\sigma\xi\) with \(\xi\) standard normal; its log-density has slope \((z'-z)/\sigma^2\) in \(z\); and for any function \(S\),

\[\mathbb{E}\Big[S(z')\,\frac{z'-z}{\sigma^2}\Big]\;=\;\nabla_z\,\mathbb{E}\big[S(z+\sigma\xi)\big]\;=\;\mathbb{E}\big[\nabla S(z+\sigma\xi)\big].\]

The reinforcement estimates the slope of the score itself, smoothed over a neighborhood of width \(\sigma\), and with a group of \(M\) reports each measured against the mean of a group that includes it, shrunk by \((M-1)/M\). It is the same slope as differentiating the score, reached by a noisier road: a lottery is being used to estimate a derivative that is available exactly.

def noisy_logit_surrogate(z, y, score, sigma=0.5, draws=4):
    """Reinforcement of a report: the policy is the distribution itself, explored by adding normal noise to its
    logits; each noisy report is paid a proper score, and the logits are moved toward the noisy reports paid more
    than their group's mean. (Laya's card describes its training so.) The log-chance of a noisy z' is
    -|z' - z|^2 / (2 sigma^2), so its slope in z is (z' - z) / sigma^2: the mean gradient is that of the score
    itself, smoothed by the noise and shrunk by (draws - 1) / draws because each report sits in its own baseline.
    The lottery adds variance to a slope that is known exactly."""
    valid = torch.isfinite(z)
    base = z.masked_fill(~valid, 0)
    noisy = base.detach() + sigma * torch.randn(draws, *z.shape, device=z.device)
    paid = score(noisy.masked_fill(~valid, float("-inf")), y.expand(draws, *y.shape))
    advantage = paid - paid.mean(0, keepdim=True)
    log_chance = -(((noisy - base) ** 2) * valid).sum(-1) / (2 * sigma ** 2)
    return -(advantage * log_chance).mean(0)

All of these were run. A copy of the released model was given adapters and post-trained for 1,000 steps, each step on 2 comments, 3 inference items and 3 reviews, 20 decisions with one outcome each, in eight ways: with the checker's reward; with the logarithm; with Brier's score; with the logarithm and the head; with Clef's loss and the head; with Laya's noisy logits paid by the logarithm and the spherical score; with the logarithm for nouls and choices and the ranked score for scores; and with the logarithm under the ordinary mask, of which more below. The second of these, adapters trained by the logarithm under the tree, is jev-pp; the rest are what it is to be compared with. A ninth network, of the same architecture with random weights, was trained on the same decisions with every weight free. In the tables, "right" is how often the first answer was the one that came, "says" the mean probability given to the first answer, "gap" the calibration gap over 15 bands, and "truth gap" the distance from the raters' shares; "acts at 0.9" is the share of questions on which the model says at least 0.9, with how often it is wrong on those; and the temperature is the one fitted on held-out cases, with the truth gap after dividing the logits by it. First the comments, all seven questions together:

trained by	right	says	gap	truth gap	acts at 0.9	wrong there	temperature	truth gap, rescaled
not reading the state	0.941	0.943	0.004	0.127	71%	2.2%
as released, no training	0.102	0.691	0.588	0.650	1%	94.2%	20.00	0.475
checker's reward	0.941	1.000	0.059	0.151	100%	5.9%	9.94	0.141
logarithm (jev-pp)	0.943	0.954	0.015	0.102	87%	3.1%	1.11	0.101
Brier	0.943	0.951	0.010	0.104	87%	3.0%	1.00	0.104
logarithm, with the head	0.942	0.953	0.014	0.102	87%	3.1%	1.11	0.101
Clef's loss, with the head	0.942	0.933	0.016	0.104	85%	2.8%	0.82	0.105
noisy logits (Laya)	0.941	0.960	0.019	0.108	89%	3.3%	1.11	0.107
logarithm and ranked score	0.943	0.955	0.015	0.100	87%	3.1%	1.11	0.100
logarithm, ordinary mask	0.942	0.956	0.014	0.108	87%	3.2%	1.11	0.107
logarithm, from random weights	0.941	0.952	0.011	0.127	74%	2.5%	1.00	0.127

The same for the one question in seven that asks whether a rater would call the comment toxic, the attribute raters dispute most:

trained by	right	says	gap	truth gap	acts at 0.9	wrong there	temperature	truth gap, rescaled
not reading the state	0.826	0.829	0.003	0.220	0%	0.0%
as released, no training	0.176	0.759	0.583	0.636	2%	88.6%	20.00	0.411
checker's reward	0.826	1.000	0.174	0.276	100%	17.4%	9.94	0.252
logarithm (jev-pp)	0.835	0.858	0.038	0.169	54%	9.9%	1.11	0.168
Brier	0.833	0.857	0.034	0.176	54%	9.6%	1.00	0.176
logarithm, with the head	0.832	0.855	0.041	0.169	55%	9.7%	1.11	0.168
Clef's loss, with the head	0.833	0.845	0.027	0.171	48%	9.2%	0.82	0.176
noisy logits (Laya)	0.827	0.877	0.050	0.185	58%	10.1%	1.11	0.182
logarithm and ranked score	0.835	0.858	0.040	0.167	56%	9.7%	1.11	0.166
logarithm, ordinary mask	0.829	0.866	0.044	0.184	54%	9.6%	1.11	0.181
logarithm, from random weights	0.826	0.849	0.028	0.220	0%	0.0%	1.00	0.220

Inference, where the question is which of three relations a reader would see:

trained by	right	says	gap	truth gap	acts at 0.9	wrong there	temperature	truth gap, rescaled
not reading the state	0.343	0.333	0.009	0.321	0%	0.0%
as released, no training	0.445	0.819	0.375	0.499	43%	56.1%	6.67	0.315
checker's reward	0.343	1.000	0.658	0.671	100%	65.8%	20.00	0.390
logarithm (jev-pp)	0.558	0.568	0.047	0.250	0%	0.0%	0.90	0.253
Brier	0.540	0.560	0.077	0.252	0%	0.0%	0.90	0.255
logarithm, with the head	0.515	0.568	0.054	0.247	0%	0.0%	1.00	0.247
Clef's loss, with the head	0.553	0.550	0.053	0.257	0%	0.0%	0.90	0.261
noisy logits (Laya)	0.420	0.425	0.059	0.304	0%	0.0%	0.74	0.308
logarithm and ranked score	0.538	0.538	0.040	0.259	0%	0.0%	0.90	0.261
logarithm, ordinary mask	0.535	0.552	0.050	0.260	0%	0.0%	0.90	0.263
logarithm, from random weights	0.330	0.366	0.036	0.320	0%	0.0%	20.00	0.321

And the reviews, where there are no shares and a model can be compared only with the stars that came:

trained by	right	says	gap	log loss	stars off	ranked loss	temperature	gap, rescaled
as released, no training	0.333	0.392	0.064	1.501	1.12	0.751	1.35	0.049
checker's reward	0.399	0.994	0.595	10.354	0.97	0.972	18.10	0.079
logarithm (jev-pp)	0.625	0.735	0.111	0.946	0.48	0.313	1.49	0.033
Brier	0.610	0.721	0.111	0.949	0.49	0.318	1.35	0.046
logarithm, with the head	0.622	0.733	0.112	0.938	0.47	0.312	1.35	0.061
Clef's loss, with the head	0.613	0.696	0.085	0.897	0.49	0.307	1.11	0.063
noisy logits (Laya)	0.597	0.650	0.053	0.937	0.51	0.325	1.11	0.028
logarithm and ranked score	0.606	0.694	0.089	0.911	0.49	0.314	1.22	0.048
logarithm, ordinary mask	0.610	0.723	0.115	0.944	0.49	0.318	1.35	0.056
logarithm, from random weights	0.419	0.427	0.047	1.345	0.89	0.576	1.11	0.047

The first row is the place to begin, because it disciplines everything under it. A reporter that never reads the comment, and gives each of the seven questions the average share of its attribute, is right on 94.1% of the questions, with a calibration gap of 0.004: most of what a moderation queue asks is answered no by nearly every rater of nearly every comment, and the commonest answer is hard to beat. Its probabilities mean exactly what they say and distinguish nothing. That is the essay's distinction between meaning what one says and sorting the cases, found in a public data set, and it is why the truth gap is in the tables: by it the reporter stands at 0.127, and at 0.220 on the toxicity question, and a model has read the comments to the extent that it stands lower.

The proper scores next. The logarithm and Brier's rule give the same model, to within the noise of one run, and it has read something: a truth gap of 0.102 and 0.104 over all the questions and 0.169 and 0.176 on toxicity, with calibration gaps of 0.015 and 0.010. Trained on one verdict a visit and never shown a share, these models have moved toward the shares, from the reporter's 0.220 to 0.169 on the disputed question. It is a modest move. 2,000 visits to comments, on a backbone of 0.6 billion weights, buy a first answer that beats the commonest one by 0.9 points on toxicity, and the last section asks what more steps and a larger backbone add. On inference, where the answer has to be read out of the two sentences, the same training goes from the released model's 44.5% to 55.8% of a possible 63.5%, with a truth gap of 0.250 against 0.321 for saying a third to everything.

Clef's loss third. Its first answers are as good as the logarithm's. What it says is lower, 0.933 where it is right 0.942 of the time, and the temperature that best corrects it is 0.82, below 1: its logits want sharpening, where the logarithm's want leaving alone or softening. This is the pull toward the toss-up that the algebra predicted, in the predicted direction. It is small at \(\varepsilon=0.05\) and costs nothing in accuracy, and on the reviews, where every training here turned out too sure, the same pull happens to help. The recipe works as a regularized classifier works, and a user of such a model should know which way its numbers lean.

Laya's lottery fourth. On the comments it arrives near where the logarithm arrives (94.1% right, a truth gap of 0.108), as the derivation said it must. On inference, the hardest of the three and the one with the fewest cases, it is behind after the same number of steps: 42.0% right against 55.8%, with a truth gap of 0.304 against 0.250. The same slope by a noisier road gets less far in the same time. Nothing in the result suggests that the lottery finds anything the derivative does not.

The reviews spoil the pattern in an instructive way. There are no shares here, only outcomes, and against outcomes every trained model is too sure: the logarithm's says 0.735 and is right on 0.625, a gap of 0.111, and the held-out temperature is 1.49. A proper score promises that the best possible report is the truth. It promises nothing about a network stopped after 1,000 steps, of which each showed it three reviews, and the longer run of the last section shows where the error comes from. One number fitted on 300 held-out reviews brings the gap to 0.033. The ranked score, used in place of the logarithm for this question, did not do better at what it is for: the mean level it reports is off by 0.49 stars on average against 0.48, and Clef's ordinal term gives 0.49. On a scale of five levels with this much data, credit for near misses bought nothing.

The last row is the one a skeptic of pretraining should look at. The same network with random weights, trained on the same decisions with every one of its 596,049,920 weights free, becomes the reporter of the first row: on the comments it is right on 94.1% with a truth gap of 0.127, the average shares and nothing more; on inference it is at chance, 33.0%; on the reviews it reaches 41.9%, where the commonest star would be right on 20.8% and the pretrained network reaches 62.5%. What the adapters taught in 1,000 steps was the request. What the model knew about comments, sentences and restaurants it had from its pretraining, and a decision model knows what its base knew.

the mask

In a model that writes, a token reads every token before it. Put several questions after a state under that rule and each question reads the questions before it: the answer to the third may hang on what the first two asked, and on their order. The essay called this arrangement the chain, and set against it the tree, in which every question reads the state and itself and nothing else. The tree needs two changes to the request and none to the network: an attention mask that hides the other questions, and position numbers that restart at the end of the state for every question, so that a question cannot tell from its own tokens' positions whether it was asked first or seventh.

def masks(b, mode):
    """Who may read whom in a request, and where each token says it stands (see world.masks).
      chain: the ordinary mask of a model that writes; a question reads the state and every question before it
      tree:  a question reads the state and itself, and is numbered as if it came straight after the state
    Returns (allowed [n, width, width] bool: row reads column, positions [n, width])."""
    owner = b["owner"]
    n, width = owner.shape
    causal = torch.tril(torch.ones(width, width, dtype=torch.bool, device=owner.device)).expand(n, -1, -1)
    positions = torch.arange(width, device=owner.device).expand(n, -1)
    if mode == "chain":
        return causal, positions
    same = owner[:, :, None] == owner[:, None, :]              # both in the state, or both in the same question
    from_state = owner[:, None, :] == -1                       # the column is a token of the state
    start = (owner == -1).sum(1, keepdim=True)                 # where the first question begins, record by record
    offset = torch.zeros_like(owner)
    for q in range(b["read"].shape[1]):                        # a question's tokens count from `start`, each time
        mine = owner == q
        first = torch.where(mine, positions, width).min(1, keepdim=True).values
        offset = torch.where(mine, positions - first + start, offset)
    return causal & (same | from_state), torch.where(owner >= 0, offset, positions)

The property can be checked exactly, by asking the same seven questions of the same comment in two orders. In 32-bit arithmetic, under the tree, the largest change in any probability averages 3.8 × 10^-6 and no first answer changes: an answer under the tree does not depend on the other questions, to the precision of the arithmetic. Under the ordinary mask the released model changes its first answer on 16.4% of the questions when they are turned by three places, and the largest change of a probability averages 0.21.

model and mask	first answers that change	mean largest change of a chance
as released, 32 bits, tree	0.0%	3.8 × 10-6
as released, 32 bits, ordinary mask	16.4%	0.210
as released, 16 bits, tree	1.5%	0.016
as released, 16 bits, ordinary mask	17.3%	0.216
trained under the tree, 16 bits	0.0%	0.001
trained under the ordinary mask, 16 bits	0.3%	0.012

Training reduces the sensitivity of the chain a great deal and does not remove it. The model trained under the ordinary mask saw the questions in a fresh order at every visit, which is the remedy Clef's makers describe for their own data, and after 1,000 steps the share of first answers that change with the order has fallen from 17.3% to 0.30%, with a mean largest change of a probability of 0.012; the tree, in the 16-bit arithmetic of training, shows 0.04% and 0.001, which is rounding. Shuffling teaches a model to care less about order. The mask makes order something it cannot see. An audit of the open decision models this month found the same thing from outside, answers that move when the order of the questions does. Julien Delaunay, "Can the Confidence and Decisions of System 1 Models Be Trusted?", arXiv:2611.12521, whose controls include the order of the questions, the order of the options and the length of the prompt. Clef's design is a chain twice over: its backbone reads the fields in sequence, and its head then lets the fields attend to one another on purpose.

What does the tree give up? Whatever one question could have learned from another. If the fields of a schema are facts about one event, the urgency of a ticket and whether it should be escalated, a chain can make the answers cohere and a tree cannot; under the tree each answer is the best report given the state alone, and two such reports may contradict each other. On these comments it gives up nothing that can be measured: on the toxicity question the tree has a truth gap of 0.169 and the chain 0.184. The seven attributes are far from independent, and the state says enough about each. I would take the tree wherever answers are to be audited one at a time, and the chain only where a joint record is the thing being decided, and then I would measure the order sensitivity before shipping it.

doubt with reasoning

Everything so far was done without a lottery, and the essay said why: a proper score of a stated distribution is smooth, and reinforcement is for what cannot be differentiated. The lottery returns when the model reasons before it answers, because a stretch of reasoning is drawn token by token. OpenJev-RLCD is the open attempt at this case, and its setting is easily stated. For a question \(x\) with outcome \(Y\), the model writes a rationale \(r\), drawn from its policy \(\pi_\theta(r\mid x)\) and ending in the marker "Answer:"; then its report \(u_\theta(x,r)\) is read at the next position, as the chances of the options' tokens. The rationale is drawn. The report is read, never drawn. The score of a report is Brier's, written so that more is better, \(J(u,y)=2u_y-\lVert u\rVert^2\).

The paper's best page is an identity. Let \(p=\mathbb{E}_r\,u(r)\) be the mixture of the reports over rationales, the model's distribution as a user who asked many times would see it, and write \(J(u,q)\) for the expected score under the truth \(q\). Since \(J(u,q)=2\,u\cdot q-\lVert u\rVert^2\) and the first term is linear,

\[J(p,q)\;=\;\mathbb{E}_r\,J\big(u(r),q\big)\;+\;\mathbb{E}_r\,\lVert u(r)-p\rVert^2 .\]

Scoring the mixture is scoring each report and adding a bonus for disagreement among them. Now take the checker's way, in which an answer \(A\) is drawn from the report and the report is in effect replaced by the corner \(e_A\). Then the first term is \(2\Pr(A=Y)-1\), the checker's objective, and the second is \(1-\lVert p\rVert^2\): the checker's reward is the mixture objective with the bonus left out, and its optimum is the mode. A single user gets a single rationale, so the paper trains the first term alone, each report on its own, and proves that its optimum is \(u(x,r)=q(x)\) for every rationale: whatever the model wrote, it should end by saying the chances.

The slope of that objective has two parts, and they are very unlike. Through the report, it is exact: differentiate \(J(u(r),y)\) with the rationale held as given. Through the rationale, it must be estimated: the reward of each rationale, less the mean reward of the others drawn for the same question, times the slope of its log-probability. For one question with \(M\) rationales, in the four arrangements compared below:

def group_loss(arm, z, log_r, y, ref_log_r, weight, tie):
    """The loss of 1 question from its M rationales. z [M, 2]: the reports, as logits; log_r [M]: the log-chances
    of the rationales (None in stage 1); y: what came; ref_log_r: the same rationales under the anchor."""
    m, u = z.shape[0], z.softmax(-1)
    if arm == "stage1":
        return -(2 * u[:, y] - (u * u).sum(-1)).mean()                             # Brier's, of each report
    shape = torch.zeros(m, device=z.device)
    if ref_log_r is not None:                                                      # stay near the anchor's rationales
        shape = -(tie / m) * (log_r.detach() - ref_log_r)
    if arm in ("two_stage", "naive"):
        score = 2 * u[:, y] - (u * u).sum(-1)
        local = score.detach() / m + shape
        advantage = local - (local.sum() - local) / (m - 1)                        # against the others of the group
        return -(score.mean() + weight * (advantage * log_r).sum())
    a = torch.multinomial(u.detach(), 1).squeeze(1)                                # the checker: an answer is drawn
    paid = (a == y).float()
    advantage = (paid - paid.mean()) / (paid.std(unbiased=False) + 1e-4)
    shape = shape - (shape.sum() - shape) / (m - 1)
    log_a = u[torch.arange(m, device=z.device), a].clamp_min(1e-12).log()
    return -(advantage * (log_r + log_a)).mean() - (shape * log_r).sum()

The first arrangement, which the paper calls stage 1, uses the exact part only: the report is trained and the rationales earn nothing. The second, the two-stage recipe, continues from stage 1 with both parts, the estimated one at weight 0.3 and with a penalty that keeps the rationales near those of the stage-1 model. The third uses both parts from the start at full weight and with no penalty, the naive reading of "reinforcement learning with a calibrated reward". The fourth is the checker.

The task is the paper's own, GSM8K-Verify: a grade-school word problem, one proposed final answer written by a small model, and a noul, is the proposed answer right. It was built so that reasoning matters, since the proposal can be checked by redoing the sum. The settings are the paper's except for the model, which here is Qwen3-0.6B where the paper's main tables use the 1.7-billion model: every weight trained, by the optimizer AdamW at a rate of \(2\times10^{-6}\), gradients clipped at a length of 1, 8 questions and 4 rationales each to a step, 400 steps, rationales drawn at temperature 1 with at most 320 new tokens, the report read from the first tokens of " yes" and " no". The paper's code could not be run for this piece and was read instead; what ran is a reimplementation of its training step of some 250 lines, checked against the paper's description and the released code's defaults. The task's data file, 8,404 problems with their proposals, is the paper's. The larger model needs about 35 gigabytes for one training run, more than one of these cards has. Stage 1 was run with the paper's three seeds, as were the two-stage recipe and the checker, and the naive arrangement once, for 200 steps.

trained by	seeds	right	says	gap	Brier	AURC	reach at 10% wrong	temperature	gap, rescaled	tokens written	marker supplied
as released	1	61.5%	0.917	0.303	0.647	0.313	0%	8.1	0.040	145	3%
checker (GRPO)	3	82.4%	0.997	0.174	0.348	0.184	0%	9.0	0.045	151	3%
stage 1: the report alone	3	81.4%	0.891	0.085	0.302	0.106	45%	1.9	0.046	188	46%
two stages	3	82.2%	0.900	0.084	0.283	0.087	62%	1.6	0.025	154	26%
naive, 200 steps	1	84.1%	0.947	0.113	0.289	0.131	33%	2.7	0.054	281	62%

"Right" is how often the answer with the larger probability was the true one, and "says", "gap" and the temperature are as before. Brier is the mean squared error of the report against what came, and AURC is the area under the curve of error against coverage when the problems are taken surest first; for both, smaller is better. "Reach at 10% wrong" is the largest share of the problems, surest first, that could be decided with no more than one in ten wrong; it is found with the test answers in hand, and so is an upper bound on what a threshold fixed beforehand would allow. "Tokens written" is the mean length of a rationale, and "marker supplied" the share of rationales that never reached the marker within the limit, so that it had to be appended for them. A seed is the number that fixes a run's random draws, and where several seeds were run the table gives their mean.

The released model is right on 61.5% of 1,260 test problems, of which 49.8% carry a correct proposal, and it says 0.92: it reasons its way to the wrong verdict on four problems in ten and is as sure of those as of the rest.

The checker teaches it to verify. Trained in the form called GRPO, group relative policy optimization, in which each drawn answer is paid one if it is right and its tokens are made more probable according as its pay beat that of the others drawn for the same problem, the model is right on 82.4% after 400 steps, the mean of three seeds, a gain of 21 points. That is what such training is for, and why it made the systems that write code. And the model says 0.997. The rule at 0.9 acts on 99% of the problems and is wrong on 17.3% of them, and next to nothing can be picked out, surest first, with one in ten wrong, between 0 and 6 problems of 1,260 over the seeds: the probabilities have all but stopped carrying information about which answers are the right ones. A temperature of 9.0 mends the average, a gap of 0.045, and leaves the order of the cases as it was.

Stage 1 reaches about the same accuracy by another road, 81.4% on average over three seeds (from 79.5% to 82.5%). Nothing in it rewards a right answer, and the rationales are paid nothing at all; only the report at the end is pressed toward what came. It says 0.89, with a gap of 0.085 that a temperature of 1.9 brings to 0.046, and its confidence sorts the cases: 45% of the problems can be decided, surest first, with one in ten wrong (from 33% to 51% over the seeds), where the checker's model allows next to none. This is the paper's central result. There it was measured on a model three times the size, with 91.8 and 92.2% right and 19 against 81% of the problems decidable at one in twenty wrong; here it survives a smaller model and an independent reimplementation. The same marks, and a confidence that can be used against one that cannot.

Why should training the report alone improve the verdicts? Because the report is where the verdict is taken. The released model writes out a check of the sum and then, at the marker, gives a verdict that its own check does not always support; pressing the final probabilities toward what came teaches the last step to read the rationale it follows. There is a cost, and the table shows it under the tokens written. The rationales earn nothing, but they are written by the same weights, and they drift: they grow from 145 tokens to 188, and 46% of them no longer reach the marker within the 320 tokens allowed, against 3% before, so that the marker has to be supplied. The authors' later manuscript adds to stage 1 a penalty that holds the rationales near the released model's, and this drift is presumably what it is for.

The second stage starts from the stage-1 model and adds the lottery. Each rationale is now paid the score of the report that follows it, less the mean pay of the others drawn for the same problem, at a weight of 0.3, with a penalty of 0.04 on straying from the rationales of the stage-1 model. Over the same three seeds it is right on 82.2% of the problems against 81.4% after stage 1, and 62% of the problems can be decided with one in ten wrong against 45%. The second difference is the firmer: the reach rose on every seed (by 5, 21 and 23 points), the accuracy on one (by 2.4 points), with the other two level. The rationales, now that they are paid, come back toward their old length, 154 tokens, and 26% of them fail to reach the marker where 46% did. The gain has the direction of the paper's and less than its size (there, 90.4 to 92.2% right, and 63 to 81% of the problems at one in twenty wrong, where these runs go from 24% to 30%), and on three seeds I would not put a number on it. What can be said is that the order of the stages does what its authors claim. Once the report means what it says, its score is a reward that can be trusted with the reasoning, and reinforcing by it leaves the calibration where it was, a gap of 0.025 after rescaling against 0.046.

The naive arrangement, loss and reward together from the released model, at full weight and with no penalty, is where the paper reports a collapse: on its examination questions the model learned within about 100 steps to write no rationale at all, the surest way to a steady reward being to draw nothing. On this task that did not happen in 200 steps. No rationale shrank to the bare marker; they grew instead, from 170 tokens to 290, until 57% of them ran into the limit of 320. The model so trained is right on 84.1% of the test problems and says 0.95, with 33% of the problems decidable at one in ten wrong, which is between the checker's model and stage 1's. What does reproduce is the measurement behind the paper's warning. Taken question by question on the released model, the slope through the lottery is 8.1 times the size of the slope through the report, where the paper found 7 to 12: whatever the rationales are being pushed toward, they are pushed far harder than the report is being corrected, and with the noise of a lottery. A reward that does not yet mean anything, steering that much force, will take the rationales somewhere; on one task it took them to silence, and on this one to length. The order of the two stages exists so that the reward means something before it is given the wheel.

on a larger backbone

Three questions were left open by a backbone of 0.6 billion weights trained for 1,000 steps: whether more steps bring the reports nearer the shares, whether a larger backbone does, and what is to be done when the options are sentences and cannot be read as single tokens.

More steps first. The training by the logarithm was run again with four times the steps, 4,000, so that a comment of the pool was visited between two and three times, each time with a fresh rater. On the toxicity question the truth gap goes from 0.169 to 0.158, where the reporter that has not read stands at 0.220; on inference the first answer is right on 59.2% of the items against 55.8%, with a truth gap of 0.259 against 0.250; on the reviews it is right on 62.9% against 62.5%.

More steps bought a little on the disputed question and cost a good deal elsewhere. On inference the first answers improved and the probabilities did not: the calibration gap rose from 0.047 to 0.070. On the reviews the model after 4,000 steps says 0.893 and is right on 0.629, a gap of 0.265 where it had been 0.111, and the temperature that mends it is 2.71. The cause is in the arithmetic of the pool. It held 4,000 reviews, and 4,000 steps of three reviews are 12,000 visits: every review was met three times on average, each time with the same stars. A proper score is proper for outcomes drawn afresh. Shown the same outcome for the same case again, a network learns the case, and the confidence it then reports on cases it has not seen is the confidence it earned on those it has. The comments suffered least, a calibration gap of 0.017 against 0.015, because every visit to a comment drew a new rater from its shares: the label of a disputed comment changed from one visit to the next, which is exactly what keeps a model honest. The rule for practice follows. With one outcome a case, pass over the cases once, or stop by the score on cases held out; and distrust any calibration measured on cases a model was trained on.

A larger backbone next. The recipe of jev-pp was run on the models of 1.7 and 4 billion weights of the same line, with the rate and the number of steps that were chosen for the smallest and nothing tuned for the larger ones.

backbone	trained by	toxicity: right	gap	truth gap	inference: right	gap	truth gap	reviews: right	gap	stars off
0.6 billion	as released	0.176	0.583	0.636	0.445	0.375	0.499	0.333	0.064	1.12
0.6 billion	checker's reward	0.826	0.174	0.276	0.343	0.658	0.671	0.399	0.595	0.97
0.6 billion	logarithm (jev-pp)	0.835	0.038	0.169	0.558	0.047	0.250	0.625	0.111	0.48
0.6 billion	Clef's loss, with the head	0.833	0.027	0.171	0.553	0.053	0.257	0.613	0.085	0.49
1.7 billion	as released	0.295	0.591	0.705	0.477	0.366	0.475	0.427	0.434	0.87
1.7 billion	checker's reward	0.826	0.174	0.276	0.382	0.617	0.609	0.566	0.402	0.50
1.7 billion	logarithm (jev-pp)	0.831	0.032	0.171	0.565	0.053	0.221	0.650	0.112	0.42
1.7 billion	Clef's loss, with the head	0.829	0.027	0.175	0.563	0.046	0.224	0.639	0.090	0.44
4 billion	as released	0.243	0.703	0.785	0.545	0.366	0.430	0.542	0.399	0.55
4 billion	checker's reward	0.826	0.174	0.276	0.343	0.658	0.671	0.533	0.455	0.59
4 billion	logarithm (jev-pp)	0.834	0.050	0.165	0.623	0.044	0.188	0.667	0.110	0.40

Four things can be read off. As released, the larger models read the request better where reading is all it takes: with no training the model of 4 billion weights is right on 54.5% of the inferences and off by 0.55 stars on the reviews, where the smallest was right on 44.5% and off by 1.12. Their confidence does not improve with their reading. At every size the released model says far more than it finds, and on the comments all three say yes to most questions.

Trained by the logarithm, the larger models do better wherever the answer has to be read out of the text. On inference they are right on 55.8%, 56.5% and 62.3% of the items at the three sizes, where 63.5% is the most that can be had, with truth gaps of 0.250, 0.221 and 0.188; on the reviews they are 0.48, 0.42 and 0.40 stars off. A decision model knows what its base knew, and a larger base knows more. On toxicity size bought nothing: truth gaps of 0.169, 0.171 and 0.165, against the 0.220 of the reporter that has not read. Whatever separates a comment that three raters in ten object to from one that six do, 2,000 visits to comments did not teach it to any of the three, and seven times the parameters did not supply it. That knowledge is in cases with their verdicts, and there is no other place to get it.

The checker's reward does at every size what it did at the smallest. The model of 4 billion weights, trained by it, answers no to every question about every comment with a probability of 1.000, and on inference gives the same letter to 100% of the items. The corruption is no weakness of small models.

And the order of the questions matters more to the larger models as released, not less. Asked the same seven questions in another order under the ordinary mask, in 32-bit arithmetic, they change 16.4%, 27.7% and 31.7% of their first answers at the three sizes, and under the tree none, at any size.

Sentences for options, then. A caller's options are seldom one token. The public set used here is banking77: the state is a message from a bank's customer, and the question is which of 77 intents it expresses, each a phrase such as "card payment fee charged", listed in a fresh order in every request. Reading the logits has nothing to hold on to, and a head over the options' spans is the only way in. The one used is the outline of Clef's, with the fields' attention to one another left out, since there is one question:

class SchemaHead(nn.Module):
    """The outline of Clef's joint schema head. Every option is a query made of the hidden states over its span, the
    output-embedding rows of its own tokens (what the backbone would have to say to write it) and its question; the
    queries read the whole sequence by cross-attention; a question is summed up from its options and reads the
    sequence too; an option's logit is a prior, the cosine of its output embedding with the question's and the last
    token's hidden state, plus a gated term from the routed question and option."""
    def __init__(self, hidden, width=512, heads=8, layers=2):
        super().__init__()
        self.norm = nn.LayerNorm(hidden)
        self.memory, self.context, self.lexical, self.ask = (nn.Linear(hidden, width, bias=False) for _ in range(4))
        self.route = nn.ModuleList([nn.TransformerDecoderLayer(width, heads, 4 * width, batch_first=True, norm_first=True, activation="gelu") for _ in range(layers)])
        self.scorer = nn.Sequential(nn.Linear(4 * width, width), nn.GELU(), nn.Linear(width, 1))
        self.prior_scale, self.joint_scale, self.gate = (nn.Parameter(torch.zeros(())) for _ in range(3))

    def forward(self, hidden, token_ids, fields, output_embedding):
        """hidden: [tokens, hidden] of ONE record; returns a list of logit vectors, one per question."""
        h = self.norm(hidden)
        memory = self.memory(h)[None]
        mean = lambda span: h[span[0]:span[1]].mean(0)
        logits = []
        for field in fields:
            question = mean(field["question"])
            context = torch.stack([mean(s) for s in field["options"]])
            lexical = torch.stack([output_embedding[token_ids[s[0]:s[1]]].mean(0) for s in field["options"]])
            options = (self.context(context) + self.lexical(lexical) + self.ask(question))[None]
            for layer in self.route:                           # options read one another, and the whole sequence
                options = layer(options, memory)
            options = options[0]
            asked = self.ask(question).expand_as(options)
            prior = self.prior_scale.exp() * F.cosine_similarity(lexical, (question + h[-1])[None], dim=-1)
            joint = self.joint_scale.exp() * F.cosine_similarity(asked, options, dim=-1)
            joint = joint + self.scorer(torch.cat([asked, options, asked * options, (asked - options).abs()], -1)).squeeze(-1)
            logits.append(prior + torch.sigmoid(self.gate) * joint)
        return logits

It was trained on the smallest backbone with a library's adapters of rank 64 on every linear layer, 40,370,176 trained numbers, and 11,556,868 more in the head, for 500 steps of 16 messages, once with the logarithm and once with Clef's sum of losses.

trained by	right	says	gap	log loss	temperature	bar for 5% wrong	acts	wrong there	answers changed by reordering the options
before training	5.5%	0.014	0.041	4.328
logarithm	90.1%	0.907	0.018	0.393	1.00	0.68	89%	5.3%	2.8%
Clef's loss	89.5%	0.879	0.028	0.444	0.90	0.64	91%	6.2%	3.3%

Before any training the head's prior, the fit between an option's text and what the backbone expects at the end of the prompt, already picks the right intent of 77 for 5.5% of the messages, some four times chance. After 500 steps, 8,000 messages, the model trained by the logarithm is right on 90.1%, says 0.907, and has a calibration gap of 0.018 with a fitted temperature of 1.00. The one trained by Clef's sum is right on 89.5% and says 0.879, with a temperature of 0.90: too modest again, and in the direction the algebra gives. Cloudflare reports 94.2% for Clef on this set, with a backbone 45 times the size and data of its own.

The last columns of that table are the protocol I would ask of any decision model before it is left alone. A bar is set on held-out messages, as the lowest confidence at or above which no more than 5 in 100 answers were wrong there:

def threshold(z, y, risk):
    """The rule's bar, set on a split kept apart: the lowest "says" such that, of the cases in which the model says
    at least as much, no more than `risk` are wrong there. 1.01 (never act) if no bar will do."""
    sure, pick = z.softmax(-1).max(-1)
    order = sure.flatten().argsort(descending=True)
    wrong = (pick != y).float().flatten()[order].cumsum(0) / torch.arange(1, order.numel() + 1)
    ok = (wrong <= risk).nonzero()
    return float(sure.flatten()[order][ok.max()]) if ok.numel() else 1.01

Then the rule "act alone at or above the bar" is applied to messages the model has not seen, and what is reported is the share it acts on and how often it is wrong on those. That is the whole of what a calibrated number is for, and it is the one test in which a failure cannot hide: the bar came from one set of cases and the errors are counted on another. Here the bar came out at 0.68 on 500 held-out messages; on 1,500 others the rule acts on 89% and is wrong on 5.3%, the promise kept to within what 500 messages can fix. For the model trained by Clef's sum it acts on 91% and is wrong on 6.2%. One more column belongs in the protocol. The same messages with their 77 options listed in another order change the first answer for 2.8% of them: the head reads the options where they stand, and where they stand is not nothing.

What a replication at Clef's own size takes can be said only in part. The trainings here took minutes: between 6 and 12 for 1,000 steps at the three sizes, each on one card of 32 gigabytes that it did not always have to itself, and 20 for the banking run with its prompts of 901 tokens. The outside reimplementation of Clef's finetuning budgets one card of 80 gigabytes for the backbone of 9 billion, with one record to a step and gradients gathered over 16, and gives its settings: rank 64 where Cloudflare used 256, a rate of 0.0001 for the adapters and half that for the head, a warm-up of 20 steps and a cosine decay, 500 to 1,000 steps, smoothing of 0.05, Brier's loss at weight 1 and the ordinal term at 0.5. It also varies each record as Cloudflare says its own data was varied: the order of the questions shuffled, a random subset of them kept, the instructions paraphrased, the options' identifiers renamed, the descriptions sometimes dropped. Its guide to data counts in records, one request with its labels: 200 to 500 for a first signal, 1,000 to 3,000 for a pilot, 50 to 100 positives for each rare class. Its authors state that none of it has been run on the released weights. None of that has been run here, and the numbers above are no forecast of it.

What stays out of reach is Jev itself: its architecture, and above all the generator of its cases. Jev-pp is no rival to its namesake and was not meant for one. It shows that the recipe is in the open, and what the recipe does and does not supply. Everything measured in this piece was measured against outcomes that happened, a rater's verdict, a reader's choice, a customer's stars. A model trained on composed outcomes can be checked only by whoever holds real ones.

what the name contains

Opened up, the name contains less machinery than it suggests and more care. The machinery is a loss: any proper score of the stated probabilities against what came, differentiated in the ordinary way. No reinforcement is needed while the answer is read and not drawn, and the forms of it that the open models describe turned out, on inspection, to be that derivative estimated by a lottery, or that loss with a regularizer added whose pull can be measured. The care is everything the loss is computed on: a request with the state first and the options read where they stand; a mask that decides whether one answer may lean on another; one outcome a case, taken from events and not from the model's maker; a reporter that has not read the state, kept in every table as the thing to beat; thresholds set on cases other than those they are judged on.

The runs add four things to the essay's argument, and take one away. A pretrained network's probabilities were off by 0.017 on text published after it was made, and its maker's post-training multiplied that by 6: the calibrated decision was there, and was spent. A checker's reward, on three kinds of real case and at three sizes of model, produced certainty and nothing else. Proper scores produced, in 1,000 steps, models that mean what they say about comments and inferences, and that one fitted number mends on the reviews. And where reasoning comes first, pressing the final report toward what came gave the checker's accuracy with a confidence that could be used, as the paper said it would. What the runs take away is any thought that meaning what one says is the hard part. The reporter that never read a comment meant what it said to within 0.004. The distance that training closed slowly, and that a larger backbone closed further, was the distance to the shares themselves, which is knowledge and not calibration; and that part of a decision model is its base and its cases, as it was for de Prony.