6. Optimization
Draft
Why plain gradient descent struggles on transformers
Four rules illustrate the problem of choosing a step. Gradient descent multiplies the gradient by one number, the learning rate. Momentum adds up recent gradients before it steps. Sign descent throws away the size of each coordinate of the gradient and keeps its sign. Adam, the rule behind the transformer of 2017 and GPT-3, divides each coordinate of the averaged gradient by a running estimate of its size.
loss ½(0.2 x² + 20 y²): a valley with curvature 0.2 along x and 20 along y
start (−1.6, 0.6), learning rate 0.05, 100 steps each
first step ÷ rate settles within 0.1 of the minimum
along x along y
gradient descent 0.32 12.0 no: 0.59 away after 100 steps
momentum 0.32 12.0 at step 50
Adam 1.00 1.00 at step 38
sign descent 1.00 1.00 at step 30
gradient descent: a step covers 1.0 % of what is left along x and 100 %
along y; above rate 0.1 it diverges along y. Adam and sign descent divide each
coordinate by the size of its own gradient: both coordinates move at the rate.
(the live figure needs JavaScript; it loads no data)
On a quadratic valley 100 times more curved across than along, the steep direction limits the rate. At a rate of 0.05 a step of gradient descent covers all of what is left across it and 1% of what is left along it: after 100 steps it is still 0.59 from the minimum, and any rate above 0.1 makes it diverge. Sign descent is within 0.1 of the minimum at step 30, and Adam at step 38. On a bent valley floor the advantage is gone: the division helps only when the bad conditioning lies along the coordinates.
The valley has two coordinates; the models of this series have between 2.2 million (gpt-pico, the smallest size of gpt-pp, the one that runs in the page) and 135 million. The difficulty is the same: a loss whose curvature differs by orders of magnitude between directions cannot be descended at one rate. The history of optimizers for transformers, from Adam in 2014 to Muon in 2024, is a history of ways to give each direction a step of its own size.
Throughout, \(\theta \in \mathbb R^d\) are the parameters, \(L(\theta)\) is the loss, the average next-token cross-entropy of Section 1, \(g_t\) is its gradient on the minibatch of step \(t\), and \(\eta\) is the learning rate. Stochastic gradient descent (SGD) is \(\theta_{t+1} = \theta_t - \eta\, g_t\).
Proposition 6.1.1 (gradient descent on a quadratic). Let \(f(\theta) = \tfrac12 \theta^{\top} H \theta\) with \(H\) symmetric and eigenvalues \(h_1 \ge \dots \ge h_d > 0\). Gradient descent, \(\theta_{t+1} = (I - \eta H)\theta_t\), converges from every start if and only if \(\eta < 2/h_1\). The best rate is \(\eta = 2/(h_1 + h_d)\), and with it the slowest coordinate is multiplied at each step by \((\kappa - 1)/(\kappa + 1)\), where \(\kappa = h_1/h_d\) is the condition number.
Proof. In the eigenbasis of \(H\) the \(i\)th coordinate is multiplied by \(1 - \eta h_i\) at each step, so the iteration converges if and only if \(\lvert 1 - \eta h_i \rvert < 1\) for every \(i\), that is, \(\eta h_1 < 2\). The slowest coordinate has the factor \(\max(1 - \eta h_d,\ \eta h_1 - 1)\), which is smallest when the two are equal. ∎
The steepest direction sets the rate and the flattest sets the wait: a factor of \(e\) in the flat direction costs about \(\kappa/2\) steps.
Remark 6.1.2 (momentum buys a square root). Momentum keeps a velocity, \(v_{t+1} = \mu v_t - \eta \nabla f(\theta_t)\) and \(\theta_{t+1} = \theta_t + v_{t+1}\). On the quadratic, the best \(\eta\) and \(\mu\) contract every coordinate by about \((\sqrt\kappa - 1)/(\sqrt\kappa + 1)\) a step: \(\sqrt\kappa/2\) steps for a factor of \(e\) where gradient descent needs \(\kappa/2\).I. Sutskever, J. Martens, G. Dahl and G. Hinton, "On the importance of initialization and momentum in deep learning", ICML 2013, Section 2, which credits the result to a paper of Polyak of 1964. It still moves every coordinate at one rate.
For convolutional networks on images one rate was enough. For transformers it has not been: the gap between SGD and Adam on attention models is among the most studied facts about them. On the 8-layer model of our ladder, the family of models on which this series measures each design choice, it is 0.35 nats. With only the rule that updates the matrices inside the blocks changed, and each rule under the better of two schedules, SGD with momentum ends at 4.536 nats per token after 50 million tokens and AdamW at 4.181, a loss that SGD never reaches, while AdamW is at SGD's final loss after six tenths of the run (Example 6.8.2).
The explanation of the gap has changed twice.
The first explanation was noise (2020). Zhang and coauthors compared the minibatch gradients of an image classifier on ImageNet with those of BERT, a transformer trained on text.
In this work, we provide one explanation for why adaptivity can facilitate convergence with theoretical and empirical evidence. The significant hint that initializes our work comes from the distribution of the stochastic gradients. For Imagenet, the norms of the mini-batch gradients are typically quite small and well concentrated around their mean. On the other hand, the mini-batch gradient norms for BERT take a wide range of values and are sometimes much larger than their mean value. More formally, while the distribution of the stochastic gradients in Imagenet is well approximated by a Gaussian, the distribution for BERT seems to be heavy-tailed. Such observation leads us to the question: does adaptivity stabilize optimization under heavy-tailed noise?
—J. Zhang and others, "Why are adaptive methods good for attention models?" (2020). NeurIPS 2020 (arXiv 1912.03194), Section 1, Assumption 1 and Theorems 2, 5 and 6. Clipping is from R. Pascanu, T. Mikolov and Y. Bengio, "On the difficulty of training recurrent neural networks", ICML 2013 (arXiv 1211.5063).
Their assumption bounds only a moment of order \(\alpha \in (1, 2]\) of the noise, so the variance may be infinite. Under it SGD with a clipped gradient, \(g \leftarrow \min(1, \tau/\lVert g \rVert)\, g\), converges at a rate with matching lower bounds, and adaptivity is read as doing the work of the clip. If noise were the whole explanation, removing it should close the gap.
The test (2023). Kunstner and coauthors removed it, by computing every gradient on the whole training set.
1.1 Contributions
The key question we wish to address is whether the gap between SGD and Adam on transformer models, as studied by Zhang et al. (2020b), is really caused by noise, or whether there already is a large difference in the deterministic setting. Our key observations are
1. Noise is not the key contributor to the performance gap between SGD and Adam on transformers. Removing noise by running in full batch does not reduce the gap (Figure 2).
2. SGD benefits less from the reduction in noise than Adam. Increasing the batch size actually increases the gap between the methods when controlling for the step-size and number of iterations (§3, Figure 3). The performance of SGD does not seem dominated by noise, as its progress per iteration does not improve with batch size as much as it does for Adam (§3.1, Figure 4).
That the performance gap between SGD and Adam grows when noise is removed suggests that the benefit of Adam over SGD can not primarily be due to a robustness to noise. This raises the question as to which component of Adam enables its good performance in the deterministic setting and whether a simpler algorithm can display similar behavior. We show that
3. Sign descent can close most of the gap between SGD and Adam in full batch. While sign descent performs poorly with small batch sizes, it improves drastically with larger batch sizes and can bridge most of the gap between gradient descent and Adam (§4, Figure 5–7).
The gap therefore also concerns optimization of the deterministic loss. In these full-batch experiments, much of it closes under the crudest adaptive rule there is, sign descent:
\[\theta_{t+1} = \theta_t - \eta\, \mathrm{sign}(g_t) . \tag{6.1}\]Why the sign helps: Zipf's law (2024). The next paper of the same group named a cause, "the heavy-tailed class imbalance found in language tasks": a few tokens are very frequent and most are rare (Section 2.2). The mechanism fits in a proposition. Take one bowl for each token, weighted by how often the token occurs.
Proposition 6.1.3 (a ladder of bowls). Let \(f(w_1, \dots, w_V) = \sum_{k=1}^{V} \pi_k \cdot \tfrac12 w_k^2\) with frequencies \(\pi_1 \ge \dots \ge \pi_V > 0\) that sum to one.
(i) Gradient descent gives \(w_k^{(t)} = (1 - \eta \pi_k)^t\, w_k^{(0)}\). It converges if and only if \(\eta < 2/\pi_1\), and at the rate \(\eta = 1/\pi_1\), which settles the first class in one step, class \(k\) keeps the factor \(1 - \pi_k/\pi_1\) a step.
(ii) Under Zipf's law with exponent one, \(\pi_k = 1/(k H_V)\) with \(H_V = \sum_{k \le V} 1/k\), and equal starts \(w_k^{(0)} = r\), the share of the initial loss left after \(t\) steps at that rate is
\[R(t) \;=\; \sum_{k=1}^{V} \pi_k \Bigl(1 - \frac1k\Bigr)^{2t} \;\approx\; \frac{\ln (V/2t) - \gamma}{H_V} \qquad (1 \ll t \ll V),\]where \(\gamma = 0.577\ldots\) is Euler's constant.
(iii) Sign descent, \(w_k \leftarrow w_k - \eta\, \mathrm{sign}(\partial f/\partial w_k)\), moves every class by \(\eta\) a step whatever \(\pi_k\) is: all classes are within \(\eta\) of the minimum after \(\lceil r/\eta \rceil\) steps.
Proof. (i) \(\partial f/\partial w_k = \pi_k w_k\). (ii) Class \(k\) keeps the share \((1 - 1/k)^{2t} \approx e^{-2t/k}\) of its loss, so \(H_V R(t) \approx \int_1^V e^{-2t/x}\, dx/x = E_1(2t/V) - E_1(2t)\) with \(E_1(z) = \int_z^\infty e^{-u}\, du/u = -\gamma - \ln z + O(z)\); the second term is negligible. (iii) The sign of \(\pi_k w_k\) is the sign of \(w_k\). ∎
Under Zipf's law every octave of rank holds the same share of the data, gradient descent conquers one octave for each doubling of the step count, and the loss falls along a straight line in \(\log t\). For \(V = 32{,}768\) the exact sum leaves 0.62, 0.41, 0.21 and 0.04 of the loss after 10, 100, 1,000 and 10,000 steps, where sign descent needs a number of steps that does not depend on \(V\). The paper proves the same separation for a softmax layer \(\mathbf W\) whose input for class \(k\) is the basis vector \(\mathbf e_k\), drawn with frequency \(\pi_k\).
Theorem 3. On the simple imbalanced setting, gradient flow and continuous time sign descent initialized at \(\mathbf{W}=0\) minimize the loss of class \(k\), \(\ell_k(t)=-\log(\sigma(\mathbf{W}(t)\mathbf{e}_k)_k)\), at the rate
Gradient flow: \(\ell_k(t)=\Theta\left(1/\pi_k t\right)\),
Continuous time sign descent: \(\ell_k(t)=\Theta\left(e^{-ct}\right)\).
The difference between the sublinear rate of gradient flow (\(1/t\)) and linear rate of sign descent (\(e^{-t}\)) is similar to existing results for overparameterized logistic regression, where normalized updates converge faster as they keep increasing the margin despite small gradients (Nacson et al., 2019). The novel element is that the convergence of gradient flow strongly depends on the class frequencies \(\pi\), while the convergence of sign descent is independent of the class frequencies.
The last layer of a transformer behaves like the bowls because, to leading order, both the gradient and the curvature of the row of token \(k\) in the unembedding carry the factor \(\pi_k\): to divide by the size of the gradient is, approximately, to divide by the curvature.Kunstner et al. (2024), Section 3.2. For a linear bigram model, F. Kunstner and F. Bach (2025, arXiv 2505.19227) find that gradient descent needs a number of iterations almost linear in the vocabulary size at exponent one, and sign descent about its square root.
Example 6.1.4 (how unequal our tokens are). The setup: the 9.90 billion training tokens of FineWeb-EDU under gpt-pp's vocabulary of 32,768 tokens. The most frequent token, " the" with its leading space, is 3.99% of the stream. The token of rank 1,000 occurs 346 times less often and the token of rank 10,000 occurs 3,885 times less often; 278 tokens make up half of the stream and 10,079 make up 90% of it. A least-squares fit of count against rank over ranks 10 to 1,000 gives a Zipf exponent of 0.883 (1.169 for TinyStories under gpt-pico's 4,096 tokens). By Proposition 6.1.3, a rate that the row of " the" tolerates is hundreds to thousands of times too small for most rows of the unembedding. ◁
Example 6.1.5 (the loss by token frequency, measured). The setup: gpt-pico trained on 100 million tokens of TinyStories under four rules, each at four or five rates half a decade apart, 18 runs of one seed: SGD with momentum 0.9 on the gradient as it is; the same with the gradient clipped at norm one; sign descent with momentum; and Adam with \(\beta = (0.9, 0.95)\), its gradient clipped too. All have one rate for every parameter, a warmup over 4 million tokens, a cosine to a tenth and no weight decay. The vocabulary, sorted by frequency, is cut into ten buckets of about a tenth of the training text each, from the two most frequent tokens to the 3,197 rarest, and the held-out loss is taken bucket by bucket.
charts held-out loss of gpt-pico against tokens seen (100M of TinyStories), one chart per optimizer at its
best rate, one curve per bucket of tokens: 10 buckets from the most frequent tokens to the rarest,
each about a tenth of the training text (bucket 1: 2 tokens; bucket 10: 3,197 tokens)
optimizer rates best all tokens most frequent rarest rarest, behind Adam
SGD with momentum 4 0.3 2.649 0.84 7.07 +3.42
SGD with momentum, clipped 5 1 2.247 0.77 5.68 +2.03
sign descent with momentum 5 0.003 1.879 0.72 3.80 +0.15
Adam 4 0.01 1.812 0.71 3.65
After 100M tokens, each at its best rate, SGD ends at 0.84 nats on the two most frequent tokens, a tenth of the training text, and 7.07 on the rarest tokens that make up a tenth; Adam at 0.71 and 3.65. SGD is 0.13 behind on the frequent tokens and 3.42 on the rare: the rarer half of the buckets makes up 87 % of its gap to Adam.
(the live figure needs JavaScript and loads about 40 kB of data)
At their best rates plain SGD ends at 2.65 nats over all tokens, clipped SGD at 2.25, sign descent with momentum at 1.88 and Adam at 1.81. SGD's gap to Adam grows from each bucket to the next without exception, from 0.13 nats on the most frequent to 3.42 on the rarest, and the rarer half of the buckets makes up 87% of it. That is the account of 2024 in a weaker form than its own figure, whose caption has SGD make "little to no progress on low-frequency classes": here its rarest bucket still ends 1.25 nats below a uniform guess, against 4.67 under Adam, on a tail far lighter than the paper's. Clipping is idle where plain SGD trains (the two end within 0.002 nats of each other at rates 0.1 and 0.3) and decisive above: at rate 1 plain SGD fails and clipped SGD has its best run. Sign descent with momentum ends 0.07 nats behind Adam. ◁
In the code. Each rule of the example is a list of settings handed to gpt-pp's trainer, and one rate is written into every group of parameters. They are OPTIMIZERS and overrides in gpt-pp's frequency_buckets.py.
N_BUCKETS = 10
PRESET, DATA = "gptpp-pico", "tinystories/bpe4k"
COMMON = ["schedule=cosine", "warmup_tokens=4e6", "final_lr_frac=0.1", "weight_decay=0.0"]
# A step is pico's 256 sequences of 256 tokens, taken as four passes of 64: the same gradient in a quarter of the
# memory, because the card is shared with other jobs.
BATCH = ["device_batch=64", "grad_accum=4"]
OPTIMIZERS = {
"sgd": ["optimizer=sgd", "momentum=0.9", "grad_clip=1.0"],
"adam": ["optimizer=adam", "beta1=0.9", "grad_clip=1.0"],
"sign": ["optimizer=lion", "beta1=0.9", "beta2=0.9"],
"sgdraw": ["optimizer=sgd", "momentum=0.9"], # the same SGD on the gradient as it is
}
...
def overrides(optimizer: str, rate: float) -> list[str]:
r = f"{rate:g}"
return [f"lr={r}", f"lr_embedding={r}", f"lr_unembedding={r}", *OPTIMIZERS[optimizer], *COMMON, *BATCH]
sgdraw is the example's plain SGD, sgd the clipped one and sign the sign descent with momentum; grad_clip=1.0 is the threshold \(\tau = 1\) of the clip, and COMMON holds the warmup, the cosine and the absent decay. overrides writes the same rate three times, for the hidden matrices, the embedding and the unembedding, which the 2026 recipe trains at three rates of their own (Section 6.5).
Quirk. The names of the example are looser than the code. "SGD with momentum" is torch.optim.SGD with Nesterov's variant of the velocity of Remark 6.1.2, the trainer's default. "Sign descent with momentum" is Lion (Section 6.2) at \(\beta_1 = \beta_2 = 0.9\), where its step is \(\eta\) times the sign of one moving average of the gradient. Adam keeps the 2026 recipe's \(\epsilon = 10^{-10}\) where PyTorch's default is \(10^{-8}\). And the buckets are tenths only roughly: frequency_buckets puts a token in the tenth that holds the middle of its own share of the training text.
Class imbalance is one of several compatible accounts given since 2023.The others: SGD's steps are measured to lie along far sharper directions than Adam's (Y. Pan and Y. Li, 2023, arXiv 2306.00204); the Hessian of a transformer has blocks of very different scale (Y. Zhang and others, NeurIPS 2024, arXiv 2402.16788); the loss is smooth in the max-norm with a much better constant than in the Euclidean norm (S. Xie, M. A. Mohamadi and Z. Li, 2024, arXiv 2410.08198). One caveat from 2025, which Example 6.1.5 repeats on gpt-pico: over more than 1,500 trained language models, tuned sign descent with momentum stayed measurably behind Adam.A. Orvieto and R. M. Gower, "In search of Adam's secret sauce", NeurIPS 2025 (arXiv 2505.21829). The sign is most of Adam and not all of it.
A transformer's loss is badly scaled, coordinate by coordinate and, as Section 6.5 adds, direction by direction. Every optimizer that works on it is a way of taking steps of equal size in a norm that fits.
Adam and AdamW in their own words
Adam is a dozen lines, and each of them has a history. This is the algorithm as Kingma and Ba printed it.
Algorithm 1: Adam, our proposed algorithm for stochastic optimization. See section 2 for details, and for a slightly more efficient (but less clear) order of computation. \(g_t^2\) indicates the elementwise square \(g_t \odot g_t\). Good default settings for the tested machine learning problems are \(\alpha = 0.001\), \(\beta_1 = 0.9\), \(\beta_2 = 0.999\) and \(\epsilon = 10^{-8}\). All operations on vectors are element-wise. With \(\beta_1^t\) and \(\beta_2^t\) we denote \(\beta_1\) and \(\beta_2\) to the power \(t\).
Require: \(\alpha\): Stepsize
Require: \(\beta_1, \beta_2 \in [0, 1)\): Exponential decay rates for the moment estimates
Require: \(f(\theta)\): Stochastic objective function with parameters \(\theta\)
Require: \(\theta_0\): Initial parameter vector
\(m_0 \leftarrow 0\) (Initialize 1st moment vector)
\(v_0 \leftarrow 0\) (Initialize 2nd moment vector)
\(t \leftarrow 0\) (Initialize timestep)
while \(\theta_t\) not converged do
\(\quad t \leftarrow t + 1\)
\(\quad g_t \leftarrow \nabla_\theta f_t(\theta_{t-1})\) (Get gradients w.r.t. stochastic objective at timestep \(t\))
\(\quad m_t \leftarrow \beta_1 \cdot m_{t-1} + (1 - \beta_1) \cdot g_t\) (Update biased first moment estimate)
\(\quad v_t \leftarrow \beta_2 \cdot v_{t-1} + (1 - \beta_2) \cdot g_t^2\) (Update biased second raw moment estimate)
\(\quad \hat m_t \leftarrow m_t/(1 - \beta_1^t)\) (Compute bias-corrected first moment estimate)
\(\quad \hat v_t \leftarrow v_t/(1 - \beta_2^t)\) (Compute bias-corrected second raw moment estimate)
\(\quad \theta_t \leftarrow \theta_{t-1} - \alpha \cdot \hat m_t/(\sqrt{\hat v_t} + \epsilon)\) (Update parameters)
end while
return \(\theta_t\) (Resulting parameters)
The last line of the loop is gradient descent with two changes: the gradient is replaced by an average of gradients, and each coordinate is divided by the root of an average of its squares. The division is the older idea. AdaGrad (2011) divided each coordinate's step by the root of the sum of all its squared gradients so far, so that a coordinate that is rarely nonzero keeps a large step. Its motive was text.
Standard stochastic subgradient methods largely follow a predetermined procedural scheme that is oblivious to the characteristics of the data being observed. In contrast, our algorithms dynamically incorporate knowledge of the geometry of the data observed in earlier iterations to perform more informative gradient-based learning. Informally, our procedures give frequently occurring features very low learning rates and infrequent features high learning rates, where the intuition is that each time an infrequent feature is seen, the learner should “take notice.” Thus, the adaptation facilitates finding and identifying very predictive but comparatively rare features.
—J. Duchi, E. Hazan and Y. Singer, "Adaptive subgradient methods for online learning and stochastic optimization" (2011). Journal of Machine Learning Research 12, Section 1, page 2122.
A sum never forgets, so every step eventually dies. The repair appeared on lecture slides in 2012, argued from sign descent, which divides the gradient by its own size. With minibatches that divisor jumps from batch to batch: "So why not force the number we divide by to be very similar for adjacent mini-batches?" RMSProp divides by the root of a moving average of the squared gradient.G. Hinton, N. Srivastava and K. Swersky, "Neural networks for machine learning", lecture 6, 2012, slides 27 to 29, which credit the method to Tijmen Tieleman, unpublished. Adam added what Algorithm 1 has beyond it: the same kind of average for the gradient itself, which is momentum, and the division of both averages by \(1 - \beta^t\).
Definition 6.2.1 (Adam and AdamW, in this part's notation). With \(\beta_1, \beta_2 \in [0, 1)\), a small \(\epsilon > 0\), a weight decay \(\lambda \ge 0\) and every operation taken coordinate by coordinate,
\[\begin{aligned} m_t &= \beta_1 m_{t-1} + (1 - \beta_1)\, g_t, & \hat m_t &= m_t / (1 - \beta_1^t),\\ v_t &= \beta_2 v_{t-1} + (1 - \beta_2)\, g_t^2, & \hat v_t &= v_t / (1 - \beta_2^t),\\ \theta_t &= (1 - \eta\lambda)\,\theta_{t-1} - \eta\, \frac{\hat m_t}{\sqrt{\hat v_t} + \epsilon} . && \end{aligned} \tag{6.2}\]Adam is (6.2) with \(\lambda = 0\). Adam with an L2 penalty keeps \(\lambda = 0\) and replaces \(g_t\) by \(g_t + \lambda^{\prime}\theta_{t-1}\). AdamW is (6.2) with \(\lambda > 0\). We write \(u_t = \hat m_t/\sqrt{\hat v_t}\) for the update before the rate, at \(\epsilon = 0\).
Proposition 6.2.2 (what the division does). For each coordinate:
(i) Unbiased averages: if \(\mathbb E\, g_i^2 = s^2\) for all \(i \le t\), then \(\mathbb E\, \hat v_t = s^2\); likewise \(\mathbb E\, \hat m_t\) is the common mean of the \(g_i\).
(ii) No scale: multiplying every gradient of the coordinate by \(c > 0\) leaves \(u_t\) unchanged.
(iii) Bounded: if \(\beta_1^2 < \beta_2\) then, for every \(t\) and every sequence of gradients,
\[\lvert u_t \rvert \;\le\; c(\beta_1, \beta_2) \;=\; \frac{1 - \beta_1}{\sqrt{(1 - \beta_2)(1 - \beta_1^2/\beta_2)}}, \qquad\text{and}\qquad u_1 = \mathrm{sign}(g_1) .\](iv) A sign with a damper: if the averages sit at their expectations for a gradient of mean \(\mu\) and variance \(\sigma^2\), \(m = \mu\) and \(v = \mu^2 + \sigma^2\), then \(u = \mathrm{sign}(\mu)\,(1 + \sigma^2/\mu^2)^{-1/2}\).
Proof. (i) \(v_t = (1 - \beta_2)\sum_{i \le t} \beta_2^{t-i} g_i^2\) has expectation \(s^2 (1 - \beta_2^t)\). (ii) \(\hat m_t\) is multiplied by \(c\) and \(\hat v_t\) by \(c^2\). (iii) Put \(\rho = \beta_1^2/\beta_2\), so that \(\beta_1^j = \rho^{j/2}\beta_2^{j/2}\). By the Cauchy–Schwarz inequality,
\[\lvert m_t \rvert \;\le\; (1 - \beta_1) \sum_{j < t} \rho^{j/2}\, \beta_2^{j/2}\, \lvert g_{t-j} \rvert \;\le\; (1 - \beta_1) \Bigl(\frac{1 - \rho^t}{1 - \rho}\Bigr)^{1/2} \Bigl(\frac{v_t}{1 - \beta_2}\Bigr)^{1/2} .\]Hence \(\lvert u_t \rvert \le c(\beta_1, \beta_2)\, \sqrt{(1 - \rho^t)(1 - \beta_2^t)}\,/\,(1 - \beta_1^t)\), and the last factor is at most one because \((1 - a)(1 - b) \le (1 - \sqrt{ab})^2\) on \([0, 1]^2\) and here \(ab = \beta_1^{2t}\). At \(t = 1\), \(\hat m_1 = g_1\) and \(\hat v_1 = g_1^2\). (iv) Substitute. ∎
The paper's words for (iii) are that Adam's "stepsizes are approximately bounded by the stepsize hyperparameter". In the form (iii) the bound is exact: the constant is 7.27 for the defaults of Algorithm 1, 1.70 for the transformer paper's \(\beta = (0.9, 0.98)\) and 1.16 for \((0.9, 0.95)\), and it is approached by a gradient that grows by the factor \(\beta_2/\beta_1\) at every step. Part (iv) is the reading of Balles and Hennig: Adam is sign descent in which each coordinate is damped by what its own noise earns.L. Balles and P. Hennig, "Dissecting Adam: the sign, magnitude and variance of stochastic gradients", ICML 2018 (arXiv 1705.07774), Section 3.1. With \(\beta_1 = \beta_2 = 0\) it is the sign descent (6.1) exactly.
The defaults of Algorithm 1 did not survive transformers. The transformer paper used \(\beta_2 = 0.98\); GPT-3 used 0.95, with a clipped gradient, and 0.95 is still the value in the open reports of 2025 and 2026. By (iii) the change tightens the bound from \(7.3\,\eta\) to \(1.2\,\eta\), and it shortens the memory of the second moment from about 1,000 steps to about 20, so that a coordinate whose gradient grows is noticed before it has taken many oversized steps.
Decoupled decay. In SGD, shrinking the weights by the factor \(1 - \eta\lambda\) at each step and adding the penalty \(\tfrac{\lambda}{2}\lVert\theta\rVert^2\) to the loss are the same thing. In 2017 the common implementations of Adam did the second and called it the first.
Now, let’s turn to adaptive gradient algorithms like the popular optimizer Adam Kingma & Ba (2014), which scale gradients by their historic magnitudes. Intuitively, when Adam is run on a loss function \(f\) plus L₂ regularization, weights that tend to have large gradients in \(f\) do not get regularized as much as they would with decoupled weight decay, since the gradient of the regularizer gets scaled along with the gradient of \(f\). This leads to an inequivalence of L₂ and decoupled weight decay regularization for adaptive gradient algorithms:
—I. Loshchilov and F. Hutter, "Decoupled weight decay regularization" (2019). ICLR 2019 (arXiv 1711.05101), Section 2; the statement below is their Propositions 1 and 2.
Proposition 6.2.3 (decay is an L2 penalty only without a preconditioner). Let an optimizer step by \(-\eta M_t \nabla f_t(\theta_{t-1})\) for matrices \(M_t\). If \(M_t\) is not a multiple of the identity, no penalty \(\tfrac{\lambda^{\prime}}{2}\lVert\theta\rVert^2\) added to \(f_t\) reproduces the weight decay of (6.2).
Proof. The penalty adds \(-\eta\lambda^{\prime} M_t \theta_{t-1}\) to the step and the decay adds \(-\eta\lambda\,\theta_{t-1}\). They agree for every \(\theta_{t-1}\) only if \(\lambda^{\prime} M_t = \lambda I\). ∎
Decoupled, every weight shrinks at the same relative rate \(\eta\lambda\) a step, whatever its gradients are. Why the decoupled form trains better had to wait for the analysis of Section 6.4.
The sign taken literally. If Adam is mostly a sign, one can keep the sign and drop the second moment. Lion, found in 2023 by a search over programs, does.
def train(weight, gradient, momentum, lr): update = interp(gradient, momentum, β1) update = sign(update) momentum = interp(gradient, momentum, β2) weight_decay = weight * λ update = update + weight_decay update = update * lr return update, momentum
Program 1: Discovered optimizer Lion. \(\beta_{1}=0.9\) and \(\beta_{2}=0.99\) by default are derived from Program 4. It only tracks momentum and uses the sign operation to compute the update. The two gray lines compute the standard decoupled weight decay, where \(\lambda\) is the strength.
Every coordinate moves by exactly \(\eta\) at every step, which is why its authors recommend a rate three to ten times smaller than AdamW's and a decay as many times larger. In the independent benchmarks of 2025, Lion and the other coordinate-wise methods ended within a factor of 1.2 of tuned AdamW in tokens.Chen et al. (2023), Section 5; K. Wen, D. Hall, T. Ma and P. Liang, "Fantastic pretraining optimizers and where to find them", 2025 (arXiv 2509.02046).
One family of update rules. The rules met so far, and the one of Section 6.5, combine many of the same operations. We test their differences on the matrix form of the ladder of Proposition 6.1.3: an \(8 \times 8\) matrix must send eight orthonormal keys to their values, each pair used a third as often as the one before, in 150 steps and at the best of 21 rates.
circuit G → (+) → [average] → [divide by root mean square] → [sign] → [orthogonalize]
→ × (−rate) → (+) → W
weight decay joins at the first (+), inside the gradient, or at the second, around
the rule; the bias correction acts on the two averages
average divide bias decay sign orthog. keys turned keys along axes
gradient descent off off off off off off 0.0077 0.0077
momentum on off off off off off 2.7 × 10⁻⁴ 2.7 × 10⁻⁴
Adam on on on off off off 3.1 × 10⁻⁴ 2.0 × 10⁻⁸
AdamW on on on outside off off 4.3 × 10⁻⁴ 1.7 × 10⁻⁷
Lion Nesterov off off outside on off 3.3 × 10⁻⁴ 2.8 × 10⁻⁷
Muon Nesterov off off off off on 7.9 × 10⁻⁷ 2.4 × 10⁻⁶
Last two columns: the loss after 150 steps over the loss at the start, on a test problem: an
8 × 8 matrix must send 8 orthonormal keys to their values, each pair used a third as often as
the one before. Every rule runs at its best rate of 21. Dividing coordinate by coordinate
repairs the problem only when the keys lie along the coordinates; orthogonalizing repairs it
either way.
(the live figure needs JavaScript; it computes everything in the page and loads no data)
Two differences stand out. With the decay inside, Adam ends at 0.13 of the starting loss, because the penalty moves the minimum; decoupled, the decay barely shows. And the division repairs the problem only where its bad scaling lies along the coordinates: with the keys along the axes Adam ends below a millionth of the starting loss, and with the keys turned it ends, like AdamW and Lion, within a factor of two of plain momentum. Only orthogonalizing the update does better there, by more than two orders of magnitude. It is the subject of Section 6.5.
Warmup and schedules
No model of this series is trained at one rate. Both recipes begin with a ramp and reduce the rate later; we measure the cost of changing each part on the same batches of training text.
chart learning rate, as a share of its peak, against tokens seen (50M a run) chart training loss of each run minus that of its recipe’s own schedule 2026 recipe held-out loss at the end trapezoid 2M tokens of warmup, flat, then a line down over the last 65 % to 0.05 4.045 cosine 2M tokens of warmup, then half a cosine down 4.076 +0.031 linear 2M tokens of warmup, then a line down 4.056 +0.011 constant 2M tokens of warmup, then flat to the end 4.195 +0.150 no warmup no warmup, flat, then a line down over the last 65 % 4.037 −0.008 2019 recipe at steps of 8,192 tokens: 8M tokens of warmup, then half a cosine down to 0.10, or no warmup pre-norm 4.581 pre-norm, no warmup 4.950 +0.369 post-norm 4.537 post-norm, no warmup 7.696 +3.159 collapsed At the ladder’s step of 65,536 tokens, where the same warmup is 122 steps, the 2019 recipe ends at 4.613 with warmup and 5.706 without, at a rate chosen with warmup; with its normalization after each sublayer it collapses either way (7.690 and 7.690), still collapses with a warmup of 32M tokens (7.691), and trains with that warmup at a rate of 0.0006, to 5.255. (the live figure needs JavaScript and loads about 51 kB of data)
The transformer paper used the ramp and gave no reason for it.
5.3 Optimizer
We used the Adam optimizer [20] with \(\beta_1=0.9\), \(\beta_2=0.98\) and \(\epsilon=10^{-9}\). We varied the learning rate over the course of training, according to the formula:
\[lrate=d_{\text{model}}^{-0.5}\cdot\min({step\_num}^{-0.5},{step\_num}\cdot{warmup\_steps}^{-1.5})\]
This corresponds to increasing the learning rate linearly for the first \(warmup\_steps\) training steps, and decreasing it thereafter proportionally to the inverse square root of the step number. We used \(warmup\_steps=4000\).
The rate rises in a straight line to its peak, \((d_{\text{model}} \cdot 4000)^{-1/2} \approx 7.0 \times 10^{-4}\) at \(d_{\text{model}} = 512\), and then decays as \(t^{-1/2}\). Three explanations of the first part followed.
The variance of the second moment (2019). The first steps of Adam divide by an average of very few squared gradients, an estimate of large variance, and "it is better to use smaller learning rates in the first few epochs of training, which justifies the warmup heuristic".L. Liu and others, "On the variance of the adaptive learning rate and beyond", ICLR 2020 (arXiv 1908.03265), Section 1; the paper's optimizer is RAdam, rectified Adam.
Curvature (2021, 2024). At initialization the loss is too sharply curved for the target rate, and a small rate lets training move to flatter regions that tolerate a larger one. Section 6.6 makes this precise.J. Gilmer and others, 2021 (arXiv 2110.04369); D. S. Kalra and M. Barkeshli, NeurIPS 2024 (arXiv 2406.09405).
The size of the first updates (2024). The third account looks at what the first steps do to the weights.
In this work, we explore warmup from this perspective, focusing on GPT2 [29] training with adaptive optimizers like AdamW [24] and Lion [3]. We identify three key issues that necessitate warmup:
- The way Adam handles momentum can lead to artificially large initial updates \(\Delta\boldsymbol w\).
- Early updates \(\Delta\boldsymbol w\) are large compared to the initial weight magnitude of \(\boldsymbol w\) for matrices.
- The gradients of early samples are highly correlated, limiting effective mini-batch sizes.
—A. Kosson, B. Messmer and M. Jaggi, "Analyzing & reducing the need for learning rate warmup in GPT training" (2024). NeurIPS 2024 (arXiv 2410.23922), Section 1. Their remedies for the first two issues reduced the benefit of warmup and did not remove it.
The first issue is a line of algebra on Definition 6.2.1.
Proposition 6.3.1 (the first step against the later ones). Let the gradients of a coordinate be independent with mean zero and variance \(\sigma^2\). Then \(u_1 = \pm 1\), while in the steady state \(\mathbb E\, m_t^2 = \sigma^2 (1 - \beta_1)/(1 + \beta_1)\) and \(v_t \approx \sigma^2\), so that the root mean square of \(u_t\) is about \(\sqrt{(1 - \beta_1)/(1 + \beta_1)}\): 0.23 at \(\beta_1 = 0.9\).
Proof. Proposition 6.2.2 (iii) gives the first step. In the steady state \(m_t = (1 - \beta_1)\sum_{j \ge 0} \beta_1^j g_{t-j}\) has variance \(\sigma^2 (1 - \beta_1)^2/(1 - \beta_1^2)\). ∎
When noise dominates, the first step is more than four times the size of a later one, in a direction that one minibatch decided, when the weights are smallest. In gpt-pp's recipe the ramp is two million tokens, about 30 steps on the ladder, and it is there for the two matrices that stay on AdamW (Section 6.5): the unembedding starts with entries of standard deviation 0.001 and a rate of 0.0098 at width 512, so one sign step at the full rate would move every entry by ten times its initial size.
The shape after the ramp. The transformer's inverse square root gave way to the cosine of a 2016 paper on restarting SGD, \(\eta_t = \eta_{\min} + \tfrac12(\eta_{\max} - \eta_{\min})(1 + \cos(\pi t/T))\) over a budget \(T\). GPT-3 fixed its form for language models: "we use cosine decay for learning rate down to 10% of its value, over 260 billion tokens".I. Loshchilov and F. Hutter, "SGDR: stochastic gradient descent with warm restarts", ICLR 2017 (arXiv 1608.03983), Equation 5; T. B. Brown and others, "Language models are few-shot learners", NeurIPS 2020 (arXiv 2005.14165), Appendix B. The 2019 recipe of our ladder is of this kind: AdamW, a linear warmup, a cosine to a tenth of the peak. The 2026 recipe is a trapezoid: the warmup, a flat top, and a straight line over the last 65% of the budget down to 5% of the peak.
In the code. Both shapes are functions of the tokens seen and not of steps, so they do not change with the batch size or the number of cards. One function returns the rate as a fraction of its peak, for these two shapes and the two others of Example 6.3.3, and a second writes it into the optimizers. They are lr_multiplier and set_lr in gpt-pp's optim.py.
def lr_multiplier(tokens: float, total_tokens: float, cfg: TrainConfig) -> float:
...
tokens = min(max(tokens, 0.0), total_tokens)
if tokens < cfg.warmup_tokens:
return tokens / cfg.warmup_tokens
if cfg.schedule == "constant":
return 1.0
if cfg.schedule == "trapezoid":
start = max(cfg.warmup_tokens, total_tokens * (1 - cfg.cooldown_frac))
else:
start = cfg.warmup_tokens
progress = (tokens - start) / max(total_tokens - start, 1e-9) # 0 when the decay begins, 1 at the end
if progress <= 0:
return 1.0
if cfg.schedule == "cosine":
decay = 0.5 * (1 + math.cos(math.pi * progress))
else: # linear, and the falling edge of the trapezoid
decay = 1 - progress
return cfg.final_lr_frac + (1 - cfg.final_lr_frac) * decay
def set_lr(optimizers: list[torch.optim.Optimizer], multiplier: float) -> None:
"""Set every group's learning rate to its peak times the schedule's multiplier."""
for optimizer in optimizers:
for group in optimizer.param_groups:
group["lr"] = group["base_lr"] * multiplier
progress is the \(t/T\) of the cosine, counted from the end of the warmup, and final_lr_frac is \(\eta_{\min}/\eta_{\max}\): 0.1 in the 2019 recipe and 0.05 in the 2026 one, whose cooldown_frac is 0.65. set_lr multiplies the peak rate base_lr of every group of every optimizer, so the ramp and the decay act on Muon's rate for the hidden matrices (Section 6.5) as they do on AdamW's.
Quirk. The trainer reads the schedule at the middle of a step: its line is multiplier = lr_multiplier((step + 0.5) * tokens_per_step, total_tokens, train_cfg), in train of train.py. The warmup is linear from zero, so the first step is taken at half a step's worth of warmup and not at zero; the recipe's 2 million tokens are 30 steps on the ladder and 14 in the long run of the 12-layer model. The trapezoid's line begins at the later of the warmup's end and the fraction 1 - cooldown_frac of the budget.
What a decaying rate does to the final weights can be written exactly when there is weight decay.
Proposition 6.3.2 (AdamW's weights are a moving average of its updates). Let \(\theta_t = (1 - \eta_t\lambda)\theta_{t-1} - \eta_t u_t\). Then
\[\theta_T = \prod_{j=1}^{T}(1 - \eta_j\lambda)\,\theta_0 \;-\; \sum_{i=1}^{T} c_i\, u_i, \qquad c_i = \eta_i \prod_{j=i+1}^{T} (1 - \eta_j\lambda) .\]At a constant rate, \(c_i = \eta (1 - \eta\lambda)^{T-i}\): the weights are \(-1/\lambda\) times an exponential moving average of the updates, with a memory of \(1/(\eta\lambda)\) steps.
Proof. Unroll the recursion. ∎
At \(\eta = 6 \times 10^{-4}\) and \(\lambda = 0.1\) the memory is 16,667 steps. Under a schedule the profile \(c_i\) says how much of each step survives in the final model, and a rate that falls at the end averages the last, noisy steps down.X. Wang and L. Aitchison, ICML 2025 (arXiv 2405.13698), in the notation of S. Bergsma and others, "Straight to zero: why linearly decaying the learning rate to zero works best for LLMs", ICLR 2025 (arXiv 2502.15938), Section 2.3; an LLM is a large language model. This is the argument made in 2025 for a straight line to zero in place of the cosine to a tenth: a model of 610 million parameters trained that way on 80 tokens per parameter beat the cosine recipe given 200. Practice has not followed: the reports on Kimi K2 and the DeepSeek models keep long flat phases and then a cosine to a tenth.
Example 6.3.3 (warmup and the shape, measured). The setup: the ladder's 8-layer model on 50 million tokens of FineWeb-EDU; the held-out loss at the end, in nats per token; one seed a run, except where a second is named. There are two yardsticks. For runs trained with AdamW it is the ladder's: ten rungs trained twice differ between their two seeds by 0.001 to 0.018 nats, by 0.006 or less on eight of the ten, so a difference under about 0.02 nats between two single runs is not established. Runs under the 2026 recipe are quieter: fourteen pairs of seeds differ by 0.001 to 0.008 nats, and a difference of 0.009 or more between two such runs is established.
The 2019 recipe was run pre-norm and post-norm (the normalization before each sublayer, or after it as in the transformer of 2017; Section 5.2), with its warmup over 8 million tokens and with none. At steps of 8,192 tokens and a rate of \(6 \times 10^{-4}\) the warmup is 977 steps. Pre-norm ends at 4.581 with it and at 4.950 without. Post-norm ends at 4.537 with it and at 7.696 without: within a hundred steps its loss falls to that of predicting each token by its frequency, 7.70 nats, and stays there. With a warmup the arrangement of 2017 trains, 0.044 better than its successor in this one pair of runs; without one it is lost, where its successor loses 0.37. At the ladder's step of 65,536 tokens and its rate of \(2.4 \times 10^{-3}\) the same warmup is 122 steps. There pre-norm ends at 4.613 with it and at 5.706 without, at a rate chosen with the warmup, and post-norm collapses within 40 steps either way. A warmup four times as long, over 32 million tokens, does not rescue it (7.691); that warmup with a rate of \(6 \times 10^{-4}\) does, to 5.255, far behind pre-norm's 4.613. At the large step post-norm needs both a long warmup and a low rate.
The 2026 recipe ends at 4.045 with its warmup and at 4.037 without; in a second seed, at 4.043 with it and 4.045 without. The difference changes sign, 0.008 one way and then 0.002 the other: at this budget leaving out the warmup does nothing that can be measured. After the same warmup, a straight line down to 5% of the peak ends at 4.056, half a cosine down to the same 5% at 4.076, and a constant rate at 4.195. The constant rate costs 0.150 nats and the cosine 0.031. The line costs 0.011, and 0.011 again in a second seed: small, and real. ◁
Weight decay with normalization
Multiply a query matrix of gpt-pp by ten and the model computes the same function: its attention scores are cosines (Section 3.4), and a cosine does not see the length of a vector. A note of 2017 drew the consequence for batch normalization: a penalty makes such weights small, and "this has no regularizing effect of making the computed function simpler".T. van Laarhoven, "L2 regularization versus batch and weight normalization", 2017 (arXiv 1706.05350), Section 2; the sentence quoted below is in Section 3.1.
Lemma 6.4.1 (scale invariance). If \(L(r\omega) = L(\omega)\) for all \(r > 0\), then \(\langle \nabla L(\omega), \omega \rangle = 0\) and \(\nabla L(r\omega) = \nabla L(\omega)/r\).
Proof. Differentiate \(L(r\omega) = L(\omega)\) in \(r\) at \(r = 1\), and in \(\omega\). ∎
In gpt-pp the lemma holds for the query and key matrices and for each row of the embedding, which is normalized as it enters the stream. For the other matrices it holds only approximately: what they write is read through a normalization, but as one term of a sum (Section 5.2).
Proposition 6.4.2 (the effective learning rate). Let \(L\) be scale-invariant and \(\hat\omega = \omega/\lVert\omega\rVert\). A gradient step \(\omega^{+} = \omega - \eta\nabla L(\omega)\) moves the direction as a gradient step on the unit sphere with the rate \(\eta/\lVert\omega\rVert^2\), to first order in \(\eta\), and lengthens the vector: \(\lVert\omega^{+}\rVert^2 = \lVert\omega\rVert^2 + \eta^2\lVert\nabla L(\omega)\rVert^2\). A step \(-\eta u\) of fixed size, \(u \perp \omega\), turns the direction by the angle \(\eta\lVert u\rVert/\lVert\omega\rVert\), to first order.
Proof. By the lemma \(\nabla L(\omega) = \nabla L(\hat\omega)/\lVert\omega\rVert\) is orthogonal to \(\omega\); divide the step by \(\lVert\omega\rVert\) and use Pythagoras. ∎
The rate that matters is the angle turned per step, and without decay it can only fall: "If no regularization is used the weights can grow unbounded, and the effective learning rate goes to 0." Decay shortens the vector, the steps lengthen it, and the two settle.
Proposition 6.4.3 (the equilibrium of AdamW on a scale-invariant vector). Let \(\omega \in \mathbb R^D\) be scale-invariant and trained by (6.2) with gradients dominated by noise, independent across steps and coordinates as in Proposition 6.3.1. Then, from any start and over a time of order \(1/(\eta\lambda)\) steps,
\[\lVert\omega\rVert \;\to\; \sqrt{\frac{\eta D}{2\lambda}}, \qquad \text{angle turned per step} \;\to\; \sqrt{2\eta\lambda\,\frac{1 - \beta_1}{1 + \beta_1}} .\]Proof sketch. In this model the updates are on average orthogonal to \(\omega\), so they add to \(\lVert\omega\rVert^2\). The moving average spreads each gradient over later steps with weights that sum to one, so over many steps the sum of the \(u_t\) grows like \(D\) independent walks of unit variance: \(\lVert\omega\rVert^2\) gains \(\eta^2 D\) a step. Decay removes \(2\eta\lambda\lVert\omega\rVert^2\) a step. Equating the two gives the norm. One step has size \(\eta\sqrt{D(1 - \beta_1)/(1 + \beta_1)}\) (Proposition 6.3.1), and the angle is that size over the norm. ∎
Neither the angle nor the time depends on the size of the gradients or on \(D\): under AdamW every layer turns at the same rate. The angle depends on the product \(\eta\lambda\), so doubling the rate and halving the decay does not change how fast the direction of the weights moves; it only makes them longer.A. Kosson, B. Messmer and M. Jaggi, "Rotational equilibrium: how weight decay balances learning across neural networks", ICML 2024 (arXiv 2305.17212), Section 2 and Table 1; for SGD the same balance gives the angle \(\sqrt{2\eta\lambda}\). A simulation of the model of Proposition 6.4.3 with \(D = 2{,}000\) reproduces both limits from starting norms of 1 and of 30. With the penalty inside Adam, both the norm and the angle depend on the size of each layer's gradients, layers turn at different rates, and the authors of this analysis take that to be why the decoupled form trains better.
Example 6.4.4 (the recipe of GPT-2's era). The setup: the model of Proposition 6.4.3 at \(\eta = 6 \times 10^{-4}\), \(\lambda = 0.1\) and \(\beta_1 = 0.9\). The equilibrium angle is \(2.5 \times 10^{-3}\) radians a step, and the direction, which diffuses by \(2\eta\lambda\) in squared angle a step, has turned by an angle of order one after \(1/(2\eta\lambda) \approx 8{,}300\) steps: the weights forget where they were on the time scale of Proposition 6.3.2. ◁
The GPT-3 paper still described its decay of 0.1 as there "to provide a small amount of regularization". For a model that sees each token once, the view since 2023 is the opposite: "weight decay is never useful as an explicit regularizer but instead changes the training dynamics in a desirable way".F. D'Angelo, M. Andriushchenko, A. Varre and N. Flammarion, "Why do we need weight decay in modern deep learning?", NeurIPS 2024 (arXiv 2310.04415), abstract.
The work of 2025 and 2026 starts from the equilibrium: scale the decay with the rate, so that the balance of Proposition 6.4.3 does not shift as the rate is scheduled down, or renormalize each matrix after each step, so that the learning rate is the angle.A. Defazio, "Why gradients rapidly increase near the end of training", 2025 (arXiv 2506.02285); K. Wen and others, "Fantastic pretraining optimizers and where to find them II: Hyperball optimization", 2026 (arXiv 2606.16899).
Remark 6.4.5 (no decay at all). The 2026 recipe of gpt-pp has no weight decay, the setting its two references started from. Nothing then shortens the weights: in the model of Proposition 6.4.3 the norm grows like the square root of the number of steps, and the angle turned per step falls accordingly, on top of the schedule. In gpt-pico's run the embedding, whose rows the model normalizes as it reads them, grew 45-fold in root mean square, where that model gives 51 for a row that moves at every step. Whether this is a defect depends on the length of the run: the competition in which Muon was introduced found zero decay best on its short track in 2024, and the reports on training at scale call decay essential (Section 6.5).
Steepest descent in another norm
The hidden matrices of gpt-pp, the four matrices of each attention layer and the two of each MLP (the two-layer perceptron of a block, Section 5.3), are not trained by Adam. Their update rule acts on a matrix as a whole. A real gradient of gpt-pico, for a \(512 \times 128\) matrix, shows what that changes.
one step of Muon on the gradient of an MLP input matrix of gpt-pico (512 × 128, trained, one batch of 65,536 held-out tokens): the 128 singular values of X after each pass of the step's Newton-Schulz loop, which maps a value s to 3.4445 s − 4.7750 s³ + 2.0315 s⁵ pass largest median smallest inside 0.68 to 1.20 stable rank 0 0.74 0.038 0.016 1 of 128 1.8 1 1.06 0.13 0.054 2 of 128 5.5 2 1.20 0.43 0.19 26 of 128 27.5 3 1.20 1.01 0.61 121 of 128 88.6 4 1.19 0.85 0.68 128 of 128 69.7 5 1.13 0.90 0.68 128 of 128 85.1 the update has the gradient's 128 directions and none of its scale. Adam's update of the same gradient: stable rank 2.7; its second singular value is 0.38 of its first, its median 0.08 (the live figure needs JavaScript and loads about 0.7 MB of data)
The gradient is nearly one direction: its largest singular value carries 55% of its squared norm, and its stable rank, the squared Frobenius norm over the squared largest singular value, is 1.8 of a possible 128. Five passes later all 128 singular values lie between 0.68 and 1.13 and the stable rank is 85. The update keeps the gradient's directions and none of their scale. This example starts from an empty momentum buffer, as the first step of a run does, so what is orthogonalized is the gradient itself; in training it is a running average of gradients. The Adam comparison uses the gradient divided, entry by entry, by the root of its second moment over 16 batches, without momentum. Its stable rank is 2.7: Adam makes coordinates equal, not directions.
This is Muon, introduced in October 2024 in a public race to train a model of GPT-2's size to a fixed loss as fast as possible.
Muon (MomentUm Orthogonalized by Newton-Schulz) optimizes 2D neural network parameters by taking the updates generated by SGD-momentum, and then applying a Newton-Schulz (NS) iteration as a post-processing step to each of them before applying them to the parameters.
—K. Jordan and others, "Muon: an optimizer for hidden layers in neural networks" (2024). A write-up of 8 December 2024 at kellerjordan.github.io/posts/muon, section "The design of Muon". The race is the modded-nanogpt speedrun: a model of 124 million parameters, a loss of 3.28 on held-out FineWeb, eight cards.
The choice of ball. Every rule of this part answers one question: among the steps of a given size, which lowers the linearized loss most? The rules differ in what size means.
Definition 6.5.1 (steepest descent under a norm). For a norm \(\lVert\cdot\rVert\) on the parameters, a gradient \(g\) and a rate \(\eta > 0\), the steepest-descent step is
\[\Delta^{\star} = \arg\min_{\Delta}\ \Bigl[\langle g, \Delta\rangle + \frac{1}{2\eta}\lVert\Delta\rVert^2\Bigr],\]and the dual norm of \(g\) is \(\lVert g\rVert_{\ast} = \max_{\lVert t\rVert \le 1}\langle g, t\rangle\).
Proposition 6.5.2 (a step is a point of the ball). \(\Delta^{\star} = -\eta\,\lVert g\rVert_{\ast}\, t^{\star}\), where \(t^{\star}\) maximizes \(\langle g, t\rangle\) over the unit ball; and \(-\eta\, t^{\star}\) minimizes \(\langle g, \Delta\rangle\) over \(\lVert\Delta\rVert \le \eta\).
Proof. Write \(\Delta = -s\,t\) with \(s \ge 0\) and \(\lVert t\rVert = 1\). The objective is \(-s\langle g, t\rangle + s^2/2\eta\). For every \(s\) the best \(t\) is \(t^{\star}\), with \(\langle g, t^{\star}\rangle = \lVert g\rVert_{\ast}\); then the best \(s\) is \(\eta\lVert g\rVert_{\ast}\). ∎
For the Euclidean ball \(t^{\star} = g/\lVert g\rVert_2\) and the step is gradient descent, \(-\eta g\). For the max-norm ball, a cube, \(t^{\star} = \mathrm{sign}(g)\) is a corner and the step is the sign descent (6.1) times \(\lVert g\rVert_1\), which is Adam with its averages switched off. Bernstein and Newhouse said it of three optimizers in 2024, Adam, Shampoo and Prodigy, in the abstract of their paper:
Deep learning optimizers are often motivated through a mix of convex and approximate second-order theory. We select three such methods—Adam, Shampoo and Prodigy—and argue that each method can instead be understood as a squarely first-order method without convexity assumptions. In fact, after switching off exponential moving averages, each method is equivalent to steepest descent under a particular norm. By generalizing this observation, we chart a new design space for training algorithms. Different operator norms should be assigned to different tensors based on the role that the tensor plays within the network. For example, while linear and embedding layers may have the same weight space of \(\mathbb{R}^{m\times n}\), these layers play different roles and should be assigned different norms. We hope that this idea of carefully metrizing the neural architecture might lead to more stable, scalable and indeed faster training.
—J. Bernstein and L. Newhouse, "Old optimizer, new norm: an anthology" (2024). arXiv 2409.20325, abstract; the statements are its Propositions 1, 2 and 5.
An optimizer is then a choice of ball. The cube has preferred axes, which is why the sign stops helping on a bent valley and when the keys are turned in the experiment of Section 6.2. The question is which ball suits a weight matrix.
The size of a matrix. A matrix acts on vectors, \(y = Wx\), and a change \(\Delta W\) moves the output by at most \(\lVert\Delta W\rVert_{\mathrm{op}}\lVert x\rVert\), with equality for the worst input. The operator norm, the largest singular value, is what the next layer can feel. The Frobenius norm, the Euclidean norm of the entries, adds up all the squared singular values, and how far the two are apart depends on the kind of matrix.
Proposition 6.5.3 (the operator norm of a Gaussian matrix). Let \(A\) be \(m \times n\) with independent standard normal entries. Then \(\mathbb E\lVert A\rVert_{\mathrm{op}} \le \sqrt m + \sqrt n\), and \(\mathbb P(\lVert A\rVert_{\mathrm{op}} \ge \sqrt m + \sqrt n + t) \le 2e^{-ct^2}\) for an absolute constant \(c\), while \(\lVert A\rVert_F\) is about \(\sqrt{mn}\).R. Vershynin, High-dimensional probability, second edition, Cambridge University Press, 2026, Theorem 7.3.1 and Corollary 7.3.2: the mean by a comparison of Gaussian processes, the tail by the concentration of Lipschitz functions (Section 2.4). The density of the singular values is the law of V. A. Marchenko and L. A. Pastur, Mat. Sb. 72, 1967. The simulated norms are means of 20 draws.
The bound is nearly attained: a simulation gives 33.60 for a \(512 \times 128\) matrix, the shape of gpt-pico's gradient matrix, against \(\sqrt{512} + \sqrt{128} = 33.94\). The singular values of such a matrix fill the interval from \(\lvert\sqrt m - \sqrt n\rvert\) to \(\sqrt m + \sqrt n\), so the stable rank of noise is about \(mn/(\sqrt m + \sqrt n)^2\): 57 of a possible 128 at that shape, where that gradient has 1.8. A gradient is a sum over the batch of products of a backpropagated error with a layer input, and the authors of Muon had observed of the updates of SGD and of Adam, by inspection, the very high condition number measured here; that orthogonalizing helps for this reason they offer as a speculation.
And for an empirically-flavored motivation, we observe that based on manual inspection, the updates produced by both SGD-momentum and Adam for the 2D parameters in transformer-based neural networks typically have very high condition number. That is, they are almost low-rank matrices, with the updates for all neurons being dominated by just a few directions. We speculate that orthogonalization effectively increases the scale of other “rare directions” which have small magnitude in the update but are nevertheless important for learning.
—K. Jordan and others, "Muon" (2024), section "Why is it good to orthogonalize the update?".
Under the Euclidean ball such a gradient is nearly all top direction, and a rate that is safe for the top direction leaves the others with far smaller steps. It is the valley of Section 6.1 again, with singular directions in place of coordinates.
The theorem. Take the ball of the operator norm instead.
Lemma 6.5.4 (the dual of the operator norm). Let \(G = U\Sigma V^{\top}\) be a reduced singular value decomposition with singular values \(\sigma_1, \dots, \sigma_r > 0\), and \(\langle A, B\rangle = \mathrm{tr}(A^{\top}B)\). Then
\[\max_{\lVert T\rVert_{\mathrm{op}} \le 1}\langle G, T\rangle \;=\; \sum_{i=1}^{r} \sigma_i, \qquad\text{attained at } T = UV^{\top},\]and the maximizer is unique if and only if \(G\) has full rank. The sum of the singular values is the nuclear norm \(\lVert G\rVert_{\ast}\).
Proof. \(\langle G, T\rangle = \sum_i \sigma_i\, u_i^{\top}T v_i \le \sum_i \sigma_i\), because \(\lvert u^{\top}Tv\rvert \le \lVert T\rVert_{\mathrm{op}}\) for unit vectors, and \(u_i^{\top}(UV^{\top})v_i = 1\). Equality forces \(Tv_i = u_i\) for every \(i\), which determines a \(T\) of norm at most one when \(G\) has full rank; if not, a term \(u_0 v_0^{\top}\) with \(u_0 \perp U\) and \(v_0 \perp V\) can be added. ∎
Theorem 6.5.5 (the orthogonalized gradient is the steepest step under the operator norm). For a gradient matrix \(G = U\Sigma V^{\top}\), the steepest-descent step of Definition 6.5.1 under the operator norm is \(\Delta W^{\star} = -\eta\,\lVert G\rVert_{\ast}\, UV^{\top}\), and \(-\eta\, UV^{\top}\) minimizes \(\langle G, \Delta W\rangle\) over \(\lVert\Delta W\rVert_{\mathrm{op}} \le \eta\). Both are unique if and only if \(G\) has full rank.
Proof. Proposition 6.5.2 with Lemma 6.5.4. ∎
In one space the theorem can be drawn. A symmetric \(2 \times 2\) matrix is three numbers, hence a point of a box, and there the unit balls of the three norms are a sphere, a double cone and a box. The step of each norm is the point of its ball that lies furthest against the gradient.
one gradient, three unit balls: the step is the point of the ball that lies furthest against G
the ball is the step singular values the rule
the gradient G [ 0.40 0.59 ] 0.90, 0.30
[ 0.59 0.20 ]
the Euclidean ball a sphere [ −0.43 −0.62 ] 0.95, 0.32 gradient descent
Δ = −G / ‖G‖ [ −0.62 −0.21 ]
the operator-norm ball a double cone [ −0.17 −0.98 ] 1, 1 Muon
Δ = −UVᵀ [ −0.98 0.17 ]
the max-norm ball a box [ −1 −1 ] 2, 0 sign descent
Δ = −sign(G) [ −1 −1 ]
A plane perpendicular to G, coming in against it, meets the box at distance 1.88, the double
cone at 1.26 and the sphere at 1.00. On the double cone it always lands on an orthogonal matrix:
both singular values of that step are 1, whatever the gradient’s are.
(the live figure needs JavaScript; it loads about 0.7 MB of three.js for the box and no data)
For a gradient with singular values 0.90 and 0.30, the three norms give different steps. The sphere's step keeps them, rescaled; the box's step, the sign of each entry, has rank one; the double cone's has both singular values equal to one, whatever the gradient's are. As an eigenvalue of the gradient passes through zero, the operator-norm step stays fixed until the sign changes. In the words of one of the theorem's authors, "we keep the singular vectors of the gradient but throw away the magnitudes of the singular values".J. Bernstein, "Deriving Muon", a note of 7 March 2025 at jeremybernste.in/writing/deriving-muon, Step 3.
Muon is to a weight matrix what the sign is to a coordinate: it keeps the directions of the gradient and makes their sizes equal.
Bernstein and Newhouse state the theorem for a list of layers, under the title "Shampoo as steepest descent under the spectral norm", which is the operator norm by another name. The link to that older optimizer is one line.
Corollary 6.5.6 (Shampoo without its memory). For \(G = U\Sigma V^{\top}\), \((GG^{\top})^{-1/4}\, G\, (G^{\top}G)^{-1/4} = UV^{\top}\), with pseudo-inverse powers if \(G\) is rank-deficient.
Proof. \((U\Sigma^2U^{\top})^{-1/4}\; U\Sigma V^{\top}\; (V\Sigma^2V^{\top})^{-1/4} = U\,\Sigma^{-1/2}\,\Sigma\,\Sigma^{-1/2}\,V^{\top}\). ∎
Shampoo (2018) multiplies a matrix gradient by these two factors, with \(GG^{\top}\) and \(G^{\top}G\) summed over all past steps, and its descendant SOAP runs Adam in the eigenbasis of the two sums. Muon averages the gradients first and orthogonalizes after.V. Gupta, T. Koren and Y. Singer, ICML 2018 (arXiv 1802.09568); N. Vyas and others, ICLR 2025 (arXiv 2409.11321). The notes of the race name the kinship where they tell how the optimizer came about:
Our use of a Newton-Schulz iteration for orthogonalization traces to Bernstein & Newhouse (2024), who suggested it as a way to compute Shampoo [5, 6] preconditioners, and theoretically explored Shampoo without preconditioner accumulation. In particular, Jeremy Bernstein @jxbz sent us the draft, which caused us to experiment with various Newton-Schulz iterations as the orthogonalization method for this optimizer. If we had used SVD instead of a Newton-Schulz iteration, this optimizer would have been too slow to be useful. Bernstein & Newhouse also pointed out that Shampoo without preconditioner accumulation is equivalent to steepest descent in the spectral norm, and therefore Shampoo can be thought of as a way to smooth out spectral steepest descent. The proposed optimizer can be thought of as a second way of smoothing spectral steepest descent, with a different set of memory and runtime tradeoffs compared to Shampoo.
—K. Jordan and others, the project page of modded-nanogpt (2024). Section "Provenance", its last paragraph, as read in November 2026. The proposed optimizer is Muon.
Which matrices get this ball is decided by what a layer maps. A hidden matrix with \(n\) rows and \(m\) columns maps vectors of unit root mean square to vectors of unit root mean square; the norm that measures this is \(\sqrt{m/n}\,\lVert W\rVert_{\mathrm{op}}\), and the step of Theorem 6.5.5 in it is \(-\eta\sqrt{n/m}\; UV^{\top}\). The code keeps that factor only where it exceeds one, as max(1.0, shape[0] / shape[1]) ** 0.5 on line 120 of optim.py (in the figure's panel, and in the listing under Definition 6.5.9): 2 for the first matrix of an MLP, which has four times as many rows as columns, and 1, not 1/2, for the second. An embedding maps one-hot vectors and gets another norm, which is the theory's reason to leave it to AdamW. For the unembedding the write-up is frank: "That such dynamics are also different for the output layer does not seem to follow from the theory, and is instead driven by empirics."
Why the loop orthogonalizes. No decomposition is computed, because of an identity.
Lemma 6.5.7 (odd polynomials act on singular values). If \(X = U\Sigma V^{\top}\) and \(p(X) = aX + b\,XX^{\top}X + c\,(XX^{\top})^2X\), then \(p(X) = U\,(a\Sigma + b\Sigma^3 + c\Sigma^5)\,V^{\top}\).
Proof. \(XX^{\top} = U\Sigma^2U^{\top}\), so \((XX^{\top})^kX = U\Sigma^{2k+1}V^{\top}\). ∎
Each pass of the loop applies the scalar map \(\varphi(x) = ax + bx^3 + cx^5\) to every singular value and leaves the singular vectors alone. Dividing by the Frobenius norm first puts every singular value in \((0, 1]\). The classical Newton-Schulz iterations are the cubic \(\tfrac32 x - \tfrac12 x^3\), for which \(1 - \varphi(x) = \tfrac12 (1 - x)^2 (x + 2)\), and the quintic \((15x - 10x^3 + 3x^5)/8\), whose derivative is \(\tfrac{15}{8}(1 - x^2)^2\). Both converge to one from any start in \((0, 1]\), and both start slowly: a small singular value grows by the slope at zero, 1.5 or 1.875, at each pass, so by 7.6 or 23 in five. Muon's coefficients make the opposite trade.
Proposition 6.5.8 (the tuned quintic). Let \(\varphi(x) = 3.4445\,x - 4.7750\,x^3 + 2.0315\,x^5\). (i) \(\varphi^{\prime}(0) = 3.4445\) and \(\varphi(x) \ge 2.16\,x\) on \((0, 0.554]\). (ii) \(\varphi(1) = 0.701\): one is not a fixed point. (iii) \(\varphi\) maps the interval \(J = [0.681, 1.203]\) into itself, and every \(x \in (0, 1]\) enters \(J\) after finitely many passes and stays.
Proof. On the positive axis \(\varphi^{\prime}\) vanishes at \(x_1 = 0.5545\), a maximum with \(\varphi(x_1) = 1.2024\), and at \(x_2 = 1.0501\), a minimum with \(\varphi(x_2) = 0.6818\). (i) \(\varphi(x)/x = a + bx^2 + cx^4\) decreases on \((0, x_1]\) from \(a\) to 2.168. (iii) On \(J\) the least value of \(\varphi\) is \(\varphi(x_2)\) and the greatest is at an end, at most 1.14. A point below \(x_1\) at least doubles at each pass until it passes \(x_1\) or lands in \(J\); a point between \(x_1\) and \(J\) is sent into \((1.13, 1.203)\); the rest of \((0, 1]\) is in \(J\). ∎
In five passes every singular value above 0.0015 of the Frobenius norm is inside \(J\), where the classical quintic would have brought 0.0015 to 0.035. The price is that inside $$J$$ nothing settles: the fixed point of \(\varphi\) at 0.868 repels, with slope \(-1.58\), and the values wander between 0.68 and 1.13. One of the method's authors says it plainly: "This iteration oscillates and in fact does not converge!", and "In short, the cursed quintic iteration sacrifices convergence for speed."J. Bernstein, "Newton-Schulz", in the documentation of Modula at docs.modula.systems. Whether accuracy matters is disputed. The write-up on Muon reports, of the distance \(\varepsilon\) from one within which the singular values have to end, that "empirically, \(\varepsilon\) can be as high as around 0.3 without harming the loss curve for Muon-based trainings"; a method of 2025 that picks the best polynomial at each pass (N. Amsel, D. Persson, C. Musco and R. M. Gower, "The Polar Express", arXiv 2505.16932) reports lower losses at every learning rate, and gained ten steps when the race adopted it, less than the paper's margin.
In the code. The whole orthogonalization is the function below: one line inside its loop, and two sets of coefficients above it. It is orthogonalize in gpt-pp's optim.py.
# Coefficients (a, b, c) of the Newton-Schulz step X <- a X + b (X X^T) X + c (X X^T)^2 X.
NS_CLASSIC = (2.0, -1.5, 0.5) # converges to singular values of exactly 1, slowly for the small ones
NS_TUNED = (3.4445, -4.7750, 2.0315) # Jordan et al.: a steeper start (a = 3.44); ends in [0.68, 1.20], not at 1
def orthogonalize(g: Tensor, steps: int = 5, coefficients: tuple[float, float, float] = NS_TUNED) -> Tensor:
...
a, b, c = coefficients
x = g.bfloat16() if g.is_cuda else g
tall = x.size(-2) > x.size(-1)
if tall: # work with the wide orientation so that the Gram matrix x x^T is the small one
x = x.mT
x = x / (x.norm(dim=(-2, -1), keepdim=True) + 1e-7)
for _ in range(steps):
gram = x @ x.mT
x = a * x + (b * gram + c * gram @ gram) @ x
if tall:
x = x.mT
return x.to(g.dtype)
The line inside the loop is \(p(X)\) of Lemma 6.5.7, with gram for \(XX^{\top}\), and NS_TUNED holds the coefficients of Proposition 6.5.8. g has the shape (..., rows, columns): each matrix of a stack is divided by its own norm and iterated on its own.
Quirk. On a card the loop runs in bfloat16, the 16-bit format of Section 6.7, and the result is cast back to the gradient's type; on a machine without one it keeps that type. A matrix with more rows than columns is transposed first, so that the Gram matrix is the smaller of the two it could be. The division is by the Frobenius norm plus \(10^{-7}\), so a zero gradient gives a zero update. And the result is not \(UV^{\top}\): by Proposition 6.5.8 its singular values end in \([0.68, 1.20]\) where they started above about 0.0015 of the Frobenius norm and are at most \(3.4445^5 = 485\) times larger where they started below, and no run was compared with an exact orthogonalization.
Quirk. NS_CLASSIC is not the classical quintic of this section: its polynomial, \(2x - 1.5x^3 + 0.5x^5\), also has the value 1 and the slope 0 at one, but its slope at zero is 2, not 1.875, and its curvature at one is 1, not 0. In five passes it would multiply a small singular value by 32 and bring 0.0015 to 0.048, where the classical quintic brings it to 0.035. Only the tests use it: Muon.step passes no coefficients, so every run used NS_TUNED.
Muon as it is run. In formulas, one step is the following.
Definition 6.5.9 (Muon, as gpt-pp runs it). For a hidden matrix \(W\) with \(n\) rows and \(m\) columns, gradient \(G_t\), momentum \(\mu = 0.95\) and rate \(\eta\),
\[\begin{aligned} M_t &= \mu M_{t-1} + (1 - \mu)\, G_t, \\ B_t &= (1 - \mu)\, G_t + \mu M_t, \\ O_t &= \varphi^{\circ 5}\bigl(B_t/\lVert B_t\rVert_F\bigr), \quad \varphi \text{ acting on the singular values},\\ W_t &= W_{t-1} - \eta\,\sqrt{\max(1, n/m)}\; O_t . \end{aligned}\]The embedding and the unembedding are trained by AdamW.
In the code. The four lines of the definition are four lines of the step, applied at once to all matrices of one shape. It is Muon.step in gpt-pp's optim.py.
@torch.no_grad()
def step(self, closure=None) -> None:
for index, group in enumerate(self.param_groups):
by_shape: dict[torch.Size, list[Tensor]] = {}
for p in group["params"]:
by_shape.setdefault(p.shape, []).append(p)
for shape, params in by_shape.items():
grad = torch.stack([p.grad for p in params]) # (K, rows, columns)
buffer = self._momentum.setdefault((index, shape), torch.zeros_like(grad))
buffer.lerp_(grad, 1 - group["momentum"])
update = grad.lerp_(buffer, group["momentum"]) if group["nesterov"] else buffer
update = orthogonalize(update, group["ns_steps"])
step_size = group["lr"] * max(1.0, shape[0] / shape[1]) ** 0.5
for p, u in zip(params, update.unbind(0)):
if group["weight_decay"] > 0:
p.mul_(1 - group["lr"] * group["weight_decay"]) # decoupled, as in AdamW
p.add_(u, alpha=-step_size)
buffer after its lerp_ is \(M_t\); update is \(B_t\), and after orthogonalize it is \(O_t\); step_size is \(\eta\sqrt{\max(1, n/m)}\), with shape[0] the \(n\) rows and shape[1] the \(m\) columns. grad stacks the \(K\) matrices of one shape, so the default model makes three calls of orthogonalize a step, one for each shape. The buffer starts at zero and is not corrected for it as Adam's averages are in (6.2): orthogonalize discards the size of \(B_t\), so the first step is the one the gradient alone would give.
Quirk. The momentum is one tensor for each shape, kept in self._momentum and not in the state that torch.optim.Optimizer saves, so state_dict() does not carry it. The decay, which the recipe leaves off, multiplies the weights by 1 - lr * weight_decay, with the rate and not with the scaled step_size. p.grad goes into torch.stack without a check for None, where Lion.step in the same file skips a parameter without a gradient, and the closure argument is ignored. The second lerp_, Nesterov's form, overwrites the stacked copy of the gradients, not the gradients.
PyTorch. PyTorch 2.9.1, the version of every run of this series, has a Muon of its own, torch.optim.Muon, with the same coefficients, five passes, momentum 0.95 and Nesterov's form; gpt-pp does not use it. It differs from the class above: its defaults are a rate of \(10^{-3}\) and a decay of 0.1; it takes one matrix at a time; its loop runs in bfloat16 without a card too, with the norm clamped from below where gpt-pp adds \(10^{-7}\); it skips parameters without a gradient; and its step scaling is a choice (adjust_lr_fn). Its docstring writes the momentum as a sum, \(B \leftarrow \mu B + g\), while its code computes the average buf.lerp_(grad, 1 - momentum), as gpt-pp's does. By the release notes, which the series did not test, PyTorch 2.14 adds a scaling option to it and 2.15 fixes "Muon overwriting momentum with bf16 grads".
The rate is \(\eta = 0.02\) at every size. AdamW has \(\beta = (0.8, 0.95)\) and \(\epsilon = 10^{-10}\),The \(\epsilon\) of (6.2) is a precision too. It is added to \(\sqrt{\hat v}\), so where gradients are smaller than \(\epsilon\) the update is \((\eta/\epsilon)\,\hat m\): momentum SGD, and a slow one. Below an output layer that starts at zero the gradients are that small, and the race that introduced Muon met them in 2025: "The speedrun is so tight now that even Adam's default epsilon parameter is already causing problems." a rate of 0.3 for the embedding and of 0.008 for the unembedding at width 768, both multiplied by \(\sqrt{768/d}\) at width \(d\). There is no weight decay and no gradient clipping. The four matrices of an attention layer are orthogonalized separately. The embedding's rate is almost forty times the unembedding's: a lookup table whose rows move only when their token occurs and a dense layer that feeds the loss are different problems, and rates set by group do by hand what Adam does by coordinate.
In the code. Which parameter goes to which rule, and at which rate, is settled in two functions: the first gives each of four groups a rate and a decay, the second hands the groups to the optimizers. They are group_settings and build_optimizers in gpt-pp's optim.py.
def group_settings(cfg: TrainConfig, tokens_per_step: Optional[float] = None) -> dict[str, dict[str, float]]:
...
scale = 1.0
if cfg.lr_ref_batch > 0 and tokens_per_step is not None:
scale = math.sqrt(tokens_per_step / cfg.lr_ref_batch)
def rate(value: Optional[float]) -> float:
return scale * (cfg.lr if value is None else value)
return {
"embedding": dict(lr=rate(cfg.lr_embedding), weight_decay=cfg.wd_embedding),
"unembedding": dict(lr=rate(cfg.lr_unembedding), weight_decay=cfg.wd_unembedding),
"matrices": dict(lr=rate(cfg.lr), weight_decay=cfg.weight_decay),
"gains": dict(lr=rate(cfg.lr_gains), weight_decay=0.0),
}
def build_optimizers(model, cfg: TrainConfig, tokens_per_step: Optional[float] = None) -> list[torch.optim.Optimizer]:
...
params = model.param_groups()
settings = group_settings(cfg, tokens_per_step)
if not params["unembedding"]:
settings["embedding"] = settings["unembedding"]
def groups(names: tuple[str, ...]) -> list[dict]:
return [dict(params=params[name], name=name, base_lr=settings[name]["lr"], **settings[name])
for name in names if params[name]]
fused = all(p.is_cuda for group in params.values() for p in group)
betas = (cfg.beta1, cfg.beta2)
...
if cfg.optimizer == "muon+adamw":
adamw = torch.optim.AdamW(groups(("embedding", "unembedding", "gains")), betas=betas, eps=cfg.eps, fused=fused)
muon = Muon(groups(("matrices",)), momentum=cfg.momentum, nesterov=cfg.nesterov, ns_steps=cfg.ns_steps)
return [adamw, muon]
The four keys of group_settings are the four groups, and rate lets a group without a rate of its own follow cfg.lr, the \(\eta\) of Definition 6.5.9. The last branch is the recipe: one AdamW of (6.2) for the three groups that are not hidden matrices, and Muon for the hidden matrices. base_lr is the peak that the schedule of Section 6.3 multiplies.
Quirk. The groups are made by GPT.param_groups in model.py. matrices is the weight of every linear map inside the blocks: query, key, value and output of each attention layer and the two or three of each MLP. embedding is the token table, with a learned position table where there is one, which therefore trains at the embedding's rate; unembedding is empty when the two tables are tied; gains is everything else: gains and biases of normalizations, biases of linear maps, a learned factor on a head's scores. In the 2026 model the gains group is empty, and its rate of 0.01 acts on nothing.
Quirk. When embedding and unembedding are one tied matrix, params["unembedding"] is empty and the embedding group takes the unembedding's rate and decay. That is the rung of the ladder that trains the 2019 model under the 2026 training recipe: its one table trains at 0.0098, and the next rung, which unties the two, gives the embedding 0.3674. Example 6.8.1 has what the embedding's rate did to the tied table.
Quirk. The names of the rules are PyTorch's. adam is torch.optim.Adam, whose weight decay is the L2 penalty of Definition 6.2.1; adamw is torch.optim.AdamW, which in PyTorch 2.9.1 is Adam with decoupled_weight_decay=True and multiplies the weights by 1 - lr * weight_decay, as (6.2) does. Every run that this part calls Adam had a decay of zero, at which the two are the same update: no model of the series was trained by Adam with an L2 penalty. The list returned is [adamw, muon], so AdamW steps first, which changes nothing, and fused asks for PyTorch's fused implementation only when every parameter is on a card. By the release notes of PyTorch 2.11, which the series did not test, that implementation now uses fused multiply-add instructions, which would change the last bits of the tables' updates.
Quirk. The recipe's constants are the defaults of TrainConfig in config.py, and they are not PyTorch's: torch.optim.AdamW has \(\beta = (0.9, 0.999)\) and \(\epsilon = 10^{-8}\). No rate is scaled with the batch either: lr_ref_batch is 0, so scale stays at one, and Muon's 0.02 was chosen at 65,536 tokens a step and used unchanged at the 147,456 of the 12-layer model's long run (Section 7.3). The probe of Example 6.8.1 on that model, at 49,152 tokens a step, could not tell 0.01 from 0.02.
Quirk. The rule in the width is not in these functions but in train_preset of config.py, which rounds: lr_embedding=round(0.3 * scale, 4) and lr_unembedding=round(0.008 * scale, 5) with scale = (768 / width) ** 0.5, which is 0.7348 and 0.0196 at width 128 and 0.3674 and 0.0098 at 512. Muon's rate has no rule, and the 2019 recipe has none at all: one rate chosen by size (\(6 \times 10^{-3}\) for gpt-pico, \(2.4 \times 10^{-3}\) on the ladder, \(6 \times 10^{-4}\) for the 12-layer model) and a warmup of 4, 8 and 32 million tokens. Until the last commit of the code the rule went by the preset's own width, so a width set on the command line kept the preset's rates; Section 7.5 has what that did to the scaling grid.
Quirk. Where a recipe clips, the trainer does it in one line, torch.nn.utils.clip_grad_norm_(model.parameters(), train_cfg.grad_clip) in train of train.py: one norm over every parameter, the \(\tau\) of Section 6.1, before either optimizer steps. The 2019 recipe clips at norm one, the 2026 recipe does not clip, and the loop that later trains the model to answer (Section 7.8) has no clipping at all. With a warmup and a clip of its own the best AdamW of Example 6.8.2 gained 0.033 nats.
Example 6.5.10 (what the loop costs). The setup: the optimizer's step and the forward and backward pass of the ladder's model and of the 12-layer model, timed apart on one card. One pass of the loop on an \(n \times m\) matrix with \(m \le n\) is three matrix products, at most \(6nm^2\) floating-point operations; the forward and backward pass through the same matrix on a batch of \(B\) tokens costs \(6nmB\). Five passes add at most the fraction \(5m/B\): 0.7% at width 768 and the race's 524,288 tokens a step, 3.9% at width 512 and the ladder's 65,536. Measured, the step takes 11 ms beside 250 ms on the ladder's model, where AdamW on every parameter would take 3 ms, and 35 ms beside 293 ms on the 12-layer model with 32 sequences of 1,024 tokens, against 6 ms. The cost of the loop does not depend on the batch, so its share falls as the batch grows. ◁
What broke at scale. The first report on a large model trained this way came in February 2025.
Weight Decay While Muon performs significantly better than AdamW on a small scale as shown by Jordan et al., 2024, we found the performance gains diminish when we scale up to train a larger model with more tokens. We observed that both the weight and the layer output’s RMS keep growing to a large scale, exceeding the high-precision range of bf16, which might hurt the model’s performance. To resolve this issue, we introduced the standard AdamW (Loshchilov & Hutter, 2019) weight decay mechanism into Muon.
\[\mathbf{W}_{t}=\mathbf{W}_{t-1}-\eta_{t}(\mathbf{O}_{t}+\lambda\mathbf{W}_{t-1})\]—J. Liu and others, "Muon is scalable for LLM training" (2025). arXiv 2502.16982, Section 2.2, the paragraph on weight decay with its Equation 3, in which \(\mathbf{O}_t\) is the orthogonalized momentum; RMS is the root mean square and bf16 the 16-bit format of Section 6.7. The 52% below is from their Section 3.2.
The cause is visible in Definition 6.5.9: every step adds a matrix of operator norm about \(\eta\), whatever the gradient, and nothing takes it away. Decay does.
Proposition 6.5.11 (decay is a constraint). Let \(\theta_t = (1 - \eta_t\lambda)\theta_{t-1} - \eta_t u_t\) with \(0 < \eta_t\lambda < 1\) and \(\lVert u_t\rVert \le s\) in some norm. Then \(\lVert\theta_t\rVert \le \max(\lVert\theta_0\rVert,\ s/\lambda)\) for every \(t\).
Proof. \(\lVert\theta_t\rVert \le (1 - \eta_t\lambda)\lVert\theta_{t-1}\rVert + \eta_t\lambda\cdot s/\lambda\), a weighted mean of \(\lVert\theta_{t-1}\rVert\) and \(s/\lambda\). ∎
For Muon the norm is the operator norm and \(s\) is about one, so with decay the largest singular value of every matrix stays under about \(1/\lambda\), and with it the attention scores. For AdamW the norm is the max-norm and \(s\) is the constant of Proposition 6.2.2. Either optimizer with decay solves a constrained problem on the ball of its own norm, of radius \(1/\lambda\).S. Xie and Z. Li, ICML 2024 (arXiv 2404.04454), Theorem 1.1; L. Chen, J. Li and Q. Liu, "Muon optimizes under spectral norm constraints", 2025 (arXiv 2506.15054).
With a decay of 0.1 and an update matched in size to AdamW's, that report puts Muon at about 52% of the training compute of AdamW for the same loss. The next failure, on the way to a model of 32 billion active parameters, was attention scores that exploded more often under Muon than under AdamW; the remedy rescales the query and key matrices of the offending heads, and the model was trained on 15.5 trillion tokens without a loss spike. gpt-pp avoids the failure by construction: its scores are 12 times a cosine (Section 3.4). Since mid-2025 four open reports on large models train their matrices with Muon and keep AdamW for the embeddings and for the few parameters that are not matrices.Kimi Team, "Kimi K2: open agentic intelligence", 2025 (arXiv 2507.20534), Section 2.1; GLM-4.5 Team, 2025 (arXiv 2508.06471); V. Singh and others, "Arcee Trinity Large technical report", 2026 (arXiv 2602.17004); DeepSeek-AI, "DeepSeek-V4", 2026 (arXiv 2606.19348), Sections 2.4 and 4.2.2.
What is and is not established. The step is the steepest one in a norm that measures what a layer does (Theorem 6.5.5), and in that norm it has the guarantee that gradient descent has in the Euclidean one.If the gradient is \(\ell_{\ast}\)-Lipschitz from the operator norm to the nuclear norm, a step lowers the loss by at least \(\eta\lVert\nabla f\rVert_{\ast} - \tfrac12\ell_{\ast}\eta^2\): W. Shen and others, "On the convergence analysis of Muon", 2025 (arXiv 2505.23737), which also exhibits a problem with Gaussian inputs on which Muon is predicted, and found, to be slower than gradient descent. And at GPT-2's size the gain has survived an adversarial test, which the write-up describes.
Muon has a slower per-step wallclock time than AdamW, so if there existed hyperparameters that could make AdamW as sample-efficient as Muon, then it would be possible to set a new record by simply chucking Muon out of the window and putting good old AdamW back in. Therefore, to trust that Muon is better than AdamW, at least for training small language models, you actually don’t need to trust me (Keller Jordan) at all. Instead, you only need to trust that there exist researchers in the community who know how to tune AdamW and are interested in setting a new NanoGPT speedrunning record. Isn’t that beautiful?
—K. Jordan and others, "Muon" (2024), section "Discussion".
No record since has done so. On the race's fixed 12-layer model, counted in 2026 in steps to the target loss: 8,225 for spectral descent without momentum, 3,250 for tuned Muon, 2,690 for the record, a hybrid with SOAP, and 5,625 for an Adam entry that its own author flags as probably undertuned.
What is not established is the size of the gain at scale. The independent benchmarks of 2025 found that "the speedup of matrix-based optimizers is inversely proportional to model scale, decreasing from 1.4x over AdamW for 0.1B parameter models to merely 1.1x for 1.2B parameter models".Wen et al. (2025), abstract. The replies: Wen et al. (2026), cited in Section 6.4; M. Khona and others, 2026 (arXiv 2607.20548); T. Shen and others, 2026 (arXiv 2606.21514). Three papers of 2026 answer. One attributes the shrinkage to the constant decoupled decay of Section 6.4 and recovers 20 to 30% by fixing the norms instead; one reports that Muon and SOAP keep their lead over AdamW at several billion parameters and batches of up to 100 million tokens; one finds the gains mixed and sensitive to the schedule.
Nor is the reason for the gain settled. A study of where it sits ties it back to Section 6.1: on heavy-tailed tasks Muon matches Adam on the frequent classes and improves on the rare ones, and applied only to the value, output and MLP matrices it recovers almost all of its effect. Those are the matrices that Section 5.4 reads as key-value memories.S. Wang and others, "Muon outperforms Adam in tail-end associative memory learning", 2025 (arXiv 2509.26030), Section 3, Observations 1 and 3. And the optimizers that ship in 2026 are no longer pure Muon: they add a second moment for each neuron after the orthogonalization and more accurate polynomials. Equal steps along singular directions and equal steps along coordinates turned out to be complements.
The edge of stability
Train a small network by gradient descent on its whole training set and track its sharpness \(S(\theta)\), the largest eigenvalue of the Hessian of the loss. The local quadratic stability threshold is \(2/\eta\), but training can keep reducing the loss while the sharpness crosses it repeatedly.
gradient descent on all 20 points at once: a tanh network 1 → 100 → 1 (301 parameters) fits 20 points of 4x³ − 3x. Sharpness: the largest eigenvalue of the Hessian of the loss, 5.8 at the start. 5,000 steps at each learning rate: rate 2 ÷ rate the sharpness reaches it and then stays between the loss, then → at the end 0.1 20 not yet: it is at 15.6 still climbing 0.26 (start) → 0.012 0.15 13.3 at step 2,520 12.2 and 14.5 0.014 → 0.0084 0.2 10 at step 1,120 8.6 and 11.3 0.022 → 0.0035 0.25 8 at step 580 6.5 and 9.4 0.039 → 0.0037 the sharpness at rate 0.2, every 100 steps, drawn between 5 and 11.5: ▂▂▂▂▂▂▃▄▅▅▆▇▇██▅▆▇▇██▅▆▆▇▇█▅▆▆▇▇██▆▆▇▇▇█▆▆▆▇▇▇▇▇▆▆▆ (the live figure needs JavaScript; it trains the network in the page and loads no data)
At the rate 0.2 the sharpness climbs from 5.8 to \(2/0.2 = 10\) in 1,120 steps and then saws between 8.6 and 11.3 for the remaining 3,880, averaging 10.0, while the loss falls another six times over, rising at 191 of those steps. At the rate 0.1 the ceiling of 20 is not reached in 5,000 steps, and the loss never once rises. Classical analysis has room only for the second behavior.
Lemma 6.6.1 (the descent lemma). If \(\lVert\nabla^2 L(\theta)\rVert_{\mathrm{op}} \le \ell\) for all \(\theta\), then \(L(\theta - \eta\nabla L(\theta)) \le L(\theta) - \eta\,(1 - \tfrac12\eta\ell)\,\lVert\nabla L(\theta)\rVert^2\), so the loss falls at every step if \(\eta < 2/\ell\).
Proof. By Taylor's formula \(L(\theta + \delta) \le L(\theta) + \langle\nabla L(\theta), \delta\rangle + \tfrac12\ell\lVert\delta\rVert^2\); put \(\delta = -\eta\nabla L(\theta)\). ∎
Proofs of convergence choose \(\eta\) below \(2/\ell\) and never look at the curvature again. In 2021 Cohen and coauthors looked, on real networks trained with the full batch.
In more detail, as we train neural networks using gradient descent with step size \(\eta\), we measure the evolution of the sharpness — the maximum eigenvalue of the training loss Hessian. Empirically, the behavior of the sharpness is consistent across architectures and tasks: so long as the sharpness is less than the value \(2/\eta\), it tends to continually rise (§3.1). We call this phenomenon progressive sharpening. The significance of the value \(2/\eta\) is that gradient descent on quadratic objectives is unstable if the sharpness exceeds this threshold (§2). Indeed, in neural network training, if the sharpness ever crosses \(2/\eta\), gradient descent quickly becomes destabilized — that is, the iterates start to oscillate with ever-increasing magnitude along the direction of greatest curvature. Yet once this happens, gradient descent does not diverge entirely or stall. Instead, it enters a regime we call the Edge of Stability (§3.2), in which (1) the sharpness hovers right at, or just above, the value \(2/\eta\); and (2) the train loss behaves non-monotonically, yet consistently decreases over long timescales. In this regime, gradient descent is constantly “trying” to increase the sharpness, but is constantly restrained from doing so. The net effect is that gradient descent continues to successfully optimize the training objective, but in such a way as to avoid further increasing the sharpness.
—J. M. Cohen, S. Kaur, Y. Li, J. Z. Kolter and A. Talwalkar, "Gradient descent on neural networks typically occurs at the edge of stability" (2021). ICLR 2021 (arXiv 2103.00065), Section 1.
Training does not stay where the lemma applies: the value at which the sharpness hovers is the one at which Proposition 6.1.1 says the iteration is about to diverge. With momentum and with Adam the threshold moves.
Proposition 6.6.2 (thresholds of stability). On a quadratic with largest curvature \(S\), gradient descent is stable if and only if \(S < 2/\eta\). With momentum as a moving average, \(m_t = \beta m_{t-1} + (1 - \beta) g_t\) and \(\theta_{t+1} = \theta_t - \eta m_t\), it is stable if and only if
\[S \;<\; \frac{2 + 2\beta}{(1 - \beta)\,\eta},\]which is \(38/\eta\) at \(\beta = 0.9\). For Adam with its second moment frozen, \(P = \mathrm{diag}(\sqrt{\hat v}) + \epsilon I\), the same holds with \(S\) the largest eigenvalue of \(P^{-1}H\), the preconditioned sharpness.
Proof. Along an eigenvector of curvature \(h\), eliminating \(m\) gives \(x_{t+1} = (1 + \beta - \eta(1 - \beta)h)\, x_t - \beta\, x_{t-1}\). Both roots of \(z^2 - pz + \beta\) are inside the unit circle if and only if \(\lvert p\rvert < 1 + \beta\), that is, \(\eta(1 - \beta)h < 2 + 2\beta\). Frozen Adam is the same iteration in the variable \(P^{1/2}\theta\), with the Hessian \(P^{-1/2}HP^{-1/2}\). ∎
In full-batch training with Adam the preconditioned sharpness is measured to settle at this value while the raw sharpness keeps rising: adaptive methods "can keep advancing into high-curvature regions, while adapting the preconditioner to compensate".J. M. Cohen and others, "Adaptive gradient methods at the edge of stability", 2022 (arXiv 2207.14484), abstract and Figure 1.
Why the iteration does not diverge once it is at the threshold was explained in 2022: as the iterates swing out along the top eigenvector of the Hessian, "the cubic term in the local Taylor expansion of the loss function causes the curvature to decrease until stability is restored".A. Damian, E. Nichani and J. D. Lee, "Self-stabilization: the implicit bias of gradient descent at the edge of stability", ICLR 2023 (arXiv 2209.15594), abstract; the map below is their Section 4.1, Equation 3, where our \(a\) and \(b\) are \(\alpha\) and \(\beta\).
Proposition 6.6.3 (the cycle of self-stabilization). Let \(x_t\) be the displacement along the top eigenvector \(u\) of the Hessian and \(y_t = S(\theta_t) - 2/\eta\) the excess of the sharpness. Let \(a = -\langle\nabla L, \nabla S\rangle > 0\), the rate at which a gradient step raises the sharpness, and \(b = \lVert\nabla S\rVert^2\). To leading order,
\[x_{t+1} = -(1 + \eta y_t)\, x_t, \qquad y_{t+1} = y_t + \eta\,\bigl(a - \tfrac12 b\, x_t^2\bigr). \tag{6.3}\]In the limit of small steps the quantity \(y^2 - a\ln x^2 + \tfrac12 b\, x^2\) is conserved, the orbits are closed curves around the point \(x^2 = 2a/b\), \(y = 0\), and over a cycle the averages are \(\langle y\rangle = 0\) and \(\langle x^2\rangle = 2a/b\).
Proof. The first equation is gradient descent on a quadratic of curvature \(2/\eta + y_t\). In the second, \(\eta a\) is the progressive sharpening of one step, and the cubic term of the loss adds to the gradient the vector \(\tfrac12 x^2\,\nabla S\), a step along which lowers \(S\) by \(\tfrac12\eta b x^2\). With \(q = \ln x^2\), (6.3) becomes \(\dot q = 2\eta y\) and \(\dot y = \eta(a - \tfrac12 b e^{q})\), which conserves \(y^2 - aq + \tfrac12 b e^q\), a function with one minimum and closed level curves. On a closed orbit the means of \(\dot q\) and \(\dot y\) vanish. ∎
Below the threshold the oscillation along \(u\) is damped and the sharpness rises; above it the oscillation grows; when \(x^2\) passes \(2a/b\) the cubic term outweighs the sharpening and the sharpness falls back. On average the sharpness sits at \(2/\eta\), as measured, and the measured oscillations follow this cycle.Averaged over the cycle, the oscillation acts as a penalty on the sharpness of exactly the strength that keeps training on the edge. A theory of 2024, the central flow, makes of this a differential equation for the mean trajectory: J. M. Cohen, A. Damian, A. Talwalkar, J. Z. Kolter and J. D. Lee, "Understanding optimization in deep learning with central flows", ICLR 2025 (arXiv 2410.24206), Section 3.2.1.
Two habits of practice become consequences. A warmup lowers the ceiling \(2/\eta_t\), or \(38/\eta_t\), slowly, and the cycle brings the sharpness down under it step by step (Section 6.3). And the second moment has to keep up: Adam's threshold applies to \(P^{-1}H\), so a preconditioner that lags behind a growing gradient pushes the preconditioned sharpness over the edge. Loss spikes have been traced to squared gradients that AdamW's second moment underestimates, one to eight steps before the spike, which is one more reason for \(\beta_2 = 0.95\). In 2026 the same behavior of the sharpness was measured up to seven billion parameters.M. Wortsman and others, NeurIPS 2023 (arXiv 2304.13013), abstract; D. S. Kalra and others, 2026 (arXiv 2601.16979), abstract.
Two limits should be kept in view. These propositions and this experiment are about the full batch. The initialization was also chosen: with starting weights of the usual size the sharpness of this network barely moves, from 38 to 43, so small ones were used. Of six seeds, all of which show the climb and the plateau, three have a loss that stalls at the edge instead of falling on. The run above is one of the other three.
Precision
gpt-pp multiplies its matrices in numbers of 16 bits and keeps its weights in 32. The 16-bit format is bfloat16: the sign and the eight exponent bits of a 32-bit float, and seven of its 23 fraction bits.
Example 6.7.1 (what 16 bits hold). The setup: the three formats as PyTorch reports them. The spacing of representable numbers just above one is \(2^{-7} = 0.0078\) in bfloat16, \(2^{-10} = 0.00098\) in the 16-bit format of the IEEE standard and \(2^{-23} = 1.2 \times 10^{-7}\) in float32. The largest number is \(3.4 \times 10^{38}\) in bfloat16 and in float32, and 65,504 in the IEEE format, whose smallest normal number is \(6.1 \times 10^{-5}\): a gradient of $$10^{-8}$$ is zero there and survives in bfloat16. In bfloat16 an addition changes a number only if the increment exceeds half the spacing, between 0.2% and 0.4% of the number: \(1 + 0.0039\) is 1, and \(1 + 0.004\) is 1.0078. ◁
Three consequences follow. The first is range: the IEEE format needed the loss scaled up so that gradients would not underflow, and bfloat16 has the range of float32. The second is that weights cannot live in 16 bits: the equilibrium step of Example 6.4.4 turns a matrix by 0.25% of its norm, at the threshold where a bfloat16 weight does not move at all. Since 2017 the rule has been a master copy of the weights in float32, with only the arithmetic of the forward and backward pass in 16 bits.P. Micikevicius and others, "Mixed precision training", ICLR 2018 (arXiv 1710.03740), abstract. The third is that 0.4% is too coarse wherever small terms are added to a large total, nearly equal numbers are subtracted, or an angle is computed. In gpt-pp the residual stream, the normalizations of the stream, the soft cap on the logits and the loss are computed in float32, and the rotation of queries and keys by their position is computed in float32 and rounded back to bfloat16; the matrix products, where the time goes, are in bfloat16, and so are the queries and keys when they are normalized and when the attention kernel reads them. So is Muon's loop, whose target is a band of width 0.5 around one (Proposition 6.5.8), far coarser than the format.
In the code. The precision of a forward pass is set outside the model: the trainer creates the weights in float32 and runs the pass inside PyTorch's autocast, which decides operation by operation. These are the lines of train in gpt-pp's train.py that set this up.
# ---- model and optimizers
torch.backends.cuda.matmul.allow_tf32 = True # TF32: float32 matrix products with a 10-bit mantissa
torch.backends.cudnn.allow_tf32 = True
torch.manual_seed(train_cfg.seed) # the same initialization on every rank
model = GPT(model_cfg).to(device)
torch.manual_seed(train_cfg.seed + 1 + rank) # different dropout masks on every rank
optimizers = build_optimizers(model, train_cfg, tokens_per_step)
compiled = torch.compile(model) if train_cfg.compile else model
net = DistributedDataParallel(compiled, device_ids=[device.index]) if world > 1 else compiled
autocast = torch.autocast("cuda", dtype=torch.bfloat16, enabled=on_gpu)
model holds the master copy in float32, and the optimizers of Section 6.5 are built on it and update it in float32. autocast is the context in which every forward pass of training runs. The model itself casts where it must: the logits to float32, self.unembed(x).float(), and the rotated queries and keys back to the type they came in, .to(x.dtype).
PyTorch. torch.autocast in PyTorch 2.9.1 casts by operation, not by model. Linear maps, matrix products and F.scaled_dot_product_attention run in bfloat16; softmax, layer_norm and the cross-entropy run in float32; F.rms_norm is on neither list and runs in the type of its input. So the stream stays in float32 because the embedding is float32 and every addition of a bfloat16 output promotes; the normalization of queries and keys, which is an F.rms_norm (Section 3.4), receives bfloat16 and returns it; and the logits are a bfloat16 product cast to float32 before the cap, so they carry the eight significant bits of bfloat16 whatever their type says. On the model's explicit attention path, which the cache and the analyses use, the scores are promoted to float32 before the softmax; the two paths therefore differ by this rounding, and Section 7.6 has what that does to sampled text.
PyTorch. The first two lines of the listing switch on TensorFloat-32, float32 products that keep ten bits of fraction, for the whole process and through flags that the documentation of PyTorch 2.9 calls "going to be deprecated"; Section 7.2 explains the mode and measures its speed.
Practice at scale has gone further: DeepSeek-V3 kept even the two moments of AdamW in bfloat16, "without incurring observable performance degradation". And the paper quoted in Section 6.4 counts precision among the things weight decay touches: "Additionally, we show that weight decay has another important practical benefit: enabling stable training with the bfloat16 precision."DeepSeek-AI, "DeepSeek-V3 technical report", 2024 (arXiv 2412.19437), Section 3.3.3; D'Angelo et al. (2024), Section 1. Where a step's time goes is measured in Section 7.2, and mixed precision on other hardware in Deep learning at scale.
Measured on the ladder
The ladder is one 8-layer, 512-wide model for each design step from the 2019 recipe to the 2026 recipe, each trained on the same tokens of FineWeb-EDU.
Example 6.8.1 (what the rates are worth, in short probes). The setup: the ladder's model on 50 million tokens of FineWeb-EDU and gpt-pico on 60 million tokens of TinyStories, one seed for each setting; the held-out loss in nats per token, against the yardsticks of Example 6.3.3. On both models Muon's rate of 0.02 was the best of 0.01, 0.02 and 0.04, by 0.02 nats on either side, and dropping the scaling of the AdamW rates by \(\sqrt{768/d}\) cost gpt-pico 0.066 nats; the rates 0.3 and 0.008 themselves were not told apart from 0.2 and 0.004 at either size. On the 12-layer model at 100 million tokens Muon's rates 0.01 and 0.02 end level, at 3.653 and 3.654, and 0.04 ends 0.025 behind. The 2019 recipe is more sensitive: at \(6 \times 10^{-4}\), the rate at which nanoGPT reproduces GPT-2 small, its 8-layer model was 0.50 nats behind its own best, which it reached at a rate four times larger. And when the embedding is tied to the unembedding, training the one matrix at the embedding's rate, 37 to 50 times the unembedding's, made the loss rise in the first 14 steps, from 10.5 to 145 in the worse of two configurations. These are short runs: they say where to start, not how the choices rank after billions of tokens. ◁
chart held-out loss against tokens seen (50M of FineWeb-EDU), one curve per rule at its best rate and schedule chart final held-out loss of every run against its rate rule schedule best rate ends at behind Muon under the recipe’s tokens to 4.181 SGD with momentum its own 1 4.536 +0.491 4.719 never Lion its own 0.0003 4.484 +0.439 4.523 never Adam the recipe’s 0.0012 4.225 +0.180 never AdamW its own 0.0024 4.181 +0.136 4.214 50M Muon the recipe’s 0.02 4.045 39M One line of the recipe changes: the rule for the hidden matrices. The embedding and the unembedding keep the recipe’s AdamW at rates 0.367 and 0.0098. SGD, Lion and AdamW each ran under the recipe’s schedule (2M tokens of warmup, no clipping) and under their own (8M of warmup, clipping at norm one) and stand at the better. Adam, which is AdamW without weight decay, ran under the recipe’s only. Muon, the recipe’s rule, ends lowest, at 4.045; AdamW follows at 4.181 under its own schedule, 0.136 behind (0.163 in a second seed). A loss of 4.181 costs AdamW 50M tokens and Muon 39M, 1.3 times fewer; SGD, Lion and Adam never reach it. (the live figure needs JavaScript and loads about 50 kB of data)
Example 6.8.2 (one model, five rules for its hidden matrices). The setup: the ladder's model with the 2026 block (58.7 million parameters) on 50 million tokens of FineWeb-EDU; the held-out loss at the end, in nats per token; one seed a run, and a second for the two best. One line of the 2026 recipe changes, the rule that updates the hidden matrices. The embedding and the unembedding keep the recipe's AdamW and its rates. The recipe's schedule, with its warmup over 2 million tokens and no clipping, was chosen with Muon. AdamW's own recipe at this size has a warmup over 8 million tokens and clips the gradient at norm one, so SGD, Lion and AdamW were each run under the recipe's schedule and again with those two, and each stands in the table at the better. Adam, which here is AdamW with its decay set to zero and not Adam with an L2 penalty, ran under the recipe's schedule only.
| rule for the hidden matrices | schedule | rates tried under it | best rate | held-out loss | behind Muon |
|---|---|---|---|---|---|
| SGD, momentum 0.9 | its own | 0.3, 1, 3 | 1 | 4.536 | 0.491 |
| Lion, \(\beta = (0.9, 0.99)\), decay 0.1 | its own | 0.0001, 0.0003, 0.001 | 0.0003 | 4.484 | 0.439 |
| Adam, \(\beta = (0.9, 0.95)\) | the recipe's | 0.0003, 0.0006, 0.0012, 0.0024, 0.0048 | 0.0012 | 4.225 | 0.180 |
| AdamW, the same with decay 0.1 | its own | 0.0006, 0.0012, 0.0024, 0.0048 | 0.0024 | 4.181 | 0.136 |
| Muon (the 2026 recipe) | the recipe's | 0.01, 0.02, 0.04 | 0.02 | 4.045 |
Muon ends lowest and AdamW second. In a second seed of those two runs AdamW ends at 4.206 and Muon at 4.043: Muon's advantage over the best AdamW is about 0.15 nats at 50 million tokens (0.136 and 0.163 in two seeds), and the two seeds are 0.025 apart under AdamW and 0.002 under Muon. Muon is at 4.181 after 38.8 million tokens, 1.3 times fewer than AdamW's 50 million, where the benchmarks of 2025 found 1.4 times at 0.1 billion parameters; no other rule reaches that loss under either schedule.
Bug, fixed. The first version of this comparison ran every rule under the recipe's schedule only, and that schedule handicaps the others; the fault was in the experiment, not in the code, and the table above is the rerun. Under the recipe's schedule AdamW ends at 4.214, 0.169 behind Muon, its loss jumping by half a nat from the rate 0.0012 to 0.0024; its own warmup and clipping move the jump up one rate, to between 0.0024 and 0.0048, and a cosine to a tenth on top of them does not help (4.207 at its best). SGD and Lion gain 0.18 and 0.04 nats from the warmup and clipping, each at a higher rate, and stay far behind: AdamW passes their final losses after 30.3 and 32.5 million tokens, so that Lion, its decay left at AdamW's 0.1, stays outside the factor of 1.2 of Section 6.2. Adam and AdamW, 0.011 apart under the recipe's schedule, are not told apart by one run each.Six more runs gave AdamW and Adam one rate for every parameter, the way of 2019, with a warmup over 4 million tokens, a cosine to a tenth and clipping. They end between 5.00 and 5.09 nats, with an embedding rate 77 to 306 times below the recipe's: a caution about the parametrization, not a comparison of rules.
The table does not show a longer run, a finer grid, a schedule tuned for each rule (all three got the warmup and clipping of AdamW's recipe), or Adam under a schedule of its own. Nor is the ladder's step from the 2019 to the 2026 training recipe (Section 5.6) Muon's alone. On a fixed model at 100 million tokens that step changes fourteen settings together (the rule, the tables' rates, decay, clipping, warmup and schedule among them), and the loss falls from 3.984 to 3.828 on the means of two seeds, by 0.155 nats. A run in each seed splits it: the 2026 recipe with AdamW in place of Muon on the hidden matrices, under AdamW's warmup and clipping, ends at 3.911 and at 3.915. So Muon is worth 0.083 and 0.086 nats, and the rest of the recipe (the tables' own rates, no decay on them, the schedule) 0.074 and 0.068. The second share is the less secure. The 2019 side's warmup over 8 million tokens was too short: with 32 million each of its rungs is 0.017 to 0.042 nats better in the first seed, and against that fairer rung the step is 0.140 and the rest of the recipe 0.057. ◁
In the code. The rule for the hidden matrices is chosen by the optimizer's name: Example 6.8.2 ran adamw+adamw, adam+adamw, lion+adamw and sgd+adamw in place of muon+adamw. It is the last branch of build_optimizers in gpt-pp's optim.py, of which Section 6.5 shows the beginning.
if cfg.optimizer.endswith("+adamw"):
recipe = TrainConfig() # the tables and gains keep the recipe's AdamW; beta1, beta2 and eps of cfg go to the matrices' rule
rest = torch.optim.AdamW(groups(("embedding", "unembedding", "gains")), betas=(recipe.beta1, recipe.beta2),
eps=recipe.eps, fused=fused)
matrices, rule = groups(("matrices",)), cfg.optimizer[: -len("+adamw")]
if rule == "adamw":
return [rest, torch.optim.AdamW(matrices, betas=betas, eps=cfg.eps, fused=fused)]
if rule == "adam":
return [rest, torch.optim.Adam(matrices, betas=betas, eps=cfg.eps, fused=fused)]
if rule == "lion":
return [rest, Lion(matrices, betas=betas)]
if rule == "sgd":
return [rest, torch.optim.SGD(matrices, momentum=cfg.momentum, nesterov=cfg.nesterov and cfg.momentum > 0)]
rest is the AdamW of the embedding and the unembedding, at their own rates; matrices goes to the rule named before the plus sign, with the run's rate, betas, \(\epsilon\) and decay.
Quirk. rest takes its betas and its \(\epsilon\) from a default TrainConfig(), \((0.8, 0.95)\) and \(10^{-10}\), whatever the run sets, so a run's beta1, beta2 and eps reach only the rule of the matrices. The AdamW runs of the table kept the recipe's \(\epsilon = 10^{-10}\) on the matrices too, because no setting changed it, and the table's SGD has Nesterov's momentum, the trainer's default. The plain names adamw and adam put all four groups into one optimizer instead, each at the rate group_settings gives it; the six runs of the sidenote above ran that way with one rate for every group, and end near 5.0 nats at every rate against 4.2 here.
Example 6.8.3 (the loss around gpt-pico). The setup: gpt-pico, 2,228,224 parameters, trained on 540 million tokens of TinyStories in 8,240 steps; the held-out loss on a plane of its weight space, on a grid of 61 by 43 points. The plane contains the initialization and the trained weights, and its second direction is the one in which the 21 saved checkpoints stray most from the line between the two. Distances are measured after each row of the embedding is divided by its root mean square: the model normalizes a row as it reads it, and measured raw, 99.9% of the squared distance from the initialization to the trained weights is the growth of the embedding of Remark 6.4.5.
map the held-out loss of gpt-pico on a plane of its weight space, seen from above: each digit is the
loss rounded down (A is 10, B is 11); I the initialization, T the trained weights, o the
checkpoints between them. Left to right: the straight line from I to T. Upward: sideways.
AAAAAAAAAAAA9999999988888753322222222222222222333
AAAAA99999999999988877764322222222222222222222223
9999999999999988887765432222222222222222222222222
9999999998888888765oooo2o222222222222222222222222
999888888888887654o322222o22222222222222222222222
8888888888887764o3222222222o22o222222211111122222
888888888887654333322222222222222o211111111111122
888888888876o443333333222222222221111111111111111
777788888o76554444333333222222221111o111111111111
77778888I776655554444333332222221111111oT11111111
7777888887766665555544444333322222111111111111112
7777788887776666665555554444333322222111111111222
7777888877777766666666655555544433332222222222222
loss at the start 8.32, at the end 1.54; the plane holds 91 % of the squared distance of the
checkpoints from the trained weights and 15 % of their squared moves from one to the next
(the live figure needs JavaScript; it loads about 0.7 MB of three.js for the surface and 21 kB of data)
Training starts at 8.318 nats, the loss of a uniform guess over 4,096 tokens, and ends at 1.540. On this plane the initialization sits on a plateau; the path leaves it sideways, drops down a steep face within its first two hundred steps, and spends the remaining eight thousand walking a shallow valley to the trained weights. The plane holds 91.0% of the squared distances of the checkpoints from the trained weights and 15.0% of their squared moves from one checkpoint to the next: most of the wandering is out of the picture. ◁
In 2026 the working recipe has two layers. Steps are made equal in size in a norm that fits the parameter, the max-norm for a table of embeddings and the operator norm for a matrix that maps the stream; and the rate, its schedule and the precision are chosen with the edge of stability in view. Section 7 trains gpt-pp with them, and Section 8 asks what the training learns.