5. The block

Draft

The residual stream

A transformer is one block, repeated. In gpt-pico, the smallest size of gpt-pp, six blocks operate on a stream 128 numbers wide at each token. Each block adds two updates: what attention gathers from the current and earlier positions, and what a small network computes at that position alone.

gpt-pico reads “Lily has a red ball. Tom wants the red” and predicts the next token.
A column a token: its residual stream, depth upward. In each block attention reads
earlier columns, then an MLP works on the column alone; both add to the stream.

predicts   Once  and   a     new   hat   .     She   likes to    ball  ball
block 6    |     |     |     |     |     ●     |     |     |     |     ‖     → ball 99.7 %
block 5    |     |     |     |     |     ●     |     |     |     |     ‖     → car 39.2 %
block 4    |     |     |     |     |     |     |     |     |     ●     ‖     → car 71.2 %
block 3    |     |     |     |     |     |     |     |     |     ●     ‖     → car 82.3 %
block 2    |     |     |     |     |     |     |     |     |     ●     ‖     → car 95.7 %
block 1    |     |     |     |     |     |     |     |     |     ●     ‖     → car 95.9 %
embedding  |     |     |     |     |     |     |     |     |     |     ‖     → ap 9.8 %
reads      start Lily  has   a     red   ball  .     Tom   wants the   red

‖ the stream of the last token   ● the word it reads hardest in that block
→ its first guess if gpt-pico stopped there
without block 6, head 4, which reads “ball”, the 99.7 % falls to 24.0 %
(the live figure needs JavaScript; it loads about 0.7 MB of three.js and, when first touched,
gpt-pico’s weights, 2.3 MB; those of the 2019 recipe, 1.8 MB, when it is picked)
gpt-pico reading a sentence. Each token is a rail, its residual stream, climbing through six blocks to a prediction of the next token. In every block attention reads earlier rails (arcs, darker for more weight), then an MLP (a square) works on the rail alone; both add to it. Tiles: the stream’s 128 numbers after each block. Arrows: the first guess if gpt-pico stopped there. Green: the selected token’s rail and the arcs that feed it. Pick a token, a block, a head, a part, a sentence or the 2019 recipe; turn the box. Measured on gpt-pico, in the page.

Consider the last position of "Lily has a red ball. Tom wants the red". Through four blocks it reads the word before it hardest, and its best guess, read from the stream early, is "car"; in blocks 5 and 6 it reads "ball", five tokens back, and only after block 6 is the guess "ball", at 99.7%. One head carries most of that: with head 4 of block 6 switched off the 99.7% falls to 24.0%, while switching off the whole MLP of any of blocks 2 to 5 costs under one point.

Three parts of the block are the subject here: the residual stream; the normalization through which each sublayer reads it; and the multilayer perceptron (MLP), or feed-forward layer. Attention was the subject of Section 3, position of Section 4.

Definition 5.1.1 (the block). Let \(X \in \mathbb R^{T \times d}\) be the residual stream of \(T\) tokens, one row of width \(d\) per token. A block with an attention sublayer, a feed-forward sublayer and a normalization \(N\) computes

\[Y = X + \mathrm{Attn}\bigl(N(X)\bigr), \qquad X^{+} = Y + \mathrm{MLP}\bigl(N(Y)\bigr). \tag{5.1}\]

\(N\) and \(\mathrm{MLP}\) act on each row separately; \(\mathrm{Attn}\) is the only map that moves information between rows. A model of \(L\) blocks starts from the embeddings of its tokens, applies (5.1) \(L\) times with different weights, and reads the logits of the next token from a row \(x\) of the last stream as \(W_U N(x)\), with \(W_U \in \mathbb R^{V \times d}\) the unembedding.

In the code. The first branch below is (5.1), one line an equation; the second is the arrangement of 2017, which Section 5.2 sets beside it. It is Block in gpt-pp's model.py.

class Block(nn.Module):
    ...
    def __init__(self, cfg: ModelConfig):
        super().__init__()
        self.pre_norm = cfg.norm_placement == "pre"
        self.norm1 = make_norm(cfg)
        self.attn = CausalSelfAttention(cfg)
        self.norm2 = make_norm(cfg) if cfg.use_mlp else None
        self.mlp = MLP(cfg) if cfg.use_mlp else None
        self.drop = nn.Dropout(cfg.dropout)

    def forward(self, x: Tensor, rot: Rotation, bias: Optional[Tensor]) -> Tensor:
        if self.pre_norm:
            x = x + self.drop(self.attn(self.norm1(x), rot, bias))
            if self.mlp is not None:
                x = x + self.drop(self.mlp(self.norm2(x)))
        else:
            x = self.norm1(x + self.drop(self.attn(x, rot, bias)))
            if self.mlp is not None:
                x = self.norm2(x + self.drop(self.mlp(x)))
        return x

The tensor x is the stream \(X\), of shape (batch, \(T\), \(d\)); after the first addition the same name holds \(Y\), and norm1 and norm2 are two separate copies of \(N\). The arguments rot and bias carry position to attention (Section 4), and drop is a dropout whose rate is zero in both recipes of this series. In an attention-only model, the kind used for the circuits of Section 9, norm2 and mlp are None and a block is one addition. The logits of the definition are made after the last block: one more norm, the model's final_norm, then the unembedding and, in the 2026 recipe, a cap on the logits that sits inside the model (Section 2). Under post-norm final_norm is the identity, since every block already ends in a norm. A second method, forward_explicit, computes the same function with a named hook on every step (resid_pre, attn_out, resid_mid, mlp_out, resid_post, each with the number of its block), where an analysis reads the stream or replaces it.

In gpt-pico a block's attention is four \(128 \times 128\) matrices, 65,536 weights, and its MLP two matrices with 131,072 between them: two thirds of the trunk is MLP, at every size.

The plus signs came first. In 2015 depth was failing for a reason that was not overfitting: with careful initialization and batch normalization, networks of tens of layers converged, and yet a plain network of 34 layers had a higher training error than one of 18. The answer made doing nothing the default.

In this paper, we address the degradation problem by introducing a deep residual learning framework. Instead of hoping each few stacked layers directly fit a desired underlying mapping, we explicitly let these layers fit a residual mapping. Formally, denoting the desired underlying mapping as \(\mathcal{H}(\mathbf{x})\), we let the stacked nonlinear layers fit another mapping of \(\mathcal{F}(\mathbf{x}):=\mathcal{H}(\mathbf{x})-\mathbf{x}\). The original mapping is recast into \(\mathcal{F}(\mathbf{x})+\mathbf{x}\). We hypothesize that it is easier to optimize the residual mapping than to optimize the original, unreferenced mapping. To the extreme, if an identity mapping were optimal, it would be easier to push the residual to zero than to fit an identity mapping by a stack of nonlinear layers.

—K. He, X. Zhang, S. Ren and J. Sun, "Deep residual learning for image recognition" (2015).CVPR 2016 (arXiv 1512.03385), Section 1. Table 2: top-1 errors on ImageNet of 27.94 and 28.54 for plain networks of 18 and 34 layers, of 27.88 and 25.03 with skips.

A year later the same authors asked what may stand on the path that skips, and the answer was: nothing.

Identity mappings in deep residual networksECCV 2016 · arXiv:1603.05027K. He, X. Zhang, S. Ren and J. Sun

If \(f\) is also an identity mapping: \(\mathbf{x}_{l+1}\equiv\mathbf{y}_{l}\), we can put Eqn.(2) into Eqn.(1) and obtain:

\[\mathbf{x}_{l+1}=\mathbf{x}_{l}+\mathcal{F}(\mathbf{x}_{l},\mathcal{W}_{l}). \qquad (3)\]

Recursively (\(\mathbf{x}_{l+2}=\mathbf{x}_{l+1}+\mathcal{F}(\mathbf{x}_{l+1},\mathcal{W}_{l+1})=\mathbf{x}_{l}+\mathcal{F}(\mathbf{x}_{l},\mathcal{W}_{l})+\mathcal{F}(\mathbf{x}_{l+1},\mathcal{W}_{l+1})\), etc.) we will have:

\[\mathbf{x}_{L}=\mathbf{x}_{l}+\sum_{i=l}^{L-1}\mathcal{F}(\mathbf{x}_{i},\mathcal{W}_{i}), \qquad (4)\]

for any deeper unit \(L\) and any shallower unit \(l\). Eqn.(4) exhibits some nice properties. (i) The feature \(\mathbf{x}_{L}\) of any deeper unit \(L\) can be represented as the feature \(\mathbf{x}_{l}\) of any shallower unit \(l\) plus a residual function in a form of \(\sum_{i=l}^{L-1}\mathcal{F}\), indicating that the model is in a residual fashion between any units \(L\) and \(l\). (ii) The feature \(\mathbf{x}_{L}=\mathbf{x}_{0}+\sum_{i=0}^{L-1}\mathcal{F}(\mathbf{x}_{i},\mathcal{W}_{i})\), of any deep unit \(L\), is the summation of the outputs of all preceding residual functions (plus \(\mathbf{x}_{0}\)). This is in contrast to a “plain network” where a feature \(\mathbf{x}_{L}\) is a series of matrix-vector products, say, \(\prod_{i=0}^{L-1}W_{i}\mathbf{x}_{0}\) (ignoring BN and ReLU).

Eqn.(4) also leads to nice backward propagation properties. Denoting the loss function as \(\mathcal{E}\), from the chain rule of backpropagation [9] we have:

\[\frac{\partial{\mathcal{E}}}{\partial{\mathbf{x}_{l}}}=\frac{\partial{\mathcal{E}}}{\partial{\mathbf{x}_{L}}}\frac{\partial{\mathbf{x}_{L}}}{\partial{\mathbf{x}_{l}}}=\frac{\partial{\mathcal{E}}}{\partial{\mathbf{x}_{L}}}\left(1+\frac{\partial}{\partial{\mathbf{x}_{l}}}\sum_{i=l}^{L-1}\mathcal{F}(\mathbf{x}_{i},\mathcal{W}_{i})\right). \qquad (5)\]

Eqn.(5) indicates that the gradient \(\frac{\partial{\mathcal{E}}}{\partial{\mathbf{x}_{l}}}\) can be decomposed into two additive terms: a term of \(\frac{\partial{\mathcal{E}}}{\partial{\mathbf{x}_{L}}}\) that propagates information directly without concerning any weight layers, and another term of \(\frac{\partial{\mathcal{E}}}{\partial{\mathbf{x}_{L}}}\left(\frac{\partial}{\partial{\mathbf{x}_{l}}}\sum_{i=l}^{L-1}\mathcal{F}\right)\) that propagates through the weight layers. The additive term of \(\frac{\partial{\mathcal{E}}}{\partial{\mathbf{x}_{L}}}\) ensures that information is directly propagated back to any shallower unit \(l\). Eqn.(5) also suggests that it is unlikely for the gradient \(\frac{\partial{\mathcal{E}}}{\partial{\mathbf{x}_{l}}}\) to be canceled out for a mini-batch, because in general the term \(\frac{\partial}{\partial{\mathbf{x}_{l}}}\sum_{i=l}^{L-1}\mathcal{F}\) cannot be always -1 for all samples in a mini-batch. This implies that the gradient of a layer does not vanish even when the weights are arbitrarily small.

Section 2, Equations 3 to 5
K. He, X. Zhang, S. Ren and J. Sun, "Identity mappings in deep residual networks" (ECCV 2016, arXiv 1603.05027), Section 2, Equations 3 to 5.

Proposition 5.1.2 (the stream is a sum, and its gradient keeps an identity). Number the \(2L\) sublayers of a model in order and write the stream after the \(i\)-th as \(x_i = x_{i-1} + F_i(x_{i-1})\), where \(F_i\) is the sublayer together with its normalization. Then for \(l < m\)

\[x_m = x_l + \sum_{i=l+1}^{m} F_i(x_{i-1}), \qquad \frac{\partial x_m}{\partial x_l} = (I + J_m) \cdots (I + J_{l+1}), \qquad J_i = \frac{\partial F_i}{\partial x_{i-1}} , \tag{5.2}\]

and the product expands into \(I\) plus \(2^{m-l} - 1\) further terms, one for each nonempty set of sublayers between \(l\) and \(m\). If the skip is scaled, \(x_i = \lambda_i x_{i-1} + F_i(x_{i-1})\), the leading \(I\) becomes \(\lambda_m \cdots \lambda_{l+1} I\).

Proof. Unroll the recursion for the first identity. For the second, one step has \(\partial x_i / \partial x_{i-1} = \lambda_i I + J_i\); multiply the steps. ∎

Each update is added to the stream, and no weight can shrink the leading \(I\) on the way down.Addition still permits erasure: a later update can subtract what an earlier one wrote. Likewise, the identity term is a direct gradient path, not a lower bound on the full gradient; the other terms can cancel it. Whatever is put on the skip (a scale, a gate, a normalization) puts a product over depth back. Highway networks, a few months before the residual network, had the skip with a learned gate on it; the gate was the dispensable part.R. K. Srivastava, K. Greff and J. Schmidhuber, "Highway networks", 2015 (arXiv 1505.00387).

In gpt-pico, with 12 sublayers, (5.2) from embedding to unembedding has \(2^{12} = 4{,}096\) terms, each a path through some of the sublayers; one path skips them all, and is a table that scores the next token from the current one. Section 9 is the study of which few paths carry a given behavior, and it starts from a reading of this sum by the researchers who set out to reverse engineer transformers:

One of the main features of the high level architecture of a transformer is that each layer adds its results into what we call the “residual stream.” The residual stream is simply the sum of the output of all the previous layers and the original embedding. We generally think of the residual stream as a communication channel, since it doesn't do any processing itself and all layers communicate through it.

—N. Elhage, N. Nanda, C. Olsson and others, "A mathematical framework for transformer circuits" (2021).Transformer Circuits Thread, 2021, the section on virtual weights and the residual stream as a communication channel.

Normalization

Each sublayer of (5.1) reads the stream through \(N\). The stream is a sum that keeps growing, and nothing else in the block fixes its scale; a normalization hands every reader an input of the same size whatever has accumulated. The device came from convolutional networks, where batch normalization rescaled each feature by its mean and variance over the mini-batch. For text the batch is the wrong place to take statistics: they depend on the batch size, differ between training and inference, and fluctuate during training. Layer normalization takes them over the coordinates of one vector: "we transpose batch normalization into layer normalization by computing the mean and variance used for normalization from all of the summed inputs to the neurons in a layer on a single training case."J. L. Ba, J. R. Kiros and G. E. Hinton, "Layer normalization", 2016 (arXiv 1607.06450), abstract. S. Ioffe and C. Szegedy, "Batch normalization: accelerating deep network training by reducing internal covariate shift", ICML 2015 (arXiv 1502.03167). The reading of it that matters here is that of S. De and S. L. Smith, "Batch normalization biases residual blocks towards the identity function in deep networks", NeurIPS 2020 (arXiv 2002.10444): in a residual network it "downscales the residual branch relative to the skip connection, by a normalizing factor on the order of the square root of the network depth".

Definition 5.2.1 (layer norm, RMS norm). For \(x \in \mathbb R^d\) let \(\mu = \tfrac1d \sum_i x_i\), \(\sigma^2 = \tfrac1d \sum_i (x_i - \mu)^2\) and \(\mathrm{rms}(x)^2 = \tfrac1d \sum_i x_i^2\), the root mean square. With a gain \(\gamma \in \mathbb R^d\) and a bias \(\beta \in \mathbb R^d\),

\[\mathrm{LN}(x) = \gamma \odot \frac{x - \mu \mathbf 1}{\sigma} + \beta, \qquad \mathrm{RMSNorm}(x) = \gamma \odot \frac{x}{\mathrm{rms}(x)} . \tag{5.3}\]

In the code. Each norm is one call to a function of PyTorch, wrapped in a module that holds its parameters, and one function builds the norm that a recipe names. They are LayerNorm, RMSNorm and make_norm in gpt-pp's model.py.

class LayerNorm(nn.Module):
    ...
    def __init__(self, d: int, bias: bool):
        super().__init__()
        self.gain = nn.Parameter(torch.ones(d))
        self.bias = nn.Parameter(torch.zeros(d)) if bias else None

    def forward(self, x: Tensor) -> Tensor:
        return F.layer_norm(x, (x.size(-1),), self.gain, self.bias, 1e-5)


class RMSNorm(nn.Module):
    ...
    def __init__(self, d: int, gain: bool):
        super().__init__()
        self.gain = nn.Parameter(torch.ones(d)) if gain else None

    def forward(self, x: Tensor) -> Tensor:
        return F.rms_norm(x, (x.size(-1),), self.gain, eps=1e-6)


...


def make_norm(cfg: ModelConfig) -> nn.Module:
    """The normalization layer chosen by `cfg.norm`; `none` is the identity."""
    if cfg.norm == "layernorm":
        return LayerNorm(cfg.d_model, cfg.bias)
    if cfg.norm == "rmsnorm":
        return RMSNorm(cfg.d_model, gain=True)
    if cfg.norm == "rmsnorm_fixed":
        return RMSNorm(cfg.d_model, gain=False)
    if cfg.norm == "dyt":
        return DynamicTanh(cfg.d_model, cfg.dyt_alpha, cfg.bias)
    return nn.Identity()

gain is \(\gamma\) and bias is \(\beta\) of (5.3), and the tuple (x.size(-1),) says that the statistics run over the last axis alone, the \(d\) coordinates of one token's vector. Both functions add a constant under the square root that (5.3) leaves out, \(10^{-5}\) in layer norm and \(10^{-6}\) in RMS norm. make_norm builds a new module at each call, two a block, one before the unembedding and, in the 2026 recipe, one after the embedding, so no gain is shared. The norm of that recipe is rmsnorm_fixed, an RMSNorm whose gain is None: it has no parameter, and the group that the optimizer keeps for gains and biases is empty (Section 6). dyt is the dynamic tanh at the end of this section, and none is the identity.

Quirk. These are gpt-pp's own modules, not nn.LayerNorm and nn.RMSNorm. The parameter that PyTorch and nanoGPT name weight is named gain; LayerNorm always has it, and has a bias only in a recipe with biases; and RMSNorm passes \(10^{-6}\) where the default of nn.RMSNorm is the machine epsilon. With these names, and with attention's four separate matrices where GPT-2 fuses queries, keys and values into one, the released weights of GPT-2 do not load into the 2019 recipe as they are, although with GPT-2's vocabulary that recipe has the 124,439,808 parameters of GPT-2 small; loaded by hand, they would also meet the exact form of GPT-2's activation where they were trained with its tanh approximation (Sections 5.3 and 5.6). The release notes of PyTorch 2.10 to 2.15, which this series did not run, report no change to the signature or the defaults of F.layer_norm and F.rms_norm.

Proposition 5.2.2 (what each norm does to a vector). Take \(\gamma = \mathbf 1\), \(\beta = 0\), and let \(P = I - \mathbf 1 \mathbf 1^{\top}/d\) be the projection that removes the mean. Then

\[\mathrm{LN}(x) = \sqrt d\, \frac{Px}{\|Px\|}, \qquad \mathrm{RMSNorm}(x) = \sqrt d\, \frac{x}{\|x\|} .\]

RMS norm sends \(x\) to the sphere of radius \(\sqrt d\). Layer norm first drops \(x\) onto the hyperplane orthogonal to \(\mathbf 1\), then sends it to the sphere of radius \(\sqrt d\) inside that hyperplane. The two agree when the coordinates of \(x\) sum to zero.

Proof. \(x - \mu \mathbf 1 = Px\), \(\sigma = \|Px\|/\sqrt d\) and \(\mathrm{rms}(x) = \|x\|/\sqrt d\). ∎

In three dimensions layer norm maps to a circle and RMS norm to a sphere. Adding the same constant to every coordinate leaves the image of layer norm unchanged; RMS norm keeps that mean, so its output moves toward the direction \(\mathbf 1\). The coordinatewise tanh considered below has a different image again: a cube, with no fixed output length.

a vector of three numbers before and after each normalization
(the live figure draws a cloud of 150 of them in three dimensions)

x              (3.21, 0.40, 1.71)       mean 1.77, length 3.66
layer norm     (1.25, −1.20, −0.05)     x − mean, then ÷ its standard deviation
RMS norm       (1.52, 0.19, 0.81)       x ÷ its root mean square
dynamic tanh   (0.92, 0.20, 0.69)       tanh(x / 2), coordinate by coordinate

               image of the cloud       lost                     left, of 768
layer norm     a circle, radius √3      the mean, the length     766
RMS norm       a sphere, radius √3      the length               767
dynamic tanh   inside the cube ±1       nothing, exactly         768

the cloud: lengths 0.48 to 4.74 before; 1.73 for all after either norm (27.7 at 768 wide)
(the live figure needs JavaScript; it loads about 0.7 MB of three.js for the box and no data)
What a normalization does to vectors of three numbers, drawn with the direction (1, 1, 1) upward. Grey: 150 vectors before. Green: the same vectors after. Layer norm drops each onto the plane where its coordinates sum to zero, then slides it to a circle; RMS norm slides it along its own ray to a sphere; dynamic tanh squashes each coordinate into a cube. Hairlines: the way each vector travels. Switch the norm, scrub the travel, stretch or shift the cloud, click a vector to follow it. Exact; the cloud is one random draw.

Proposition 5.2.3 (invariance, and what it does to gradients). Let \(N\) be either map of Proposition 5.2.2 and \(y = N(x)\).

(a) \(N(ax) = N(x)\) for \(a > 0\), and \(\mathrm{LN}(x + b \mathbf 1) = \mathrm{LN}(x)\).

(b) The Jacobians are

\[J_{\mathrm{RMS}}(x) = \frac{\sqrt d}{\|x\|}\Bigl(I - \frac{y y^{\top}}{d}\Bigr), \qquad J_{\mathrm{LN}}(x) = \frac{\sqrt d}{\|Px\|}\Bigl(P - \frac{y y^{\top}}{d}\Bigr), \tag{5.4}\]

each a multiple of an orthogonal projection, of rank \(d - 1\) and \(d - 2\), that sends \(x\) to zero.

(c) If a function sees \(x\) only through \(N\), \(f(x) = g(N(x))\), then \(x^{\top} \nabla f(x) = 0\) and \(\nabla f(ax) = \nabla f(x)/a\).

Proof. (a) is read off Proposition 5.2.2. (b): the derivative of \(z \mapsto z/\|z\|\) is \((I - \hat z \hat z^{\top})/\|z\|\) with \(\hat z = z/\|z\|\); take \(z = x\), or \(z = Px\) and compose with \(P\), and note that \(\hat z = y/\sqrt d\). (c): \(\nabla f = J^{\top} \nabla g\) with \(J\) symmetric, \(Jx = 0\) and \(J(ax) = J(x)/a\). ∎

Part (c) is the lever of what follows. A stream that has grown by a factor \(a\) passes back gradients smaller by the factor \(a\): a normalization is an automatic control of step size for whatever stands in front of it.

Where it sits. The transformer of 2017 normalized after the addition, as its paper says where it describes the encoder:

Encoder: The encoder is composed of a stack of \(N=6\) identical layers. Each layer has two sub-layers. The first is a multi-head self-attention mechanism, and the second is a simple, position-wise fully connected feed-forward network. We employ a residual connection [11] around each of the two sub-layers, followed by layer normalization [1]. That is, the output of each sub-layer is \(\mathrm{LayerNorm}(x+\mathrm{Sublayer}(x))\), where \(\mathrm{Sublayer}(x)\) is the function implemented by the sub-layer itself. To facilitate these residual connections, all sub-layers in the model, as well as the embedding layers, produce outputs of dimension \(d_{\text{model}}=512\).

—A. Vaswani and others, "Attention is all you need" (2017).NIPS 2017 (arXiv 1706.03762), Section 3.1, the paragraph on the encoder, where \(N\) counts layers; references [11] and [1] are the papers on residual networks and on layer normalization. The warmup is in its Section 5.3.

That puts the normalization \(N\) on the skip, which Proposition 5.1.2 warns against, and the design needed warmup: a learning rate raised gradually over the first 4,000 steps. By 2019 several groups had moved the normalization inside the branch, GPT-2 among them, with one more normalization before the unembedding.

Definition 5.2.4 (post-norm, pre-norm). With sublayers \(F_i\) and stream \(x_i\): post-norm is \(x_i = N\bigl(x_{i-1} + F_i(x_{i-1})\bigr)\); pre-norm is \(x_i = x_{i-1} + F_i\bigl(N(x_{i-1})\bigr)\) with the output read from \(N(x_{2L})\), which is (5.1).

In Block.forward (Section 5.1) pre-norm is the first branch and post-norm the second, and the last \(N\) of pre-norm is the model's final_norm, which post-norm does without.

Proposition 5.2.5 (the gradient argument). Let \(N\) be RMS norm with unit gain.

(a) Suppose that at initialization each sublayer writes a vector with \(\mathbb E\|F_i\|^2 = c\,d\) that is uncorrelated with the stream it is added to, and that \(\|x_0\|^2 = d\). Then in pre-norm \(\mathbb E\|x_m\|^2 = (1 + cm)\,d\), while in post-norm \(\|x_m\|^2 = d\) for every \(m \ge 1\) and the sum that is normalized has \(\mathbb E\|u_m\|^2 = (1 + c)\,d\).

(b) If the loss sees a vector \(u\) only through \(N\), \(E = g(N(u))\), then \(\|\partial E/\partial u\| \le \sqrt d\, \|\nabla g\| / \|u\|\).

(c) Hence, at the scales of (a), the gradient that reaches the last block is of order \(\|\nabla g\|/\sqrt{1 + c}\) in post-norm whatever the depth, and of order \(\|\nabla g\|/\sqrt{1 + 2cL}\) in pre-norm: with \(c\) fixed it falls like \(1/\sqrt L\).

Proof. (a): when the cross term vanishes, \(\mathbb E\|x_{m-1} + F_m\|^2 = \mathbb E\|x_{m-1}\|^2 + c\,d\); in pre-norm induct on \(m\), in post-norm \(N\) returns each sum to the sphere of radius \(\sqrt d\). (b): \(\partial E/\partial u = \nabla g^{\top} J_N(u)\), and by (5.4) \(J_N(u)\) is \(\sqrt d/\|u\|\) times a projection. (c): apply (b) to the last vector that is normalized before the unembedding. ∎

The assumption carries the weight: a fixed \(c\) means that a block writes at the scale of the stream whatever the depth, which is what the weights of 2017 did, drawn by the rule of Glorot and Bengio. And (b) is not special to the last reader. Every reader divides by the size of what it reads, so a block should receive a gradient that goes as one over the size of the stream where it sits: about the same in every block of a post-norm stack, whose stream is reset after every sublayer, and falling toward the output in a pre-norm stack, whose stream grows on the way up. The paper that proved (c), with the concentration made precise, drew the consequence for training:

The Transformer is widely used in natural language processing tasks. To train a Transformer however, one usually needs a carefully designed learning rate warm-up stage, which is shown to be crucial to the final performance but will slow down the optimization and bring more hyper-parameter tunings. In this paper, we first study theoretically why the learning rate warm-up stage is essential and show that the location of layer normalization matters. Specifically, we prove with mean field theory that at initialization, for the original-designed Post-LN Transformer, which places the layer normalization between the residual blocks, the expected gradients of the parameters near the output layer are large. Therefore, using a large learning rate on those gradients makes the training unstable. The warm-up stage is practically helpful for avoiding this problem. On the other hand, our theory also shows that if the layer normalization is put inside the residual blocks (recently proposed as Pre-LN Transformer), the gradients are well-behaved at initialization. This motivates us to remove the warm-up stage for the training of Pre-LN Transformers. We show in our experiments that Pre-LN Transformers without the warm-up stage can reach comparable results with baselines while requiring significantly less training time and hyper-parameter tuning on a wide range of applications.

—R. Xiong and others, "On layer normalization in the transformer architecture" (2020).ICML 2020 (arXiv 2002.04745), abstract. Its Lemma 2 is part (a) and its Theorem 1 part (c): with high probability the gradient of the last feed-forward matrix is at most \(O(d\sqrt{\ln d})\) in post-norm and \(O(d\sqrt{\ln d/L})\) in pre-norm. The rule of 2017: X. Glorot and Y. Bengio, "Understanding the difficulty of training deep feedforward neural networks", AISTATS 2010.

We test both halves of the argument on our own models: at initialization, the gradient that each block receives and the size of the stream after it, for stacks of 12, 24 and 48 blocks; and, trained, the model on which this series measures each design choice (8 layers, 512 wide) in the 2019 recipe, the design and training of GPT-2, with the norm in either place, with a warmup and without.The 2020 result does not make warmup dispensable for every pre-norm recipe. Kalra and Barkeshli (2024) find that warmup chiefly helps a network tolerate a larger target learning rate, with its behavior depending on initialization and parameterization. Normalization placement alone therefore does not settle the schedule.

gradient reaching each block at initialization: 48 blocks, 256 wide,
one batch of real text, mean of 8 random initializations

block               1       2      12      24      36      48
initialization of 2019 (GPT-2: 0.02, writes into the stream ÷ √(2L))
  post-norm      0.94   0.079   0.079   0.079   0.079   0.079    stream 1.00 to 1.00
  pre-norm       3.59    3.36    2.24    1.72    1.43    1.25    stream 0.030 to 0.088
initialization of 2017 (Glorot: a block writes at the scale of the stream)
  post-norm      1.99    1.97    1.75    1.65    1.63    1.53    stream 1.00 to 1.00
  pre-norm       1.31    1.13    0.61    0.39    0.31    0.27    stream 1.14 to 6.62
  pre-norm, last block of 12 | 24 | 48 blocks: 0.58 | 0.42 | 0.27

the 2019 recipe (8 blocks, 512 wide) in steps of 8,192 tokens:
training loss at 10M tokens, held-out loss at the end
  pre-norm, warmup            5.86   4.581
  pre-norm, no warmup         6.01   4.950
  post-norm, warmup           5.79   4.537
  post-norm, no warmup        7.69   7.696
in steps of 65,536 tokens, held-out loss at the end: pre-norm 4.613, 5.706;
post-norm 7.690, 7.690 (token frequencies alone give 7.701)
post-norm there with a warmup of 32M tokens: 7.691 at rate 0.0024, 5.255 at rate 0.0006
(the live figure needs JavaScript and loads about 37 kB of data)
Where the norm sits. Green: pre-norm, the norm before each sublayer; ink: post-norm, the norm after each addition. Left and middle, at initialization, for stacks of 12, 24 and 48 blocks: the gradient each block receives from one batch of text, and the size of the stream after it, on log scales. Right: the training loss of four runs of the 2019 recipe in steps of 8,192 tokens, with a warmup (solid) and without (dashed). Change the initialization and the depth. Measured on our model: 256 wide at initialization, a few random seeds; the runs 512 wide, one seed each.

At initialization the law of the reader holds almost to the letter. Under the initialization of GPT-2 a post-norm stack hands every block the same gradient, at 12 blocks or at 48: for the output matrix of the MLP, the last block receives 1.00, 1.00 and 1.00 of what the first does. The one exception is the attention of the first block, which reads the raw embedding before any norm has touched it and gives that block twelve times the gradient of the others. A pre-norm stack hands its last block less than its first: for the same matrix 0.44, 0.40 and 0.37 of it at 12, 24 and 48 blocks, and for a whole block the product of gradient and stream size is the same at the two ends of the stack to within two percent.

The fall with depth that (c) predicts is there in the setting of the proposition and nowhere else. With the weights of 2017 the gradient of the last block of a pre-norm stack shrinks by a factor of 0.47 from 12 blocks to 48, against a predicted \(1/\sqrt 4 = 0.50\), and that of a post-norm stack by 0.94. Under the initialization of GPT-2 the two factors are 1.01 and 1.00: its depth factor (Section 5.5) makes \(c\) shrink as the stack deepens, the stream ends at the same size in every stack, and the \(1/\sqrt L\) is gone.

Trained, the two placements follow the usual account, in which a warmup rescues post-norm and pre-norm can do without one, only at the smaller of the two step sizes tried, where the warmup has more steps and the rate is lower. These runs use the 2019 recipe at steps of 8,192 tokens and a rate of 0.0006, where a warmup of 8M tokens is 977 steps (50M tokens, one seed each). There post-norm reaches 4.537 nats a token with the warmup and collapses without it to the frequencies of the tokens (7.696, where token frequencies alone give 7.70); pre-norm reaches 4.581 with it and trains without, to 4.950: it survives, 0.37 behind, which is less than the "comparable results" of the abstract's last sentence.

Known issue. At steps of 65,536 tokens, the step of every run of the ladder (Section 5.6), and a rate of 0.0024, on the same budget, the same warmup is 122 steps, and post-norm collapses with the warmup or without: its loss is at the frequencies of the tokens within 40 steps and stays there, to 7.690 at the end both times. Neither a rate four or ten times lower (in runs of 10M tokens) nor a warmup of 32M tokens, 488 steps, rescues it alone (7.691 after 50M tokens); that warmup and a rate of 0.0006 together do, to 5.255. So the two runs at 7.690 do not show that post-norm needs a warmup, and they do not show that it cannot train. Pre-norm trains at that step either way, to 4.613 with the warmup and 5.706 without. In the 2026 recipe, that of gpt-pp, post-norm trains at the large step too: 4.133, against 4.026 for the same layer norm before the sublayers. That run changes two switches of the recipe, the placement and the kind of norm, which is why it is set beside layer norm and not beside the recipe's own norm.

The stream gets louder. After training, a pre-norm stream grows faster than the square-root law of Proposition 5.2.5: by about 4.5% a layer in GPT-2 XL between layers 5 and 41, close to a factor of five. In gpt-pico, on the sentence about Lily and Tom, the last token's stream is 11.3 long after the embedding and 3,594 after the sixth block. Proposition 5.2.3(c) is the mechanism: every reader is blind to scale, so a sublayer that wants its write to count must write at the scale of what is already there. One study of 2025 finds that deep pre-norm layers then contribute little, calls it a curse of depth, and divides the output of the normalization in layer \(\ell\) by \(\sqrt\ell\).S. Heimersheim and A. Turner, "Residual stream norms grow exponentially over the forward pass", LessWrong, 2023. W. Sun and others, "The curse of depth in large language models", NeurIPS 2025 (arXiv 2502.05795).

Less of it. Is the centering needed? A paper of 2019 supposed that it is not, in the third sentence of its abstract:

Layer normalization (LayerNorm) has been successfully applied to various deep neural networks to help stabilize training and boost model convergence because of its capability in handling re-centering and re-scaling of both inputs and weight matrix. However, the computational overhead introduced by LayerNorm makes these improvements expensive and significantly slows the underlying network, e.g. RNN in particular. In this paper, we hypothesize that re-centering invariance in LayerNorm is dispensable and propose root mean square layer normalization, or RMSNorm. RMSNorm regularizes the summed inputs to a neuron in one layer according to root mean square (RMS), giving the model re-scaling invariance property and implicit learning rate adaptation ability. RMSNorm is computationally simpler and thus more efficient than LayerNorm. We also present partial RMSNorm, or \(p\)RMSNorm where the RMS is estimated from \(p\)% of the summed inputs without breaking the above properties. Extensive experiments on several tasks using diverse network architectures show that RMSNorm achieves comparable performance against LayerNorm but reduces the running time by 7%\(\sim\)64% on different models. Source code is available at https://github.com/bzhangGo/rmsnorm.

—B. Zhang and R. Sennrich, "Root mean square layer normalization" (2019).NeurIPS 2019 (arXiv 1910.07467), abstract.

The hypothesis held. On a GPT-2 of 124M parameters trained on OpenWebText, a comparison of 2026 reports validation losses of 2.94 with layer norm and 2.95 with RMS norm.M. Chen and others, "Stronger normalization-free transformers", CVPR 2026 (arXiv 2512.10938), Tables 12 and 26. On our 8-layer model, with nothing else changed in the 2026 recipe (50M tokens, two seeds each), layer norm with its gain ends at 4.026 and 4.023 nats and RMS norm with a gain at 4.025 and 4.023: with or without the centering, the two cannot be told apart.

The small fast recipes of 2024 then dropped the gain: it is absorbed by the matrix that reads the normalized vector, \(W(\gamma \odot y) = (W \operatorname{diag}\gamma)\, y\), and without it the layer has no parameters at all. Absorbed as a function, that is; training is another matter.Removing a parameter can leave the set of possible functions unchanged while changing the route gradient descent takes through that set. Expressive equivalence is not a promise of equal training loss. The norm of the 2026 recipe, without the gain, ends at 4.045 and 4.043, behind them by 0.019 to 0.021 in the two seeds.

Is 0.02 a difference? There are two yardsticks, one for each way of training. Fourteen runs in the 2026 recipe were made twice, with seeds that differ in the initial weights and the order of the data: the two end 0.001 to 0.008 nats apart, 0.005 in root mean square, so between two such runs a difference of 0.009 or more is established. The ladder, the 8-layer model retrained once for each change between the two recipes (Section 5.6), has two seeds a rung: they end 0.001 to 0.006 apart on eight of the ten rungs and 0.015 and 0.018 on two of the seven trained the old way, 0.008 in root mean square, so there, and wherever the old training recipe is used, a difference under 0.016 is not established. By the first yardstick the gain's 0.02 is a difference, in both seeds. The ladder agrees: under the old training recipe, the step from layer norm with its gain to the norm without parameters cost 0.026 and 0.037 in its two seeds. At this size the recipe is the worse for dropping the gain.

What the norm without parameters does have is a symmetry that layer norm lacks: it commutes with every rotation of the stream, \(\mathrm{RMSNorm}(Rx) = R\,\mathrm{RMSNorm}(x)\), where by Proposition 5.2.2 layer norm commutes only with the rotations that fix \(\mathbf 1\) up to sign. Rotate the stream, and with it every matrix that reads or writes it, and the function is unchanged: the statement that the residual stream has no privileged basis is exact for the block of 2026. The hidden layer of an MLP does have one, since its nonlinearity acts coordinate by coordinate, which is why Section 5.4 can speak of single neurons there and nowhere else.

The block of 2026 also normalizes the embedding before the first block, because an untied embedding trained at its own high learning rate does not keep its scale (in a probe of 763 steps the root mean square of its rows went from 1 to between 2.5 and 5.1). The same growth makes fixed sinusoids fade when they are added before this norm, which is where the code adds them unless a switch says otherwise (Section 4). Without that norm the 8-layer model is 0.014 and 0.008 worse in its two seeds: small, just past the yardstick in the mean, and there both times.

None. The last step asks whether a statistic is needed at all. Dynamic tanh (DyT) replaces the layer by \(\gamma \odot \tanh(\alpha x) + \beta\) with one learned scalar \(\alpha\); it "is inspired by the observation that layer normalization in Transformers often produces tanh-like, \(S\)-shaped input-output mappings".J. Zhu, X. Chen, K. He, Y. LeCun and Z. Liu, "Transformers without normalization", CVPR 2025 (arXiv 2503.10622), abstract, Table 4 and Section 9: "we find that DyT offers no speedup over models with normalization layers when properly compiled/optimized". Proposition 5.2.6 is from F. Stollenwerk, "On the mathematical relationship between layer normalization and dynamic activation functions", EACL 2026 (arXiv 2503.21708). The observation has an exact form.

Proposition 5.2.6 (RMS norm, one coordinate at a time). Write \(\rho_i^2 = \sum_{j \ne i} x_j^2\). Then \(\mathrm{RMSNorm}(x)_i = \sqrt d\, x_i / \sqrt{\rho_i^2 + x_i^2}\): with the other coordinates held fixed, an odd, increasing S-curve in \(x_i\), of slope \(\sqrt d/\rho_i\) at zero, that saturates at \(\pm \sqrt d\).

The S-shape needs no learning to explain it: a coordinate that is large dominates its own norm and is squashed, a small one is scaled linearly, and DyT's \(\alpha\) stands in for \(1/\rho_i\). What DyT gives up is the coupling between coordinates: its image is a cube and not a sphere, so the size of its output is no longer fixed. On LLaMA models from 7B to 70B parameters it matches RMS norm to within 0.01 in loss; in the GPT-2 comparison above it reaches 2.97 against 2.94. On our 8-layer model, at the best of three starting slopes, it ends at 4.077, 0.031 behind the norm of the recipe. And with no normalization at all the run does not finish: its loss is 5.98 at step 70 and no longer finite at step 80, although its writing matrices start at zero (Section 5.5).

In the code. Dynamic tanh is the formula above and little else. It is DynamicTanh in gpt-pp's model.py.

class DynamicTanh(nn.Module):
    ...
    def __init__(self, d: int, alpha: float, bias: bool):
        super().__init__()
        self.alpha = nn.Parameter(torch.tensor([alpha]))
        self.gain = nn.Parameter(torch.ones(d))
        self.bias = nn.Parameter(torch.zeros(d)) if bias else None

    def forward(self, x: Tensor) -> Tensor:
        x = self.gain * torch.tanh(self.alpha * x)
        return x if self.bias is None else x + self.bias

alpha is \(\alpha\), one number a layer, and starts at the configuration's dyt_alpha, 0.5 unless set; gain is \(\gamma\) and bias is \(\beta\). The gain is always there, where the norm of the recipe has none, and the bias exists only in a recipe with biases: in the 2026 recipe, which has none, the layer has a slope and a gain and no \(\beta\).

A deep network trains when a path of controlled scale runs from input to output and every block is a small correction to it. The history of the block is the history of where that control is kept: in the statistics of a batch (2015), in each token's own statistics (2016), in the initial weights (2019), in a pointwise function (2025).

The MLP and its activation

The paper of 2017 describes the MLP in two sentences: a network "applied to each position separately and identically. This consists of two linear transformations with a ReLU activation in between."Vaswani and others (2017), Section 3.3; the perplexities are rows (C) of Table 3. ReLU is the rectified linear unit, \(\max(0, z)\).

Definition 5.3.1 (the feed-forward layer). With an activation \(\phi\) applied coordinate by coordinate and a hidden width \(d_m\),

\[\mathrm{MLP}(x) = W_{\mathrm{out}}\, \phi(W_{\mathrm{in}} x), \qquad W_{\mathrm{in}} \in \mathbb R^{d_m \times d}, \quad W_{\mathrm{out}} \in \mathbb R^{d \times d_m}. \tag{5.5}\]

The designs of 2017 to 2019 add a bias to each map.

The hidden width there was 2,048 for a stream of 512. The ratio of four became the default and is derived nowhere: the paper's own ablation shows only that wider is better, with perplexities of 5.12, 4.92 and 4.75 at hidden widths of 1,024, 2,048 and 4,096. What changed, four times, is \(\phi\). We compare six activations in one and the same model below.

the activations of the MLP, in the order they entered transformers

        function      since                          loss    zero  <1 % rms    dead  kurtosis
ReLU    max(0, x)     2017 · the Transformer        4.063    86 %      86 %   0.0 %      49.2
GELU    x Φ(x)        2018 · GPT and BERT           4.046   1.0 %      21 %   0.0 %      42.9
SiLU    x σ(x)        2020 · in T5, as Swish        4.052   0.0 %     0.7 %   0.0 %      44.2
GEGLU   GELU(g) · v   2020 · Shazeer; Gemma         4.043   0.0 %      14 %   0.0 %       306
SwiGLU  SiLU(g) · v   2020 · Shazeer; PaLM, Llama   4.037   0.0 %     5.0 %   0.0 %       289
ReLU²   max(0, x)²    2021 · Primer; the speedrun   4.045    83 %      85 %   0.0 %     1,306

loss: validation loss of the 2026 recipe (8 layers, 512 wide) with only the
activation changed, 50M tokens; zero, <1 % of rms: shares of the hidden units' activations
on 102,400 held-out tokens; dead: units that never fire on them
GELU dips to −0.170 at x = −0.75, SiLU to −0.278 at −1.28; a gated unit is its curve times a second read v
(the live figure needs JavaScript and loads about 11 kB of data)
The activations of the MLP. Green: the selected activation (solid) and its slope (dashed), against its input; the others are hairlines. A gated unit multiplies its curve by a second linear read v of the input, drawn at v = 1, 2 and −1. The wash is measured: where the inputs to the nonlinearity fall in the trained model. Table: for the 2026 recipe trained with each activation and nothing else changed, the validation loss, and what its hidden units do on held-out text. Select a row. Curves exact; units measured in one run, losses in one or two seeds.

The rectifier. The rectified linear unit won on speed: a convolutional network reached a fixed training error six times faster with it than with tanh. Its price is the unit that is off on every input, which "a gradient-based optimization algorithm will not adjust".A. Krizhevsky, I. Sutskever and G. E. Hinton, "ImageNet classification with deep convolutional neural networks", NeurIPS 2012, Figure 1. A. L. Maas, A. Y. Hannun and A. Y. Ng, "Rectifier nonlinearities improve neural network acoustic models", 2013, Section 2. In trained language models: E. Voita, J. Ferrando and C. Nalmpantis, "Neurons in large language models: dead, n-gram, positional", 2023 (arXiv 2309.04827), find more than 70% of the neurons of some early layers never active.

A smooth gate. The Gaussian error linear unit (GELU) reads the rectifier as the input times a gate, \(z \cdot \mathbf 1[z > 0]\), and softens the gate: it "weights inputs by their value, rather than gates inputs by their sign as in ReLUs".

Proposition 5.3.2 (GELU is the mean of a random gate). Let \(\Phi\) be the distribution function of the standard normal law. If \(m \sim \mathrm{Bernoulli}(\Phi(z))\), then \(\mathbb E[mz] = z\,\Phi(z) = \mathrm{GELU}(z)\). It is smooth, and negative for \(z < 0\), with minimum \(-0.170\) at \(z = -0.752\).

Proof. \(\mathbb E[m] = \Phi(z)\). ∎

GPT used it in 2018, BERT "following OpenAI GPT", and GPT-2 ships a tanh approximation of it that is off by at most \(4.7 \times 10^{-4}\), for a reason Section 5.6 comes to. A year after GELU an automated search found the same shape with a logistic gate, \(z\,\sigma(\beta z)\), called Swish, or the sigmoid linear unit (SiLU) at \(\beta = 1\), and its authors drew the lesson:

Like ReLU, Swish is unbounded above and bounded below. Unlike ReLU, Swish is smooth and non-monotonic. In fact, the non-monotonicity property of Swish distinguishes itself from most common activation functions. The derivative of Swish is

\[\begin{aligned}f^{\prime}(x)&=\sigma(\beta x)+\beta x\cdot\sigma(\beta x)(1-\sigma(\beta x))\\ &=\sigma(\beta x)+\beta x\cdot\sigma(\beta x)-\beta x\cdot\sigma(\beta x)^{2}\\ &=\beta x\cdot\sigma(x)+\sigma(\beta x)(1-\beta x\cdot\sigma(\beta x))\\ &=\beta f(x)+\sigma(\beta x)(1-\beta f(x))\end{aligned}\]

The first derivative of Swish is shown in Figure 5 for different values of \(\beta\). The scale of \(\beta\) controls how fast the first derivative asymptotes to \(0\) and \(1\). When \(\beta=1\), the derivative has magnitude less than \(1\) for inputs that are less than around \(1.25\). Thus, the success of Swish with \(\beta=1\) implies that the gradient preserving property of ReLU (i.e., having a derivative of 1 when \(x>0\)) may no longer be a distinct advantage in modern architectures.

—P. Ramachandran, B. Zoph and Q. V. Le, "Searching for activation functions" (2017).arXiv 1710.05941, Section 4. D. Hendrycks and K. Gimpel, "Gaussian error linear units (GELUs)", 2016 (arXiv 1606.08415). J. Devlin and others, "BERT: pre-training of deep bidirectional transformers for language understanding", NAACL 2019 (arXiv 1810.04805), Appendix A.2.

The skip and the normalization now carry the signal (Proposition 5.1.2, Proposition 5.2.3), and the activation is free to be chosen for the function it computes.

A gate with its own weights. A gated linear unit multiplies one linear projection of its input by a squashed second projection. In 2020 it was put inside the transformer, \(\bigl(\mathrm{Swish}(xW) \otimes xV\bigr) W_2\) in place of \(\phi(xW_1) W_2\), a variant named SwiGLU, with the hidden width cut to two thirds so that three matrices cost what two did. On a T5 model at equal parameters the gated variants came out a few hundredths of a nat ahead (1.636 for SwiGLU against 1.677 for ReLU), and the paper ends:

We have extended the GLU family of layers and proposed their use in Transformer. In a transfer-learning setup, the new variants seem to produce better perplexities for the de-noising objective used in pre-training, as well as better results on many downstream language-understanding tasks. These architectures are simple to implement, and have no apparent computational drawbacks. We offer no explanation as to why these architectures seem to work; we attribute their success, as all else, to divine benevolence.

—N. Shazeer, "GLU variants improve transformer" (2020).arXiv 2002.05202, Section 4 and Table 1. The gated unit: Y. N. Dauphin, A. Fan, M. Auli and D. Grangier, "Language modeling with gated convolutional networks", ICML 2017 (arXiv 1612.08083). A. Chowdhery and others, "PaLM: scaling language modeling with pathways", 2022 (arXiv 2204.02311).

PaLM and LLaMA adopted it, the second writing the accounting into its architecture:

SwiGLU activation function [PaLM]. We replace the ReLU non-linearity by the SwiGLU activation function, introduced by Shazeer (2020) to improve the performance. We use a dimension of \(\frac{2}{3}4d\) instead of \(4d\) as in PaLM.

—H. Touvron and others, "LLaMA: open and efficient foundation language models" (2023).arXiv 2302.13971, Section 2.2, the paragraph on the activation.

The square. In parallel a search over transformer components reported that "The most effective modification is the improvement from a ReLU activation to a squared ReLU activation in the Transformer’s feed forward block", and said what the square is: it "is equivalent when ReGLU’s \(U\) and \(V\) weight matrices are the same and squared ReLU is immediately preceded by a linear transformation with weight matrix \(U\)".D. R. So, W. Mańke, H. Liu, Z. Dai, N. Shazeer and Q. V. Le, "Primer: searching for efficient transformers for language modeling", NeurIPS 2021 (arXiv 2109.08668), Section 3. ReGLU is the gated unit with a ReLU gate.

Proposition 5.3.3 (the price of a gate, and the gate that is free). (a) A gated layer with three matrices of hidden width \(h\) has \(3dh\) weights, as many as (5.5) when \(h = \tfrac23 d_m\); for \(d_m = 4d\) that is \(h = \tfrac83 d\), or 2,048 at \(d = 768\). (b) A gated unit with a ReLU gate and tied projections is a squared ReLU: \(\max(0, w \cdot x)\,(w \cdot x) = \max(0, w \cdot x)^2\).

So the block of 2026 has two dialects, SwiGLU and the squared ReLU, and they are one idea: each hidden unit is a product of two linear forms of the input, with separate weights at two thirds of the width or with tied weights at the full width. The square has a condition: homogeneous of degree two, a bare stack of such units explodes or dies doubly exponentially, and it is the normalization in front of each that keeps the recursion from starting.

In the code. The plain layer of (5.5) and the gated one are one class with two return lines, and the square is a ReLU followed by a square. They are relu2, ACTIVATIONS and MLP in gpt-pp's model.py.

def relu2(x: Tensor) -> Tensor:
    ...
    return F.relu(x).square()


ACTIVATIONS: dict[str, Callable[[Tensor], Tensor]] = {
    "relu": relu, "gelu": gelu, "silu": silu, "relu2": relu2, "swiglu": silu, "geglu": gelu,
}


class MLP(nn.Module):
    ...
    def __init__(self, cfg: ModelConfig):
        super().__init__()
        self.fc = nn.Linear(cfg.d_model, cfg.d_mlp, bias=cfg.bias)
        self.gate = nn.Linear(cfg.d_model, cfg.d_mlp, bias=cfg.bias) if cfg.act in GATED else None
        self.proj = nn.Linear(cfg.d_mlp, cfg.d_model, bias=cfg.bias)
        self.act = ACTIVATIONS[cfg.act]

    def forward(self, x: Tensor) -> Tensor:
        if self.gate is None:
            return self.proj(self.act(self.fc(x)))
        return self.proj(self.act(self.gate(x)) * self.fc(x))

    def forward_explicit(self, x: Tensor, hook: Hooks, layer: int) -> Tensor:
        """`mlp.l.pre` is what enters the nonlinearity (the gate, in a gated unit); `mlp.l.post` the neurons."""
        if self.gate is None:
            post = self.act(hook(f"mlp.{layer}.pre", self.fc(x)))
        else:
            post = self.act(hook(f"mlp.{layer}.pre", self.gate(x))) * self.fc(x)
        return self.proj(hook(f"mlp.{layer}.post", post))

fc is \(W_{\mathrm{in}}\), proj is \(W_{\mathrm{out}}\) and act is \(\phi\), so the first return is (5.5); the maps act on the last axis of a tensor of shape (batch, \(T\), \(d\)), one token at a time. The second return is the gated layer, with gate, fc and proj in the places of \(W\), \(V\) and \(W_2\) above, and for the two gated names ACTIVATIONS gives the function that squashes the gate; its six keys are the six models of Example 5.3.4 below. The hidden width is ModelConfig.d_mlp, which for a gated unit is round(width * 2 / 3): 2,048 at \(d = 768\), as in Proposition 5.3.3, and 1,365 at \(d = 512\), where two thirds of 2,048 is not a whole number, so that a gated block of the example has 512 weights fewer than a plain one. forward_explicit is the same function with two named hooks, mlp.l.pre on what enters the nonlinearity and mlp.l.post on the hidden layer, which holds the neurons of Section 5.4.

Example 5.3.4 (six activations in one model). The setup: the 8-layer model in the 2026 recipe, 50M tokens of FineWeb-EDU, two seeds each (one for GEGLU); the activation is ReLU, GELU, SiLU, squared ReLU, SwiGLU or GEGLU, its sibling with a GELU gate, the gated two at two thirds of the width, so that every model has 58.7M parameters. Against the square, which ends at 4.045 and 4.043 nats in the two seeds: ReLU is 0.018 and 0.016 behind, twice the yardstick of Section 5.2; SiLU is 0.007 and 0.006 behind and SwiGLU 0.008 and 0.011 ahead, small, with the same sign both times; GELU is level (0.001 behind, then 0.002 ahead), as is GEGLU in its one seed. From ReLU to SwiGLU there are 0.026 in both seeds, the sign of the T5 experiment's 0.041. So the square is worth 0.017 over ReLU and nothing over GELU, and at this size and budget only ReLU costs more than a hundredth of a nat.

What differs more is what the units do, read on 102,400 held-out tokens. In every ungated model 83% to 86% of what enters the nonlinearity is below zero. ReLU and its square answer with an exact zero (86% and 83% of all activations), GELU with a small negative number (1% exact zeros), SiLU with numbers that are almost never negligible. Squaring makes a few activations very large: the kurtosis of the activations is 1,306 for the square, about 300 for the two gated units and under 50 for the other three. Units that never fire are rare in all six: none in five of the models, 0.03% of the units of the square. ◁

Others' measurements agree that the gaps are small and disagree on their sign: 23 of the 24 open models released since January 2025 gate their MLP, the small fast recipes of Section 5.6 square it, and at 1.1B parameters one comparison has the square at 2.337 ± 0.004 against 2.353 ± 0.004 for SwiGLU.A. H. Huang and I. Schlag, "Deriving activation functions using integration", 2025 (arXiv 2411.13010), Table 2. No theory predicts the sign.

What the MLP stores

Neuron 1,445 of block 8 of gpt-pp has a recognizable job: the 25 passages of training text on which it fires hardest all end in "on the other", and the token that its output makes most likely is "hand". Those are two different observations about a neuron: where it fires, and what it promotes.

block 8, neuron 1,445 of gpt-pp, of 200 sampled in each of 12 blocks; text: 2.0M training tokens
key    the text on which it fires hardest, the trigger token in brackets
         …. Non-epileptic seizures, on the[ other] hand,
       …itutions. The family waqfs, on the[ other] hand,
       …ration. Sandia’s technique, on the[ other] hand,
       …des the spinal cord. Steak, on the[ other] hand,
       all 25 strongest triggers end in “ other”
value  promotes “ hand” “ Hand” “hand” “-hand” “ hands” “Hand”; 25 of 25 triggers are followed by one of its 30 most promoted tokens

block                                      1   2   3   4   5   6   7   8   9  10  11  12
top triggers ending in the same token, %  29  41  40  40  37  39  36  35  34  32  28  26   (8 at random)
top token follows the top trigger, %       1   1   2   3   3   5   5   8  13   7   5   5
top triggers followed by a top token, %    5   6   6   6   9   9  12  16  21  17  15  11
(the live figure needs JavaScript and loads about 0.4 MB of data)
Neurons of gpt-pp’s MLPs read as key-value memories. Left: the training text on which the picked neuron fires hardest, aligned on the token where it fires. Green: those trigger tokens. Right: the tokens its output weights promote and suppress, read through the unembedding, with the number of triggers each one follows. Above: over 200 sampled neurons a block, how often a neuron’s top triggers end in the same token, and how often its value continues its key. Pick a block and one of ten neurons that span the range. Measured on one model over 2.0M tokens.

The two lists are the two halves of an idea of 2020. It starts from a complaint, "Feed-forward layers constitute two-thirds of a transformer model’s parameters, yet their role in the network remains under-explored", and from the form of (5.5):

Transformer feed-forward layers are key-value memoriesEMNLP 2021 · arXiv:2012.14913M. Geva, R. Schuster, J. Berant and O. Levy

2 Feed-Forward Layers as Unnormalized Key-Value Memories

Feed-forward layers A transformer language model Vaswani et al. (2017) is made of intertwined self-attention and feed-forward layers. Each feed-forward layer is a position-wise function, processing each input vector independently. Let \(\mathbf{x}\in\mathbb{R}^{d}\) be a vector corresponding to some input text prefix. We can express the feed-forward layer \(\text{FF}(\cdot)\) as follows (bias terms are omitted):

\[\text{FF}(\mathbf{x})=f(\mathbf{x}\cdot K^{\top})\cdot V \qquad (1)\]

Here, \(K,V\in\mathbb{R}^{d_{m}\times d}\) are parameter matrices, and \(f\) is a non-linearity such as ReLU.

Neural memory A neural memory Sukhbaatar et al. (2015) consists of \(d_{m}\) key-value pairs, which we call memories. Each key is represented by a \(d\)-dimensional vector \(\mathbf{k}_{i}\in\mathbb{R}^{d}\), and together form the parameter matrix \(K\in\mathbb{R}^{d_{m}\times d}\); likewise, we define the value parameters as \(V\in\mathbb{R}^{d_{m}\times d}\). Given an input vector \(\mathbf{x}\in\mathbb{R}^{d}\), we compute a distribution over the keys, and use it to compute the expected value:

\[p(k_{i}\mid x)\propto\exp(\mathbf{x}\cdot\mathbf{k}_{i})\]

\[\text{MN}(\mathbf{x})=\sum_{i=1}^{d_{m}}p(k_{i}\mid x)\mathbf{v}_{i}\]

With matrix notation, we arrive at a more compact formulation:

\[\text{MN}(\mathbf{x})=\text{softmax}(\mathbf{x}\cdot K^{\top})\cdot V \qquad (2)\]

Feed-forward layers emulate neural memory Comparing equations 1 and 2 shows that feed-forward layers are almost identical to key-value neural memories; the only difference is that neural memory uses softmax as the non-linearity \(f(\cdot)\), while the canonical transformer does not use a normalizing function in the feed-forward layer. The hidden dimension \(d_{m}\) is essentially the number of memories in the layer, and the activation \(\mathbf{m}=f(\mathbf{x}\cdot K^{\top})\), commonly referred to as the hidden layer, is a vector containing an unnormalized non-negative coefficient for each memory. We refer to each \(\mathbf{m}_{i}\) as the memory coefficient of the \(i\)th memory cell.

Section 2, its first three paragraphs
M. Geva, R. Schuster, J. Berant and O. Levy, "Transformer feed-forward layers are key-value memories" (EMNLP 2021, arXiv 2012.14913), Section 2, its first three paragraphs. MN is a key-value memory network. The sentence quoted before the fragment opens the abstract.

The paper writes vectors as rows. In the convention of (5.5), where they are columns:

Definition 5.4.1 (keys, values, memory coefficients). Let \(k_i\) be row \(i\) of \(W_{\mathrm{in}}\) and \(v_i\) column \(i\) of \(W_{\mathrm{out}}\). Then

\[\mathrm{MLP}(x) = \sum_{i=1}^{d_m} m_i(x)\, v_i, \qquad m_i(x) = \phi(k_i \cdot x). \tag{5.6}\]

\(k_i\) is the key of neuron \(i\), \(v_i\) its value, \(m_i(x)\) its memory coefficient, and \(m_i(x)\, v_i\) its sub-update.

The first paper studies the keys. In a 16-layer language model with \(d_m = 4{,}096\) it collects, for sampled neurons, the sentence prefixes with the largest memory coefficient, and hands them to annotators.

Key Pattern Example trigger prefixes
\(\mathbf{k}_{449}^{1}\) Ends with “substitutes” (shallow) At the meeting, Elton said that “for artistic reasons there could be no substitutes / In German service, they were used as substitutes / Two weeks later, he came off the substitutes
\(\mathbf{k}_{2546}^{6}\) Military, ends with “base”/“bases” (shallow + semantic) On 1 April the SRSG authorised the SADF to leave their bases / Aircraft from all four carriers attacked the Australian base / Bombers flying missions to Rabaul and other Japanese bases
\(\mathbf{k}_{2997}^{10}\) a “part of” relation (semantic) In June 2012 she was named as one of the team that competed / He was also a part of the Indian delegation / Toy Story is also among the top ten in the BFI list of the 50 films you should
\(\mathbf{k}_{2989}^{13}\) Ends with a time range (semantic) Worldwide, most tornadoes occur in the late afternoon, between 3 pm and 7 / Weekend tolls are in effect from 7:00 pm Friday until / The building is open to the public seven days a week, from 11:00 am to
\(\mathbf{k}_{1935}^{16}\) TV shows (semantic) Time shifting viewing added 57 percent to the episode’s / The first season set that the episode was included in was as part of the / From the original NBC daytime version , archived
Table 1 of Geva, Schuster, Berant and Levy (2021), reproduced: "Examples of human-identified patterns that trigger different memory keys." The superscript of a key is its layer, the subscript its neuron; a slash separates three prefixes on which the key's memory coefficient is among the largest.

Annotators found at least one pattern for every key sampled, 3.6 a key on average; shallow patterns such as a shared last word dominate layers 1 to 9, semantic ones layers 10 to 16. In at least about 68% of inputs the layer's top prediction is the top prediction of none of its memories: the output is a composition. Read through the unembedding, a value agrees with the token that follows its key's top trigger in 3.5% of neurons from layer 11 on and almost never below (chance: 0.0004%).Geva and others (2021), Sections 3 to 5; a model of 247M parameters trained on WikiText-103.

The second paper makes the output side exact.

Transformer feed-forward layers build predictions by promoting concepts in the vocabulary spaceEMNLP 2022 · arXiv:2203.14680M. Geva, A. Caciularu, K. R. Wang and Y. Goldberg

FFN Outputs as Linear Vector Combinations. Each FFN at layer \(\ell\) consists of two linear transformations with a point-wise activation function in between (bias terms are omitted):

\[\texttt{FFN}^{\ell}(\mathbf{x}^{\ell})=f\left(W_{K}^{\ell}\mathbf{x}^{\ell}\right)W_{V}^{\ell},\]

where \(W_{K}^{\ell},W_{V}^{\ell}\in\mathbb{R}^{d_{m}\times d}\) are parameter matrices, and \(f\) is a non-linearity function. Previous work proposed this module can be cast as an emulated neural key-value memory Sukhbaatar et al. (2015); Sukhbaatar et al. (2019), where rows in \(W_{K}^{\ell}\) and columns in \(W_{V}^{\ell}\) are viewed as keys and values, respectively Geva et al. (2021). For an input \(\mathbf{x}^{\ell}\), the keys produce a vector of coefficients \(\mathbf{m}^{\ell}:=f\left(W_{K}^{\ell}\mathbf{x}^{\ell}\right)\in\mathbb{R}^{d_{m}}\), that weighs the corresponding values in \(W_{V}^{\ell}\). Denoting by \(\mathbf{k}_{i}^{\ell}\) the \(i\)-th row of \(W_{K}^{\ell}\) and by \(\mathbf{v}_{i}^{\ell}\) the \(i\)-th column of \(W_{V}^{\ell}\), we can then use the following formulation:

\[\texttt{FFN}^{\ell}(\mathbf{x}^{\ell})=\sum_{i=1}^{d_{m}}f(\mathbf{x}^{\ell}\cdot\mathbf{k}_{i}^{\ell})\mathbf{v}_{i}^{\ell}=\sum_{i=1}^{d_{m}}m_{i}^{\ell}\mathbf{v}_{i}^{\ell}.\]

Therefore, a FFN update can be viewed as a collection of sub-updates, each corresponding to a weighted value vector in the FFN output.

[…]

Interpreting Sub-Updates in the Vocabulary Space. Consider a sub-update \(m_{i}^{\ell}\mathbf{v}_{i}^{\ell}\) for a given input, we can estimate its influence on the representation \(\mathbf{x}^{\ell}\) (before the FFN update) by analyzing the change it induces on the output distribution. Concretely, we isolate the effect of \(m_{i}^{\ell}\mathbf{v}_{i}^{\ell}\) on the probability \(\mathbf{p}^{\ell}_{w}\) of \(w\in\mathcal{V}\):

\[p\big(w\;|\;\mathbf{x}^{\ell}+m_{i}^{\ell}\mathbf{v}_{i}^{\ell},E\big)=\frac{\exp{\big(\mathbf{e}_{w}\cdot\mathbf{x}^{\ell}+\mathbf{e}_{w}\cdot m_{i}^{\ell}\mathbf{v}_{i}^{\ell}\big)}}{Z\big(E(\mathbf{x}^{\ell}+m_{i}^{\ell}\mathbf{v}_{i}^{\ell}))}\propto\exp{\big(\mathbf{e}_{w}\cdot\mathbf{x}^{\ell}\big)}\cdot\exp{\big(\mathbf{e}_{w}\cdot m_{i}^{\ell}\mathbf{v}_{i}^{\ell}\big)}, \qquad (2)\]

where \(\mathbf{e}_{w}\) is the embedding of \(w\), and \(Z\big(\cdot\big)\) is the constant softmax normalization factor.

This implies that each sub-update \({m_{i}^{\ell}\mathbf{v}_{i}^{\ell}}\) introduces a scaling factor to the probability of every token \(w\) based on its dot product with \(\mathbf{e}_{w}\). Specifically, having \({\mathbf{e}_{w}\cdot m_{i}^{\ell}\mathbf{v}_{i}^{\ell}>0}\) increases the probability of \(w\), and having \({\mathbf{e}_{w}\cdot m_{i}^{\ell}\mathbf{v}_{i}^{\ell}<0}\) decreases it. This scaling factor can be split into two parts:

• The term \({\mathbf{e}_{w}\cdot\mathbf{v}_{i}^{\ell}}\) can be viewed as a static score of \(w\) that is independent of the input to the model. Thus, the projection \({\mathbf{r}_{i}^{\ell}=E\mathbf{v}_{i}^{\ell}\in\mathbb{R}^{|\mathcal{V}|}}\) induces a ranking over the vocabulary that allows comparing the scores by \(\mathbf{v}_{i}^{\ell}\) w.r.t different tokens.

• The term \(m_{i}^{\ell}\) is the dynamic coefficient of \(\mathbf{v}_{i}^{\ell}\), which is fixed for all tokens for a given input. Thus, these coefficients allow comparing the contribution of value vectors in a specific update.

Section 3, Equation 2
M. Geva, A. Caciularu, K. R. Wang and Y. Goldberg, "Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space" (EMNLP 2022, arXiv 2203.14680), Section 3, Equation 2. E is the output embedding, with rows e, and x the stream at layer l.

Proposition 5.4.2 (a sub-update multiplies every probability). Let the logits of a stream \(r\) be \(Er\), where \(E\) is the output embedding of the fragment (the unembedding \(W_U\) of Definition 5.1.1, not the input embedding) and \(e_w\) its row for token \(w\), and let \(p(\cdot \mid r)\) be their softmax. For any \(u \in \mathbb R^d\),

\[p(w \mid r + u) = \frac{p(w \mid r)\, e^{e_w \cdot u}}{\sum_{w^{\prime}} p(w^{\prime} \mid r)\, e^{e_{w^{\prime}} \cdot u}} .\]

With \(u = m_i v_i\), the static scores \(E v_i\) rank the vocabulary once and for all, and the input decides only the coefficient. For \(m_i > 0\) the tokens at the top of the ranking are promoted and those at its bottom demoted.

Proof. \(E(r + u) = Er + Eu\), and the softmax is the normalized exponential. ∎

Two things depend on the block. With ReLU or its square \(m_i \ge 0\), so a neuron pushes one way only; a gated unit is signed, and both ends of its ranking matter. And logits are read through a normalization: RMS norm without gain only rescales a lone vector, so the ranking that \(v_i\) induces is unchanged, and to first order the sub-update acts through \(E\, J_N(r)\, m_i v_i\), which by (5.4) is \(v_i\) with its component along \(r\) removed, divided by \(\mathrm{rms}(r)\). With layer norm the ranking can change: the paper finds 64.5% of the top 30 tokens in common in GPT-2.

The first paper had conjectured that the layer "acts as an elimination mechanism to “veto” the top prediction in the residual". The second measured the opposite: tokens that end up predicted are pushed up by the dominant sub-updates, and those that drop out receive scores near zero. "This suggests that FFN updates work in a promotion mechanism, where top-candidate tokens are those being pushed by dominant sub-updates."Geva and others (2022), Table 4 and Appendix A.1; the conjecture is in Section 5.2 of the first paper. A qualification from Voita and others (2023): in early layers, neurons that detect a token have updates that "explicitly focus on removing the information about triggering them tokens, i.e., current input".

Example 5.4.3 (the keys and values of gpt-pp). The setup: gpt-pp, 200 neurons sampled in each of its 12 blocks; for a neuron, the 25 positions in 2.0M tokens of training text with the largest memory coefficient, and the 30 tokens ranked highest by \(W_U v_i\). Neuron 1,445 of block 8 is a memory in the sense of both papers: all 25 of its triggers end in "other", and each is followed by a token that its value promotes. A neuron of block 12, number 1,960, has the other kind of key: its triggers do not end in one token (a fifth of them end in "broken", three in "bogged"), and all 25 are followed by a token that its value promotes, "down" first among them. Over the sample, a neuron's strongest triggers end in one and the same token 41% of the time at block 2 and 26% at block 12, against 8% for positions drawn at random: the keys of the lower blocks are the shallower ones, as in the table of 2021, by a modest margin. The token that a value promotes most is the token after its key's strongest trigger for 1% of the neurons of block 1, 13% at block 9 and 5% at block 12, and almost never when values are dealt to other neurons: values continue their keys in the upper half of the model, not at its very top. ◁

In the code. Both readings of a neuron are short. Its memory coefficient is what MLP.forward_explicit (Section 5.3) hands to the hook mlp.l.post, and the scores of its value are one matrix product. They are value_vectors and value_scores in gpt-pp's memories.py.

def value_vectors(model: GPT, layer: int, neurons: Optional[Tensor] = None) -> Tensor:
    """The values of a layer, one row a neuron: columns of the MLP's output matrix, (n, d)."""
    w = model.blocks[layer].mlp.proj.weight.float().T
    return w if neurons is None else w[neurons.to(w.device)]


@torch.no_grad()
def value_scores(model: GPT, layer: int, neurons: Optional[Tensor] = None) -> Tensor:
    """W_U v_i for the neurons of a layer: (n, V), the static score of every token under each value."""
    return value_vectors(model, layer, neurons) @ model.unembed.weight.float().T

proj.weight is \(W_{\mathrm{out}}\), of shape (\(d\), \(d_m\)), so the rows of its transpose are the values \(v_i\), and the product with the transposed unembedding is \(W_U v_i\) for each of the \(n\) neurons asked for: a table of shape (\(n\), \(V\)) in which a row ranks the vocabulary.

Quirk. The score is the bare product: the final norm is left out, and so is the cap that the model puts on its logits (Section 2). The norm of the 2026 recipe leaves the ranking as it is, by the argument before the example; for a model with a layer norm the value would have to be normalized first, which the code does not do. Gated MLPs are not covered: there the hook mlp.l.pre holds the gate, and a coefficient has a sign. The pass that collects the triggers stops the model after the last block it needs by raising an exception inside that block's hook, and none of this changes a number of the example.

Editing a memory. If the layer is a memory, a fact is a pair that can be rewritten. The step from reading to surgery changed the unit: "We hypothesize that MLPs can be modeled as a linear associative memory; note that this differs from Geva et al. 2021’s per-neuron view."K. Meng, D. Bau, A. Andonian and Y. Belinkov, "Locating and editing factual associations in GPT", NeurIPS 2022 (arXiv 2202.05262), Section 3 and Equation 2. The caution: P. Hase and others, "Does localization inform editing?", NeurIPS 2023 (arXiv 2301.04213). The memory is the whole matrix \(W = W_{\mathrm{out}}\), a key is the vector of memory coefficients at the last token of a subject, and its value is what the matrix then writes.

If \(W\) is the least-squares memory of keys \(K\) and values \(V\), held as columns, and \(C = KK^{\top}\) is invertible, the matrix that minimizes \(\|\hat W K - V\|_F\) subject to \(\hat W k_* = v_*\) for a new pair is, by a Lagrange multiplier,

\[\hat W = W + \Lambda\, (C^{-1} k_*)^{\top}, \qquad \Lambda = \frac{v_* - W k_*}{k_*^{\top} C^{-1} k_*} .\]

The edit is one outer product, and an old key \(k_j\) now returns its old value plus \(\Lambda\,(k_*^{\top} C^{-1} k_j)\): the whitening by \(C^{-1}\) aims the change where the other keys are not. One caution stands: where a fact is located by tracing does not predict which layer is best to edit.

How much fits. Store \(n\) pairs in one matrix as a sum of outer products, \(W = \sum_i v_i k_i^{\top}\) with unit keys, and read one back:

\[W k_j = v_j + \sum_{i \ne j} (k_i \cdot k_j)\, v_i . \tag{5.7}\]

With orthonormal keys the second term vanishes, and \(n \le d\). With keys and values drawn at random from the unit sphere it is noise of expected squared norm \((n - 1)/d\), since \(\mathbb E (k_i \cdot k_j)^2 = 1/d\) (Section 2). That ruins \(v_j\) as a vector once \(n\) nears \(d\), but the output of a layer is only compared with a finite list, and the largest inner product over a list tolerates far more.Recovering a vector accurately is a stronger demand than choosing its name from a list. Interference can spoil the vector while leaving the correct value ranked first; the apparent capacity depends on which task counts as recall.

memory  n pairs of unit vectors (key k, value v) in d dimensions, stored as W = sum of v kᵀ
read    W k_j = v_j + what leaks from the other pairs; read back when v_j scores highest

random keys, d = 128  n = 64      n = 512     n = 2,048
linear, read back     100 %       97.1 %      22.5 %
relu, read back       100 %       100 %       62.5 %
relu², read back      100 %       100 %       100 %

rewrite one of 512 pairs: W += r aᵀ; every other pair moves by a share of the change r
plain, a = k          0.092 (root mean square)
with the covariance C 0.079
(the live figure needs JavaScript; it computes in the page and loads no data)
A matrix as a memory: n random key-value pairs in d dimensions stored as a sum of outer products, each key read back against all n values. Green: the keys whose own value scores highest, as dots over the diagonal and as their share against n. Grey curves: the same pairs with a ReLU, or its square, between key and value. Right: one pair rewritten by a rank-one edit, plain and with the keys’ covariance, and how far the others move. Change n, the memory, the keys, d. Computed in the page from one draw.

Count a key as recalled when its own value scores higher than every other value. At gpt-pico's width, \(d = 128\), the matrix returns all of 64 random pairs, 497 of 512 and 460 of 2,048; at \(d = 768\), all of 2,048. The score of the right value is 1 and the leak from the other pairs has a spread of \(\sqrt n/d\), so the matrix starts to fail not near \(n = d\) but near \(n = d^2/(4 \ln n)\). A ReLU between key and value, one unit a pair, moves the fall to the right, and its square, the unit of our block, further still: at \(d = 128\) all of 8,192 pairs come back, at the price of \(2nd\) numbers in place of \(d^2\). Rewriting one pair with the edit above also changes the answers to other keys. With random keys the other pairs move by 0.092 of the change, and by 0.079 with the covariance of the keys; with keys that share a direction, as activations do, by 0.54 and by 0.085.

Theory agrees with the count: when the association is one to one, "the storage capacities of both linear and MLP associative memories scale linearly with parameter count". Measured on GPT-style models of up to 1.5B parameters, the figure is about 3.6 bits a parameter.E. Nichani, J. D. Lee and A. Bietti, "Understanding factual recall in transformers via associative memories", 2024 (arXiv 2412.06538), abstract and Theorems 1 and 2. J. X. Morris and others, "How much do language models memorize?", 2025 (arXiv 2505.24832), abstract.

Two later turns belong to other sections. Neurons answer to several unrelated patterns at once, so since 2024 the layer is rewritten as a wider, sparser layer trained to imitate it, a transcoder, which is (5.6) with learned memory cells (Section 10). And large models make the layer wide and sparse in the architecture itself: a mixture of experts runs a few of many feed-forward layers for each token, in 19 of the 24 open models released since January 2025. Read this way, the two halves of the block are one object.

Attention is a key-value memory written at inference time, out of the context. The MLP is the same kind of memory written by training, out of the corpus.

Initialization and signal propagation

Before the first step every weight is a random number, and its scale decides whether anything can be learned. For a network without skips the classical rule sets the variance of the weights so that neither the signal on its way up nor the gradient on its way down dies or explodes: \(2/n\) for a layer of fan-in \(n\) with ReLU units, the two paying for the half of the units that are off. Mean-field theory made of the rule a criterion: "random networks may be trained precisely when information can travel through them".K. He, X. Zhang, S. Ren and J. Sun, "Delving deep into rectifiers: surpassing human-level performance on ImageNet classification", 2015 (arXiv 1502.01852); S. S. Schoenholz, J. Gilmer, S. Ganguli and J. Sohl-Dickstein, "Deep information propagation", ICLR 2017 (arXiv 1611.01232), abstract. A block with a skip changes the question: the skip carries lengths and angles through unchanged, and what must be controlled is how much the branches add.

The 0.02. GPT drew every weight from a normal law of standard deviation 0.02, whatever the layer, and gave its reason in one sentence of the paragraph that holds its whole recipe:

Model specifications Our model largely follows the original transformer work [62]. We trained a 12-layer decoder-only transformer with masked self-attention heads (768 dimensional states and 12 attention heads). For the position-wise feed-forward networks, we used 3072 dimensional inner states. We used the Adam optimization scheme [27] with a max learning rate of 2.5e-4. The learning rate was increased linearly from zero over the first 2000 updates and annealed to 0 using a cosine schedule. We train for 100 epochs on minibatches of 64 randomly sampled, contiguous sequences of 512 tokens. Since layernorm [2] is used extensively throughout the model, a simple weight initialization of N(0, 0.02) was sufficient. We used a bytepair encoding (BPE) vocabulary with 40,000 merges [53] and residual, embedding, and attention dropouts with a rate of 0.1 for regularization. We also employed a modified version of L2 regularization proposed in [37], with w = 0.01 on all non bias or gain weights. For the activation function, we used the Gaussian Error Linear Unit (GELU) [18]. We used learned position embeddings instead of the sinusoidal version proposed in the original work. We use the ftfy library to clean the raw text in BooksCorpus, standardize some punctuation and whitespace, and use the spaCy tokenizer.

—A. Radford, K. Narasimhan, T. Salimans and I. Sutskever, "Improving language understanding by generative pre-training" (2018).OpenAI, 2018, Section 4.1, page 5, with two footnote marks dropped; reference [2] is the paper on layer normalization.

It is a constant and not a fan-in rule: \(1/\sqrt d\) is 0.036 at \(d = 768\) and 0.025 at 1,600, so 0.02 lies below the variance-preserving scale at every width of GPT-2, and the normalization absorbs the difference.

The depth factor. GPT-2 added one sentence: the weights of residual layers are scaled at initialization by \(1/\sqrt N\), with \(N\) the number of residual layers. The reason was printed two months later by the same laboratory: "We scale the initialization of \(W_{2}\) and \(W_{p}\) in Eq. 5 by \(\frac{1}{\sqrt{2N}}\) to keep the ratio of input embedding scale to residual block scale invariant across values of \(N\)".R. Child, S. Gray, A. Radford and I. Sutskever, "Generating long sequences with sparse transformers", 2019 (arXiv 1904.10509), Section 5.2; there \(N\) counts blocks, and the two matrices are the last ones of the MLP and of attention.

Proposition 5.5.1 (why \(1/\sqrt{2L}\)). Let the \(2L\) sublayers of a pre-norm model write, at initialization, vectors whose coordinates have variance \(s^2\) and are uncorrelated across sublayers and with the embedding, whose coordinates have variance \(e^2\). Then the coordinates of the last stream have variance \(e^2 + 2L s^2\). Scaling the last matrix of every sublayer by \(1/\sqrt{2L}\) makes it \(e^2 + s^2\) at every depth.

Proof. Variances of uncorrelated terms add; each sublayer reads a normalized input, so \(s\) does not depend on what has accumulated; and its output is linear in its last matrix. ∎

The code released with GPT-2 draws every matrix at 0.02; the ports and reproductions read \(N\) as \(2L\) and scale the two matrices that write to the stream, the writers from here on, to a standard deviation of 0.0041 at 12 blocks and of 0.0020 at 48.

Zero. In the same year a paper showed that such a scaling can stand in for normalization altogether, and boxed its recipe.

Fixup initialization: residual learning without normalizationICLR 2019 · arXiv:1901.09321H. Zhang, Y. N. Dauphin and T. Ma

Put together, we propose the following method to train residual networks without normalization:

Fixup initialization (or: How to train a deep residual network without normalization)

1. Initialize the classification layer and the last layer of each residual branch to \(0\).

2. Initialize every other layer using a standard method (e.g., He et al. (2015)), and scale only the weight layers inside residual branches by \(L^{-\frac{1}{2m-2}}\).

3. Add a scalar multiplier (initialized at 1) in every branch and a scalar bias (initialized at 0) before each convolution, linear, and element-wise activation layer.

Section 3, the boxed recipe
H. Zhang, Y. N. Dauphin and T. Ma, "Fixup initialization: residual learning without normalization" (ICLR 2019, arXiv 1901.09321), Section 3, the boxed recipe. L is the number of branches and m the number of layers in a branch; for the two matrices of an MLP the exponent is one half, GPT-2's.

Rule 1 is the one that lasted. It is the limit of the depth factor, which is what batch normalization had been supplying in residual networks all along (De and Smith, in Section 5.2), and it is what a scalar on the branch that starts at zero does.T. Bachlechner and others, "ReZero is all you need: fast convergence at large depth", 2020 (arXiv 2003.04887). In the block of 2026 the two writers start at zero.

Example 5.5.2 (gpt-pico before its first step). The setup: gpt-pico at initialization, scored on held-out TinyStories. With both writers at zero every block is the identity, and the model is exactly embed, normalize, unembed. Its unembedding is drawn at a standard deviation of 0.001, so the logits are nearly zero and the loss should be \(\log 4{,}096 = 8.318\) nats; measured, 8.318. The 0.001 has a reason: from an unembedding of exactly zero the first step would move the unembedding and nothing else. ◁

In the code. The 0.02 with its depth factor and the zero of the 2026 recipe are the two values of one switch of the configuration, init: gpt2 and zero_proj. It is GPT.init_weights in gpt-pp's model.py.

@torch.no_grad()
def init_weights(self) -> None:
    ...
    cfg = self.cfg
    gpt2 = cfg.init == "gpt2"
    self.embed_std = 0.02 if gpt2 or cfg.tie_embeddings else 1.0
    nn.init.normal_(self.embed.weight, std=self.embed_std)
    if self.pos_embed is not None:
        nn.init.normal_(self.pos_embed.weight, std=self.embed_std)
    if not cfg.tie_embeddings:
        nn.init.normal_(self.unembed.weight, std=0.02 if gpt2 else 0.001)
    n_residual = cfg.n_layer * (2 if cfg.use_mlp else 1)
    for block in self.blocks:
        writers = [block.attn.o] + ([block.mlp.proj] if block.mlp is not None else [])
        for module in block.modules():
            if not isinstance(module, nn.Linear):
                continue
            writes = any(module is w for w in writers)
            if gpt2:
                nn.init.normal_(module.weight, std=0.02 / math.sqrt(n_residual) if writes else 0.02)
            elif writes:
                nn.init.zeros_(module.weight)
            else:
                bound = 1 / math.sqrt(module.in_features)
                nn.init.uniform_(module.weight, -bound, bound)
            if module.bias is not None:
                nn.init.zeros_(module.bias)

n_residual is the \(2L\) of Proposition 5.5.1, or \(L\) in a model without MLPs, and the writers are attn.o and mlp.proj, the last matrix of each sublayer. The two tables are drawn before the loop: untied, as in the 2026 recipe, the embedding at a standard deviation of one, the scale of the normalized stream, and the unembedding at the 0.001 of Example 5.5.2. Biases, where a recipe has them, start at zero.

Quirk. The switch moves more than the writers. Under zero_proj every other matrix of a block is drawn uniformly between \(\pm 1/\sqrt n\), with \(n\) its fan-in, which is the width \(d\) for each of them (a standard deviation of 0.0255 at width 512, where gpt2 has 0.02), and the untied tables at 1 and 0.001, where gpt2 has 0.02 for both; a tied table keeps 0.02 whatever the switch says. So each comparison of the two that this series reports, in the next paragraph and as change 6 of Section 5.6, sets one whole scheme against the other: on the ladder, where the table is still tied, the writers and the draw of the other matrices change together, and in the finished recipe the two tables change as well.

Put to the test in the 2026 recipe (8 layers, 50M tokens, one seed each), the two schemes cannot be told apart: the model ends at 4.044 nats under gpt2, its writers scaled by \(1/\sqrt{2L}\), and at 4.045 under zero_proj, its writers at zero, and the two curves cross on the way. Plain 0.02 on every matrix was not run, so what the depth factor is worth is not measured here. On the ladder, under the old training recipe, the step from the one scheme to the other did help, by 0.025 and 0.030 in its two seeds, and by 0.008 in one seed once both rungs have a longer warmup; Section 5.6 sets the measurements side by side.

What the skip does on the way up. The skip has a second job, which is not about gradients.

Attention-based architectures have become ubiquitous in machine learning. Yet our understanding of the reasons for their effectiveness remains limited. This work proposes a new way to understand self-attention networks: we show that their output can be decomposed into a sum of smaller terms, each involving the operation of a sequence of attention heads across layers. Using this decomposition, we prove that self-attention possesses a strong inductive bias towards “token uniformity”. Specifically, without skip connections or multi-layer perceptrons (MLPs), the output converges doubly exponentially to a rank-1 matrix. On the other hand, skip connections and MLPs stop the output from degeneration. Our experiments verify the identified convergence phenomena on different variants of standard transformer architectures.

—Y. Dong, J.-B. Cordonnier and A. Loukas, "Attention is not all you need: pure attention loses rank doubly exponentially with depth" (2021).arXiv 2103.03404, abstract. The consequences for training: L. Noci and others, "Signal propagation in transformers: theoretical perspectives and the role of rank collapse", 2022 (arXiv 2206.03126), Theorem 3.1 and Lemma 3.2. A second failure, attention that saturates on single tokens when queries and keys are large: A. Giorlandino and S. Goldt, "Two failure modes of deep transformers and how to avoid them", ICLR 2026 (arXiv 2505.24333).

The part of the argument worth seeing is that attention averages.

Proposition 5.5.3 (attention contracts). Let \(A\) be a \(T \times T\) matrix with nonnegative rows that sum to one, as every attention pattern is, and for a stream \(X\) with rows \(x_i\) let \(\operatorname{diam}(X) = \max_{i,j} \|x_i - x_j\|\). Then

\[\operatorname{diam}(AX) \le \delta(A)\, \operatorname{diam}(X), \qquad \delta(A) = 1 - \min_{i,j} \sum_k \min(A_{ik}, A_{jk}),\]

and \(\delta(A) < 1\) whenever every two rows of \(A\) put weight on a common token, as under a causal mask, where every row puts weight on the first.

Proof. Row \(i\) of \(AX\) minus row \(j\) is \(\sum_k a_k x_k\) with \(a_k = A_{ik} - A_{jk}\), which sum to zero. The positive and the negative part of \(a\) each have total mass \(s = 1 - \sum_k \min(A_{ik}, A_{jk})\), so the sum is \(s\,(p - q)\) with \(p\) and \(q\) convex combinations of the rows of \(X\), and \(\|p - q\| \le \operatorname{diam}(X)\). ∎

Without skips the stream after \(\ell\) attention layers is \(A_\ell \cdots A_1 X\) times value matrices on the right: the tokens are pulled together layer after layer. With a skip, the first identity of (5.2) keeps the embedding in the stream at every depth, with all the differences between its tokens. The stakes are in training: when all tokens coincide the gradient of the queries and keys is zero, since attention cannot learn where to look when every key is the same. We test this on random weights at the initialization of GPT-2: 32 tokens through 48 layers of attention alone, of attention with skips, with skips and MLPs, and of the full pre-norm block.

mean cosine between 32 random token vectors, 128 wide, after each layer
(random weights at the initialization of GPT-2, causal attention)

layer                          0      1      2      4      8     16     48
attention alone             0.00   0.73   0.89   0.98   1.00   1.00   1.00   one direction
with skip connections       0.00   0.00   0.00   0.00   0.00   0.00   0.00
with skips and MLPs         0.00   0.00   0.00   0.00   0.00   0.00   0.00
the full pre-norm block     0.00   0.00   0.01   0.02   0.05   0.12   0.22

attention alone: above 0.99 after 5 layers; what tells the tokens apart
shrinks 2.0-fold a layer, to 5.3 × 10⁻¹⁴ of their size at layer 48
residual weights not scaled by 1/√(2L): the full block reaches 0.86
(the live figure needs JavaScript; it computes everything in the page and loads no data)
Rank collapse at initialization. Green: the mean cosine between 32 random token vectors after each layer of attention alone; within a few layers they are one direction. Ink: the same stack with skip connections, with MLPs, and as the full pre-norm block. The squares are the cosines of every pair of tokens at the selected layer, darker for more alike. Drag the depth, change the width, drop the scaling of the residual weights, redraw. Computed in the page from random weights at the initialization of GPT-2; one draw.

Through attention alone the tokens point one way after five layers (mean cosine above 0.99), and what still tells them apart halves at every further layer: with scores near zero, causal attention is the matrix whose row \(i\) averages the first \(i\) tokens, and its eigenvalues are \(1, \tfrac12, \tfrac13, \dots\) With the skip the tokens are as far apart at layer 48 as at the start. In the full pre-norm block they do drift together, to a mean cosine of 0.22 at layer 48 with the depth factor of Proposition 5.5.1 and to 0.86 without it, which is the case for scaling what a block writes.

The 2019 block beside the 2026 block

The nine changes from GPT-2 to gpt-pp are measured twice on an 8-layer model: on the way up a ladder of ten rungs, each rung one change added to the last (100M tokens a rung), and the other way round, the finished 2026 recipe with that one old part put back (50M tokens). The seven rungs of the 2019 side keep its learning rate and its warmup of 8M tokens, which was too short: with 32M tokens each is 0.017 to 0.042 nats better in one seed, and the steps between them hardly move, but for the last.

the blocks of GPT-2 (2019) and gpt-pp (2026), part by part

                2019                                2026
position        a learned vector a position         queries and keys turned by position
norm            LayerNorm: gain, bias               RMSNorm, no parameters
score           q · k / √64                         12 · cos(q, k)
activation      GELU                                squared ReLU
writes          W z + b, from N(0, 0.005²)          W z, from 0
unembedding     the embedding matrix, tied          a matrix of its own
logits          as they come                        15 · tanh(logits / 15)
trained by      AdamW, cosine decay                 Muon and AdamW, flat then linear decay

validation loss after each change (8 layers, 512 wide, 100M tokens a rung)
0 the 2019 recipe  4.135
1 no biases        4.135   0.000
2 rotary           4.024   −0.110
3 RMSNorm          4.056   +0.031
4 ReLU²            4.023   −0.033
5 QK norm          4.011   −0.011
6 zero init        3.984   −0.027
7 2026 training    3.828   −0.155
8 untied           3.817   −0.011
9 soft-cap         3.814   −0.003
with a warmup of 32M tokens the 2019 rungs are 0.017 to 0.042 better
of the 0.321, the training recipe 0.155, the block’s eight changes 0.165
Muon for AdamW 0.083 and 0.086, the rest 0.074 and 0.068
old part back in the 2026 recipe (50M tokens):
  no biases −0.017, rotary +0.110, RMSNorm −0.020, ReLU² −0.001
  QK norm +0.031, zero init −0.001, untied 0.000, soft-cap −0.002
(the live figure needs JavaScript and loads about 76 kB of data)
From the block of 2019 to the block of 2026 in nine changes. Green: the parts the selected change replaces, in both blocks. Left, GPT-2; right, gpt-pp; the stream runs down the left of each, and each sublayer reads it at a tap and adds to it at a plus. Below: the validation loss of an 8-layer, 512-wide model after each change, 100M tokens a rung, a tick a seed; under it, the finished recipe with the old part put back; beside it, the selected change during training. Step through the changes or click a part. Measured on our runs.

This is all that GPT-2's paper says about its block.

Language models are unsupervised multitask learnersOpenAI 2019A. Radford, J. Wu, R. Child, D. Luan, D. Amodei and I. Sutskever

2.3 Model

We use a Transformer (Vaswani et al., 2017) based architecture for our LMs. The model largely follows the details of the OpenAI GPT model (Radford et al., 2018) with a few modifications. Layer normalization (Ba et al., 2016) was moved to the input of each sub-block, similar to a pre-activation residual network (He et al., 2016) and an additional layer normalization was added after the final self-attention block. A modified initialization which accounts for the accumulation on the residual path with model depth is used. We scale the weights of residual layers at initialization by a factor of 1/√N where N is the number of residual layers. The vocabulary is expanded to 50,257. We also increase the context size from 512 to 1024 tokens and a larger batchsize of 512 is used.

Section 2.3, the whole section
A. Radford, J. Wu, R. Child, D. Luan, D. Amodei and I. Sutskever, "Language models are unsupervised multitask learners" (OpenAI 2019), Section 2.3, the whole section.

Seven sentences, and none on the optimizer. The code released with the weights is an inference script. Its block is the pre-norm block of (5.1), and its activation is not quite GELU:

gpt-2, the code released with the modelGitHub 2019OpenAI
def gelu(x):
    return 0.5*x*(1+tf.tanh(np.sqrt(2/np.pi)*(x+0.044715*tf.pow(x, 3))))

[…]

def block(x, scope, *, past, hparams):
    with tf.variable_scope(scope):
        nx = x.shape[-1].value
        a, present = attn(norm(x, 'ln_1'), 'attn', nx, past=past, hparams=hparams)
        x = x + a
        m = mlp(norm(x, 'ln_2'), 'mlp', nx*4, hparams=hparams)
        x = x + m
        return x, present
the model file, lines 25 to 26 and 123 to 130
OpenAI, "gpt-2, the code released with the model" (GitHub 2019), the model file, lines 25 to 26 and 123 to 130.

So a GPT-2 that can be trained is a reconstruction: from this paragraph, from the 2018 paper that it "largely follows", from the inference code and, for the optimization, from the appendix of GPT-3 (Section 6). The reproductions that became the common starting point, nanoGPT and its successor, made each choice in the open and left a reason for each.A. Karpathy, nanoGPT (GitHub, 2022 to 2023), with its commit messages, and build-nanogpt with the lecture "Let's reproduce GPT-2 (124M)" (2024). The Hugging Face port of GPT-2 reads the depth factor the same way. Tying: O. Press and L. Wolf, "Using the output embedding to improve language models", EACL 2017 (arXiv 1608.05859).

  GPT-2 paper code released with it reproductions, 2022 to 2024 reason given
normalization at the input of each sub-block, one more after the last the same the same a skip with nothing on it carries the gradient from the loss to the tokens
depth factor \(1/\sqrt N\) on residual layers absent: 0.02 everywhere, 0.01 for positions \(0.02/\sqrt{2L}\) on the two writers Proposition 5.5.1; one commit records that it "makes a ton of difference"
activation not stated (GPT: GELU) the tanh approximation exact GELU in one, the tanh form in the other the tanh form is kept only where the released weights must load exactly
biases not stated in every linear map and normalization optional, off for training in one "code runs faster, identical results"
unembedding not stated the embedding, transposed the same 38.6M fewer parameters of 124.4M; similar tokens are similar at both ends
vocabulary 50,257 50,257 padded to 50,304 a multiple of 64 sped up the last matrix product, "about 25% faster" in 2023 and about 4% by mid 2024
The block of 2019 as three sources give it: the GPT-2 paper, the inference code released with the weights, and the two reproductions. "GPT" is the paper of 2018; quoted words are from commit messages of the first reproduction.

On the activation, an author of GELU settled the matter in 2020, in a comment of two sentences:

I used the tanh approximation simply because the error function erf was slow in tensorflow some years ago. If the exact version is fast enough now and does not have numerical issues, I do not see a reason to use an inexact version.

—D. Hendrycks, comment on issue 39853 of the PyTorch repository (2020).The whole comment, of 22 June 2020.

From there to the block of 2026 every component was replaced, most of them within two months of a public speedrun in which a change survives only if it beats the previous record on the same hardware.K. Jordan and contributors, modded-nanogpt (GitHub, 2024 to 2026): a GPT-2 of 124M parameters to a validation loss of 3.28 in 45 minutes in May 2024 and in 0.665 minutes in August 2026. nanochat (GitHub, 2025 to 2026), by the author of nanoGPT, keeps the same block, and its experiment log records what it rejected. On biases, PaLM (2022): "We found this to result in increased training stability for large models."

change 2019 2026 since made, on the ladder old part put back
1 biases in every linear map and norm none 2022 0.000 −0.017
2 position a learned vector per position, added to the stream queries and keys rotated by their position 2021 −0.110 +0.110
3 normalization layer norm with a gain RMS norm without parameters 2019; gain dropped 2024 +0.031 −0.020
4 activation GELU squared ReLU 2021; adopted 2024 −0.033 −0.001
5 attention score \(q \cdot k / \sqrt{d_{\mathrm{head}}}\) \(12 \cos(q, k)\) 2020; adopted 2024 −0.011 +0.031
6 writers at initialization 0.02 scaled by \(1/\sqrt{2L}\), the other matrices at 0.02 zero, the other matrices of a block uniform in \(\pm 1/\sqrt d\) 2019; adopted 2024 −0.027 −0.001
7 training recipe AdamW on every weight, cosine decay Muon on the matrices of the blocks, AdamW on the rest, with its own rates and schedule 2024 −0.155 not run as one change
8 embedding, unembedding one matrix two, and a norm on the embedding 2024 −0.011 0.000
9 logits unbounded \(15 \tanh(z/15)\) 2024 −0.003 −0.002
The nine changes from the block of 2019 to the block of 2026, in the order of the ladder, each with the change in held-out loss (nats a token; 8 layers, 512 wide; negative is better) when it is made on the way up, at 100M tokens a rung and the mean of two seeds, and when the old part alone is put back into the finished recipe, at 50M tokens (the mean of two seeds for changes 1 to 5, one seed for changes 6, 8 and 9). A step of the ladder under 0.016 is not established, nor a put-back under 0.009 (Section 5.2). Change 7 replaces the whole training recipe, which is the subject of Section 6; the bias of the norm goes with change 1.

In the code. The two columns of the table are one dictionary: the 2026 recipe is the set of defaults of the model's configuration, and the 2019 recipe is the list of what differs. It is RECIPES in gpt-pp's config.py.

# The 2026 recipe is the set of defaults of ModelConfig; the 2019 recipe is written as its differences from it.
RECIPES: dict[str, dict] = {
    "gptpp": dict(),
    "gpt2-2019": dict(pos="learned", norm="layernorm", norm_placement="pre", embed_norm=False, act="gelu",
                      qk_norm=False, bias=True, tie_embeddings=True, logit_softcap=0.0, init="gpt2"),
}

Ten entries and nine differences, since norm_placement="pre" restates the default. Changes 1 to 6 of the table set bias, pos, norm, act, qk_norm and init to their defaults, one a rung; change 8 sets tie_embeddings and embed_norm together, change 9 logit_softcap, and change 7 is not in the dictionary, since it is the training configuration (Section 6). Three entries hold more than their row of the table says: init is a whole scheme (Section 5.5); qk_norm normalizes queries and keys to unit root mean square and puts the gain on the query, which gives the score of the table by another route (Section 3); and gpt2-2019 is the design of 2019 as the reproductions read it, not the released model, whose weights do not load into it (Section 5.2).

On the ladder the nine changes take the held-out loss from 4.135 to 3.814, 0.321 nats in all. The training recipe of 2026 is 0.155 of it, 48%; the eight changes of the block are the other 52%, and rotary positions alone 0.110. Two changes of nine carry 83% of the way.

Known issue. The larger of the two is not Muon alone, and it is not one change: change 7 moves 14 training settings together (the rule for the matrices, the rates of the tables, the momentum coefficient, the decay, the clipping, the warmup, the schedule). With AdamW kept on the matrices under the new recipe, on a warmup of 8M tokens and a clipping of its own, without which the comparison handicaps it (Section 6), Muon's part of the step is 0.083 and 0.086 in the two seeds and the rest of the recipe's (the tables' rates, the decay, the schedule) 0.074 and 0.068. With the longer warmup on the 2019 side the whole way is 0.286 in place of one seed's 0.328 and this step 0.140 in place of 0.157, the rest of the recipe's part 0.057.

Put back one at a time into the finished recipe, two old parts clearly cost, in both seeds: the learned position table (0.110) and unnormalized queries and keys (0.031). And three parts of the block of 2026 are not better than what they replaced, at this size. GELU is level with the squared ReLU (Section 5.3). With a learned gain in its norm the recipe is 0.020 better, centered or not, and with biases 0.017 better (0.021 and 0.013): each past the yardstick of Section 5.2 in both seeds, and at three times the rate for gains and biases still 0.013 and 0.023 in one. On the biases the ladder says nothing: dropping them moved the loss by 0.010 one way in one seed and by 0.010 the other way in the other. The cap on the logits does nothing either way; the speedrun's own author counts it among the methods that "impose additional structure" and are "unlikely to scale". Change 8 adds 16.8M parameters and gives the embedding a learning rate of its own, 0.3674 where the one tied table of the rung before trains at the rate of an unembedding, 0.0098 (Section 6); it gained 0.012 and 0.010 on the ladder in the two seeds, and its put-back, which changes four things at once (the tying, the norm on the embedding, the table's rate, its initialization), loses nothing.

The two columns disagree on three parts, and the disagreement is the lesson. The squared ReLU gained 0.033 on the way up and is worth nothing over GELU at the end. Writers at zero, with the rest of their scheme, gained 0.027, or 0.008 in one seed when the rungs have the longer warmup, and are worth nothing at the end. The normalization of queries and keys gained 0.011, under the ladder's yardstick, and costs 0.031 to take out. A change made on the way up is measured in the company of the old training recipe and of what the earlier rungs left behind (the square came right after the norm had lost its gain); a change put back is measured in the finished recipe. Neither number is the effect of the part: a part has effects only in a recipe.The ladder and the put-back experiments ask different questions. A useful change can become redundant after another change, so the measured gains are not independent quantities that can be carried into any model and added up. The recipe of 2026 is the series' own, and by these measurements two of its parts do not earn their place in an 8-layer model, the norm without a gain and the maps without biases, which measure worse than what they replaced in both seeds; a third, the square, gained on the way up and is no better than GELU at the end. Section 11 says what would be changed.

Example 5.6.1 (gpt-pico in both recipes). The setup: two models of gpt-pico's shape on the same 540M tokens of TinyStories, one pass, one seed each. One has the block of 2019 and its training, AdamW with the learning rate retuned for this size (0.006; of four rates tried in short probes the largest, 0.012, was 0.012 nats better, so its best rate is not bracketed); the other is gpt-pico, with the block of 2026 and the training recipe of Section 6.

model parameters in attention and MLPs held-out loss, nats a token bits per byte
gpt-pico's shape, 2019 recipe 1,746,944 1,186,560 1.6329 0.5849
gpt-pico, the smallest gpt-pp 2,228,224 1,179,648 1.5398 0.5515
The two recipes at the size that runs in the page, scored on 1,048,576 held-out tokens of TinyStories. One seed each.

The trunks are the same to within biases and gains; the model of 2026 is larger by its second vocabulary matrix. Its loss is lower by 0.093 nats a token. This is the whole recipe at the smallest size, block and training together, on one seed each; Section 7 repeats the comparison at the shape of gpt-pp. ◁

The large open models agree with the small recipes on the frame and differ in one component. Of 24 released since January 2025, 22 use RMS norm, 21 rotary positions in at least some layers, and 18 untied embeddings; query-key normalization is visible in six, introduced in one of them "to ensure stable training".Yang, Li, Yang, Zhang and others, "Qwen3 technical report", 2025 (arXiv 2505.09388), Section 2. The counts are read from the configuration files and reports of 24 open models released from January 2025 on; a feature counts only where it is visible. The placement is that of 2019. The difference is the activation: 23 of the 24 gate their MLP where the small recipes square it, two members of one family by Proposition 5.3.3.

What is being reopened in 2026 is the oldest component. An abstract of March 2026 opens with the complaint:

Residual connections with PreNorm are standard in modern LLMs, yet they accumulate all layer outputs with fixed unit weights. This uniform aggregation causes uncontrolled hidden-state growth with depth, progressively diluting each layer's contribution. We propose Attention Residuals (AttnRes), which replaces this fixed accumulation with softmax attention over preceding layer outputs, allowing each layer to selectively aggregate earlier representations with learned, input-dependent weights. To address the memory and communication overhead of attending over all preceding layer outputs for large-scale model training, we introduce Block AttnRes, which partitions layers into blocks and attends over block-level representations, reducing the memory footprint while preserving most of the gains of full AttnRes. Combined with cache-based pipeline communication and a two-phase computation strategy, Block AttnRes becomes a practical drop-in replacement for standard residual connections with minimal overhead. Scaling law experiments confirm that the improvement is consistent across model sizes, and ablations validate the benefit of content-dependent depth-wise selection. We further integrate AttnRes into the Kimi Linear architecture (48B total / 3B activated parameters) and pre-train on 1.4T tokens, where AttnRes mitigates PreNorm dilution, yielding more uniform output magnitudes and gradient distribution across depth, and improves downstream performance across all evaluated tasks.

—Kimi Team, "Attention residuals" (2026).arXiv 2603.15031, abstract.

That is the louder stream of Section 5.2, now treated as a fault of the plus sign: three of the 24 models replace the identity skip, with several streams mixed by a constrained matrix or with attention over the outputs of earlier layers. Whether a decade of Proposition 5.1.2 gives way is open. The other half of the 2026 recipe, its training, with an optimizer that takes these matrices as matrices, is Section 6.

← Back to all posts