7. Guidance
Conditioning and latent spaces Draft
Conditional scores and classifier guidance
Often we want more than a random sample: an image of a particular class, an image matching a caption, or a reconstruction from a damaged observation. The condition \(c\) enters the denoiser as another input, \(D_\theta(x, t, c)\). The earlier mathematics then applies to the conditional law \(p_0(\cdot \mid c)\).
Guidance goes further. It strengthens the condition's influence during sampling, often making outputs more recognizably match the request at the cost of diversity. We will derive the guided score and distinguish the density it describes at one noise level from the distribution the sampler actually produces. We will then look at the architectures and latent spaces that implement these ideas.
Conditioning changes what counts as a plausible source. An ambiguous noisy patch might have several explanations in the full dataset but fewer within a chosen class. The conditional denoiser averages over that smaller, reweighted set of explanations; the right panel of the first figure of Section 7.2 draws that reweighting at one noisy point. The classifier term below measures how a small change in the noisy observation changes the evidence for the requested class.
Proposition 7.1.1 (Bayes' rule for scores). Let \(p_t(x \mid c)\) be the noised law of the class \(c\) and \(p_t(x) = \sum_c p(c)\, p_t(x \mid c)\) the noised law of all the data, and let \(p_t(c \mid x)\) be the posterior of the class given the noised point. Then
\[\nabla_x \log p_t(x \mid c) \;=\; \nabla_x \log p_t(x) + \nabla_x \log p_t(c \mid x) . \tag{7.1}\]Proof. \(p_t(x \mid c) = p_t(c \mid x)\, p_t(x)/p(c)\) by Bayes' rule, and \(p(c)\) does not depend on \(x\). ∎
The second term in (7.1) is the gradient of a classifier's log probability on noisy inputs. Dhariwal and Nichol train such a classifier and use it to steer a diffusion model. Added once to an unconditional score, the exact classifier gradient gives the conditional score by Bayes' rule. Starting from a conditional model, their guided score has the form \(s_t(x \mid c) + w\, \nabla_x \log p_\phi(c \mid x_t)\), with \(w > 0\) adding emphasis to the class.P. Dhariwal and A. Nichol, "Diffusion models beat GANs on image synthesis", NeurIPS 2021 (arXiv 2105.05233), Section 4: classifier guidance, with the scaled gradient; "We achieve an FID of 2.97 on ImageNet 128×128, 4.59 on ImageNet 256×256, and 7.72 on ImageNet 512×512, and we match BigGAN-deep even with as few as 25 forward passes per sample". The 4.59 is their ADM-G model with 250 sampling steps.
Increasing that emphasis tends to produce more characteristic, less varied samples. It also requires a second network trained across noise levels. The classifier gradient follows whatever raises the class probability most quickly, which need not coincide with a direction toward realistic data.
Classifier-free guidance
Ho and Salimans obtain the guidance direction from the denoiser itself. During training, they randomly replace the condition with a null token \(\varnothing\) for some examples. The same network therefore learns a conditional denoiser, \(D_\theta(x, t, c)\), and an unconditional one, \(D_\theta(x, t, \varnothing)\). Their difference supplies a direction in which the condition changes the prediction.
Subtracting the unconditional prediction isolates what the condition contributes at the current point. Adding that difference once gives the conditional prediction; adding it more than once extrapolates beyond it. For example, \(w=2\) gives twice the conditional score minus the unconditional score. This is why a larger guidance scale changes the sampling rule rather than merely supplying the condition more accurately.
left R3 at sigma = 0.5 with the probe x = (-0.20, 0.10) between the moons: the 24 centres with
their posterior weights, 15.8 of them effective, 9.6 within the upper moon, which holds
0.534 of the posterior; the unconditional estimate D_u = (-0.17, 0.11), the class estimate
D_c = (-0.13, 0.40), and the chord D_u + w (D_c - D_u) through them, ticked at w = 0, 1,
2, 3: the guided estimate at w = 2 sits at (-0.09, 0.68), 0.061 from a centre, the w = 3
tick 0.343 off the arc
right the 24 responsibilities r_k(x) over all the data, the class's 12 renormalized over them
strip log p_0(D(w) | c) against w: -1.39 at w = 1, -0.06 at w = 2, -4.03 at w = 3, against
0.09 at a centre
(the live figure needs JavaScript)
Definition 7.2.1 (classifier-free guidance). With \(s_t(x \mid c)\) and \(s_t(x)\) the conditional and unconditional scores and a guidance scale \(w \ge 1\), the guided score is
\[\tilde s_t(x) \;=\; (1 - w)\, s_t(x) + w\, s_t(x \mid c) \;=\; s_t(x) + w\,\bigl(s_t(x \mid c) - s_t(x)\bigr), \tag{7.2}\]and guided sampling runs any sampler of Sections 3 to 5 with \(\tilde s_t\) in place of the score. The scale \(w = 1\) is conditional sampling. Ho and Salimans write the same in the noise units as \(\tilde\varepsilon = (1 + w')\varepsilon_c - w'\varepsilon_\varnothing\), whose \(w'\) is \(w - 1\) here.J. Ho and T. Salimans, "Classifier-free diffusion guidance", NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications, arXiv 2207.12598 (26 July 2022), Equation 6 and Algorithm 2: \(\tilde\varepsilon_\theta(z_\lambda, c) = (1 + w)\varepsilon_\theta(z_\lambda, c) - w\varepsilon_\theta(z_\lambda)\), with the unconditional model "obtained by a null token" and \(w\) the "guidance strength". The scale \(w\) of this series, in which \(w = 1\) is the conditional score, is the convention of most later papers and of the figures below.
Proposition 7.2.2 (what the guided score is a score of). At each fixed \(t\), \(\tilde s_t\) is the score of the law
\[\tilde p_t(x) \;\propto\; p_t(x \mid c)^{\,w}\, p_t(x)^{\,1 - w} \;\propto\; p_t(x \mid c)\, p_t(c \mid x)^{\,w - 1},\]so that classifier-free guidance is classifier guidance with the implicit classifier \(p_t(c \mid x) = p_t(x \mid c)\, p(c)/p_t(x)\) and the weight \(w - 1\).
Proof. Take the gradient of the logarithm of the first expression and compare with (7.2); the second expression follows from (7.1). ∎
At each fixed noise level, the guided field is a score of a well-defined density. That does not yet tell us the sampler's endpoint. To conclude that it draws from \(\tilde p_0 \propto p_0(x \mid c)^w p_0(x)^{1 - w}\), we would need the whole family of guided densities to evolve consistently under the forward noising process. In general it does not.
The consistency issue is about the order of two operations. We could sharpen the clean distribution and then add noise, or add noise first and sharpen each resulting density. These operations generally give different answers. The reverse-process theorem requires the first kind of family: all its densities must come from one common forward evolution. A valid score at each separate time is not sufficient.
panel two Gaussian classes at -1 and +1 of width s = 0.35, guided toward A = -1 at w = 2, at
sigma = 1: the sharpened clean law noised to sigma (ink), mean -1.003; the law the guided
score belongs to there (green), mean -1.321, 0.318 from the first in W_1; the sampler's
own marginal at its node sigma = 1.01 (dashed), mean -1.608; the two noised classes as
hairlines
strip the three means against sigma: the first flat at -1.003, the second falling to -1.819 at
sigma = 3 toward the proof's limit -2, the sampler's between; the sampler's endpoint is
0.214 from the sharpened clean law
(the live figure needs JavaScript)
Proposition 7.2.3 (the guided laws are not a diffusion of anything). For \(w \ne 0, 1\) and data with more than one class, the family \(\{\tilde p_t\}_{t \in (0, 1]}\) is in general not the family of noised laws of any data law. Consequently the reverse process run with \(\tilde s_t\) does not have the marginals \(\tilde p_t\), its endpoint is not \(\tilde p_0\), and the law of its samples depends on the sampler and not only on \(w\).
Proof. Theorem 4.2.2 gives a process with marginals \(q_t\) from a family of scores only when the \(q_t\) are the forward marginals of \(q_0\); a reverse process run with the scores of some other family has different marginals, so the second statement follows from the first. For the first, take a variance-exploding process, under which the forward kernel preserves means, and two Gaussian classes \(p_0(\cdot \mid A) = \mathcal N(m_A, s^2 I)\), \(p_0(\cdot \mid B) = \mathcal N(m_B, s^2 I)\), \(m_A \ne m_B\), with equal weights, and guide toward \(c = A\) with \(w = 2\), so that \(\tilde p_t \propto p_t(x \mid A)^2/p_t(x)\). If \(\tilde p_t\) were the noised family of \(\tilde p_0\), its mean would be the mean of \(\tilde p_0\) for every \(t\), by the mean-preserving property of the kernel.
But as \(t \to \infty\) the posterior flattens: with \(v_t = s^2 + \sigma_t^2\) and \(\bar m = (m_A + m_B)/2\), \(p_t(A \mid x) = \tfrac12 + (x - \bar m) \cdot (m_A - m_B)/(4 v_t) + O(v_t^{-2})\), so the mean of \(\tilde p_t = p_t(x \mid A)\, p_t(A \mid x)/\int p_t(y \mid A)\, p_t(A \mid y)\, dy\), computed under \(x \sim \mathcal N(m_A, v_t I)\), tends to \(m_A + \tfrac12 (m_A - m_B)\): the slope of the posterior falls like \(1/v_t\) while the covariance of \(x\) with \(x \cdot (m_A - m_B)\) grows like \(v_t\), and the tilt moves the mean by a fixed amount. The mean of \(\tilde p_0 \propto p_0(x \mid A)\, p_0(A \mid x)\) is displaced from \(m_A\) away from \(m_B\), since \(p_0(A \mid x)\) is increasing along \(m_A - m_B\), by an amount that vanishes as \(s \to 0\), when \(p_0(A \mid x)\) is already \(1\) near \(m_A\). The two means differ, so the family is not a noised family. The same argument applies to every \(w \notin \{0, 1\}\). ∎
Guidance modifies the drift of the sampling process. Its output law depends on how that modified drift is used, including the choice of sampler. The two-moon example makes the distinction visible because we can evaluate both the guided density and the distribution of the sampler's outputs.
left 1,024 samples of the guided process on the two moons R3 (256 drawn), toward the chosen
moon, by Heun in 32 steps from fixed noise, for the guidance scale w; hollow: on the wrong
moon; one sample followed, its guided trajectory green, the same noise at w = 1 dashed
right the law the guided score belongs to at sigma, p_t(.|c)^w p_t^{1-w}, normalized on the box,
or the sampler's own marginal at that sigma, which differs from it for w != 1
strip the mean of the guided law against sigma for w and for w = 1, which does not move
(the live figure needs JavaScript)
Example 7.2.4 (guidance on the two moons). At \(w = 1\) the sampler draws from the upper moon: the 1,024 samples have a spread of \(0.741\) about their centroid and a mean log density of \(-0.511\) under the class law, and \(5\) of them land on the lower moon, having started from noise on the wrong side of the gap. At \(w = 3\), the figure's default, the samples move toward the far side of the moon, away from the other class: the spread falls to \(0.597\), none lands on the wrong moon, and the mean log density rises to \(-0.358\), every sample being more typical of its class. The right panel shows why: the law the guided score belongs to is the class law multiplied by the posterior of the class raised to the power \(w - 1\), which is small where the other moon is near.
The price is diversity, measured by the spread and by the nearest-neighbour distance between samples, which shrinks as \(w\) grows; at \(w = 6\) the samples crowd the outer rim of the moon. Restricting the guidance to the interval \(\sigma \in [0.1, 1]\), the figure's second mode, keeps most of the typicality and returns some of the spread, which is the finding of Kynkäänniemi et al. on ImageNet. The view 'the sampler's marginal' measures Proposition 7.2.3: the sliced Wasserstein-1 distance between the sampler's states at \(\sigma = 0.08\) and draws from the law the guided score belongs to there is \(0.077\) at \(w = 1\), the sampler's own error, which comes from its start at \(\mathcal N(0, 9I)\) rather than at the class law noised to \(\sigma = 3\) and not from its 32 steps, and \(0.207\) at \(w = 3\); and the strip shows why: at \(w = 3\) the mean of the guided law moves from \((-0.51, 0.34)\) at \(\sigma = 0.05\) to \((-1.11, 0.75)\) at \(\sigma = 1.2\), while a noised family keeps its mean.
Example 7.2.8 below, on the trained model, writes the same score in the noise units of Ho and Salimans, \((1 + w')\,\varepsilon_c - w'\,\varepsilon_\varnothing\) with \(w' = w - 1\), and its sweep \(w' \in \{-2, -1, -0.5, 0, 0.5, 1, 2, 4, 8\}\) is \(w \in \{-1, 0, 0.5, 1, 1.5, 2, 3, 5, 9\}\) here, of which the slider reaches \(w \le 6\): its first two rows are anti-guided and unconditional samples, which the slider reaches at \(w \le 0\). ◁
Remark 7.2.5 (what has been found about where and how to guide). Kynkäänniemi et al. measured the effect of guidance by noise level and found it "clearly harmful toward the beginning of the chain (high noise levels), largely unnecessary toward the end (low noise levels), and only beneficial in the middle"; applying it only in that interval took the record FID on ImageNet at \(512 \times 512\) from \(1.81\) to \(1.40\) with the same model. Karras et al. then replaced the unconditional model in (7.2) by a smaller or less-trained version of the conditional model itself, so that the extrapolation is away from the errors of a weaker model rather than away from the unconditional law, and reached FIDs of \(1.01\) at \(64 \times 64\) and \(1.25\) at \(512 \times 512\).T. Kynkäänniemi, M. Aittala, T. Karras, S. Laine, T. Aila and J. Lehtinen, "Applying guidance in a limited interval improves sample and distribution quality in diffusion models", NeurIPS 2024 (arXiv 2404.07724): the quotation, and "This limited guidance interval improves the record FID in ImageNet-512 significantly, from 1.81 to 1.40"; the \(1.81\) is the EDM2-XXL model of Section 7.3 with guidance. T. Karras, M. Aittala, T. Kynkäänniemi, J. Lehtinen, T. Aila and S. Laine, "Guiding a diffusion model with a bad version of itself", NeurIPS 2024 (arXiv 2406.02507): "guiding generation using a smaller, less-trained version of the model itself rather than an unconditional model … setting record FIDs of 1.01 for 64x64 and 1.25 for 512x512". Both findings are consistent with Proposition 7.2.3: since guidance is a sampler and not a target, where along the path it is applied, and against what, are free parameters, and their effect is empirical.
Algorithm 7.2.6 GUIDED-HEUN
(classifier-free guidance on the EDM clock)
Picture: each evaluation of the score is two network
calls, conditional and unconditional, combined by
the scale w before the step is taken.
x --+--> D_theta(x, sigma, c) --.
| +--> (1-w) s_u + w s_c --> step
'--> D_theta(x, sigma, null) --'
Input a network D_theta(x, sigma, c) trained with
condition dropout; a class c; a scale w >= 1;
the grid of Definition 5.3.1; optionally an
interval [a, b] of noise levels for guidance.
Output a guided sample.
1. x ~ N(0, sigma_0^2 I)
2. for i = 0, ..., N-1:
3. w_i <- w if sigma_i in [a, b] else 1
4. s <- (1 - w_i) s_theta(x, sigma_i, null)
+ w_i s_theta(x, sigma_i, c)
5. take the Heun step of Algorithm 5.3.2 with
d = -sigma_i s, re-evaluating s at the
predicted point in the same way
6. return x
Remark 7.2.7 (confusion: guidance and temperature). Raising \(p_0(x \mid c)\) to a power, a temperature, sharpens the class law toward its modes and keeps it where it is; guidance multiplies the class law by the posterior of the class, which moves it away from the other classes. The right panel of the two-moons figure above makes the difference visible: at \(w = 3\) the guided law is not the moon made thinner, it is the moon made thinner and displaced toward its outer edge. The second figure of Section 7.2 draws the two sharpenings side by side in one dimension. And by Proposition 7.2.3 neither description is the law of the samples; it is the law whose score the sampler follows at each noise level, and the samples are what the reverse process makes of a sequence of such scores.
Example 7.2.8 (guidance on the class-conditional model). We use the class-conditional model, the small U-Net of Section 11 at the widths \((64, 64, 128)\) with 2,547,457 parameters, trained on FashionMNIST at \(28 \times 28\) along DDPM's chain of 1,000 steps with the context dropped with probability \(0.1\), and sample it by Algorithm 3.3.2 along that chain under the combination \((1 + w')\,\varepsilon_c - w'\,\varepsilon_\varnothing\) at \(w' \in \{-2, -1, -0.5, 0, 0.5, 1, 2, 4, 8\}\), the series' \(w = 1 + w'\), for the ten classes, from the same noise at every scale. On 510 samples per scale, we measure how often a classifier trained on the data assigns a sample its intended class, the within-class distance from a sample to its nearest neighbour, and the Fréchet distance FD_clf to held-out images in that classifier's features.
The classifier assigns the samples their intended class \(0.757\) of the time at \(w = 1\), \(0.908\) at \(w = 3\) and \(0.884\) at \(w = 9\), while the within-class distance between a sample and its nearest neighbour falls from \(0.340\) to \(0.323\), and the Fréchet distance to held-out images falls from \(88.8\) to \(77.3\) and is smallest, \(70.3\), at \(w = 2\), then rises to \(101.0\) at \(w = 5\) and \(143.6\) at \(w = 9\): typicality increases as diversity falls, as on the two moons. Beyond \(w = 3\) the accuracy no longer rises while the distance to the data grows.
Applying guidance only between indices 100 and 739 of the chain lowers the Fréchet distance at every scale from \(w = 2\) on, to \(66.6\) at \(w = 2\), the smallest of the sweep, to \(74.8\) at \(w = 3\) and to \(88.2\) in place of \(101.0\) at \(w = 5\), at an accuracy of \(0.896\) at \(w = 3\) and \(0.908\) at \(w = 5\), and leaves the diversity at \(0.323\) at \(w = 3\), where guidance at every step had it. This resembles Kynkäänniemi et al.'s result: a better distribution at nearly the same typicality, though this experiment recovers no diversity.
The film shows where guidance acts: along one chain the clean estimates of the guided and the unguided runs are \(2.164\) apart per pixel at the first step, index \(999\), where the estimate \(\hat x_0 = (x_t - \sigma_t\hat\varepsilon)/\alpha_t\) divides by \(\alpha = 0.006\) and magnifies a small guidance direction, and they are never so far apart again: the distance falls to \(0.273\) at index \(7\). The direction itself, \(\|\varepsilon_c - \varepsilon_\varnothing\|\) per pixel along the guided chain, is \(0.008\) at the start, largest at index \(143\), \(0.031\), and \(0.025\) at the end: small where the chain begins, which the interval leaves out, and largest inside the interval. ◁
left the guidance grid: samples of the class-conditional model, (64, 64, 128), trained with the
context dropped with probability 0.1, by the ancestral sampler along DDPM's chain of 1,000
steps, at w' = -2 to 8 (rows) for the ten classes (columns), one of four seeds, the same
noise for every w'; under it, the change map: the chosen row minus the w' = 0 row
right class accuracy, Frechet distance FD_clf and within-class diversity against w' on 510 samples
per w', with guidance at every step (solid) and only at indices 100 to 739 (dashed); accuracy
rises from 0.76 at w' = 0 to 0.91 at 2 and is 0.88 at 8, FD_clf falls from 89 to 77 (smallest,
70, at w' = 1; 144 at 8), diversity from 0.34 to 0.32
film two guided chains of class 3 at w' = 0 and w' = 2 from the same noise, every eighth state,
their clean estimates, the guidance direction eps_c - eps_u at the guided chain's state as a
change map, its size per pixel along the chain (largest at index 143) and the distance
between the clean estimates
(the live figure needs JavaScript and loads 1.54 MB of data)
Architectures: from the U-Net to the diffusion transformer
The equations specify what the network should predict, but leave much of its architecture open. A denoiser maps \((x, t, c) \mapsto D_\theta\). Since 2020, work on that map has changed its backbone, the way it receives the condition, and the representation in which it predicts the answer.
Time tells the network how to interpret its input. A small feature in an almost clean image is evidence to preserve; the same feature at high noise may be a fluctuation to average away. The architecture must also combine scales: local pixels constrain edges and texture, while distant regions help determine the object's shape. The coarse and fine paths of a U-Net, and the interactions supplied by attention, address those different sources of evidence.
DDPM uses a U-Net: a convolutional encoder–decoder with skip connections between resolutions and a time embedding added to each block. Dhariwal and Nichol's ADM adds attention at several resolutions and adaptive group normalization.
Peebles and Xie's diffusion transformer replaces the convolutional backbone with a transformer over patches of a latent image. Time and class enter through adaptive layer normalization with zero-initialized scales. Their experiments show improving FID as model compute increases; the largest model reaches an FID of \(2.27\) on ImageNet at \(256 \times 256\) with classifier-free guidance.W. Peebles and S. Xie, "Scalable diffusion models with transformers", ICCV 2023 (arXiv 2212.09748): DiT-XL/2 at \(118.6\) Gflops and \(675\)M parameters, FID \(2.27\) at \(256 \times 256\) with guidance scale \(1.5\), and \(3.04\) at \(512 \times 512\). The architecture is agnostic to the schedule: N. Ma, M. Goldstein, M. S. Albergo, N. M. Boffi, E. Vanden-Eijnden and S. Xie, "SiT: exploring flow and diffusion-based generative models with scalable interpolant transformers", ECCV 2024 (arXiv 2401.08740), keep it and change the path to the linear interpolant of Section 6, reaching \(2.06\) at \(256 \times 256\) and \(2.62\) at \(512 \times 512\) "with the same structure, parameters and GFLOPs".
Text conditioning introduces a sequence of tokens. Latent diffusion models use cross-attention from image tokens to text tokens. Esser et al.'s MM-DiT instead applies joint attention to the concatenated image and text tokens, with separate weights for the two modalities, and scales to eight billion parameters.Esser et al. (2024, cited in Section 6.3): "a novel transformer-based architecture for text-to-image generation that uses separate weights for the two modalities and enables a bidirectional flow of information between image and text tokens", models "up to a model size of 8B parameters and \(5 \times 10^{22}\) training FLOPs". Black Forest Labs' FLUX.1 models, announced on 1 August 2024, are "based on a hybrid architecture of multimodal and parallel diffusion transformer blocks and scaled to 12B parameters", trained "by building on flow matching" (bfl.ai/announcing-black-forest-labs, read 24 October 2026).
Example 7.3.1 (the small U-Net behind the examples). The browser-sized class-conditional model, \((16, 32, 64)\), trained on FashionMNIST at \(28 \times 28\) along DDPM's chain of 1,000 steps with the context dropped with probability \(0.1\), runs in the figure below from weights the page loads when you press run, and draws two samples of a chosen class by DDIM in \(K = 8\) steps from fixed draws of noise, with the guidance direction \(\varepsilon_c - \varepsilon_\varnothing\), its size per pixel and the network's activations shown at every step.
Both it and the \((64, 64, 128)\) model of Example 7.2.8 are U-Nets of the kind above at a small scale: a residual input block of two 3 × 3 convolutions with GroupNorm and GELU, two down blocks that convolve twice and pool by rearranging each 2 × 2 cell into channels, a dense bottleneck, two up blocks that transpose-convolve and convolve four times, and a last skip from the input block to the output convolution. Time enters as a sinusoidal embedding of dimension \(8\) passed through a two-layer perceptron and added to the up path, and the class as a one-hot vector, multiplied by a mask that drops it with probability \(0.1\), passed through the same kind of perceptron and multiplied into the up path. At the widths \((64, 64, 128)\) the network has \(2{,}547{,}457\) parameters; at \((16, 32, 64)\), \(583{,}169\) parameters.
Its time embedding varies less than its size might suggest: the sinusoidal embedding's four frequencies are \(1\), \(0.046\), \(0.0022\) and \(0.0001\) on \(t/T \in [0, 1]\), so six of its eight coordinates are constant to within \(0.046\) and the network reads the noise level from two numbers. Section 11 describes the network operation by operation and gives the training recipe; the page's own evaluation of it, in JavaScript, agrees with PyTorch's on the same inputs to within \(2.2 \times 10^{-3}\) in every coordinate, with a root-mean-square error of \(4 \times 10^{-4}\). ◁
the browser-sized class-conditional model, (16, 32, 64), 583,169 parameters, trained on DDPM's chain of 1,000 steps with the context dropped with probability 0.1, whose weights the page loads when you press run: two guided samples (or four) by DDIM in K = 8 steps (or 10 or 20), in a Web Worker, with their clean estimates and the guidance direction eps_c - eps_u as a change map at every step; under them, for one sample, eight channels of four layers of the network, each over its own range, its eight time features, and the size of eps_c - eps_u per pixel against the step (the live figure needs JavaScript; its weights, about 1.2 MB, load when you press run)
More recent work changes the representations learned inside the model. Yu et al. align a diffusion transformer's hidden states with features from a pretrained self-supervised encoder. They report reaching the same FID more than seventeen times faster and reaching \(1.42\) with guidance restricted to an interval. Wang et al. find that alignment helps early but hurts later, and stop it partway through training for a further speedup.
Other work changes the space in which generation happens. Zheng, Ma, Tong and Xie replace the latent space with the representation space of a frozen encoder and train a decoder, reaching \(1.51\) without guidance and \(1.13\) with it. Li and He return to pixels, using large patches and predicting the clean image rather than the noise. They report \(1.82\) and \(1.78\) for \(256\) and \(512\) pixels without a tokenizer, pretrained encoder, or auxiliary loss.S. Yu, S. Kwak, H. Jang et al., "Representation alignment for generation: training diffusion transformers is easier than you think", ICLR 2025 (arXiv 2410.06940): "speed up SiT training by over 17.5×, matching the performance (without classifier-free guidance) of a SiT-XL model trained for 7M steps in less than 400K steps", and "FID=1.42 using classifier-free guidance with the guidance interval", \(1.80\) with plain guidance. Z. Wang, W. Zhao, Y. Zhou et al., "REPA works until it doesn't: early-stopped, holistic alignment supercharges diffusion training", NeurIPS 2025 (arXiv 2505.16792): "it reaches the vanilla SiT-XL/2 baseline FID in 50 epochs and matches REPA's best FID in 500 epochs, amounting to a 28X reduction in optimization steps". B. Zheng, N. Ma, S. Tong and S. Xie, "Diffusion transformers with representation autoencoders", ICLR 2026 (arXiv 2510.11690): "1.51 FID at 256x256 (no guidance) and 1.13 at both 256x256 and 512x512 (with guidance)". T. Li and K. He, "Back to basics: let denoising generative models denoise", CVPR 2026 (arXiv 2511.13720): JiT-G/16 FID \(1.82\) at \(256\) and JiT-G/32 FID \(1.78\) at \(512\), "using no tokenizer, no pre-training, and no extra loss".
| date | model | backbone, space | FID, ImageNet 256 × 256 | note |
|---|---|---|---|---|
| 2021 | ADM-G, Dhariwal and Nichol | U-Net, pixels | 4.59 | classifier guidance, 250 steps |
| 2022 | LDM-4-G, Rombach et al. | U-Net, latents at \(f = 4\) | 3.60 | classifier-free guidance 1.5, 250 DDIM steps |
| 2022 | DiT-XL/2, Peebles and Xie | transformer, latents | 2.27 | guidance 1.5; 675M parameters |
| 2024 | SiT-XL/2, Ma et al. | the same, linear path | 2.06 | the schedule changed, nothing else |
| 2024 | SiT-XL/2 + REPA, Yu et al. | the same, aligned to DINOv2 | 1.42 | guidance in an interval |
| 2025 | RAE, Zheng et al. | transformer, representation latents | 1.13 | 1.51 without guidance |
| 2025 | JiT-G/16, Li and He | transformer, pixels, patch 16 | 1.82 | no tokenizer; 1.78 at 512 × 512 |
Latent diffusion
Rombach, Blattmann, Lorenz, Esser and Ommer move diffusion into the latent space of an autoencoder. First, train the autoencoder with a perceptual loss and light regularization to compress an image by a factor \(f\) on each side. Then train diffusion on those compressed representations and decode the generated latents into pixels. The latent grid has \(f^2\) fewer spatial locations, reducing the work of each diffusion step.
The decoder sets the range of images the latent generator can produce. A useful diagnostic is therefore to encode and decode real held-out images before judging the diffusion model. Details already lost in those reconstructions reveal a limitation of the representation. Improving the latent sampler alone does not give the decoder a new way to express them.
Compression creates a tradeoff. With too little, the diffusion model spends capacity on detail that is barely visible. With too much, the autoencoder has already discarded information the generator needs. Their experiments favour \(f = 4\) and \(f = 8\), and their ImageNet model at \(256 \times 256\) reaches \(3.60\) with 400 million parameters.R. Rombach, A. Blattmann, D. Lorenz, P. Esser and B. Ommer, "High-resolution image synthesis with latent diffusion models", CVPR 2022 (arXiv 2112.10752), Section 4.1: models with \(f \in \{1, 2, 4, 8, 16, 32\}\), "a significant FID gap of 38 between pixel-based diffusion (LDM-1) and LDM-8 after 2M training steps", and "In summary, LDM-4 and -8 offer the best conditions"; Table 3: LDM-4-G, FID \(3.60\), 400M parameters, 250 DDIM steps, classifier-free guidance \(1.5\). Stable Diffusion is this design at scale.
Moving to latents changes both the data law and the meaning of the noise level. The new distribution lives in a smaller space shaped by the autoencoder. Its signal-to-noise ratio is measured in latent units, so a schedule tuned for one resolution may need adjustment at another. Esser et al. shift timesteps with resolution to keep the fraction of signal destroyed at a given time comparable across latent grids.Esser et al. (2024), Section 5.3, "Resolution-dependent shifting of timestep schedules": for an image of \(n\) pixels the path's time is reparameterized so that the signal-to-noise ratio at a given \(t\) falls as \(n\) grows, a correction that the second figure of Section 3.1 draws with its shift slider as a vertical shift of the \(\lambda_t\) curve.
The results in Section 7.3 leave the choice of space open. Zheng et al. replace the perceptual autoencoder with a semantic representation; Li and He remove the autoencoder and design the backbone and prediction target for pixels. Section 10.4 returns to the question of which space makes generation easiest to learn.
Conditioning without retraining
A trained diffusion model can also use information that was not supplied as a condition during training. The three methods below modify the sampling procedure to incorporate known pixels, a guide image, or a measurement. Each uses the denoiser we already have, with a different approximation for the extra information.
Inpainting by replacement. Suppose the pixels in a set \(\Omega\) are known. After each reverse step, overwrite those coordinates with their forward-noised values, \(\alpha_t x_0^\Omega + \sigma_t z\). The model fills in the unknown coordinates while repeatedly seeing a noisy version of the observed surroundings. Song et al. describe this procedure in their appendix. Its exactness requires restrictive conditional-independence assumptions; in general it is an approximation.
Editing by partial noising. Meng et al.'s SDEdit adds noise to a guide image—a sketch or a rough edit—up to an intermediate time \(t_0\), then runs the reverse process. Structure that survives the noise level \(\sigma_{t_0}\) can guide the output, while finer details are regenerated. This uses the coarse-to-fine interpretation of sampling discussed by Dieleman.C. Meng, Y. He, Y. Song, J. Song, J. Wu, J.-Y. Zhu and S. Ermon, "SDEdit: guided image synthesis and editing with stochastic differential equations", ICLR 2022 (arXiv 2108.01073). The invertibility of the flow, Theorem 4.3.3, gives the other editing primitive, DDIM inversion, which maps an image to its noise and back.
Posterior sampling for inverse problems. For a measurement \(y = A x_0 + \text{noise}\), Bayes' rule gives \(\nabla \log p_t(x \mid y) = s_t(x) + \nabla_x \log p_t(y \mid x)\). The difficulty is the second term: it asks for the measurement likelihood given a noisy state, rather than a clean image.
The exact measurement term averages over the possible clean sources:
\[p_t(y\mid x) = \mathbb E\bigl[p(y\mid x_0)\mid x_t=x\bigr].\]Evaluating a likelihood at the posterior mean is generally different from averaging that likelihood. The distinction matters most when the posterior contains several incompatible explanations: its mean can be an unlikely clean image, as in the two-point example of Section 2. The approximation below becomes more plausible when the clean posterior is tightly concentrated; the figure below draws both terms on that two-point example.
Chung et al. approximate that likelihood by evaluating it at the Tweedie estimate, \(p(y \mid D_t(x))\), and differentiating through the denoiser. The approximation has no general error guarantee. Much of the work on diffusion posterior sampling studies when it is reliable and how to improve it.H. Chung, J. Kim, M. T. Mccann, M. L. Klasky and J. C. Ye, "Diffusion posterior sampling for general noisy inverse problems", ICLR 2023 (arXiv 2209.14687).
panel R1 at sigma = 1 from the noisy state x = 0.3, with the measurement y = x_0 + tau xi at
tau = 0.5: the posterior puts 0.646 on +1 and 0.354 on -1, its mean D(x) = 0.291; the
exact term p_sigma(y | x), two bumps at -1 and +1 of those heights (green), against the
approximation N(y; D(x), tau^2), one bump at 0.291 (ink); at y = 1 they read 0.515 and
0.292, a ratio of 1.76; or, in the second view, the pulls on x at y = 1, 0.708 exact
against 2.594 from the approximation
strip the ratio of the exact term to the approximation against sigma: 1.00 at 0.3, 0.97 at
0.5, 1.76 at 1, 2.98 at 2
(the live figure needs JavaScript)
Remark 7.5.1 (takeaway). Conditioning and guidance assemble a sampling drift from the learned score and additional information. Bayes' rule supplies a classifier term; classifier-free guidance combines conditional and unconditional predictions; inverse-problem methods use observations. Once the needed predictions are available, the samplers of Sections 4 and 5 can use the modified field. The distinction between learning a denoiser and deciding how to sample from it remains useful throughout.