10. What is known
Convergence guarantees and recent results Draft
Convergence: from a score to a sample
The earlier parts established exact identities between scores, denoisers, velocities, and sampling processes. A trained model uses approximations to all of them. This part asks what we can still guarantee, and which assumptions those guarantees require.
There are three distinct questions. Given a score accurate in \(L^2\), how many sampling steps are needed, and how close is the output to the desired law? Given \(n\) observations, how accurately can we learn the score? And when the exact empirical objective favours memorization, what allows a trained network to generalize? We will treat these questions in turn, then collect the open problems and dated results.
The first step is to separate the error already present at the sampler's start, the error in the learned score, and the error introduced by finite steps. The convergence bounds differ mainly in how they control the last term.
Notice the endpoint \(\delta>0\). The theorem compares the sampler with a slightly noised version of the data, where a smooth density and a score are available. Replacing that law by the original data requires a separate control of the remaining smoothing. This distinction is essential for discrete or lower-dimensional data: a smooth density and an atomic law can be close in a transport distance while having infinite relative entropy in a relevant direction.
Theorem 10.1.1 (the decomposition, informal). Let the reverse SDE (4.2) with \(\lambda = 1\) be run from \(\gamma = \mathcal N(0, I)\) on a grid \(1 = t_0 > t_1 > \dots > t_N = \delta\) with a learned score \(s_\theta\), the drift frozen over each step. Let \(\hat p_\delta\) be the law of its output and \(p_\delta\) the true marginal at time \(\delta\). Then
\[\begin{aligned} \mathrm{KL}(p_\delta \,\|\, \hat p_\delta) \;\le\; &\underbrace{\mathrm{KL}(p_1 \,\|\, \gamma)}_{\text{prior}} \;+\; \underbrace{\sum_k h_k\, g_{t_k}^2\, \mathbb E_{p_{t_k}}\|s_\theta - s_{t_k}\|^2}_{\text{score error}} \\ &+\; \underbrace{\text{(steps, dimension, regularity of } s_t)}_{\text{discretization}} , \end{aligned}\]up to constants. The prior term is Theorem 1.4.3; the score term is the appropriately weighted excess denoising loss above the irreducible floor of Section 2.3 on the sampler's grid; the discretization term is where the theorems differ.
The argument first compares entire random trajectories. Local errors in the drift accumulate into a discrepancy between these path distributions. Keeping only the endpoint throws away information, so it cannot increase relative entropy. This explains why a sum of errors incurred along the path can bound the final sample error, and also why the bound can be loose.
Where it comes from. The relative entropy of the laws of two diffusions on path space with the same diffusion coefficient \(g_t\) and drifts \(b^1_t\), \(b^2_t\), started from the same law, is \(\tfrac12 \int_0^1 \mathbb E\, \|b^1_t - b^2_t\|^2/g_t^2\, dt\) under the first law, by Girsanov's theorem; applied to the true reverse SDE and to the algorithm's chain, interpolated as a diffusion with a piecewise-constant drift, the difference of the drifts is \(g_t^2\) times the difference between \(s_\theta\) at the start of the step and \(s_t\) at the current point, which splits into the score error and the change of the true score along a step. The prior term enters because the two processes start from different laws, \(p_1\) and \(\gamma\), and the data-processing inequality carries their relative entropy to the end. ∎S. Chen, S. Chewi, J. Li, Y. Li, A. Salim and A. R. Zhang, "Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions", ICLR 2023 (arXiv 2209.11215), whose main result holds "for an \(L^2\)-accurate score estimate (rather than \(L^\infty\)-accurate)", does "not require restrictive functional inequality conditions that preclude substantial non-log-concavity", and scales "polynomially in all relevant problem parameters". The Girsanov route is theirs and that of the later papers; the control of the score's change along a step is where stochastic localization enters in Benton et al.
left the output law of the reverse SDE (lambda = 1) on the variance-preserving schedule, run on
R2 from N(0, 1) in N = 128 steps to delta = 0.02 with the drift frozen over each step and the
exact score of the empirical measure of 64 training points in place of the true score (green),
over the true marginal p_delta (ink) and the output of the exact score (grey), with the 64
points as a rug; or KL(p_t || p^_t) along the run against half the running score term
right KL(p_delta || p^_delta) against the bound's score term, log-log, for the empirical scores of
n = 16 to 1024 points and R2's score shifted by 0.1, at N = 32 and 128: with the exact score
128 steps leave 0.0051 nats and 32 steps 0.0402 (the floors); with 64 points at N = 128 the
error is 0.0828 against a term of 0.1686, the floor plus half the term (0.0894); the filled
points from n = 64 on and the shifted score stay within 12 % of that line at both step counts,
while 32 points at 32 steps reach 0.65 of theirs; 75 % of the term accrues in the last quarter
of the steps; in transport distance the comb is 0.335 from p_delta and the shifted law 0.114,
against 0.0828 and 0.0974 nats
(the live figure needs JavaScript)
What is proved, in order of appearance, is the following; the step counts are for a reverse SDE sampler unless stated.
| result | assumptions on \(p_0\) | what is bounded | rate or step count |
|---|---|---|---|
| Chen, Chewi, Li, Li, Salim, Zhang, ICLR 2023 | finite second moment; \(L^2\) score error | total variation to a smoothed \(p_0\) | polynomial in \(d\), \(1/\varepsilon\) and the score error |
| Chen, Lee, Lu, ICML 2023 | finite second moment | reverse relative entropy to \(p_0 * \mathcal N(0, \delta)\) | \(\tilde O(d \log(1/\delta)/\varepsilon)\) steps |
| Benton, De Bortoli, Doucet, Deligiannidis, ICLR 2024 | finite second moment | relative entropy to \(p_0 * \mathcal N(0, \delta)\), accuracy \(\varepsilon^2\) | \(\tilde O(d \log^2(1/\delta)/\varepsilon^2)\) steps, linear in \(d\) |
| Conforti, Durmus, Gentiloni Silveri, SIMODS 2025 | finite Fisher information relative to \(\gamma\) | relative entropy, fixed step size, OU and its kinetic version | "simple, improved and sharp" constants |
| Li, Wei, Chen, Chi, ICLR 2024 | no smoothness | total variation | \(1/T\) for the probability-flow ODE, \(1/\sqrt T\) for the DDPM-type sampler; \(1/T^2\) and \(1/T\) for accelerated variants |
| Li, Wei, Chi, Chen, 2024 | mild | total variation, the probability-flow ODE | \(d/\varepsilon\) iterations up to logarithms |
| Li and Yan, JMLR 2025 | \(\ell_2\)-accurate scores | total variation, DDPM | \(O(d/T)\); \(O(k/T)\) for data of intrinsic dimension \(k\) |
| Chen, Daras, Dimakis, ICML 2023 | mild | DDIM-type deterministic samplers, non-linear forward processes | the first polynomial bounds for these samplers |
| Gentiloni Silveri, Conforti, Durmus, ICML 2026 | finite moments, score integrability | relative entropy and 2-Wasserstein for Brownian-motion-based flow matching | improved dimension dependence over earlier work |
| Pham, Conforti, Ren, Durmus, May 2026 | approximation error only | relative entropy (early stopped) and total variation, discrete flow matching on \(\mathbb Z_m^d\) | explicit in \(m\) and \(d\) |
These are worst-case step counts over all laws satisfying the assumptions. They can be far larger than the counts used in practice: with \(d\) in the hundreds of thousands and \(\varepsilon\) around a tenth, \(d/\varepsilon^2\) is of the order of \(10^7\) steps, while the samplers of Section 5 use tens.
The bounds leave out structure that may help explain the gap. Heun's method benefits from regularity along the flow lines, and image data may have a much smaller intrinsic dimension than their pixel count. Li and Yan's \(O(k/T)\) bound begins to capture the latter advantage.H. Chen, H. Lee and J. Lu, "Improved analysis of score-based generative modeling: user-friendly bounds under minimal smoothness assumptions", ICML 2023 (arXiv 2211.01916): "under only a finite second moment condition, approximating … in reverse KL divergence in \(\varepsilon\)-accuracy can be done in \(\tilde O(d\log(1/\delta)/\varepsilon)\) steps". J. Benton, V. De Bortoli, A. Doucet and G. Deligiannidis, ICLR 2024 (arXiv 2308.03686): "diffusion models require at most \(\tilde O(d\log^2(1/\delta)/\varepsilon^2)\) steps to approximate an arbitrary distribution on \(\mathbb R^d\) corrupted with Gaussian noise of variance \(\delta\) to within \(\varepsilon^2\) in KL divergence". G. Conforti, A. Durmus and M. Gentiloni Silveri, "KL convergence guarantees for score diffusion models under minimal data assumptions", SIAM Journal on Mathematics of Data Science 7 (2025), 86–109 (arXiv 2308.12240). G. Li, Y. Wei, Y. Chen and Y. Chi, "Towards non-asymptotic convergence for diffusion-based generative models", ICLR 2024 (arXiv 2306.09251); G. Li, Y. Wei, Y. Chi and Y. Chen, "A sharp convergence theory for the probability flow ODEs of diffusion models", arXiv 2408.02320: "\(d/\varepsilon\) iterations – modulo some logarithmic and lower-order terms – are sufficient to approximate the target distribution to within \(\varepsilon\) total-variation distance". G. Li and Y. Yan, "\(O(d/T)\) convergence theory for diffusion probabilistic models under minimal assumptions", Journal of Machine Learning Research 26 (2025), 1–55 (arXiv 2409.18959). S. Chen, G. Daras and A. G. Dimakis, "Restoration-degradation beyond linear diffusions: a non-asymptotic analysis for DDIM-type samplers", ICML 2023 (arXiv 2303.03384). M. Gentiloni Silveri, G. Conforti and A. Durmus, "Diffusion flow matching: dimension-improved KL bounds and Wasserstein guarantees", ICML 2026 (arXiv 2606.16610). L.-T.-N. Pham, G. Conforti, Z. Ren and A. Durmus, "Discrete flow matching: convergence guarantees under minimal assumptions", arXiv 2605.08882 (May 2026).
When reading a rate, keep its target law and metric attached to it. A bound in Wasserstein distance controls a cost of moving probability; a bound in relative entropy controls log probability ratios. Neither is the same as a guarantee about the visual appearance of one image. Likewise, an error bound for one fixed exact trajectory answers a different question from a bound on a stochastic sampler's output distribution.
Remark 10.1.2 (takeaway). The bounds connect a weighted score error to an error in the generated law, while accounting separately for initialization and discretization. This is the sense of Chen et al.'s phrase "sampling is as easy as learning the score". Several results need only moment or Fisher-information assumptions on the data, rather than log-concavity or a bound on the number of modes.
These guarantees make score estimation central to the problem. Once an accurate score is available, they quantify the additional cost of turning it into samples. They do not, by themselves, explain how a finite training set and a neural network produce that score.
Estimation: learning the score from samples
Sampling guarantees begin by assuming that the score is already accurate. Learning it is a statistical problem of its own. Without useful structure in the data, high dimension makes that task prohibitively expensive; the next result makes this limitation precise.
Theorem 10.2.1 (the rate of score estimation; Wibisono, Wu and Yang). For data in \(\mathbb R^d\) whose law is sub-Gaussian with a Lipschitz score \(s^*\), the minimax rate of estimating \(s^*\) from \(n\) samples in squared \(L^2(p)\) error is \(\tilde\Theta(n^{-2/(d + 4)})\), attained by a regularized kernel estimator; no estimator does better uniformly over the class.A. Wibisono, Y. Wu and K. Y. Yang, "Optimal score estimation via empirical Bayes smoothing", COLT 2024 (arXiv 2402.07747): "the optimal rate of \(\tilde\Theta(n^{-2/(d+4)})\) … highlighting the curse of dimensionality"; the extension to Hölder scores of exponent \(\beta \le 1\) is in the same paper.
The dimension dependence reflects how little nearby data a generic estimator can use. Subdividing each coordinate into twice as many intervals creates \(2^d\) times as many cells. Without structure relating those cells, resolving finer features demands many more observations. Low-dimensional support, smoothness, locality, and shared features can reduce the effective problem, but that extra structure must enter the assumptions or the estimator.
For an image, the exponent \(2/(d + 4)\) is extremely small. The worst-case guarantee would require an impractical number of samples. It tells us that useful learning must exploit structure beyond the generic assumptions of the theorem.
left the true score of R2 (ink) and the smoothed empirical score of n = 64 training points, the
score of the training set convolved with a Gaussian of width h = 0.1 (green), with p_0 and
the integrand p_0 (s^ - s*)^2 as washes and the 64 points as a rug
right the error E_{p_0}|s^_{n,h} - s*|^2 against h for n = 16 to 4096 (means over 4 samples): a U
per n, best at h = 0.14 for n = 64, where the error is 7.8 against 16,500 at h = 0.02 and
24.0 at h = 1; the best error falls from 13.4 at n = 16 to 1.27 at n = 4096 like n^-0.41 and
the best width from 0.25 to 0.081 like n^-0.18; or the best error against n with the
theorem's worst-case slopes; or the expected number of 60,000 training points within sqrt(d)
of a typical point against d: 31,230 in d = 1, 1,910 in d = 20, 72 in d = 50 and 3.2e-30 in
d = 784, where one point would take about 10^34 samples
(the live figure needs JavaScript)
Several results identify structure that helps. Oko, Akiyama and Suzuki study densities in a Besov smoothness class and empirical denoising over a suitable network class. They obtain minimax density-estimation rates in total variation and Wasserstein-1 distance, with adaptation to low-dimensional support.
Chen, Huang, Zhao and Wang show that, for data on an unknown linear subspace, a suitable network can achieve error controlled by the subspace dimension. Li and Yan's \(O(k/T)\) result carries intrinsic dimension into the sampling bound. Under the manifold hypothesis, Huang, Han, Bai et al. describe two phases of learning: at small noise the denoising map contracts toward the manifold, while at moderate noise it refines the density along it. Their sample complexity depends on intrinsic dimension.K. Oko, S. Akiyama and T. Suzuki, "Diffusion models are minimax optimal distribution estimators", ICML 2023 (arXiv 2303.01861): "when the true density function belongs to the Besov space and the empirical score matching loss is properly minimized, the generated data distribution achieves the nearly minimax optimal estimation rates in the total variation distance and in the Wasserstein distance of order one … diffusion models adapt to low-dimensional data distributions". M. Chen, K. Huang, T. Zhao and M. Wang, "Score approximation, estimation and distribution recovery of diffusion models on low-dimensional data", ICML 2023 (arXiv 2302.07194): "The convergence rate depends on the subspace dimension, indicating that diffusion models can circumvent the curse of data ambient dimensionality". W. Huang, A. Han, M. Bai et al., "Provably learning diffusion models under the manifold hypothesis: collapse and refine", arXiv 2605.20235 (May 2026): "at small noise scales, the diverging singularity of the score drives a rapid dimensional collapse of the induced denoising map onto the data manifold projection; at moderate noise scales, training refines the intrinsic density on the learned manifold", with "sample complexity [that] depends on the intrinsic dimension rather than the ambient dimension". J. Pidstrigach, "Score-based generative models detect manifolds", NeurIPS 2022 (arXiv 2206.01018), gave the first precise conditions under which the samples lie on the data's manifold, and noted that with the training set as the manifold the same result describes memorization.
Remark 10.2.2 (confusion: the loss has the wrong minimizer). These results restrict the estimator: they use a network class of a chosen size and architecture, or an estimator with smoothing. Section 2.5 considered minimization over all functions and obtained the empirical score, which memorizes the training set. The restriction is therefore part of the explanation for generalization. It determines which aspects of the empirical denoiser the learned model can represent and fit.
Memorization and generalization
The exact denoiser of a finite training set averages over that set. As the noise falls, its posterior can concentrate on a single training point. Following this empirical score all the way to the data therefore leads back to the observations. Real networks often produce new samples instead. To understand the difference, we first need to locate where the individual training points become distinguishable.
A finite dataset and the population it was sampled from define different denoising problems. If we declare the dataset itself to be the complete distribution, then returning one of its members is exactly correct generation. Generalization instead asks us to infer structure that extends beyond those observed points. A network can help by imposing a restricted, shared form on its denoising rule, even while fitting the finite dataset less exactly.
Proposition 10.3.1 (the empirical score memorizes). Let \(\hat p_0 = \tfrac1n\sum_i\delta_{x^{(i)}}\) be the empirical measure of \(n\) training points and \(\hat s_t\) its exact score after noising. Exact reverse sampling from the corresponding terminal law returns \(\hat p_0\) in the endpoint limit, whether by the probability flow or the reverse SDE. Its samples therefore belong to the training set with probability one.
Proof. At every positive time, the noised empirical law is a smooth Gaussian mixture. The probability flow and reverse SDE transport its marginals back toward \(\hat p_0\), by Theorems 1.3.3 and 4.2.2, with the atomic endpoint understood as a limit. ∎
The concentration on a particular training point can be read from its posterior weights. For two training points \(i\) and \(j\),
\[\frac{r_j(x)}{r_i(x)} = \exp\!\left(-\frac{\|x-\alpha_t x^{(j)}\|^2-\|x-\alpha_t x^{(i)}\|^2}{2\sigma_t^2}\right).\]What matters is the gap between squared distances relative to the noise variance. A nearest point dominates when its competitors' combined weight becomes small. This explains why sample spacing matters, while also showing why a single nearest-neighbour distance is only a rough indicator of the transition.
left two clusters N(+-1.5, 0.45^2 I) and n = 64 training points from them, the probe (1.4, 0.1)
with its nearest point (green, d1 = 0.130) and second-nearest point (d2 = 0.135) and a
circle of radius sigma = 0.3
right the posterior over the 64 points at the probe against sigma, stacked in order of distance:
the nearest point's share reaches 1/2 at sigma_C = 0.077, below d1, where the odds of the
second point are still 0.88 and 2.3 points remain; the population law's cluster posterior
(blue) reaches 0.9 at sigma_S = 1.31, 17 times higher; over 64 held-out probes sigma_C/d1
runs from 0.67 to 1.42 between quartiles and exceeds 1 at 31 of them
(the live figure needs JavaScript)
An unrestricted model that learns the exact empirical score memorizes. The useful questions are when individual training points begin to determine the reverse trajectory, and how a trained network behaves below that noise level. Our small examples let us compare the empirical and population scores directly.
left 24 of 64 flow lines on two clusters N(+-1.5, 0.45^2 I) under the empirical score of n training
points (ending on training points; one path followed in green) and under the population
score (blue)
right the speciation noise level, where a path commits to a cluster, and the collapse noise level,
where one training point takes the posterior, against n, with the exact bimodality threshold
sigma* = 1.431; or the squared distance between the two scores against sigma for four n; or the
collapse level against n in d = 2, 4, 8 and 16 dimensions: 37-fold down from n = 4 to 1024 in
d = 2, like n^-0.58, and 4.0-fold in d = 16, like n^-0.24, where it is still 0.44 at n = 1024,
0.32 times the training set's nearest-neighbour distance of 1.39
(the live figure needs JavaScript)
Example 10.3.2 (two regimes on two clusters). In the figure's default view, \(n = 64\), the two scores agree down to a noise level of \(\sigma = 1.345\), the mean over the sixty-four paths of the level at which they commit to a cluster under either of them, and part only below \(\sigma \approx 0.128\), where the empirical posterior begins to be owned by single training points; the training set's mean nearest-neighbour distance is \(0.21\), and the collapse happens at a noise level of that order. All sixty-four endpoints under the empirical score are training points. With \(n = 1024\) the collapse waits until \(\sigma \approx 0.0313\) while the speciation level stays near \(1.35\): the first depends on the sample, the second on the clusters. The population law stops being bimodal along the cluster axis at \(\sigma^* = \sqrt{1.5^2 - 0.45^2} = 1.43\), which sits above the figure's speciation level because the criterion waits for a decided posterior.
The second view of the right panel draws the squared distance between the two scores against \(\sigma\) for four training-set sizes: it is small at large \(\sigma\), grows roughly like \(\sigma^{-2}\) as \(\sigma\) falls, with no threshold at the collapse level, and at every \(\sigma\) it falls with \(n\), so that the level below which it is appreciable moves toward zero with \(n\), the statement of Section 2.5 that the two scores agree wherever the noise exceeds the spacing of the training points, with the spacing on the axis; at \(\sigma = 0.1\) the error is \(15.264\) for \(n = 16\), \(2.773\) for \(n = 64\) and \(0.142\) for \(n = 1{,}024\). Between the two levels the empirical and the population flows are indistinguishable, and that interval is where a model's generalization is decided. ◁
Biroli, Bonnaire, De Bortoli and Mézard gave this picture its theory. For a data law with a covariance whose top eigenvalue is \(\Lambda\), the speciation time of a variance-preserving process, the time at which the backward trajectories commit to a class, solves \(\Lambda e^{-2t_S} = 1\), a spectral quantity of the data alone; for two Gaussian clusters it is \(t_S = \tfrac12\log(d\tilde\mu^2)\) in their scaling. The collapse time, at which trajectories fall into the attractor of a single training point, is where an excess entropy density, which at large times equals \((\log n)/d\), crosses zero, so that \((\log n)/d\) is the parameter that controls memorization: "one needs \((\log n)/d \gg 1\) to push \(t_C\) to zero and avoid the collapse", which for images is never the case. Lu and Tang showed in 2026 that a non-reversible perturbation of the reverse process can move the speciation earlier but leaves the collapse where it is, and Takahashi, Takahashi and Kabashima found the two transitions, with the same structure, in discrete diffusion.G. Biroli, T. Bonnaire, V. de Bortoli and M. Mézard, "Dynamical regimes of diffusion models", Nature Communications 15, 9957 (17 November 2024) (arXiv 2402.18491): "The speciation time \(t_S\) is the transition from pure noise to the commitment of a trajectory towards one category. The collapse time \(t_C\) is the time where the backward trajectory falls into the attractor of one given data point"; Equation 4, "\(\Lambda e^{-2t_S} = 1\)"; "The parameter \(\alpha = (\log n)/d\) plays a key role in the collapse". H. Lu and Y. Tang, "Steering dynamical regimes of diffusion models by breaking detailed balance", arXiv 2602.15914 (February 2026, presented at an ICML 2026 workshop): "suitable non-reversible perturbations can accelerate speciation. In contrast … the corresponding collapse time remains unchanged". T. Takahashi, T. Takahashi and Y. Kabashima, "Dynamical regimes of discrete diffusion models", Journal of Statistical Mechanics (2026) 083402.
Experiments help reveal what a network learns below the collapse level. Kadkhodaie, Guth, Simoncelli and Mallat train two networks on disjoint halves of a dataset and compare samples generated from the same noise. With up to a hundred training images, the networks memorize. Around a thousand, their outputs resemble distorted training images. At \(10^5\), the two networks produce nearly identical samples that no longer resemble their respective training examples. This suggests convergence toward a function shaped by the data law and architecture, rather than the particular sample. They explain the behaviour through shrinkage in a basis adapted to each image's geometry.
Kamb and Ganguli derive the optimal denoiser subject to locality and translation equivariance, calling it the "equivariant local score machine". After calibrating one time-dependent parameter, it predicts trained convolutional models' outputs with a median \(r^2\) of \(0.95\) on CIFAR-10 and \(0.94\) to \(0.96\) on three other datasets. The resulting samples combine local patches of training images into new arrangements. The account is less complete for self-attention networks, where the median \(r^2\) is about \(0.77\).Z. Kadkhodaie, F. Guth, E. P. Simoncelli and S. Mallat, "Generalization in diffusion models arises from geometry-adaptive harmonic representations", ICLR 2024, outstanding paper (arXiv 2310.02557): "two DNNs trained on non-overlapping subsets of a dataset learn nearly the same score function, and thus the same density, when the number of training images is large enough", and "for our examples, roughly \(10^5\) images suffices". M. Kamb and S. Ganguli, "An analytic theory of creativity in convolutional diffusion models", ICML 2025 (arXiv 2412.20292): median \(r^2\) "of 0.95, 0.94, 0.94, 0.96 for our top model on CIFAR10, FashionMNIST, MNIST, and CelebA", "a locally consistent patch mosaic mechanism of creativity", and "median \(r^2 \sim 0.77\) on CIFAR10" for attention U-Nets.
Generalization does not rule out copying. Carlini et al. use a generate-and-filter procedure to extract more than a thousand training examples from existing models, including photographs and company logos. Somepalli et al. also find copying in models including Stable Diffusion, with rates affected by dataset size and duplication. These observations are consistent with the empirical-score picture, especially where repeated or isolated examples strongly influence the posterior.N. Carlini, J. Hayes, M. Nasr et al., "Extracting training data from diffusion models", USENIX Security 2023 (arXiv 2301.13188); G. Somepalli, V. Singla, M. Goldblum, J. Geiping and T. Goldstein, "Diffusion art or digital forgery? Investigating data replication in diffusion models", CVPR 2023 (arXiv 2212.03860).
Remark 10.3.3 (takeaway). The examples distinguish a scale associated with the broad structure of the data from a scale at which individual training points become visible. At noise levels large relative to sample spacing, empirical and population denoisers can be close. At smaller noise levels, fitting the empirical denoiser more accurately can pull samples toward particular observations.
On the 150-step chain the training set becomes visible to the empirical law's exact denoiser from index \(25\) down, where one training image owns the posterior of a held-out noised image (the probes of Example 4.4.5). The evidence for convolutional networks suggests that locality and equivariance help them combine patterns across observations. This is a partial explanation of their behaviour, not a guarantee against copying, and a comparably complete account for transformers remains open. Generalization depends on which structure the model learns to share across examples, especially at noise levels where exact fitting would distinguish individual training points.
Example 10.3.4 (a network memorizes, live). We return to R3, the two moons, and draw the noise level \(\sigma\) log-uniform on \([0.02, 3]\), using 64 fixed training points drawn from the law. The network of Example 2.3.9, a multilayer perceptron with layers \(3 \to 64 \to 64 \to 2\) and 4,546 parameters, trains in the page below with Adam on batches of 256 for as many steps as the reader adds. At every step we measure its training loss, its held-out loss, its distances to the population score and to the empirical score of the 64 points, and the share of its 512 samples, by Heun in 48 steps from fixed noise, that land within \(0.02\) of a training point: the two clocks of Remark 10.3.3 on a learned score.
Its distance to the population score falls first, while the training set is invisible at the noise levels where the loss is largest, and throughout the run the learned score stays closer to the population score than to the empirical score of the 64 points. From about step 11,000 on, its training loss stays below the \(0.60\) that the exact denoiser of R3 scores on the same 64 points, a level that no denoiser chosen without the sample beats in expectation, and the endpoints of its flow lines, redrawn as it trains, begin, slowly, to move onto the training points: \(0.029\) of 512 samples within 0.02 of a training point at 3,000 steps and \(0.043\) at 10,000, against \(0.027\) for fresh draws of R3. What makes a network of 2.5 million parameters trained on 60,000 images stop short of this is the subject of the next example. ◁
left twenty-four flow lines of a 4,546-parameter network trained in the page on 64 points of
R3 (Adam, batches of 256, sigma log-uniform on [0.02, 3]), from fixed noise, redrawn every
hundred steps over the training points; or its field over the exact one, its error over
the plane or 512 of its samples
right its held-out loss by noise level against the floor of Proposition 2.3.5 (Monte Carlo
with the exact score), and the training loss (green), the excess over the floor, the
distances to the population and to the empirical score of the 64 points and the loss of
the exact denoiser on them, against the training step
(the live figure needs JavaScript; it trains in a Web Worker)
Using the same starting noise compares two learned maps point by point. This is a stronger comparison than asking whether their output distributions agree: two different maps can generate the same law. Nearest-training-image distances answer another question and need a held-out reference, because ordinary new samples can also resemble training examples. The experiment below uses both measurements rather than treating either as a complete test of generalization.
Example 10.3.5 (memorization on FashionMNIST). We repeat the comparison on FashionMNIST at \(28 \times 28\). For each \(n\) in \(\{128, 1{,}024, 8{,}192, 30{,}000\}\), we train pairs of networks with the \(\varepsilon\)-model's architecture (Section 11, widths (64, 64, 128)). Each member sees \(n\) images from its own half of the training set; the two halves are disjoint. Training uses DDPM's chain of 1,000 steps for fifteen thousand updates. We use one pair at \(n = 128\) and two at each larger \(n\).
Both networks in a pair generate samples by DDIM in fifty steps from the same 96 draws of \(\mathcal N(0, I)\). We compare their outputs by root-mean-square distance per pixel and by how often a FashionMNIST classifier assigns them the same class. We also measure each sample's distance to its nearest training image, using held-out images' distance to the full 60,000-image training set as a reference. Finally, we compare the networks' held-out losses by noise level.
This is Kadkhodaie et al.'s experiment at the scale of this series. In the figure below, the change map \(A - B\) shows, pixel by pixel, where the two samples from one noise differ, and following one pair gives its own distance, its two nearest-image distances and its two classes.
At \(n = 128\) the two models' samples are \(0.581\) apart per pixel and agree in class \(0.396\) of the time, and each sample sits \(0.226\) from its nearest training image, closer than a held-out image is to the training set, \(0.256\); at \(n = 30{,}000\) the two samples from one noise are \(0.536\) apart, agree \(0.427\) of the time, and sit \(0.313\) from any training image, farther than a held-out one. At both sizes each model is closer to its own training set than to the other model, and the class agreement is between four and five in ten at every size, against one in ten by chance.
The clearest signal is the distance to the training set: it crosses the held-out reference once, from \(0.226\) at \(n = 128\) to \(0.320\) at \(n = 1{,}024\), and stays above it, consistent with stronger attraction to the training set at the smallest size. This distance alone neither proves exact copying nor rules out occasional copies at the larger sizes.
The distance between the two models' samples, which Kadkhodaie et al. saw vanish at \(10^5\) images, does not fall here in any way this measurement can rank: averaged over the pairs of models at each size, the distance between the two samples is \(0.638\) at \(n = 1{,}024\), \(0.613\) at \(n = 8{,}192\) and \(0.536\) at \(n = 30{,}000\), and the two pairs at \(n = 8{,}192\) differ by \(0.035\) but those at \(n = 1{,}024\) and at \(n = 30{,}000\) by \(0.334\) and \(0.384\), more than any two sizes differ, so two pairs per size are too few to rank the sizes. At this scale, with networks of two and a half million parameters trained for fifteen thousand steps, two models trained on disjoint data still disagree over most of every image.
The held-out loss at the cleanest bin says the same about memorization, \(1.849\) for the pair of models trained on \(128\) images each against \(0.239\) at \(n = 30{,}000\). ◁
top twenty-four samples of a network trained on n images (A), of a network trained on n other
images from the other half of the training set, from identical noise (B), the change map
A − B (green where A is brighter, ink where B is) and the nearest training image of each A,
for n = 128 to 30,000, on DDPM's chain of 1,000 steps at 15,000 steps
bottom against n: the distance between A and B, 0.581 per pixel at n = 128 and 0.536 at 30,000;
the distance to the nearest training image, 0.226 and 0.313, against 0.256 for held-out images;
and the class agreement of A and B, 0.396 and 0.427
(the live figure needs JavaScript and loads about 0.3 MB of data)
What remains
The sources reviewed for this series leave the following questions open as of 24 October 2026. Each concerns a gap between the exact constructions, the available guarantees, and what trained models do in practice.
-
The inductive bias of attention. Kamb and Ganguli's theory accounts for convolutional networks and not for self-attention, and the models of Section 7.3 are transformers. What function does a diffusion transformer converge to below the collapse level, and why does it generalize?
-
A theory of guidance. Proposition 7.2.3 says that guidance is a sampler and not a target. What law do the guided samples have, as a function of the scale, the interval and the sampler, and why does the interval of Kynkäänniemi et al. or the weak model of Karras et al. do what it does?
-
Learning the flow map. The one-step methods of Section 8 train on self-referential targets, and their convergence is not proved. When does a fastforward training, in Geng et al.'s term, converge, and to what, and which of the three characterizations of Proposition 8.1.2 is the right one to impose?
-
The step counts. The theorems of Section 10.1 need \(d/\varepsilon^2\) steps and practice uses twenty. Which property of the data law, or of the learned velocity, is responsible, and can it be measured on a trained model? Example 5.4.2 gives one data point: on the ε-model trained on DDPM's chain of 1,000 steps, DDIM's Fréchet distance is \(363.1\) at ten steps, \(251.0\) at twenty-five and \(212.7\) at the full thousand, most of the way in twenty-five steps and the rest slowly.
-
The sampler with an imperfect score. With the exact score every member of the family (4.2) gives the same law; with a learned one they do not, and Remark 4.2.5 is an argument, not a theorem. Which \(\lambda\), which grid and which solver minimize the error of a given imperfect score, and does the answer depend on where the score is wrong?
-
Parallel decoding. Theorem 9.3.4 forbids correct sequences in sublinear steps for a particular data law. For natural language, how much parallelism is compatible with a given error, and do editing and block structure change the exponent or only the constant?
-
The space. Pixels, perceptual latents and representation latents have each produced the best ImageNet result of some year of Section 7.3. Is there a principle that says which space a diffusion model should live in for a given data law and compute budget?
-
Evaluation. FID below \(2\) on ImageNet, the scale of every record in Section 10.5, is within the range where the protocol matters as much as the model, and the likelihood of Section 4.4 does not rank models the way samples do; Dupuis et al.'s gap is a first measurement of the distance between the two. What should the next benchmark measure?
The latest results, dated
The table gathers the records and recent results cited throughout the series, grouped by task. Dates and publication status reflect the sources read on 24 October 2026. The sidenotes in the referenced sections give the sources and the conditions behind each result.
| task | result | by | date | status | where in this series |
|---|---|---|---|---|---|
| CIFAR-10, unconditional, FID | 1.97 at 35 evaluations | EDM, Karras et al. | 2022 | NeurIPS 2022 | 5.3 |
| CIFAR-10, 2 steps | 2.06 | sCM, Lu and Song | Oct 2024 | ICLR 2025 | 8.2 |
| CIFAR-10, 1 step, no teacher | 2.51 | iCT, Song and Dhariwal | Oct 2023 | ICLR 2024 | 8.2 |
| CIFAR-10, 20 evaluations, training-free solver | 2.87 | DPM-Solver, Lu et al. | 2022 | NeurIPS 2022 | 5.2 |
| CIFAR-10, bits per dimension | 2.65 (2.49 with augmentation) | VDM, Kingma et al. | 2021 | NeurIPS 2021 | 3.4 |
| ImageNet 64, FID | 1.01 | autoguidance, Karras et al. | Jun 2024 | NeurIPS 2024 | 7.2 |
| ImageNet 64, 2 steps, distilled | 1.25 | Align Your Flow, Sabour et al. | Jun 2025 | NeurIPS 2025 | 8.2 |
| ImageNet 256, multi-step, FID | 1.13 | RAE, Zheng et al. | Oct 2025 | ICLR 2026 | 7.3 |
| ImageNet 256, pixels, multi-step | 1.82 | JiT, Li and He | Nov 2025 | CVPR 2026 | 7.3 |
| ImageNet 256, 1 step, no teacher | 1.72 | improved MeanFlow, Geng et al. | Dec 2025 | CVPR 2026 | 8.3 |
| ImageNet 256, 1 step, no teacher | 1.19 | priming models, Park et al. | Oct 2026 | under review | 8.3 |
| ImageNet 512, FID | 1.25 | autoguidance, Karras et al. | Jun 2024 | NeurIPS 2024 | 7.2 |
| ImageNet 512, 2 steps | 1.88 | sCM, Lu and Song | Oct 2024 | ICLR 2025 | 8.2 |
| ImageNet 512, pixels | 1.78 | JiT, Li and He | Nov 2025 | CVPR 2026 | 7.3 |
| text diffusion, speed | 1,479 tokens/s, average over evaluations | Gemini Diffusion, DeepMind | May 2025 | vendor page | 9.5 |
| text diffusion, open weights | about 1,500 tokens/s on one H100 | DiffusionGemma | Jul 2026 | technical report | 9.5 |
| text diffusion, 100B | 892 tokens/s on HumanEval+ | LLaDA2.1 | Feb 2026 | arXiv | 9.5 |
| convergence, reverse SDE | \(\tilde O(d\log^2(1/\delta)/\varepsilon^2)\) steps | Benton et al. | 2023 | ICLR 2024 | 10.1 |
| convergence, DDPM, total variation | \(O(d/T)\), \(O(k/T)\) with intrinsic dimension \(k\) | Li and Yan | 2024 | JMLR 2025 | 10.1 |
| convergence, flow matching | dimension-improved KL and \(W_2\) bounds | Gentiloni Silveri et al. | Jun 2026 | ICML 2026 | 10.1 |
| convergence, discrete flow matching | KL and total variation on \(\mathbb Z_m^d\) | Pham et al. | May 2026 | arXiv | 10.1 |
| the score-matching gap | tighter bounds in three metrics | Dupuis et al. | Jul 2026 | arXiv | 4.4 |
| generalization, theory | ELS machine, median \(r^2\) 0.95 on CIFAR-10 | Kamb and Ganguli | Dec 2024 | ICML 2025 | 10.3 |
| memorization, theory | speciation and collapse times | Biroli et al. | Feb 2024 | Nature Communications 2024 | 10.3 |