Noises and flows
Draft
A diffusion model learns to make new samples by learning to denoise. We start with samples from a distribution \(p_0\) on \(\mathbb R^d\) whose density we do not know. We make a noisy copy of each sample and train a network to estimate the clean original from the copy and its noise level. Because we kept the original, we can measure the estimate's squared error. Adding noise gives us training inputs with known targets, so we can train by regression without evaluating \(p_0\).
We do this at many noise levels, from almost clean data to almost pure noise. The best estimate under squared error is the conditional mean of the clean original given the noisy observation. As we will derive, these denoising estimates determine how to reverse the change in distribution caused by adding noise. A sampler starts from fresh noise and uses the estimates to guide successive steps toward the data distribution.
Throughout this series, \(x_0 \sim p_0\) is a data point and \(z \sim \mathcal N(0, I)\) is a standard Gaussian vector. At time \(t\), the noisy point is
\[x_t \;=\; \alpha_t\, x_0 + \sigma_t\, z, \qquad t \in [0, 1],\]The schedule \((\alpha_t, \sigma_t)\) controls how much signal remains and how much noise has been added. It runs from \((\alpha_0, \sigma_0) = (1, 0)\), the data, toward \((\alpha_1, \sigma_1) \approx (0, 1)\), noise. Write \(p_t\) for the law of \(x_t\). A stochastic differential equation, \(dx = f_t\, x\, dt + g_t\, dW\), produces the same distribution at each time.
left 1,500 pairs (x_t, x₀) of R2 at t = 0.3 of the variance-preserving schedule (α_t = 0.630,
σ_t = 0.777), three bands of dots; through them the conditional mean E[x₀ | x_t], a soft
staircase, with the least-squares fit to the pairs alone on top of it (within 0.25 between
the pairs' 1st and 99th percentiles); the diagonal x₀ = x_t/α_t; p_t under the axis; the
probe at x_t = 0.80 with its 87 residuals
right the slice p(x₀ | x_t = 0.80): bumps at 0.22, 1.58 and −1.29 with weights 0.51, 0.40 and
0.10, the rescaled copy 1.27 as a tick, the conditional mean 0.61 as a dot, shift −0.66
squared errors: conditional mean 0.707 (the floor), x_t/α_t read back 1.523, the mean −0.24
guessed 1.404; the fit errs 0.725 on 20,000 held-out pairs
(the live figure needs JavaScript)
There are several ways to travel back through these distributions. The three formulas below describe the main ones we will study. They may look unfamiliar now; the series builds them one piece at a time.
\[\begin{aligned} &\text{the reverse SDE:}\quad && dx = \bigl[f_t\, x - g_t^2\, \nabla \log p_t(x)\bigr]\, dt + g_t\, d\bar W, \\ &\text{the probability-flow ODE:}\quad && \frac{dx}{dt} = f_t\, x - \tfrac12\, g_t^2\, \nabla \log p_t(x) \;=\; u_t(x), \\ &\text{the flow-matching velocity:}\quad && u_t(x) = \mathbb E\bigl[\dot\alpha_t\, x_0 + \dot\sigma_t\, z \,\big|\, x_t = x\bigr]. \end{aligned}\]The first line describes Anderson's reverse-time diffusion, which adds fresh randomness along the way. The second describes a deterministic flow with the same distribution at each time. The third writes that flow's velocity as a conditional expectation: an average that a neural network can learn from pairs \((x_0, z)\) by least squares.
The shared ingredient is the score, \(\nabla \log p_t\). It points in the direction in which the log density rises fastest. Tweedie's formula connects it to denoising: \(\mathbb E[x_0 \mid x_t] = (x_t + \sigma_t^2 \nabla \log p_t(x_t))/\alpha_t\). Learn the best estimate of the clean point from a noisy one, and you have also learned the score.B. D. O. Anderson, "Reverse-time diffusion equation models", Stochastic Processes and their Applications 12 (1982), 313–326, for the first line; Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon and B. Poole, "Score-based generative modeling through stochastic differential equations", ICLR 2021 (arXiv 2011.13456), for the second and for the use of both as generative models; Y. Lipman, R. T. Q. Chen, H. Ben-Hamu, M. Nickel and M. Le, "Flow matching for generative modeling", ICLR 2023 (arXiv 2210.02747), for the third, there with time running the other way. Tweedie's formula is B. Efron, "Tweedie's formula and selection bias", Journal of the American Statistical Association 106 (2011), 1602–1614, after H. Robbins (1956) and K. Miyasawa (1961).
Three ideas tie the methods together. First, adding noise moves probability. The Fokker–Planck equation describes that movement, and we can read it either as a stochastic diffusion or as transport along the velocity \(u_t\). Individual paths can look very different while sharing every marginal distribution \(p_t\).
Second, the score is a conditional expectation. We can therefore learn it from samples without evaluating the unknown density. A network may predict the clean point, the noise, the score, or the velocity; these four predictions are affine transformations of one another.
Third, sampling takes time. We must choose a schedule, a solver, and a number of steps. We can also choose how much fresh noise to add during reversal, from none in the deterministic flow to the amount in Anderson's reverse SDE. Few-step and one-step methods try to learn larger moves along the path, replacing many small solver steps with a learned map.
A trained diffusion model supports a family of samplers. Much of the progress since 2020 has come from learning which sampler to use and how to reach the data in fewer steps.
forward 2,000 samples of R2 diffuse into N(0, 1) under the variance-preserving process: their
density as a wash under the exact p_t drawn as contours, forty of them as lines
backward 2,000 draws of N(0, 1) return to R2 by the reverse SDE (rough paths) and by the
probability-flow ODE (smooth paths that never cross); the marginals agree, the
couplings do not: the strip plots each path's start against its end
(the live figure needs JavaScript)
The change in sampling cost has been striking. On class-conditional ImageNet at 256 × 256 pixels, Peebles and Xie's diffusion transformer reached a Fréchet inception distance of 2.27 in December 2022 with a 250-step sampler. In May 2025, the MeanFlow model of Geng, Deng, Bai, Kolter and He reached 3.43 with a single network evaluation, trained from scratch. By October 2026, the best one-step result readable on arXiv, a preprint under review, stood at 1.19, below the 250-step figure of 2022.W. Peebles and S. Xie, "Scalable diffusion models with transformers", ICCV 2023 (arXiv 2212.09748, December 2022): FID 2.27 for DiT-XL/2 with classifier-free guidance at scale 1.5, 250 sampling steps. Z. Geng, M. Deng, X. Bai, J. Z. Kolter and K. He, "Mean flows for one-step generative modeling", NeurIPS 2025 (arXiv 2505.13447, 19 May 2025): "an FID of 3.43 with a single function evaluation (1-NFE) on ImageNet 256x256 trained from scratch". W. Park, J. Kim, Y. Du, S. Chen and B. Han, "Are we overthinking one-step generation?", arXiv 2610.20331 (14 October 2026, an ICLR 2027 submission under review when read on 24 October 2026): "a state-of-the-art FID of 1.19 among one-step generators". Section 8 tabulates the one-step results between these dates with their status.
Text presents a related problem in a different state space. Tokens are discrete, so the forward process can replace them with masks instead of adding Gaussian noise. Diffusion language models can then unmask many tokens per step. In 2025, reported speeds reached 1,109 tokens per second for Mercury Coder Mini on a single H100, as measured by Artificial Analysis, and 1,479 tokens per second for Gemini Diffusion, as reported by Google DeepMind. An autoregressive model generates one token per step.Inception Labs, S. Khanna et al., "Mercury: ultra-fast language models based on diffusion", arXiv 2506.17298 (17 June 2025): "Mercury Coder Mini and Mercury Coder Small achieve state-of-the-art throughputs of 1109 tokens/sec and 737 tokens/sec, respectively, on NVIDIA H100 GPUs", from an endpoint hosted in February 2025. Google DeepMind, "Gemini Diffusion", deepmind.google/models/gemini-diffusion, read 24 October 2026: "1479 tokens / sec", the "average sampling speed across reported evals", with an overhead of 0.84 s; the model was announced at Google I/O on 20 May 2025. Both figures are the vendors' own. Section 9 explains what parallel unmasking costs in exactness, with an exact computation on six tokens. We will examine the mathematics behind these results, what has been proved, and what the reported numbers establish.
Sections 1 and 2 build the foundation: how noise changes a distribution, and how denoising regression learns its score. Section 3 turns those ideas into DDPM's discrete chain. Section 4 derives the reverse SDE and the probability-flow ODE, and shows how the latter gives a likelihood.
Sections 5 and 6 compare ways to move along the path. We first study DDIM, numerical solvers, and the design choices behind EDM. We then derive flow matching and connect its velocity to the score. Section 7 adds conditioning, guidance, architectures, and latent spaces. Section 8 asks how much of the sampling path can be learned in advance, down to a single step.
Section 9 moves to discrete states and masked language models. Section 10 gathers the convergence and estimation guarantees, examines memorization, and separates established results from open questions. Section 11 offers further reading and a reproducible recipe for the models used in the worked examples.
By the end, you should be able to read a diffusion library's sampling loop and identify its reverse process, solver, and coefficients. You should also be able to translate a training objective into a denoising regression, adapt one sampler to DDPM, EDM, or flow matching, and assess what a claim such as "one-step generation at FID 1.2" actually measures. The recipe in Section 11 lets you retrain the worked examples and connect each line of their training and sampling loops to the mathematics.
The figures let you work through the claims. Most compute their answers in the page using Gaussian mixtures, whose densities, scores, denoisers, and velocities have closed forms. When a figure shows a law rather than samples, it computes that law on a grid or by dynamic programming and labels the result accordingly. The prose quotes the default settings, so you can check the numbers before changing them.
Nine figures test the same ideas on real data. They use small U-Nets trained for this series on FashionMNIST at its native \(28 \times 28\) pixels, and carry the tag "FashionMNIST 28 × 28 · the small U-Net of Section 11". The architecture and recipe follow the NVIDIA Deep Learning Institute course "Generative AI with Diffusion Models". Each model took under forty minutes to train on one GPU. Section 11 gives the data, network, training recipe, chains, and seeds needed to reproduce them. These figures load small arrays as they come into view, usually a few hundred kilobytes and at most about two megabytes. Their numbers are measurements on particular trained networks with fixed seeds.
Some figures also train or run a network in your browser. The figure in Section 2.3 trains one while you watch. Two others, in Sections 5.4 and 7.3, run a trained network of 576,897 parameters in the page, 583,169 for the class-conditional one. They load the weights when you press run.NVIDIA Deep Learning Institute, "Generative AI with Diffusion Models": a course on U-Nets, DDPM with \(T = 150\), the GroupNorm, GELU and sinusoidal-embedding refinements, and classifier-free guidance, on FashionMNIST and on flowers. The architecture and the samplers are re-implemented here in the series' own words; nothing of its code is reproduced. The data are FashionMNIST, H. Xiao, K. Rasul and R. Vollgraf, arXiv 1708.07747 (2017), MIT licence, 60,000 training and 10,000 test images of \(28 \times 28\) pixels in ten classes, used at that resolution.
Four small examples recur. R1 is the two-point law \(\tfrac12 \delta_{-1} + \tfrac12 \delta_{+1}\) on the line; every formula can be worked out by hand. R2 is a mixture of three Gaussians on the line, with weights \(0.45, 0.35, 0.20\), means \(-1.4, 0.2, 1.6\), and standard deviations \(0.25, 0.15, 0.30\). Its mean is \(-0.24\) and its variance is \(1.40\).
R3 is the "two moons" in the plane: 24 isotropic Gaussians of standard deviation \(0.12\), twelve along each of two half circles of radius 1. The upper half circle is centred at \((-0.5, -0.3)\) and opens downward; the lower one is its reflection through the origin. The moons serve as two classes when we need labels. R4 transports one Gaussian to another in the plane, \(\mathcal N(0, I)\) to \(\mathcal N(\mu, \Sigma)\), where \(\mu = (1.1, 0.5)\) and \(\Sigma\) has standard deviations \(1.4\) and \(0.45\) along axes rotated by \(35^\circ\). Here the transports we study are affine, and their costs can be computed explicitly.
The real-data example, \(L\), is FashionMNIST at \(28 \times 28\) pixels, scaled to \([-1, 1]\): a law in \(784\) dimensions of which only samples are known. It lets us measure in a trained model the quantities we can compute exactly on R1 to R4. The opening figure uses one additional example, 64 Gaussians of standard deviation \(0.055\) arranged on a curve, as a visual introduction.
Notation. Data are \(x_0 \sim p_0\) in \(\mathbb R^d\) and noise is \(z \sim \mathcal N(0, I)\). Time is \(t \in [0, 1]\) with the data at \(t = 0\) and noise at \(t = 1\), the convention of Sohl-Dickstein et al., of DDPM and of Song et al.; the flow-matching literature runs time the other way, and Section 6 gives the dictionary.
The schedule is \((\alpha_t, \sigma_t)\), with \(\alpha\) decreasing from 1 and \(\sigma\) increasing from 0, and \(x_t = \alpha_t x_0 + \sigma_t z\) has law \(p_t\); the kernel \(p_t(x \mid x_0) = \mathcal N(x; \alpha_t x_0, \sigma_t^2 I)\). The signal-to-noise ratio is \(\alpha_t^2/\sigma_t^2\) and its logarithm is \(\lambda_t\). The forward SDE is \(dx = f_t\, x\, dt + g_t\, dW\) with \(f_t = \dot\alpha_t/\alpha_t\) and \(g_t^2 = -\sigma_t^2 \dot\lambda_t\) (Section 1.1).
The score is \(s_t(x) = \nabla \log p_t(x)\); the denoiser is \(D_t(x) = \mathbb E[x_0 \mid x_t = x]\); the noise estimate is \(\hat\varepsilon_t(x) = \mathbb E[z \mid x_t = x] = -\sigma_t s_t(x)\); the velocity is \(u_t(x) = \dot\alpha_t D_t(x) + \dot\sigma_t \hat\varepsilon_t(x)\) (Sections 2.2 and 4.3). A learned network carries the subscript \(\theta\): \(D_\theta\), \(\varepsilon_\theta\), \(s_\theta\), \(u_\theta\).
Three schedules recur: variance preserving, \(\alpha_t^2 + \sigma_t^2 = 1\), with \(\alpha_t = e^{-B(t)/2}\) and \(B(t) = \int_0^t \beta\) for a rate \(\beta_t\) (DDPM's choice is \(\beta_t = 0.1 + 19.9\, t\) in the continuous limit); variance exploding, \(\alpha_t = 1\) with \(\sigma_t\) growing, of which the EDM clock \(\sigma = t\) is a special case; and the linear path \(\alpha_t = 1 - t\), \(\sigma_t = t\) of flow matching.
The discrete chain of DDPM has \(T\) steps, rates \(\beta_i\), and \(\bar\alpha_i = \prod_{j \le i}(1 - \beta_j)\), so that \(x_i = \sqrt{\bar\alpha_i}\, x_0 + \sqrt{1 - \bar\alpha_i}\, \varepsilon\) (Section 3.1). The worked examples use two chains, the 150-step chain and DDPM's chain of 1,000 steps, both with \(\beta_i\) linear from \(10^{-4}\) to \(0.02\) (Section 11); where a figure uses the 150-step chain, \(i\) runs from \(0\) to \(149\), \(i = 149\) being the first index of the sampler, and a trajectory's row \(r\) is the state at index \(149 - r\).
The flow map of the ODE from time \(s\) to time \(t\) is \(X_{s \to t}\) (Section 8.1). A reverse process with noise multiplier \(\lambda \ge 0\) is \(dx = [f_t x - \tfrac{1 + \lambda^2}{2} g_t^2 s_t(x)]\, dt + \lambda g_t\, d\bar W\), run with \(dt < 0\); \(\lambda = 0\) is the flow and \(\lambda = 1\) is Anderson's reverse SDE (Section 4.2).
The relative entropy of \(p\) with respect to \(q\) is \(\mathrm{KL}(p \,\|\, q)\) and the Wasserstein-1 distance is \(W_1\). Numbered statements are Definition, Theorem, Proposition, Example and Remark \(N.S.K\), with \(N\) the part and \(S\) the section.
Three expositions are useful companions. Nakkiran, Bradley, Zhou and Advani derive DDPM, DDIM, and flow matching with little machinery and the same time convention used here. Fjelde, Mathieu and Dutordoir's Cambridge Machine Learning Group post approaches flow matching through normalizing flows, deriving the marginalization theorem and Gaussian paths. Dieleman's essay shows how the same model can be understood in many ways: as a score estimator, reverse SDE, flow, autoencoder, latent-variable model, recurrent network, autoregressive model in frequency space, or estimator of expectations.P. Nakkiran, A. Bradley, H. Zhou and M. Advani, "Step-by-step diffusion: an elementary tutorial", arXiv 2406.08929 (June 2024); T. Fjelde, E. Mathieu and V. Dutordoir, "An introduction to flow matching", Cambridge MLG blog, 20 January 2024, mlg.eng.cam.ac.uk/blog/2024/01/20/flow-matching.html; S. Dieleman, "Perspectives on diffusion", 20 July 2023, sander.ai/2023/07/20/perspectives.html. The reference of the mathematical side is Y. Lipman, M. Havasi, P. Holderrieth, N. Shaul, M. Le, B. Karrer, R. T. Q. Chen, D. Lopez-Paz, H. Ben-Hamu and I. Gat, "Flow matching guide and code", arXiv 2412.06264 (December 2024), whose chapters 6 to 9 extend the construction to Markov chains and to arbitrary Markov processes.
The companion to Section 1 is this project's earlier post on Markov processes and the Kolmogorov equations. It derives the generator and the forward and backward equations from the Markov property. Here we take the stochastic differential equation as our starting point.
(General disclaimers apply.) Nothing here is investment, legal or tax advice. The paper results, benchmark figures and vendor speed claims are as published at the dates given, and the sidenotes say where each was read; a preprint under review is marked as such.