11. Further reading
Draft
The earlier parts' sidenotes provide the full bibliography. The table below offers a shorter route into it, organized by what you want to study next: introductions, the mathematics behind Sections 1 and 4, the original methods, few-step and discrete models, and the theory.
Benchmark results and reported speeds carry dates throughout this series. Records changed repeatedly in the year before it was written, so a number needs its date and experimental setting to be useful.
| purpose | reading |
|---|---|
| a first course | Nakkiran, Bradley, Zhou and Advani, "Step-by-step diffusion: an elementary tutorial", arXiv 2406.08929 (2024): DDPM, DDIM and flow matching with the least machinery, in the time convention of this series |
| Holderrieth and Erives, "An introduction to flow matching and diffusion models", arXiv 2506.02070 (2025, revised March 2026): a self-contained tutorial from first principles, with exercises | |
| Fjelde, Mathieu and Dutordoir, "An introduction to flow matching", Cambridge MLG blog (20 January 2024): the marginalization theorem and the Gaussian paths from the side of normalizing flows | |
| Dieleman, "Perspectives on diffusion", sander.ai (20 July 2023): the dozen readings of one model, and why the choice of reading matters in practice | |
| the reference | Lipman, Havasi, Holderrieth, Shaul, Le, Karrer, Chen, Lopez-Paz, Ben-Hamu and Gat, "Flow matching guide and code", arXiv 2412.06264 (2024): the theory in full generality, from Euclidean flows to Riemannian ones, Markov chains and generator matching, with a PyTorch library |
| stochastic analysis | Øksendal, Stochastic Differential Equations, 6th ed. (Springer, 2003): Brownian motion, the Itô integral and the existence theorem of Section 1.1 |
| Karatzas and Shreve, Brownian Motion and Stochastic Calculus, 2nd ed. (Springer, 1991): the same with proofs, and Girsanov's theorem, on which Section 10.1 rests | |
| Pavliotis, Stochastic Processes and Applications (Springer, 2014), and Risken, The Fokker–Planck Equation, 2nd ed. (Springer, 1989): the Fokker–Planck equation, the Ornstein–Uhlenbeck process and the approach to equilibrium of Section 1 | |
| Bakry, Gentil and Ledoux, Analysis and Geometry of Markov Diffusion Operators (Springer, 2014): the logarithmic Sobolev inequality and the Bakry–Émery method of Theorem 1.4.3 | |
| Nelson, Dynamical Theories of Brownian Motion (Princeton University Press, 1967): why the noise is Gaussian, Theorem 5.1, quoted in the companion post Markov processes and Kolmogorov equations | |
| Anderson, "Reverse-time diffusion equation models", Stochastic Processes and their Applications 12 (1982), and Haussmann and Pardoux, "Time reversal of diffusions", Annals of Probability 14 (1986): the theorem of Section 4.2 in its original and general forms | |
| transport | Villani, Optimal Transport: Old and New (Springer, 2009), and Ambrosio, Gigli and Savaré, Gradient Flows (Birkhäuser, 2008): the continuity equation, push-forwards and the Wasserstein geometry behind Sections 6.1 and 6.4 |
| the original papers, diffusion | Sohl-Dickstein, Weiss, Maheswaranathan and Ganguli, ICML 2015; Ho, Jain and Abbeel, NeurIPS 2020; Song and Ermon, NeurIPS 2019; Song, Sohl-Dickstein, Kingma, Kumar, Ermon and Poole, ICLR 2021; Song, Meng and Ermon, ICLR 2021; Kingma, Salimans, Poole and Ho, NeurIPS 2021; Karras, Aittala, Aila and Laine, NeurIPS 2022: Sections 3, 4 and 5 in the order the ideas appeared |
| the practical side | NVIDIA Deep Learning Institute, "Generative AI with Diffusion Models": U-Nets, DDPM with \(T = 150\), the GroupNorm, GELU and sinusoidal-embedding refinements, classifier-free guidance; the architecture and the training recipe that the worked examples of Sections 2, 3, 5 and 7 follow, described in full in the section below |
| the original papers, flows | Lipman, Chen, Ben-Hamu, Nickel and Le, ICLR 2023; Liu, Gong and Liu, ICLR 2023; Albergo and Vanden-Eijnden, ICLR 2023, and Albergo, Boffi and Vanden-Eijnden, JMLR 26 (2025); Tong et al., TMLR; Pooladian et al., ICML 2023: Section 6 and its couplings |
| the score and its estimation | Hyvärinen, JMLR 6 (2005); Vincent, Neural Computation 23 (2011); Efron, JASA 106 (2011); Raphan and Simoncelli, Neural Computation 23 (2011): Section 2 from the statistics side |
| guidance and architectures | Dhariwal and Nichol, NeurIPS 2021; Ho and Salimans, arXiv 2207.12598; Kynkäänniemi et al., NeurIPS 2024; Karras et al., "Guiding a diffusion model with a bad version of itself", NeurIPS 2024; Rombach et al., CVPR 2022; Peebles and Xie, ICCV 2023; Ma et al., ECCV 2024; Esser et al., ICML 2024; Yu et al., ICLR 2025; Zheng, Ma, Tong and Xie, ICLR 2026; Li and He, CVPR 2026: Section 7 |
| few steps | Salimans and Ho, ICLR 2022; Song, Dhariwal, Chen and Sutskever, ICML 2023; Song and Dhariwal, ICLR 2024; Lu and Song, ICLR 2025; Boffi, Albergo and Vanden-Eijnden, TMLR and NeurIPS 2025; Frans, Hafner, Levine and Abbeel, ICLR 2025; Sabour, Fidler and Kreis, NeurIPS 2025; Geng, Deng, Bai, Kolter and He, NeurIPS 2025, and Geng et al., CVPR 2026; Zhou, Ermon and Song, ICML 2025: Section 8, with the preprints of 2026 listed in its table |
| discrete states | Austin et al., NeurIPS 2021; Campbell et al., NeurIPS 2022; Lou, Meng and Ermon, ICML 2024; Sahoo et al. and Shi et al., NeurIPS 2024; Ou et al., ICLR 2025; Gat et al., NeurIPS 2024; Arriola et al., ICLR 2025; Feng et al., NeurIPS 2025: Section 9; the systems are in its table with their reports |
| convergence theory | Chen, Chewi, Li, Li, Salim and Zhang, ICLR 2023; Chen, Lee and Lu, ICML 2023; Benton, De Bortoli, Doucet and Deligiannidis, ICLR 2024; Conforti, Durmus and Gentiloni Silveri, SIMODS 7 (2025); Li, Wei, Chen and Chi, ICLR 2024, and arXiv 2408.02320; Li and Yan, JMLR 26 (2025); Gentiloni Silveri, Conforti and Durmus, ICML 2026: Section 10.1 |
| estimation and generalization | Oko, Akiyama and Suzuki, ICML 2023; Chen, Huang, Zhao and Wang, ICML 2023; Wibisono, Wu and Yang, COLT 2024; Pidstrigach, NeurIPS 2022; Kadkhodaie, Guth, Simoncelli and Mallat, ICLR 2024; Kamb and Ganguli, ICML 2025; Biroli, Bonnaire, De Bortoli and Mézard, Nature Communications 15 (2024): Sections 10.2 and 10.3 |
| pages that carry dates | the arXiv abstract pages of the papers above, whose submission histories date every version; the OpenReview records, which name the venue and the decision; deepmind.google/models/gemini-diffusion and inceptionlabs.ai/blog for the vendor figures of Section 9.5, read on 24 October 2026 |
Working through the examples. Use the figures to make predictions before reading their numerical answers. Four short experiments cover the main distinctions in the series.
- In Section 2, place the two-point example's observation at zero. Predict the denoiser's answer, then reduce the noise. Explain why the estimate at the exact midpoint remains zero even when almost every generated sample should be near one of the two clean points.
- In Section 5, keep the exact denoiser and starting noises fixed, and increase the number of Euler and Heun steps. Separate the distance to the reference flow's endpoints from the distance between sample distributions. Decide which remaining discrepancy more steps can remove.
- In Section 6, compare independent endpoint pairs with the alternative coupling. Follow both a straight training segment and a learned flow line. Explain why the network can see only straight segments during training and still generate curved trajectories.
- In Section 9, keep the token predictor exact and change how many positions are revealed together. Predict which dependencies those simultaneous draws lose. Then compare neighbouring and interleaved groups in the six-token example.
For each experiment, name what stayed fixed, what changed, and which formula explains the result. If the result disagrees with your prediction, locate the missing assumption before trying another setting. These small laws make it possible to separate a mistaken intuition from an imperfectly trained network.
The model behind the examples
The nine worked examples in Sections 2 to 10 use small diffusion models trained for this series. This section gives the information needed to reproduce them: the data, each network operation, the training recipe, the two chains, and the classifier used to measure sample quality. The architecture and recipe follow the source acknowledged in the introduction, described here in this series's notation.
Each model trains in under forty minutes on the GPU specified below. Their purpose is to make the mathematics measurable at a manageable scale.
Data. FashionMNIST contains 60,000 training images and 10,000 test images of garments in ten classes. We use the native \(28 \times 28\) resolution, one channel, and scale pixels to \([-1, 1]\), with \(-1\) black and \(+1\) white. Training includes random horizontal flips.
Held-out losses use the first 2,000 test images with fixed noise draws. References to the posterior mean over "the 60,000 training images" use the original training set without flips.
Read the network as one function reused at every noise level. Its output has the same spatial shape as the noisy image because it predicts a noise vector in that image's coordinates. The shrinking and expanding spatial grids are internal computations; they do not change the state space of the diffusion process. Time and class tell this shared function which conditional prediction to make.
map the browser-sized U-Net, widths (16, 32, 64), as two rows of boxes whose area is proportional to
the tensor's value count: the state x, 1 x 28 x 28 (784 values); the down path d_0 16 x 28 x 28
(12,544), d_1 32 x 14 x 14 (6,272), d_2 64 x 7 x 7 (3,136); the dense layers 3,136 -> 32 -> 32
-> 3,136, two bars and two dots; then u_0 64 x 7 x 7, u_1 32 x 14 x 14, u_2 16 x 28 x 28, the
head's concatenation 32 x 28 x 28 (25,088) and the noise estimate, 1 x 28 x 28 again; the three
skips d_2, d_1, d_0 join the up path, the only full-resolution paths around the dense layers
counts input 2,544, Down 1 50,976, Down 2 203,328, dense 204,928, Up 0 37,056, time 6,080, Up 1 53,664,
Up 2 13,520, head 4,801: 576,897 in all, of which the dense layers and Down 2 hold 408,256 (71 %);
(64, 64, 128) gives 2,524,673 with a 64-wide bottleneck and 64 % in those two blocks; the class
embeddings add 6,272 and 22,784
clock the eight features te(tau), tau = i/T, that the network reads: at i = 500 of the chain of 1,000
steps, sin 0.479, 0.023, 0.0011, 0.00005 and cos 0.878, 0.9997, 1.0000, 1.0000; along the whole
chain sin tau moves by 0.841 and cos tau by 0.460, the other six features by at most 0.046
(the live figure needs JavaScript)
The network, operation by operation. The U-Net has widths \((c_0, c_1, c_2)\). Its basic unit, the ConvBlock, applies a \(3 \times 3\) convolution with stride 1 and padding 1, followed by GroupNorm and GELU. GroupNorm uses \(\min(8, c_0)\) groups in the input block and output head, and \(\min(32, c_0, c_1, c_2)\) groups elsewhere.
The input has shape \(1 \times 28 \times 28\). A residual input block computes \(x_1 = \mathrm{ConvBlock}(1 \to c_0)(x)\) and \(d_0 = x_1 + \mathrm{ConvBlock}(c_0 \to c_0)(x_1)\) at full resolution.
Two Down blocks reduce the spatial size. Each applies two ConvBlocks, \(c_{\mathrm{in}} \to c_{\mathrm{out}}\) and \(c_{\mathrm{out}} \to c_{\mathrm{out}}\), then rearranges every \(2 \times 2\) cell into four channels. A ConvBlock reduces those \(4 c_{\mathrm{out}}\) channels back to \(c_{\mathrm{out}}\). The outputs are \(d_1\) of shape \(c_1 \times 14 \times 14\) and \(d_2\) of shape \(c_2 \times 7 \times 7\).
The bottleneck flattens \(d_2\), applies GELU, and passes through three dense layers, \(49\, c_2 \to c_1 \to c_1 \to 49\, c_2\), each followed by ReLU. It reshapes the result to \(c_2 \times 7 \times 7\) and applies another ConvBlock to obtain \(u_0\).
Time enters through \(\tau = i/T\), the chain index divided by the chain length. We form eight sinusoidal features, \(\sin(\tau f_k)\) and \(\cos(\tau f_k)\), at four frequencies \(f_k = 10000^{-k/3}\) for \(k = 0, 1, 2, 3\): \(1, 0.046, 0.0022, 0.0001\). Two embeddings, each a dense layer, GELU, and a second dense layer, map those features to \(c_2\) and \(c_1\) channels. They are broadcast over the pixels and added at two depths: \(t_1\) to \(u_0\) and \(t_2\) to \(u_1\) below.
The class-conditional model embeds a masked one-hot class vector into \(c_2\) and \(c_1\) channels in the same way. These embeddings multiply the up-path features before the time embeddings are added: \(k_1 \odot u_0 + t_1\) and \(k_2 \odot u_1 + t_2\). During training, each one-hot entry is zeroed independently with probability \(0.1\). Setting the mask to zero at sampling time gives the unconditional estimate needed for classifier-free guidance (Section 7.2).
Two Up blocks restore the spatial resolution. Each concatenates its input with the matching skip, \(d_2\) and then \(d_1\), applies a \(2 \times 2\) transposed convolution with stride 2 to \(c_{\mathrm{out}}\) channels, and follows it with four ConvBlocks. The outputs are \(u_1\) of shape \(c_1 \times 14 \times 14\) and \(u_2\) of shape \(c_0 \times 28 \times 28\).
The head concatenates \(u_2\) with \(d_0\), then applies a \(3 \times 3\) convolution to \(c_0\) channels, GroupNorm, ReLU, and a final \(3 \times 3\) convolution to one channel. It outputs the noise estimate \(\varepsilon_\theta(x_i, i)\). The one-step denoiser uses the same architecture with \(x_0\) as its target.
The widths are \((64, 64, 128)\) for the \(\varepsilon\)-model and the class-conditional model, and \((16, 32, 64)\) for the browser-sized models and the one-step denoiser.
There are two separate loops to keep track of. A training update draws a clean image, a time, and fresh noise, then makes one prediction with a known target. A sampling run starts without a clean image and feeds each predicted update into the next. A successful reconstruction of a corrupted held-out image tests the first task; generating from fresh noise also tests the starting law and the whole sequence of updates.
left one training example with the trained weights held fixed: training image 1 of the 200 shipped
(a top) and a draw of epsilon give x_500 = 0.279 x_0 + 0.960 epsilon (abar_500 = 0.0778); the
browser-sized epsilon-model's prediction eps-hat = eps_theta(x_500, 500) is drawn beside its
known target as the change map eps-hat - eps over +-0.5, with the one-example loss
||eps-hat - eps||^2/784 = 0.0204 beside the held-out mean 0.0116 of indices 467 to 533, the
loss an update would reduce
right one sampling run of the same model, DDIM with eta = 0 in 8 steps at indices 999, 856, 714,
571, 428, 285, 143, 0: the eight states, from the starting noise z, over the eight clean
estimates x_0-hat made at those indices, each estimate fed with its noise estimate into the
next state; the state keeps an RMS of 1.00 at 999, 0.75 at 143 and 0.67 at the end, while
x_0-hat reaches |x_0-hat| = 42 at index 999 and stays below 3.5 from index 285
check on the two test inputs shipped with the weights, at indices 37 and 120, the page's network
agrees with the training framework to 1.2 x 10^-3 in the worst coordinate and 3.6 x 10^-4
root mean square
(the live figure needs JavaScript and loads about 0.3 MB of data, and about 1.2 MB of weights when
you press run)
The recipe. Noise-prediction models use Algorithm 2.3.7 in noise units. Take a batch of 128 training images, flip each horizontally with probability one half, and draw a chain index \(i\) uniformly from \(\{0, \dots, T - 1\}\) for each image. Form \(x_i = \sqrt{\bar\alpha_i}\, x_0 + \sqrt{1 - \bar\alpha_i}\, \varepsilon\), then take one Adam step at learning rate \(10^{-3}\) on the mean squared error between \(\varepsilon_\theta(x_i, i)\) and \(\varepsilon\).
The full models train for forty epochs, \(18{,}720\) steps. The one-step denoiser trains for six epochs, \(2{,}808\) steps, using the single corruption \(0.5\, x_0 + 0.5\, z\) and the clean image as its target. The memorization pairs of Section 10.3 train for \(15{,}000\) steps on DDPM's chain of 1,000 steps. Each member sees \(n\) images from one of two disjoint halves of the training set. A fixed random permutation defines the halves, and each run's seed selects its images; \(n = 30{,}000\) uses the whole half.
Seeds are fixed: 0 for each full model, 1 and 2 for the first memorization pair, and 3 and 4 for the second. Rerunning on the same hardware reproduces the numbers to floating-point noise. Each run records training loss and held-out noise-prediction error in fifteen index bins. Held-out measurements are taken every thousand steps, or every three thousand for memorization runs, producing the curves in Section 3.3.
Sampling begins with a draw of \(\mathcal N(0, I)\) at index \(T - 1\). The ancestral sampler of Algorithm 3.3.2 uses the mean \((x_i - \beta_i\, \varepsilon_\theta(x_i, i)/\sqrt{1 - \bar\alpha_i})/\sqrt{1 - \beta_i}\) and adds Gaussian noise of variance \(\beta_{i-1}\), the previous step's rate, rather than the posterior variance \(\tilde\beta_i\). It adds no noise at \(i = 0\). The alternative is DDIM from Proposition 5.1.3, with \(\eta\) specified by each example and a subsequence of indices when taking fewer than \(T\) steps.
On one NVIDIA A10G the \(\varepsilon\)-model trains in forty epochs (\(2{,}262\) s on the 150-step chain and \(2{,}222\) s on DDPM's chain of 1,000 steps), the class-conditional model in \(1{,}987\) s and \(2{,}379\) s on the two chains, the browser-sized models in \(820\) s and \(879\) s, and the one-step denoiser in \(113\) s.
Two chains. Both chains take \(\beta_i\) linear from \(10^{-4}\) to \(0.02\) over their \(T\) steps. The 150-step chain, \(T = 150\), ends at \(\bar\alpha_T = 0.219\): its last state is still \(0.47\, x_0 + 0.88\, z\), far from the \(\mathcal N(0, I)\) the sampler starts from, the prior mismatch of Remark 1.4.5 that Example 3.1.5 measures. DDPM's chain of 1,000 steps, \(T = 1000\), ends at \(\bar\alpha_T = 4.0 \times 10^{-5}\). The \(\varepsilon\)-model and the class-conditional model are trained on both, so that the figures can show what the length of the chain changes and what it does not: Examples 3.1.5, 3.3.5, 4.4.5 and 5.4.2 measure the \(\varepsilon\)-model of each chain side by side, the film of Example 3.3.4 runs the \(\varepsilon\)-model trained on DDPM's chain of 1,000 steps and keeps every fifth state of six chains, and the guidance grid of Example 7.2.8 runs the class-conditional model trained on that chain. The browser-sized models, which Example 7.3.1 and the live run of Section 5.4 evaluate in the page, and the memorization pairs are trained on DDPM's chain of 1,000 steps only. On either chain the index \(i\) runs from \(0\), almost clean, to \(T - 1\).
The models. The table lists every model the figures draw on, with its widths, its number of parameters, its training length and its chain.
| model | widths | parameters | training | chain |
|---|---|---|---|---|
| the ε-model | (64, 64, 128) | 2,524,673 | 40 epochs, 18,720 steps | the 150-step chain and DDPM's chain of 1,000 steps |
| the class-conditional model | (64, 64, 128) | 2,547,457 | 40 epochs, 18,720 steps | the 150-step chain and DDPM's chain of 1,000 steps |
| the browser-sized ε-model, which the page runs in Section 5.4 | (16, 32, 64) | 576,897 | 40 epochs, 18,720 steps | DDPM's chain of 1,000 steps |
| the browser-sized class-conditional model, which the page runs in Section 7.3 | (16, 32, 64) | 583,169 | 40 epochs, 18,720 steps | DDPM's chain of 1,000 steps |
| the one-step denoiser of Section 2.2 | (16, 32, 64) | 576,897 | 6 epochs, 2,808 steps | none: the one level \(0.5\, x_0 + 0.5\, z\) |
| the memorization pairs of Section 10.3: two ε-models trained on n images each, drawn from the two disjoint halves of the training set, n = 128, 1,024, 8,192 or 30,000 (a whole half); one pair for n = 128 and two pairs otherwise | (64, 64, 128) | 2,524,673 | 15,000 steps | DDPM's chain of 1,000 steps |
To reproduce a comparison, first hold the trained weights, starting noises, and evaluation images fixed while changing only the sampler. Then hold the sampler fixed when comparing models. Inspect both the noisy state and the clean estimate along a run: the state is supposed to contain noise at intermediate times, while the estimate expresses the model's current guess. This avoids mistaking an expected noisy intermediate state for a failed denoiser.
The classifier and FD_clf. A small convolutional classifier supplies the class accuracies and the feature distances labelled FD_clf. It uses two \(3 \times 3\) convolutions, first to 32 and then to 64 channels, each followed by GroupNorm with eight groups, GELU, and \(2 \times 2\) max pooling. A dense layer produces 128 features, followed by GELU and a final dense layer to the ten classes. Training uses Adam at \(10^{-3}\) for ten epochs, batches of 256, and seed 0, with test accuracy \(0.920\).
FD_clf is the Fréchet distance between Gaussians fitted to those 128-dimensional features for \(1{,}024\) samples and \(1{,}024\) held-out images. Its scale belongs to this classifier. FID in the literature instead uses features from an Inception network trained on ImageNet, so the two sets of values cannot be compared directly.
The network in the page. The figures in Sections 5.4 and 7.3 evaluate the browser-sized networks as you watch. The weights of the \((16, 32, 64)\) models, \(576{,}897\) and \(583{,}169\) parameters stored as 16-bit floats, take about 1.2 MB each. They load when you press run, and a background thread evaluates the operations described above. On test inputs shipped with the weights, its results agree with the training framework to \(2.2 \times 10^{-3}\) in the worst coordinate and \(4 \times 10^{-4}\) root mean square.
Each guided step needs two forward passes: one with the class and one with a zero mask. A pass takes a fraction of a second in the browser, so the figures show progress and default to modest sample and step counts.
How the pages draw. The recorded model outputs are stored as small binary arrays, with one byte per value and a stated numerical range. When a figure comes into view, the page decodes the arrays and draws them on a canvas. The readouts use those same arrays, so the displayed measurements and images come from the same data.
You can reproduce the closed-form examples with the programs in Sections 1 to 9. They use Gaussian-mixture formulas and ordinary arithmetic, with no Python imports beyond the standard library, and print the numbers quoted in the prose.
The trained examples require the models described above. With the data, a GPU, and this recipe, you can retrain them and compare the same quantities the figures measure.