1. Brownian motion
The Fokker–Planck equation Draft
Brownian motion and the Itô equation
To understand how a diffusion model creates a sample, we first need to understand how it adds noise. The forward process gradually erases information about the data. Its transition kernel tells us what remains at each time; the Fokker–Planck equation describes how the whole distribution changes; and an equilibrium bound tells us how close the result is to pure noise. These are the ingredients we will later use to build the reverse process.
Definition 1.1.1 (Brownian motion). A standard Brownian motion in \(\mathbb R^d\) is a process \((W_t)_{t \ge 0}\) with \(W_0 = 0\), with independent increments, with \(W_t - W_s \sim \mathcal N(0, (t - s) I)\) for \(s < t\), and with continuous paths.
Over a short interval of length \(h\), each coordinate receives a fresh Gaussian displacement with standard deviation \(\sqrt h\). The square root matters: splitting an interval into smaller pieces preserves the total variance, because independent variances add. Over \(n\) intervals of length \(h\), the variance is \(nh\). This is how many small, unbiased kicks accumulate into a visible spread of positions.
This project's earlier post derives Brownian motion from assumptions about how a process moves. A theorem of Nelson says that the only continuous-time process on the line with independent, symmetric, stationary increments that moves by \(o(t)\) in time \(t\) is Brownian motion with some diffusion constant, or a process that stays still.E. Nelson, Dynamical Theories of Brownian Motion (Princeton University Press, 1967), Theorem 5.1, quoted in full in the post Markov processes and Kolmogorov equations of this project. For the construction of the process and the Itô integral the series relies on B. Øksendal, Stochastic Differential Equations, 6th ed. (Springer, 2003), Chapters 2 to 5, and I. Karatzas and S. E. Shreve, Brownian Motion and Stochastic Calculus, 2nd ed. (Springer, 1991), Chapters 2 and 3. Under those assumptions, Gaussian noise follows from the requirements on the process. Here we take Brownian motion as given.
Definition 1.1.2 (Itô equation, strong solution). Let \(f: \mathbb R^d \times [0, 1] \to \mathbb R^d\) and \(g: [0, 1] \to \mathbb R\) be measurable. A strong solution of the stochastic differential equation
\[dx_t = f(x_t, t)\, dt + g(t)\, dW_t, \qquad x_0 \sim p_0,\]is a continuous adapted process with \(x_t = x_0 + \int_0^t f(x_s, s)\, ds + \int_0^t g(s)\, dW_s\) for all \(t\), the second integral being Itô's. An Itô integral of a deterministic integrand is a Gaussian random variable with mean zero and variance \(\int_0^t g(s)^2\, ds\), the Itô isometry; this is the only property of the integral the series needs beyond its existence.
To read the equation, imagine taking a small step:
\[x_{t+h} \approx x_t + f(x_t, t)\,h + g(t)\sqrt h\,\xi, \qquad \xi \sim \mathcal N(0, I),\]with a fresh \(\xi\) at each step. The drift \(f\) gives the predictable motion from the current position; \(g\) controls the random displacement. The integral equation defines the limit as the steps become arbitrarily small. An adapted solution uses only randomness available up to the current time. A strong solution is built from the specified initial point and Brownian motion; uniqueness means these determine a single path.
Theorem 1.1.3 (existence and uniqueness). If \(f\) is Lipschitz in \(x\) uniformly in \(t\), of linear growth, and \(g\) is bounded on \([0, 1]\), then the equation of Definition 1.1.2 has a unique strong solution for every initial law with finite second moment.Øksendal, Stochastic Differential Equations, Theorem 5.2.1; Karatzas and Shreve, Theorem 5.2.9. The hypothesis fails for the linear path of flow matching at \(t = 1\), where \(f_t = -1/(1 - t)\), which is why samplers for that path start at \(t = 1 - \epsilon\) or treat the endpoint by the formula of Proposition 1.1.5 rather than by the equation.
Theorem 1.1.4 (Itô's formula). If \(x_t\) solves the equation of Definition 1.1.2 and \(\phi: \mathbb R^d \times [0, 1] \to \mathbb R\) is \(C^{2, 1}\), then
\[d\phi(x_t, t) \;=\; \Bigl(\partial_t \phi + f \cdot \nabla \phi + \tfrac12 g^2 \Delta \phi\Bigr)(x_t, t)\, dt \;+\; g(t)\, \nabla\phi(x_t, t) \cdot dW_t .\]The last term has mean zero whenever \(\nabla\phi\) is bounded. The operator \(\mathcal L_t \phi = f \cdot \nabla\phi + \tfrac12 g^2 \Delta\phi\) in the drift is the generator of the process; Section 1.2 builds the Fokker–Planck equation from it.
The extra term \(\tfrac12 g^2\Delta\phi\) comes from the second-order term in a Taylor expansion. A Brownian displacement has size \(\sqrt h\), so its square has size \(h\) and contributes at the same order as the drift. In one dimension, take \(\phi(x) = x^2\) and \(dx = dW\). The formula gives \(d(x^2) = 2x\,dW + dt\): the position has no average drift, but its expected square grows at rate one. The extra term accounts for that growing spread.
left x against t from x_0 = 1.6 under the Ornstein-Uhlenbeck process: one Brownian path stepped by
x_{k+1} = x_k + f x_k h + g sqrt(h) xi_k at h = 1/20, 1/80 and 1/320, and eight more paths at
h = 1/20; at step 6 (t = 0.30) a particle at the kernel's mean 1.007 drifts by −0.15 and is
kicked with standard deviation 0.55, 3.6 times the drift (at h = 1/320: −0.010 and 0.14,
14 times); under it the step magnified on the kick's scale
right the chain's mean and variance against t by the recursion m <- (1 + f h) m, v <- (1 + f h)^2 v
+ g^2 h, as rings over the kernel's alpha_t x_0 and sigma_t^2: at t = 0.30, 1.065 and 0.587
against 1.007 and 0.604; at t = 1 the variance is 1.30 against 1.00 (1.015 at h = 1/320); the
chains at h = 1/20 and 1/80 end 0.25 and 0.05 from the chain at h = 1/320 (rms over 2,000 paths)
(the live figure needs JavaScript)
The forward process we need has a particularly simple form: its drift is linear in the current point, and its noise level changes with time. We can solve it explicitly. The result is the Gaussian kernel \(x_t = \alpha_t x_0 + \sigma_t z\) from the introduction, with the two SDE coefficients determined by the schedule.
For a fixed clean point \(x_0\), this kernel is a Gaussian centred at \(\alpha_t x_0\). Across the data distribution, we mix one such Gaussian for each possible clean point, so the noisy density \(p_t\) can still have many peaks. Knowing how to corrupt an individual sample therefore lets us draw noisy samples even when their overall density is unknown.
Proposition 1.1.5 (the Gaussian kernel as an SDE). Let \(\alpha, \sigma: [0, 1] \to \mathbb R\) be \(C^1\) with \(\alpha_0 = 1\), \(\sigma_0 = 0\), \(\alpha_t > 0\) on \([0, 1)\), and with \(\sigma_t/\alpha_t\) strictly increasing. Set
\[f_t \;=\; \frac{\dot\alpha_t}{\alpha_t}, \qquad g_t^2 \;=\; \alpha_t^2\, \frac{d}{dt}\Bigl(\frac{\sigma_t^2}{\alpha_t^2}\Bigr) \;=\; 2\sigma_t \dot\sigma_t - 2\sigma_t^2 \frac{\dot\alpha_t}{\alpha_t} \;=\; -\sigma_t^2\, \dot\lambda_t,\]where \(\lambda_t = \log(\alpha_t^2/\sigma_t^2)\) is the log signal-to-noise ratio. Then the solution of \(dx_t = f_t x_t\, dt + g_t\, dW_t\) started at \(x_0\) satisfies, for every \(t \in [0, 1)\),
\[x_t \;\overset{d}{=}\; \alpha_t x_0 + \sigma_t z, \qquad z \sim \mathcal N(0, I) \text{ independent of } x_0 .\]Proof. Divide by \(\alpha_t\) to cancel the drift: the product rule gives \(d(x_t/\alpha_t) = (g_t/\alpha_t)\,dW_t\), because \(d(1/\alpha_t) = -(\dot\alpha_t/\alpha_t^2)\,dt\). Integrating and multiplying back by \(\alpha_t\) gives
\[x_t = \alpha_t x_0 + \alpha_t\int_0^t \frac{g_s}{\alpha_s}\,dW_s.\]The integral has a deterministic integrand, so it is Gaussian with mean zero. Its variance per coordinate is
\[\int_0^t \frac{g_s^2}{\alpha_s^2}\,ds = \int_0^t \frac{d}{ds}\Bigl(\frac{\sigma_s^2}{\alpha_s^2}\Bigr)\,ds = \frac{\sigma_t^2}{\alpha_t^2},\]since \(\sigma_0 = 0\). Multiplying by \(\alpha_t\) gives variance \(\sigma_t^2\) per coordinate. The monotonicity of \(\sigma/\alpha\) is what makes \(g_t^2 \ge 0\). ∎
Dividing by \(\alpha_t\) removes the signal's changing scale. In these coordinates, the process only accumulates independent noise, whose variance is \(\sigma_t^2/\alpha_t^2\). This explains the monotonicity assumption: accumulated noise variance must increase. The coefficient \(g_t^2\) specifies how quickly to inject fresh variance while accounting for the rescaling.
The equality in distribution is for each fixed time. Reusing one \(z\) in \(\alpha_t x_0 + \sigma_t z\) at every time produces a different collection of paths from the SDE, which receives fresh kicks throughout. For training, we usually need only a noisy sample at a chosen time, so the kernel lets us generate it directly without simulating the intervening path.
left p_t of R2 at t = 0.10 under the variance-preserving process (α = 0.947, σ = 0.322) as a curve, its
three weighted components, and the kernel of the clean point x_0 = 1.6: the bell at 1.51 with
standard deviation 0.32, narrower than the component it sits in (0.43); the three bumps become
two at t = 0.142 and one at t = 0.201
right x against t: the three centre tracks α_t m_k, the track α_t x_0 with its ±σ_t band, 24 of the
2,000 SDE paths from x_0 (fresh kicks at h = 1/320), and at t = 0.10 the histogram of their
positions over the bell N(α_t x_0, σ_t²): cloud mean 1.510 and standard deviation 0.335 against
the bell's 1.515 and 0.322 (seed 29)
(the live figure needs JavaScript)
Example 1.1.6 (the three schedules). The proposition is applied to the schedules of the notation paragraph of the introduction. For the variance-preserving schedule, \(\alpha_t = e^{-B(t)/2}\) with \(B(t) = \int_0^t \beta_s\, ds\) and \(\sigma_t^2 = 1 - \alpha_t^2\), one has \(\dot\alpha_t/\alpha_t = -\beta_t/2\) and \(\sigma_t^2/\alpha_t^2 = e^{B(t)} - 1\), whose derivative is \(\beta_t e^{B(t)}\), so \(g_t^2 = \alpha_t^2 \beta_t e^{B(t)} = \beta_t\). The forward equation is the Ornstein–Uhlenbeck process \(dx = -\tfrac12 \beta_t x\, dt + \sqrt{\beta_t}\, dW\), which Song et al. call the VP SDE; DDPM's chain corresponds to \(\beta_t = 0.1 + 19.9\, t\), so that \(B(1) = 10.05\), \(\alpha_1 = e^{-5.025} = 0.0066\) and \(\lambda_1 = -10.05\).Song et al., "Score-based generative modeling through stochastic differential equations" (2021), Section 3.4 and Appendix B, where the chain of J. Ho, A. Jain and P. Abbeel, "Denoising diffusion probabilistic models", NeurIPS 2020 (arXiv 2006.11239), with \(\beta_i\) rising linearly from \(10^{-4}\) to \(0.02\) over \(T = 1000\) steps, is shown to be the Euler–Maruyama discretization of this equation with \(\beta_t = \bar\beta_{\min} + t(\bar\beta_{\max} - \bar\beta_{\min})\), \(\bar\beta_{\min} = 0.1\) and \(\bar\beta_{\max} = 20\). Section 3.1 states the correspondence as a proposition. The 150-step chain of the worked examples of Section 3 keeps the same range of \(\beta_i\) on \(150\) steps, so it discretizes \(\bar\beta_t = 0.015 + 2.985\,t\) with \(B(1) = 1.5075\) and ends at \(\alpha_1 = 0.468\): it is a different process, which Example 3.1.5 measures.
The name variance preserving refers to the balance \(\alpha_t^2 + \sigma_t^2 = 1\). If the data have unit covariance, shrinking them removes exactly as much covariance as the fresh noise adds. For other data, the covariance moves toward \(I\), as Proposition 1.2.6 will show.
For a variance-exploding schedule, \(\alpha_t = 1\), so \(f_t = 0\) and \(g_t^2 = d\sigma_t^2/dt\): the process is a time-changed Brownian motion, \(dx = \sqrt{d\sigma_t^2/dt}\, dW\), and with the clock \(\sigma = t\) of Karras et al. it is \(dx = \sqrt{2t}\, dW\). The signal keeps its original scale while noise accumulates around it. Reaching a noise-dominated state requires making that noise large relative to the data.
For the linear path \(\alpha_t = 1 - t\), \(\sigma_t = t\) of flow matching, \(f_t = -1/(1 - t)\) and \(\sigma_t^2/\alpha_t^2 = t^2/(1 - t)^2\) has derivative \(2t/(1 - t)^3\), so \(g_t^2 = 2t/(1 - t)\). The kernel reaches \(x_1 = z\) exactly: the signal coefficient becomes zero at a finite time. Both SDE coefficients blow up as \(t \to 1\), although the kernel itself has a well-defined limit.
On R2 at \(t = 0.3\) under the variance-preserving schedule, \(B(0.3) = 0.03 + 0.8955 = 0.9255\), \(\alpha_t = 0.6296\), \(\sigma_t = 0.7770\), and the variance of \(p_t\) is \(\alpha_t^2 \cdot 1.4044 + \sigma_t^2 = 1.160\), which is the number the figure of Section 1.2 reports.
So far the kernel takes us from clean data to time \(t\). To describe successive noising steps, we also need to start at an intermediate time \(s\). The next formula separates the noise already present at \(s\) from the fresh noise added between \(s\) and \(t\).
Definition 1.1.7 (transition kernel, semigroup). The transition kernel of the forward process from time \(s\) to time \(t > s\) is the law \(p_{t \mid s}(x_t \mid x_s)\) of \(x_t\) given \(x_s\). The kernels form a semigroup: \(p_{u \mid s} = \int p_{u \mid t}(\cdot \mid y)\, p_{t \mid s}(dy \mid \cdot)\) for \(s < t < u\), the Chapman–Kolmogorov equation.
Proposition 1.1.8 (the kernel in closed form). For the process of Proposition 1.1.5 and \(0 \le s < t < 1\),
\[p_{t \mid s}(x_t \mid x_s) \;=\; \mathcal N\bigl(x_t;\, \alpha_{t \mid s}\, x_s,\; \sigma_{t \mid s}^2 I\bigr), \qquad \alpha_{t \mid s} = \frac{\alpha_t}{\alpha_s}, \qquad \sigma_{t \mid s}^2 = \sigma_t^2 - \alpha_{t \mid s}^2\, \sigma_s^2 .\]Proof. By the proof of Proposition 1.1.5, \(x_t/\alpha_t - x_s/\alpha_s = \int_s^t (g_r/\alpha_r)\, dW_r\), a Gaussian independent of \(x_s\) with variance \(\sigma_t^2/\alpha_t^2 - \sigma_s^2/\alpha_s^2\) per coordinate. Multiply by \(\alpha_t\). The Chapman–Kolmogorov identity is then the statement that \(\alpha_{u \mid t}\alpha_{t \mid s} = \alpha_{u \mid s}\) and \(\alpha_{u \mid t}^2 \sigma_{t \mid s}^2 + \sigma_{u \mid t}^2 = \sigma_{u \mid s}^2\), both of which are immediate from the formulas. ∎
The subtraction in \(\sigma_{t \mid s}^2\) prevents us from counting old noise twice. Scaling \(x_s\) by \(\alpha_{t \mid s}\) carries forward noise of variance \(\alpha_{t \mid s}^2\sigma_s^2\). We add just enough independent noise to bring the total to \(\sigma_t^2\). Chapman–Kolmogorov says that taking two such steps gives the same transition law as taking one combined step.
This is the identity on which DDPM's reverse conditionals (Section 3.2) and the DDIM family (Section 5.1) rest, in the notation of Kingma, Salimans, Poole and Ho.D. P. Kingma, T. Salimans, B. Poole and J. Ho, "Variational diffusion models", NeurIPS 2021 (arXiv 2107.00630), Section 2, where \(\alpha_{t \mid s}\) and \(\sigma_{t \mid s}^2\) are introduced in this form and the signal-to-noise ratio \(\alpha_t^2/\sigma_t^2\) is made the primary object.
Example 1.1.9 (Proposition 1.1.8 on the DDPM chain). The chain of Ho, Jain and Abbeel moves from step \(i - 1\) to step \(i\) by \(x_i = \sqrt{1 - \beta_i}\, x_{i-1} + \sqrt{\beta_i}\, \varepsilon_i\). In the notation of the proposition this is the kernel with \(\alpha_{i \mid i-1} = \sqrt{1 - \beta_i}\) and \(\sigma_{i \mid i-1}^2 = \beta_i\), and the semigroup identity composes these into \(\alpha_{i \mid 0} = \sqrt{\bar\alpha_i}\) with \(\bar\alpha_i = \prod_{j \le i}(1 - \beta_j)\) and \(\sigma_{i \mid 0}^2 = 1 - \bar\alpha_i\): the schedule \(\alpha_i = \sqrt{\bar\alpha_i}\), \(\sigma_i = \sqrt{1 - \bar\alpha_i}\) is variance preserving. With \(T = 1000\) and \(\beta_i\) linear from \(10^{-4}\) to \(0.02\), \(\bar\alpha_T = 4.04 \times 10^{-5}\), so the last state is \(0.0064\, x_0\) plus noise of variance \(0.99996\). Section 3.1 returns to this.
Remark 1.1.10 (confusion: the clock is a convention). Three clocks appear in the literature for the same progression of distributions. One may use \(t \in [0, 1]\) with any monotone reparameterization; one may use the noise level \(\sigma\) itself when \(\alpha = 1\), as Karras et al. do; or one may use the log signal-to-noise ratio \(\lambda\), as Kingma et al. do. Changing the clock changes how quickly we traverse the distributions and the coefficients used to describe that motion.
Proposition 1.1.5 shows how to compare them: the law of \(x_t\) depends on \((\alpha_t, \sigma_t)\) through the ratio \(\sigma_t/\alpha_t\) and the overall scale \(\alpha_t\). Remove the scale by looking at \(x_t/\alpha_t = x_0 + (\sigma_t/\alpha_t) z\). At matching values of \(\sigma/\alpha\), a variance-preserving process and a variance-exploding process are related by a time-dependent magnification; Section 5.2 uses exactly this to show that DDIM is Euler's method.
The Fokker–Planck equation
So far we have followed individual random paths. Imagine running the process on many samples and making a histogram of their positions at each time. The Fokker–Planck equation describes how the density behind that histogram changes. We will derive it from Itô's formula, because the same calculation with a different drift will establish the reverse-time equation in Section 4.2.
Definition 1.2.1 (generator). The generator of the process \(dx = f(x, t)\, dt + g(t)\, dW\) is the differential operator \(\mathcal L_t \phi = f(\cdot, t) \cdot \nabla \phi + \tfrac12 g(t)^2 \Delta \phi\) on \(C^2\) functions. Its formal adjoint in \(L^2(dx)\) is \(\mathcal L_t^* p = -\nabla \cdot (f p) + \tfrac12 g^2 \Delta p\).
Think of \(\phi\) as a measurement of position: a coordinate, a squared distance, or a smooth window onto a small region. The generator tells us the expected rate of change of that measurement. Its adjoint describes the same change by evolving the density instead. Integration by parts is what transfers the derivatives from the measurement to the density in the proof below.
Theorem 1.2.2 (Fokker–Planck). Suppose the solution \(x_t\) of the equation of Definition 1.2.1 has a density \(p_t\) that is \(C^2\) in \(x\) and \(C^1\) in \(t\) and that \(f\) is \(C^1\). Then
\[\partial_t p_t(x) \;=\; -\nabla \cdot \bigl(f(x, t)\, p_t(x)\bigr) + \tfrac12 g(t)^2\, \Delta p_t(x) \;=\; \mathcal L_t^* p_t(x). \tag{1.1}\]Proof. Let \(\phi \in C_c^\infty(\mathbb R^d)\), a smooth function of compact support, independent of \(t\). By Itô's formula, \(\phi(x_t) - \phi(x_0) = \int_0^t \mathcal L_s \phi(x_s)\, ds + \int_0^t g(s) \nabla\phi(x_s) \cdot dW_s\), and the last term has mean zero because \(\nabla\phi\) is bounded. Taking expectations and differentiating in \(t\),
\[\frac{d}{dt} \int \phi\, p_t\, dx \;=\; \mathbb E\bigl[\mathcal L_t \phi(x_t)\bigr] \;=\; \int \Bigl(f \cdot \nabla\phi + \tfrac12 g^2 \Delta\phi\Bigr) p_t\, dx .\]Integrate by parts, once in the first term and twice in the second; the boundary terms vanish because \(\phi\) has compact support. The right side becomes \(\int \phi\, \bigl(-\nabla \cdot (f p_t) + \tfrac12 g^2 \Delta p_t\bigr)\, dx\). Since \(\partial_t p_t\) is continuous and the identity holds for every test function \(\phi\), the two integrands agree pointwise. ∎
The drift term \(-\nabla\cdot(f p_t)\) carries probability along the field \(f\): density rises where more probability arrives than leaves. The diffusion term \(\tfrac12 g^2\Delta p_t\) smooths the density. In one dimension, a peak curves downward, so its second derivative is negative and diffusion lowers it; a trough curves upward and diffusion fills it. Unbiased random motion can therefore change the density even though it has no preferred direction.
The equation also preserves total probability: the right side is a divergence, so \(\int p_t\, dx\) is constant when no probability escapes at the boundary. And it is linear in \(p\): a mixture of initial laws evolves into the same mixture of evolved laws. We can evolve each group of particles separately and then combine them with the original weights, which is why the running examples stay Gaussian mixtures.
Example 1.2.3 (the kernel solves it). For the process of Proposition 1.1.5 and a point mass at \(x_0\), the claimed solution is \(p_t(x) = \mathcal N(x; m_t, v_t I)\) with \(m_t = \alpha_t x_0\) and \(v_t = \sigma_t^2\). We can check it by tracking the Gaussian's centre and width. Differentiating \(\log p = -\|x - m\|^2/(2v) - \tfrac d2\log(2\pi v)\) gives
\[\partial_t p = -\dot m_t\cdot\nabla p + \tfrac12\dot v_t\,\Delta p.\]For a Gaussian, \(x\cdot\nabla p = m\cdot\nabla p - v\Delta p - d\,p\). Substituting this into the right side of (1.1) gives
\[-f_t\,\nabla\cdot(xp) + \tfrac12 g_t^2\Delta p = -f_t m_t\cdot\nabla p + \tfrac12(2f_t v_t + g_t^2)\,\Delta p.\]The two expressions agree when
\[\dot m_t = f_t m_t, \qquad \dot v_t = 2f_t v_t + g_t^2.\]The centre follows the drift. The variance changes for two reasons: the drift stretches or contracts the existing cloud, contributing \(2f_t v_t\), and fresh noise adds variance at rate \(g_t^2\). The factor two appears because variance scales as the square of a displacement. Substituting \(m_t = \alpha_t x_0\) and \(v_t = \sigma_t^2\) recovers exactly the coefficients of Proposition 1.1.5.
For a mixture \(p_0 = \sum_k w_k\,\mathcal N(m_k, s_k^2 I)\) such as R2 or R3, linearity gives
\[p_t = \sum_k w_k\,\mathcal N\bigl(\alpha_t m_k,\; (\alpha_t^2 s_k^2 + \sigma_t^2) I\bigr).\]Each component keeps its weight while its centre and width change. Under the variance-preserving schedule, the centres move toward the origin and the component variances approach one, so the initially separate bumps merge. This is the formula the figures evaluate.
The figure below solves (1.1) numerically on R2, with no knowledge of the closed form, and lays the result over it. The scheme is conservative by construction: each grid cell exchanges a flux with its neighbours, so the total mass cannot drift.
left p_t(x) on R2 at the chosen t: the exact mixture kernel (line) and the finite-difference
solution of the Fokker-Planck equation (rings); under it the current J and its drift and
diffusion parts as curves, or the rate d_t p = -d_x J and its two terms
right log10 KL(p_t || N(0, 1)) against t (solid) and the bound KL_0 e^{-B(t)} (dashed); or both
sides of the entropy-production identity; or the Fisher information and the loss floors
(the live figure needs JavaScript)
Example 1.2.4 (R2 by finite differences). In the figure's default view, \(t = 0.3\), the three bumps of R2 have merged into one hump of variance \(1.160\), and the largest difference between the finite-difference density and the exact one, over the 241 grid points, is \(0.0028\), against a peak density of \(0.353\). Reaching \(t = 1\) takes \(10{,}077\) explicit steps in the figure's run, most of them late, when \(\beta_t\) is large and the diffusion limit on the step is tight. Refining the grid to 481 and 961 points brings the largest difference to \(0.0014\) and \(0.0007\): the error halves with the spacing, as a first-order scheme's must, and the step count quadruples. The scheme is the following.
Algorithm 1.2.5 FP-EXPLICIT
(one dimension, flux form, explicit in time)
Picture: each cell i holds mass p_i dx; a flux F
crosses each face between cells; a cell changes by
the difference of the fluxes through its two faces,
so what leaves one cell enters the next.
cell i-1 cell i cell i+1
[ p_{i-1} ]----[ p_i ]----[ p_{i+1} ]
F_{i-1/2} F_{i+1/2}
------> ------>
F = (f x p)_upwind - (g^2/2) (p_i - p_{i-1})/dx
p_i <- p_i - dt (F_{i+1/2} - F_{i-1/2}) / dx
Input grid x_0 < ... < x_{N-1} with spacing dx; the
drift f(t) x and diffusion g(t)^2; a density p
at time t_0; a final time t_1.
Output the density at t_1.
1. t <- t_0
2. while t < t_1:
3. dt <- min(t_1 - t, 0.4 dx^2 / g(t)^2,
0.4 dx / max_i |f(t) x_i|)
4. for each face i = 1 .. N-1:
5. v <- f(t) (x_{i-1} + x_i) / 2
6. adv <- v p_{i-1} if v >= 0 else v p_i
7. F_i <- adv - (g(t)^2 / 2) (p_i - p_{i-1}) / dx
8. F_0 <- 0; F_N <- 0 (walls: no flux)
9. for each cell i: p_i <- p_i - dt (F_{i+1} - F_i)/dx
10. t <- t + dt
The two limits on the step in line 3 are the stability conditions of the explicit scheme: the diffusive one, \(dt \le \tfrac12 dx^2/g^2\) with a safety factor, and the advective one, \(dt \le dx/\max\lvert f x\rvert\). The upwind choice in line 6 takes the density from the cell the flow comes from, which keeps the scheme positive. The program below runs it on R2 and prints the figure's numbers together with the relative entropies of Section 1.4; it needs nothing beyond the standard library.
# The Fokker-Planck equation of the variance-preserving process on R2,
# solved by finite differences and compared with the exact kernel.
#
# d_t p = -d_x( f(t) x p ) + (1/2) g(t)^2 d_xx p,
# f = -beta/2, g^2 = beta, beta(t) = 0.1 + 19.9 t.
#
# Picture: the grid, its faces and the two fluxes of one cell.
#
# x: -6 ---+---+---+-- ... --+---+---+--- 6
# | | i | | | |
# F_{i} F_{i+1} (241 cells, dx = 0.05)
#
# Flux form on a grid of 241 points over [-6, 6]: the flux through the
# face between cells i-1 and i is the upwind advective part minus the
# diffusive part, and each cell loses the difference of its two fluxes,
# so mass is conserved to rounding. The time step obeys the diffusion
# limit 0.4 dx^2 / g^2 and the advection limit 0.4 dx / max|f x|.
import math
W, M, S = (0.45, 0.35, 0.20), (-1.4, 0.2, 1.6), (0.25, 0.15, 0.30)
N, L = 241, 6.0
grid = [-L + 2 * L * i / (N - 1) for i in range(N)]
dx = grid[1] - grid[0]
def alpha(t): return math.exp(-0.5 * (0.1 * t + 9.95 * t * t))
def sigma(t): return math.sqrt(1 - alpha(t) ** 2)
def beta(t): return 0.1 + 19.9 * t
def exact(t): # the mixture kernel at time t
a, s = alpha(t), sigma(t)
out = []
for x in grid:
p = 0.0
for w, m, sd in zip(W, M, S):
v = a * a * sd * sd + s * s
p += w * math.exp(-0.5 * (x - a * m) ** 2 / v) / math.sqrt(2 * math.pi * v)
out.append(p)
return out
def solve(p, t0, t1):
p, t, steps = list(p), t0, 0
while t < t1 - 1e-12:
g2, f = beta(t), -0.5 * beta(t)
dt = min(t1 - t, 0.4 * dx * dx / g2, 0.4 * dx / (abs(f) * L), 1e-2)
flux = [0.0] * (N + 1)
for i in range(1, N):
xf = 0.5 * (grid[i - 1] + grid[i])
vel = f * xf
adv = vel * p[i - 1] if vel >= 0 else vel * p[i]
flux[i] = adv - 0.5 * g2 * (p[i] - p[i - 1]) / dx
p = [p[k] - dt * (flux[k + 1] - flux[k]) / dx for k in range(N)]
t += dt
steps += 1
return p, steps
def trapz(y): return dx * (sum(y) - 0.5 * (y[0] + y[-1]))
def kl(p, q): return trapz([a * math.log(a / b) if a > 1e-300 and b > 1e-300 else 0.0 for a, b in zip(p, q)])
gauss = [math.exp(-0.5 * x * x) / math.sqrt(2 * math.pi) for x in grid]
p0 = exact(0.0)
kl0 = kl(p0, gauss)
print(f"t = 0.0 KL(p_t || N(0,1)) = {kl0:.4f} mass = {trapz(p0):.6f}")
p, steps = p0, 0
for t_prev, t_next in ((0.0, 0.3), (0.3, 1.0)):
p, n = solve(p, t_prev, t_next)
steps += n
ex = exact(t_next)
err = max(abs(a - b) for a, b in zip(p, ex))
B = 0.1 * t_next + 9.95 * t_next ** 2
print(f"t = {t_next:.1f} max |fd - exact| = {err:.4f} mass = {trapz(p):.6f} "
f"KL = {kl(ex, gauss):.4f} bound {kl0:.4f} e^-B = {kl0 * math.exp(-B):.5f} steps so far {steps}")
t = 0.0 KL(p_t || N(0,1)) = 0.7124 mass = 1.000000
t = 0.3 max |fd - exact| = 0.0028 mass = 1.000000 KL = 0.0208 bound 0.7124 e^-B = 0.28236 steps so far 923
t = 1.0 max |fd - exact| = 0.0052 mass = 1.000000 KL = 0.0000 bound 0.7124 e^-B = 0.00003 steps so far 10047
The step count differs slightly from the figure's because the figure stops at sixty intermediate times to store the density; the error at \(t = 0.3\) and the entropies are the same.
Proposition 1.2.6 (moments). Under the process of Proposition 1.1.5, the mean and covariance of \(p_t\) are \(m_t = \alpha_t m_0\) and \(C_t = \alpha_t^2 C_0 + \sigma_t^2 I\), where \(m_0\) and \(C_0\) are those of \(p_0\). This is immediate from \(x_t = \alpha_t x_0 + \sigma_t z\) and gives the variance \(1.160\) of Example 1.1.6 in one line; the Fokker–Planck equation gives the same by multiplying (1.1) by \(x\) and \(x x^\top\) and integrating by parts, which yields \(\dot m_t = f_t m_t\) and \(\dot C_t = 2 f_t C_t + g_t^2 I\), the equations of Example 1.2.3 for a general law. The total variance of R2 under the variance-preserving schedule therefore moves monotonically from \(1.404\) to \(1\): the process contracts laws whose variance exceeds one and inflates the others.
Remark 1.2.7 (confusion: forward and backward equations, and the backward process). The companion post of this project treats two equations: the backward Kolmogorov equation, \(-\partial_s u = \mathcal L_s u\), for the function \(u(x, s) = \mathbb E[\phi(x_t) \mid x_s = x]\), and the forward equation, (1.1), for the density. The words "forward" and "backward" there refer to the direction in which each equation is solved in time. The "reverse" or "backward" process of Section 4 is a third thing: a new stochastic process, running from \(t = 1\) to \(0\), whose density at each time is \(p_t\). Its own Fokker–Planck equation is a forward equation for a different drift. The three must be kept apart; the series writes "reverse process" for the third and never uses "backward" for it.
The continuity equation and the probability current
The Fokker–Planck equation also admits a deterministic interpretation. Its diffusion term can be written as a flow of probability from one region to another. This simple rearrangement leads to the probability-flow ODE and, later, to flow matching.
Definition 1.3.1 (continuity equation, current). A family of densities \(p_t\) and a velocity field \(v_t(x)\) satisfy the continuity equation if \(\partial_t p_t + \nabla \cdot (p_t v_t) = 0\). The vector field \(J_t = p_t v_t\) is the probability current: the amount of probability crossing a unit surface per unit time.
In one dimension, integrate over an interval \([a,b]\) to get
\[\frac{d}{dt}\int_a^b p_t(x)\,dx = J_t(a) - J_t(b).\]The probability inside changes by what enters at the left minus what leaves at the right. This is the continuous version of the flux update in Algorithm 1.2.5. The product \(p_t v_t\) accounts for both how much probability is present and how fast it moves.
Proposition 1.3.2 (the diffusion term is a transport). Equation (1.1) is the continuity equation with the velocity
\[v_t(x) \;=\; f(x, t) - \tfrac12 g(t)^2\, \nabla \log p_t(x). \tag{1.2}\]Proof. Since \(\nabla p_t = p_t \nabla \log p_t\) wherever \(p_t > 0\), the diffusion term is \(\tfrac12 g^2 \Delta p_t = \nabla \cdot (\tfrac12 g^2 \nabla p_t) = \nabla \cdot (p_t \cdot \tfrac12 g^2 \nabla \log p_t)\). Substituting into (1.1) gives \(\partial_t p_t = -\nabla \cdot \bigl(p_t\, (f - \tfrac12 g^2 \nabla \log p_t)\bigr)\). ∎
Why does unbiased noise produce a directed current? Across a boundary between crowded and sparse regions, more particles are available to cross from the crowded side. Their individual kicks are unbiased, but their net movement carries probability toward lower density. That current is \(-\tfrac12 g^2\nabla p_t\). Dividing by the local density turns it into a velocity, \(-\tfrac12 g^2\nabla\log p_t\).
This is why the score, \(\nabla\log p_t\), appears. It points toward increasing log density, and its negative supplies the spreading part of the forward velocity. Section 2 will show how to estimate it from samples.
left p_t of R2 at t = 0.3 under the variance-preserving process, a boundary at b = 2.0, and under it
the window b ± 0.99 magnified with the jumps of every tenth particle that crosses b in one step
of h = 1/100: 731 of 100,000 cross from left to right and 414 from right to left (738 and 422
expected), a net of 317 toward the sparse side, against n h times the diffusion part of the
current, −½g²∂_x p_t = 0.316, which gives 316
right the expected crossings against h on log-log axes: both directions, their sum, which grows like
√h (563 at h = 1/400), and their difference, which grows like h (79 at h = 1/400), with the
first-order count n J h dashed
(the live figure needs JavaScript)
Theorem 1.3.3 (a flow with the same marginals). Let \(p_t\) solve (1.1) with \(p_t > 0\) and let \(v_t\) be the velocity (1.2), assumed locally Lipschitz in \(x\). If \(X_0 \sim p_0\) and \(\dot X_t = v_t(X_t)\), then \(X_t \sim p_t\) for every \(t\).
Proof. For a test function \(\phi\), the law \(q_t\) of \(X_t\) satisfies \(\frac{d}{dt}\int \phi\, q_t = \mathbb E[\nabla\phi(X_t) \cdot v_t(X_t)] = \int \nabla\phi \cdot v_t\, q_t\), which after integration by parts says that \(q_t\) solves the continuity equation \(\partial_t q + \nabla \cdot (q v_t) = 0\) in the weak sense, with \(q_0 = p_0\). By Proposition 1.3.2, \(p_t\) solves the same linear equation with the same initial condition, and for a locally Lipschitz field the weak solution is unique, being the push-forward of \(p_0\) by the flow of \(v\). Hence \(q_t = p_t\). Section 6.1 restates this as the correspondence between flows and the continuity equation and proves the uniqueness it uses. ∎
The theorem compares distributions at each time, called marginals. If we photograph a cloud of SDE particles and a cloud of ODE particles at the same time, both photographs sample the same density. Tracking an individual particle through successive photographs reveals different paths. The ODE needs randomness only in its starting point; once that point is chosen, its path is determined. Sections 4 to 6 use this freedom to construct different samplers for the same model.
An especially clear case is the variance-preserving process already at \(\mathcal N(0,I)\). Its score is \(-x\), so (1.2) gives \(v_t(x) = -\tfrac12\beta_t x + \tfrac12\beta_t x = 0\). Every ODE particle stays still. The SDE particles continue to move, with inward drift balancing the spread from their random kicks. Both clouds retain exactly the same Gaussian density. The second figure of Section 4.2 draws this case when its data law is set to \(\mathcal N(0, I)\).
Example 1.3.4 (the current on R2). For the variance-preserving process, \(J_t = p_t\, (-\tfrac12 \beta_t x) - \tfrac12 \beta_t \nabla p_t\): a drift part that carries mass toward the origin, in proportion to \(x\), and a diffusion part that carries it from the bumps outward. In the figure's default view at \(t = 0.3\) the two nearly cancel across the middle of the hump and the net current points inward at the two flanks, where \(p_t\) still has more mass than \(\mathcal N(0, 1)\) would; the strip under the density draws the two parts as curves, dashed and dotted, each several times larger than their sum across the middle of the hump, where the sum is close to zero. At \(t = 0.3\) the net current peaks at \(0.400\) while its drift and diffusion parts peak at \(0.924\) and \(0.675\).
At \(t = 1\), where \(p_t\) is within a relative entropy of \(10^{-5}\) of \(\mathcal N(0, 1)\), the net current is small. At exact equilibrium it vanishes everywhere: drift and diffusion cancel pointwise, \(\tfrac12\beta x\, p + \tfrac12 \beta \nabla p = 0\), whose only normalized solution is the standard Gaussian.
Example 1.3.5 (the velocity on R1). For the two-point law R1 under a variance-exploding process, Section 2.2 computes \(\nabla \log p_\sigma(x) = (\tanh(x/\sigma^2) - x)/\sigma^2\), so with the noise level as the clock, \(g^2 = 2\sigma\), the velocity (1.2) is \(v_\sigma(x) = (x - \tanh(x/\sigma^2))/\sigma\). Its flow lines, drawn in the right panel of the first figure of Section 2.2 with the clock reversed, fan out from \(\pm 1\) at small \(\sigma\) and never meet.
Remark 1.3.6 (takeaway: one family of densities, many processes). During reversal, we can choose how much fresh noise to add. More noise spreads the particles more, so the drift must compensate if we want to keep the same densities. The score supplies that compensation.
Algebraically, write \(\tfrac12 g^2 \Delta p = \tfrac{1 + \lambda^2}{2} g^2 \Delta p - \tfrac{\lambda^2}{2} g^2 \Delta p\). Turn the first piece into transport as in Proposition 1.3.2. The second becomes an ordinary diffusion when time is reversed. This gives a reverse process with drift \(f - \tfrac{1 + \lambda^2}{2} g^2 \nabla \log p\) and noise \(\lambda g\), for every \(\lambda \ge 0\), all with the marginals \(p_t\). These are the processes of Theorem 4.2.3, with \(\lambda = 0\) the flow and \(\lambda = 1\) Anderson's SDE. The second figure of Section 4.2 draws four of them on R2 from the same noise.
Equilibrium and the approach to it
As a variance-preserving process adds more noise, it loses information about its starting distribution and approaches \(\mathcal N(0, I)\). A sampler starts from that Gaussian, so we need to know how close the forward process has come to it. Stopping too early leaves a mismatch before sampling even begins.
The kernel already suggests why this happens: as \(B(t)\) grows, \(\alpha_t = e^{-B(t)/2}\) shrinks the contribution of the data and \(\sigma_t^2 = 1-e^{-B(t)}\) approaches one. To turn that picture into a quantitative guarantee, we need a measure of the discrepancy between distributions. The bound below uses relative entropy and shows that it decays exponentially in \(B(t)\). This will supply the first of the three error terms in the convergence guarantees of Section 10.
Definition 1.4.1 (relative entropy, relative Fisher information). For densities \(p\) and \(q\) on \(\mathbb R^d\) with \(p \ll q\),
\[\mathrm{KL}(p \,\|\, q) = \int p \log \frac{p}{q}\, dx, \qquad I(p \,\|\, q) = \int p\, \Bigl\|\nabla \log \frac{p}{q}\Bigr\|^2 dx .\]The relative Fisher information with respect to Lebesgue measure, \(I(p) = \int p\, \|\nabla \log p\|^2\), is the expected squared norm of the score.
The ratio \(p/q\) tells us where \(p\) places more or less density than the reference \(q\). Relative entropy averages its logarithm under \(p\), giving a nonnegative discrepancy that is zero only when the laws agree. Relative Fisher information measures how sharply that log ratio varies with position. The next identity connects the remaining discrepancy to the rate at which diffusion removes it.
Proposition 1.4.2 (entropy production under the variance-preserving process). Let \(p_t\) be the law of the Ornstein–Uhlenbeck process \(dx = -\tfrac12 \beta_t x\, dt + \sqrt{\beta_t}\, dW\) and \(\gamma = \mathcal N(0, I)\). Then
\[\frac{d}{dt}\, \mathrm{KL}(p_t \,\|\, \gamma) \;=\; -\tfrac12\, \beta_t\, I(p_t \,\|\, \gamma).\]Proof. Since \(\nabla \log \gamma = -x\), the Fokker–Planck equation (1.1) with \(f = -\tfrac12 \beta_t x\) and \(g^2 = \beta_t\) reads \(\partial_t p = \tfrac12 \beta_t \nabla \cdot (x p + \nabla p) = \tfrac12 \beta_t\, \nabla \cdot \bigl(p\, \nabla \log (p/\gamma)\bigr)\). Differentiating the relative entropy, the term from \(\partial_t p\) inside the logarithm integrates to \(\frac{d}{dt}\int p = 0\), so
\[\begin{aligned} \frac{d}{dt}\, \mathrm{KL}(p_t \,\|\, \gamma) &= \int \partial_t p\, \log\frac{p}{\gamma}\, dx \\ &= \tfrac12 \beta_t \int \nabla \cdot \bigl(p\, \nabla \log \tfrac{p}{\gamma}\bigr) \log \tfrac{p}{\gamma}\, dx \\ &= -\tfrac12 \beta_t \int p\, \bigl\|\nabla \log \tfrac{p}{\gamma}\bigr\|^2 dx, \end{aligned}\]by one integration by parts, valid when \(p\) decays fast enough at infinity, as every Gaussian mixture does. ∎
The right side is nonpositive, so the process continually reduces relative entropy. Its rate is controlled by the difference between the two score fields, since \(\nabla\log(p_t/\gamma) = \nabla\log p_t + x\). At equilibrium this difference vanishes. To bound how long the approach takes, we still need to relate the rate of decrease to the discrepancy that remains. The next inequality does exactly that: \(I(p\|\gamma) \ge 2\,\mathrm{KL}(p\|\gamma)\).
The figure of Section 1.2 checks the identity in its second view: differencing the exact relative entropy at \(t = 0.3\) gives \(dKL/dt = -0.2135\), and \(-\tfrac12 \beta_t I(p_t \| \gamma)\) by quadrature gives \(-0.2135\) with \(I(p_t \| \gamma) = 0.0703\); the two curves coincide at every \(t\).
Theorem 1.4.3 (Bakry–Émery for the Gaussian). Every density \(p\) with finite relative Fisher information satisfies the logarithmic Sobolev inequality \(\mathrm{KL}(p \,\|\, \gamma) \le \tfrac12 I(p \,\|\, \gamma)\). Consequently, for the process of Proposition 1.4.2,
\[\mathrm{KL}(p_t \,\|\, \gamma) \;\le\; e^{-B(t)}\, \mathrm{KL}(p_0 \,\|\, \gamma), \qquad B(t) = \int_0^t \beta_s\, ds .\]Proof. The inequality is Gross's logarithmic Sobolev inequality for the Gaussian measure, in the form Bakry and Émery proved for every measure whose potential has Hessian bounded below by the identity.L. Gross, "Logarithmic Sobolev inequalities", American Journal of Mathematics 97 (1975), 1061–1083; D. Bakry and M. Émery, "Diffusions hypercontractives", Séminaire de Probabilités XIX, Lecture Notes in Mathematics 1123 (Springer, 1985), 177–206. The semigroup treatment used here is D. Bakry, I. Gentil and M. Ledoux, Analysis and Geometry of Markov Diffusion Operators (Springer, 2014), Chapter 5, where the inequality for \(\gamma\) reads \(\mathrm{Ent}_\gamma(f^2) \le 2 \int \|\nabla f\|^2 d\gamma\), which with \(f^2 = p/\gamma\) is the statement above. Given it, Proposition 1.4.2 yields \(\frac{d}{dt}\mathrm{KL}(p_t \| \gamma) \le -\beta_t\, \mathrm{KL}(p_t \| \gamma)\), and Grönwall's inequality gives the bound. ∎
The effective duration here is \(B(t)\), the accumulated noise rate. For finite initial relative entropy, each extra unit of \(B\) reduces this upper bound by a factor of \(e^{-1}\). Merely calling the endpoint \(t = 1\) says nothing about its proximity to Gaussian noise; the schedule must accumulate enough \(B\) to make the remaining discrepancy small.
Example 1.4.4 (Theorem 1.4.3 on R2). On R2 under DDPM's schedule the relative entropy of the data law with respect to \(\gamma\) is \(0.712\). The figure's right panel plots \(\log_{10} \mathrm{KL}(p_t \| \gamma)\) against the bound. At \(t = 0.3\) the actual value is \(0.0208\) and the bound is \(0.282\); at \(t = 0.5\), \(0.0025\) against \(0.056\); at \(t = 1\), where \(B(1) = 10.05\), the bound is \(0.712\, e^{-10.05} = 3.1 \times 10^{-5}\) and the actual value is below the resolution of the grid. The bound is loose by an order of magnitude along the way, because R2 is far from the extremal laws of the logarithmic Sobolev inequality, and it is exactly what is needed: a guarantee that depends on \(p_0\) only through one number. The figure's second view draws the ratio of the two: \(0.074\) at \(t = 0.3\) and \(0.045\) at \(t = 0.5\). ◁
Remark 1.4.5 (warning: the prior mismatch). A variance-preserving sampler starts its reverse process from \(\gamma\). The ideal starting law is \(p_1\), the actual endpoint of the forward process. Replacing it by a Gaussian introduces an error even if the reverse dynamics are otherwise exact. The bound is
\[\mathrm{KL}(p_1 \| \gamma) \le e^{-B(1)}\,\mathrm{KL}(p_0 \| \gamma).\]Applying the same exact reverse transition to both starting laws cannot increase their relative entropy, by the data-processing inequality. This starting discrepancy therefore bounds its contribution to the final error; Section 10.1 includes it as a separate term. With \(B(1) = 10.05\), the bound is \(4.3 \times 10^{-5}\) times the initial relative entropy, whenever that initial quantity is finite.
On the 150-step chain of Example 1.1.6, which stops at \(\bar\alpha_T = 0.219\), substantial signal remains. The average conditional divergence \(\mathrm{KL}(q(x_T \mid x_0) \| \mathcal N(0, I))\) is \(70.05\) nats per image on FashionMNIST at \(28 \times 28\), \(0.09\) per pixel, against \(0.0109\) for the thousand-step chain; Example 3.1.5 measures the effect of changing the sampler's starting law. This average includes the information the noisy point retains about its clean source, so it is an upper bound on the marginal mismatch \(\mathrm{KL}(p_1\|\gamma)\).
For a variance-exploding schedule there is no equilibrium: the cloud keeps spreading as noise increases. The sampler starts from \(\mathcal N(0, \sigma_{\max}^2 I)\). For a centred data law with covariance \(C\), the mismatch of the Gaussian with covariance \(C + \sigma_{\max}^2 I\) from this starting law is \(\tfrac14 \mathrm{tr}(C^2)/\sigma_{\max}^4\) to leading order, of the order of \(d\,(s/\sigma_{\max})^4\) for data of standard deviation \(s\) per coordinate. This compares the Gaussian approximations; the full noised law may also differ in shape. Karras et al. take \(\sigma_{\max} = 80\) for data scaled to \(s = 0.5\), so the leading term is below \(d \times 10^{-9}\): for a \(64 \times 64\) colour image, a few parts in a million.T. Karras, M. Aittala, T. Aila and S. Laine, "Elucidating the design space of diffusion-based generative models", NeurIPS 2022 (arXiv 2206.00364), Table 1: \(\sigma_{\min} = 0.002\), \(\sigma_{\max} = 80\), \(\sigma_{\mathrm{data}} = 0.5\). The expansion is the second-order term of \(\mathrm{KL}(\mathcal N(0, \Sigma_1) \| \mathcal N(0, \Sigma_2)) = \tfrac12[\mathrm{tr}(\Sigma_2^{-1}\Sigma_1) - d - \log\det(\Sigma_2^{-1}\Sigma_1)]\) in the eigenvalues of \(C/\sigma_{\max}^2\).
Remark 1.4.6 (takeaway: the score's size is a Fisher information). For the variance-exploding process in the clock \(\tau = \sigma^2/2\), (1.1) is the heat equation \(\partial_\tau p = \Delta p\). The computation of Proposition 1.4.2 with \(q\) the Lebesgue measure gives de Bruijn's identity, \(\frac{d}{d\tau}\int p\log p = -I(p)\). Thus the entropy \(H(p) = -\int p\log p\) grows at rate \(I(p)\). Large log-density gradients mean that smoothing increases entropy quickly.
The same quantity enters the irreducible error of score regression in Section 2.3. There we train on the known target \(-z/\sigma\), but the noisy observation alone does not determine which \(z\) was added. Even the best predictor leaves a mean squared error of \(d/\sigma^2 - I(p_\sigma)\) per sample. This is the uncertainty that remains about the target after observing the noisy point.The identity goes back to N. M. Blachman, "The convolution inequality for entropy powers", IEEE Transactions on Information Theory 11 (1965), 267–271, who attributes it to N. G. de Bruijn; A. J. Stam, "Some inequalities satisfied by the quantities of information of Fisher and Shannon", Information and Control 2 (1959), 101–112, is the earlier source. The loss floor is Proposition 2.3.5. The third view of the figure of Section 1.2 expresses the residual errors in the units of the noise and of the clean point, and draws both against \(t\) on R2.
Where this is used
The results above do different jobs in a sampler. The Gaussian kernel lets us make training pairs, the continuity equation gives a deterministic sampling path, and the equilibrium bound controls the error in the starting distribution. The table shows where each result returns, and offers a way back if you skip ahead.
| object | where it is used |
|---|---|
| the kernel \(x_t = \alpha_t x_0 + \sigma_t z\) and its coefficients \(f_t = \dot\alpha_t/\alpha_t\), \(g_t^2 = -\sigma_t^2 \dot\lambda_t\) (Proposition 1.1.5) | the forward SDE of Section 4.1, the regression targets of Section 2.4, the dictionary of Section 6.3 |
| the composed kernel \(\alpha_{t\mid s}\), \(\sigma_{t\mid s}\) (Proposition 1.1.8) | DDPM's reverse conditionals in Section 3.2, the DDIM family in Section 5.1 |
| the Fokker–Planck equation (Theorem 1.2.2) and its linearity | Anderson's theorem in Section 4.2, the mixture formulas of every figure |
| the continuity form with velocity \(f - \tfrac12 g^2 \nabla\log p\) (Proposition 1.3.2) | the probability-flow ODE of Section 4.3, the family of reverse processes, the marginal velocity of Section 6.2 |
| the entropy decay \(e^{-B(t)}\) (Theorem 1.4.3) | the prior term of the convergence bounds of Section 10.1 |
| the empirical and the population score of a mixture | the regimes of Section 10.3 |
| the posterior mean over a training set (Example 2.1.6 with \(s_k = 0\)) | the memorizing minimizer of Sections 2.5, 3.3 and 10.3 |