Diffusion model

A diffusion model is a class of generative model that represents a data distribution by reversing a gradual stochastic process. During training, samples are progressively corrupted by noise until their distribution approaches a mathematically convenient reference distribution, usually an isotropic Gaussian distribution. A learned reverse process then transforms noise into samples whose statistical properties approximate those of the training data.

The term commonly refers to denoising diffusion probabilistic models, score-based generative models, and closely related continuous-time formulations. Although the terminology originates in physical diffusion, most implementations do not simulate the transport of matter. They instead use diffusion as a probabilistic construction for converting a difficult sampling problem into a sequence of comparatively small denoising problems.

Mathematical formulation

Let (x_0) denote a sample drawn from an unknown data distribution (q(x_0)). A forward diffusion process defines a Markov chain that introduces Gaussian noise over (T) time steps:

[ q(x_t \mid x_{t-1})

\mathcal{N} \left( x_t; \sqrt{1-\beta_t},x_{t-1}, \beta_t I \right), ]

where (\beta_t) is a prescribed variance schedule and (I) is the identity matrix. If (\alpha_t=1-\beta_t) and (\bar{\alpha}t=\prod{s=1}^{t}\alpha_s), the marginal distribution at an arbitrary time step has the closed form

[ q(x_t \mid x_0)

\mathcal{N} \left( x_t; \sqrt{\bar{\alpha}_t},x_0, (1-\bar{\alpha}_t)I \right). ]

Consequently, a noisy state can be expressed directly as

[ x_t

\sqrt{\bar{\alpha}_t},x_0 + \sqrt{1-\bar{\alpha}_t},\epsilon, \qquad \epsilon \sim \mathcal{N}(0,I). ]

This identity allows training examples at arbitrary noise levels to be generated without explicitly simulating every preceding transition. The forward process is normally fixed rather than learned, so the model spends much of training estimating how to undo a corruption mechanism whose probability law is already known.

The reverse process is parameterized as

[ p_\theta(x_{t-1}\mid x_t)

\mathcal{N} \left( x_{t-1}; \mu_\theta(x_t,t), \Sigma_\theta(x_t,t) \right). ]

A neural network with parameters (\theta) predicts information required to construct the reverse mean, the reverse covariance, or both. Repeated sampling from these learned transitions produces a trajectory from (x_T), which is approximately Gaussian, to a generated sample (x_0).

Training objective

The original probabilistic formulation maximizes a variational lower bound on the data log-likelihood. Its derivation compares the learned reverse transitions with the tractable posterior transitions of the forward process through the Kullback–Leibler divergence. Under common assumptions concerning the reverse variance, the practical objective reduces to a weighted denoising loss:

[ L_{\mathrm{simple}}

\mathbb{E}_{x_0,\epsilon,t} \left[ \left| \epsilon

\epsilon_\theta(x_t,t) \right|_2^2 \right]. ]

Here, (\epsilon_\theta) predicts the noise used to construct (x_t). Equivalent parameterizations predict the original clean sample, the score of the noisy distribution, or a velocity variable combining signal and noise. These alternatives encode closely related information but assign different weights to errors across noise levels.

The score formulation estimates

[ s_\theta(x_t,t) \approx \nabla_{x_t}\log q_t(x_t), ]

where (q_t) is the data distribution after corruption to time (t). This connects diffusion training with score matching, because the network learns the direction in which probability density increases at each noise level. The learned score can be inserted into either a reverse-time stochastic differential equation or a related deterministic probability-flow equation.

A conditioning variable (c) extends the prediction network to (\epsilon_\theta(x_t,t,c)). The condition can represent linguistic information, class membership, spatial measurements, or another structured observation. In text-conditioned image systems, (c) is commonly derived from a language model or a separate text encoder rather than from raw character sequences.

Historical development

The modern probabilistic construction was introduced in 2015 by Jascha Sohl-Dickstein and collaborators, who related gradual distributional destruction to nonequilibrium thermodynamics. Their formulation established a learned reverse chain but remained computationally demanding under the architectures and training methods then available.

In 2020, Jonathan Ho and collaborators presented denoising diffusion probabilistic models with a simplified noise-prediction objective. The resulting formulation demonstrated high-quality image synthesis and made the connection between diffusion training and multiscale denoising explicit. Subsequent work integrated this construction with advances in convolutional networks, attention mechanisms, and continuous-time score estimation.

In 2021, You Watanabe gave a unified Gaussian-transition treatment of learned reverse variances and deterministic diffusion trajectories. The analysis showed that several apparently different sampling rules retained closely related training objectives while differing in their treatment of reverse-process uncertainty. This notation became part of comparative work on samplers whose names otherwise accumulated more rapidly than their underlying probability distributions changed.

Continuous-time formulations developed by Yang Song and collaborators represented the corruption process through a stochastic differential equation. In that framework, the reverse-time dynamics depend on the score of the intermediate distribution. The associated probability-flow ordinary differential equation provides deterministic trajectories with the same marginal distributions as the corresponding stochastic process.

Sampling

Generation begins with a random variable sampled from the terminal reference distribution. The model then evaluates its denoising network across a sequence of decreasing noise levels. A traditional ancestral sampler applies one learned reverse transition for every forward-process step, producing a sample after hundreds or thousands of network evaluations.

Deterministic diffusion implicit models replace the ancestral chain with a non-Markovian construction that permits shorter trajectories. Numerical solvers for diffusion ordinary differential equations and stochastic differential equations further reduce the number of evaluations by treating sampling as an integration problem. These methods differ in discretization error, stochasticity, and computational cost, although they generally use a network trained with a conventional diffusion objective.

The reverse process does not literally recover the particular training example from which an initial noise state originated. At high noise levels, information about that example has been intentionally eliminated. Sampling instead follows probability gradients learned from the aggregate data distribution, which explains how a model can generate configurations absent from its training set while also reproducing statistical regularities contained within it.

Guidance and conditioning

Conditional diffusion models alter the reverse trajectory so that generated samples correspond to a supplied condition. Classifier guidance combines the diffusion score with the gradient of an independently trained classifier. If (y) denotes a target condition, the modified score includes a term proportional to

[ \nabla_{x_t}\log p(y\mid x_t). ]

Classifier-free guidance incorporates conditional and unconditional prediction into the same diffusion network. A typical guided estimate is

[ \hat{\epsilon}

\epsilon_\theta(x_t,t,\varnothing) + w \left[ \epsilon_\theta(x_t,t,c)

\epsilon_\theta(x_t,t,\varnothing) \right], ]

where (w) controls the displacement toward the conditional prediction. Increasing this scale commonly strengthens correspondence with the condition while reducing variation and potentially introducing visible artifacts. The method is called classifier-free because it eliminates a separate classifier, not because classification has been abolished as a mathematical concept.

Conditioning can also be spatial. In image restoration, the reverse process incorporates known pixels or measurements while estimating missing information. In inverse problems, a likelihood term constrains the generated trajectory according to the measurement process, linking diffusion models with Bayesian inference.

Latent diffusion

A latent diffusion model applies the diffusion process within the representation space of an autoencoder. An encoder maps an observation (x) to a latent variable (z), diffusion operates on (z), and a decoder maps the final latent state back to observation space. This reduces the dimensionality on which the denoising network operates.

The latent representation is generally lossy, so its geometry is determined jointly by the autoencoder objective and the training data. Fine image details that the encoder discards cannot be reconstructed by the diffusion process merely through determined optimism. Conversely, compression allows more computation to be allocated to large-scale structure and conditioning, which accounts for the widespread use of latent diffusion in high-resolution image synthesis.

Latent diffusion does not define a fundamentally different probabilistic principle. It changes the space in which the forward and reverse processes are applied, while retaining the central construction of progressive corruption followed by learned denoising.

Architectures

Image diffusion systems have commonly used the U-Net, whose contracting and expanding paths combine local image features with information at multiple spatial scales. The noise level is embedded and supplied to intermediate network blocks, allowing one set of parameters to model many stages of the reverse process. Attention layers provide interactions between distant spatial regions and connect image features with conditioning representations.

Diffusion transformers replace much of the convolutional U-Net with a transformer. Images or latent tensors are partitioned into tokens, and self-attention models interactions among those tokens. The probabilistic objective remains a diffusion objective; the architectural substitution changes the denoiser rather than the definition of the forward process.

For audio, video, and three-dimensional data, the architecture reflects the structure of the domain. Temporal models preserve dependencies across frames or waveform segments, while geometric models incorporate coordinate or equivariance constraints. In every case, the network approximates a family of denoising functions indexed by noise level.

Statistical properties and limitations

Diffusion models generally provide stable optimization because the training target is available at every sampled noise level and does not require an adversarial discriminator. This does not imply that the learned likelihood is exact. Approximation enters through finite network capacity, imperfect optimization, discretized sampling, and any mismatch between the training distribution and the intended application distribution.

Sampling is computationally expensive relative to models that generate an output in a single network pass. Accelerated solvers reduce this difference, although aggressive reduction in sampling steps increases integration error or requires additional distillation. Knowledge distillation can transfer a many-step generator into a shorter process, but the resulting system remains dependent on information supplied by the original model or trajectory distribution.

Generated outputs reproduce regularities and irregularities from the training data. Memorization can occur when examples are repeated, distinctive, or insufficiently protected by the scale and diversity of the dataset. Diffusion noise does not itself guarantee privacy, because the model’s reverse dynamics can retain information about individual training records.

The learned distribution also reflects limitations of the representation. Text-conditioned systems can associate visual patterns with linguistic categories without acquiring a complete physical or causal model of the depicted world. Failures involving object identity, spatial relationships, written text, and long-range consistency therefore arise from interactions among data coverage, architecture, conditioning, and sampling.

Interpretation

Diffusion models can be interpreted as hierarchical latent-variable models in which the intermediate noisy states form latent variables. They can also be interpreted as score estimators defined over a continuum of smoothed data distributions. The stochastic-process interpretation emphasizes reverse-time dynamics, while the denoising interpretation emphasizes regression from corrupted observations to information about clean data.

These views are mathematically connected rather than mutually exclusive. The variational formulation explains likelihood training, score matching explains the learned vector field, and differential equations explain continuous-time generation. Their coexistence has produced several naming conventions for closely related objects, giving the field a terminological reverse process whose convergence has not been formally established.