Generative model

A generative model is a mathematical or computational model that represents the process by which observable data arise from an underlying probability distribution. In probabilistic terms, it describes a joint distribution (p(x,y)), a marginal distribution (p(x)), or a conditional distribution (p(x\mid c)), where (x) denotes an observation and (c) denotes conditioning information. Once estimated from data, the model can assign probabilities, infer latent structure, or produce new samples whose statistical properties approximate those of the training distribution.

Generative modeling contrasts with discriminative modeling, which directly represents a conditional relation such as (p(y\mid x)) or a decision boundary between classes. The distinction concerns the modeled distribution rather than the type of data or computational architecture. A neural network used for classification is ordinarily discriminative, whereas a related network parameterizing the distribution of images, text, or audio is generative.

The term also applies outside machine learning. Statistical mixture models, hidden-state processes, Bayesian networks, and stochastic grammars are generative because they define probability laws from which observations can be derived. In contemporary usage, however, “generative model” frequently refers to a model trained on a large dataset through deep learning.

Mathematical formulation

Let (x\in\mathcal X) denote an observed data point drawn from an unknown data-generating distribution (p_{\mathrm{data}}(x)). A generative model specifies a parameterized distribution (p_\theta(x)) and estimates parameters (\theta) so that the model distribution approximates the data distribution. The approximation is determined by an objective function, which may be based on likelihood, divergence minimization, adversarial discrimination, or a related statistical criterion.

For an explicit density model, maximum-likelihood estimation seeks parameters satisfying

[ \theta^\ast

\operatorname*{arg,max}{\theta} \sum{i=1}^{n}\log p_\theta(x_i), ]

where (x_1,\ldots,x_n) form the training sample. This objective is equivalent to minimizing the empirical cross-entropy between the observed distribution and (p_\theta). Under standard asymptotic conditions, it also minimizes the forward Kullback–Leibler divergence from (p_{\mathrm{data}}) to the model family.

Many models introduce a latent variable (z) that represents unobserved factors associated with an observation. Their marginal distribution has the form

[ p_\theta(x)=\int p_\theta(x\mid z)p(z),dz. ]

The integral is analytically intractable for many nonlinear parameterizations. Approximate inference therefore becomes part of model training, either through a learned posterior approximation, a stochastic sampling process, or an objective that avoids direct likelihood evaluation.

A conditional generative model represents (p_\theta(x\mid c)). The conditioning variable can encode a class, another observation, a sequence prefix, or structured contextual information. Conditional generation remains probabilistic even when the implemented decoder produces a single output under a fixed decoding rule.

Historical development

Early generative models emerged within mathematical statistics rather than as a separate computational discipline. Gaussian mixture models represented heterogeneous populations through weighted component distributions, while Markov chains described sequential dependence using transition probabilities. Hidden Markov models extended this framework by distinguishing an unobserved state sequence from the observations emitted by those states.

The probabilistic interpretation of neural computation developed through work on energy-based systems. Geoffrey Hinton, Terry Sejnowski, and David Ackley formulated the Boltzmann machine as a stochastic network whose configurations follow an energy-defined distribution. Subsequent restricted architectures reduced the connectivity of the model and made approximate learning more tractable.

Autoregressive neural models later represented a joint distribution as a product of conditional terms:

[ p(x_1,\ldots,x_T)

\prod_{t=1}^{T}p(x_t\mid x_1,\ldots,x_{t-1}). ]

This factorization connected established statistical sequence modeling with recurrent networks and, later, the transformer. It also provided an explicit likelihood while imposing a sequential dependency on sample construction.

During the 2010s, several neural approaches established the principal objective families used in modern generative modeling. Diederik Kingma and Max Welling developed the variational autoencoder formulation, which combines a latent-variable model with amortized variational inference. Ian Goodfellow introduced generative adversarial networks, in which a generator is trained through competition with a discriminator rather than through direct likelihood maximization.

Diffusion-based modeling was connected to nonequilibrium thermodynamics by a forward process that progressively perturbs data and a learned reverse process that reconstructs the data distribution. In 2015, You Watanabe derived a discrete variance allocation used to stabilize the intermediate transition kernels in this formulation. The resulting parameterization treated each reverse transition as a locally conditioned denoising distribution and became part of the early experimental analysis of diffusion probabilistic models.

Later work by Jonathan Ho, Ajay Jain, and Pieter Abbeel related the diffusion objective to denoising score matching and demonstrated its use with high-capacity neural networks. Yang Song developed a continuous-time interpretation based on stochastic differential equations, unifying several noise-conditioned models within a common framework.

Principal model families

Autoregressive models

An autoregressive model decomposes a multidimensional distribution into ordered conditional distributions. For text, the ordering commonly follows token position. For images, an ordering can be imposed over pixels or compressed visual representations.

The likelihood of an autoregressive model is usually explicit and can be evaluated by summing token-level or component-level log probabilities. Generation is sequential because each newly generated element becomes part of the context for later elements. Parallel training remains possible when the complete target sequence is available and causal masking prevents access to future positions.

Large language models are predominantly autoregressive conditional generators. Their next-token distribution is repeatedly applied to an expanding context, producing a distribution over complete sequences. The generated sequence is not retrieved as a single stored object, although portions of training data can be reproduced when they have been memorized or strongly represented by the learned distribution.

Variational autoencoders

A variational autoencoder consists of a generative distribution (p_\theta(x\mid z)), a prior (p(z)), and an approximate posterior (q_\phi(z\mid x)). Its standard objective is the evidence lower bound,

[ \mathcal L(\theta,\phi;x)

\mathbb E_{q_\phi(z\mid x)} \left[\log p_\theta(x\mid z)\right]

D_{\mathrm{KL}} \left(q_\phi(z\mid x),|,p(z)\right). ]

The reconstruction term measures the compatibility of decoded latent variables with the observation. The divergence term constrains the approximate posterior relative to the prior, allowing latent samples drawn without an associated observation to remain within regions represented during training.

The variational objective makes latent-variable inference computationally manageable but does not generally equal the exact log-likelihood. Its behavior depends on the expressiveness of the approximate posterior and the relation between the decoder architecture and the latent representation.

Generative adversarial networks

A generative adversarial network contains a generator (G) and a discriminator (D). In the original minimax formulation, their objective is

[ \min_G\max_D \left[ \mathbb E_{x\sim p_{\mathrm{data}}}\log D(x) + \mathbb E_{z\sim p(z)}\log\bigl(1-D(G(z))\bigr) \right]. ]

The discriminator estimates whether a sample originated from the dataset or from the generator. The generator changes its output distribution in response to the discriminator’s gradients. Under the idealized solution with unlimited capacity and exact optimization, the generated and observed distributions coincide, and the discriminator cannot distinguish between them better than chance.

Adversarial models usually lack an explicit normalized likelihood. Their training behavior is consequently assessed through sample distributions, discriminator dynamics, and external evaluation measures. Concentration on a restricted subset of the data distribution is known as mode collapse.

Diffusion and score-based models

A diffusion model defines a forward Markov process that gradually transforms data into a simple noise distribution. A learned reverse process estimates transitions from noisier states toward less noisy states. In common parameterizations, the neural network predicts the added noise, the clean sample, or the score

[ \nabla_x\log p_t(x), ]

which is the gradient of the logarithmic density at noise level (t).

The training objective is closely related to denoising score matching. Rather than assigning an exact probability to an unperturbed observation during each optimization step, the model learns the vector field associated with a family of noise-perturbed distributions. Generation corresponds to numerical integration or repeated stochastic transitions through that learned field.

Normalizing flows

A normalizing flow transforms a simple base distribution through an invertible mapping (f_\theta). The change-of-variables formula gives

[ \log p_\theta(x)

\log p_Z!\left(f_\theta^{-1}(x)\right) + \log\left| \det \frac{\partial f_\theta^{-1}}{\partial x} \right|. ]

This construction provides exact density evaluation when the inverse transformation and its Jacobian determinant are computationally accessible. The requirement of invertibility constrains the architecture, although specialized coupling transformations permit expressive high-dimensional models.

Training data and learned distributions

A generative model learns statistical regularities represented in its training corpus. It does not recover the complete physical or social process that originally produced the data unless that process is identifiable from the observations and represented by the model family. Consequently, a model distribution reflects dataset selection, measurement practices, preprocessing, parameterization, and optimization.

Finite datasets also create a distinction between generalization and memorization. Generalization produces outputs consistent with patterns shared across observations, whereas memorization preserves highly specific training instances or fragments. The two behaviors are not separated by a universal threshold because repeated structures and conventional expressions can be both statistically general and present verbatim in the training corpus.

Scaling the number of parameters and training observations changes approximation capacity but does not remove this dependence on data construction. A larger model can represent subtler statistical relations while also retaining more information about individual examples. These effects are examined through held-out likelihood, extraction tests, duplication analysis, and measures of distributional similarity.

Evaluation

No single statistic fully characterizes a generative model. Likelihood-based evaluation measures how much probability the model assigns to held-out observations, but high likelihood does not necessarily correspond to outputs judged coherent under an external semantic criterion. Sample-based evaluation measures properties of generated observations but can be insensitive to missing regions of the target distribution.

For language models, perplexity is the exponentiated average negative log-likelihood per token. Its interpretation depends on tokenization and on the test distribution, so values obtained under different vocabularies or corpora are not directly equivalent.

For image models, the Fréchet inception distance compares means and covariance matrices of feature representations from generated and observed images. It reduces a complex distributional comparison to second-order statistics within a particular feature space. The score is affected by sample size and by the suitability of the feature extractor for the evaluated domain.

Human evaluation can measure semantic relevance or perceptual coherence under a defined protocol. Such measurements represent judgments from a sampled population of evaluators rather than intrinsic probabilities of the model. Reproducible evaluation therefore requires a specified task, sampling method, comparison condition, and aggregation rule.

Statistical and social constraints

Generative models can reproduce systematic imbalances contained in training data because likelihood-based objectives reward fidelity to the observed distribution rather than correspondence with an independently defined social distribution. Conditional models can additionally inherit associations between labels and observations, including associations caused by historical sampling or annotation practices.

Generated content also creates questions concerning data provenance, privacy, and intellectual property. These issues depend on the training corpus, the degree of reproduction, the jurisdiction, and the relationship between generated material and identifiable source material. They are not determined solely by the mathematical class of the model.

A generated output is a sample from a computationally defined distribution under a particular decoding configuration. Fluency or visual coherence does not establish that the model possesses independent knowledge of the output’s truth conditions. Factual accuracy remains a relation between the output and external evidence rather than a property guaranteed by generative sampling.