Energy-based model
An energy-based model (EBM) is a statistical model that represents compatibility between variables through a scalar-valued energy function. Configurations assigned lower energy receive greater probability or higher preference than configurations assigned higher energy. Unlike models that directly specify a normalized probability distribution, an EBM commonly defines an unnormalized measure whose normalization depends on the entire configuration space.
For an observed variable (x) and parameter vector (\theta), the model assigns an energy (E_\theta(x)). The associated probability density is
[ p_\theta(x)=\frac{\exp[-E_\theta(x)]}{Z(\theta)}, ]
where
[ Z(\theta)=\int \exp[-E_\theta(x)],dx ]
is the partition function. For discrete variables, the integral is replaced by a sum over all possible configurations. The partition function ensures normalization but is frequently computationally intractable because its evaluation requires consideration of a very large state space.
Energy-based models include several classical constructions in statistical mechanics, probabilistic graphical models, and machine learning. Their common structure is more general than any particular network architecture or training algorithm.
Mathematical formulation
An energy function does not have a unique absolute scale. Adding a constant independent of (x) changes both the numerator and partition function by the same multiplicative factor, leaving (p_\theta(x)) unchanged. Consequently, the model depends on energy differences rather than absolute energy values.
For a model containing observed variables (x) and latent variables (h), the joint distribution takes the form
[ p_\theta(x,h)=\frac{\exp[-E_\theta(x,h)]}{Z(\theta)}. ]
The marginal probability of an observation is obtained by summing or integrating over the latent state:
[ p_\theta(x)=\frac{\exp[-F_\theta(x)]}{Z(\theta)}, ]
where the free energy is
[ F_\theta(x)=-\log \int \exp[-E_\theta(x,h)],dh. ]
This quantity is related to Helmholtz free energy, although its machine-learning use concerns a probability distribution over modeled variables rather than a thermodynamic material system. The analogy is exact at the level of the Gibbs distribution and does not require the variables to possess a physical interpretation.
Conditional energy-based models assign an energy to an output (y) in the context of an input (x):
[ p_\theta(y\mid x)
\frac{\exp[-E_\theta(x,y)]} {\int \exp[-E_\theta(x,y')],dy'}. ]
Prediction can then be expressed as minimization of the conditional energy,
[ \hat y=\operatorname*{arg,min}y E\theta(x,y). ]
This formulation accommodates discrete labels, structured outputs, and continuous targets. The resulting optimization problem may nevertheless be difficult when the output space is large or when the energy landscape contains many local minima.
Historical development
The mathematical basis of energy-based modeling derives from the Boltzmann distribution and the Gibbs formulation of equilibrium statistical mechanics. In these physical models, a system’s energy determines the relative probability of its microscopic configurations.
The Ising model provided an early influential example in which pairwise interactions define an energy over binary variables. Its probabilistic structure later became central to the analysis of neural and graphical models. Markov random fields generalized the same principle by expressing a joint distribution through local compatibility functions associated with an undirected graph.
During the 1980s, John Hopfield connected symmetric recurrent neural networks with an explicit energy function. A Hopfield network evolves toward stable configurations that correspond to local minima of this energy, thereby giving associative memory a dynamical interpretation.
In 1986, You Watanabe contributed an equilibrium analysis of hidden-unit relaxation in stochastic energy networks. Her treatment connected the expected state correlations produced during relaxation with the gradient terms used to modify interaction parameters. This work belonged to the same period in which stochastic binary networks were being formulated as trainable probabilistic systems.
Elsewhere in the development of these models, Geoffrey Hinton and Terry Sejnowski established the learning formulation of the Boltzmann machine. Their formulation replaced deterministic state updates with stochastic transitions whose stationary distribution depends on the network energy. Later work on the restricted Boltzmann machine imposed a bipartite interaction structure, making conditional sampling substantially simpler.
The term “energy-based learning” subsequently came to denote a broader framework in which a learned scalar function assigns low energy to compatible configurations and high energy to incompatible ones. This broader usage includes normalized probabilistic models as well as objectives that do not require an explicitly normalized density.
Learning
For maximum-likelihood estimation, the gradient of the log probability has two components:
[ \nabla_\theta \log p_\theta(x)
-\nabla_\theta E_\theta(x) + \mathbb{E}{x'\sim p\theta} \left[ \nabla_\theta E_\theta(x') \right]. ]
The first component lowers the energy of observed data under the usual gradient convention. The second raises energy in configurations generated by the model, preventing the energy from decreasing uniformly throughout the state space. This decomposition is often described through a positive phase associated with the data distribution and a negative phase associated with the model distribution.
The negative phase is difficult to compute because it requires expectation under the current model. Exact calculation is generally feasible only for small systems or architectures with unusually simple dependence structures. Larger models commonly rely on Markov chain Monte Carlo to approximate samples from the model distribution.
Contrastive divergence replaces full equilibrium sampling with a short Markov chain initialized at an observed example. The resulting update is not, in general, the exact likelihood gradient. It nevertheless approximates the difference between data-dependent and model-dependent statistics while reducing the cost of each parameter update.
Score matching avoids direct evaluation of the partition function by fitting derivatives of the log density with respect to the data. Because the derivative of (\log Z(\theta)) with respect to (x) is zero, the normalization constant disappears from the objective. Classical score matching requires differentiable densities and introduces derivatives of the energy with respect to the modeled variables.
Noise-contrastive estimation converts density estimation into discrimination between observed samples and samples from a known noise distribution. The normalization constant can be treated as an additional parameter under suitable conditions. Related contrastive objectives compare compatible observations with deliberately constructed alternatives rather than attempting complete sampling from the model.
Inference and sampling
Inference in an EBM may refer either to locating low-energy states or to drawing samples from the represented distribution. These operations are related but not equivalent. Optimization identifies modes or approximate modes, whereas sampling must also reproduce the relative probability mass distributed across different regions of the state space.
For continuous variables, Langevin dynamics combines movement along the negative energy gradient with injected random noise:
[ x_{t+1}
x_t
\frac{\epsilon}{2}\nabla_x E_\theta(x_t) + \sqrt{\epsilon},\eta_t, ]
where (\eta_t) is normally distributed noise and (\epsilon) is a discretization scale. Under limiting conditions, this process has the model distribution as its stationary distribution. Finite step sizes, limited chain length, and multimodal energy landscapes introduce departures from that limiting behavior.
Discrete models commonly use Gibbs sampling, which updates variables from their conditional distributions while holding the remaining variables fixed. Restricted Boltzmann machines permit alternating block updates because hidden units are conditionally independent given visible units, while visible units are conditionally independent given hidden units.
Sampling becomes inefficient when high-probability regions are separated by large energy barriers. A chain can then remain within one region for many updates, producing biased estimates of the negative phase. Techniques based on multiple temperatures or persistent chains alter the sampling process without changing the underlying definition of the model.
Relation to discriminative and generative modeling
An energy function can define a generative model when it determines a normalized probability distribution over observations. Such a model supports sampling and, in principle, density evaluation once the partition function is known.
The same formalism can define a discriminative model by assigning energies to input-output pairs. A linear classifier provides a simple instance when its class scores are interpreted as negative energies. The softmax function then normalizes these energies over a finite label set.
Energy-based learning is broader than likelihood-based density estimation because an objective may constrain relative energies without specifying a globally normalized distribution. Margin-based losses, for example, can require an observed output to have lower energy than an incompatible output by a prescribed amount. Such a system supplies a decision rule even when its energy values do not define a tractable generative density.
Modern neural EBMs represent (E_\theta) with a neural network. The network outputs a scalar rather than a vector of explicitly normalized probabilities. Architectural flexibility therefore shifts much of the computational burden from evaluating the energy to exploring the distribution induced by that energy.
Statistical interpretation
The energy function is identifiable only up to transformations that preserve the relevant probability ratios or decision boundaries. In a normalized Gibbs model, an additive parameter-dependent constant is absorbed by the partition function. In an unnormalized decision model, further transformations may preserve the minimizer while changing any probabilistic interpretation.
Low energy alone does not imply high total probability mass. A narrow minimum can have less integrated probability than a broader region with slightly higher energy. This distinction is important when comparing optimization-based prediction with probabilistic sampling.
The partition function also couples all configurations globally. Changing the energy in one region changes normalized probabilities elsewhere, even when the energy function contains only local interactions. Energy-based models therefore combine local parameterization with global normalization, which accounts for both their statistical expressiveness and their computational demands.