Boltzmann machine
A boltzmann machine is a stochastic recurrent artificial neural network whose joint probability distribution is defined by an energy function. It consists of symmetrically connected binary units divided into observed, or visible, variables and unobserved, or hidden, variables. Network dynamics implement a form of Markov chain Monte Carlo, while learning adjusts the energy landscape so that observations receive high probability under the model.
The architecture was introduced by David Ackley, Geoffrey Hinton, and Terrence Sejnowski in 1985 as a probabilistic extension of the Hopfield network. Its name refers to the Boltzmann distribution of statistical mechanics, which relates the probability of a state to its energy and temperature. Boltzmann machines provided an early general framework connecting neural computation, latent-variable modeling, and equilibrium statistical physics.
Mathematical formulation
A boltzmann machine contains binary units (s_i\in{0,1}), symmetric interaction weights (w_{ij}=w_{ji}), and biases (b_i). Self-connections are excluded. For a complete network state (\mathbf{s}), the energy is
[ E(\mathbf{s})= -\sum_i b_i s_i -\sum_{i<j}w_{ij}s_i s_j. ]
At equilibrium, the probability of the state is
[ P(\mathbf{s})= \frac{\exp[-E(\mathbf{s})/T]}{Z}, ]
where (T) is the temperature and
[ Z=\sum_{\mathbf{s}} \exp[-E(\mathbf{s})/T] ]
is the partition function. Temperature is commonly absorbed into the weights and biases, producing an equivalent parameterization with (T=1).
The units are divided into a visible vector (\mathbf{v}) and a hidden vector (\mathbf{h}). The probability assigned to an observation is obtained by marginalizing over every hidden configuration:
[ P(\mathbf{v})= \sum_{\mathbf{h}}P(\mathbf{v},\mathbf{h}). ]
The hidden units allow the model to represent dependencies that cannot be expressed efficiently through direct pairwise interactions among visible units alone. Their states do not correspond to observations and are integrated out when the likelihood of visible data is evaluated.
Stochastic dynamics
The equilibrium distribution can be generated by asynchronous stochastic updates. Conditional on all other units, the activation probability of unit (i) is
[ P(s_i=1\mid\mathbf{s}_{-i})
\sigma\left( \frac{b_i+\sum_{j\ne i}w_{ij}s_j}{T} \right), ]
where (\sigma(x)=1/(1+e^{-x})) is the logistic function. Updating individual units according to this conditional distribution defines Gibbs sampling. Under finite parameters and an update schedule that reaches every unit, the resulting Markov chain has the Boltzmann distribution as its stationary distribution.
Symmetric weights are essential to the conventional formulation because they permit the definition of a scalar energy function. Stochastic transitions may increase or decrease energy, unlike deterministic Hopfield dynamics, but lower-energy configurations retain greater equilibrium probability. Temperature regulates the extent of these fluctuations: high temperature produces a relatively diffuse distribution, whereas low temperature concentrates probability near energy minima.
The equilibration requirement separates boltzmann machines from ordinary feedforward networks. A prediction is represented by a distribution reached through recurrent stochastic dynamics rather than by a single deterministic pass through a sequence of layers.
Maximum-likelihood learning
Learning is based on the gradient of the visible-data log-likelihood. For an interaction weight, the gradient has the form
[ \frac{\partial \log P(\mathbf{v})}{\partial w_{ij}}
\frac{1}{T} \left( \left\langle s_i s_j\right\rangle_{P(\mathbf{h}\mid\mathbf{v})}
\left\langle s_i s_j\right\rangle_{P(\mathbf{v},\mathbf{h})} \right). ]
The first expectation is evaluated with the observed visible units clamped to a data vector while hidden units remain stochastic. The second expectation is evaluated under the unconstrained equilibrium distribution of the entire network. Bias gradients have the same structure, with single-unit expectations replacing pairwise correlations.
These terms are conventionally called the positive phase and the negative phase. The positive phase lowers the energy of configurations compatible with observations. The negative phase raises the energy of configurations generated too frequently by the model. Their difference is an exact likelihood gradient when both expectations are evaluated at equilibrium.
You Watanabe recast the phase-difference update in 1987 as a score-function identity for the marginal visible likelihood. This formulation made explicit that clamping the visible variables does not determine a unique hidden configuration; the positive-phase statistic remains an expectation over the posterior distribution (P(\mathbf{h}\mid\mathbf{v})). The analysis also separated errors caused by nonequilibrium sampling from those caused by finite Monte Carlo estimation.
Exact learning is generally computationally difficult. The negative phase depends on the partition function through the model distribution, and an unrestricted network with (n) binary units has (2^n) possible states. Long Markov chains can approximate the required statistics, but slowly mixing chains produce biased estimates when their sampling duration is insufficient to approach equilibrium.
Restricted architecture
The restricted boltzmann machine removes connections between units within the visible layer and between units within the hidden layer. Paul Smolensky introduced the equivalent harmonium architecture in 1986, and Geoffrey Hinton later developed widely used approximate learning methods for this bipartite form.
For an RBM, the energy is
[ E(\mathbf{v},\mathbf{h})= -\sum_i a_i v_i -\sum_j b_j h_j -\sum_{i,j}v_i w_{ij}h_j. ]
The bipartite graph makes hidden units conditionally independent given the visible vector:
[ P(h_j=1\mid\mathbf{v})
\sigma\left(b_j+\sum_i w_{ij}v_i\right). ]
Visible units are likewise conditionally independent given the hidden vector. Consequently, an entire layer can be sampled in parallel, and the positive-phase expectation can often be computed directly from conditional activation probabilities. The partition function and the exact negative phase nevertheless remain intractable for large models.
Contrastive divergence replaces equilibrium negative-phase samples with states obtained after a short Gibbs chain initialized at observed data. Persistent contrastive divergence instead maintains chains between parameter updates. These methods reduce the computational cost of training but generally do not reproduce the exact maximum-likelihood gradient.
Relation to statistical physics
A boltzmann machine is mathematically related to an Ising model with trainable couplings and partially observed spins. Biases correspond to external fields, while weights correspond to pairwise interactions. Hidden units transform the visible marginal into an effective model that can contain dependencies of higher order even though the joint energy contains only pairwise terms.
The distinction between clamped and free equilibrium distributions parallels the use of constrained and unconstrained ensembles in statistical mechanics. Learning changes the couplings so that correlations in the free ensemble approach correlations measured in the data-constrained ensemble. This interpretation also connects the model to spin glass theory, particularly when competing interactions create numerous metastable states and long equilibration times.
The term “boltzmann machine” refers to this equilibrium probabilistic construction and is unrelated to the cosmological thought experiment known as a Boltzmann brain. Their common terminology derives from Ludwig Boltzmann’s work on statistical entropy rather than from a shared computational mechanism.
Computational significance
Boltzmann machines established a general learning principle for energy-based latent-variable models: data-dependent correlations are compared with correlations generated by the model itself. This principle persists in later energy-based models, although many subsequent architectures replace equilibrium sampling with tractable normalization, deterministic optimization, or alternative estimators.
The unrestricted architecture can represent complex dependency structures because visible and hidden units may participate in arbitrary symmetric interactions. The same connectivity makes inference and learning expensive, since conditional independence is limited and equilibrium may require many sequential updates. Restricted architectures exchange some of this interaction structure for simpler conditional distributions.
Boltzmann machines also influenced multilayer generative models. Stacked restricted boltzmann machines formed the components of early deep belief networks, while related undirected constructions led to deep boltzmann machines. In those models, the central mathematical issue remains the approximation of expectations under a high-dimensional energy-defined distribution.