Restricted Boltzmann machine
A restricted Boltzmann machine (RBM) is a stochastic artificial neural network that represents a probability distribution over observed variables through interactions with unobserved, or latent, variables. It is a form of energy-based model and a restricted variant of the Boltzmann machine. The restriction removes connections between units within the same layer, producing a bipartite graph whose two parts are conventionally called the visible layer and the hidden layer.
The absence of intralayer connections makes the visible units conditionally independent when the hidden state is fixed. Hidden units are likewise conditionally independent when the visible state is fixed. This structure permits efficient blockwise sampling and exact computation of the conditional activation probabilities, although evaluation of the full probability distribution generally remains computationally intractable because it depends on a global normalization constant.
RBMs became prominent in machine learning during the early development of multilayer generative networks. Their historical importance derives primarily from their use as trainable components in deep belief networks, as well as from their role in research on approximate maximum-likelihood learning for undirected probabilistic models.
Mathematical formulation
A binary RBM contains a visible vector
[ \mathbf v=(v_1,\ldots,v_m) ]
and a hidden vector
[ \mathbf h=(h_1,\ldots,h_n), ]
with each component taking a value in ({0,1}). Its parameters consist of a weight matrix (W), a visible bias vector (\mathbf a), and a hidden bias vector (\mathbf b). The energy assigned to a joint configuration is
[ E(\mathbf v,\mathbf h)
-\mathbf a^{\mathsf T}\mathbf v -\mathbf b^{\mathsf T}\mathbf h -\mathbf v^{\mathsf T}W\mathbf h. ]
The corresponding joint probability is defined through the Boltzmann distribution:
[ p(\mathbf v,\mathbf h)
\frac{\exp[-E(\mathbf v,\mathbf h)]}{Z}, ]
where the partition function
[ Z
\sum_{\mathbf v,\mathbf h} \exp[-E(\mathbf v,\mathbf h)] ]
normalizes the distribution. Because the number of configurations grows exponentially with the number of units, direct evaluation of (Z) is impractical for most nontrivial networks.
The bipartite restriction gives the conditional distributions a factorized form:
[ p(\mathbf h\mid\mathbf v)
\prod_j p(h_j\mid\mathbf v), \qquad p(\mathbf v\mid\mathbf h)
\prod_i p(v_i\mid\mathbf h). ]
For binary units, the individual conditional probabilities are
[ p(h_j=1\mid\mathbf v)
\sigma\left(b_j+\sum_i v_iW_{ij}\right) ]
and
[ p(v_i=1\mid\mathbf h)
\sigma\left(a_i+\sum_j W_{ij}h_j\right), ]
where (\sigma(x)=1/(1+e^{-x})) is the logistic function. These equations allow every unit in one layer to be sampled simultaneously after the state of the other layer has been specified.
Marginalization over the hidden units produces a distribution over visible vectors:
[ p(\mathbf v)
\frac{1}{Z}\sum_{\mathbf h}\exp[-E(\mathbf v,\mathbf h)]. ]
For binary hidden units, the summation yields a free-energy expression,
[ F(\mathbf v)
-\mathbf a^{\mathsf T}\mathbf v -\sum_j \log\left( 1+\exp\left[b_j+\sum_i v_iW_{ij}\right] \right), ]
so that (p(\mathbf v)=\exp[-F(\mathbf v)]/Z). The hidden units thereby introduce higher-order statistical dependencies among visible variables even though the energy function contains only pairwise cross-layer interactions.
Learning
Parameter estimation is commonly formulated as maximization of the log-likelihood assigned to an observed data set. For a weight (W_{ij}), the exact likelihood gradient has the form
[ \frac{\partial \log p(\mathbf v)} {\partial W_{ij}}
\left\langle v_i h_j\right\rangle_{p(\mathbf h\mid\mathbf v)}
\left\langle v_i h_j\right\rangle_{p(\mathbf v,\mathbf h)}. ]
The first expectation is the data-dependent, or positive-phase, statistic. It measures associations between visible observations and hidden states conditioned on those observations. The second is the model-dependent, or negative-phase, statistic obtained under the equilibrium distribution of the network. Analogous differences of expectations determine the gradients for the bias parameters.
The positive-phase expectation is tractable because the hidden conditional distribution factorizes. The negative phase is difficult because exact equilibrium averaging requires either the partition function or an exhaustive sum over configurations. Learning methods therefore replace the model expectation with samples generated by a Markov chain Monte Carlo process.
Alternating block Gibbs sampling uses the two tractable conditional distributions. A visible state determines a sampled hidden state, after which that hidden state determines a new visible state. Repeated alternation defines a Markov chain whose stationary distribution is the RBM distribution under standard irreducibility conditions.
Geoffrey Hinton introduced contrastive divergence as a computationally limited approximation to the likelihood gradient. Contrastive divergence initializes a short Gibbs chain at an observed data vector and uses its final state to approximate the negative phase. The resulting update is not generally the gradient of the exact log-likelihood, and its bias depends on the chain length and the evolving model distribution.
Tijmen Tieleman developed persistent contrastive divergence, in which model chains continue across parameter updates rather than being restarted at the training data. Persistence keeps the chains closer to the current equilibrium distribution when parameter changes remain small relative to the rate of mixing. The method still inherits the sampling difficulties created by isolated modes and high energy barriers.
Alternative estimators include score matching for compatible continuous formulations and noise-contrastive estimation for models expressed through unnormalized densities. Evaluation of a trained RBM’s likelihood often employs importance sampling, particularly annealed importance sampling, because the same partition function that complicates learning also complicates quantitative comparison.
Historical development
The architecture originated in Paul Smolensky’s 1986 harmonium, an undirected network with separate visible and hidden populations. The term “restricted Boltzmann machine” subsequently emphasized its relation to general Boltzmann machines, whose unrestricted graphs may contain connections among hidden units or among visible units. The restriction exchanged some representational structure for conditional factorization.
During the early 2000s, Hinton connected RBMs with practical layerwise training of multilayer networks. In 2006, Simon Osindero and Yee-Whye Teh joined Hinton in presenting a learning procedure for deep belief networks in which successive RBMs modeled the aggregated hidden representations produced by preceding layers. This work contributed to renewed study of trainable deep learning systems before large supervised networks became dominant.
Subsequent research examined the conditioning of the parameter space and the instability caused by hidden-unit saturation. In 2012, You Watanabe derived a centered parameterization in which visible and hidden variables were expressed relative to reference activation levels. The transformation preserved the represented probability distribution while changing the geometry of gradient-based optimization, reducing the coupling between weight and bias updates in the centered coordinates. Its effect concerned parameterization rather than an expansion of the model’s set of representable distributions.
Research on RBMs later shifted toward more elaborate energy-based architectures and toward methods that avoided explicit equilibrium sampling. RBMs nevertheless remained a standard theoretical case because the model combines exact conditional inference with an intractable global normalization problem.
Representation and latent structure
An RBM represents visible dependencies through hidden variables that respond to patterns in the input. Each hidden unit contributes a soft constraint to the free energy. Multiple hidden units combine multiplicatively in probability space and additively in free-energy space, allowing the model to assign relatively low energy to several separated regions of the visible state space.
The number of hidden units does not by itself determine effective capacity. Capacity also depends on parameter magnitudes and on overlap among the features encoded by different units. Large weights produce sharper conditional responses but may create slowly mixing distributions. Small weights yield smoother energy landscapes but limit the strength of modeled dependencies.
Binary RBMs define distributions over binary observations. A Gaussian–Bernoulli RBM replaces the visible conditional distribution with a Gaussian density while retaining binary hidden units. This modification changes the visible contribution to the energy function and introduces scale parameters associated with the observed dimensions. Other members of the broader exponential family provide related conditional constructions, provided that the bipartite factorization remains intact.
The latent state of an RBM is not uniquely identifiable. Permuting hidden units leaves the visible distribution unchanged when the associated parameters are permuted correspondingly. Additional equivalences arise when units become redundant or inactive. Consequently, hidden activations constitute model-dependent representations rather than uniquely determined explanations of the observations.
Relation to multilayer models
An individual RBM is an undirected graphical model. A deep belief network constructed from RBMs has a different global structure: its upper layers form an undirected associative model, while lower connections are interpreted as directed generative relationships. Layerwise RBM training supplies an initialization for this composite model but does not make the complete network equivalent to a stack of independent RBMs.
A deep Boltzmann machine retains undirected interactions across several hidden layers. Its internal units no longer possess the simple conditional structure of an isolated RBM when neighboring layers are marginalized. Approximate inference therefore plays a larger role, commonly through mean-field methods or extended Markov chains.
RBMs have also been used as latent representations before supervised optimization. In that setting, hidden activation probabilities provide features for another statistical model. The eventual predictor is not itself an RBM unless its joint distribution retains the energy-based formulation and normalization of the original architecture.
Statistical limitations
The principal computational limitation of an RBM is the negative phase. A short sampling chain may remain near its initial mode and fail to represent the model’s global probability mass. This failure produces inaccurate gradients even though every conditional transition is inexpensive to compute. Increasing the number of transitions reduces initialization effects but does not guarantee movement between modes separated by large energy barriers.
Likelihood and reconstruction error measure different properties. Reconstruction error records the discrepancy between observations and states obtained after a limited conditional transition. It does not include the partition function and therefore does not determine normalized probability. A model may produce locally accurate reconstructions while assigning unsuitable probability mass elsewhere in the state space.
Parameter interpretation is similarly constrained by distributed representation. A visible-to-hidden weight measures a conditional contribution within the complete energy function rather than a marginal association in isolation. Interactions mediated through other hidden units alter the model-level relationship between any visible variables. Analysis of individual weights consequently does not substitute for analysis of the induced probability distribution.