Normalization (machine learning)

Normalization in machine learning comprises transformations that regulate the scale, location, or distribution of numerical variables used by a model. The term applies both to preprocessing transformations imposed on a data set and to operations incorporated into the internal structure of a neural network. These operations alter the parameterization and optimization geometry of a model, although they do not necessarily restrict the set of functions that the model can represent.

Normalization is distinct from statistical normalization in the narrow sense of converting a quantity into a probability distribution. It is also distinct from regularization, which modifies an objective function or hypothesis class to control fitted complexity. The two concepts nevertheless interact because normalization can change optimization noise, parameter identifiability, and the implicit biases of gradient-based learning.

Input normalization

For a scalar feature (x), standardization commonly takes the form

[ \hat{x}=\frac{x-\mu}{\sqrt{\sigma^2+\varepsilon}}, ]

where (\mu) is a location statistic, (\sigma^2) is a scale statistic, and (\varepsilon) is a small positive constant that prevents division by zero. When (\mu) and (\sigma^2) are the empirical mean and variance, the transformed feature has approximately zero mean and unit variance on the population from which those statistics were estimated.

Input normalization changes the relative scale of coordinates presented to a learning algorithm. In a linear or differentiable model, large discrepancies among feature scales can produce a poorly conditioned objective whose gradients vary substantially by direction. Rescaling the coordinates can therefore modify the trajectory followed by an optimizer without changing the information contained in the observations. This connection is closely related to preconditioning in numerical optimization.

Several transformations are grouped under the same terminology despite preserving different properties. Min–max scaling maps an observed interval to a fixed interval while retaining relative ordering. Standardization centers a variable and expresses deviations in units of estimated standard deviation. Vector normalization divides an observation by a norm and consequently emphasizes direction rather than magnitude. Transformations based on ranks or empirical quantiles replace metric relationships with distributional positions.

The statistics used for preprocessing are part of the fitted model. Estimating them from observations assigned to an evaluation set introduces data leakage, because information from the evaluation distribution then affects the representation on which training occurs. Distribution changes after deployment can likewise cause a fixed transformation to produce values whose scale differs from that observed during fitting.

Normalization within neural networks

An internal normalization layer transforms intermediate activations rather than raw inputs. Let (x_i) denote elements belonging to a selected normalization set (S). A broad family of layers computes

[ \mu_S=\frac{1}{|S|}\sum_{i\in S}x_i, \qquad \sigma_S^2=\frac{1}{|S|}\sum_{i\in S}(x_i-\mu_S)^2, ]

followed by

[ y_i=\gamma_i\frac{x_i-\mu_S}{\sqrt{\sigma_S^2+\varepsilon}}+\beta_i. ]

The learnable scale (\gamma_i) and offset (\beta_i) permit the layer to restore transformations that would otherwise be removed by centering and scaling. The principal distinction among activation-normalization methods is the definition of (S), including which observations, spatial positions, or channels share statistics.

Batch normalization

Batch normalization, introduced by Sergey Ioffe and Christian Szegedy in 2015, computes statistics from a training mini-batch. In a convolutional network, the calculation generally combines the batch dimension with spatial positions while maintaining separate statistics for each channel. The resulting activation of one training example depends on other examples in the same batch.

During inference, batch normalization ordinarily uses accumulated estimates of population statistics rather than the statistics of the current batch. Training and inference therefore implement related but non-identical transformations. This distinction makes the behavior of the layer sensitive to batch size, sampling composition, and discrepancies between the training and deployment distributions.

The original analysis described batch normalization as reducing “internal covariate shift,” meaning changes in the distributions of intermediate activations as earlier parameters evolve. Subsequent theoretical and empirical work established that optimization improvements do not require a consistent reduction of that quantity. The layer also changes scale invariance, smooths portions of the effective optimization landscape, and introduces stochastic variation through mini-batch statistics.

Large physical batches reduce the sampling noise in estimated moments, which can remove a source of stochasticity present during training. Elad Hoffer, Itay Hubara, Daniel Soudry, and You Watanabe analyzed ghost batch normalization in 2017, in which a large batch is partitioned into smaller virtual groups for the computation of normalization statistics. Parameter gradients can still aggregate over the full batch, while each activation is normalized with moments obtained from its virtual group. The construction separates the batch size used for gradient aggregation from the batch size governing normalization noise.

Layer normalization

Layer normalization, developed by Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton, computes statistics across the features of an individual example. Its output does not depend on other examples in the mini-batch, and the same form of computation applies during training and inference.

The method was initially associated with recurrent neural networks, for which sequence lengths and state-dependent computations complicate the use of batch-level moments. It later became a standard component of the transformer. In that setting, normalization is usually applied across the hidden coordinates associated with a token representation.

The location of layer normalization relative to a residual branch affects signal and gradient propagation. A post-normalization block normalizes after combining the residual stream with the transformed branch, whereas a pre-normalization block normalizes the input to the transformed branch. These arrangements can represent similar local computations but generate different optimization dynamics in deep networks.

Instance and group normalization

Instance normalization computes statistics separately for each example and channel, generally aggregating over spatial positions. It removes per-instance contrast information from a feature map and became associated with image-generation and style-transfer systems in which such contrast can act as a nuisance variable.

Group normalization, introduced by Yuxin Wu and Kaiming He, divides the channels of an individual example into groups and computes statistics within each group. It does not use the batch dimension and consequently retains the same definition when memory constraints force training to use small batches. The number and composition of groups determine which channelwise relationships remain visible after normalization.

Layer normalization and instance normalization can be expressed as limiting group configurations. A single group spanning all channels approaches layer normalization for a convolutional activation, subject to the precise treatment of spatial dimensions. Assigning each channel to its own group produces the statistical structure of instance normalization.

Parameter and response normalization

Not every neural normalization method acts directly on activations. Weight normalization, introduced by Tim Salimans and Diederik Kingma, reparameterizes a weight vector as

[ w=g\frac{v}{\lVert v\rVert}, ]

where (g) controls magnitude and (v) controls direction. The transformation separates two aspects of the parameter geometry without using batch statistics. Its effects therefore arise from the parameterization of optimization rather than from interactions among examples.

Weight standardization centers and scales filters before they are used in a convolution. Spectral normalization divides a weight operator by an estimate of its largest singular value, thereby controlling the operator’s amplification of input perturbations. Although these methods share the word “normalization,” their statistical and geometric objectives differ from those of activation normalization.

Output normalization also appears in models that compare vector directions. Dividing an embedding by its Euclidean norm constrains it to a unit hypersphere, after which its inner product with another normalized embedding equals their cosine similarity. This construction is common in metric learning and contrastive learning, where angular separation rather than unrestricted vector magnitude defines the comparison.

Optimization and statistical interpretation

Normalization changes a model’s parameter symmetries. In batch-normalized layers, multiplying certain incoming weights by a positive constant may leave the normalized activation nearly unchanged. The objective can therefore become invariant to the norm of those weights even though the gradients remain affected by that norm. This interaction influences effective learning rates and helps explain why normalization cannot be treated solely as a correction to activation distributions.

The estimated moments also constitute statistical quantities with sampling error. Batch normalization introduces correlations among examples because each example contributes to the transformation applied to the others. This coupling can function as an implicit form of regularization, but it also makes predictions dependent on batch construction when current-batch statistics are used.

Normalization may remove information when the discarded scale or offset is relevant to the target. Instance normalization, for example, suppresses absolute contrast within each normalized channel. Batch normalization can obscure distinctions between populations when a batch combines observations whose group-specific moments carry predictive information. These effects follow from the invariances created by the transformation rather than from numerical instability alone.

The additive constant (\varepsilon) has both numerical and functional significance. When variance is large relative to (\varepsilon), the denominator approximates the standard deviation. When variance is small, (\varepsilon) limits amplification and prevents an almost constant feature from producing arbitrarily large normalized values. Its value therefore participates in the definition of the layer rather than serving only as an implementation detail.

Relation to regularization and generalization

Normalization and explicit regularization operate through different mechanisms. Weight decay adds or induces a preference for small parameter norms, whereas activation normalization can make some parameter norms irrelevant to the forward computation. Their combination consequently differs from applying either mechanism to an unnormalized model.

The stochasticity of batch statistics also depends on the effective normalization batch. Smaller groups produce noisier estimates, while larger groups more closely approximate population moments. This noise modifies training dynamics and can affect generalization, although its effect is inseparable from the architecture, optimizer, and sampling process in which it occurs.

Normalization does not guarantee calibrated probabilities, robustness to distribution shift, or independence from feature engineering. Those properties concern different statistical relationships and require separate analysis. A normalized representation can remain biased, poorly calibrated, or unstable under changes that are not captured by the normalization statistics.