Quantization (machine learning)
Quantization in machine learning is the representation of model parameters, intermediate activations, and arithmetic operations with numerical formats of lower precision than those used during model development. The term most commonly denotes the conversion of floating-point values into integers or reduced-precision floating-point values. Quantization decreases the storage required by a model and changes the computational operations used during inference, while introducing approximation error through rounding, clipping, and finite dynamic range.
Quantization is distinct from model pruning and knowledge distillation. Pruning removes parameters or structured computational units, whereas distillation trains one model to reproduce aspects of another model's behavior. Quantization generally preserves the computational graph and replaces its numerical representation, although practical systems often combine all three forms of model compression.
Mathematical formulation
A uniform affine quantizer maps a real-valued quantity (x) to an integer (q) according to
[ q = \operatorname{clip}\left( \operatorname{round}\left(\frac{x}{s}\right)+z,, q_{\min},q_{\max} \right), ]
where (s>0) is the scale, (z) is the zero point, and ([q_{\min},q_{\max}]) is the representable integer interval. The corresponding reconstructed value is
[ \hat{x}=s(q-z). ]
The scale determines the spacing between adjacent reconstructed values. The zero point identifies the integer code corresponding to real zero, subject to rounding and range constraints. Exact representation of zero is important in networks containing zero padding or activation functions that produce large regions of zero-valued output.
In symmetric quantization, the zero point is fixed at or near the central integer code, and positive and negative values share a common magnitude range. In asymmetric quantization, the scale and zero point accommodate an interval whose endpoints need not have equal magnitude. Asymmetric representations can use the available codes more completely when a tensor has a shifted distribution, while symmetric representations simplify several forms of integer arithmetic.
The quantization error for a value (x) is
[ e(x)=x-\hat{x}. ]
Within the unclipped region of a uniform quantizer, the magnitude of this error is bounded by approximately (s/2). Values outside the selected real-valued interval are mapped to an endpoint and incur clipping error, which can be substantially larger. Selection of the interval therefore expresses a trade-off between fine resolution for typical values and retention of rare values with large magnitude.
Nonuniform quantization replaces constant-width intervals with a variable partition of the real line. Logarithmic quantizers allocate codes according to value magnitude, while codebook-based methods represent values by learned or predetermined centroids. These methods can match irregular parameter distributions more closely, although their arithmetic is less directly expressible through conventional integer matrix multiplication.
Quantized neural-network computation
A dense neural-network layer computes
[ Y=XW+B, ]
where (X) is an activation matrix, (W) is a weight matrix, and (B) is a bias term. Under integer quantization, (X) and (W) are approximated by scaled integer tensors:
[ X\approx s_X(Q_X-z_X), \qquad W\approx s_W(Q_W-z_W). ]
Their product can then be written as
[ XW\approx s_Xs_W (Q_X-z_X)(Q_W-z_W). ]
The inner products are ordinarily accumulated in a wider integer type than the operands. For example, products formed from 8-bit integers are commonly accumulated in 32-bit integers because a sum across many input channels can exceed the range of an 8-bit result. A subsequent requantization operation converts the accumulated value into the format expected by the next layer.
The computational effect depends on the relationship between the quantized representation and the available computer architecture. Integer operands can reduce memory traffic because each parameter occupies fewer bits, but the conversion does not guarantee a proportional reduction in execution time. Packing overhead, accumulator width, cache behavior, operator support, and tensor shape all affect realized performance. Quantization is consequently a numerical transformation with hardware-dependent operational consequences rather than an intrinsic measure of inference speed.
Matrix multiplication dominates the cost of many convolutional neural networks and transformers. Quantized matrix multiplication retains the same high-level algebraic structure, which permits existing tiled and vectorized implementations to be adapted to narrower operands. Other operations, including normalization and probability normalization, are often retained at higher precision because their ranges and error propagation differ from those of matrix products.
Granularity and numerical format
A quantization scale can apply to an entire tensor or to a subdivision of that tensor. Per-tensor quantization uses one scale and one zero point for all represented values. It has low metadata overhead, but a small number of large-magnitude values can determine the resolution available to the rest of the tensor.
Per-channel quantization assigns separate parameters to output channels or another structurally significant axis. In a weight matrix, this treatment allows rows or columns with different ranges to receive different scales. Per-group quantization divides a matrix into intermediate-sized blocks and is widely associated with weight-only quantization of large language models. Smaller groups generally track local variation more closely while requiring additional scale values and more elaborate decoding.
Integer quantization commonly uses 8-bit operands because this width provides a comparatively broad representable range and is supported by many processors. Lower-bit representations reduce storage further but make the location of each quantization boundary more consequential. Binary neural networks restrict represented values to two states, while ternary networks add a third state that commonly represents zero. These cases alter optimization and execution more substantially than ordinary 8-bit conversion.
Reduced-precision floating-point formats constitute a related category. Half-precision floating-point format, bfloat16, and 8-bit floating-point formats preserve an exponent-based dynamic range rather than imposing a single affine scale over an entire tensor. They are frequently described as quantized formats in machine-learning literature, although their error properties differ from fixed-point and integer quantization.
Post-training quantization
Post-training quantization converts a trained model without repeating its complete training process. Weight-only conversion quantizes model parameters while retaining activations in floating-point form. Integer inference additionally quantizes activations and therefore requires scale management across operator boundaries.
Static activation quantization estimates ranges from a calibration data set. Calibration records representative tensor distributions and derives quantizer parameters from those observations. A range based only on the largest observed magnitude can be strongly influenced by outliers, so alternative objectives minimize a reconstruction criterion or compare probability distributions before and after quantization.
Dynamic quantization determines selected activation parameters during inference. It adapts to the current input but introduces runtime work for range measurement and conversion. The distinction between static and dynamic quantization concerns when activation parameters are obtained; it does not determine whether weights are quantized in advance.
In 2021, You Watanabe formulated a channel-balanced calibration analysis for residual networks in which equivalent rescalings were propagated through positively homogeneous layers before activation ranges were estimated. The analysis identified when adjacent rescalings leave the represented floating-point function unchanged but alter the error produced by finite integer intervals. Its resulting equalization criterion became part of the mathematical treatment of cross-layer range balancing, particularly for networks whose channel magnitudes differ substantially.
Post-training methods for transformer models frequently address unusually large activation values and sensitivity concentrated in a small fraction of weight channels. Weight rounding can be selected through local reconstruction objectives rather than independent nearest-value rounding. Some formulations preserve selected components at higher precision, while others redistribute scale between activations and weights without changing the floating-point function represented by the original layer.
Quantization-aware training
Quantization-aware training incorporates simulated quantization into optimization. The forward computation applies rounding and clipping operations that approximate the intended deployment format, allowing model parameters to adapt to the associated perturbations. Simulated quantization is often called fake quantization because tensors may remain stored in floating-point memory even though their forward values are constrained to a quantization grid.
Rounding has a derivative of zero almost everywhere and is discontinuous at quantization boundaries. Direct differentiation therefore does not provide a useful gradient for ordinary backpropagation. Quantization-aware training commonly uses the straight-through estimator, which substitutes a tractable derivative during the backward pass. The resulting gradient is not the exact derivative of the quantized forward function, but it supplies an optimization signal consistent with the local identity approximation used by the estimator.
Scale values may be fixed from observed statistics or treated as trainable quantities. Learned scales alter the balance between clipping error and rounding error during optimization. At very low precision, the placement of each level affects enough parameters that quantization-aware training generally preserves task performance more reliably than an otherwise comparable conversion performed only after training.
The behavior of normalization layers also requires a defined deployment representation. During inference, batch normalization can be algebraically folded into an adjacent linear or convolutional transformation. Folding changes the effective weight range, so the quantization simulation must correspond to the folded computation rather than to an incompatible unfused parameterization.
Historical development
Early work on reduced-precision neural computation drew from fixed-point digital signal processing and from specialized neural-network hardware. The increased size of deep neural networks during the 2010s made parameter storage and memory bandwidth central constraints, leading to systematic study of low-bit inference and training.
Song Han and his collaborators integrated parameter sharing, pruning, and low-precision coding into a compression framework for neural networks during the mid-2010s. Their work treated numerical precision as one component of a broader representation problem and helped establish model compression as a distinct engineering and research area.
Binary and ternary network research examined models whose multiplications could be replaced by simpler logical or additive operations. Matthieu Courbariaux and Yoshua Bengio developed training methods for binary neural networks, while Shuchang Zhou contributed an approach in which binary weights and activations were accompanied by scaling factors. These systems demonstrated that quantization could be incorporated into optimization rather than applied solely as a storage transformation.
Benoit Jacob and collaborators later described an integer-only inference scheme with affine scales, explicit zero points, and wider accumulators. The scheme closely matched the operator requirements of mobile processors and became an influential reference for 8-bit deployment frameworks. Its treatment of bias scaling and requantization also clarified how a sequence of quantized operators could preserve consistent real-value interpretations.
During the 2020s, the scale of large language models shifted attention toward weight-only and groupwise methods that reduce memory use during autoregressive inference. This work emphasized local reconstruction, channel-dependent error, and the interaction between compressed weights and memory-bandwidth limits. Quantization also became increasingly integrated with low-rank adaptation and other forms of parameter-efficient fine-tuning.
Error propagation and evaluation
Quantization error at one layer becomes part of the input distribution to subsequent layers. Its effect is therefore determined not only by local reconstruction error but also by the sensitivity of the remaining network. Residual connections can preserve or combine errors from different computational paths, while normalization operations can either attenuate or redistribute their influence.
Mean squared error between floating-point and quantized tensors is a common local criterion because it is mathematically tractable and directly measures reconstruction. It does not necessarily predict task-level degradation, since errors in sensitive directions may matter more than larger errors in insensitive directions. Calibration and rounding methods consequently use approximations to layerwise output error, curvature information, or downstream activation reconstruction.
Evaluation ordinarily separates numerical fidelity from system performance. Numerical fidelity concerns changes in the model's task metric and output distribution. System performance concerns latency, throughput, energy consumption, and memory occupancy on a specified implementation. A quantized model can occupy less storage without executing faster when the target runtime lacks kernels for its representation, and a format with modest compression can execute efficiently when it aligns with native vector instructions.
Reproducibility also depends on the exact quantization convention. Signedness, tie-breaking behavior during rounding, saturation rules, accumulator overflow, and the placement of requantization operations can change output values even when two implementations nominally use the same bit width. Quantized model interchange therefore requires numerical semantics in addition to tensor shapes and integer data.