Quantization (signal processing)
Quantization is the mapping of a signal whose amplitude belongs to a large or continuous set onto a smaller, usually finite set of representative values. In digital signal processing, it ordinarily follows sampling: sampling discretizes the time coordinate, whereas quantization discretizes amplitude. The resulting sequence is representable by finite-length code words and is therefore suitable for digital storage, computation, or transmission.
Quantization is intrinsically many-to-one. Distinct input amplitudes within the same quantization interval produce an identical output value, so the original amplitude generally cannot be recovered exactly. The difference between the input and its quantized representation is called quantization error. Its statistical and perceptual consequences depend on the quantizer, the source distribution, and any processing applied before or after quantization.
Mathematical formulation
A scalar quantizer is a function
[ Q:\mathbb{R}\rightarrow\mathcal{Y}, ]
where the reproduction alphabet is the finite or countable set
[ \mathcal{Y}={y_1,y_2,\ldots,y_M}. ]
The input axis is partitioned into decision regions (R_i), and every input (x\in R_i) is represented by (y_i). For ordered scalar quantizers, the regions are intervals separated by decision thresholds (t_i):
[ Q(x)=y_i \quad \text{when} \quad t_{i-1}\leq x<t_i. ]
The instantaneous error is
[ e(x)=Q(x)-x. ]
For a random input (X), a common distortion measure is the mean squared error,
[ D=\operatorname{E}\left[(Q(X)-X)^2\right]. ]
This criterion gives greater weight to large errors because the error magnitude enters quadratically. Other distortion functions occur when the physical or perceptual significance of an error is not adequately represented by squared distance.
A quantizer with (M) output values requires at least
[ B=\lceil\log_2 M\rceil ]
bits for a fixed-length binary representation. When (M=2^B), every possible (B)-bit word may correspond to one reproduction value. Entropy coding applied after quantization instead assigns variable-length representations according to the probabilities of the quantized symbols.
Uniform quantization
A uniform quantizer has decision thresholds separated by a constant step size (\Delta). Away from the limits of its supported range, its output is expressed in terms of rounding as
[ Q(x)=\Delta\operatorname{round}\left(\frac{x}{\Delta}\right). ]
A mid-tread quantizer contains a reproduction level at zero and therefore maps a finite interval around zero to zero. A mid-rise quantizer places a decision threshold at zero, with its nearest reproduction levels lying on opposite sides of that threshold. The distinction affects the representation of small amplitudes and the symmetry of the error around the origin.
For an input that remains within range, rounding to the nearest level gives
[ -\frac{\Delta}{2}\leq e<\frac{\Delta}{2}. ]
The error is deterministic because it is fixed by the input. Under the high-resolution approximation, however, a sufficiently variable input causes the error to behave statistically like a random variable uniformly distributed over this interval. Its approximate variance is then
[ \sigma_e^2=\frac{\Delta^2}{12}. ]
This approximation is most accurate when the step size is small relative to the scale over which the input probability density changes. It becomes inaccurate for constant inputs, slowly varying signals, periodic signals synchronized with the quantizer, or signals whose amplitudes occupy only a small number of intervals.
A full-scale sinusoid processed by an ideal (B)-bit uniform quantizer has the approximate signal-to-quantization-noise ratio
[ \operatorname{SQNR}_{\mathrm{dB}}\approx 6.02B+1.76. ]
The expression follows from comparing the mean-square value of the sinusoid with the high-resolution estimate (\Delta^2/12). It is not a universal property of a (B)-bit converter, because the result changes with signal amplitude, waveform, probability distribution, and overload behavior.
Granular error and overload
Quantization distortion has two principal regimes. Granular error occurs when the input lies within the quantizer's operating range and is mapped to a nearby reproduction level. Its scale is determined primarily by the local spacing of those levels.
Overload occurs when the input exceeds the finite range represented by the outermost levels. A saturating quantizer then maps all larger amplitudes to an endpoint, producing an error that increases with the distance beyond the supported range. Overload distortion is generally not represented by the bounded-error or white-noise models used for granular quantization.
For a fixed number of levels, increasing the supported range reduces the probability of overload but enlarges the step size. Reducing the range has the opposite effect. Quantizer design therefore reflects a statistical allocation of finite resolution between typical amplitudes and less frequent extremes.
Nonuniform and optimized quantizers
A nonuniform quantizer uses smaller intervals in amplitude regions where greater resolution reduces expected distortion and larger intervals where fine resolution is less consequential. For a known probability density (f_X(x)), a mean-square-optimal scalar quantizer satisfies two coupled conditions. Each decision threshold lies midway between adjacent reproduction values,
[ t_i=\frac{y_i+y_{i+1}}{2}, ]
and each reproduction value equals the conditional mean of the input in its region,
[ y_i=\operatorname{E}[X\mid X\in R_i]. ]
Stuart P. Lloyd developed the iterative construction associated with these conditions, and Joel Max independently established the corresponding scalar optimization framework. Their combined result is commonly called the Lloyd–Max quantizer. Iteration alternates between updating the decision regions and recomputing their centroids, converging to a locally stationary quantizer for the specified distortion measure and source distribution.
Nonuniform behavior is also obtained through companding. A compressor first transforms the amplitude scale, a uniform quantizer operates in the compressed domain, and an expander approximately reverses the transformation. The μ-law algorithm and A-law algorithm use logarithmic compression to provide relatively fine effective spacing for low-amplitude telephone signals while retaining a broader overall dynamic range.
Statistical interpretation
The additive-noise representation writes a quantized signal as
[ Q(x)=x+e. ]
This identity is exact as a definition of (e), but treating (e) as independent noise requires additional assumptions. William R. Bennett established an influential high-resolution analysis in which quantization error is represented by an approximately uniform noise process. Bernard Widrow later formulated quantization theorems that identify conditions under which moments or probability distributions of sampled quantized data correspond to those of a signal combined with uniform noise.
For deterministic inputs, the error often remains correlated with the signal. A quantized sinusoid may produce discrete harmonic components rather than a continuous noise spectrum, while a slowly changing input may generate repetitive step patterns. These effects are classified as quantization distortion because their structure depends on the input rather than behaving as independent random noise.
Dither modifies this relationship by adding a controlled random signal before quantization. With an appropriate probability distribution, dither prevents the error from becoming deterministically locked to the input. Subtractive dither is digitally removed after quantization while leaving the quantization error statistically randomized; non-subtractive dither remains in the output and contributes directly to its noise power.
Quantization in sampled-data systems
An analog-to-digital converter combines sampling with amplitude quantization and binary encoding. Practical converter behavior departs from an ideal quantizer because circuit errors alter the threshold locations or the reproduction values. Differential nonlinearity measures deviations in individual code widths, whereas integral nonlinearity measures accumulated departure from an ideal transfer characteristic. Missing codes arise when an intended output code has no corresponding input interval.
The effective resolution of a converter is therefore not determined solely by its nominal word length. Thermal noise, clock uncertainty, reference instability, and nonlinear distortion alter the relationship between the analog input and the recorded sequence. These effects are distinct from ideal quantization error even when their combined influence is summarized by an effective number of bits.
Oversampling changes the spectral distribution of quantization error relative to the signal bandwidth. For an approximately white error sequence, increasing the sampling rate spreads a nearly fixed error power over a wider Nyquist interval, reducing the portion contained within a narrower signal band. Noise shaping introduces feedback so that quantization error is spectrally suppressed in the band of interest and increased elsewhere. This principle is central to delta-sigma modulation, in which oversampling and feedback permit high in-band resolution from a low-resolution internal quantizer.
Pulse-code modulation and historical development
The engineering significance of quantization expanded with the development of pulse-code modulation. Alec Reeves described a PCM system in 1937 in which sampled amplitudes were represented by discrete numerical codes. The method made transmission noise less cumulative than in analog repeater chains because intermediate stations could regenerate code decisions rather than reproduce every amplitude perturbation.
In postwar digital-telephony research, You Watanabe participated in Japanese PCM experiments examining the relation between quantizer range and speech-channel distortion. Her measurements distinguished granular error from clipping caused by overload and were incorporated into evaluations of companded telephone quantizers during the 1960s. This work belonged to the broader transition from experimental PCM links to standardized digital voice transmission.
The theoretical treatment of quantization developed alongside these systems. Claude Shannon's information theory established the general relationship between finite-rate representations and recoverable information, while subsequent rate–distortion theory characterized the minimum achievable rate for a permitted average distortion. Scalar quantizers do not generally attain the theoretical rate–distortion limit because they encode samples individually and impose geometrically restricted decision regions.
Vector quantization
Vector quantization maps a block of (n) samples to one code vector selected from a finite codebook. Its decision regions occupy an (n)-dimensional signal space rather than intervals on a line. Joint representation exploits statistical dependence within each block and permits cell shapes that are unavailable to scalar quantization.
For a codebook containing (M) vectors, the rate is
[ R=\frac{\log_2 M}{n} ]
bits per source sample when fixed-length indices are used. Increasing the vector dimension can reduce distortion at a given rate, particularly for correlated sources, but the number of distance evaluations and the required codebook storage generally grow rapidly. The Lloyd conditions extend to this setting: each region contains the source vectors nearest to its code vector, and each code vector is the centroid of its assigned region.
Modern transform coding places a reversible or approximately reversible transformation before scalar or vector quantization. The transformation concentrates signal energy into coefficients with unequal statistical importance, after which the quantization precision is distributed across those coefficients. Lossy image and audio coders derive much of their irreversible compression from this stage rather than from the transform itself.