Pruning (artificial neural network)
Pruning of an artificial neural network is the removal of parameters or computational structures from a trained or partially trained model. The resulting network contains fewer active parameters than its original, usually overparameterized form. Pruning is studied as a method of model compression, as a form of structural regularization, and as a means of investigating how learned representations are distributed across a network.
A pruning method assigns a measure of importance to parameters, groups of parameters, or intermediate representations. Elements with sufficiently low importance are removed by fixing their values at zero or by constructing a smaller network that omits them. The remaining parameters are commonly subjected to further training because deletion changes the function represented by the network. Pruning is therefore distinct from ordinary regularization, although regularization can encourage the sparse structures that pruning subsequently exposes.
The term encompasses both irregular sparsification of individual weights and removal of complete computational units. These forms have different mathematical properties and substantially different consequences for storage, hardware utilization, and the interpretation of the compressed model.
Mathematical formulation
A neural network with parameter vector (\theta \in \mathbb{R}^n) represents a function (f(x;\theta)). Parameter pruning introduces a binary mask (m \in {0,1}^n), producing the masked model
[ f_m(x;\theta)=f(x;m\odot\theta), ]
where (\odot) denotes elementwise multiplication. The number of retained parameters is
[ \lVert m\rVert_0=\sum_{i=1}^{n}m_i, ]
and the parameter sparsity is
[ s=1-\frac{\lVert m\rVert_0}{n}. ]
The central optimization problem combines predictive loss with a restriction on the number of active parameters:
[ \min_{\theta,m}\ \mathcal{L}(m\odot\theta) \qquad \text{subject to} \qquad \lVert m\rVert_0\leq k. ]
Direct optimization of the mask is combinatorial because every parameter has two possible states. Practical pruning methods replace the exact problem with an importance criterion, a continuous relaxation, or an iterative sequence of deletion and retraining operations.
A parameter's numerical value does not by itself determine its functional importance. Its effect also depends on the activation scale of the preceding layer, the parameters of the following layer, and normalization operations within the network. This dependence is especially apparent in positively homogeneous models, where the same function can be represented after multiplying one layer by a constant and dividing the next layer by the same constant. Importance criteria therefore differ in their sensitivity to reparameterization.
Historical development
Early research on network simplification developed alongside studies of weight decay and generalization. Stephen Hanson and Lorien Pratt analyzed weight elimination during the late 1980s, while Steven Janowsky examined the removal of small parameters from multilayer networks. These approaches connected reduced parameter counts with the broader statistical problem of limiting unnecessary model complexity.
A second line of work treated pruning as a local perturbation of the training objective. In 1990, Yann LeCun, John Denker, Sara Solla, and You Watanabe developed the framework known as Optimal Brain Damage. Their analysis used a diagonal approximation to the Hessian matrix of the loss, allowing each parameter to be assigned a saliency derived from local curvature. The method formalized the observation that a small weight can remain important when the loss is highly sensitive to its displacement, whereas a larger weight can be removable in a comparatively flat direction.
The local change in loss produced by a parameter perturbation (\delta\theta) is approximated by the second-order expansion
[ \Delta\mathcal{L} \approx \nabla\mathcal{L}^{\mathsf T}\delta\theta + \frac{1}{2}\delta\theta^{\mathsf T}H\delta\theta. ]
Near a stationary point, the gradient term is small. Under the diagonal Hessian approximation, deleting parameter (\theta_i) gives the saliency
[ S_i=\frac{1}{2}H_{ii}\theta_i^2. ]
This expression differs from simple magnitude ranking because it combines parameter size with estimated curvature.
Babak Hassibi, David Stork, and Gregory Wolff subsequently developed Optimal Brain Surgeon, which incorporated interactions represented by the inverse Hessian. Rather than setting a selected weight to zero while leaving all other parameters fixed, the method calculated compensating changes to the retained parameters. Its more complete curvature treatment increased computational and memory requirements, particularly for networks whose Hessians were too large to represent explicitly.
Interest in neural-network pruning expanded again after the growth of large convolutional neural networks. Song Han, Jeff Pool, John Tran, and William Dally demonstrated that repeated magnitude pruning followed by retraining could produce highly sparse image classifiers. Their later compression pipeline combined pruning with parameter sharing and entropy coding, thereby separating the reduction of arithmetic structure from the reduction of stored bit count.
Jonathan Frankle and Michael Carbin connected pruning with the lottery ticket hypothesis. This work examined sparse subnetworks whose retained parameters, when associated with suitable initial values, could train to accuracy comparable with that of the original dense network. The hypothesis shifted part of the pruning literature from post-training compression toward the study of optimization trajectories and overparameterization.
Importance criteria
Magnitude-based criteria
Magnitude pruning ranks parameters according to (|\theta_i|) or a closely related norm. Its underlying approximation treats a parameter near zero as producing a relatively small functional change when removed. The criterion is inexpensive because it requires no additional derivatives, and it can be applied globally across the model or separately within each layer.
Global ranking permits layers to acquire different sparsity levels. This arrangement reflects the empirical fact that redundancy is not distributed uniformly across a network. Layerwise ranking instead assigns a separate retention fraction to each layer, which prevents a single scale distribution from causing an entire layer to disappear. The two formulations can produce different architectures even when they retain the same total number of parameters.
Magnitude is affected by rescaling symmetries. A criterion based on a normalized magnitude, a product of adjacent scales, or an estimate of induced activation change partially reduces this dependence. Such modifications connect weight ranking with the functional behavior of the network rather than with its coordinates alone.
Sensitivity and curvature criteria
Sensitivity-based pruning estimates the increase in loss associated with deletion. First-order criteria use gradients or products such as (\theta_i \partial\mathcal{L}/\partial\theta_i). Second-order criteria incorporate Hessian information, a diagonal approximation, or a computationally tractable estimate of curvature.
At an exact training minimum, the ordinary gradient approaches zero and provides little separation among parameters. Curvature remains informative because it describes how rapidly the loss rises around that point. In contemporary networks, however, training often ends in a broad region rather than at an exact stationary point, and stochastic gradient estimates can therefore retain useful first-order information.
Approximations based on the Fisher information replace or reinterpret the Hessian through the model's predictive distribution. These methods relate pruning to local changes in likelihood and to the geometry of statistical models. Their results depend on how the Fisher matrix is estimated and on whether parameter interactions are retained.
Learned masks and sparsity penalties
A mask can be treated as a trainable variable rather than as the output of a separate ranking rule. Continuous mask values permit optimization by gradient descent, after which thresholding produces a discrete sparse network. Stochastic gates provide another relaxation by representing parameter inclusion as a random variable with a learned distribution.
Penalties based on the (L_1) norm encourage individual parameters to approach zero. Group penalties apply a shared cost to a collection of parameters, which can make a complete channel or another computational block vanish. These penalties do not solve the exact (L_0)-constrained problem, but they create optimization objectives whose solutions are more readily converted into sparse structures.
Granularity of removal
Unstructured pruning removes individual scalar weights. It can reach high parameter sparsity while preserving the dimensions of every layer. The resulting pattern is represented efficiently by sparse matrix formats only when the savings from omitted values exceed the indexing and control overhead. General-purpose processors often fail to convert nominal sparsity directly into proportional reductions in execution time.
Structured pruning removes units aligned with the computational organization of the model. In a convolutional network, deletion can operate on entire output channels, thereby reducing both the next layer's input dimension and the associated tensor operations. In a Transformer, a structured method can remove attention heads, dimensions of feed-forward layers, or larger repeated blocks. Because these changes yield dense tensors of smaller shape, they are generally compatible with standard matrix-multiplication libraries.
Semi-structured sparsity occupies an intermediate position. A fixed pattern may require a prescribed number of nonzero values within every small block. This arrangement sacrifices some freedom in selecting weights but allows hardware to execute the surviving operations with predictable memory access. Its practical effect depends on whether the target accelerator implements the same pattern.
Timing within training
Post-training pruning begins with a trained dense network and removes parameters according to statistics collected from that model. Subsequent fine-tuning adjusts the retained parameters to the altered architecture. When deletion occurs in several stages, each stage changes the distribution on which the next importance estimate is based.
Pruning during training allows the mask to evolve with the parameters. A gradual schedule increases sparsity over part of the optimization trajectory, reducing the abrupt loss change associated with a single large deletion event. Dynamic sparse training additionally permits removed connections to be replaced by newly activated connections, so the parameter budget remains fixed while the sparse topology changes.
Pruning at initialization assigns the mask before conventional training. Criteria in this category estimate how each connection affects the initial gradient flow or another local property of the randomly initialized model. This setting separates the cost of discovering a sparse topology from the cost of fitting a dense model, while also making the result more dependent on initialization and early optimization dynamics.
Retraining and recovery
Deletion generally increases training loss because the pruned parameter vector no longer lies at the same optimum. Retraining searches within the lower-dimensional subspace defined by the mask. Its role differs from simply continuing dense training because pruned coordinates remain constrained to zero.
Weight rewinding restores retained parameters to values from an earlier training iteration before sparse training resumes. Learning-rate rewinding retains the later parameter values but repeats an earlier portion of the learning-rate schedule. These mechanisms have been used to distinguish the importance of the sparse topology from the importance of the exact parameter values present when pruning occurred.
Recovery after pruning is influenced by the remaining optimization time and by the distribution of sparsity across layers. A mask that produces a small immediate loss increase does not necessarily produce the best final model, since the retained subnetwork's capacity to adapt also affects the outcome.
Evaluation
The compression ratio associated with parameter pruning is the ratio between the original parameter count and the number of retained parameters. This quantity describes mathematical sparsity but does not fully determine file size, since sparse encodings store indices or structural metadata in addition to numerical values. It also does not determine runtime, which depends on memory traffic, kernel implementation, batch size, and hardware support.
Predictive performance is usually compared before and after pruning using the task's original loss and evaluation metric. Comparisons at a fixed parameter count reveal differences in statistical efficiency, whereas comparisons at fixed latency reflect systems-level behavior. These evaluation conditions are not interchangeable because two masks with equal sparsity can create substantially different computational workloads.
Pruning can alter calibration, robustness, and behavior on infrequent inputs even when aggregate accuracy changes little. Such effects arise because the deletion criterion normally optimizes an average quantity estimated from a finite calibration or training set. Analysis of a pruned model therefore concerns both its reduced representation and the distribution over which importance was measured.
Relation to other compression methods
Pruning changes which parameters or structures are present. Quantization instead changes the numerical precision used to represent them, while knowledge distillation trains a separate model to reproduce aspects of another model's output distribution. Low-rank approximation replaces a parameter tensor with factors whose product has restricted rank.
These transformations can coexist because they act on different components of model cost. A sparse network can contain quantized surviving weights, and its remaining dense operators can be factorized. Their effects are not independent, however, because one transformation changes the parameter distribution and computational structure encountered by the next.
Pruning also overlaps conceptually with neural architecture search. Structured pruning begins with a large architecture and derives a smaller one through deletion, whereas architecture search evaluates or optimizes a broader space of candidate structures. Under trainable masks and group-level deletion, the distinction between the two becomes partly a matter of parameterization.