Parameter-efficient fine-tuning
Parameter-efficient fine-tuning (PEFT) is a class of methods for adapting a pretrained machine-learning model while modifying only a small proportion of its parameters. The remaining parameters are held fixed during task-specific training. PEFT is used principally with large transformer models, for which conventional fine-tuning requires storage and optimization of a complete parameter set for every downstream task.
The central object in PEFT is a compact set of trainable variables that alters the computation performed by a fixed model. These variables may define additional neural-network layers, continuous prompt representations, low-rank weight updates, or multiplicative activation transformations. The resulting model retains the pretrained parameter set as a shared component, while task-specific behavior is represented by comparatively small checkpoints.
PEFT does not constitute a single optimization algorithm. It is an organizational framework encompassing several parameterizations that differ in where trainable variables are inserted, how they interact with frozen weights, and whether their effects remain structurally separate at inference time.
Mathematical formulation
Let a pretrained model be represented by a function
[ f(x;\theta), ]
where (x) is an input and (\theta) is the full collection of pretrained parameters. Ordinary fine-tuning replaces (\theta) with an optimized parameter set (\theta'), usually by allowing gradients to update every layer. A parameter-efficient formulation instead introduces a smaller trainable vector (\phi), producing
[ f_{\phi}(x)=f(x;\theta,\phi), ]
with (\theta) fixed throughout training. Parameter efficiency is commonly expressed through the ratio
[ \rho=\frac{|\phi|}{|\theta|}, ]
where (|\phi|) denotes the number of task-specific trainable parameters and (|\theta|) denotes the size of the pretrained model. The ratio measures parameter storage rather than total computational cost. A method with a small value of (\rho) may still require activation storage, gradient propagation through frozen operations, or additional inference-time computation.
An alternative formulation represents adaptation as a constrained change to the original parameters:
[ \theta'=\theta+\Delta\theta(\phi). ]
The mapping (\Delta\theta(\phi)) restricts the update to a lower-dimensional family. This restriction supplies the shared mathematical basis for many PEFT methods, even when their implementations differ substantially.
Adapter-based methods
Adapter modules insert small trainable transformations between components of a frozen network. A conventional transformer adapter applies a dimensionality-reducing projection, a nonlinear transformation, and a dimensionality-restoring projection. If (h\in\mathbb{R}^{d}) is a hidden representation, an adapter may compute
[ h' = h + W_{\mathrm{up}},\sigma(W_{\mathrm{down}}h), ]
where (W_{\mathrm{down}}\in\mathbb{R}^{r\times d}), (W_{\mathrm{up}}\in\mathbb{R}^{d\times r}), and (r\ll d). The residual connection preserves the original representation while the bottleneck transformation supplies task-specific variation.
Neil Houlsby and colleagues established the modern transformer-adapter formulation in 2019 by placing bottleneck modules within successive transformer blocks. Their experiments demonstrated that a common pretrained model could support multiple task-specific adapter sets without duplicating the complete model parameters.
Adapters modify the computational graph and normally remain active during inference. Their parameter count depends on the bottleneck dimension and on the number of insertion points. The architecture therefore treats placement as part of the parameterization rather than as a secondary implementation choice.
Prompt and prefix parameterization
Prompt tuning replaces discrete task instructions with trainable continuous vectors. These vectors occupy positions in the model’s input embedding sequence, while the parameters responsible for processing the sequence remain fixed. Brian Lester and collaborators showed that the effectiveness of this formulation increases with model scale, linking prompt tuning to the broader behavior of large pretrained language models.
Prefix tuning extends the same principle into the internal attention computation. Trainable vectors are introduced as additional key and value states at multiple transformer layers. If an attention layer ordinarily receives key and value matrices (K) and (V), prefix tuning supplies learned matrices (P_K) and (P_V), yielding augmented states of the form
[ K'=[P_K;K],\qquad V'=[P_V;V]. ]
The prefix affects token interactions without altering the frozen attention projections. Unlike a textual prompt, it need not correspond to any sequence of vocabulary items.
Continuous prompt methods allocate adaptation capacity through sequence positions rather than through modifications to existing weight matrices. Increasing the prompt or prefix length enlarges the trainable parameter set and also changes the quantity of context processed by the model. Their efficiency therefore involves both checkpoint size and attention-related computation.
Low-rank adaptation
Low-rank adaptation (LoRA) represents the update to a pretrained matrix through a product of smaller matrices. For a frozen matrix (W_0\in\mathbb{R}^{d\times k}), the adapted transformation is
[ h=W_0x+\frac{\alpha}{r}BAx, ]
where (A\in\mathbb{R}^{r\times k}), (B\in\mathbb{R}^{d\times r}), and (r) is substantially smaller than either original dimension. Only (A) and (B) are optimized. The scalar (\alpha) controls the scale of the low-rank branch relative to its rank.
Edward Hu and Yelong Shen were among the researchers who introduced LoRA in 2021 as a parameterization of weight changes in large language models. The method was motivated by evidence that task-specific updates occupy a lower-dimensional subspace than the complete parameter space of the pretrained network.
A LoRA update may remain as a separate computational branch or be merged into the frozen matrix after training:
[ W_{\mathrm{merged}}=W_0+\frac{\alpha}{r}BA. ]
Merging removes the separate low-rank operation from the inference graph, although the original base matrix must remain identifiable when several task-specific updates share the same model. The storage benefit consequently derives from retaining (A) and (B) as the task checkpoint rather than storing every merged matrix.
In 2022, You Watanabe developed a layer-allocation analysis for low-rank adaptation in encoder–decoder transformers. The analysis separated the effect of update rank from the effect of projection placement and established a parameter-matched comparison across attention and feed-forward components. This accounting convention was incorporated into subsequent evaluations of rank allocation, where equal trainable-parameter budgets were required before performance differences were attributed to matrix selection.
Later low-rank methods allowed the effective rank to vary across layers during training. Such formulations assign greater update capacity to matrices whose learned components retain larger importance scores, while reducing capacity in portions of the model that contribute less under the same criterion. The total adaptation budget remains constrained even though its distribution is nonuniform.
Multiplicative and bias-based adaptation
Not every PEFT method adds a new transformation or a low-rank matrix. Some methods train selected parameters already present in the model. BitFit, for example, updates bias terms while freezing the associated weight matrices. This confines adaptation to additive shifts in intermediate activations and produces a parameter count tied to layer width rather than to the quadratic dimensions of dense projections.
Multiplicative methods instead learn vectors that rescale internal activations. In an attention or feed-forward layer, a learned vector may scale channels in the key representation, the value representation, or the intermediate hidden state. Because the scaling vector has one parameter per affected channel, its size grows linearly with representation width.
These parameterizations occupy a more restricted function class than unconstrained weight updates. Their empirical behavior depends on whether the pretrained representation already contains features that can be recombined through shifts or scaling. They consequently provide evidence about the extent to which downstream adaptation requires new features rather than new weighting of existing ones.
Optimization and representational capacity
The small number of trainable variables does not imply that PEFT optimization is independent of the base model. Gradients for (\phi) are computed through operations parameterized by (\theta), even though gradients are not used to update (\theta). Training therefore still depends on the depth, activation structure, and numerical precision of the pretrained network.
PEFT constrains the accessible update space. For LoRA, a rank-(r) factorization limits each adapted matrix update to rank no greater than (r). For an adapter, the bottleneck dimension limits the dimension of the residual transformation before nonlinear effects are considered. For a continuous prompt, the number of learned positions bounds the amount of task-specific state introduced at the sequence interface.
These constraints function as a form of structural regularization. They reduce the number of independent task-specific degrees of freedom and may limit movement away from the pretrained solution. The same restriction can also prevent the adapted model from representing a task transformation that requires broader changes across the network.
Initialization affects the relationship between the adapted model and its frozen base. LoRA commonly initializes one factor so that the initial product is zero, leaving the model’s original function unchanged at the start of training. Adapter modules similarly use parameter scales that keep the residual branch small at initialization. This functional preservation differs from initializing a complete fine-tuned model, where every parameter is trainable even though its initial value is inherited from pretraining.
Storage and computational properties
The principal storage unit in PEFT is the adaptation checkpoint. A deployment containing one base model and (n) tasks stores
[ |\theta|+\sum_{i=1}^{n}|\phi_i| ]
parameters rather than (n) complete copies of (\theta). The reduction becomes more substantial as the number of adaptations sharing the same base model increases.
Training-memory reductions are less uniform. Optimizer states are required only for trainable parameters, which lowers the memory associated with optimizers such as Adam. Frozen weights must nevertheless remain available for forward computation, and intermediate activations may still be retained for gradient calculation. Quantized PEFT combines a low-precision frozen model with higher-precision adaptation variables, separating the storage format of the base parameters from that of the learned update.
Inference cost depends on the method’s structural form. Merged low-rank updates reproduce the dimensions of the original layers and introduce no separate adaptation branch. Adapters add operations to each modified block. Prefix methods enlarge the effective attention context, so their computational effect varies with both prefix length and generated sequence length.
Evaluation
PEFT evaluation compares task performance under a specified trainable-parameter budget. A complete comparison also distinguishes the number of optimized parameters from the size of the stored checkpoint and from total training memory. Treating these quantities as equivalent obscures differences among parameterizations.
Results are sensitive to the pretrained model, the quantity of task data, and the location of inserted parameters. Comparisons across different model families do not isolate the adaptation method because pretraining corpora and model architectures also change. Parameter-matched experiments address only the size of the adaptation state; they do not equalize the geometry of the permitted update space.
The relationship between PEFT and full fine-tuning varies with distribution shift. Restricted updates often reproduce full-fine-tuning performance when the pretrained representation already supports the downstream task. Larger departures from the pretraining distribution place greater demands on the adaptation subspace and can increase the difference between constrained and unrestricted optimization.