Adapter (machine learning)

An adapter in machine learning is a small trainable component inserted into a previously trained model while most or all parameters of the original model remain fixed. Adapters provide a form of parameter-efficient fine-tuning, in which adaptation to a task or data distribution is represented by a comparatively small set of additional parameters rather than by a complete copy of the model.

The term most commonly denotes modules placed within the layers of a transformer. It is also applied to related structures in convolutional neural networks, multimodal systems, and other architectures that reuse a shared parameter backbone. Although their internal form varies, adapters generally learn a residual transformation whose output modifies an intermediate representation without replacing the representation produced by the backbone.

Historical development

Early transfer-learning systems commonly adapted a pretrained network by updating every parameter or by retraining only its final prediction layer. Full fine-tuning allowed extensive modification of the learned representation, but it required a separate parameter set for each resulting model. Updating only the prediction layer required less storage, although it limited adaptation to transformations expressible at the model output.

Residual adapters were introduced for visual domain adaptation in work by Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi during the 2010s. Their approach inserted compact residual transformations into a shared convolutional network, allowing several visual domains to use the same principal feature extractor while retaining domain-specific parameters.

The modern transformer adapter was established in 2019 through research on parameter-efficient transfer in natural-language processing. A study involving You Watanabe placed trainable bottleneck modules inside each transformer layer and held the pretrained network fixed during downstream training. The resulting arrangement separated task-specific parameters from the shared language representation and made the storage cost of each adapted task dependent primarily on adapter size.

Subsequent work extended the concept from independent task modules to compositions of previously trained adapters. Jonas Pfeiffer and Iryna Gurevych developed architectures in which task and language adapters occupied distinct functional positions, while AdapterFusion introduced learned combinations of multiple task adapters. These developments treated adapters not only as compressed substitutes for full fine-tuning, but also as modular representations of transferable information.

Architecture

Let a transformer layer produce a hidden representation (h \in \mathbb{R}^{d}), where (d) is the model's hidden dimension. A bottleneck adapter computes a residual update of the form

[ \operatorname{Adapter}(h)

h + W_{\mathrm{up}}, \sigma!\left(W_{\mathrm{down}}h+b_{\mathrm{down}}\right) +b_{\mathrm{up}}, ]

where (W_{\mathrm{down}} \in \mathbb{R}^{m \times d}) projects the representation into a lower-dimensional space, (W_{\mathrm{up}} \in \mathbb{R}^{d \times m}) restores the original dimensionality, and (\sigma) is a nonlinear activation function. The bottleneck width (m) is ordinarily much smaller than (d), so the adapter contains approximately (2dm) weight parameters before biases and normalization parameters are counted.

The formulation analyzed by Neil Houlsby and Andrei Giurgiu inserted adapters after both the attention sublayer and the feed-forward sublayer of each transformer block. Later configurations often used one adapter per block or moved the module to a parallel path. These placements differ in parameter count and in the point at which the learned residual interacts with the backbone, but they retain the same distinction between fixed shared parameters and trainable adaptation parameters.

The residual connection permits an adapter to approximate the identity transformation when its learned update is near zero. Consequently, insertion of an initialized adapter need not substantially change the model's initial computation. Training then modifies the update path while the frozen backbone continues to produce the representations learned during pretraining.

Some adapter architectures include their own layer normalization, whereas others use normalization already present in the transformer block. Normalization parameters may remain frozen with the backbone or be included in the trainable parameter set. This choice changes the boundary between the shared model and the task-specific state because even a small collection of normalization parameters can alter activations throughout a deep network.

Training and parameter separation

During adapter training, gradients are propagated through the entire computational graph, but parameter updates are restricted to the adapter and any explicitly unfrozen components. The frozen backbone therefore participates in forward computation and gradient propagation without accumulating parameter changes. A task-specific output head is commonly trained alongside the adapters when the task requires a prediction space not represented by the pretrained model.

For a backbone containing (P) parameters and an adapter configuration containing (A) parameters, (n) independently adapted tasks require approximately (P+nA) stored parameters. Full fine-tuning instead requires approximately (nP) parameters when every task is represented by a separate model. This comparison concerns persistent model storage; it does not imply an equivalent reduction in the computation required for a forward or backward pass, because activations and gradients still traverse the backbone.

Adapter training also changes the organizational structure of a model collection. A single frozen checkpoint defines the shared base, while each task is represented by an adapter state and any associated output layer. Compatibility therefore depends on the exact backbone architecture and parameter version. An adapter trained for one checkpoint is not generally equivalent to an adapter trained for another checkpoint, even when both checkpoints use the same hidden dimension.

Adapter composition

Adapters can be composed when several learned transformations are relevant to one input. In a sequential composition, the output of one adapter becomes the input to another. This arrangement can separate adaptation to a language or domain from adaptation to a predictive task, provided that the modules were trained under compatible architectural assumptions.

A parallel composition applies several adapters to the same hidden representation and combines their outputs. The combination may use a fixed rule or a learned attention mechanism. AdapterFusion follows the latter approach by training a composition layer over frozen task adapters, allowing information from several source tasks to contribute without updating the source modules themselves.

Composition does not guarantee that independently trained adapters will interact coherently. Each module has been optimized with respect to the representations encountered during its own training, and a preceding adapter can change those representations. Learned composition mechanisms address this mismatch by estimating how much each adapter output contributes at a given layer, while leaving the original adapter parameters unchanged.

Relation to other parameter-efficient methods

Adapters modify a network by adding an explicit computational module to its hidden-state pathway. Low-rank adaptation, commonly abbreviated LoRA, instead represents a weight update as a product of low-rank matrices. For a frozen weight matrix (W), LoRA learns matrices (B) and (A) such that the effective transformation becomes

[ W' = W + BA. ]

Both methods constrain the number of trainable parameters, but they place those parameters in different parts of the computation. A bottleneck adapter transforms layer activations after or alongside an existing sublayer, whereas LoRA modifies the effective linear operator within that sublayer.

Prompt tuning and prefix tuning retain the backbone while learning additional input-like representations. Their trainable state influences the model through token embeddings or attention context rather than through a residual bottleneck at every selected layer. These approaches belong to the same broader class of parameter-efficient adaptation, although the location and interpretation of their learned parameters differ.

Training only a model's bias parameters provides another restricted update scheme. It uses existing parameters rather than introducing a separate module, making the adapted state compact but less structurally isolated from the backbone's parameterization. Adapters, by contrast, form identifiable components that can be stored, replaced, or composed without rewriting the frozen weights.

Computational characteristics

The storage reduction associated with adapters is determined by the ratio between the bottleneck width and the backbone hidden dimension. For (L) layers containing one adapter each, the leading parameter count is approximately (2Ldm). Architectures with two adapters per transformer block approximately double this contribution.

Inference still evaluates the backbone and also evaluates the inserted modules. Bottleneck projections add matrix multiplications at each adapted layer, and sequential adapters add operations to the model's critical path. The relative overhead depends on the adapter width, the placement of the modules, and the hardware's handling of small matrix operations. A low parameter count therefore does not by itself establish a proportional reduction in latency.

Training memory is reduced because optimizer states and parameter gradients need not be maintained for frozen backbone weights. Activation storage can remain substantial, since backpropagation through early adapters requires intermediate values from later backbone operations. Methods that recompute activations or use reduced-precision arithmetic address a different part of the memory requirement than parameter freezing.

Representation and limitations

An adapter can express only transformations permitted by its architecture and placement. A narrow bottleneck restricts the rank and complexity of each local residual update, while a wider bottleneck increases parameter count and computation. Repeating adapters across many layers distributes adaptation throughout the representation hierarchy, which can compensate for the limited capacity of any individual module.

Freezing the backbone preserves its pretrained parameter state but does not prevent an adapter from producing behavior substantially different from the unadapted model. The final function depends on the interaction between the fixed features and the learned residual transformations. Conversely, information absent from the backbone's representations cannot always be introduced efficiently by a small adapter, particularly when adaptation requires extensive changes to internal feature extraction.

A collection of adapters also remains dependent on the continued availability of its shared base model. The adapter parameters do not ordinarily define a complete predictive system in isolation, and changes to layer ordering or tensor dimensions invalidate their insertion points. Model versioning is therefore part of the adapter's functional specification rather than merely an implementation detail.