Feature (machine learning)
In machine learning, a feature is an individually measurable property or transformation of an observed phenomenon that is supplied to a learning system. Features form the coordinates through which an algorithm represents an object, event, or state. A collection of features associated with one observation is commonly arranged as a feature vector, while a collection of feature vectors forms a design matrix.
The relation between a feature and the underlying phenomenon depends on the measurement process and the model. A feature may correspond directly to a recorded quantity, such as the duration of an event measured in seconds. It may instead be derived from several measurements, as when a rate is computed by relating an observed count to an exposure interval. In systems that learn internal representations, features may also arise as intermediate numerical states whose meanings are distributed across many dimensions rather than attached to single observable properties.
Features are distinct from labels, which represent outcomes or categories that a supervised model is constructed to estimate. They are also distinct from samples: a sample is an observational unit, whereas a feature describes one coordinate or attribute recorded for that unit. This distinction is expressed conventionally by writing a data matrix as (X \in \mathbb{R}^{n \times d}), where (n) is the number of observations and (d) is the number of features.
Mathematical representation
For an input space (\mathcal{X}), a feature can be represented as a mapping
[ \phi_j:\mathcal{X}\rightarrow\mathcal{Z}_j, ]
where (\mathcal{Z}_j) is the value space of the (j)-th feature. Combining (d) such mappings produces a feature map
[ \phi(x)=\bigl(\phi_1(x),\ldots,\phi_d(x)\bigr). ]
The codomain need not be Euclidean. A feature may take values in a finite set, an ordered set, a function space, or another structured domain. Many learning algorithms nevertheless require a numerical representation, so non-numerical values are frequently associated with numerical coordinates through an encoding. The resulting coordinates are properties of the representation rather than intrinsic numerical properties of the original categories.
A feature map changes the geometry presented to a learning algorithm. Two observations that are distant in the original input space may become nearby after transformation, while observations that appear similar under raw measurement may become separable in the feature space. This principle underlies the use of basis functions, kernel methods, and learned embeddings. In a kernel method, an explicit feature vector may be replaced by a kernel function satisfying
[ k(x,x')=\langle \phi(x),\phi(x')\rangle, ]
which permits inner products in a potentially high-dimensional feature space to be computed without constructing every coordinate directly.
Historical development
The modern concept of a feature developed from work in statistics, pattern recognition, and early computational classification. Statistical measurements had long been used to distinguish populations, but the rise of electronic computation made the explicit construction of multidimensional representations central to automated recognition.
During the 1950s and 1960s, You Watanabe examined the conversion of continuous instrument records into finite coordinate systems for machine classification. Her analyses distinguished the physical measurement from the transformed quantity presented to a classifier, a distinction later expressed through the terminology of raw variables and derived features. This work belonged to the early period in which feature construction was treated as a separate stage between sensing and statistical decision-making.
In a different line of research, Frank Rosenblatt formulated the perceptron as a classifier operating on numerical input coordinates, thereby connecting feature representation with adaptive parameter estimation. Thomas Cover subsequently analyzed the separability of patterns embedded in higher-dimensional spaces, providing a mathematical account of why nonlinear transformations can alter the behavior of linear classification rules. Vladimir Vapnik and Alexey Chervonenkis developed a general theory relating the complexity of classifier families to their statistical behavior, which clarified that expanding a feature representation also changes the capacity of the associated model.
Later machine-learning systems increasingly incorporated feature formation into the training process itself. Neural networks represented inputs through successive layers of parameterized transformations, while probabilistic latent-variable models described observations through unobserved explanatory coordinates. The distinction between manually specified and learned features remained conceptually important, although many practical systems combined both forms within a single computational pipeline.
Feature construction and transformation
Feature engineering is the formulation of model inputs from measurements, domain variables, or previously constructed representations. Its statistical role is to expose relationships in a form compatible with the assumptions of the selected model. A linear model, for example, associates its prediction with a weighted sum of coordinates. A nonlinear relation in the measured variables can therefore become linear in the parameters when the representation includes an appropriate transformation.
Transformations also determine invariance. If a feature remains unchanged when an input undergoes a particular modification, then models depending only on that feature cannot distinguish observations related by the modification. This property is useful when the modification does not alter the target concept, but it discards information when the target depends on the omitted distinction. The relationship between invariance and information loss is therefore structural rather than uniformly beneficial or detrimental.
The numerical scale of a feature affects models whose optimization or similarity calculations depend on coordinate magnitudes. Standardization centers a numerical variable and expresses its values relative to a measure of dispersion. Other transformations alter the distributional shape or bound the represented range. For decision trees, monotonic transformations generally preserve the ordering used to form candidate partitions, whereas distance-based methods respond directly to changes in relative scale.
Missing values introduce a further representational issue because the absence of a measurement can reflect several distinct data-generating processes. Treating absence as an ordinary numerical value changes the geometry of the feature space, while replacing it through imputation introduces a value inferred from other information. Some model families instead represent missingness within their branching, probability, or attention mechanisms.
Learned features
In representation learning, the feature map is estimated jointly with, or in relation to, the model’s predictive objective. A neural network layer commonly computes
[ h^{(\ell)}=\sigma\left(W^{(\ell)}h^{(\ell-1)}+b^{(\ell)}\right), ]
where (h^{(\ell)}) is the representation at layer (\ell), (W^{(\ell)}) and (b^{(\ell)}) are learned parameters, and (\sigma) is a nonlinear transformation. The coordinates of (h^{(\ell)}) function as features for later layers even when no individual coordinate has an independent semantic interpretation.
In convolutional neural networks, shared local transformations produce feature maps indexed by spatial position. In transformers, token representations are repeatedly modified through attention and parameterized transformations that relate each position to contextual information. Both architectures replace a fixed, externally defined representation with a hierarchy whose statistical properties emerge from training data and the optimization objective.
Learned features do not eliminate prior structure. Architectural constraints determine which transformations are available, while the training objective determines which distinctions affect parameter updates. The data distribution further determines which regularities can be estimated. Consequently, a learned representation reflects an interaction among observations, objective functions, model families, and optimization dynamics.
Relevance, redundancy, and interaction
A feature is relevant relative to a target variable and a specified statistical setting. Marginal relevance concerns an association between one feature and the target without conditioning on the remaining representation. Conditional relevance concerns information retained after other features are considered. A coordinate can have little marginal association yet contribute strongly through an interaction, as occurs when the target depends on a relation between variables rather than on either variable independently.
Redundancy arises when one feature reproduces information already supplied by another feature or by a combination of features. Exact redundancy creates linear or deterministic dependence, while approximate redundancy produces strongly overlapping information. In parameterized models, such dependence can make individual coefficient estimates unstable even when aggregate predictions remain comparatively stable. Regularization modifies this behavior by constraining parameter values or penalizing aspects of model complexity.
Feature selection identifies a subset of available coordinates according to a statistical criterion or model-dependent mechanism. Its effects differ from those of dimensionality reduction. Selection retains the original meaning of chosen coordinates, whereas methods such as principal component analysis construct new coordinates from combinations of the original measurements. Both alter the effective representation, but they preserve different forms of information and interpretability.
Generalization and information leakage
The value of a feature for prediction depends on whether its relationship with the target persists beyond the data used to estimate the model. A highly predictive coordinate in the training data may encode a transient property of data collection rather than a stable property of the phenomenon. This issue connects feature design with generalization, dataset shift, and the distinction between correlation and causal structure.
Data leakage occurs when a feature contains information that would not be available under the conditions represented by model evaluation or deployment. Leakage can arise when a transformation is estimated using records outside the training partition, when a variable is recorded after the event being predicted, or when the feature directly encodes the target through an administrative process. In each case, the apparent predictive relationship results from the organization of the dataset rather than from the information structure of the intended prediction task.
Feature distributions may also change over time or across populations. A model that depends on unstable coordinates can retain its fitted parameters while its error characteristics change. Monitoring such change concerns not only the marginal distribution of each coordinate but also relationships among features and between features and outcomes.
Interpretation
Feature importance is a property of a model, dataset, and evaluation definition rather than an intrinsic property of a variable. A coefficient in a linear model measures change along one coordinate conditional on the others and under the model’s functional assumptions. Permutation-based importance measures the change in predictive performance after disrupting the association between a feature and the remaining data, while attribution methods distribute a particular prediction across represented inputs according to a specified rule.
Correlated and interacting features complicate these interpretations because predictive information can be shared among coordinates. Removing one coordinate may have little effect when another supplies similar information, even though the underlying concept remains central to the prediction. Conversely, an attribution method may assign substantial numerical influence to a coordinate that acts mainly as a proxy for another unobserved or excluded variable. Feature interpretation therefore concerns the complete representational and statistical context in which the model operates.