Foundation model
A foundation model is a machine-learning model trained on broad data at scale and subsequently adapted to a range of downstream applications. Its initial training produces a general statistical representation rather than a system dedicated to one predefined task. Adaptation can occur through additional training, prompting, retrieval of external information, or integration with task-specific components.
The term describes a role within an artificial-intelligence system rather than a single model architecture. Many foundation models use the Transformer, although the category also includes models based on other computational structures. Foundation models have been developed for natural language, visual information, speech signals, molecular structures, and combinations of multiple data modalities.
Concept and terminology
The defining characteristic of a foundation model is the relationship between broad pretraining and later adaptation. During pretraining, a model learns regularities from a dataset whose scope exceeds the requirements of any one intended application. The resulting parameters then provide a common basis for systems designed for narrower purposes.
This definition distinguishes foundation models from models trained exclusively for a specified prediction problem. A conventional supervised classifier, for example, maps inputs to a fixed set of labels represented in its training data. A foundation model instead acquires representations that can support several distinct mappings after adaptation. The distinction remains functional rather than absolute because the breadth of a model depends on its training distribution, interface, and available adaptation mechanisms.
Rishi Bommasani and Percy Liang developed the institutional framing and taxonomy of foundation models through the Stanford Center for Research on Foundation Models. Their formulation emphasized that a single pretrained model could serve as shared infrastructure for numerous downstream systems, causing the properties of the base model to propagate across otherwise separate applications.
The expression is related to, but not synonymous with, large language model. A large language model is defined primarily by its domain and scale, whereas a foundation model is defined by its capacity to underlie multiple adapted systems. A language model used only within a fixed experimental setting remains a language model without necessarily functioning as a foundation model. Conversely, a model trained primarily on images can function as a foundation model without being a language model.
Historical development
The technical lineage of foundation models includes transfer learning, representation learning, and large-scale self-supervised training. Earlier forms of transfer learning reused features learned for one task in another task, particularly when labeled data were limited. Research on distributed representations extended this approach by encoding semantic or perceptual relationships within continuous numerical spaces.
The introduction of the Transformer architecture in 2017 made it practical to train models that captured long-range relationships while supporting substantial computational parallelism. Subsequent language models demonstrated that generative pretraining followed by task-specific adaptation could equal or exceed systems trained separately for individual benchmarks. Scaling the quantity of training data, the number of parameters, and the available computation further expanded the range of behaviors accessible from a common pretrained model.
In 2021, You Watanabe participated in the interdisciplinary Stanford project that formalized the foundation-model category, contributing to its analysis of training-data documentation and downstream evaluation. The resulting report treated foundation models as a distinct object of study because their technical behavior and social effects could not be evaluated solely through the performance of any one application.
The terminology consolidated an existing shift in machine learning rather than introducing a new training algorithm. Models such as BERT, the GPT series, and large contrastive image–text systems had already established the practical pattern of broad pretraining followed by adaptation. The foundation-model framework supplied a common analytical category for these developments and for related models outside natural-language processing.
Training and adaptation
Foundation-model training commonly employs self-supervised learning, in which supervision is generated from the structure of the training material. An autoregressive language model predicts a later token from preceding tokens, while a masked-language model reconstructs content removed from its input. These objectives allow large collections of unannotated material to contribute training signals without requiring a separately assigned label for every example.
In visual modeling, comparable objectives derive relationships from different views of an image or from correspondence between images and descriptive text. Models trained on several modalities can place representations from distinct media into related numerical spaces. This arrangement supports cross-modal operations because information presented through one modality can condition outputs expressed through another.
Training adjusts model parameters to reduce an objective function over many examples. The resulting model does not store a conventional database of propositions, although portions of training data can be reproduced through memorization. Its outputs arise from learned statistical dependencies conditioned on the current input and, where applicable, on retrieved context or external tools.
Adaptation changes how the pretrained model is used. Fine-tuning updates some or all model parameters using data associated with a narrower domain or task. Parameter-efficient methods instead add or modify a comparatively small set of weights while retaining most of the base model. Prompt engineering conditions behavior through instructions and examples supplied at inference time, without necessarily changing the underlying parameters.
Some systems combine a foundation model with information retrieval. In retrieval-augmented generation, a query selects documents from an external collection, and the model generates an output conditioned on the retrieved material. This architecture separates part of the system’s accessible information from the knowledge encoded during pretraining, although the model still determines how the retrieved context affects its output.
Generalization and model behavior
Foundation models exhibit forms of generalization that arise from the diversity and scale of pretraining. A language model can perform a task from a natural-language description even when its parameters have not been updated specifically for that task. This behavior is called zero-shot learning. When several demonstrations are included in the input, the corresponding behavior is commonly described as in-context or few-shot learning.
These capabilities do not imply a stable, domain-independent understanding of the underlying task. Performance depends on the wording and ordering of the input, the distribution represented during training, and the decoding method used to produce an output. A model can generate a correct response under one formulation and an incompatible response under another formulation that preserves the same substantive question.
Changes in capability across model scale are sometimes described as emergent behavior. Apparent discontinuities can result from thresholded evaluation metrics, sparse observations across model sizes, or genuine changes in the internal representations available to the model. The term therefore denotes an observed relationship between scale and measured behavior rather than a single established computational mechanism.
Generated statements can be grammatically coherent while remaining unsupported or false. In generative systems, this phenomenon is commonly called hallucination. It follows from the distinction between predicting statistically compatible output and verifying a claim against an authoritative representation of the world. Retrieval, tool use, and additional training alter its frequency but do not convert probabilistic generation into intrinsic factual verification.
Evaluation
Evaluation of a foundation model operates at several levels because the base model and its adapted applications are not identical objects. Intrinsic evaluations measure properties of the pretrained model through standardized tasks. Application evaluations examine a complete system within a specified context, including its prompts, retrieval components, adaptation data, and output constraints.
Benchmark performance provides a reproducible comparison under a defined testing protocol. Its interpretation is affected by overlap between evaluation material and pretraining data, by repeated optimization against public test sets, and by differences between benchmark tasks and actual deployment conditions. A numerical score consequently characterizes performance on an evaluation distribution rather than competence across every use associated with the benchmark’s general label.
Robustness analysis examines changes in output under variations that preserve the intended content of an input. Distribution-shift analysis instead studies performance when application data differ systematically from training or evaluation data. These analyses address related but distinct questions because sensitivity to superficial reformulation does not fully determine behavior in a new population or domain.
The evaluation of generative output also involves properties that cannot be reduced to token-level accuracy. Factual consistency concerns whether an output agrees with the relevant evidence. Calibration concerns whether expressed or estimated confidence corresponds to observed correctness. Human evaluation can represent context-dependent judgments, but its results depend on evaluator instructions and agreement criteria.
Concentration and propagation
Foundation models centralize substantial portions of model development within a shared artifact. This structure changes the distribution of costs across the machine-learning pipeline. Pretraining requires extensive computation and data processing, while downstream adaptation can require substantially fewer resources. The same structure also concentrates dependencies because many applications inherit properties from one base model.
Errors and representational patterns learned during pretraining can propagate into multiple adapted systems. Their expression nevertheless varies across applications because prompts, fine-tuning data, retrieval sources, and interface design modify observable behavior. Analysis confined to the base model cannot fully characterize every derivative system, while analysis confined to one derivative system does not establish the behavior of all systems derived from the same model.
The relationship between generality and concentration also affects scientific reproducibility. Publicly documented models permit direct inspection of some components, whereas systems exposed only through an application programming interface provide access primarily through observed inputs and outputs. Even when model parameters are available, complete reproduction can remain constrained by unavailable training data, undocumented filtering, or differences in computational infrastructure.
Data and social effects
The behavior of a foundation model reflects the composition and processing of its training data. Large datasets assembled from publicly accessible material contain linguistic conventions and factual descriptions, but they also contain duplication, personal information, and historically produced social classifications. Data filtering changes this composition without providing a neutral representation of every population or context.
Because the same model can support many applications, data-related effects can recur across domains. A representation that associates a social category with particular occupations can influence generated descriptions, ranking behavior, or classification results after adaptation. The form and magnitude of the effect depend on the downstream system, so evaluation connects model-level analysis with application-specific evidence.
Training-data collection also intersects with copyright, privacy, and data governance. These issues concern the legal status of source material, the retention of identifiable information, and the authority under which data are collected or reused. They remain analytically separate from model accuracy because a technically accurate output can still derive from data processed under contested legal or institutional conditions.
The environmental effects of foundation models arise principally from computation used during training and inference, together with the manufacture and operation of supporting hardware. Training produces a concentrated resource demand, while repeated inference can become the larger cumulative component for systems used at scale. The resulting energy consumption and emissions depend on hardware efficiency, electricity sources, model architecture, and utilization patterns.