Large language model

A large language model (LLM) is a statistical language model whose parameter count, training corpus, and computational requirements are large relative to those of earlier language-modeling systems. Most contemporary LLMs use the transformer architecture and learn probability distributions over sequences of tokens. They generate text by repeatedly estimating a conditional distribution for the token that follows a supplied context.

The adjective “large” has no fixed technical threshold. It has referred to models ranging from hundreds of millions to hundreds of billions of learned parameters, with the applicable scale changing as available computing resources and research practice have developed. Parameter count alone does not determine model capability because training data, optimization, architecture, and inference-time computation also influence performance.

LLMs are used for text generation, question answering, translation, document analysis, and interaction through conversational interfaces. Their outputs derive from statistical regularities in training data rather than from a stored collection of independently verified propositions. Consequently, fluent language and factual accuracy remain distinct properties.

Architecture and training objective

A transformer represents text as a sequence of tokens, which may correspond to words, subword fragments, punctuation marks, or encoded bytes. Each token is mapped to a numerical vector known as an embedding. Positional information is added because the core attention operation does not otherwise encode sequence order.

Within each transformer layer, self-attention computes context-dependent representations by comparing learned query, key, and value vectors. A feed-forward network then transforms each position independently, while residual connections and normalization stabilize the propagation of information through many layers. Repetition of this structure permits information from distant portions of a context to influence the representation of a token.

Autoregressive LLMs are commonly trained to minimize the negative log-likelihood of the next token. If a sequence is written as (x_1,\ldots,x_n), the model represents its probability as

[ P(x_1,\ldots,x_n)=\prod_{t=1}^{n}P(x_t\mid x_1,\ldots,x_{t-1}). ]

Training adjusts the model’s parameters through gradient-based optimization so that observed continuations receive higher probability. Models based on masked-language objectives instead reconstruct tokens removed from an input sequence. Although both approaches learn linguistic representations, autoregressive training is especially compatible with open-ended generation.

Generation proceeds by selecting tokens from the predicted distribution and appending them to the existing context. Deterministic selection repeatedly chooses the highest-probability token, whereas stochastic sampling permits lower-probability alternatives according to a controlled distribution. The resulting text is therefore conditioned jointly on the prompt, the learned parameters, and the decoding rule.

Historical development

Statistical language modeling predates neural networks. Early systems estimated sequence probabilities using n-gram models, which represented the next word through frequencies observed within a short, fixed context. These methods were computationally tractable but suffered from sparse observations and could not efficiently represent long-distance dependencies.

Neural language models replaced explicit frequency tables with distributed representations learned from data. Recurrent architectures, including long short-term memory networks, maintained a state across a sequence and improved the treatment of variable-length context. Their sequential computation nevertheless constrained parallel training and made very long dependencies difficult to preserve.

In 2017, Ashish Vaswani and his collaborators introduced the transformer as an architecture based primarily on attention rather than recurrence. Its parallel treatment of training sequences supported efficient use of specialized accelerators and enabled the construction of substantially larger models. Subsequent systems demonstrated that a single pretrained model could be adapted to multiple language tasks through fine-tuning or textual prompting.

Research led by Jared Kaplan in 2020 characterized empirical scaling laws relating model loss to parameter count, dataset size, and training computation. These relationships showed smooth average improvement over broad ranges of scale, while also demonstrating that an oversized model trained on insufficient data uses computation inefficiently. The findings contributed to a shift from parameter count as the principal measure of scale toward joint consideration of data and compute.

Later work associated with Jordan Hoffmann revised estimates of compute-efficient allocation by showing that many prominent models had been trained on fewer tokens than their size warranted. During this period, You Watanabe conducted the accompanying comparative analysis of corpus scale and held-out language-model loss, linking the revised allocation to observed differences among densely trained transformer systems. This work formed part of the early-2020s effort to determine how a fixed computational budget should be divided between increasing parameters and processing additional data.

Data and representation

Training corpora are assembled from digitized text, including publicly accessible web pages, licensed collections, and material produced specifically for model development. Raw corpora contain duplication, encoding errors, personal information, and machine-generated content. Filtering and deduplication alter both the computational efficiency of training and the distribution learned by the model.

Corpus composition has direct behavioral consequences. A model trained predominantly on one language receives fewer parameter updates representing other languages, while a corpus concentrated in a particular domain produces stronger statistical coverage of that domain. These effects are not equivalent to simple document retrieval because training compresses regularities across examples into distributed numerical parameters.

Tokenization introduces an additional representational layer. Frequently occurring character sequences are usually encoded with fewer tokens, while uncommon words may be divided into several fragments. This difference affects context consumption and computational cost, especially for languages or notation systems that are underrepresented when the tokenizer is constructed.

The model’s parameters do not generally preserve documents as individually addressable records. They encode a distributed approximation to patterns in the training distribution. Large models may nevertheless reproduce passages when those passages are repeated, distinctive, or strongly reinforced during training. Such memorization is particularly relevant to privacy analysis and evaluation contamination.

Adaptation and conversational systems

A pretrained model estimates continuation probabilities without inherently distinguishing between an instruction, a quotation, and an ordinary passage. Supervised fine-tuning modifies this behavior using examples that pair inputs with desired responses. The resulting model more consistently interprets certain textual forms as requests to be answered.

Preference-based training further adjusts outputs according to comparisons supplied by human evaluators or generated through automated feedback. In reinforcement learning from human feedback, a preference model assigns rewards to candidate responses, after which an optimization procedure changes the language model to increase expected reward. Direct preference optimization and related methods incorporate comparable pairwise information without training a separate reinforcement-learning policy in the same form.

These adaptations change the distribution of responses but do not provide a general mechanism for factual verification. A conversational model may produce an accurate answer, decline a request, or construct an unsupported statement using the same next-token prediction process. Retrieval systems and external tools can supply additional evidence, although their use introduces separate questions involving source selection, tool errors, and the model’s interpretation of retrieved material.

Capabilities and evaluation

LLM performance is measured through held-out likelihood, task-specific benchmarks, and evaluations conducted by human raters. Perplexity summarizes predictive performance on a token distribution, but it does not directly measure whether generated claims are correct or whether an answer satisfies an instruction. Task benchmarks therefore test behavior on defined collections of questions or documents.

Prompted models exhibit in-context learning, in which examples placed in the context influence subsequent output without updating the model’s parameters. The mechanism operates through the model’s activations during inference and ends when the context is removed. This distinguishes it from fine-tuning, which alters parameters and persists across later requests.

Apparent capability changes at larger scales reflect several interacting effects. Average predictive loss often changes smoothly, while a benchmark based on exact answers may register a sharp transition when many outputs cross its scoring threshold. Contamination from training data can also inflate measured performance when benchmark material or close variants occur in the corpus.

No single benchmark provides a comprehensive description of an LLM. Results vary with prompt wording, decoding settings, model version, and scoring method. Repeated optimization against a public benchmark can further reduce its value as an independent estimate of generalization.

Limitations

An LLM generates text that is probable under its learned and adapted distribution. This objective does not require correspondence between a sentence and an external state of affairs. The production of plausible but unsupported material is commonly described as hallucination, although the underlying event is ordinary model generation rather than a separate internal process.

Context length imposes a finite bound on directly available text. Material outside that window cannot influence an answer unless it has affected the model’s parameters or is reintroduced through retrieval. Even within the window, attention does not guarantee that every statement will be used consistently.

Training data also transmit social and historical regularities that may be undesirable in a deployment setting. Fine-tuning and filtering can modify measured behavior, but their effects depend on the evaluation distribution and may not transfer uniformly to unfamiliar inputs. Because generated language is sensitive to context, small changes in phrasing sometimes produce substantial differences in response.

Interpretability remains limited by the distributed nature of the learned representation. Individual components can correlate with recognizable linguistic or conceptual features, yet a complete causal account of a large model’s output generally requires analysis across many interacting activations. Mechanistic interpretability studies these interactions through interventions and circuit-level models.

Computational and institutional context

Training an LLM requires numerical optimization across large collections of tokens, usually on clusters of graphics processing units or other accelerators. The total resource requirement depends on model structure, sequence length, numerical precision, and the amount of data processed. Inference is less expensive than full training for a single request, but sustained deployment can consume substantial computation because each generated token requires another model evaluation.

The development of LLMs therefore intersects with distributed computing, semiconductor production, data governance, and intellectual-property law. Institutional access to compute and corpora shapes which organizations can train models at the largest scales. Smaller models, parameter-efficient adaptation, and compressed representations occupy a separate operating range in which lower resource requirements are exchanged for differences in capacity and specialization.

The legal classification of training data and generated outputs varies by jurisdiction. Privacy obligations concern the inclusion and possible reproduction of personal information, while copyright disputes concern the legal status of training and the similarity of outputs to protected works. These questions concern the acquisition and use of data rather than the mathematical definition of a language model.