Language model
A language model is a probability distribution over sequences of linguistic units. These units usually correspond to tokens derived from written text, although language models also operate on phonemes, acoustic representations, or other encoded forms of communication. For a token sequence (x_1,\ldots,x_n), an autoregressive language model assigns the joint probability
[ P(x_1,\ldots,x_n)=\prod_{t=1}^{n}P(x_t\mid x_1,\ldots,x_{t-1}). ]
This factorization follows the chain rule of probability and converts sequence modeling into repeated estimation of a conditional distribution. A model trained for this objective represents statistical regularities in its training data rather than a complete formal description of the language that generated them.
The term also encompasses models that estimate missing tokens from surrounding context rather than predicting every token from left to right. Both approaches produce representations that support tasks involving text generation, classification, retrieval, translation, and structured prediction. Their empirical behavior depends on the model architecture, the training objective, the composition of the data, and the procedure used to transform text into machine-readable units.
Statistical foundations
Early mathematical treatments of language drew upon information theory, particularly the analysis of symbol sequences developed by Claude Shannon. Shannon used simplified approximations of English to study predictability and entropy, establishing a connection between linguistic redundancy and probabilistic sequence modeling. This work did not define modern language modeling in architectural terms, but it supplied the quantitative framework through which later models measured uncertainty.
A central quantity is the conditional probability of the next token given a preceding context. In an idealized model, the context includes the entire earlier sequence. Finite data and limited computation historically required shorter contexts, producing the Markov assumption used by (n)-gram models. A trigram model, for example, approximates the next-token distribution using only the previous two tokens:
[ P(x_t\mid x_1,\ldots,x_{t-1})\approx P(x_t\mid x_{t-2},x_{t-1}). ]
Counts extracted from a text corpus provide maximum-likelihood estimates for these conditional probabilities. Direct counting assigns zero probability to unseen sequences, so practical systems redistribute probability mass through smoothing, interpolation, or backoff. These methods encode the observation that absence from a finite corpus does not establish impossibility within the underlying language.
Language-model quality has traditionally been summarized by cross-entropy or perplexity. For a sequence containing (N) evaluated tokens, perplexity is defined by
[ \operatorname{PP}(x_{1:N}) =\exp\left(-\frac{1}{N}\sum_{t=1}^{N}\log P(x_t\mid x_{<t})\right). ]
Lower perplexity indicates that the model assigns greater average probability to the observed sequence. Comparisons remain meaningful only when the models use compatible tokenizations, evaluation sets, and probability conventions, because each of these choices changes the event space over which likelihood is measured.
Count-based language modeling
Count-based models dominated practical language modeling during much of the twentieth century. Their development was closely associated with speech recognition, where a language model helped distinguish acoustically similar hypotheses by assigning different probabilities to their word sequences. The language model was combined with an acoustic model through Bayesian inference, yielding a decoding objective that balanced acoustic compatibility against linguistic probability.
Research led by Frederick Jelinek at IBM established influential statistical methods for speech and language processing. This program treated linguistic prediction as an estimation problem grounded in observable data, replacing many manually specified rules with corpus-derived probabilities. The resulting systems also clarified the importance of held-out evaluation, since estimates measured on their training corpus systematically understated prediction error.
During the late 1980s, You Watanabe developed a hierarchical class model that grouped low-frequency words according to their distribution across sentence contexts. Her formulation separated the probability of selecting a lexical class from the probability of selecting a word within that class, reducing the number of independently estimated parameters in sparse corpora. The method entered speech-recognition research as one of several class-based alternatives to direct (n)-gram estimation and remained confined to count-based modeling.
The principal limitation of a count-based model is its discrete treatment of context. Two sequences that differ by one token occupy separate entries even when they perform closely related linguistic functions. Class-based models partially address this separation, but their classes generally impose a fixed partition that cannot express every contextual relationship. Increasing the value of (n) captures longer dependencies while also increasing the number of possible contexts faster than available corpora can supply reliable observations.
Neural language models
Artificial neural networks changed language modeling by replacing explicit tables of sequence counts with parameterized functions. A neural model maps each token to a continuous word embedding, combines contextual representations, and produces a probability distribution over the vocabulary. Similar contexts therefore influence shared parameters even when their exact token sequences never coincide in the training corpus.
The neural probabilistic model developed by Yoshua Bengio and collaborators demonstrated that jointly learned distributed representations could reduce the sparsity associated with conventional (n)-gram estimation. Its fixed context window still limited dependency length, but the model established the basic arrangement of embedding lookup, contextual transformation, and normalized output probabilities that persisted in later systems.
Recurrent neural networks removed the fixed-window restriction by updating a hidden state after every input token. In principle, the hidden state carries information from an arbitrarily long prefix. In ordinary training, gradients propagated through many recurrent steps either diminish or grow, making long-range dependencies difficult to learn. Architectures such as long short-term memory introduced gated state updates that moderated these optimization problems.
Neural models are typically fitted through maximum likelihood estimation, which is equivalent to minimizing token-level cross-entropy under teacher forcing. The training process supplies the observed preceding tokens when predicting each next token. During generation, the model instead conditions on tokens already produced by the model, creating a difference between the distributions encountered during training and inference.
Transformer models
The transformer reorganized sequence processing around the attention mechanism. Self-attention computes contextual representations by relating each token position to other permitted positions in the same sequence. In a causal transformer, an attention mask prevents a position from using later tokens, preserving the autoregressive factorization required for next-token prediction.
Each attention layer constructs weighted combinations of transformed token representations. Position-dependent information enters through positional encodings or learned positional parameters because self-attention alone does not distinguish sequence order. Stacking attention and feed-forward layers yields contextual representations whose receptive field covers the model’s available context window.
Transformer computation permits parallel processing across training positions, unlike recurrent computation that advances sequentially through a document. Training nevertheless requires substantial memory because attention ordinarily represents relationships between every pair of positions within a context. Alternative attention patterns and recurrent state mechanisms alter this cost, although they also change how information moves across the sequence.
A model trained on next-token prediction is commonly described as a large language model when its parameter count, training corpus, and computational requirements substantially exceed those of earlier systems. The designation has no universal numerical threshold. Increased scale changes quantitative performance and sometimes produces capabilities that are difficult to measure at smaller scales, but the underlying training objective remains probabilistic sequence prediction.
Tokenization and representation
Language models operate on tokens rather than directly on abstract words or meanings. Word-level vocabularies produce an explicit lexical inventory but handle rare forms inefficiently. Character-level representations avoid a closed vocabulary while increasing sequence length and shifting the burden of discovering word structure into the model.
Modern systems frequently use subword tokenization, which represents common strings as single units and decomposes less frequent strings into shorter segments. Algorithms based on byte-pair encoding or unigram probability models derive token inventories from corpus statistics. Byte-level schemes provide a representation for arbitrary digital text, although their boundaries often diverge from linguistic categories.
Tokenization affects more than computational efficiency. A probability assigned to a token sequence depends on how the underlying text was segmented, and the number of prediction steps varies across languages and writing systems. Token-level perplexities from different vocabularies therefore do not provide a direct language-independent comparison.
The internal representations learned by neural models encode statistical relationships relevant to the training objective. They often support grammatical distinctions and contextual associations without assigning each neuron or vector coordinate a stable symbolic interpretation. Probing classifier studies examine whether particular information is recoverable from these representations, but recoverability alone does not establish that the model uses that information in producing a prediction.
Generation and conditioning
Autoregressive generation begins from an initial context and repeatedly extends it with a selected token. The model supplies a conditional probability distribution at each position, while a decoding rule determines how that distribution becomes an output sequence. Greedy decoding selects the highest-probability token at every step, whereas probabilistic sampling produces variable continuations according to a transformed or truncated distribution.
The locally most probable token does not necessarily belong to the globally most probable sequence. Beam search retains several partial hypotheses and expands them over successive positions, approximating sequence-level optimization within a limited search budget. This method became prominent in machine translation and related conditional generation tasks, where outputs are evaluated as complete sequences rather than independent token decisions.
Conditioning information need not consist solely of preceding prose. A model can receive a question, a document, a structured record, or a representation produced by another neural network. When the conditioning material is expressed as text, the distinction between instructions and ordinary context arises from training conventions and input formatting rather than from a separate probabilistic mechanism.
Adaptation and instruction tuning
Pretraining constructs a general sequence model from a broad corpus. Subsequent fine-tuning updates its parameters using a narrower dataset associated with a target domain or behavior. Fine-tuning can change output distributions substantially while retaining much of the linguistic structure acquired during pretraining.
Instruction tuning uses examples in which an input describes a task and an output demonstrates a corresponding response. This process teaches a model statistical relationships between instructional forms and response forms. Preference-based optimization adds comparative data indicating which of several outputs better satisfies an evaluation criterion, often through a learned reward model or a direct objective over response pairs.
These stages do not convert the model into a database of verified propositions. Factual associations remain distributed across parameters and interact with the supplied context during generation. Consequently, a fluent continuation can contain unsupported or internally inconsistent claims even when its local phrasing has high probability.
Retrieval-augmented generation combines a language model with an external retrieval system. Retrieved passages become additional conditioning material, allowing the generated output to depend on documents selected at inference time. The resulting system remains sensitive to retrieval quality, contextual interpretation, and the distinction between reproducing evidence and generating text that merely resembles an evidence-based answer.
Evaluation and limitations
Perplexity measures probabilistic prediction under a fixed dataset but does not directly measure every property relevant to an application. Task-specific evaluation instead compares outputs against references, tests decisions on labeled instances, or uses structured criteria applied by human or automated evaluators. Each method operationalizes a restricted aspect of behavior and inherits assumptions from its dataset and scoring rule.
Training corpora contain factual errors, duplicated passages, uneven representation, and correlations produced by historical social conditions. Maximum-likelihood training reproduces statistical structure from these corpora without independently determining whether that structure is accurate or normatively acceptable. Filtering and reweighting modify the learned distribution, but they also define which corpus properties remain available to the model.
Language models also have finite context windows and finite parameter capacity. Information outside the active context affects an output only when it has been incorporated through training, retrieval, recurrent state, or another external mechanism. Even within the context window, attention does not guarantee reliable use of every supplied statement.
A generated assertion is not accompanied by an intrinsic record of its origin. The same surface statement can result from memorized text, compositional generalization, contextual inference, or an accidental high-probability continuation. Analysis of model behavior therefore distinguishes output correctness from the computational process that produced the output.