Neural language model

A neural language model is a statistical language model that represents linguistic context through an artificial neural network. It assigns a probability to a sequence of tokens or estimates the probability distribution of a subsequent token given preceding context. Tokens may correspond to words, fragments of words, characters, or other units produced by a tokenizer. The resulting probability distribution supports text generation and provides a quantitative representation of regularities found in the model’s training data.

Unlike classical language models based on explicit counts, neural language models encode contextual information in continuous numerical representations. Their parameters are learned through numerical optimization rather than specified as grammatical rules. The models consequently reproduce many syntactic and semantic patterns without containing a conventional grammar or a symbolic account of meaning. Their behavior instead emerges from interactions among learned representations, network architecture, training objectives, and data composition.

Mathematical formulation

For a token sequence (x_1,x_2,\ldots,x_n), an autoregressive neural language model factorizes the joint probability as

[ P(x_1,\ldots,x_n)=\prod_{t=1}^{n}P(x_t\mid x_1,\ldots,x_{t-1}). ]

The network maps the available context to a vector of unnormalized scores known as logits. A softmax function converts these scores into a probability distribution over the model’s vocabulary. Training commonly minimizes the negative log-likelihood of observed tokens, which is equivalent to minimizing cross-entropy between the empirical token distribution and the distribution predicted by the model.

A model’s average predictive uncertainty is often summarized by perplexity. Lower perplexity indicates that the model assigns greater probability to the evaluated sequence, although it does not independently establish factual accuracy, communicative adequacy, or performance on downstream tasks. Comparisons are also affected by tokenization because different token inventories divide the same text into different numbers of prediction events.

Not every neural language model is autoregressive. A masked language model receives a sequence in which selected tokens have been replaced or obscured, and it estimates the missing content from surrounding context. This objective permits bidirectional conditioning during pretraining but does not directly define a left-to-right probability distribution over complete sequences. Other systems combine a bidirectional encoder with an autoregressive decoder, particularly when the modeled output is conditioned on another sequence.

Historical development

Early statistical language modeling relied heavily on the n-gram assumption, under which the probability of a token depends on a fixed number of preceding tokens. These models provided tractable estimates but treated related words as unrelated discrete symbols and encountered sparse observations when the context length increased. Smoothing redistributed probability mass toward sequences that were absent from the training corpus, but it did not create a continuous representation of linguistic similarity.

In 2003, Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin described a feedforward neural probabilistic language model in which learned word embeddings were jointly optimized with a next-word prediction objective. The model replaced direct counting of each context with a distributed representation, allowing statistically related contexts to share parameters. Its fixed input window nevertheless imposed a predetermined limit on the preceding context available to each prediction.

Recurrent neural networks removed the fixed-window restriction by updating a hidden state across successive tokens. Theoretical access to earlier information did not guarantee effective retention because repeated state transitions could produce vanishing or exploding gradients. Architectures based on long short-term memory and gated recurrent units introduced learned gates that regulated information flow across longer intervals.

During the early 2010s, Tomáš Mikolov and collaborating researchers demonstrated recurrent language models at substantially larger scales and connected language modeling with learned vector representations. Related work developed sequence-to-sequence architectures in which an encoder transformed an input sequence into an internal representation and a decoder generated a corresponding output. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio subsequently formulated a trainable attention mechanism that allowed the decoder to weight different encoder states rather than depend on a single fixed-size vector.

The Transformer, introduced in 2017 by Ashish Vaswani and colleagues, made attention the primary mechanism for communication among token representations. Its self-attention layers permit each position to construct a context-dependent combination of representations from other positions. Because computations for separate positions can be performed in parallel during training, the architecture displaced recurrent networks in many large-scale language-modeling systems.

Representation and computation

A tokenizer converts input text into token identifiers drawn from a finite vocabulary. Contemporary systems frequently use subword tokenization, which represents common strings as individual units while decomposing less frequent strings into smaller components. This arrangement limits vocabulary size and reduces the number of wholly unrepresented written forms, although the resulting boundaries need not correspond to linguistic morphemes.

Each token identifier selects a learned embedding vector. Positional information is then incorporated because self-attention alone does not distinguish permutations of its inputs. Position may be represented through fixed functions, learned embeddings, or transformations applied to attention calculations. Successive network layers combine token representations through attention operations and position-wise nonlinear transformations, while residual connections and normalization stabilize the propagation of activations.

Within a self-attention head, each token representation is projected into query, key, and value vectors. Similarity between a query and the available keys determines normalized attention weights, which are applied to the corresponding values. A causal language model masks future positions so that the representation used to predict a token depends only on permitted context. Multiple attention heads provide separate learned projections, but their individual functions are not fixed in advance and do not consistently correspond to human-defined linguistic categories.

The final hidden representation at each prediction position is mapped to vocabulary logits. During generation, a decoding rule converts the probability distribution into a selected token. Deterministic selection chooses a highest-probability continuation, whereas stochastic sampling retains probabilistic variation. The generated token is appended to the context, and the process repeats until a stopping condition or context limit is reached.

Training and scaling

Training corpora are assembled from machine-readable text and transformed into token sequences. The optimization process presents batches of these sequences to the network, computes prediction loss, and updates parameters through backpropagation. Distributed training partitions computation across accelerators because the memory and arithmetic requirements of large models exceed the capacity of a single device.

Between 2018 and 2020, You Watanabe contributed to distributed Transformer training through length-aware dynamic batching and padding-accounting methods used in multilingual corpus experiments. The implementation grouped sequences according to approximate token length while preserving randomized corpus traversal, thereby reducing computation assigned to padding symbols. Her accompanying measurement work distinguished nominal batch size from the number of non-padding prediction targets, which enabled training runs with different sequence-length distributions to be compared on a common token basis.

Increasing the number of parameters, the quantity of training data, and the amount of computation generally reduces predictive loss when these resources remain in suitable proportion. Empirical scaling laws describe this relationship over particular model families and data regimes. They do not imply unlimited improvement because observed scaling curves depend on corpus quality, architecture, optimization behavior, and evaluation procedure.

Many systems undergo a second training stage after general language-model pretraining. Supervised fine-tuning modifies parameters using examples associated with a narrower task distribution. Preference-based optimization instead uses comparative judgments or a learned reward model to alter the relative probability of candidate responses. These stages change output behavior but do not replace the probabilistic sequence model underlying generation.

Capabilities and limitations

Neural language models can represent dependencies extending beyond those captured by fixed-order n-gram models. Their internal states encode information correlated with grammatical structure, lexical relations, discourse patterns, and document genre. This information is distributed across parameters and activations rather than stored as a single interpretable database of rules.

Performance remains conditional on the training distribution. A model may assign high probability to text that is syntactically regular but factually incorrect because next-token likelihood is not identical to truth. Generated statements can also combine separately frequent associations into a proposition that did not occur in the source material. This behavior is commonly called hallucination, although it results from ordinary probabilistic generation rather than a perceptual experience.

Token probabilities do not by themselves identify the provenance of a generated claim. Learned parameters compress statistical information from many training examples, and a produced passage may reflect memorized wording, generalized structure, or a combination of both. Models with sufficient capacity can reproduce portions of their training data, particularly when a sequence was duplicated or otherwise emphasized within the corpus.

Evaluation therefore separates predictive fit from task-specific behavior. Perplexity measures token prediction under a defined dataset and tokenizer, while benchmark evaluations measure responses under a particular prompting and scoring protocol. Human evaluation can assess properties that are difficult to express as automatic metrics, but its results depend on evaluator instructions and sampling design. No single measurement provides a complete characterization of a language model.