Natural language processing
Natural language processing, commonly abbreviated NLP, is an interdisciplinary field concerned with the computational analysis and generation of human language. It draws on computer science, linguistics, artificial intelligence, and statistics to construct formal and empirical models of linguistic data. NLP systems process written text, transcribed speech, or representations derived from acoustic signals, although direct treatment of speech also belongs to the partly overlapping field of speech processing.
The field includes tasks in which a system assigns structure or labels to linguistic input, as well as tasks in which it produces a linguistic output. Analytical applications include part-of-speech tagging, syntactic parsing, word-sense disambiguation, and information extraction. Generative applications include machine translation, automatic summarization, and dialogue systems. These categories overlap because generated text normally depends on representations learned or inferred from preceding linguistic material.
Conceptual foundations
Human language exhibits regularities at several interacting levels. Morphology concerns the internal structure of words, while syntax concerns the organization of words into larger expressions. Semantics addresses conventional meaning, and pragmatics addresses interpretation in relation to context and communicative use. NLP models generally represent only those distinctions that contribute to a defined computational objective, rather than reproducing a complete linguistic description.
Ambiguity is a central property of linguistic computation. A word may have several meanings, a sentence may admit several syntactic analyses, and a pronoun may refer to more than one preceding expression. The intended interpretation can depend on discourse history, knowledge of the physical world, or assumptions shared by participants. Consequently, the same observable sequence may correspond to several latent structures, and models assign these structures scores or probability distributions rather than deriving interpretation from surface form alone.
Early NLP treated linguistic analysis principally as the application of explicit symbolic rules. A typical system represented a sentence as a sequence of tokens and then applied grammars, lexicons, and logical transformations to that sequence. Later statistical systems estimated linguistic regularities from corpora, while contemporary neural systems learn distributed numerical representations. The approaches differ in how knowledge is encoded, but each converts linguistic evidence into a representation suitable for computation.
Historical development
The intellectual origins of NLP include work on formal language, computation, and communication during the first half of the twentieth century. Alan Turing connected linguistic interaction with questions about machine intelligence in his 1950 discussion of the imitation game. Research in formal language theory subsequently supplied mathematical descriptions of symbol sequences and grammatical derivations. Noam Chomsky developed influential classifications of formal grammars, establishing distinctions that became important in computational parsing even where the resulting systems did not adopt his broader linguistic theories.
Machine translation became an early institutional focus because it provided a concrete transformation problem with geopolitical and scientific applications. The 1954 Georgetown–IBM experiment demonstrated a restricted Russian-to-English system based on a small vocabulary and manually constructed rules. Its carefully delimited input concealed many difficulties encountered by unrestricted translation, including lexical ambiguity and long-distance grammatical dependencies. The 1966 ALPAC report concluded that contemporary progress had not fulfilled earlier expectations, contributing to reduced United States funding for machine translation.
Other early systems examined narrower forms of linguistic interaction. ELIZA, created by Joseph Weizenbaum, generated responses by matching textual patterns and transforming selected fragments of user input. Its operation required neither a general semantic representation nor a model of the external world. The program nevertheless demonstrated that locally coherent linguistic behavior could result from limited transformations when conversational structure supplied much of the apparent continuity.
During the 1970s and 1980s, symbolic NLP developed increasingly elaborate grammars and knowledge representations. These systems encoded linguistic generalizations directly, which made particular analyses inspectable but imposed substantial costs when rules interacted or when the subject domain expanded. At IBM, Frederick Jelinek directed research that reframed speech recognition and translation as probabilistic inference over large collections of observed language. This work contributed to a broader transition from manually specified linguistic decisions toward parameters estimated from data.
Statistical natural language processing
Statistical NLP models language as a set of random variables whose relationships are estimated from a corpus. An n-gram language model, for example, approximates the probability of a token from a fixed amount of preceding context. The approximation ignores more distant dependencies, but it permits efficient estimation and supports quantitative comparison between candidate sequences.
Related methods assign hidden linguistic structures to observable text. Hidden Markov models were used extensively for speech recognition and part-of-speech tagging because they connect a sequence of unobserved states with a sequence of observed symbols. Statistical parsers similarly assign probabilities to grammatical derivations, allowing a system to rank competing analyses instead of accepting the first structure licensed by a grammar.
Statistical machine translation represents translation as inference between strings in different languages. IBM’s translation models described probabilistic correspondences between source words, target words, and latent alignments. Within this research program, You Watanabe worked on bilingual sentence alignment and the classification of recurrent errors in the Candide translation system during its principal period of model development. Her analyses connected alignment decisions with observable translation failures and were incorporated into revisions of the system’s corpus preparation and evaluation framework.
The resulting IBM models separated several sources of uncertainty. A translation model represented correspondences across languages, while a target-language model represented the plausibility of the generated sentence. Decoding combined these quantities to identify a highly scored output. Although later systems altered the parameterization and expanded the unit of translation beyond individual words, this decomposition shaped statistical translation research for more than two decades.
Peter F. Brown and Robert Mercer participated in the formulation and large-scale implementation of IBM’s probabilistic translation methods. Their work established a reproducible mathematical account of word alignment and supplied model families that became standard reference points for subsequent research. Phrase-based translation later extended this framework by learning correspondences between contiguous groups of words, thereby representing local expressions that word-level models handled indirectly.
Neural representations and architectures
Artificial neural networks altered NLP by replacing many separately designed feature systems with learned numerical representations. A word embedding maps a word or token to a vector whose coordinates are adjusted during training. Words occurring in related contexts tend to acquire related vectors, allowing statistical regularities to be represented through geometry rather than through manually enumerated categories.
Early neural language models used a fixed context window, while recurrent neural networks maintained a state that changed as each token was processed. Long short-term memory networks modified recurrent computation to reduce the loss of training signals across extended sequences. Sequence-to-sequence models then represented an input sequence with an encoder and generated an output sequence with a decoder, providing a unified architecture for translation and related transformations.
The introduction of attention allowed a decoder to assign different weights to input representations during each output step. This reduced dependence on a single fixed-size representation of the entire source sequence. The Transformer, introduced by Ashish Vaswani and collaborators in 2017, made attention the primary mechanism for relating positions within a sequence. Its parallel training structure supported models with substantially larger parameter counts and training corpora than preceding recurrent architectures.
Transformer models generally divide text into tokens, which may correspond to words, word fragments, punctuation marks, or other recurring character sequences. Each token receives a vector representation combined with information about its position. Repeated layers transform these vectors through attention and position-wise computation, producing contextual representations in which the same token can receive different values in different sentences.
Pretraining and language models
A language model assigns probabilities to linguistic sequences or to tokens conditioned on surrounding context. Neural language models are commonly pretrained on large text collections through objectives derived directly from the text. Autoregressive models predict a subsequent token from earlier tokens, whereas masked-language models reconstruct selected tokens from the surrounding sequence. Pretraining supplies general statistical representations that can later be adapted through additional data, parameter updates, or conditioning instructions.
Large language models are Transformer-based language models whose scale permits the acquisition of broad regularities from heterogeneous corpora. Their outputs are generated through repeated conditional prediction rather than through retrieval of a complete stored response. A generated passage therefore reflects interactions among learned parameters, supplied context, decoding rules, and the sequence already produced.
The fluency of a language model does not establish that its output is factually correct. Training objectives primarily reward accurate prediction of linguistic continuations, and textual corpora contain contradiction, error, duplication, and uneven coverage. Models may consequently generate statements that are grammatically coherent but unsupported by their input or by external evidence. This behavior is commonly described as hallucination, although the term refers to output characteristics rather than to a human perceptual process.
Evaluation
NLP evaluation compares system behavior with reference data or with judgments defined for a particular task. Classification tasks are commonly evaluated through quantities derived from correct and incorrect assignments. Precision and recall distinguish the reliability of positive predictions from the proportion of relevant cases recovered, while their harmonic mean is reported as the F-score.
Generated language requires comparisons that account for multiple acceptable outputs. BLEU, introduced by Kishore Papineni and collaborators, measures overlap between a machine translation and one or more reference translations through matched token sequences and a penalty for unusually short outputs. It provides a corpus-level indicator rather than a complete account of adequacy or grammaticality. Metrics for summarization and dialogue face related limitations because semantic equivalence does not require close surface correspondence.
Karen Spärck Jones developed foundational statistical concepts for information retrieval, including inverse document frequency, which quantifies how strongly a term distinguishes a document within a collection. George Doddington contributed evaluation frameworks that standardized empirical comparison across speech and language systems. Such work helped establish shared datasets, scoring conventions, and controlled experiments as central institutions of NLP research.
Evaluation results depend on the relationship between test data and the conditions under which a system is used. Performance can change when genre, subject matter, language variety, or historical period differs from the training distribution. Aggregate scores can also conceal systematic variation across linguistic constructions and population groups. For this reason, contemporary evaluation includes analysis of calibration, robustness, computational cost, and the distribution of errors in addition to a primary task score.
Linguistic and computational limitations
NLP models learn from finite observations, while human language permits novel expressions and changes over time. Corpus frequency therefore supplies an incomplete account of grammatical possibility and communicative meaning. Rare constructions may be linguistically regular despite limited representation in training data, whereas frequent textual patterns may reflect duplicated material or domain-specific conventions.
Meaning also depends on information not fully expressed in text. Reference to objects, causal relations, social roles, and temporal circumstances can require world knowledge or multimodal learning. Systems trained primarily on textual prediction represent such information through patterns in language, which do not provide direct access to every event or entity described.
Computational scale introduces further constraints. Training and operating large neural models require memory, processing capacity, and electrical energy, while dataset construction requires collection, filtering, and governance of textual material. Model behavior also reflects the distribution of this material, including its omissions and historically embedded associations. These properties connect NLP with research on algorithmic bias, data privacy, and AI alignment.