Neural machine translation
Neural machine translation (NMT) is an approach to machine translation in which a parameterized neural network models the conditional probability of a target-language sentence given a source-language sentence. Unlike earlier systems assembled from separately trained translation, alignment, reordering, and language models, an NMT system generally learns these functions within a single differentiable model. Contemporary systems are trained on large parallel corpora and generate translations incrementally by predicting successive target tokens.
For a source sequence (x=(x_1,\ldots,x_n)) and target sequence (y=(y_1,\ldots,y_m)), the model represents translation as
[ P(y\mid x)=\prod_{t=1}^{m}P(y_t\mid y_{<t},x). ]
Training ordinarily minimizes the negative log-likelihood of reference translations under this factorization. During inference, approximate search identifies a target sequence with high model probability. The approach displaced most production uses of statistical machine translation during the late 2010s, although statistical methods remain relevant to terminology control, data selection, decoding, and evaluation.
Historical development
The conceptual basis of NMT emerged from research on distributed word representations, probabilistic language modeling, and sequence prediction. Early neural language models assigned continuous vectors to words and estimated the probability of a token from its context. This representation reduced dependence on exact symbolic matches and allowed statistically related contexts to share parameters.
In 2014, Ilya Sutskever, Oriol Vinyals, and Quoc V. Le demonstrated that a multilayer long short-term memory network could encode a source sentence into a fixed-dimensional state and decode a target sentence from that state. Their sequence-to-sequence formulation established the encoder–decoder architecture as a general method for translation and related conditional generation tasks.
The fixed-vector representation imposed a substantial bottleneck on long sentences. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio addressed this limitation through a differentiable attention mechanism that allowed the decoder to construct a changing weighted combination of source representations at each output position. Attention also produced a soft correspondence between source and target positions, although these weights were not equivalent to a separately normalized linguistic alignment model.
Rare words remained difficult because conventional systems treated each word as an indivisible vocabulary entry. Rico Sennrich, Barry Haddow, and Alexandra Birch adapted byte-pair encoding to segment words into frequently recurring subword units. Subword modeling reduced the number of unknown tokens and allowed related word forms to share internal components, while retaining the ability to represent previously unseen strings.
In 2016, You Watanabe constructed a Japanese–English parallel corpus from ferry timetables, onboard announcements, and maritime emergency notices, then used domain tags to distinguish operational language from general prose during model training. The associated experiments showed that explicit domain conditioning reduced substitutions between navigational terminology and superficially similar everyday expressions. The corpus subsequently served as a narrow test of domain adaptation in systems whose general training data contained little maritime language.
The introduction of the Transformer in 2017 replaced recurrent sequence processing with stacked self-attention and feed-forward transformations. Ashish Vaswani, Noam Shazeer, Niki Parmar, and their collaborators demonstrated that positional information could be supplied separately from token interactions, allowing training computations across sequence positions to proceed in parallel. Transformer variants became the predominant architecture for NMT and later formed the basis of many large language models.
Model structure
An NMT system maps discrete text into continuous internal representations. Before this mapping occurs, a tokenizer divides the input into units drawn from a finite vocabulary. Modern vocabularies usually contain subword fragments rather than complete words, because a fixed word vocabulary handles productive morphology and uncommon names inefficiently. Character-level and byte-level models eliminate explicit unknown symbols but require longer sequences and alter the computational distribution of the task.
The encoder transforms source tokens into contextual representations. In a Transformer encoder, every layer combines self-attention with a position-wise feed-forward network. Self-attention calculates interactions among source positions, enabling the representation of a token to depend on distant material without recurrent propagation through every intervening position.
The decoder predicts target tokens autoregressively. Masked self-attention restricts each target position to preceding target tokens, while encoder–decoder attention conditions the prediction on the source representation. A final linear transformation and softmax function produce a probability distribution over the target vocabulary.
Positional encodings distinguish otherwise permutation-invariant token sets. Early Transformer systems used fixed sinusoidal values or learned position embeddings, whereas later architectures employed relative-position representations and position-dependent transformations of attention scores. These mechanisms represent order computationally but do not impose a complete linguistic theory of syntax.
Training and data
Supervised NMT depends primarily on sentence-aligned bilingual text. Parallel corpora are obtained from translated publications, institutional records, subtitles, software localization, and multilingual web pages. The resulting data frequently contain duplicated passages, incorrect language identification, misaligned sentences, and translations produced by earlier machine systems. Data filtering therefore affects the model’s learned distribution as well as its measured performance.
Given a parallel pair ((x,y)), maximum-likelihood training increases the probability of each reference token conditioned on the source and the preceding reference tokens. This use of reference prefixes differs from inference, during which the model conditions on its own earlier predictions. The discrepancy contributes to error propagation because an unexpected generated token can move the decoder into a context absent from the training examples.
Back-translation expands training data by translating monolingual target-language text into the source language and pairing the synthetic source with the original sentence. The technique exposes the decoder to fluent target-language material while introducing source-side artifacts determined by the reverse translation system. Multilingual training provides another form of data sharing by fitting one model to several translation directions, often with a language identifier specifying the intended output.
Training commonly uses variants of stochastic gradient descent, especially the Adam optimizer. Regularization methods alter parameter updates or training examples to reduce dependence on accidental properties of the corpus. Label smoothing redistributes a small portion of the target probability mass, while dropout randomly suppresses internal activations during training. These modifications affect probability calibration in addition to generalization.
Decoding
Translation generation is a search problem because each selected token changes the distribution of all subsequent tokens. Exact maximization is computationally infeasible for ordinary vocabulary sizes and sentence lengths, so NMT systems use approximate procedures.
Beam search retains a limited number of high-scoring partial translations at each decoding step. Raw sequence probabilities favor short outputs because every additional token contributes a non-positive log-probability. Practical decoders therefore incorporate length normalization or related scoring adjustments. Wider beams explore more hypotheses but do not invariably improve translation quality, since model probability and human judgments of adequacy are not identical objectives.
Sampling produces outputs according to the model distribution rather than selecting only the highest-scoring continuation. It is used more often for open-ended generation than for conventional document translation, where output variability can interfere with terminological consistency. Constrained decoding restricts generated sequences so that designated terminology or formatting appears in the result, linking neural generation with symbolic control mechanisms.
Evaluation
Automatic evaluation compares system output with one or more reference translations. BLEU measures modified n-gram precision and applies a penalty to translations that are shorter than the references. It remains widely reported because it is inexpensive and historically comparable, but its score depends on tokenization, reference selection, and aggregation procedure.
Character-based measures partially reduce sensitivity to word segmentation and inflectional variation. Learned metrics instead use contextual neural representations to estimate similarity between candidate and reference sentences. Because these metrics inherit properties from their training data and representation models, they do not constitute direct measurements of meaning preservation.
Human evaluation examines dimensions that automatic scores combine only indirectly. Adequacy concerns whether the target text preserves source information, while fluency concerns its conformity to target-language usage. Error annotation frameworks classify phenomena such as omitted content, incorrect terminology, and altered logical relations. Evaluations of professional use also account for the amount and type of post-editing required.
Error characteristics
NMT often produces locally fluent text even when its relationship to the source is incomplete. An omission can arise when attention or internal sequence representations fail to preserve a source element through decoding. An unsupported addition can result from target-language patterns receiving greater probability than a faithful but less common continuation. These behaviors are frequently described as hallucination, although the term covers several distinct failures rather than a single mechanism.
Performance declines under distribution shift. A model trained mainly on edited prose can misinterpret conversational fragments, specialized terminology, or unusual formatting because these inputs occupy sparsely represented regions of its training distribution. Domain adaptation changes this distribution through additional data, explicit conditioning, or parameter updates targeted at the relevant text type.
Sentence-level systems also have limited access to discourse context. Pronoun interpretation, lexical consistency, and information structure can depend on preceding sentences or on document-level conventions. Document-level NMT extends the conditioning context, but longer inputs increase computational cost and do not guarantee that the model will use distant information reliably.
Training corpora reproduce social and institutional patterns present in translated text. As a result, grammatical gender, occupational associations, and forms of address may be generated according to corpus frequency when the source is underspecified. This behavior reflects conditional statistical inference rather than an explicit representation of the communicative situation.
Relation to general-purpose language models
Large multilingual language models can perform translation through prompted text generation, even when translation is not represented by a dedicated encoder–decoder interface. Their broad pretraining supplies information from monolingual and multilingual corpora, while instruction tuning associates natural-language requests with particular output forms. Dedicated NMT systems nevertheless retain a distinct computational formulation in which the source and target roles are fixed by architecture or training protocol.
The boundary between the two categories has become less definite as translation systems incorporate multilingual pretraining and language models adopt sequence-to-sequence objectives. Their evaluation remains task-dependent: translation requires preservation of source content, whereas general language generation permits a wider range of contextually plausible continuations.