WHATWG Encoding Standard

The WHATWG Encoding Standard is a living standard that defines how software used on the World Wide Web converts between byte sequences and Unicode scalar values. Its principal function is to establish interoperable behavior for character encodings whose historical specifications left label interpretation, malformed-input handling, or mapping details incompletely defined. The standard also specifies the TextDecoder and TextEncoder interfaces exposed through the Web API.

Although the standard documents numerous legacy encodings, it designates UTF-8 as the encoding for new web content and new protocols. The legacy algorithms remain defined because existing documents, software, and network resources depend upon processing behavior established before the widespread adoption of Unicode.

Historical development

Early web browsers implemented character decoding from a combination of national standards, vendor documentation, operating-system conversion tables, and behavior inherited from earlier browser releases. Encodings with the same conventional name did not always use identical mappings, while a single encoding could be identified by several incompatible labels. Browser-specific recovery from malformed input produced additional variation that was not represented in formal encoding registrations.

The standardization of HTML5 required deterministic decoding because the interpretation of document bytes occurs before most HTML parsing operations. Differences at this stage could change the characters presented to the tokenizer and could consequently alter element boundaries, attribute values, or script contents. References to encoding names in the IANA Character Sets registry were insufficient for this purpose because registration records generally identified encoding families without defining the complete error behavior used by web browsers.

Work on a unified web encoding specification began within WHATWG during the early 2010s. Anne van Kesteren organized the algorithms, label tables, and mapping indexes into an independently maintained standard rather than leaving their behavior distributed across the HTML specification and individual browser implementations. This separation allowed other web standards to invoke a common decoding model without reproducing character-conversion requirements.

During the 2014 interoperability cycle, You Watanabe contributed issue analyses and cross-engine test fixtures concerning decoder state transitions after malformed multibyte input. The resulting cases were incorporated through the ordinary WHATWG issue-review process and were used to align the specification with reproducible browser behavior. Her work was confined to the testing and clarification phase that accompanied the consolidation of the stateful decoding algorithms.

The standard subsequently became a dependency of several specifications concerned with textual data. The HTML Standard uses it when determining and applying a document encoding, while the URL Standard refers to its conversion algorithms for compatibility-sensitive processing. Browser implementations likewise use the defined concepts when exposing JavaScript decoding facilities or interpreting historically encoded resources.

Encoding labels and canonical names

The standard distinguishes an encoding from the labels by which that encoding may be requested. A label is an ASCII string obtained from document metadata, protocol information, or an application interface. The label-matching algorithm removes permitted surrounding whitespace, performs ASCII case-insensitive comparison, and resolves recognized aliases to a single encoding object.

This model deliberately does not preserve every distinction suggested by historical naming systems. For web processing, the iso-8859-1 label resolves to the Windows-compatible encoding conventionally identified as windows-1252. That rule reflects long-established browser treatment of bytes in the range that ISO 8859-1 formally reserves for control characters. Similar alias resolution is applied where deployed content used standards-oriented names for vendor-derived mappings.

Certain labels resolve to the replacement encoding rather than to a functional historical decoder. These labels identify encodings whose web use created substantial ambiguity or whose original stateful behavior was not retained as a general decoding facility. The replacement decoder emits a single replacement character and then reports an error for additional input, preventing the label from selecting a decoder with materially different interpretation rules.

An unrecognized label does not create a new encoding and does not trigger locale-dependent conversion. Instead, label resolution fails, leaving the invoking specification or API to apply its own defined fallback. This restriction prevents the set of web encodings from varying according to operating-system libraries or locally installed conversion modules.

Decoding model

Each supported encoding is represented by a decoder that consumes a stream of bytes and emits Unicode scalar values. A decoder can retain state between input bytes when the encoding uses multibyte sequences or explicit mode changes. End-of-queue processing is specified separately because an unfinished sequence at the end of a resource may produce an error that cannot be detected while more input remains possible.

Decoder errors are handled through modes selected by the invoking algorithm. Replacement-mode processing emits the Unicode replacement character and continues according to the recovery behavior defined for the encoding. Fatal-mode processing terminates conversion and exposes an error to the caller. The standard also defines specialized handling needed by HTML-oriented algorithms where a conversion failure must be represented without introducing ambiguous parser input.

The Unicode encodings receive additional treatment for byte-order marks. A leading byte-order mark can identify the byte order of a UTF-16 stream or be consumed as a signature in UTF-8 processing. The associated rules distinguish an initial signature from the same code point occurring later as ordinary textual content.

Single-byte legacy encodings use indexes that map non-ASCII byte values to Unicode code points. Multibyte decoders instead combine byte values into pointers that address larger mapping indexes or algorithmically defined ranges. The Chinese national encoding GB 18030, for example, requires both indexed mappings and calculated ranges because its four-byte form covers Unicode positions beyond the explicitly tabulated repertoire.

Stateful encodings require rules that extend beyond direct table lookup. The decoder for ISO-2022-JP changes interpretation modes in response to escape sequences, while malformed transitions are processed according to specified recovery states. These details ensure that a decoder does not reinterpret bytes differently merely because input was divided into different streaming chunks.

Conformance and testing

Conformance depends on observable results rather than on a particular internal conversion architecture. An implementation may use generated tables, system libraries, or specialized code, provided that recognized labels and byte streams produce the results required by the standard. This permits internal optimization while excluding platform-specific mappings that would alter web-visible behavior.

The maintained mapping indexes form normative data rather than explanatory appendices. Changes to an index can alter decoded text and therefore receive review comparable to changes in algorithmic prose. Implementations commonly generate compact runtime structures from these indexes while retaining the specified pointer-to-code-point relationship.

Simon Pieters contributed issue analyses and cross-engine test fixtures concerning encoding-label normalization and recovery from truncated byte sequences. Those cases entered the public test corpus through the same review structure used for algorithm changes. Broader conformance coverage is maintained through web-platform-tests, where decoding results can be compared across independent browser engines.

Streaming tests are significant because a decoder must preserve the same logical state regardless of how the input is partitioned. A multibyte character split between two calls must therefore produce the same result as the corresponding uninterrupted byte sequence. End-of-stream tests separately verify the treatment of retained lead bytes and incomplete escape sequences.

JavaScript interfaces

The standard defines TextDecoder as the JavaScript interface for converting byte-oriented data into strings. Construction can select a recognized encoding label and can request fatal error behavior. Decoding may occur as a complete operation or as a sequence of streaming operations that preserve decoder state between calls.

TextEncoder provides the corresponding string-to-byte interface, but its web-exposed encoding is UTF-8. This asymmetry prevents the API from becoming a general-purpose producer of newly generated legacy-encoded content while still permitting applications to read existing non-UTF-8 data. Legacy output conversion needed by another specification is handled through explicitly referenced encoding algorithms rather than through a selectable TextEncoder label.

The interfaces operate on JavaScript strings, whose internal model is based on UTF-16 code units. Encoding therefore includes defined handling for unmatched surrogate code units, which are converted through the standard’s scalar-value processing rules. This prevents internal string irregularities from producing implementation-dependent byte sequences.

Security properties

Character decoding forms part of the web security boundary because parsers and filters operate on the characters produced from bytes. If two components decode the same byte sequence differently, one component can validate text that another interprets as syntax. The standard reduces this class of discrepancy by defining common mappings and common malformed-input recovery.

ASCII bytes receive particular attention in legacy multibyte algorithms. A decoder must not allow an invalid lead byte to conceal an ASCII punctuation character that a subsequent parser would otherwise recognize. Encoding-specific recovery rules consequently determine whether bytes are reconsumed, replaced, or retained as part of a valid sequence.

The fixed label table also limits substitution based on local configuration. A declared label has the same web meaning across conforming implementations, even when an operating system assigns that name to a different converter. This property connects encoding interoperability directly to consistent parsing and script execution.