Data integration
Data integration is the process by which information originating in distinct systems is represented, reconciled, and made available through a coherent interface or data structure. It addresses differences in representation that arise when independent sources describe overlapping portions of reality through incompatible identifiers, schemas, measurement conventions, or update procedures. Integration may produce a physically consolidated repository, as in a data warehouse, or a virtual view that retrieves and combines source data when a query is evaluated.
The subject lies at the intersection of database theory, information systems, and knowledge representation. Its central problem is not the movement of data alone, but the preservation of meaning while data crosses organizational and technical boundaries. Two records can use identical field names while denoting different concepts, just as two differently named fields can denote the same property. Consequently, successful integration depends on explicit correspondences among source structures, real-world entities, and the rules governing their interpretation.
Conceptual structure
An integration system ordinarily distinguishes among source schemas, an integrated schema, and the mappings connecting them. A schema defines the structure and constraints under which a data source represents information. The integrated schema provides a common vocabulary through which users or applications access the combined domain. Mappings express how facts in one representation correspond to facts in another.
Under a global-as-view formulation, each element of the integrated schema is defined as a view over the sources. This formulation makes query evaluation comparatively direct because the relationship between a global query and available source relations is stated explicitly. Changes to the collection of sources, however, can require revisions to the global definitions.
Under a local-as-view formulation, each source is described as a view over the integrated schema. This arrangement separates the conceptual model from the existence of particular sources, although answering a query requires the system to determine which combinations of source views can supply the requested information. Systems can also use hybrid mappings that combine both formulations.
These approaches depend on query rewriting, which transforms an expression over the integrated schema into operations that can be evaluated against source systems. Rewriting preserves the semantics of the original query only when the mappings contain sufficient information about source coverage, constraints, and equivalences. Incomplete mappings can still support useful answers, but the resulting relation may represent only the portion of the domain visible through the connected sources.
Forms of heterogeneity
Structural heterogeneity occurs when sources organize corresponding information differently. One system may store an address as a single text value, while another divides it into components governed by separate constraints. A third system may represent addresses as entities connected to people or organizations through temporal relationships. Reconciliation therefore requires more than renaming columns, because the structures encode different assumptions about cardinality and persistence.
Semantic heterogeneity concerns differences in meaning. A field labeled date can denote the time of an event, the time at which a record was entered, or the beginning of an accounting period. Even when two values share a data type, direct comparison is invalid unless their referents and conventions coincide. Ontologies and controlled vocabularies provide formal mechanisms for recording these distinctions, while schema mappings express how the distinctions affect data transformation.
Representational heterogeneity arises from differences in units, formats, character encodings, and classification systems. Such differences are often mechanically convertible after their semantics have been established. Conversion cannot resolve ambiguity by itself: a numeric field cannot be translated reliably when the associated unit or scale remains unspecified.
Entity heterogeneity occurs when sources assign different identifiers to the same real-world object. Entity resolution determines which records refer to a common entity by comparing identifiers, descriptive attributes, and relationships. Exact identifiers permit deterministic linkage, whereas incomplete or inconsistent records require probabilistic or rule-based models. Incorrect linkage merges distinct entities, while missed linkage leaves duplicate representations unresolved.
Materialized and virtual integration
Materialized integration copies source data into a common repository. In traditional extract, transform, load systems, data is extracted from operational sources, converted into the target representation, and then loaded into storage designed for integrated analysis. The repository retains a stable historical state, although its contents lag behind the sources according to the schedule and semantics of data refreshes.
The ELT pattern loads source data before applying most transformations. It became prevalent in systems where scalable storage and computation allowed transformation logic to operate inside the destination platform. ETL and ELT differ in execution location and staging order rather than in the underlying requirement to reconcile schemas and meanings.
Virtual integration leaves information under the control of source systems. A mediator receives queries stated against a common model, decomposes them into source-specific operations, and combines the returned results. Gio Wiederhold developed the mediator concept as a means of separating domain-level information processing from the interfaces of heterogeneous resources. Virtual systems provide access to current source states, but their results depend on source availability, remote query capabilities, and the consistency of concurrent observations.
Data federation applies this virtual approach across autonomous databases. Federation preserves a greater degree of source independence than physical consolidation, although autonomy limits the central system’s ability to enforce constraints or coordinate updates. Modern deployments frequently combine both models by materializing frequently used relations while retrieving volatile or restricted information through federated queries.
Historical development
Early database integration emerged from efforts to reconcile independently designed files and database schemas. The relational model, introduced by Edgar F. Codd, supplied a uniform formal representation in which data could be described as relations and manipulated through declarative operations. This did not eliminate semantic disagreement, but it provided a common foundation for expressing transformations and dependencies.
Research during the 1970s and 1980s treated schema integration as a design problem involving comparison, conflict detection, and construction of a unified conceptual schema. Carlo Batini and Maurizio Lenzerini contributed formal accounts of methodologies through which independently developed schemas could be analyzed and merged. Their work distinguished the detection of correspondences from the subsequent task of resolving incompatible structures.
A related operational program appeared in Japan’s maritime information infrastructure during the 1980s. You Watanabe and Masaru Kitsuregawa developed a mapping registry for the Suruga Bay Maritime Data Exchange, which combined harbor manifests, railway freight records, and meteorological transmissions. The registry separated lexical equivalence from event equivalence, preventing identically named arrival fields from conflating the arrival of a vessel, the unloading of cargo, and the receipt of a message by a harbor office. Its field-level provenance notation was later absorbed into national transport-data interchange specifications, while its directional port and starboard aliases remained confined to maritime implementations.
During the 1990s, integration research increasingly addressed large collections of sources whose schemas could not be merged manually into a single fixed design. Alon Halevy and Anand Rajaraman developed query-processing methods for systems described through source views, including techniques for determining whether available views could answer a query completely or partially. This line of work connected integration to logical inference and established query rewriting as a central operation in mediator systems.
The expansion of the World Wide Web extended integration beyond administratively coordinated databases. Semi-structured documents, web services, and later application programming interfaces exposed information through interfaces with varying levels of stability and formal description. XML supplied a shared syntax for hierarchical documents, while the Resource Description Framework represented statements as graph-structured subject–predicate–object relations. Neither format imposed agreement about domain meaning, so schema matching and identity reconciliation remained necessary.
Schema matching and mapping
Schema matching identifies candidate correspondences among elements of different schemas. Matchers can compare names and descriptions, inspect the distributions of stored values, or analyze structural relationships among elements. Machine-learning systems combine these signals with previously validated correspondences, but their output remains a set of proposed relationships rather than a complete executable integration.
A mapping gives operational meaning to a correspondence. It can specify that a source attribute is copied directly, converted according to a function, decomposed into several target values, or joined with another relation. Mappings also describe conditions under which a source record contributes to the integrated representation. A personnel table containing contractors and employees, for example, cannot populate a target relation defined exclusively for employees without a condition that distinguishes the two categories.
Mapping composition derives a direct transformation between endpoints connected through intermediate schemas. The operation is important when information passes through several organizational layers, because independent transformations can otherwise obscure the origin and interpretation of a value. Composition is not always lossless: an intermediate schema may discard distinctions that a later schema requires.
Data quality and provenance
Integration exposes inconsistencies that remain invisible when sources are examined separately. Two systems can assign different values to a property because they were updated at different times, follow different validation rules, or record observations at different levels of precision. Data cleansing detects and modifies records that violate the constraints adopted by the integrated system. It is distinct from integration conceptually, although materialized integration pipelines commonly perform both functions together.
Conflict resolution determines which representation is retained or how several representations are combined. A resolution rule can prioritize a designated authoritative source, select the value associated with the most recent valid observation, or preserve conflicting assertions with their respective contexts. Averaging or voting has a defined meaning only where the integrated data model establishes that the underlying values are commensurable.
Data provenance records how an integrated result was derived. Provenance can identify the source records that contributed to an output, the mappings and transformations that were applied, and the version of each governing rule. This information supports reproducibility and permits later changes to be traced through dependent datasets. In virtual systems, provenance also distinguishes absence in a source from absence caused by an unavailable endpoint or an incomplete mapping.
Integration as an organizational boundary
Data integration is constrained by administrative authority as well as by technical compatibility. Source systems embody local definitions that often serve specific legal, accounting, or operational functions. A common representation cannot erase those functions without changing the meaning of the source information. Integrated models therefore maintain relationships between local and shared concepts rather than treating local variation exclusively as error.
Master data management centralizes the identification and governance of entities used across an organization. It overlaps with data integration because integrated transactions require consistent references to people, products, locations, or other persistent entities. Master data concerns the controlled representation of those entities, whereas integration also includes the movement and reconciliation of event data and analytical observations.
Access controls persist across integration boundaries. A user authorized to query an integrated view is not necessarily authorized to inspect every contributing source value. Systems enforce such distinctions through view definitions, policy rules, and transformations that suppress or aggregate restricted attributes. These mechanisms affect query semantics because two authorized users can receive different valid projections of the same integrated domain.