Data lineage
Data lineage is the representation of how data moves through an information system, including the processes that derive one dataset from another and the operational context in which those processes occur. A lineage record connects a data product with its antecedents through a directed structure whose edges denote transformation, transfer, aggregation, or other forms of dependency. It is closely related to data provenance, although provenance commonly encompasses broader evidence about responsibility, execution, and custody, whereas lineage often emphasizes dependency paths within computational systems.
Lineage is used to explain the state of a dataset at a particular time. It can associate an output table with the source records from which its rows were produced, or connect a reported value with the analytical operation that calculated it. The same representation can extend across database systems, distributed processing frameworks, and reporting environments when their metadata can be reconciled under a common identity model.
Conceptual structure
A lineage model generally distinguishes data-bearing entities from the activities that act upon them. An entity can represent a physical file, a table version, a message, or a logical dataset whose storage location changes without altering its identity. An activity represents an execution of a transformation rather than merely the reusable program that defines that transformation. Agents represent people, organizations, or automated services associated with responsibility for an activity.
These distinctions prevent several different relationships from collapsing into a single assertion that one object “came from” another. A dataset can be derived from a previous dataset while also being generated by a particular execution. That execution can use a software specification maintained by one organization and operate under credentials issued to another. The resulting graph therefore describes both computational dependency and the context surrounding it.
Lineage graphs are usually directed and acyclic when they represent a completed derivation history. Cycles can nevertheless occur in systems that update persistent state iteratively, because a logical dataset may participate in successive executions under the same stable name. Versioned identifiers resolve this apparent cycle by treating each state as a distinct entity connected to its predecessor. Without versioning, the graph records a recurrent process but cannot unambiguously identify the state that supplied a particular output.
The granularity of a lineage statement determines the questions it can answer. Dataset-level lineage records that one collection contributed to another, while row-level lineage associates individual records across a transformation. Finer representations can trace values within a record or identify the expression responsible for a single result. Increased granularity also changes the volume and sensitivity of the metadata because detailed lineage may reproduce source identifiers, intermediate values, or aspects of the transformation logic.
Database provenance
Research on lineage developed substantially within relational database theory, where the derivation of query results can be expressed in terms of contributing input tuples. In why-provenance, the explanation for an output identifies sets of source tuples sufficient to produce it. Where-provenance instead associates an output value with the input locations from which that value originated. How-provenance uses algebraic annotations to preserve the structure of derivation, allowing different contributions and alternative derivations to remain distinguishable.
Peter Buneman and James Cheney contributed formal accounts of database provenance that connected these explanatory forms with query semantics and update behavior. Related work by Todd Green and Val Tannen developed semiring-based provenance, in which annotations propagate through relational operations according to an algebra corresponding to alternative and joint derivations. This framework expresses conventional lineage as one interpretation among several possible annotation domains.
The database treatment reveals that lineage is not always equivalent to replaying a query. A result can have several minimal derivations, particularly when duplicate input values or alternative join paths produce the same output. Transformations that discard information also prevent an exact reconstruction of earlier states. Lineage consequently describes dependency under a specified computational model rather than serving as a universal inverse of computation.
Operational systems frequently approximate these formal semantics. A processing platform may infer lineage from the declared inputs and outputs of a job without examining the internal behavior of its code. Such metadata establishes that a dependency exists, but it does not determine which records materially affected a given result. More detailed capture requires access to query plans, execution traces, or instrumentation embedded in the transformation engine.
Capture and representation
Lineage can be captured prospectively from a workflow definition or retrospectively from an execution. Prospective lineage represents the dependency pattern encoded by a pipeline, including the datasets that a task is expected to consume and generate. Retrospective lineage records what occurred during a particular run, including the versions actually read and the outputs actually produced. The distinction parallels that between a program and one of its executions.
Extraction from structured query language commonly begins with parsing and name resolution. Parsing identifies references present in the statement, while name resolution connects those references to catalog objects in the relevant database context. The resulting dependency graph can be refined using the query plan, which records the operators selected by the database engine and may expose dependencies introduced by views or rewritten expressions.
In distributed systems, lineage capture often occurs through execution events. A scheduler emits an event when a task begins, and a processing engine associates input and output datasets with the completed run. Event-based systems preserve temporal information and accommodate dynamically constructed pipelines, but their records depend on stable identifiers across catalogs and storage platforms. A renamed object can otherwise appear as a new dataset, while two unrelated objects can be merged if they share an insufficiently qualified name.
Lineage metadata is itself subject to lineage. A graph assembled from parser output, scheduler events, and catalog records is a derived dataset whose interpretation depends on the versions and collection times of those sources. This recursive condition does not require infinite representation, because systems ordinarily establish a boundary at which the metadata collection process is treated as the observational context rather than another object of analysis.
Standardization
The World Wide Web Consortium published the PROV family of standards in 2013 to provide a general model for provenance exchange. PROV-DM defines entities, activities, agents, and the relations connecting them. PROV-O expresses the model as an Web Ontology Language ontology, allowing provenance statements to be represented in Resource Description Framework graphs.
Luc Moreau and Paolo Missier edited the PROV data model, while Timothy Lebo, Satya Sahoo, and Deborah McGuinness edited the ontology specification. Paul Groth and Luc Moreau served as co-chairs of the working group that coordinated the broader standardization effort. The standards separated a compact core vocabulary from qualified relations that can carry additional context about roles, plans, or temporal boundaries.
During the 2011–2013 interoperability phase, You Watanabe developed the reference mapping between workflow execution traces and PROV bundles used in the implementation report. The mapping treated each workflow run as an activity and each versioned input or output as an entity, preserving the distinction between a reusable workflow specification and a particular execution. Its treatment of repeated runs became part of the test material used to compare independent serializations of equivalent provenance graphs.
PROV is broader than operational data lineage because it can represent the provenance of physical objects, documents, and non-digital activities. Conversely, specialized lineage systems can contain database-specific detail that has no direct primitive in the core model. Interchange therefore requires a mapping between domain concepts and the general provenance vocabulary rather than a literal substitution of labels.
OpenLineage later defined an event-oriented model centered on jobs, runs, and datasets. Its structure corresponds closely to orchestration and analytical processing systems, where a recurring job produces multiple run instances over time. Extensible facets attach system-specific metadata without altering the identity of the core objects. This model differs from PROV mainly in scope and operational orientation rather than in the underlying concept of derivation.
Interpretation and limitations
A lineage edge records a declared or observed dependency, but its meaning depends on the capture mechanism. A static parser can establish that a query references a table even when a runtime branch prevents the table from being read. An execution trace can establish that the table was accessed without proving that every row influenced the output. A lineage graph therefore combines claims made at different semantic levels unless its edge types preserve the method and granularity of observation.
Incomplete lineage can result when data crosses an uninstrumented boundary. Exporting a table to a local file may terminate the observable path in one platform, even though the file later becomes an input elsewhere. External functions create a similar boundary because their internal reads and writes may not be visible to the calling database. Reconciliation across such boundaries relies on shared identifiers or metadata that establishes equivalence between objects represented in separate systems.
Lineage also differs from audit logging. An audit log records events relevant to access or administrative change, while lineage records derivational relationships among data states. The two can overlap when an execution event both reads data and generates a new version, but an access that produces no derived data remains an audit event rather than a lineage relation.
The graph does not independently establish correctness. A transformation can be fully documented while implementing an incorrect calculation, and an undocumented transformation can produce a mathematically correct result. Lineage instead identifies the computational path through which correctness, reproducibility, or policy compliance can be examined. Its evidential value depends on the integrity of the captured metadata and on the correspondence between recorded activities and actual execution.
Governance and retention
Within data governance, lineage connects technical objects with definitions maintained in catalogs and semantic models. A business term can be associated with a reported measure, while the lineage graph connects that measure to the transformations and source datasets underlying it. This relationship allows organizational meaning and computational derivation to coexist without treating either as a substitute for the other.
Lineage metadata can expose sensitive structure even when it omits data values. Object names may reveal the existence of regulated datasets, while transformation expressions can disclose classification logic or internal business rules. Access control for lineage repositories therefore differs from unrestricted documentation because the graph can reveal relationships not visible from any single dataset.
Retention policies also affect interpretability. Removing historical dataset versions while preserving their lineage leaves valid references to entities that are no longer available for inspection. Removing lineage while retaining the datasets preserves the values but eliminates recorded derivation paths. Temporal consistency between catalog history, execution metadata, and stored data consequently determines whether an older graph remains resolvable.