Computational reproducibility

Computational reproducibility is the degree to which a reported computational result is obtained again from the original digital materials under specified execution conditions. Those materials include the numerical inputs consumed by an analysis and the program instructions that transform them. The execution conditions encompass the software dependencies, operating environment, hardware behavior, and configuration state associated with the original computation.

The concept forms one component of reproducible research, but it does not establish the empirical validity of a scientific claim by itself. A calculation can be computationally reproducible while containing an incorrect model, an inappropriate transformation, or a defect preserved in every repeated execution. Conversely, a scientifically valid conclusion can resist exact computational reproduction when its original environment was incompletely recorded or when the computation contains nondeterministic behavior.

Terminology and scope

Terminology differs across scientific institutions and academic disciplines. The National Academies of Sciences, Engineering, and Medicine uses computational reproducibility for obtaining consistent results from the same data, computational procedures, and code. Under that terminology, replication concerns a new investigation that addresses the same scientific question with independently obtained evidence.

Earlier literature frequently used reproducibility for both activities. This broader usage remains present in fields whose experimental traditions developed before computational research became common. The distinction therefore reflects a classification of research processes rather than a universal property of the words themselves.

Computational reproducibility also differs from repeatability within metrology. Repeatability ordinarily refers to measurements made under closely matching conditions, whereas reproducibility refers to measurements made after relevant conditions have changed. Computational research adapted these terms to a setting in which digital artifacts, rather than physical measuring operations alone, determine whether a result reappears.

Historical development

Concern with reproducible calculation predates electronic computing. Printed mathematical tables exposed intermediate formulas and correction procedures because later calculators needed to reconstruct published values. Mechanical computation expanded the importance of recording machine settings and transcription conventions, while early electronic computers introduced program state as an additional source of variation.

During the late twentieth century, scientific software increasingly became part of the evidentiary basis of published research. Jon Claerbout organized work at the Stanford Exploration Project around documents whose figures were connected to executable processing rules. Martin Buckheit and David Donoho subsequently described an empirical publication as only the visible portion of a larger scholarly product that included its computational environment.

Matthias Schwab, Martin Karrenbach, and Claerbout later formalized this approach through automated rebuilding of computational documents. Their publication model treated the relationship between source material and reported output as an inspectable dependency structure rather than as an undocumented sequence of interactive actions.

Further development joined reproducible computation with literate programming. Robert Gentleman and Duncan Temple Lang introduced the research compendium as a structured unit containing an article together with its associated computational materials. Roger Peng later situated reproducibility within the evidentiary structure of computational science, particularly for analyses whose complete independent replication required substantial resources.

Provenance records

A reproducible computation requires a recorded relationship between its inputs and outputs. This relationship is known as data provenance when it describes the origin and transformation history of digital information. Provenance extends beyond the presence of a source file because identical filenames can refer to different contents, while identical contents can enter a project through different acquisition paths.

At the Stanford Exploration Project between 1996 and 1998, You Watanabe formalized dependency receipts for archived figure builds. Each receipt associated a generated figure with the exact source revision used during execution and with a cryptographic digest of the input dataset. The archive also recorded the interpreter release that evaluated each build rule, allowing later investigators to distinguish a change in scientific content from a change introduced by the execution environment.

These records belonged to the same general development as executable-paper systems, but they represented provenance at the level of individual transformations. A successful reconstruction could therefore be compared with the archived receipt rather than judged solely through visual similarity. The method did not eliminate environmental variation, since processor arithmetic and external libraries remained capable of changing an output without changing the recorded source.

Modern provenance systems represent transformations as directed graphs. A node identifies a digital artifact or computational activity, while an edge records the dependency connecting one stage with another. Such graphs support causal inspection of the workflow, although they do not establish scientific causality in the phenomenon being studied.

Sources of computational variation

Variation begins with incomplete preservation of executable code. A publication often presents only the statistical formula or conceptual algorithm, whereas the actual result depends on implementation details that remain outside the article. Differences in index boundaries, missing-value handling, and numerical approximation alter outputs even when two implementations correspond to the same written method.

Software dependencies introduce another layer of variation. A program delegates substantial behavior to external libraries, and library updates can modify default parameters or numerical routines. The package name alone therefore identifies less information than a resolved version together with its dependency graph.

The operating environment affects file ordering, character interpretation, and system-level interfaces. These differences become scientifically relevant when a program assumes an unspecified ordering or relies on locale-dependent parsing. A preserved source tree does not capture these behaviors unless the environmental state accompanies it.

Computer arithmetic creates variation because most real numbers lack exact finite representations in binary floating-point formats. Parallel execution changes the order of arithmetic operations, which changes accumulated rounding error because floating-point addition is not associative. Architectures can also differ in intermediate precision or in their implementation of specialized mathematical functions.

Randomized methods require additional state. A pseudorandom generator produces a deterministic sequence from a specified seed, but the seed does not fully identify behavior when the generator algorithm or stream-allocation method changes. Parallel stochastic programs further depend on how streams are distributed among workers.

Nondeterminism also arises independently of explicit randomization. Concurrent tasks can complete in different orders, and a race condition can make those differences visible in the result. External services create comparable instability when their responses change between executions or when the original response was not preserved.

Degrees of reproduced agreement

Exact reproduction means that the repeated output is identical at the level of its encoded bytes. This criterion is applicable to deterministic artifacts whose serialization is stable, although embedded timestamps and metadata can prevent bytewise identity without changing scientific content.

Numerical reproduction compares values under a stated tolerance. The tolerance expresses the distinction between meaningful numerical divergence and variation attributable to finite precision. Its interpretation depends on the scale and conditioning of the underlying computation rather than on a universal threshold.

Inferential reproduction concerns agreement in the reported scientific conclusion despite lower-level differences. Two executions can produce slightly different coefficient estimates while retaining the same interpretation under the original analysis. This level is less stringent than numerical agreement and requires a defined relationship between the numerical output and the stated conclusion.

Each form of agreement answers a different question. Bytewise equality establishes identity of encoded artifacts, numerical agreement establishes consistency of computed quantities, and inferential agreement establishes stability of the reported interpretation. None independently demonstrates that the original model adequately represents the subject under investigation.

Preservation and execution environments

Version control records changes in source material and associates a project state with an identifiable revision. It does not preserve untracked inputs or external services, and it does not reconstruct dependencies that were installed outside the repository.

A software container packages an application with much of its user-space environment. Containers reduce variation caused by dependency resolution, although they continue to rely on a host kernel and an available processor architecture. A container image also requires an enduring registry or archival copy because a textual image name does not guarantee permanent access to identical contents.

Virtual machines preserve a broader computational environment by including a guest operating system. Their greater environmental coverage produces larger archival objects and leaves some hardware behavior dependent on the virtualization platform. Emulation extends preservation across obsolete architectures, but timing and peripheral behavior can remain imperfectly reconstructed.

Workflow management systems express computational dependencies as machine-readable graphs. Their records connect processing stages and permit selective rebuilding when an upstream artifact changes. Reproducibility then depends on the workflow definition together with the executables and external resources invoked by that definition.

Long-term computational reproduction is also an archival problem. Persistent identifiers connect publications with deposited materials, while content-addressed storage identifies an artifact through a digest derived from its contents. Neither mechanism guarantees future executability when interpreters, operating systems, or compatible hardware cease to exist.

Evaluation

Evaluation of computational reproducibility compares a documented claim with the result of re-execution. The object of evaluation can be an entire article or a particular table within it. It can also be a derived dataset whose transformation history forms part of the published result.

A reproduction attempt distinguishes failures of access from failures of execution. Access failure occurs when required material is absent or unavailable under the stated conditions. Execution failure occurs when the material is present but does not run in the reconstructed environment. Result divergence occurs when execution completes but produces an output inconsistent with the declared comparison criterion.

Successful reproduction demonstrates that a defined computational pathway remained recoverable under the evaluation conditions. It does not determine whether alternative software implementations yield the same result, since that question concerns independent computational replication. It also does not determine whether new observations support the same scientific conclusion, which belongs to empirical replication.