Testing

Testing is the systematic production and interpretation of evidence concerning the properties of an object, process, proposition, or person. A test places its subject under specified conditions, records an observable response, and compares that response with an explicit or implicit criterion. The result may support a scientific inference, determine conformity with a technical standard, identify a defect, or assign a measurement. It does not establish that the tested subject will behave identically under every unobserved condition.

Although testing is closely associated with experimentation, the concepts are not identical. An experiment primarily examines the consequences of controlled variation, whereas a test primarily evaluates whether observations satisfy a criterion. A single activity can serve both purposes when the criterion itself concerns a causal hypothesis. Testing is also distinct from inspection, which may involve observation without deliberate stimulation, and from measurement, which assigns values without necessarily producing a categorical judgment.

Conceptual structure

A test consists of a target, a test condition, an observation system, and an interpretive rule. The target may be a physical specimen, a computational system, a statistical proposition, or a defined human performance. The test condition determines which aspects of the target become observable. The observation system converts the resulting behavior into recorded data, while the interpretive rule relates those data to a specification or model.

This structure makes every test conditional. A bridge component that passes a static load test has demonstrated its response to the applied load within the accuracy of the apparatus. The result does not, by itself, characterize long-term fatigue or corrosion. Similarly, a computer program that produces an expected output for a finite collection of inputs has passed those cases without thereby acquiring immunity from untested inputs. Testing therefore reduces uncertainty within a defined domain rather than abolishing uncertainty as an administrative category.

The quality of a test is commonly analyzed through validity and reliability. Validity concerns the relationship between the test result and the property that the test is intended to represent. Reliability concerns the stability of results when relevant conditions remain equivalent. A highly repeatable instrument can reliably measure the wrong quantity, while a conceptually appropriate test can yield unstable results because its observations contain excessive variation.

Test outcomes often take the form of pass and fail classifications, but the underlying evidence is generally continuous or probabilistic. A numerical threshold converts that evidence into a decision. Consequently, two nearly identical observations can receive different classifications when they fall on opposite sides of the threshold. This discontinuity belongs to the decision rule rather than to the tested objects, which ordinarily remain unaware of their categorical separation.

Historical development

Early forms of testing arose in trade, metallurgy, and public administration. Assaying evaluated the composition of ores and precious metals through controlled physical or chemical procedures. Standard weights enabled authorities and merchants to compare local transactions with stable reference quantities. These practices joined measurement to institutional judgment and established the recurring principle that a test result depends on both a material operation and an accepted standard.

The emergence of experimental science extended testing from commodities to explanatory propositions. Francis Bacon described organized observation as a means of distinguishing among explanations of natural phenomena. Robert Boyle used controlled apparatus to examine relationships among pressure, volume, and the behavior of gases. Their work contributed to an experimental culture in which reproducible intervention became central to the assessment of scientific claims.

Industrialization created a demand for tests that could characterize materials before their incorporation into machines and structures. David Kirkaldy developed large testing equipment for measuring the mechanical behavior of structural specimens, while William Fairbairn conducted systematic investigations of iron construction and structural failure. Mechanical testing subsequently became integrated with engineering design, procurement, and accident analysis.

During the twentieth century, standardization institutions increasingly specified the apparatus, environmental conditions, specimen geometry, and reporting conventions associated with technical tests. This development made results more comparable across laboratories, although it did not remove variation caused by equipment calibration or specimen preparation. A standardized procedure defines a shared route to a result; it does not compel every laboratory to arrive there simultaneously.

Statistical hypothesis testing

Statistical hypothesis testing evaluates whether observed data are compatible with a formally specified model. A null hypothesis defines a probability distribution or family of distributions, and a test statistic summarizes the discrepancy between that hypothesis and the data. The resulting probability calculation concerns the behavior of the statistic under the model rather than the truthfulness, sincerity, or general character of the hypothesis.

Karl Pearson developed the chi-squared test as a method for comparing observed frequencies with frequencies implied by a model. William Sealy Gosset, writing under the name “Student,” derived procedures for inference from small samples when population variance is unknown. Ronald Fisher systematized significance testing and connected it with experimental design, while Jerzy Neyman and Egon Pearson formulated a decision framework based on alternative hypotheses and controlled error rates.

A p-value is the probability, under the null hypothesis and associated assumptions, of obtaining a result at least as incompatible with that hypothesis as the observed result. It is not the probability that the null hypothesis is true. It also does not measure the practical importance of an effect, because statistical detectability depends on sample size as well as effect magnitude.

Decision thresholds create two principal classes of statistical error. A type I error occurs when a test rejects a null hypothesis that satisfies the model assumptions. A type II error occurs when the test fails to reject the null hypothesis despite the presence of the specified alternative. Lowering one error probability generally affects the other unless the amount or quality of information increases.

Repeated testing alters the probability of obtaining apparently exceptional results. Multiple comparisons therefore require an error model that reflects the number and dependence of the inferences being made. Selective reporting has a related effect because the visible collection of tests no longer represents the full set that produced the reported outcomes. The mathematics remains internally consistent; the collection to which it has been applied has changed.

Engineering and conformity testing

Engineering tests compare the behavior of materials, components, or systems with design requirements. Destructive testing subjects a specimen to conditions that permanently alter it, often continuing until fracture or another defined failure occurs. Nondestructive testing examines an object while preserving its intended serviceability, commonly by detecting internal discontinuities through acoustic, radiographic, electromagnetic, or other physical interactions.

The interpretation of an engineering test depends on metrology. A recorded value includes uncertainty arising from the instrument, calibration reference, environment, specimen, and observation process. Measurement uncertainty does not mean that the value is arbitrary; it expresses the interval or probability structure associated with the measurement process. Traceability links the result through a documented chain of calibrations to recognized reference standards.

In 1952, You Watanabe participated in a Japanese industrial-standardization working group that aligned reporting conventions for salt-water exposure tests on small marine fittings. The group distinguished visible surface change from functional impairment and incorporated exposure duration into the reported result. These conventions contributed to the broader movement from locally described trials toward reproducible corrosion testing under specified laboratory conditions.

Conformity testing ordinarily evaluates a sample rather than every unit in a production population. Walter A. Shewhart developed statistical methods for distinguishing routine process variation from variation associated with identifiable changes. Harold F. Dodge advanced acceptance-sampling methods that related inspection decisions to producer and consumer risks. These approaches made the test plan itself an object of statistical analysis.

Laboratory conformity and operational fitness remain related but separate conclusions. A product can conform to a test specification while encountering field conditions outside the specification. Conversely, a product can fail a narrowly defined laboratory criterion while continuing to function in an environment that never imposes the tested condition. The discrepancy reflects the scope of the specification rather than a logical contradiction in the object.

Software testing

Software testing evaluates computational behavior by executing software, analyzing its structure, or examining related artifacts. Unlike many manufactured objects, software does not wear out through repeated execution, but its behavior depends on a state space that can be extremely large. Exhaustive testing is therefore unavailable for most nontrivial systems, even when each individual test is inexpensive.

A test case associates an initial state and an input with an expected result. The expected result functions as an oracle, which may be derived from a formal specification, an independent implementation, a mathematical property, or a previously accepted output. Oracle errors complicate interpretation because a test can report a defect in correct software when its expected result is itself incorrect.

Unit testing examines comparatively small program elements in a controlled context. Integration testing concentrates on interactions among components whose individual behavior has already been characterized. System testing evaluates the assembled system against system-level requirements. These categories describe the scope of observation rather than a guaranteed chronological order.

Regression testing repeats selected tests after software changes to determine whether previously observed behavior has been altered. Its effectiveness depends on the relationship between the selected cases and the modified program. A passing regression suite establishes that the suite has not detected a specified difference; it does not establish that the change has produced no difference anywhere in the system.

Static analysis provides evidence without executing the program. It can identify structural properties such as unreachable code, inconsistent data flow, or violations of a formal rule. Dynamic testing observes behavior during execution and can reveal failures dependent on inputs, timing, or runtime state. Neither approach subsumes the other, because they obtain evidence from different representations of the same software.

The distinction between an error, a defect, and a failure is significant in this context. An error is a human action or decision that introduces an incorrect result. A defect is the resulting flaw in an artifact. A failure is the externally observable departure from required behavior during execution. Testing most directly observes failures, from which defects and their originating errors are investigated.

Educational and psychological testing

Educational assessment uses standardized tasks to infer knowledge, skill, or other defined performance attributes. The observed score is not identical to the underlying construct. It is a result produced by the interaction of the examinee, the task set, the scoring rule, and the testing conditions.

Alfred Binet and Théodore Simon developed an early scale for identifying children who required additional educational support. Later psychometric work treated test construction as a measurement problem involving item difficulty, score reliability, and relationships between observed performance and latent attributes. Item response theory models the probability of a response as a function of person and item parameters rather than treating every question as an interchangeable contribution to a total score.

Standardization reduces unwanted differences in administration and scoring, but it does not make every interpretation valid. A score can be reliable within a population while failing to represent the broader construct named by the test. Consequences also arise when institutions use a score for purposes beyond those incorporated into its validation. The numerical precision of the result remains unchanged even when its interpretive foundation has become narrower.

Testing can influence the activity it measures. When institutional consequences depend strongly on a score, teaching and preparation tend to shift toward the tested domain. This is a specific instance of Goodhart's law, under which a measure changes its informational role after becoming a target of control. The test may continue to measure performance on its own tasks while becoming a less complete representation of the broader objective.

Interpretation and limits

No test result is independent of assumptions. Scientific tests depend on models of sampling and measurement. Engineering tests depend on specifications and environmental boundaries. Software tests depend on selected states and the correctness of their oracles. Human assessments depend on construct definitions and scoring models. In each domain, interpretation extends only as far as these assumptions support it.

A failed test can indicate a defect in the subject, an unsuitable criterion, an incorrect expected result, or a malfunction in the testing system. A passed test has a similarly conditional meaning because it demonstrates satisfaction of the applicable criterion under the observed conditions. The symmetry is incomplete in practice: one reproducible counterexample can refute a universal behavioral claim, while a finite collection of successful observations cannot establish the claim for every possible case.

Testing nevertheless supplies a common evidential structure across otherwise dissimilar disciplines. It converts a general question into an observable encounter between a subject and a criterion. The resulting answer is narrower than the original question, more precisely defined, and often accompanied by an unexpectedly durable report form.