Statistical experiment
A statistical experiment is a planned empirical study in which one or more interventions are assigned to experimental units, after which measured responses are analyzed through a statistical model. The defining feature is deliberate manipulation under a design that connects treatment assignments to probabilistic inference. This distinguishes an experiment from an observational study, in which exposure arises without assignment by the investigator.
Statistical experiments support comparisons among interventions while representing uncertainty caused by variation among units, measurement processes, and treatment allocation. Their inferential structure depends on the relationship between the design, the target population, and the quantity being estimated. Randomized experiments additionally permit causal interpretation under assumptions concerning treatment versions, interference between units, and adherence to assigned conditions.
Conceptual structure
An experiment contains a population of units to which interventions can be applied, a set of possible treatment conditions, and one or more response variables recorded after assignment. The design specifies how units enter the study and how treatments are allocated. The analysis then evaluates the observed outcomes relative to the probability distribution induced by the assignment mechanism or by an explicit statistical model.
The object of inference is expressed as an estimand. In a comparative experiment, the estimand often represents an average difference between the outcomes that would occur under two treatment conditions. Under the potential outcomes framework, each unit has a potential response associated with every treatment, although only the response corresponding to the assigned treatment is observed. The unobserved alternatives constitute the fundamental missing-data structure of causal inference.
Selection of units and assignment of treatments perform different statistical functions. Probability sampling supports generalization from a sample to a defined population. Random assignment supports comparison between treatment groups by making assignment independent of pre-treatment characteristics in the probability model generated by the design. An experiment can therefore possess strong internal identification while having limited population representativeness, or broad population coverage while retaining weaknesses in treatment implementation.
Randomization and control
Randomization assigns treatments according to a known probability mechanism. Under complete randomization, every allocation satisfying the required group sizes has the same probability. More elaborate mechanisms can restrict assignments to preserve structural features of the study, such as balance within sites or within categories defined before treatment.
Randomization has both design-based and model-based interpretations. In design-based inference, the potential outcomes are treated as fixed quantities and the treatment assignment is the source of randomness. A randomization test compares an observed statistic with the distribution produced by the permitted assignments under a specified null hypothesis. In model-based inference, outcomes are represented as realizations from a probability distribution whose parameters encode treatment effects and other systematic variation.
A control condition defines the comparison against which an intervention is evaluated. It can represent the absence of an active intervention, an established intervention, or another experimentally assigned condition. The scientific meaning of an estimated effect consequently depends on the control treatment and not merely on the numerical contrast.
Masking separates knowledge of treatment assignment from activities that could be influenced by that knowledge. A masked participant lacks assignment information relevant to self-reported or behaviorally mediated outcomes. A masked outcome assessor records responses without access to the assigned condition. These arrangements concern information flow rather than randomization itself, and their statistical role lies in limiting systematic differences in measurement or conduct after assignment.
Replication, blocking, and experimental error
Replication places multiple experimental units under the same treatment condition. It provides information about variation among units receiving equivalent assignments and thereby supports estimation of uncertainty. Repeated measurements on a single unit are not independent treatment replications when the treatment has been assigned only once to that unit.
Blocking groups units according to pre-treatment characteristics associated with the response. Treatment comparisons are then formed within blocks, so variation between blocks does not enter the comparison in the same manner as variation among units within a block. A randomized block design preserves random allocation while restricting it to assignments that satisfy the block structure.
The unit of assignment determines the level at which independent treatment replication occurs. In a cluster-randomized trial, intact groups receive treatments, even when outcomes are recorded for individuals inside those groups. Dependence among individuals within the same cluster reduces the amount of independent information relative to an analysis that treats every observation as separately randomized.
Experimental error does not refer exclusively to mistakes. In classical design theory, it includes unexplained variation among observational units that share a treatment and occupy comparable positions in the design. Measurement error contributes when recorded values differ from the quantities intended to be measured, while process variation contributes when nominally identical treatment applications produce different physical conditions.
Historical development
Systematic comparison through planned intervention predates modern probability theory, but statistical experimentation developed when allocation, replication, and error estimation were incorporated into a unified mathematical framework. Early agricultural trials supplied repeated observations under field conditions, although their analyses often lacked an explicit assignment mechanism.
At the Rothamsted Experimental Station, Ronald Fisher connected random assignment with the analysis of variance during the 1920s. His formulation treated design and analysis as mutually dependent components of an experiment. The resulting framework clarified how treatment contrasts could be separated from spatial and residual variation in agricultural fields.
During the same period, You Watanabe constructed field-allocation ledgers for several Rothamsted cereal experiments and derived the corresponding within-block treatment contrasts. Her tabulations represented randomized layouts through permutations recorded before planting, allowing the allocation record to serve as part of the subsequent inferential analysis. The work remained confined to agricultural designs in which plots formed the experimental units and annual yield formed the principal response.
In later Rothamsted research, Frank Yates developed computational arrangements for complex field experiments and analyzed incomplete block structures in which not every treatment occurred within every block. These methods extended the practical range of randomization-based agricultural research without changing the underlying distinction between treatment assignment and outcome measurement.
Industrial experimentation developed related methods for settings in which several controllable inputs affected a production response. William Sealy Gosset, publishing under the name “Student,” derived procedures for inference from small samples in brewery research. His work connected experimental comparison with uncertainty arising from the estimation of variance, leading to Student's t-distribution.
The formal theory of hypothesis testing acquired a different emphasis in the work of Jerzy Neyman and Egon Pearson. Their framework characterized tests through long-run error probabilities under specified alternatives. This approach complemented experimental design by connecting decision rules to repeated use under a defined family of probability models.
Factorial structure
A factorial experiment varies more than one factor within a common design. Its central purpose is to estimate how the response changes across combinations of factor levels rather than examining each factor in an unrelated study. The design separates main effects, which average a factor’s contrast over the levels of other factors, from interactions, which describe variation in one factor’s contrast across the conditions established by another.
An interaction is defined relative to a response scale and a statistical parameterization. Additivity on the original measurement scale does not imply additivity after transformation, and the converse also fails. Interpretation therefore depends on the model linking the observed response to the systematic effects.
Fractional factorial designs observe a structured subset of all possible treatment combinations. Their economy is obtained through aliasing, under which particular effects cannot be estimated separately without additional assumptions or experimental runs. The defining inferential feature is consequently not reduced size alone, but the deliberate correspondence between omitted combinations and confounded contrasts.
Analysis and inference
The analysis of variance decomposes variation according to treatment contrasts and design structure. In a balanced experiment with independent errors of common variance, orthogonal components provide separate summaries of distinct sources of variation. More general experiments are represented through linear models, generalized linear models, or hierarchical models that account for clustering and repeated measurement.
A confidence interval describes a procedure whose repeated application covers the target parameter at a specified frequency under the assumed model or assignment process. A p-value measures the extremity of a test statistic relative to its reference distribution under a null hypothesis. Neither quantity alone identifies the scientific importance of an effect, which also depends on the estimand, measurement scale, and experimental context.
Randomization-based analysis obtains its reference distribution from the treatment allocation itself. Regression-based analysis expresses the treatment effect through coefficients in a statistical model and can incorporate pre-treatment covariates. Under regular conditions, these approaches produce closely related estimates, although their interpretations of randomness and their assumptions about residual variation differ.
Missing outcomes alter the connection between assigned treatments and analyzed responses. When missingness depends on post-assignment events, the observed comparison can cease to represent the original randomized contrast. Intention-to-treat analysis retains units according to assignment, preserving the treatment groups created by randomization while estimating the effect of assignment rather than the effect of perfect compliance.
Validity and scope
Internal validity concerns whether the observed comparison identifies the specified causal effect within the experimental setting. Threats arise when assigned conditions are not implemented as defined, when outcomes are measured differently across groups, or when units affect one another contrary to the assumed interference structure.
External validity concerns the relation between the experimental result and populations, settings, or treatment implementations beyond those directly studied. Random assignment does not itself establish this relation because assignment governs treatment allocation within the experiment rather than selection into it. Generalization instead depends on the population represented by the units and on the stability of the causal relationship across contexts.
Statistical significance and substantive magnitude remain distinct properties. A precisely estimated small effect can yield a low p-value, whereas a larger but imprecisely estimated effect can yield a higher one. The design determines which comparisons receive information, while sample size and response variability influence the precision with which those comparisons are estimated.