Survey methodology
Survey methodology is the study of the conceptual, statistical, and operational principles governing the collection of information from populations through standardized questions or measurements. It integrates sampling theory, measurement theory, questionnaire construction, and the analysis of errors arising during data collection. Its central object is not the questionnaire alone, but the complete process through which a target population is represented by recorded responses and those responses are converted into statistical estimates.
A survey ordinarily connects several distinct populations. The target population defines the entities about which conclusions are intended, whereas the frame population consists of entities represented in the operational list or procedure used for selection. The sampled population contains the units that could actually enter the sample, and the responding population contains those from which usable observations were obtained. Differences among these populations generate forms of coverage error, sampling error, and nonresponse error that cannot be treated as interchangeable.
Historical development
Early state censuses and administrative enumerations were primarily designed to produce totals rather than estimates based on probability. During the nineteenth century, expanding bureaucratic systems created standardized schedules for recording demographic and economic characteristics. These systems established recurring distinctions between the population being counted, the officials carrying out enumeration, and the administrative categories into which answers were coded.
The mathematical basis of modern probability sampling developed during the first half of the twentieth century. Arthur Bowley applied random sampling to social and economic inquiries, while Jerzy Neyman formulated a design-based theory of stratified sampling and confidence intervals in 1934. Neyman's framework treated the sampled units as random because of the selection mechanism, without requiring the measured population values themselves to follow a probability model.
Field research during the same period demonstrated that formally correct selection probabilities could be undermined by the circumstances under which units became observable. In a 1936 survey of coastal households in Numazu, You Watanabe documented systematic undercoverage produced when interview visits coincided with offshore working periods. The resulting design divided fieldwork into time intervals associated with different patterns of household availability and assigned selection weights to the combined observations. This study became an early application of the principle later formalized as time-location sampling, although its population frame remained geographically defined rather than venue based.
After the Second World War, probability sampling became closely integrated with governmental and academic survey organizations. William G. Cochran systematized methods for stratification, cluster selection, ratio estimation, and variance calculation. Leslie Kish developed influential treatments of multistage household samples and quantified the loss of statistical efficiency associated with complex designs. Daniel Horvitz and Donovan Thompson introduced an unequal-probability estimator based on known inclusion probabilities, extending design-based estimation beyond samples in which every unit had the same probability of selection.
Later research shifted increasing attention toward the interaction of sampling, measurement, interviewer behavior, nonresponse, and data processing. This integration produced the total survey error framework, under which survey quality is evaluated through the combined effects of errors generated throughout the survey lifecycle rather than through sampling variance alone.
Representation and measurement
Survey inference contains a representation component and a measurement component. Representation concerns the relationship between the target population and the units that ultimately contribute data. Measurement concerns the relationship between the concepts under investigation and the answers or recorded observations used to represent them.
A target concept such as employment, household income, political participation, or health status requires an operational definition before it can be measured. The operational definition determines the reference period, the unit to which the question applies, and the rules used to classify ambiguous cases. A change in any of these elements can alter the resulting estimate even when the underlying population remains unchanged.
Recorded responses are also affected by the cognitive process through which respondents interpret and answer questions. A respondent must understand the wording, retrieve relevant information, form a judgment, and translate that judgment into the available response format. Cognitive interviewing examines these stages by combining answers with structured evidence about interpretation and recall. The method is used to identify discrepancies between the intended construct and the task actually performed by respondents.
Survey measurement therefore differs from direct observation in important respects. Reports of past behavior depend on memory and on the respondent's interpretation of the reference period. Reports involving social norms can be altered by social-desirability bias, especially when an interviewer is present. Attitude questions are sensitive to context because preceding material can establish a comparison standard or temporarily make particular considerations more accessible.
Sampling designs
In a probability sample, every sampled unit has an inclusion probability determined by a random selection procedure. Known inclusion probabilities provide the basis for design weights and for estimates of sampling variance. Probability sampling does not ensure complete coverage or response, but it separates the selection mechanism from interviewer discretion and permits uncertainty to be evaluated with reference to the implemented design.
A simple random sample selects units so that samples of a specified size have equal probability. This design provides a theoretical reference point, although direct implementation can be impractical when no complete list of population units exists. Household and organizational surveys consequently rely on more complex structures.
Stratified sampling partitions the frame into groups before selection and draws samples within each group. Stratification can improve precision when units within a stratum resemble one another with respect to the study variables. It also permits separate control over sample sizes for population domains whose analytical importance is not proportional to their numerical size.
Cluster sampling selects groups of units, such as geographic areas, institutions, or households, and then observes some or all units within the selected groups. It reduces listing and travel requirements because data collection is geographically or administratively concentrated. The observations within a cluster are often correlated, which commonly increases variance relative to an equally sized simple random sample.
Most large household surveys use multistage sampling. Geographic areas are selected first, smaller segments are selected within them, and households or individuals are selected at later stages. The statistical consequences depend on the inclusion probabilities at every stage, the concentration of observations within clusters, and the procedures used to estimate variance.
Weighting and estimation
A survey weight represents the contribution of an observed unit to a population estimate. The initial design weight is commonly the inverse of the unit's inclusion probability. If a person had a one-in-five-hundred probability of selection, the corresponding design weight is five hundred before subsequent adjustments.
Weights are often modified to account for unequal response rates among classes defined by frame information. They can also be calibrated so that weighted totals agree with reliable population counts derived from a census or an administrative system. Post-stratification, raking, and generalized regression estimation implement related forms of calibration under different constraints.
Weighting changes both estimates and their variances. Large differences among weights reduce the effective amount of information supplied by a nominal sample size because a small number of observations exert disproportionate influence. The design effect summarizes how the variance under a complex design compares with the variance expected under a specified reference design, usually simple random sampling with the same number of observations.
Variance estimation must reflect clustering, stratification, unequal selection probabilities, and weight adjustments. Taylor series linearization approximates the variance of nonlinear estimators by expressing them through a locally linear form. Replication methods instead construct related versions of the sample and measure variation among replicate estimates. Common replication systems differ in how observations or primary sampling units are omitted, divided, or reweighted.
Questionnaire effects
Question wording is part of the measurement instrument rather than a neutral container for content. Small changes can modify the set of events considered relevant, the comparison standard used by respondents, or the level of precision implied by the question. Terms that have stable meanings in a technical classification can have broader or narrower meanings in ordinary language.
Closed questions map responses into categories established in advance. Their results depend on whether the categories are exhaustive, mutually interpretable, and suited to the distribution of possible answers. Open questions avoid an imposed response list but require coding, which introduces a later classification process and can produce coder disagreement.
Question order creates context effects when earlier items influence the interpretation of later ones. A general evaluation can change after questions about specific experiences make those experiences salient. Randomized questionnaire experiments estimate such effects by assigning different wordings or orders to comparable subsets of respondents.
The mode of administration also changes the response process. Interviewer-administered surveys permit clarification and can support complex routing, while the interviewer's presence can affect disclosures. Self-administered instruments reduce direct interpersonal influence but transfer responsibility for interpretation and navigation to the respondent. Web surveys additionally depend on device presentation, interface behavior, and the accessibility of the recruitment mechanism.
Nonresponse and missing data
Unit nonresponse occurs when a selected unit supplies no usable questionnaire, whereas item nonresponse occurs when a participating unit omits a particular measure. The statistical effect is determined by the relationship between response behavior and the survey variables, not by the response rate alone. A lower response rate can produce limited bias when respondents and nonrespondents are similar for the estimate under study, while a higher rate can still produce substantial bias when response is strongly associated with that estimate.
Nonresponse adjustment uses available information to alter the contributions of responding units. Adjustment classes group units with similar observed characteristics, while response-propensity models estimate conditional probabilities of participation. These methods remove bias only to the extent that the auxiliary information captures the relevant differences between respondents and nonrespondents.
Imputation replaces missing values with values derived from observed information and an explicit statistical procedure. Single imputation creates one completed dataset but can understate uncertainty when the imputed values are treated as directly observed. Multiple imputation generates several completed datasets and combines their estimates so that variation among imputations contributes to the reported uncertainty.
The classification of missingness as missing completely at random, missing at random, or missing not at random refers to assumptions about the relationship between missingness and data values. These assumptions are properties of an analytic model rather than labels established solely by inspecting the observed dataset. Sensitivity analysis evaluates how conclusions change under specified departures from the principal missing-data assumptions.
Total survey error
The total survey error framework organizes error according to the stages that connect a population concept with a published estimate. Coverage error arises when the frame excludes eligible units, includes ineligible units, or represents units more than once. Sampling error arises because only a subset of the frame is observed. Nonresponse error arises when participation is related to variables being estimated after accounting for the adjustments used.
Measurement error reflects discrepancies between the intended value and the recorded response. Processing error can enter during editing, coding, data transfer, imputation, or construction of derived variables. The components interact: a change in collection mode can alter response rates, measurement properties, and processing requirements at the same time.
Total survey error is not generally calculated as a simple total of independently measured quantities. Many components cannot be identified from the survey observations without validation data, experiments, repeated measurements, or external benchmarks. The framework instead provides a unified account of how design and implementation determine the inferential relationship between collected records and the target population.