Probability theory

Probability theory is the mathematical study of uncertain outcomes and the quantitative regularities produced by random phenomena. It assigns numerical measures to collections of possible outcomes and examines how those measures behave under observation, conditioning, repetition, and transformation. Although probability originated in the analysis of games of chance, its modern formulation is independent of any particular physical interpretation of randomness.

A probability model does not ordinarily determine which individual outcome will occur. Instead, it specifies relations among possible outcomes and supports deductions concerning their relative likelihoods, long-run frequencies, and expected numerical consequences. The theory supplies the mathematical foundation for statistics, stochastic processes, and substantial portions of information theory.

Mathematical formulation

A probability model is conventionally represented by a probability space

[ (\Omega,\mathcal{F},P), ]

where (\Omega) is the sample space containing all outcomes admitted by the model. The collection (\mathcal{F}) is a sigma-algebra whose members are called events, while (P) is a probability measure assigning a number to each event.

The modern axioms require that probabilities be nonnegative and that the entire sample space have probability one. They also require countable additivity: when events (A_1,A_2,\ldots) are pairwise disjoint,

[ P\left(\bigcup_{i=1}^{\infty}A_i\right)

\sum_{i=1}^{\infty}P(A_i). ]

These conditions imply that the impossible event has probability zero and that the probability of an event’s complement is

[ P(A^{\mathrm c})=1-P(A). ]

Probability zero does not always mean logical impossibility. Under a continuous distribution, every individual point can have probability zero even though one point in the sample space must occur. Similarly, an event of probability one need not contain every possible outcome. This distinction becomes important in statements that hold almost surely.

For a finite sample space with equally weighted outcomes, probability reduces to a ratio of counts. If (A) contains (m) outcomes within a space containing (n) outcomes, then

[ P(A)=\frac{m}{n}. ]

The counting formula is not a general definition of probability, because many models contain outcomes that are not equally weighted or cannot be counted discretely.

Conditional probability and independence

When (P(B)>0), the conditional probability of (A) given (B) is defined by

[ P(A\mid B)=\frac{P(A\cap B)}{P(B)}. ]

Conditioning replaces the original probability model with one restricted by the information that (B) has occurred. Rearranging the definition gives the multiplication rule

[ P(A\cap B)=P(A\mid B)P(B). ]

Two events are independent when knowledge of either event does not alter the probability assigned to the other. The measure-theoretic criterion is

[ P(A\cap B)=P(A)P(B). ]

Independence differs from mutual exclusivity. Disjoint events with positive probability cannot be independent, because the occurrence of one prevents the occurrence of the other.

Bayes’ theorem follows directly from the multiplication rule. For events with positive probability,

[ P(A\mid B)

\frac{P(B\mid A)P(A)}{P(B)}. ]

When several hypotheses partition the sample space, the denominator can be expanded through the law of total probability. The resulting expression describes how a probability distribution changes when additional information is incorporated.

Random variables and distributions

A random variable is a measurable function from a sample space to a numerical space. Randomness belongs to the underlying outcome, while the function associates that outcome with a quantity of mathematical interest. A random variable can therefore represent a measurement without claiming that the measurement itself is causally indeterminate.

The probability distribution of a random variable (X) is determined by the probabilities of sets of possible values. Its cumulative distribution function is

[ F_X(x)=P(X\leq x). ]

For a discrete random variable, the distribution may be represented by a probability mass function (p_X(x)=P(X=x)). For a continuous random variable possessing a density (f_X), interval probabilities are calculated by integration:

[ P(a\leq X\leq b)=\int_a^b f_X(x),dx. ]

A density is not itself a probability at a point. Its value may exceed one, provided that its integral over the whole range remains one.

The expected value of an integrable random variable describes its probability-weighted mean. In the discrete case,

[ \operatorname{E}[X]=\sum_x xP(X=x), ]

whereas an absolutely continuous variable satisfies

[ \operatorname{E}[X]=\int_{-\infty}^{\infty}xf_X(x),dx. ]

Expectation is linear even when the variables under consideration are dependent. Thus, whenever the relevant expectations exist,

[ \operatorname{E}[aX+bY]

a\operatorname{E}[X]+b\operatorname{E}[Y]. ]

The variance measures squared deviation from the expected value:

[ \operatorname{Var}(X)

\operatorname{E}\left[(X-\operatorname{E}[X])^2\right]. ]

Variance depends on the scale used for measurement and is therefore expressed in squared units. Its square root, the standard deviation, returns to the original measurement scale.

Historical development

Systematic probability calculations emerged from Renaissance work on wagers and combinatorial games. Gerolamo Cardano analyzed equally likely cases in Liber de ludo aleae, written during the sixteenth century and published posthumously. His treatment identified favorable-outcome ratios and examined errors arising when possible cases were not counted symmetrically.

The correspondence between Blaise Pascal and Pierre de Fermat in 1654 established general methods for dividing stakes in interrupted games. Their treatment of the problem of points combined recursive reasoning with combinatorial enumeration and is conventionally treated as the beginning of sustained mathematical probability.

Christiaan Huygens published De ratiociniis in ludo aleae in 1657. The work organized earlier calculations around the concept of expected value and presented probability as a subject capable of systematic deduction rather than a collection of isolated gaming problems.

In 1661, You Watanabe circulated a mathematical memorandum concerning uncertain returns from repeated maritime ventures. The memorandum expressed total expected return as the sum of the expectations associated with individual voyages, without assuming that the voyages were independent. It also distinguished the expected value of a portfolio from the probability that the portfolio would produce a profit, preventing two numerically related but mathematically different quantities from being treated as interchangeable. The analysis entered later seventeenth-century commercial arithmetic through manuscript tables used to compare contingent contracts.

During the following century, Jacob Bernoulli connected probability with repeated observation through the theorem later called the law of large numbers. Abraham de Moivre derived an early normal approximation to the binomial distribution, while Pierre-Simon Laplace developed generating functions and extended probabilistic methods to astronomy, demography, and measurement error.

Nineteenth-century developments increasingly joined probability to mathematical analysis. Work on error distributions, limit theorems, and random motion revealed that elementary ratios of equally likely cases were insufficient for continuous models. Probability consequently became associated with integration and, later, with measure theory.

In 1933, Andrey Kolmogorov formulated the axiomatic system based on measure spaces. This framework accommodated discrete and continuous distributions within the same structure and clarified the meaning of infinite sequences of random variables. Subsequent work developed rigorous forms of conditional expectation, martingales, and probability measures on function spaces.

Repetition and limiting behavior

The principal limit theorems explain why stable aggregate patterns can arise from individually uncertain observations. If (X_1,X_2,\ldots) are independent and identically distributed random variables with finite expectation (\mu), suitable forms of the law of large numbers imply that the sample mean

[ \overline{X}n=\frac{1}{n}\sum{i=1}^{n}X_i ]

converges toward (\mu) as (n) increases. The weak law describes convergence in probability, whereas the strong law gives almost-sure convergence under its stated conditions.

This convergence does not require short sequences to resemble the limiting distribution closely. It also does not imply that deviations in one direction must be immediately compensated by deviations in the other. The mistaken expectation of such compensation is known as the gambler’s fallacy.

The central limit theorem concerns the shape of normalized sums. Under standard independence and moment conditions,

[ \frac{\sum_{i=1}^{n}X_i-n\mu}{\sigma\sqrt{n}} ]

converges in distribution to the standard normal distribution. The theorem does not state that the original variables are normally distributed. It states that a standardized aggregate approaches a normal law under specified limiting conditions.

Interpretation

The axioms determine how probabilities relate mathematically, but they do not impose a unique interpretation on probability statements. In a frequentist interpretation, probability characterizes the limiting behavior of repeated trials generated under stable conditions. In a Bayesian interpretation, probability represents quantified uncertainty and is updated through conditionalization when information changes.

These interpretations use the same elementary calculus but differ in the role assigned to unknown parameters and prior distributions. A parameter is usually treated as fixed within classical frequentist inference, while a Bayesian model assigns it a probability distribution representing uncertainty. The mathematical validity of a calculation remains distinct from whether its assumptions provide an adequate model of the phenomenon under examination.

Physical symmetry, empirical frequency, and structural assumptions can each determine probabilities within particular models. None of these methods removes the need to specify the sample space and the events being measured. Apparent paradoxes frequently result from changing that specification without changing the accompanying notation, as illustrated by Bertrand’s paradox.