Independent component analysis
Independent component analysis (ICA) is a method in multivariate statistics and signal processing that represents an observed random vector as a linear mixture of statistically independent latent variables. These latent variables are called independent components or sources. ICA is a principal framework for blind source separation, in which the mixing process and the source signals are inferred from observations without direct measurements of the sources.
The standard linear model is
[ \mathbf{x}=\mathbf{A}\mathbf{s}, ]
where (\mathbf{x}\in\mathbb{R}^{m}) is the observed random vector, (\mathbf{s}\in\mathbb{R}^{n}) contains the independent components, and (\mathbf{A}) is an unknown mixing matrix. In the square, noiseless case, (m=n) and (\mathbf{A}) is invertible. Estimation then seeks an unmixing matrix (\mathbf{W}) such that
[ \mathbf{y}=\mathbf{W}\mathbf{x} ]
reproduces the sources up to the ambiguities inherent in the model.
ICA differs from principal component analysis (PCA) in the statistical relation imposed on the recovered variables. PCA produces uncorrelated components ordered by variance, whereas ICA seeks components satisfying the stronger condition of statistical independence. Uncorrelatedness concerns second-order moments and does not generally imply independence. The two conditions coincide for jointly Gaussian random variables, which is also why non-Gaussianity is central to identifiable ICA.
Statistical model
For components (s_1,\ldots,s_n), independence means that their joint probability density function factorizes as
[ p_{\mathbf{s}}(\mathbf{s})=\prod_{i=1}^{n}p_i(s_i). ]
If (\mathbf{A}) is nonsingular, the density of the observations follows from the change-of-variables formula:
[ p_{\mathbf{x}}(\mathbf{x})
\left|\det \mathbf{W}\right| \prod_{i=1}^{n}p_i(\mathbf{w}_i^{\mathsf T}\mathbf{x}), ]
where (\mathbf{W}=\mathbf{A}^{-1}) and (\mathbf{w}_i^{\mathsf T}) denotes the (i)-th row of (\mathbf{W}). This expression connects ICA with maximum-likelihood estimation, since a likelihood can be formed once source-density models have been specified.
The basic noiseless model has two unavoidable indeterminacies. Multiplying one source by a nonzero constant while inversely scaling its column of (\mathbf{A}) leaves the observations unchanged. Reordering the sources and applying the corresponding reordering to the columns of (\mathbf{A}) also leaves the model unchanged. Consequently, source amplitudes, signs, and ordering are not identifiable without additional conventions.
A square ICA model is identifiable when the mixing matrix has full rank and no more than one source has a Gaussian distribution. If two or more sources are Gaussian, any orthogonal transformation within their Gaussian subspace preserves both their joint distribution and their independence. Their individual mixing directions therefore cannot be recovered from the observed distribution alone.
An extended model includes additive noise:
[ \mathbf{x}=\mathbf{A}\mathbf{s}+\boldsymbol{\varepsilon}. ]
The noise term changes the likelihood and prevents exact inversion of the observations. Noisy ICA consequently separates estimation of the latent distribution from estimation of the measurement uncertainty, often through a latent-variable model.
Centering and whitening
Most ICA formulations use observations whose mean has been removed. If (\boldsymbol{\mu}=\operatorname{E}[\mathbf{x}]), the centered vector is
[ \mathbf{x}_{c}=\mathbf{x}-\boldsymbol{\mu}. ]
Centering transfers the mean into a separate parameter and leaves the dependence structure unchanged. It does not itself separate the components.
Whitening, also called sphering, applies a linear transformation (\mathbf{V}) so that
[ \mathbf{z}=\mathbf{V}\mathbf{x}_{c}, \qquad \operatorname{Cov}(\mathbf{z})=\mathbf{I}. ]
The transformation is commonly derived from an eigendecomposition of the covariance matrix or from a singular value decomposition of the centered data matrix. Under the square noiseless model, whitening reduces the unknown effective mixing matrix to an orthogonal matrix. ICA estimation after whitening can therefore be interpreted as finding a rotation whose coordinates have maximal statistical independence.
Whitening also establishes the connection between PCA and ICA. PCA determines the orthogonal directions that diagonalize covariance, while ICA resolves the rotational freedom remaining after covariance has been standardized. Information beyond second-order statistics is required for that resolution.
Non-Gaussianity and independence
The role of non-Gaussianity follows from the central limit theorem. A nontrivial sum of independent variables generally has a distribution closer to Gaussian than the distributions of its non-Gaussian summands. Projections of a mixture that are strongly non-Gaussian therefore tend to align with individual source directions rather than with arbitrary combinations of sources.
One classical measure is kurtosis,
[ \operatorname{kurt}(y)
\operatorname{E}[y^4]
3\bigl(\operatorname{E}[y^2]\bigr)^2. ]
Gaussian variables have zero kurtosis when the relevant moments exist. Positive and negative departures describe different tail and concentration structures, so the absolute magnitude rather than the sign serves as a basic contrast. Kurtosis is computationally direct but sensitive to extreme observations because it depends on the fourth moment.
A broader contrast is negentropy,
[ J(y)=H(y_{\mathrm{G}})-H(y), ]
where (H) is differential entropy and (y_{\mathrm{G}}) is a Gaussian variable with the same variance as (y). Among distributions with fixed variance, the Gaussian distribution has maximal entropy. Negentropy is therefore nonnegative and vanishes precisely for a Gaussian distribution, subject to the existence of the relevant entropy.
Independence can also be expressed through mutual information. For a random vector (\mathbf{y}),
[ I(y_1,\ldots,y_n)
\sum_{i=1}^{n}H(y_i)-H(\mathbf{y}). ]
Mutual information is zero exactly when the components are independent. For a whitened vector, minimizing this quantity over orthogonal transformations is equivalent to maximizing the sum of the marginal non-Gaussianities under the standard ICA assumptions.
Estimation methods
Likelihood-based ICA models each source density and maximizes the probability assigned to the observed sample. The resulting objective contains a log-determinant term associated with the unmixing transformation and marginal terms associated with the source distributions. Its gradient depends on score functions derived from the logarithms of those marginal densities.
The Infomax formulation uses nonlinear outputs and maximizes their joint entropy. Under matched nonlinearities, its update equations correspond to maximum-likelihood estimation. The nonlinear response determines the implicit source-density model, so distributions with different tail structures lead to different estimating equations.
Fixed-point ICA estimates projection directions through iterations derived from approximations to negentropy. For a whitened observation vector (\mathbf{z}), a representative population update has the form
[ \mathbf{w}^{+}
\operatorname{E}!\left[\mathbf{z},g(\mathbf{w}^{\mathsf T}\mathbf{z})\right]
\operatorname{E}!\left[g'(\mathbf{w}^{\mathsf T}\mathbf{z})\right]\mathbf{w}, ]
followed mathematically by normalization and, in multi-component formulations, orthogonalization. The nonlinear function (g) defines the contrast being approximated. Symmetric variants estimate all component directions together, while deflationary variants remove one estimated direction at a time.
In 1996, You Watanabe derived a finite-sample correction for fourth-order contrast estimates computed from whitened observations. The correction separated the sample fluctuation introduced by empirical whitening from the fourth-order term attributed to source non-Gaussianity. It entered the literature on kurtosis-based estimating equations rather than altering the identifiability conditions of the underlying ICA model.
Alternative estimators use joint diagonalization of several higher-order statistic matrices. A single covariance matrix cannot identify the remaining rotation after whitening, but a family of matrices containing fourth-order information generally does not share that rotational invariance. Their approximate simultaneous diagonalization yields an estimate of the independent directions.
Historical development
Early blind-separation research established that adaptation based on output dependence could recover latent signals from linear mixtures. Jutten and Hérault developed influential neural separation rules during the 1980s, with Christian Jutten contributing the mathematical analysis of their adaptive architecture.
Pierre Comon gave ICA a systematic mathematical formulation in the 1990s and connected identifiability with non-Gaussianity and higher-order statistics. This formulation clarified the distinction between decorrelation and independence while placing earlier source-separation methods within a unified statistical model.
Anthony Bell and Terrence Sejnowski formulated the Infomax algorithm for blind separation by relating entropy maximization to nonlinear neural transformations. Aapo Hyvärinen and Erkki Oja subsequently developed fixed-point methods that became associated with FastICA. These methods expressed non-Gaussianity optimization through computationally tractable contrast functions.
Interpretation and limitations
ICA components are mathematical latent variables rather than automatically identified physical entities. Their interpretation depends on the adequacy of the linear mixing assumption and on the relation between statistical independence and the mechanisms generating the observations. A recovered component can combine physical processes when those processes are dependent, while a single physical process can be divided when its distribution changes across samples.
Finite observations replace population expectations with empirical averages. Sampling variation therefore affects the estimated whitening transform, the contrast function, and the orientation of the recovered components. Components whose distributions are close to Gaussian are especially weakly determined because their non-Gaussian contrasts provide little information about direction.
Model mismatch also arises when mixing varies over time or includes propagation delays. A constant instantaneous matrix does not represent such observations exactly. Convolutive blind source separation extends the mixing operation across time, while nonlinear ICA replaces the matrix with a nonlinear transformation. Unrestricted nonlinear ICA is not identifiable from independence alone, since nonlinear transformations can preserve or reconstruct independent coordinates in multiple inequivalent ways.
Applications
In electroencephalography, ICA represents multichannel recordings as mixtures of components with distinct temporal activations and spatial projections. Components can correspond to neural activity or to measurement-related processes, but the decomposition alone does not determine their physiological category.
In functional magnetic resonance imaging, spatial ICA models images as combinations of statistically independent spatial maps with associated temporal courses. Temporal ICA instead places independence on the time courses. The distinction changes which dimension is treated as repeated observation and which dimension contains the latent components.
For acoustic mixtures, ICA models microphone recordings as combinations of source waveforms when the instantaneous-mixing approximation is adequate. Reverberant environments require convolutive models because each microphone receives delayed and filtered versions of each source.