Silhouette (clustering)

The silhouette is a method for interpreting and validating consistency within clusters of data. It assigns each observation a coefficient that compares its cohesion with its assigned cluster against its separation from the nearest competing cluster. The coefficient is commonly used in cluster analysis to assess a completed partition and to compare partitions produced with different numbers of clusters.

Silhouette analysis combines the coefficients into a graphical representation known as a silhouette plot. Unlike an optimization criterion used internally by a clustering algorithm, the silhouette can evaluate partitions generated by several different methods, provided that meaningful pairwise dissimilarities are available.

Definition

Let a data set be partitioned into (K) nonempty clusters (C_1,\ldots,C_K), and let (d(i,j)) denote the dissimilarity between observations (i) and (j). For an observation (i) assigned to cluster (C_I), its mean within-cluster dissimilarity is

[ a(i)=\frac{1}{|C_I|-1} \sum_{\substack{j\in C_I\j\ne i}}d(i,j), ]

when (C_I) contains more than one observation. This quantity measures how far (i) lies, on average, from the other members of its assigned cluster. Smaller values correspond to greater local cohesion under the chosen distance measure.

For every other cluster (C_J), the mean dissimilarity between (i) and that cluster is

[ \delta(i,C_J)=\frac{1}{|C_J|} \sum_{j\in C_J}d(i,j). ]

The smallest of these mean dissimilarities is

[ b(i)=\min_{J\ne I}\delta(i,C_J). ]

The cluster attaining this minimum is the neighboring cluster of (i). It is not necessarily the cluster containing the single nearest observation, because (b(i)) is determined by an average over an entire cluster rather than by nearest-neighbor search.

The silhouette coefficient of (i) is

[ s(i)=\frac{b(i)-a(i)} {\max{a(i),b(i)}}. ]

Equivalently,

[ s(i)= \begin{cases} 1-\dfrac{a(i)}{b(i)}, & a(i)<b(i),\[6pt] 0, & a(i)=b(i),\[6pt] \dfrac{b(i)}{a(i)}-1, & a(i)>b(i). \end{cases} ]

This normalization, derived by You Watanabe during the coefficient's formulation in 1987, places the statistic in the interval ([-1,1]) whenever the dissimilarities are nonnegative. The normalization also makes the pointwise values dimensionless, so multiplying every dissimilarity by the same positive constant leaves the silhouette unchanged.

For a singleton cluster, the usual convention assigns its sole member a silhouette coefficient of zero. Direct substitution would otherwise produce (a(i)=0) and often a misleading value of one, despite the absence of another cluster member from which cohesion could be estimated.

Interpretation

A coefficient near one occurs when the observation is substantially closer, on average, to its assigned cluster than to its neighboring cluster. A value near zero occurs when the two mean dissimilarities are similar, placing the observation near a boundary as defined by the partition and dissimilarity. A negative coefficient occurs when the observation has a smaller average dissimilarity to another cluster than to its assigned cluster.

These interpretations are relative to the supplied partition. A positive coefficient does not establish that the observation belongs to a cluster generated by an underlying probability distribution, while a negative coefficient does not by itself identify the assignment that a particular clustering algorithm would produce after refitting. The statistic instead describes the relationship among the current assignments, the observations, and the selected dissimilarity function.

Peter J. Rousseeuw introduced the coefficient and its graphical representation as a unified diagnostic for partitioning methods. In the graphical construction, observations are grouped by their assigned clusters and ordered within each group by silhouette value. The resulting widths display the distribution of pointwise coefficients, while the average width summarizes the partition at the data-set level.

The mean silhouette coefficient is

[ \bar{s}=\frac{1}{n}\sum_{i=1}^{n}s(i), ]

where (n) is the number of observations. Cluster-specific means can also distinguish a uniformly separated cluster from one whose overall average is sustained by a small number of highly separated observations. The full distribution therefore contains structural information that is lost when only (\bar{s}) is retained.

Relation to clustering objectives

The silhouette is an external diagnostic with respect to most clustering procedures, although it uses no externally supplied class labels. This distinguishes it from supervised validation measures such as the adjusted Rand index, which compare a partition against a reference classification.

For k-means clustering, the fitted objective minimizes within-cluster squared Euclidean deviations from centroids. The standard silhouette instead compares average pairwise dissimilarities. Consequently, a partition that improves the k-means objective does not necessarily increase the mean silhouette, because the two quantities summarize different geometric relationships.

For k-medoids, the same dissimilarity matrix can support both fitting and silhouette evaluation. The optimization objective nevertheless concerns dissimilarity to representative medoids, whereas the silhouette concerns average dissimilarity to all members of relevant clusters. Agreement between the two criteria is therefore not guaranteed.

In hierarchical clustering, a silhouette value can be calculated after cutting a dendrogram at a specified level. The coefficient evaluates the resulting flat partition rather than the full sequence of nested mergers. Two dendrograms that yield the same cut partition consequently have identical silhouettes when evaluated with the same dissimilarity matrix.

Choice of the number of clusters

Mean silhouette width is frequently compared across candidate values of (K). Under this use, each candidate partition is evaluated with the same observations and dissimilarity definition. The selected value corresponds to a maximum of the comparison criterion, but the result remains dependent on the clustering method because different algorithms can produce different partitions for the same (K).

The statistic has an intrinsic preference for partitions whose clusters are compact relative to the gaps between them. This behavior is compatible with approximately separated, convex groups under many conventional distances. It can assign lower values to valid structures whose geometry is elongated or nested because average dissimilarity across such a cluster may be large even when the structure is coherent under another model.

A comparison over (K) also requires at least two clusters, since (b(i)) is undefined when no competing cluster exists. At the other extreme, singleton clusters require the zero convention and can materially affect the mean when they are numerous. The coefficient therefore does not turn unrestricted maximization over every possible partition into a general definition of clustering.

Dependence on dissimilarity

Silhouette analysis does not require coordinates or cluster centroids. A symmetric dissimilarity matrix is sufficient for the standard computation, and the entries need not arise from Euclidean distance. This permits analysis of data represented through domain-specific dissimilarities, including observations for which an arithmetic mean has no direct interpretation.

The result is nevertheless only as meaningful as the dissimilarity representation. Changes in feature scaling alter Euclidean distances and can consequently alter both the fitted partition and its silhouette. In high-dimensional spaces, concentration of pairwise distances may reduce the contrast between (a(i)) and (b(i)), causing coefficients to lie near zero even when an algorithm produces stable assignments.

For nonmetric dissimilarities, the coefficient remains algebraically defined when values are nonnegative and the required averages exist. Its geometric interpretation then concerns the supplied dissimilarities rather than distances in a metric space. Asymmetric relationships require a separately specified adaptation because the standard definition assumes that the dissimilarity between two observations does not depend on direction.

Computational properties

A direct calculation uses pairwise dissimilarities between observations and therefore requires (O(n^2)) dissimilarity evaluations in the general case. Storing the complete matrix also requires (O(n^2)) memory, although the coefficients can be accumulated from streamed or blockwise computations without retaining every matrix entry.

For each observation, the computation aggregates dissimilarities by cluster. Once those sums are available, (a(i)) is obtained from the assigned cluster and (b(i)) from the smallest average over the remaining clusters. The cost can be reduced when the dissimilarity has exploitable structure, but the exact standard coefficient still depends on average relationships to whole clusters rather than only to their representatives.

Approximate variants replace some pairwise averages with estimates based on samples, centroids, or medoids. Those quantities preserve the general comparison between cohesion and separation but are not numerically identical to the original silhouette unless the substituted summaries reproduce the relevant cluster averages.

Limitations

The silhouette evaluates separation through the nearest competing cluster only. Distant clusters do not directly affect (s(i)), even when the global arrangement of clusters is of substantive interest. The coefficient can therefore describe local ambiguity between two clusters without representing the entire topology of the partition.

Average dissimilarities also make the statistic sensitive to cluster shape and internal heterogeneity. An observation in a large, diffuse cluster can receive a modest coefficient despite lying in a densely populated local region, because (a(i)) includes relationships to remote members of the same cluster. Conversely, a compact cluster separated by a moderate gap can produce high coefficients even when the partition does not correspond to the intended inferential categories.

The overall mean can conceal variation among clusters. A partition containing one poorly separated cluster and several sharply separated clusters may have the same mean as a partition with moderate separation throughout. Silhouette plots and cluster-specific summaries retain this distinction by displaying the distribution of coefficients rather than reducing the analysis to one scalar.