Evaluating Clustering: Silhouettes, Stability, External Labels and Useful Partitions

Begin with the intended claim

Clustering can compress vectors, identify operating regimes, organise documents or suggest biological populations. Success differs across these tasks. A partition that minimises squared reconstruction error may be excellent compression and poor evidence for distinct populations.

Write down what a useful result would enable before examining a coloured plot. Specify the representation, distance, treatment of noise and whether every observation must be assigned. If external labels exist, decide whether they represent the desired grouping or merely one convenient annotation.

Why training inertia cannot choose k

For k-means, adding centres cannot worsen the globally optimal training objective. At k=n, every observation can have its own centre and zero error. Therefore the smallest training inertia is not a useful selection rule by itself.

An elbow searches for diminishing returns, but a smooth continuum may have no natural kink. Separate local optimisation runs can also produce an irregular curve because some runs find poorer minima. Use restarts and distinguish optimisation variation from genuine changes in representational capacity.

Held-out quantisation error measures how well a codebook compresses new observations. It is valuable when compression is the goal, but still does not prove that its cells are natural populations.

Calculate a silhouette

For an observation i, let a(i) be mean distance to other members of its own cluster and b(i) the smallest mean distance to another cluster. Then

Take clusters \(A=\{0,1\}\) and \(B=\{5,6\}\) on the number line. For point 1, a=1 and b=4.5, so \(s=3.5/4.5\approx0.778\). It is much closer on average to its own group.

Singleton clusters need a stated convention; common implementations assign silhouette zero. A partition with only one cluster has no alternative group for b. Noise requires a policy too: treating all noise as one cluster and excluding it answer different questions.

Silhouette rewards compact separated groups under the selected distance. It can penalise valid curved density-connected groups because distant points on the same curve inflate a. The score is therefore not a neutral judge between k-means and DBSCAN.

External agreement must ignore label names

Cluster numbers are arbitrary. Predictions labelled 0,0,1,1 and 7,7,3,3 describe the same partition. A naive classification accuracy can therefore be meaningless unless labels are aligned, and alignment itself can conceal split or merge errors.

Pair-based measures ask whether two observations are together in both partitions or separate in both. Adjusted Rand index corrects agreement relative to a random-partition baseline with fixed cluster sizes. With contingency counts \(n_{ij}\), row totals \(a_i\) and column totals \(b_j\), define \(A=\sum_{ij}\binom{n_{ij}}2\), \(B=\sum_i\binom{a_i}2\), \(C=\sum_j\binom{b_j}2\) and \(T=\binom n2\):

For truth \(\{1,2\},\{3,4\}\) and prediction \(\{1,3\},\{2,4\}\), every contingency cell is 1. Thus A=0, B=C=2, T=6, and ARI=−0.5. Below-zero values are possible; chance adjustment is not a transformation forcing every result into [0,1]. Degenerate partitions need explicit conventions.

Stability tests sensitivity

Resample observations, perturb plausible measurement noise, change initialisation and vary reasonable preprocessing choices. Compare partitions on a shared set using a permutation-invariant measure. If separately fitted models support prediction, evaluate them on the same held-out observations; otherwise define overlap and correspondence carefully.

Stability is informative but insufficient. A single-cluster method can be very stable. So can an algorithm consistently splitting on an irrelevant feature. Distinguish stability across seeds from stability across new data, because the first probes optimisation and the second probes sampling uncertainty.

For hierarchical methods, compare meaningful cuts or branch relationships. For density methods, report changes in the assigned population as well as agreement among the points both runs retained. Dropping difficult noise points can artificially improve a score.

Visualisation is a diagnostic, not a verdict

A t-SNE or UMAP plot can help locate examples for inspection. The embedding itself changes neighbourhoods, density and distances, so islands on the plot are not independent validation of clusters fitted on that same plot.

Inspect original-feature summaries, representative observations and boundary cases. Check whether duplicates, batch effects, missingness or source identifiers explain the groups. A visually strong separation caused by acquisition equipment may be real geometry and still irrelevant to the scientific question.

Report a comparison that can be reproduced

Keep the same preprocessing and evaluation population when comparing algorithms, or state the differences. Report parameter search, computational budget, noise fraction, cluster sizes, several seeds and relevant uncertainty. Selecting the best of many configurations on one score can overfit that evaluation criterion.

A useful report includes a compactness measure where appropriate, an external or application-based assessment when available, stability results and qualitative inspection. Explain disagreements among these signals rather than compressing them into one unsupported ranking of algorithms.

References

1. scikit-learn. Clustering performance evaluation.
2. James, Witten, Hastie, Tibshirani and Taylor. An Introduction to Statistical Learning with Applications in Python. Springer, 2023.