Clustering and PCA

Leakage checks protect labeled evaluation. Untagged data asks a quieter question. A developer-support team has embedded thousands of billing, login, and deploy-policy messages, and nobody has assigned reliable topics yet. Before you build search or train a classifier, you need to know whether the representation contains useful neighborhoods at all.

You'll group a tiny unlabeled corpus with k-means, inspect it from fewer dimensions with principal component analysis (PCA), and run failure tests that stop a pretty plot from becoming a false claim. Those habits matter later when a retrieval-augmented generation system selects evidence for an LLM.

Six messages without topic labels

Suppose a text embedding model has already converted six developer messages into two-number vectors. Real embeddings might contain hundreds or thousands of coordinates. Two dimensions let us compute every important step by hand first.

The human-readable names are available here so you can check the geometry. Pretend the algorithm sees only coordinates. It doesn't receive billing, login, or deploy as labels.

The first diagnostic is nearest neighbors. For two points $a=(a_x,a_y)$ and $b=(b_x,b_y)$, Euclidean distance is:

$$d(a,b)=\sqrt{(a_x-b_x)^2 + (a_y-b_y)^2}$$

That formula asks how far apart two messages lie in the current representation. Run it before clustering so you know whether local pairs look sensible.

The intended local structure appears if each message's closest neighbor belongs to the same readable theme. That isn't a proof that all future messages will organize cleanly. It's a first representation check.

K-means turns nearby points into hypotheses

K-means is an unsupervised algorithm: it receives vectors and a requested number of groups, $k$, but no correct topic names. It repeats two operations:

1. Assignment: send every vector to its nearest centroid.
2. Update: replace each centroid with the mean of its assigned vectors.

The algorithm minimizes inertia, the within-cluster sum of squared distances:

$$J = \sum_{i=1}^{n} \left\lVert x_i - \mu_{z_i} \right\rVert_2^2$$

Here, $x_i$ is message vector $i$, $z_i$ is its assigned cluster, and $\mu_{z_i}$ is that cluster's centroid. Lower inertia means points lie closer to their selected centers. It doesn't mean the resulting groups deserve human topic names.^{[1]Reference 1The Elements of Statistical Learning.https://hastie.su.domains/ElemStatLearn/}

Start with $k=3$ and choose one seed from each visible area:

• $c_0=(1.0,1.0)$ from billing_A
• $c_1=(4.0,4.2)$ from login_A
• $c_2=(8.0,1.0)$ from deploy_A

After assignment, each pair stays together. Averaging coordinates moves the three centroids to:

Notice the wording: the algorithm found left, upper, and right groups. A reviewer reads sampled messages and proposes the topic labels afterward.

Build k-means instead of hiding the loop

A production implementation uses a tested library, but writing the small loop once makes later debugging far easier. This function accepts a matrix of vectors and initial centers. It stops when the centers no longer move.

One defensive detail is missing on purpose: if a center gets no assigned vectors, its mean becomes undefined. Library implementations handle empty clusters and initialization more carefully. Focus on the alternating assignment/update mechanism, not a full clustering package.

Choose k by Usefulness, Not by Wishful Naming

You chose $k=3$ because the map visibly contained three pairs. On a new corpus, $k$ isn't supplied by nature. A small audit combines several signals:

• Inertia: lower is better, but always falls as $k$ rises, eventually reaching zero when every point gets its own center.
• Silhouette score: for each point, compares cohesion within its assigned group against separation from neighboring groups. Higher is better, but it still judges geometry rather than business meaning.
• Sample review: read messages nearest each centroid and messages close to cluster boundaries.
• Stability: rerun with different seeds or resampled messages and check whether the interpretation survives.

The scikit-learn version below evaluates possible values for $k$.^{[2]Reference 2Scikit-learn: Machine Learning in Python.https://jmlr.org/papers/v12/pedregosa11a.html} Setting n_init=20 makes the choice explicit: try several starting configurations and retain the lowest-inertia result for each $k$.

On this unusually neat six-point example, $k=3$ should separate the three visible pairs. On a real developer-message corpus, no single metric gives permission to publish topic names. Read the examples.

Equal scores can still tell different stories

Even a stable-looking objective can hide ambiguity. Four messages positioned at the corners of a square have two equally valid two-cluster solutions: split left from right, or split top from bottom.

Both answers optimize the same squared-distance objective equally well. If each axis meant a different product concern, only human review or a downstream task could say which grouping is useful.

PCA Gives You a Smaller View

K-means assigns groups. Principal component analysis (PCA) does something different: it replaces the original coordinate axes with new perpendicular axes that preserve as much variance as possible in the first few dimensions.

First center the six vectors by subtracting their mean, $(4.7, 2.27)$. A centered coordinate describes how a message differs from the average message. Centering isn't scaling: subtracting a mean doesn't stop a high-unit column from dominating variance. Then factor the centered matrix using singular value decomposition (SVD):

$$X_{\text{centered}} = U \Sigma V^\top$$

The rows of $V^\top$ are principal directions. The first direction, PC1, captures the greatest remaining linear spread. Its share of total variance is the squared first singular value divided by the sum of squared singular values. Keeping the top components gives the best rank-limited reconstruction in squared-error terms, but it doesn't guarantee that the retained directions preserve a rare semantic distinction.^{[1]Reference 1The Elements of Statistical Learning.https://hastie.su.domains/ElemStatLearn/[3]Reference 3The approximation of one matrix by another of lower rankhttps://doi.org/10.1007/BF02288367}

For our map, PC1 runs almost horizontally:

PCA has created a new feature, pc1. It didn't pick an existing embedding coordinate. In high-dimensional embeddings, each principal component is a weighted mixture of many original coordinates.

Measure what compression discards

Reconstruction makes the tradeoff concrete. Project onto one component, map back into the original space, and measure squared error. Two components can perfectly reconstruct this two-dimensional input; one component can't.

Explained variance measures geometry, not business value. A rare fraud or policy-escalation direction may carry little total variance and still matter greatly.

Failure test 1: mixed scales manufacture clusters

Embeddings often appear next to metadata such as message length, incident count, or time since the policy changed. If you concatenate features with unlike numeric scales, both k-means and PCA can chase the largest units rather than the concept you care about.

The script below creates two intended topics (billing and deploy) while message length cuts across both. Without scaling, length dominates. Standardizing each column to comparable scale recovers the topic split in this constructed check.

A negative adjusted Rand score means the partition disagrees with the intended topic split more than random labeling would. This isn't a rule to standardize every embedding blindly. If a model's embedding vectors already come with a specified normalization and similarity contract, follow that contract. During evaluation, fit any learned scaler on the development corpus and reuse it for held-out examples. The rule is simpler: don't mix measurements and then pretend their scales are irrelevant.

Failure test 2: the similarity rule changes neighbors

Text retrieval frequently compares vector direction with cosine similarity, while ordinary k-means minimizes Euclidean distance to centroids. Dot product also rewards magnitude. A model or index must use the similarity rule its representations were built for.

The retrieval check gives a query two candidate policy snippets. One candidate points almost exactly in the query direction; the other has a much larger norm.

On unit-normalized vectors, cosine ranking and Euclidean nearest-neighbor ranking are directly related: maximizing cosine similarity minimizes squared Euclidean distance. That relation is useful, but only after normalization is explicit.

That equivalence is about nearest neighbors, not about ordinary k-means. Standard k-means still minimizes squared Euclidean distance to means. The mean of several L2-normalized vectors is generally not unit-length, so cluster centers drift off the sphere your cosine index uses. If you cluster the same vectors you will cosine-search later, either:

• run spherical k-means (assign by cosine / max dot product on unit vectors, then re-normalize each centroid after the mean update), or
• treat L2-normalizing points, running Euclidean k-means, and re-normalizing only the final centroids before serving as a deployment heuristic, not as spherical k-means. Compare held-out cosine neighbor rankings or retrieval metrics against a spherical k-means baseline before keeping it.

Without that bridge, "cluster the embedding space" and "cosine retrieval" can disagree on neighborhood structure even when every vector started unit-norm.

K-means also prefers roughly spherical clusters of similar volume. Elongated topic clouds, nested subtopics, or one giant "misc" region can look poor under inertia or silhouette even when a human would split differently. Treat inertia as an optimization diagnostic, not as proof that topic geometry matches the product.

Centering changes cosine geometry

Cosine similarity measures the angle between two vectors from the origin:

$$\text{cosine\_similarity}(u, v) = \frac{u \cdot v}{\|u\| \|v\|}$$

A common pitfall is applying principal component analysis (PCA) to embeddings and assuming cosine retrieval keeps its old meaning. PCA centers the dataset by subtracting the column-wise mean from each vector, which shifts the origin to the center of the data cloud.

Shifting the origin alters angles between vectors, so original cosine distances and neighborhood relationships aren't generally preserved. Two vectors that point in nearly identical directions from the original origin can point in different directions after centering.

This check calculates cosine similarities before and after mean-centering:

If you must reduce dimensions for cosine retrieval, treat the reduced space as a new representation and re-validate held-out query rankings under the same cosine metric, rather than assuming the projection preserved the model's angles.

Failure test 3: PCA can hide the dimension your query needs

A retrieval system doesn't need a globally beautiful map. It needs the right document for a specific question. Suppose horizontal variation across billing and deploy policies dominates a corpus, while the distinction between password-reset and billing pages sits mostly in a quieter vertical direction.

This diagnostic projects documents to only one principal component, then runs a nearest-document lookup for a login question. The compressed view preserves most overall spread but collapses a distinction that mattered for this query.

After projection, billing_help, login_2fa, and runbook_note land at the same one-dimensional coordinate. The failure isn't that billing became uniquely closest. PCA removed the quieter vertical distinction that separated the useful login page from two tied alternatives.

PCA is excellent for quick inspection and sometimes useful for compression. Before deploying compressed retrieval vectors, measure retrieval quality on labeled queries. In the next chapter, you'll build exactly those ranking checks.

Run a Representation Audit Before Deploying Search

An unsupervised plot is most useful when it produces testable questions. For a developer-support or policy corpus, write an audit record before claiming that the embedding space is organized:

Use a handful of reviewed examples as an audit set, not as hidden training labels. The audit set can reject a broken representation before a downstream LLM begins quoting the wrong policy.

On this tiny teaching set, perfect purity is expected because the coordinates were designed to be readable. A production audit should be harder: sample boundary cases, unusual languages, policy updates, and short messages with ambiguous intent.

Practice tasks

1. Change the initial centers in kmeans-from-scratch.py so one cluster becomes empty. Add a guard that either reseeds the empty center or raises a clear error.
2. Add one metadata column with a much larger numeric scale to the six-message matrix. Compare cluster assignments before and after deliberate scaling.
3. Keep two PCA components in compression-can-change-retrieval.py, then confirm the login query recovers login_2fa. Explain why explained variance alone couldn't guarantee that result.
4. Run write-a-cluster-audit-summary.py with one mislabeled or boundary message. Identify which samples you'd inspect before assigning topic names.
5. Write a representation-audit record for a small policy corpus: metric contract, normalization rule, cluster stability check, sampled neighbors, and held-out retrieval metric.

Practice guidance

1. mean(axis=0) on an empty slice produces invalid centers. A learning implementation should detect not np.any(labels == index) before averaging; a production library may reseed more carefully.
2. Without scaling, the large-unit metadata can dominate squared distance. After scaling, inspect whether recovered groups reflect useful semantics rather than assuming standardization is always correct.
3. With both components, this two-dimensional fixture is reconstructed exactly and login retrieval returns login_2fa. In a real index, compare held-out retrieval metrics before and after compression.
4. A single conflicting message should reduce purity or create a boundary case. Inspect centroid-near examples and edge cases; don't attach a permanent topic name from one clean run.
5. Record embedding model/version, similarity metric, normalization, sample strategy, stability runs, and before/after retrieval quality if compression is proposed.