Clustering and PCA

In the last chapter, you learned to protect evaluation from leakage when labels exist. Now a support team has embedded thousands of untagged return and delivery messages, and nobody has assigned reliable topics yet. Before you build search or train a classifier, you need to ask a quieter question: does this representation contain useful neighborhoods at all?

This chapter gives you a disciplined answer. You'll group a tiny unlabeled corpus with k-means, inspect it from fewer dimensions with principal component analysis (PCA), and run failure tests that stop a pretty plot from becoming a false claim. Those habits matter later when a retrieval-augmented generation system selects evidence for an LLM.

Six Messages Without Topic Labels

Suppose a text embedding model has already converted six customer messages into two-number vectors. Real embeddings might contain hundreds or thousands of coordinates. Two dimensions let us compute every important step by hand first.

The human-readable names are available here so you can check the geometry. Pretend the algorithm sees only coordinates. It doesn't receive refund, login, or shipping as labels.

The first diagnostic is simply nearest neighbors. For two points $a=(a_x,a_y)$ and $b=(b_x,b_y)$, Euclidean distance is:

$$d(a,b)=\sqrt{(a_x-b_x)^2 + (a_y-b_y)^2}$$

That formula asks how far apart two messages lie in the current representation. Run it before clustering so you know whether local pairs look sensible.

The intended local structure appears if each message's closest neighbor belongs to the same readable theme. That isn't a proof that all future messages will organize cleanly. It's a first representation check.

K-Means Turns Nearby Points Into Hypotheses

K-means is an unsupervised algorithm: it receives vectors and a requested number of groups, $k$, but no correct topic names. It repeats two operations:

1. Assignment: send every vector to its nearest centroid.
2. Update: replace each centroid with the mean of its assigned vectors.

The algorithm minimizes inertia, the within-cluster sum of squared distances:

$$J = \sum_{i=1}^{n} \left\lVert x_i - \mu_{z_i} \right\rVert_2^2$$

Here, $x_i$ is message vector $i$, $z_i$ is its assigned cluster, and $\mu_{z_i}$ is that cluster's centroid. Lower inertia means points lie closer to their selected centers. It doesn't mean the resulting groups deserve human topic names.^[1]

Start with $k=3$ and choose one seed from each visible area:

• $c_0=(1.0,1.0)$ from refund_A
• $c_1=(4.0,4.2)$ from login_A
• $c_2=(8.0,1.0)$ from shipping_A

After assignment, each pair stays together. Averaging coordinates moves the three centroids to:

Notice the wording: the algorithm found left, upper, and right groups. A reviewer reads sampled messages and proposes the topic labels afterward.

Build K-Means Instead of Hiding the Loop

A production implementation uses a tested library, but writing the small loop once makes later debugging far easier. The function below accepts a matrix of vectors and initial centers. It stops when the centers no longer move.

One defensive detail is missing on purpose: if a center gets no assigned vectors, its mean becomes undefined. Library implementations handle empty clusters and initialization more carefully. Your learning target here is the alternating assignment/update mechanism, not a full clustering package.

Choose k by Usefulness, Not by Wishful Naming

You chose $k=3$ because the map visibly contained three pairs. On a new corpus, $k$ isn't supplied by nature. A small audit combines several signals:

• Inertia: lower is better, but always falls as $k$ rises, eventually reaching zero when every point gets its own center.
• Silhouette score: for each point, compares cohesion within its assigned group against separation from neighboring groups. Higher is better, but it still judges geometry rather than business meaning.
• Sample review: read messages nearest each centroid and messages close to cluster boundaries.
• Stability: rerun with different seeds or resampled messages and check whether the interpretation survives.

The scikit-learn version below evaluates possible values for $k$.^[2] Setting n_init=20 makes the choice explicit: try several starting configurations and retain the lowest-inertia result for each $k$.

On this unusually neat six-point example, $k=3$ should separate the three visible pairs. On a real support corpus, no single metric gives permission to publish topic names. Read the examples.

Equal Scores Can Still Tell Different Stories

Even a stable-looking objective can hide ambiguity. Four messages positioned at the corners of a square have two equally valid two-cluster solutions: split left from right, or split top from bottom.

Both answers optimize the same squared-distance objective equally well. If each axis meant a different product concern, only human review or a downstream task could say which grouping is useful.

PCA Gives You a Smaller View

K-means assigns groups. Principal component analysis (PCA) does something different: it replaces the original coordinate axes with new perpendicular axes that preserve as much variance as possible in the first few dimensions.

First center the six vectors by subtracting their mean, $(4.7, 2.27)$. A centered coordinate describes how a message differs from the average message. Centering isn't scaling: subtracting a mean doesn't stop a high-unit column from dominating variance. Then factor the centered matrix using singular value decomposition (SVD):

$$X_{\text{centered}} = U \Sigma V^\top$$

The rows of $V^\top$ are principal directions. The first direction, PC1, captures the greatest remaining linear spread. Its share of total variance is the squared first singular value divided by the sum of squared singular values. Keeping the top components gives the best rank-limited reconstruction in squared-error terms, but it doesn't guarantee that the retained directions preserve a rare semantic distinction.^[1][3]

For our map, PC1 runs almost horizontally:

PCA has created a new feature, pc1. It didn't pick an existing embedding coordinate. In high-dimensional embeddings, each principal component is a weighted mixture of many original coordinates.

Measure What Compression Discards

Reconstruction makes the tradeoff concrete. Project onto one component, map back into the original space, and measure squared error. Two components can perfectly reconstruct this two-dimensional input; one component can't.

Explained variance measures geometry, not business value. A rare fraud or policy-escalation direction may carry little total variance and still matter greatly.

Failure Test 1: Mixed Scales Manufacture Clusters

Embeddings often appear next to metadata such as message length, order value, or age of the shipment. If you concatenate features with unlike numeric scales, both k-means and PCA can chase the largest units rather than the concept you care about.

The script below creates two intended topics (refund and shipping) while message length cuts across both. Without scaling, length dominates. Standardizing each column to comparable scale recovers the topic split in this constructed check.

This isn't a rule to standardize every embedding blindly. If a model's embedding vectors already come with a specified normalization and similarity contract, follow that contract. During evaluation, fit any learned scaler on the development corpus and reuse it for held-out examples. The rule is simpler: don't mix measurements and then pretend their scales are irrelevant.

Failure Test 2: The Similarity Rule Changes Neighbors

Text retrieval frequently compares vector direction with cosine similarity, while ordinary k-means minimizes Euclidean distance to centroids. Dot product additionally rewards magnitude. A model or index must use the similarity rule its representations were built for.

This tiny example gives a query two candidate policy snippets. One candidate points almost exactly in the query direction; the other has a much larger norm.

On unit-normalized vectors, cosine ranking and Euclidean nearest-neighbor ranking are directly related: maximizing cosine similarity minimizes squared Euclidean distance. That relation is useful, but only after normalization is explicit.

Failure Test 3: PCA Can Hide the Dimension Your Query Needs

A retrieval system doesn't need a globally beautiful map. It needs the right document for a specific question. Suppose horizontal variation across refund and shipping policies dominates a corpus, while the distinction between password-reset and billing pages sits mostly in a quieter vertical direction.

The following diagnostic projects documents to only one principal component, then runs a nearest-document lookup for a login question. The compressed view preserves most overall spread but collapses a distinction that mattered for this query.

After projection, billing_help, login_2fa, and warehouse_note land at the same one-dimensional coordinate. The failure isn't that billing became uniquely closest. PCA removed the quieter vertical distinction that separated the useful login page from two tied alternatives.

PCA is excellent for quick inspection and sometimes useful for compression. Before deploying compressed retrieval vectors, measure retrieval quality on labeled queries. In the next chapter, you'll build exactly those ranking checks.

Run a Representation Audit Before Shipping Search

An unsupervised plot is most useful when it produces testable questions. For a support or policy corpus, write an audit record before claiming that the embedding space is organized:

Use a handful of reviewed examples as an audit set, not as hidden training labels. It gives you a way to reject a broken representation before a downstream LLM begins quoting the wrong policy.

On this tiny teaching set, perfect purity is expected because the coordinates were designed to be readable. A production audit should be harder: sample boundary cases, unusual languages, policy updates, and short messages with ambiguous intent.

Practice Tasks

1. Change the initial centers in kmeans-from-scratch.py so one cluster becomes empty. Add a guard that either reseeds the empty center or raises a clear error.
2. Add one metadata column with a much larger numeric scale to the six-message matrix. Compare cluster assignments before and after deliberate scaling.
3. Keep two PCA components in compression-can-change-retrieval.py, then confirm the login query recovers login_2fa. Explain why explained variance alone couldn't guarantee that result.
4. Run write-a-cluster-audit-summary.py with one mislabeled or boundary message. Identify which samples you'd inspect before assigning topic names.
5. Write a representation-audit record for a small policy corpus: metric contract, normalization rule, cluster stability check, sampled neighbors, and held-out retrieval metric.

Practice guidance

1. mean(axis=0) on an empty slice produces invalid centers. A learning implementation should detect not np.any(labels == index) before averaging; a production library may reseed more carefully.
2. Without scaling, the large-unit metadata can dominate squared distance. After scaling, inspect whether recovered groups reflect useful semantics rather than assuming standardization is always correct.
3. With both components, this two-dimensional fixture is reconstructed exactly and login retrieval returns login_2fa. In a real index, compare held-out retrieval metrics before and after compression.
4. A single conflicting message should reduce purity or create a boundary case. Inspect centroid-near examples and edge cases; don't attach a permanent topic name from one clean run.
5. Record embedding model/version, similarity metric, normalization, sample strategy, stability runs, and before/after retrieval quality if compression is proposed.

Key Concepts

• unlabeled embeddings and representation inspection
• k-means assignments, centroids, and inertia
• choosing $k$ with geometry plus human audit
• PCA, SVD, projection, and reconstruction error
• explained variance versus semantic usefulness
• feature-scale and similarity-metric failures
• compression risk before retrieval

Evaluation rubric

• Foundational: Computes one centroid update and explains why a cluster integer isn't a topic label.
• Intermediate: Runs k-means and PCA diagnostics, then distinguishes explained variance from semantic evidence.
• Advanced: Designs an embedding audit that catches scale, metric, or compression failures before an LLM retrieval pipeline uses the vectors.

Follow-up questions

Common pitfalls