Linear Algebra for ML

Access-ticket features can already be arranged into tensors and projected through a matrix. A harder question comes next: if two feature columns mostly repeat the same pattern, do you need to keep both?

That question appears everywhere in machine learning. An embedding table can contain redundant directions. A learned update can be almost low-rank. A data matrix can have a weak direction that makes a numerical solve unreliable. Singular value decomposition (SVD) gives you a precise way to find those directions, measure their strengths, and decide what to keep.

A matrix can hide repeated directions

Use a tiny matrix with the same named-axis discipline as the previous lesson. Each row is one ticket. The two columns record whether the ticket contains a latency signal or a rollback signal:

$$A = \begin{bmatrix} 1 & 0 \\ 0 & 1 \\ 1 & 1 \end{bmatrix}$$

The third ticket contains both signals. That makes the combined direction [latency + rollback] stronger than the contrast direction [latency - rollback].

To compare directions fairly, give each one length 1:

$$v_{\text{common}} = \frac{1}{\sqrt{2}} \begin{bmatrix}1 \\ 1\end{bmatrix}, \qquad v_{\text{contrast}} = \frac{1}{\sqrt{2}} \begin{bmatrix}1 \\ -1\end{bmatrix}$$

Multiply A by either direction. The result tells you how strongly each ticket activates that pattern; its length tells you the pattern's total strength in the matrix.

The common direction has strength $\sqrt{3}$; the contrast direction has strength $1$. The third row reinforces the common direction and cancels in the contrast direction. Nothing is compressed yet. You're only measuring which feature combinations the rows support most strongly.

The Gram matrix reveals those directions by hand

For a data matrix A, multiply it by its transpose on the feature side:

$$G = A^T A = \begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix}$$

G has shape [2, 2] because it compares feature directions with feature directions. A direction v that obeys $Gv = \lambda v$ is an eigenvector of G: G scales it without turning it. Its eigenvalue $\lambda$ is squared strength, because

$$\lVert A v \rVert^2 = v^T A^T A v = \lambda$$

when v has length 1.

For this matrix:

$$G \begin{bmatrix}1 \\ 1\end{bmatrix} = 3 \begin{bmatrix}1 \\ 1\end{bmatrix}, \qquad G \begin{bmatrix}1 \\ -1\end{bmatrix} = 1 \begin{bmatrix}1 \\ -1\end{bmatrix}$$

SVD names the directions and strengths

Every real matrix has a singular value decomposition:^{[1]Reference 1Numerical Linear Algebra.https://people.maths.ox.ac.uk/trefethen/books.html}

$$A = U \Sigma V^T$$

For the incident matrix A with shape [3, 2], use the compact (or reduced) form. It keeps only the two singular directions that can fit in a matrix with two feature columns:

Our right singular vectors are the normalized common and contrast directions. Our singular values are:

$$\sigma_1 = \sqrt{3}, \qquad \sigma_2 = 1$$

The corresponding left singular vectors come from $u_i = A v_i / \sigma_i$. Put the factors back together and they reconstruct every original entry.

Use the direct SVD routine for computation. It returns U, the one-dimensional singular-value array S, and Vt, which is $V^T$.

The sign of a singular vector may differ between correct implementations. If both matching directions in U and V flip sign, their product stays the same. Compare singular values and reconstruction, not raw signs.

Rank counts independent patterns

The rank of a matrix is the number of non-zero singular values. It counts how many independent directions the matrix really needs.

Suppose every rollback feature exactly repeats the latency feature:

$$B = \begin{bmatrix} 1 & 1 \\ 0 & 0 \\ 1 & 1 \end{bmatrix}$$

The second column adds no new information. B has two columns, but only one independent direction, so its rank is 1.

A feature table with rank below its column count deserves inspection. Sometimes it contains a deliberate encoding. Sometimes two features duplicate each other and waste compute. Rank exposes the question; domain meaning answers it.

Exact rank is a mathematical definition. Floating-point data rarely produces a perfect zero, so np.linalg.matrix_rank treats sufficiently small singular values as zero using a tolerance. Measurement noise or downstream requirements may justify a different cutoff.

Truncation trades detail for a smaller matrix

Return to the full-rank incident matrix A. Keeping both singular directions reconstructs it exactly. Keeping only the strongest direction creates a rank-one approximation:

$$A_1 = U_{:,1} \sigma_1 V_{:,1}^T$$

Among all rank-one matrices, this approximation has the smallest Frobenius reconstruction error: the square root of the sum of squared entry errors.^{[2]Reference 2The approximation of one matrix by another of lower rankhttps://doi.org/10.1007/BF02288367} Equivalently, it minimizes that squared sum. The guarantee is about reconstructing A. It doesn't promise that a downstream classifier or retriever keeps its quality.

Here the discarded direction has strength 1, so the rank-one approximation loses meaningful contrast between a latency-only ticket and a rollback-only ticket. A small matrix makes that cost visible before you try the same idea on embeddings.

PCA is SVD after centering each feature

So far the matrix contained signal counts and we kept its origin at zero. Principal component analysis (PCA) answers a different question: which directions describe variation around the average row?

That means centering first. Given a matrix X of tickets by numeric features:

$$X_{\text{centered}} = X - \text{column means}$$

Then SVD on the centered matrix gives principal-component directions in V. If you skip centering, a large average offset can dominate the first direction even when it doesn't describe variation between tickets.

This leading direction points roughly along [delay evidence + escalation evidence], because those two signals rise together in the sample. PCA uses variance, not meaning. A direction that explains a lot of variation still needs human interpretation and task evaluation.

Condition number warns about solving through weak directions

SVD also reveals a failure mode. If a matrix stretches one direction strongly and almost erases another, an operation that tries to reverse that transformation must amplify the weak direction.

The 2-norm condition number is:

$$\kappa_2(A) = \frac{\sigma_{\max}}{\sigma_{\min}}$$

when the smallest singular value is non-zero. A condition number near 1 means all directions have comparable scales. A large condition number warns that solving, inverting, whitening (rescaling projected components toward unit variance), or other operations that divide by small singular values can magnify perturbations.^{[1]Reference 1Numerical Linear Algebra.https://people.maths.ox.ac.uk/trefethen/books.html}

Notice the scope of the claim. Merely storing vectors in a matrix with a weak direction doesn't automatically make cosine or dot-product retrieval unstable. The amplification appears when a pipeline performs an inverse-like operation, such as a solve or whitening transform, along that weak direction.

Use the Gram matrix for insight, not as a numerical shortcut

A.T @ A made our hand calculation easy, but it isn't the safe general implementation of SVD. Its condition number is squared:

$$\kappa_2(A^T A) = \kappa_2(A)^2$$

for a full-column-rank matrix. A matrix with condition number 10,000 turns into a Gram matrix with condition number 100,000,000. That can discard useful precision before the decomposition even begins. Use direct SVD implementations in numerical code; keep A.T @ A as a teaching device for the relationship between eigenvectors and singular vectors.

Choosing a cutoff uses squared singular values

For centered data, the squared singular values are proportional to how much variation each principal direction explains. For an uncentered count or embedding matrix, they still measure Frobenius energy captured by each direction. In either case, the cumulative ratio gives a transparent reconstruction-based cutoff, where r is the number of singular values:

$$\text{retained ratio at } k = \frac{\sigma_1^2 + \cdots + \sigma_k^2} {\sigma_1^2 + \cdots + \sigma_r^2}$$

This rule answers a reconstruction question. For a retrieval system, follow it with ranking metrics on held-out queries rather than assuming that matrix energy equals relevance.

Build it: compress a tiny retrieval matrix and check rankings

Represent four incident tickets with three term features: latency, rollback, and trace. The query is mostly about a latency trace with a smaller rollback signal. Project both tickets and query onto the top two right-singular directions, then compare rankings before and after compression with cosine similarity. Cosine similarity divides each dot product by both vector lengths, so the score reflects direction rather than raw magnitude.

For this tiny fixture, compression drops a real tail direction and changes scores while preserving the ranking. The explicit norm guard matters too: cosine similarity is undefined for a zero-length query or ticket vector. A production pipeline should reject, quarantine, or replace that vector instead of returning NaN scores.

This is a useful sanity check, not a credible evaluation. On a real retrieval set, measure Recall@k, the fraction of relevant tickets recovered in the top k returned candidates, and inspect failures before accepting reduced dimensions.

Where these ideas return in LLM engineering

LoRA doesn't compute the SVD of a full trained update during fine-tuning. It learns thin factors directly. SVD gives you the language to understand what "low rank" means; later training lessons explain how those factors are optimized.

Practice with measured answers

1. For $C = \begin{bmatrix}3 & 1 \\ 1 & 3\end{bmatrix}$, compute $C^T C$, its two eigenvalues, and its singular values. Then compute the condition number.
2. Modify the retrieval build exercise to keep only one singular direction. Did the first result stay the same? How large did the maximum score change become?
3. Add a large constant offset [100, 100] to every row of the PCA example. Compare SVD before centering and after centering. Which one describes differences between rows?

Answers to check:

1. $C^T C = \begin{bmatrix}10 & 6 \\ 6 & 10\end{bmatrix}$, eigenvalues are 16 and 4, singular values are 4 and 2, and $\kappa_2(C) = 2$.
2. A one-direction representation throws away the latency-versus-rollback contrast. With this fixture, every compressed cosine score becomes 1.0, so the ranking stops being meaningful. The exact code above returns order [3, 2, 1, 0], doesn't preserve the first result, and reports a maximum score change of 0.7113.
3. Uncentered SVD is pulled toward the large offset. Centered SVD removes that shared mean and recovers the variation direction.

Mastery check

Key concepts

• SVD as feature directions in $V$, strengths in $\Sigma$, and row participation in $U$
• rank as the number of independent non-zero singular directions
• truncated SVD as best fixed-rank reconstruction, not automatic downstream success
• PCA as SVD of a centered data matrix
• condition number as a warning for inverse-like operations along weak directions
• direct SVD as the computational default instead of manually decomposing $A^T A$

Evaluation rubric

• Foundational: Starting from the three-incident matrix, computes $A^T A$, identifies common and contrast directions, and explains why the singular values are $\sqrt{3}$ and 1.
• Intermediate: Uses NumPy SVD to choose a low-rank basis, compresses both tickets and query, and reports a retrieval metric before accepting the reduced representation.
• Advanced: Builds an ill-conditioned matrix, demonstrates amplification during a solve or whitening-like operation, and explains why forming $A^T A$ worsens the numerical problem.

Follow-up questions

Common pitfalls

• Symptom: PCA's first direction represents a shared baseline rather than differences among rows. Cause: The data wasn't centered. Fix: Subtract feature means before applying the PCA interpretation.
• Symptom: A hand-built SVD becomes unreliable on near-dependent columns. Cause: It decomposes A.T @ A, squaring the condition number. Fix: Use a direct SVD routine for computation.
• Symptom: Compression has low reconstruction error but misses important tickets. Cause: The cutoff optimized matrix energy instead of relevance. Fix: Measure retrieval metrics after projection.
• Symptom: Similarity scores become NaN after projection. Cause: A query or ticket vector has zero length, so cosine similarity is undefined. Fix: Guard norms before division and define a reject, quarantine, or replacement policy.
• Symptom: Engineers read columns of U as embedding feature axes. Cause: Row and feature spaces were confused. Fix: Use V for feature directions and U for row participation.