Linear Algebra for ML

Access-ticket features can already be arranged into tensors and projected through a matrix. A harder question comes next: if two feature columns mostly repeat the same pattern, do you need to keep both?

That question appears everywhere in machine learning. An embedding table can contain redundant directions. A learned update can be almost low-rank. A data matrix can have a weak direction that makes a numerical solve unreliable. Singular value decomposition (SVD) gives you a precise way to find those directions, measure their strengths, and decide what to keep.

A matrix can hide repeated directions

Use a tiny matrix that builds on the named-axis discipline from the earlier vectors-and-tensors lesson. Each row is one ticket. The two columns record whether the ticket contains a latency signal or a rollback signal:

$$A = \begin{bmatrix} 1 & 0 \\ 0 & 1 \\ 1 & 1 \end{bmatrix}$$

The third ticket contains both signals. That makes the combined direction [latency + rollback] stronger than the contrast direction [latency - rollback].

To compare directions fairly, give each one length 1:

$$v_{\text{common}} = \frac{1}{\sqrt{2}} \begin{bmatrix}1 \\ 1\end{bmatrix}, \qquad v_{\text{contrast}} = \frac{1}{\sqrt{2}} \begin{bmatrix}1 \\ -1\end{bmatrix}$$

Multiply A by either direction. The result tells you how strongly each ticket activates that pattern; its length tells you the pattern's total strength in the matrix.

The common direction has strength $\sqrt{3}$; the contrast direction has strength $1$. The third row reinforces the common direction and cancels in the contrast direction. Nothing is compressed yet. You're only measuring which feature combinations the rows support most strongly.

The Gram matrix reveals those directions by hand

For a data matrix A, multiply it by its transpose on the feature side:

$$G = A^T A = \begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix}$$

G has shape [2, 2] because it compares feature directions with feature directions. A direction v that obeys $Gv = \lambda v$ is an eigenvector of G: G scales it without turning it. Its eigenvalue $\lambda$ is squared strength, because

$$\lVert A v \rVert^2 = v^T A^T A v = \lambda$$

when v has length 1.

For this matrix:

$$G \begin{bmatrix}1 \\ 1\end{bmatrix} = 3 \begin{bmatrix}1 \\ 1\end{bmatrix}, \qquad G \begin{bmatrix}1 \\ -1\end{bmatrix} = 1 \begin{bmatrix}1 \\ -1\end{bmatrix}$$

Why do eigenvalues of A.T @ A become squared singular values?

Answer

For a unit direction v, the number ||A @ v|| is how much A stretches that direction. Squaring the length gives (A @ v).T @ (A @ v), which equals v.T @ A.T @ A @ v. If v is an eigenvector, that value is its eigenvalue. SVD reports the stretch, so it takes the square root.

SVD names the directions and strengths

Every real matrix has a singular value decomposition:^{[1]Reference 1Numerical Linear Algebra.https://people.maths.ox.ac.uk/trefethen/books.html}

$$A = U \Sigma V^T$$

For the incident matrix A with shape [3, 2], use the compact (or reduced) form. It keeps only the two singular directions that can fit in a matrix with two feature columns:

Our right singular vectors are the normalized common and contrast directions. Our singular values are:

$$\sigma_1 = \sqrt{3}, \qquad \sigma_2 = 1$$

The corresponding left singular vectors come from $u_i = A v_i / \sigma_i$. Put the factors back together and they reconstruct every original entry.

Use the direct SVD routine for computation. It returns U, the one-dimensional singular-value array S, and Vt, which is $V^T$.

The sign of a singular vector may differ between correct implementations. If both matching directions in U and V flip sign, their product stays the same. Compare singular values and reconstruction, not raw signs.

Rank counts independent patterns

The rank of a matrix is the number of non-zero singular values. It counts how many independent directions the matrix really needs.

Suppose every rollback feature exactly repeats the latency feature:

$$B = \begin{bmatrix} 1 & 1 \\ 0 & 0 \\ 1 & 1 \end{bmatrix}$$

The second column adds no new information. B has two columns, but only one independent direction, so its rank is 1.

A feature table with rank below its column count deserves inspection. Sometimes it contains a deliberate encoding. Sometimes two features duplicate each other and waste compute. Rank exposes the question; domain meaning answers it.

Exact rank is a mathematical definition. Floating-point data rarely produces a perfect zero, so np.linalg.matrix_rank treats sufficiently small singular values as zero using a tolerance. Measurement noise or downstream requirements may justify a different cutoff.

Truncation trades detail for a smaller matrix

Return to the full-rank incident matrix A. Keeping both singular directions reconstructs it exactly. Keeping only the strongest direction creates a rank-one approximation:

$$A_1 = U_{:,1} \sigma_1 V_{:,1}^T$$

Among all rank-one matrices, this approximation has the smallest Frobenius reconstruction error: the square root of the sum of squared entry errors.^{[2]Reference 2The approximation of one matrix by another of lower rankhttps://doi.org/10.1007/BF02288367} Equivalently, it minimizes that squared sum. The guarantee is about reconstructing A. It doesn't promise that a downstream classifier or retriever keeps its quality.

Here the discarded direction has strength 1, so the rank-one approximation loses meaningful contrast between a latency-only ticket and a rollback-only ticket. A small matrix makes that cost visible before you try the same idea on embeddings.

PCA is SVD after centering each feature

So far the matrix contained signal counts and we kept its origin at zero. Principal component analysis (PCA) answers a different question: which directions describe variation around the average row?

That means centering first. Given a matrix X of tickets by numeric features:

$$X_{\text{centered}} = X - \text{column means}$$

Then SVD on the centered matrix gives principal-component directions in V. If you skip centering, a large average offset can dominate the first direction even when it doesn't describe variation between tickets.

This leading direction points roughly along [delay evidence + escalation evidence], because those two signals rise together in the sample. PCA uses variance, not meaning. A direction that explains a lot of variation still needs human interpretation and task evaluation.

Sparse matrices need a different projection path

PCA centers every feature. On a dense table, that's routine. In a sparse term-document or ticket-log matrix, subtracting a nonzero column mean turns each implicit zero into $-\mu_j$, so a naively centered representation becomes dense. scikit-learn distinguishes PCA from TruncatedSVD, which doesn't center its input and can work efficiently with sparse matrices.^{[3]Reference 3sklearn.decomposition.PCAhttps://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html}

A 100,000 by 10,000 matrix contains one billion cells. At 1% density, a sparse representation stores about 10 million values plus indices; a dense array stores all one billion values. Dense values alone require about 4 GB as float32 or 8 GB as float64. Actual sparse memory depends on index widths and format.

Use TruncatedSVD when an uncentered latent semantic analysis is the intended representation. It answers a different question from PCA: directions relative to the origin, not deviations around feature means. This small check counts how many explicit values centering creates:

The example is deliberately materialized at a safe size. Production code should avoid creating X_dense or X_centered_dense for a large sparse matrix.

Condition number warns about solving through weak directions

SVD also reveals a failure mode. If a matrix stretches one direction strongly and almost erases another, an operation that tries to reverse that transformation must amplify the weak direction.

The 2-norm condition number is:

$$\kappa_2(A) = \frac{\sigma_{\max}}{\sigma_{\min}}$$

when the smallest singular value is non-zero. A condition number near 1 means all directions have comparable scales. A large condition number warns that solving, inverting, whitening (rescaling projected components toward unit variance), or other operations that divide by small singular values can magnify perturbations.^{[1]Reference 1Numerical Linear Algebra.https://people.maths.ox.ac.uk/trefethen/books.html}

Notice the scope of the claim. Merely storing vectors in a matrix with a weak direction doesn't automatically make cosine or dot-product retrieval unstable. The amplification appears when a pipeline performs an inverse-like operation, such as a solve or whitening transform, along that weak direction.

A matrix has singular values 1 and 0.0001. Why can a solve be sensitive even when the input perturbation is small?

Answer

Its condition number is 10,000. Reversing the weak direction divides by 0.0001, so a small perturbation along that direction can become a large change in the solution.

Why direct SVD is more stable

A.T @ A made our hand calculation easy, but it isn't the safe general implementation of SVD. Its condition number is squared:

$$\kappa_2(A^T A) = \kappa_2(A)^2$$

for a full-column-rank matrix. A matrix with condition number 10,000 turns into a Gram matrix with condition number 100,000,000. That can discard useful precision before the decomposition even begins. Use direct SVD implementations in numerical code; keep A.T @ A as a teaching device for the relationship between eigenvectors and singular vectors.

Why is np.linalg.svd(A) a better computational default than finding eigenvectors of A.T @ A yourself?

Answer

The Gram-matrix route is excellent for understanding the formula, but it squares the condition number. Direct SVD routines are designed to avoid throwing away as much numerical precision on weak directions.

Choosing a cutoff uses squared singular values

For centered data, the squared singular values are proportional to how much variation each principal direction explains. For an uncentered count or embedding matrix, they still measure Frobenius energy captured by each direction. In either case, the cumulative ratio gives a transparent reconstruction-based cutoff, where r is the number of singular values:

$$\text{retained ratio at } k = \frac{\sigma_1^2 + \cdots + \sigma_k^2} {\sigma_1^2 + \cdots + \sigma_r^2}$$

This rule answers a reconstruction question. For a retrieval system, follow it with ranking metrics on held-out queries rather than assuming that matrix energy equals relevance.

Build it: compress a tiny retrieval matrix and check rankings

Represent four incident tickets with three term features: latency, rollback, and trace. The query is mostly about a latency trace with a smaller rollback signal. Project both tickets and query onto the top two right-singular directions, then compare rankings before and after compression with cosine similarity. Cosine similarity divides each dot product by both vector lengths, so the score reflects direction rather than raw magnitude.

For this tiny fixture, compression drops a real tail direction and changes scores while preserving the ranking. The explicit norm guard matters too: cosine similarity is undefined for a zero-length query or ticket vector. A production pipeline should reject, quarantine, or replace that vector instead of returning NaN scores.

This is a useful sanity check, not a credible evaluation. On a real retrieval set, measure Recall@k, the fraction of relevant tickets recovered in the top k returned candidates, and inspect failures before accepting reduced dimensions.

Where these ideas return in LLM engineering

LoRA doesn't compute the SVD of a full trained update during fine-tuning. It learns thin factors directly. SVD gives you the language to understand what "low rank" means; later training lessons explain how those factors are optimized.

Practice with measured answers

1. For $C = \begin{bmatrix}3 & 1 \\ 1 & 3\end{bmatrix}$, compute $C^T C$, its two eigenvalues, and its singular values. Then compute the condition number.
2. Modify the retrieval build exercise to keep only one singular direction. Did the first result stay the same? How large did the maximum score change become?
3. Add a large constant offset [100, 100] to every row of the PCA example. Compare SVD before centering and after centering. Which one describes differences between rows?

Answers to check:

1. $C^T C = \begin{bmatrix}10 & 6 \\ 6 & 10\end{bmatrix}$, eigenvalues are 16 and 4, singular values are 4 and 2, and $\kappa_2(C) = 2$.
2. A one-direction representation throws away the latency-versus-rollback contrast. With this fixture, every compressed cosine score becomes 1.0, so the ranking stops being meaningful. The exact code above returns order [3, 2, 1, 0], doesn't preserve the first result, and reports a maximum score change of 0.7113.
3. Uncentered SVD is pulled toward the large offset. Centered SVD removes that shared mean and recovers the variation direction.