Linear Algebra for ML

You're investigating a production incident during a high-traffic release. Pull three customer support tickets into an incident table. Each row tracks two binary signals: did the user report high latency, and did they request an emergency rollback? Ticket t2 flags both issues at once. Intuitively, the joint latency-plus-rollback pattern dominates your dataset, while the contrast between latency-only and rollback-only looks like a secondary detail.

What happens if you compress this table to save memory or speed up vector retrieval? If you discard the weaker contrast direction, a latency-only ticket and a rollback-only ticket collapse into identical representations. Your search or routing pipeline suddenly can't tell an API timeout from a broken deployment.

Singular value decomposition (SVD) finds perpendicular input directions and measures the output length produced by each unit direction. For this uncentered table, those strengths describe magnitude relative to zero, rather than variation around the average ticket. Choosing which directions to keep turns linear algebra into a compression decision whose cost we can calculate.

In the vectors, matrices, and tensors lesson, we worked with rows, columns, and dot products. Keep those operations close: three tickets and two features give us everything we need to compute, visualize, and diagnose a full SVD by hand.

A matrix can hide repeated directions

Write those flags as a 3×2 matrix. Each row is one ticket; each column records a latency or rollback signal:

$$A = \begin{bmatrix} 1 & 0 \\ 0 & 1 \\ 1 & 1 \end{bmatrix}$$

Look at t2 and predict its score along two diagonals. Because it contains both signals, it should reinforce [latency + rollback]; its contribution to [latency - rollback] should cancel.

First compare the raw patterns [1, 1] and [1, -1]. Ticket t2 = [1, 1] scores 1 + 1 = 2 on the first and 1 - 1 = 0 on the second. Both pattern vectors have length $\sqrt{1^2 + 1^2} = \sqrt{2}$. Divide by that length to compare unit vectors, whose length is 1:

$$v_{\text{common}} = \frac{1}{\sqrt{2}} \begin{bmatrix}1 \\ 1\end{bmatrix}, \qquad v_{\text{contrast}} = \frac{1}{\sqrt{2}} \begin{bmatrix}1 \\ -1\end{bmatrix}$$

Multiply A by either direction. Each entry is one ticket's score for that pattern. The common scores are $[1/\sqrt{2}, 1/\sqrt{2}, 2/\sqrt{2}]$, so their squared lengths add to $1/2 + 1/2 + 2 = 3$. For contrast, the scores are $[1/\sqrt{2}, -1/\sqrt{2}, 0]$ and the squared lengths add to $1$. Taking square roots gives strengths $\sqrt{3}$ and $1$. NumPy checks each step:

The common direction has strength $\sqrt{3}$; the contrast direction has strength $1$. Nothing has been compressed yet. This is a measurement of which feature mix the rows support most strongly.

Coordinates come from a basis

The common and contrast directions are linear combinations of the original feature axes: scale each axis, then add. The span is every vector reachable by those combinations.

These directions are orthogonal: their dot product is $(1 - 1)/2 = 0$, so they meet at a right angle. Neither is a multiple of the other, which makes them linearly independent. Together they form a basis: a set of independent vectors that spans the entire two-dimensional feature space. Because they're also unit vectors, this basis is orthonormal.

To practice changing coordinates, allow mention counts instead of yes-or-no flags for a new row $x = [3, 1]$. Its common score is $(3 + 1)/\sqrt{2} = 2\sqrt{2}$; its contrast score is $(3 - 1)/\sqrt{2} = \sqrt{2}$.

Because this basis is orthonormal, its coordinates are dot products with the two unit directions:

$$c_{\text{common}} = x \cdot v_{\text{common}} = 2\sqrt{2}, \qquad c_{\text{contrast}} = x \cdot v_{\text{contrast}} = \sqrt{2}$$

Each coefficient is an orthogonal projection coordinate. Multiply a coefficient by its unit direction to recover the part of x carried by that pattern:

So $x = (x \cdot v_{\text{common}})v_{\text{common}} + (x \cdot v_{\text{contrast}})v_{\text{contrast}}$. The A @ v scores above measure every ticket along one direction; these coordinates rebuild one ticket from its directions. Later, projecting onto fewer SVD directions will keep selected coordinates and deliberately discard the rest.

We chose this basis by inspection. For real data, you need the matrix to reveal it. A.T @ A makes that choice visible without changing the feature space.

The Gram matrix reveals those directions by hand

For a data matrix A, compare feature columns with one another by multiplying on the feature side:

$$G = A^T A = \begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix}$$

The first diagonal entry is the latency column's dot product with itself: $1^2 + 0^2 + 1^2 = 2$. The off-diagonal entry is the columns' overlap: $1\cdot0 + 0\cdot1 + 1\cdot1 = 1$. G has shape [2, 2] because its rows and columns both index features.

A nonzero vector v satisfying $Gv = \lambda v$ is an eigenvector. Multiplying by G keeps it on the same line through the origin. Here the eigenvalue $\lambda$ is nonnegative because G is a Gram matrix, so it stretches or shrinks the vector without reversing it. For a unit eigenvector, $\lambda$ is squared strength because

$$\lVert A v \rVert^2 = v^T A^T A v = \lambda$$

when v has length 1.

For this matrix, the common direction is scaled by 3 and the contrast direction by 1:

$$G \begin{bmatrix}1 \\ 1\end{bmatrix} = 3 \begin{bmatrix}1 \\ 1\end{bmatrix}, \qquad G \begin{bmatrix}1 \\ -1\end{bmatrix} = 1 \begin{bmatrix}1 \\ -1\end{bmatrix}$$

SVD names the directions and strengths

Any real matrix, including this 3×2 incident table, has a singular value decomposition (SVD):^{[1]Reference 1Numerical Linear Algebra.https://people.maths.ox.ac.uk/trefethen/books.html}

$$A = U \Sigma V^T$$

The incident matrix A has shape [3, 2]. Its reduced SVD keeps min(3, 2) = 2 directions. Reduced SVD still includes zero singular values if the matrix has dependent columns; it hasn't made a compression decision yet.^{[2]Reference 2numpy.linalg.svdhttps://numpy.org/doc/stable/reference/generated/numpy.linalg.svd.html}

Here, the right singular vectors are the normalized common and contrast directions. Their singular values tell us how strongly A stretches those directions:

$$\sigma_1 = \sqrt{3}, \qquad \sigma_2 = 1$$

For each positive singular value, the corresponding left singular vector is $u_i = A v_i / \sigma_i$. For the common pattern, dividing its three scores by $\sqrt{3}$ gives $u_1 = [1, 1, 2]^T/\sqrt{6}$. For contrast, $u_2 = [1, -1, 0]^T/\sqrt{2}$. Columns of U have unit length; the scaled columns U * S contain the actual ticket scores. Put all three factors back together and they reconstruct every original entry.

For computation, call the SVD routine directly. It returns U, the one-dimensional singular-value array S, and Vt, which is $V^T$. Before running it, predict the shapes: reduced U should have one row per ticket, and Vt should have one row per retained feature direction.

The sign of a singular vector may differ between correct implementations. If matching directions in U and V flip sign together, their product stays the same. Repeated singular values also permit rotations within the tied subspace. Compare singular values and reconstruction, not raw vectors. The tiny printed reconstruction error is floating-point roundoff, not discarded information.

Rank counts independent patterns

The rank of a matrix is the number of non-zero singular values. It counts how many independent directions the matrix really needs, not how many columns happen to be present.

Now make the failure concrete. Suppose every rollback feature exactly repeats the latency feature:

$$B = \begin{bmatrix} 1 & 1 \\ 0 & 0 \\ 1 & 1 \end{bmatrix}$$

The second column adds no new information. B still has two columns, but only one independent direction, so its rank is 1. If you had to predict a new row from one column, the duplicate couldn't improve that prediction.

A feature table with rank below its column count deserves inspection. It may contain a deliberate encoding, or two features may duplicate each other and waste compute. Rank exposes the structural fact; domain meaning tells you whether that fact is a bug.

In general, rank is the dimension of a matrix's column space, the span of its columns, and also the dimension of its row space.

Exact rank is a mathematical definition. Floating-point calculations needn't produce a perfect zero for a dependent direction, so np.linalg.matrix_rank treats sufficiently small singular values as zero using a tolerance. Its default accounts for roundoff and matrix dimensions; a measurement-noise threshold may need to be larger. Numerical rank depends on that threshold.^{[3]Reference 3numpy.linalg.matrix_rankhttps://numpy.org/doc/stable/reference/generated/numpy.linalg.matrix_rank.html}

Truncation trades detail for a smaller matrix

Return to full-rank incident matrix A. Keeping both singular directions reconstructs it exactly. What would one direction lose? Keeping only the strongest creates a rank-one approximation:

$$A_1 = U_{:,1} \sigma_1 V_{:,1}^T$$

Among all rank-one matrices, this approximation has the smallest Frobenius reconstruction error: the square root of the sum of squared entry errors.^{[4]Reference 4The approximation of one matrix by another of lower rankhttps://doi.org/10.1007/BF02288367} Equivalently, it minimizes that squared sum. Notice what the guarantee covers: reconstructing A. It doesn't promise that a downstream classifier or retriever keeps its quality.

The discarded direction has strength 1. Look at the top two rows of A₁: a latency-only ticket and a rollback-only ticket both become [0.5, 0.5]. Their contrast is gone. Four entries each have absolute error 0.5, so the Frobenius error is $\sqrt{4\cdot0.5^2} = 1$.

A₁ is still a 3×2 matrix. The smaller representation is its one coordinate per ticket, A @ Vt[:1].T, plus the saved feature direction needed for reconstruction. Rebuilding the full matrix doesn't save storage. This distinction matters when compressing embeddings.

PCA is SVD after centering each feature

So far, zero meant "no signal." Principal component analysis (PCA) changes the question: which directions describe how tickets vary around their average row?

That question requires centering first. Given a matrix X of tickets by numeric features, subtract each column's average:

$$X_{\text{centered}} = X - \text{column means}$$

For our three tickets, both column means are $2/3$. Subtraction changes t0 from [1, 0] to [1/3, -2/3]: it has more latency and less rollback than the average ticket. The centered rows are [1/3, -2/3], [-2/3, 1/3], and [1/3, 1/3].

Run SVD on those rows and the right singular vectors give principal-component directions. They're also eigenvectors of the sample covariance matrix centered.T @ centered / (n - 1), where n is the number of rows. Its entries measure how feature columns vary together. For this example, centering makes the contrast direction strongest. Check which two tickets differ most from each other before running the code.^{[5]Reference 5sklearn.decomposition.PCAhttps://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html}

Uncentered SVD favored common signal because the rows sit mostly in the positive diagonal direction from the origin. Centered PCA favors contrast because the latency-only and rollback-only tickets are the most separated around their mean. Both results are correct: we asked different questions of the same rows.

PCA finds variance, not relevance. It also centers without automatically rescaling feature units. If one feature is measured in milliseconds and another in seconds, choose the scaling deliberately before fitting. For a new row, reuse the training column means and the fitted directions; don't fit a fresh PCA basis per query.^{[5]Reference 5sklearn.decomposition.PCAhttps://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html}

Avoid materializing a centered sparse matrix

PCA centers every feature mathematically. In a sparse term-document matrix, subtracting a nonzero column mean turns each implicit zero into $-\mu_j$. Explicitly building that centered matrix destroys sparsity. This doesn't make PCA impossible on sparse input: scikit-learn supports sparse inputs with selected solvers, including arpack. The implementation can handle centering without constructing the full centered data table.^{[5]Reference 5sklearn.decomposition.PCAhttps://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html}

A 100,000 by 10,000 matrix has one billion cells. At 1% density, a sparse format stores about 10 million values plus indices. The dense values alone take about 4 GB as float32, or about 8 GB as float64. Those are order-of-magnitude values; index widths and format add their own cost.

If directions relative to the origin are what you want, TruncatedSVD works directly on sparse counts or TF-IDF values (term counts reweighted by how common a term is across documents). Unlike PCA, it doesn't center; this is the usual latent semantic analysis setup.^{[6]Reference 6TruncatedSVDhttps://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html} The choice changes the question, not just memory use. Count the nonzero entries before and after centering a small dense array with roughly 1% nonzero values:

The example is deliberately small. For a large sparse matrix, don't materialize X or X - mean; use a sparse-aware decomposition path.

Least squares projects a target onto a feature span

SVD also supports the fitting problem behind linear regression. Suppose the three tickets have target severities

$$y = \begin{bmatrix} 1 \\ 2 \\ 2 \end{bmatrix}$$

The two columns of A are the available latency and rollback features. The first two rows demand weights 1 and 2, but those weights predict 3 for the last row rather than its target 2. No combination matches all three targets. Choose weights $w$ that make the squared residual as small as possible:

$$\underset{w}{\operatorname{argmin}}\;\lVert Aw - y \rVert_2^2$$

This is least squares.^{[1]Reference 1Numerical Linear Algebra.https://people.maths.ox.ac.uk/trefethen/books.html} The fitted vector $Aw$ is the orthogonal projection of y onto the span of A's columns. In other words, choose the point in that feature span closest to the target. For this small example, the hand calculation uses the Gram matrix already computed:

$$A^T A w = A^T y, \qquad \begin{bmatrix}2 & 1 \\ 1 & 2\end{bmatrix}w = \begin{bmatrix}3 \\ 4\end{bmatrix}, \qquad w = \begin{bmatrix}2/3 \\ 5/3\end{bmatrix}$$

That gives $Aw = [2/3, 5/3, 7/3]$ and residual $y - Aw = [1/3, 1/3, -1/3]$. The residual is perpendicular to both feature columns. Move anywhere else inside their span and the squared error can only stay the same or grow.

NumPy's least-squares solver checks the arithmetic. It finds the weights that minimize the Euclidean residual norm; that also minimizes its square. Inspect the prediction and residual, not just a single loss number:^{[7]Reference 7numpy.linalg.lstsqhttps://numpy.org/doc/stable/reference/generated/numpy.linalg.lstsq.html}

The normal equations make the geometry easy to see, but forming $A^T A$ can square a condition number. For a large or ill-conditioned design matrix, use a direct least-squares or SVD-based routine instead of building the Gram matrix yourself. That numerical warning is next.

Condition number warns about solving through weak directions

SVD also reveals a failure mode. If a matrix stretches one direction strongly and almost erases another, a solve that tries to reverse that transformation must amplify whatever landed in the weak direction.

For example, a diagonal matrix with entries 1 and 0.0001 maps [1, 1] to [1, 0.0001]. Reversing it divides the second output by 0.0001. An output error of only 0.0001 therefore adds 1 to the recovered second coordinate.

The ratio between the largest and smallest stretch measures this imbalance. It's the 2-norm condition number:

$$\kappa_2(A) = \frac{\sigma_{\max}}{\sigma_{\min}}$$

when the smallest singular value is nonzero. For an invertible square matrix, a condition number near 1 means directions have comparable scales; it doesn't mean the matrix is close to the identity. A large condition number warns about sensitivity of inverse-like operations. If a singular value is zero, a square matrix has no inverse and its condition number is infinite. Whitening (rescaling projected components toward unit variance) also divides by their strengths, so weak directions need care.^{[8]Reference 8numpy.linalg.condhttps://numpy.org/doc/stable/reference/generated/numpy.linalg.cond.html} The next example isolates that sensitivity:

Storing vectors in a matrix with a weak direction doesn't automatically make cosine or dot-product retrieval unstable. Amplification appears when a pipeline performs an inverse-like operation, such as a solve or whitening transform, along that weak direction.

Forming AᵀA can square the condition number

A.T @ A made our hand calculation easy, but it isn't the safe general implementation of SVD. For a full-column-rank matrix, forming it squares the 2-norm condition number:

$$\kappa_2(A^T A) = \kappa_2(A)^2$$

A condition number of 10,000 becomes 100,000,000 in the Gram matrix. The full-column-rank assumption matters: a wide matrix can have a finite ratio of singular values while A.T @ A is singular. Weak directions can lose useful precision before the decomposition begins. Use a direct SVD implementation in numerical code; keep A.T @ A as a teaching device for the relationship between eigenvectors and singular vectors.

Choosing a cutoff uses squared singular values

For centered data, squared singular values are proportional to the variation explained by each principal direction. For an uncentered count or embedding matrix, they still measure Frobenius energy captured by each direction. In either case, the cumulative ratio gives a reconstruction-based cutoff, where r is the number of singular values. Predict the smallest k that should pass a 95% target from the spectrum below, then check it:

$$\text{retained ratio at } k = \frac{\sigma_1^2 + \cdots + \sigma_k^2} {\sigma_1^2 + \cdots + \sigma_r^2}$$

This rule answers a reconstruction question. For a retrieval system, follow it with ranking metrics on held-out queries. Matrix energy isn't a proxy for relevance by itself.

Compress a retrieval matrix and check the ranking

Represent four incident tickets with three term features: latency, rollback, and trace. The query is mostly about a latency trace with a smaller rollback signal. We'll project tickets and query onto the top two right-singular directions, then compare rankings before and after compression with cosine similarity.

Cosine similarity divides each dot product by both vector lengths, so it compares direction rather than raw magnitude. Before running the fixture, predict whether dropping only the weakest direction should change the top ticket.

For this tiny fixture, dropping the tail direction changes scores a little and keeps the same order. The explicit norm guard matters too: cosine similarity is undefined for a zero-length query or ticket vector. A serving pipeline should reject, quarantine, or replace that vector instead of returning NaN scores.

The fixture is a sanity check, not a credible evaluation. On a real retrieval set, measure Recall@k (the fraction of relevant tickets recovered in the top k returned candidates) and inspect failures before accepting reduced dimensions.

Low-rank factors in LoRA and embedding retrieval

LoRA doesn't compute the SVD of a full trained update during fine-tuning. It learns thin factors directly. SVD gives you the language for "low rank"; later training lessons explain how those factors are optimized.

Practice with measured answers

Use the same fixture to test each idea under a small change. Start with the symmetric matrix, then stress the interpretation rather than the arithmetic:

1. Start with the symmetric matrix $C = \begin{bmatrix}3 & 1 \\ 1 & 3\end{bmatrix}$: compute $C^T C$, its two eigenvalues, singular values, and condition number.
2. In the retrieval exercise, keep only one singular direction. Did the first result stay the same? How large did the maximum score change become?
3. Add a large constant offset [100, 100] to every row of the PCA example. Compare SVD before and after centering. Which version describes differences between rows?
4. Fit y = [1, 2, 2] with the columns of A using least squares. What are the weights, prediction, and residual?

Check your reasoning against these answers:

1. $C^T C = \begin{bmatrix}10 & 6 \\ 6 & 10\end{bmatrix}$, eigenvalues are 16 and 4, singular values are 4 and 2, and $\kappa_2(C) = 2$.
2. In one dimension, nonzero scalars with the same sign have cosine similarity 1. All four tickets and this query project to the same sign, so all four scores are 1 up to roundoff. The order of those ties isn't evidence of relevance and can vary with numerical precision or sorting behavior. The maximum score change is approximately 0.7113.
3. Uncentered SVD is pulled toward [1, 1] by the offset. Centering removes it and recovers the contrast direction [1, -1]/sqrt(2), up to sign, just as before.
4. The weights come out to [2/3, 5/3], yielding prediction [2/3, 5/3, 7/3] with residual [1/3, 1/3, -1/3]. That residual is orthogonal to both columns of A.

Our three-ticket matrix leaves one concrete warning: its weaker contrast direction is exactly what distinguishes a latency-only ticket from a rollback-only ticket. SVD tells you how much reconstruction error discarding it costs. Only the downstream task can tell you whether losing that distinction is acceptable.