Vectors, Matrices & Tensors

The gradients lesson computed [4, 2] for a CodeAssist latency prediction: one loss slope for the prompt-length weight and one for the queue-delay weight. That ordered list has a name. It's a vector. The gradient tells us how loss changes; the learning rate turns it into a weight update.

What changes when you process three requests together, or keep a separate vector for every token in a prompt? The arithmetic stays familiar, but extra axes answer a different question: which request, which token, and which feature does each number belong to? Once you see how one latency prediction turns into a row-column dot product, stacking multiple requests into a matrix and entire prompt batches into a 3D tensor becomes second nature.

Start with the vector you already computed

Before stacking anything, name the object in front of you. One number is a scalar, such as a loss of 1.0. An ordered list of numbers is a vector. A rectangular table of numbers is a matrix. The numbers don't carry their own meaning; axis labels and position conventions do.

Reuse the two-weight latency row:

Each vector above has length 2, but they play distinct physical roles. Shape tells you how values sit in memory; names and coordinates tell you what they actually mean.

The NumPy lesson introduced .shape. A shape of (2,) denotes a single axis containing two items. That trailing comma is Python's syntax for a one-element tuple so it isn't confused with grouped parentheses. A raw scalar has shape () because it has zero axes.

Use NumPy arrays throughout this lesson; ordinary Python lists don't implement @. This first example repeats the learning-rate update with arrays. All three inputs should have shape (2,), and subtracting 0.1 * gradient should give [1.6, 1.8]:

A vector isn't automatically an embedding. Here it holds two named features or two parameters. Later, a token embedding is a longer vector whose positions are learned through training rather than labeled by hand.

Those updated weights form a fresh vector. How does that weight vector turn into a concrete latency prediction?

Let aligned features make one prediction

The latency model uses a dot product: multiply matching positions, then add. Let $x$ name the features, $w$ the weights, and $\hat{y}$ ("y hat") the predicted latency. Subscripts 1 and 2 below identify the two coordinates; Python indexes those same positions with 0 and 1.

$$\hat{y} = x \cdot w = x_1w_1 + x_2w_2$$

With the last chapter's starting weights [2, 2]:

$$[2, 1] \cdot [2, 2] = (2 \times 2) + (1 \times 2) = 6$$

That reproduces the 6-second prediction from the previous lesson. For the remaining latency examples, freeze a different toy rule, w = [2, 1]. These are chosen coefficients, not the result of the update above; we're studying array operations rather than training:

$$[2, 1] \cdot [2, 1] = (2 \times 2) + (1 \times 1) = 5$$

Prompt length contributes four seconds; queue delay contributes one. Two vectors of length two produce one scalar.

Check the difference between *, which keeps the two contributions, and @, which sums them. Both should agree with the hand calculation:

Element-wise multiplication stops at a contribution vector. The dot product adds those contributions into one number. For two real-valued 1-D arrays, NumPy's @ computes this inner product.^{[1]Reference 1numpy.matmulhttps://numpy.org/doc/stable/reference/generated/numpy.matmul.html}

Why can't you swap the feature order to [queue_delay, prompt_blocks] while keeping weights as [seconds per prompt block, seconds per second of queue delay]?

Answer

The lengths would still match, so the code would run, but positions would no longer describe the same quantities. You'd multiply queue delay by the prompt-length weight and prompt length by the queue-delay weight. Length checks catch structural errors; clear axis meaning catches semantic errors.

Now apply that same rule to several requests. We don't need a different weight vector for every row.

Stack requests into a matrix

Put three observed requests into a matrix: a rectangular array whose rows are requests and whose columns are features. The row axis lets one rule run across all requests.

$$X = \begin{bmatrix} 1 & 0 \\ 2 & 1 \\ 3 & 1 \end{bmatrix} \quad w = \begin{bmatrix} 2 \\ 1 \end{bmatrix}$$

Read conceptual shape [3, 2] as three requests by two features. Multiplying every row by the same weight vector returns one prediction per request:

$$[3, 2] \; @ \; [2] \rightarrow [3]$$

The @ notation is the contract you'll see in library code. Here it means "dot each row with w." A length-two NumPy vector has shape (2,) even though we drew its values vertically in the equation. An explicit column matrix would have shape (2, 1) and would preserve a length-one output axis.

Predict the three results before running this cell: one short prompt with no queue gives 2; the other rows give 5 and 7:

The one-request computation repeats across each row. Feature lengths must agree, and the row axis survives. The final assertion adds a singleton axis with [:, None] to highlight the distinction between a 1D vector and a 2D column matrix.

Those three numbers provide one output per request. But what happens if each request needs two distinct outputs instead of one?

Use a matrix to make several outputs

A weight vector produces one output. A weight matrix produces several. Suppose the latency service also wants an internal escalation-priority score from the same two features. This score uses arbitrary units and handpicked weights; it isn't a probability or a trained decision rule.

Use one column of W for each output:

$$W = \begin{bmatrix} 2.0 & 0.2 \\ 1.0 & 1.5 \end{bmatrix}$$

The first column computes response seconds; the second computes priority. For x = [2, 1], run both dot products side by side:

$$xW = [2, 1] \begin{bmatrix} 2.0 & 0.2 \\ 1.0 & 1.5 \end{bmatrix} = [5.0, 1.9]$$

The seconds column is the familiar [2, 1] rule. The priority column is a second dot product against the same features. The shape contract is:

$$[2] \; @ \; [2, 2] \rightarrow [2]$$

For the whole three-row matrix:

$$[3, 2] \; @ \; [2, 2] \rightarrow [3, 2]$$

This lesson stores examples as rows and outputs as columns, so it writes X @ W. Textbooks that store each example as a column may write W x instead. Both conventions work if axis meanings stay attached to the numbers.

In the code, W[:, 0] selects the seconds column. Compare its predictions with the corresponding column of the full result:

Neural-net layers use the same idea at larger width: one learned matrix mixes input features into many output features for every row at once.

The useful debugging question is now concrete: what if a teammate hands you a W whose inner size doesn't match?

Match inner dimensions before multiplying

Suppose W expects three input features:

$$[3, 2] \; @ \; [3, 2]$$

The first object ends with feature count 2; the second starts with 3. There's no third feature to pair, so the multiply has to fail.

Run the invalid multiplication deliberately, then check a compatible matrix. The catch prints our own short diagnosis because NumPy's full error wording can vary by release:

When multiplication fails, don't randomly transpose arrays until the error disappears. Name the axes first: request rows, input features, and output features. Then fix the object whose meaning is wrong.

Requests aren't the only extra axis. A prompt is a sequence of tokens, so a third axis must say which token each feature vector belongs to.

Add token and batch axes with tensors

In library code, a tensor means an array with any number of axes. NumPy calls its array type ndarray; PyTorch calls its type torch.Tensor. The practical array definition is the one used here and in the opening linear algebra chapter of Deep Learning.^{[2]Reference 2Deep Learning.https://www.deeplearningbook.org/}

A scalar has zero axes, a vector one, and a matrix two. The number of axes is often called tensor rank, available as .ndim in NumPy. Don't confuse it with matrix rank, which measures independent directions and comes in the next lesson.

So far, we represented each request with two summary features: prompt length and queue delay. To represent its text, switch to one vector per token. These token coordinates aren't the latency features, even though we'll also give them length two. The NumPy lesson used the same (B, T, D) convention on log snippets:

Use tiny handcrafted token features so every number stays readable. This isn't a trained embedding table; it's a shape exercise:

Two prompts, each containing three token-feature vectors, therefore have shape [2, 3, 2].

Read that shape as B = 2 prompts, T = 3 token positions, and D = 2 features per token. Indexing is zero-based, so H[1, 2, 0] selects the first feature of the third token in the second prompt. A colon keeps all entries along its axis: H[:, :, 0] therefore keeps batch and sequence but fixes the feature. Run both selections:

The last axis still contains features. Extra axes only organize which prompt and token each vector belongs to. A learned projection should change that last axis and leave the others alone.

Project each token without mixing positions

In this lesson, a projection is a matrix transformation of each token's features. Neural networks learn its coefficients; we'll choose small ones to inspect the arithmetic. Let matrix P map two input features into three output features:

$$[B, T, 2] \; @ \; [2, 3] \rightarrow [B, T, 3]$$

Predict the output before multiplying: batch and sequence should remain intact, while the token-feature size changes from 2 to 3.

Read the inner calculation as "multiply input coordinate d by its coefficient for output coordinate j, then sum over d." The three nested lists repeat that calculation over output features, tokens, and prompts. np.testing.assert_allclose checks the library result against this explicit version, allowing for small floating-point differences.

For a shared projection with shape [D, output_width], each token is transformed independently. Prompts don't mix, and token positions don't mix. More generally, @ multiplies the last two axes and broadcasts compatible leading batch axes.^{[1]Reference 1numpy.matmulhttps://numpy.org/doc/stable/reference/generated/numpy.matmul.html} Here P has no batch axis, so the same matrix serves both prompts.

This also explains an important limit: making the last dimension wider doesn't create new information. Each output is still a weighted combination of the two inputs. A later layer must do more than change width if it needs interactions between token positions.

A token tensor has shape [4, 128, 768] and a projection has shape [768, 256]. What output shape should you expect?

Answer

[4, 128, 256]. The meeting feature dimension 768 contracts, while batch and sequence axes remain intact.

Multiplication usually fails loudly when inner sizes disagree. Addition can stay quiet while still attaching a number to the wrong axis. That makes broadcasting a semantic debugging problem, not only an arithmetic one.

Treat broadcasting as an axis decision

Broadcasting lets a smaller array participate in element-wise operations against a larger one. First choose what should repeat. An output bias [seconds_offset, priority_offset] should repeat across every request row:

$$[3, 2] + [2] \rightarrow [3, 2]$$

NumPy compares dimensions from the right: equal sizes match, a size of 1 can expand, and missing leading dimensions act like size 1. The rule is useful, but it can also hide an axis mistake.^{[3]Reference 3Broadcasting.https://numpy.org/doc/stable/user/basics.broadcasting.html}

Write both additions explicitly, then predict the second request. Feature bias should produce [5.1, 1.8]; row offsets should produce [4.9, 1.8] while still looking valid:

Both results have three rows and two columns, but only one uses the intended axis. Checking only the result shape misses this bug. Requiring a one-dimensional bias with length equal to the output width catches the (3, 1) input. You still need named columns and a known-value example to catch two feature biases supplied in the wrong order.

The same dot product that scored one request can compare token vectors with each other. The result changes from one score to a grid of scores.

Read a score grid as repeated dot products

For three token-feature vectors in one prompt, multiply the token matrix by its transpose, which swaps rows and columns. The transpose turns each token row into a column so a row-column dot product compares two tokens. Predict the result shape first:

$$[T, D] \; @ \; [D, T] \rightarrow [T, T]$$

Every entry in the output is one dot product between two token positions. Rows name the first token; columns name the second.

These are raw alignment scores, not automatically cosine similarities. Notice the middle token's self-score: 0.5² + 0.5² = 0.5, not 1. A dot product depends on vector length as well as direction.

For a real vector $v$, its Euclidean length, written $\|v\|$, is the square root of the sum of its squared coordinates. For nonzero vectors, cosine similarity is $a \cdot b / (\|a\|\|b\|)$.^{[2]Reference 2Deep Learning.https://www.deeplearningbook.org/} Dividing each vector by its length makes its self-score 1. A zero vector has no direction, so cosine similarity needs an explicit policy for that case.

You can draw a two-coordinate vector as an arrow from (0, 0) to its coordinates. Cosine similarity measures the angle between two such arrows: 1 for the same direction, 0 for perpendicular directions, and -1 for opposite directions. Scaling an arrow by a positive number changes its length but not that angle.

Compute both grids and compare their diagonals. keepdims=True gives lengths shape [3, 1], keeping each length attached to its row during normalization:

Later, attention will create separate query and key vectors, scale the Q @ K.T score grid, and convert each row into weights that sum to one.^{[4]Reference 4Attention Is All You Need.https://arxiv.org/abs/1706.03762} That expression is for one 2-D pair of matrices. For a batch shaped [B, T, D], use H @ H.swapaxes(-1, -2) to get [B, T, T]. Plain H.T reverses all three axes and would put batch size in the wrong place.

If Q has shape [5, 8] and K has shape [5, 8], why does Q @ K.T have shape [5, 5]?

Answer

Transposing K turns [5, 8] into [8, 5]. The feature dimensions 8 meet, and the surviving dimensions describe five query positions by five key positions.

Build it: trace one tiny prompt projection

Now combine the pieces in a function that accepts a batch of token sequences and a projection matrix. Convert the inputs to numeric arrays first, reject the wrong number of axes, then check feature width. np.asarray(..., dtype=float) also rejects ragged lists such as [[1, 0], [0.5]]; inspecting only the first row wouldn't catch that missing coordinate.

Use the same two-to-three projection as before. After projecting, average over token positions to produce one summary vector per prompt. NumPy's mean(axis=1) removes the sequence axis and retains batch and features.^{[5]Reference 5numpy.meanhttps://numpy.org/doc/stable/reference/generated/numpy.mean.html}

The two summaries are identical even though the token order differs. That's expected: a shared token projection followed by a mean is insensitive to permutations of the tokens. Shape checks can't make this summary understand order. Later sequence models introduce position information and interactions between tokens for that reason.

Exclude padding from the mean

Real batches often add padding to make sequences equally long. Even zero-valued padding changes an ordinary mean: two real vectors [1, 0] and [0, 1] average to [0.5, 0.5], but adding two zero rows changes that to [0.25, 0.25].

A mask marks real positions. Add a singleton feature axis so one validity bit applies to every coordinate of its token. Sum only real vectors and divide by the count of real tokens. Reject an all-padding prompt because its mean has no valid denominator:

The mask fixes padding bias, not lost word order. Those are separate properties to test. This example assumes finite real-token values; a mask isn't a substitute for validating the numeric data you keep.

Real token vectors won't be handwritten, and learned projections will be much wider. The invariant stays small enough to say aloud: preserve batch and sequence; match and transform feature size.

Recognize the contract inside a neural-network layer

You've now executed four related operations. In this table, N counts rows, D is input width, and P is output width; B and T still mean batch and sequence. Read each contract by identifying the feature axis that's summed away:

One orientation detail matters when you read model code. PyTorch's torch.nn.Linear(in_features, out_features) stores weight as [out_features, in_features], the transpose of our W. It computes input @ weight.T + bias and preserves all input axes except the last, which becomes out_features.^{[6]Reference 6Linearhttps://docs.pytorch.org/docs/stable/generated/torch.nn.Linear.html} So Linear(2, 3) can receive [B, T, 2] directly and return [B, T, 3]. You don't need to flatten the batch or tokens first.

That's an array convention, not a different kind of algebra. Before transposing anything, check how the API names its input and output axes.

Test shapes before code

Keep the convention fixed: rows represent requests or token positions, and the final dimension represents features. Predict each result before checking the sketch:

1. A latency feature vector has shape [3], and a weight vector has shape [3]. What does x @ w return?
2. A training table X has shape [20, 3], and W has shape [3, 4]. What does X @ W mean and what shape does it produce?
3. Prompt hidden states have shape [8, 12, 64], and a projection has shape [64, 16]. Which axes remain unchanged?
4. Scores have shape [20, 4]. Why can adding a value of shape [20, 1] be a silent bug when you meant a four-feature bias?
5. Token vectors have shape [12, 64]. What transpose is needed to create a pairwise score grid?
6. For H with shape [2, 3, 2], what scalar does H[1, 2, 0] select, and what shape does a slice that keeps every prompt and token but fixes feature 0 have?

Solution sketches

1. It returns a scalar because matching feature positions are multiplied and summed.
2. It applies one four-output transformation to each of 20 rows, producing [20, 4].
3. Batch size 8 and sequence length 12 stay unchanged; feature size changes from 64 to 16, giving [8, 12, 16].
4. [20, 1] legally broadcasts one different value across all four output columns in each row. The output shape looks right even though the adjustment is attached to requests instead of features.
5. Use the transpose [64, 12]; then [12, 64] @ [64, 12] produces [12, 12].
6. H[1, 2, 0] selects the first feature of the third token in the second prompt, which is 1.0 in the running tensor. The feature-0 slice keeps prompt and token axes, giving shape [2, 3].

Carry shape contracts forward

Every model operation now reads as a contract:

Before running a multiply, predict its output. When it fails, name the meeting axes. When it broadcasts legally, ask whether the repeated value still means what you intended.

You can now name a latency row, stack rows into a matrix, and project a token tensor without guessing shapes. The next question is whether the matrix carries genuinely different feature directions or repeats the same mixture. Matrix rank counts independent directions; singular value decomposition (SVD) reveals their strengths. Those are the next lesson's questions.