Vectors, Matrices & Tensors

CodeAssist's previous latency-gradient calculation produced [4, 2]: one correction for its prompt-length weight and one for its queue-delay weight. That list already has a mathematical name. It's a vector.

One list can become a row in a dataset, learned weights can change its features, and batches of token sequences need extra axes. These shapes are the notation behind training code, similarity search, and later Transformer chapters.

Start with the vector you already computed

A scalar is one number, such as a loss of 1.0. A vector is an ordered list of numbers. Its order matters because each position has a meaning.

Reuse the response-latency example from the gradient calculation:

Every vector above has shape [2], but they play different roles. Shape tells you how values are arranged. Names tell you what those values mean.

A vector isn't automatically an embedding. Here it holds two features or two parameters. Much later, a token embedding will be a longer vector whose positions are learned rather than named by hand.

A dot product makes one weighted prediction

The latency model uses a dot product to combine two features into one prediction:

$$\hat{y} = x \cdot w = x_1w_1 + x_2w_2$$

Here x is the feature vector, w is the weight vector, and \hat{y} is the predicted response latency. To isolate the operation, use a calibrated rule with x = [2, 1] and w = [2, 1]:

$$[2, 1] \cdot [2, 1] = (2 \times 2) + (1 \times 1) = 5$$

The dot product multiplies matching positions, then adds the contributions. Two vectors of length two produce one scalar prediction.

The output shape is (): NumPy's representation for a scalar. Multiplying element by element produced a vector of contributions; applying @ summed them into one number.

Several requests make a matrix

Training from one request would be fragile. Put three observed requests into a matrix: a rectangular array whose rows are requests and whose columns are features.

$$X = \begin{bmatrix} 1 & 0 \\ 2 & 1 \\ 3 & 1 \end{bmatrix} \quad w = \begin{bmatrix} 2 \\ 1 \end{bmatrix}$$

Read conceptual shape [3, 2] as three requests by two features. NumPy prints the same shape as tuple (3, 2). Multiplying every row by the same weight vector returns one prediction per request:

$$[3, 2] \; @ \; [2] \rightarrow [3]$$

This is the same computation you did for one request, repeated efficiently for every row. In this matrix-vector case, the feature dimensions must agree and the row axis survives in the result.^{[1]Reference 1Array programming with NumPy.https://www.nature.com/articles/s41586-020-2649-2}

A matrix can transform features into new features

The weight vector above produces one output. A model often wants several outputs from the same input. Suppose CodeAssist needs a predicted response latency and an internal escalation-priority score.

Use one column of a weight matrix for each output:

$$W = \begin{bmatrix} 2.0 & 0.2 \\ 1.0 & 1.5 \end{bmatrix}$$

The first column computes response seconds; the second computes priority. For x = [2, 1]:

$$xW = [2, 1] \begin{bmatrix} 2.0 & 0.2 \\ 1.0 & 1.5 \end{bmatrix} = [5.0, 1.9]$$

The shape contract is:

$$[2] \; @ \; [2, 2] \rightarrow [2]$$

For the whole three-row matrix:

$$[3, 2] \; @ \; [2, 2] \rightarrow [3, 2]$$

Matrices matter in neural networks because one learned transform can mix input features into many output features for every row at once.

Meeting dimensions are a contract, not a suggestion

Suppose a teammate replaces W with a matrix expecting three input features:

$$[3, 2] \; @ \; [3, 2]$$

The first object ends with feature count 2; the second expects feature count 3. No computation can match the missing feature, so NumPy raises an error.

When you see a multiplication error, don't randomly transpose arrays until it disappears. Name the axes first: request rows, input features, and output features. Then fix the object whose meaning is wrong.

Framework tensors add axes for sequences and batches

In NumPy and PyTorch code, a tensor is an array with any number of axes. Use that framework meaning here: a vector is a one-axis tensor, and a matrix is a two-axis tensor. Text models commonly need three named axes:

Use tiny handcrafted token features so every number stays readable. This isn't a trained embedding table; it's a shape exercise:

Two prompts, each containing three token-feature vectors, have shape [2, 3, 2].

The last axis still contains features. Extra axes organize which prompt and which token each vector belongs to.

A token projection keeps leading tensor axes

Once a prompt tensor has token features, a learned projection can change only its final feature dimension. Let P map two input features into three output features:

$$[B, T, 2] \; @ \; [2, 3] \rightarrow [B, T, 3]$$

Batch and sequence remain intact. Only the token-feature size changes.

For this [B, T, D] @ [D, P] case, NumPy applies the same two-dimensional projection at every leading [B, T] position, preserving the batch and sequence axes. More generally, higher-rank @ treats operands as stacks of matrices on their last two axes and broadcasts compatible leading axes.^{[1]Reference 1Array programming with NumPy.https://www.nature.com/articles/s41586-020-2649-2}

Broadcasting: valid arithmetic can still be the wrong meaning

Broadcasting lets a smaller array participate in element-wise operations against a larger one. Adding an output bias [seconds_offset, priority_offset] to every request row is useful:

$$[3, 2] + [2] \rightarrow [3, 2]$$

NumPy compares dimensions from the right: equal dimensions match, a dimension of size 1 can expand, and missing leading dimensions act like size 1. That flexibility helps, but it can hide an axis mistake.^{[2]Reference 2Broadcasting.https://numpy.org/doc/stable/user/basics.broadcasting.html}

Both additions return shape [3, 2], yet only one uses the intended axis. A failing operation is easy to debug. A legal broadcast on the wrong axis is more dangerous, so assertions should state meaning, not final size alone.

A score grid is repeated dot products

The latency predictor used a dot product to combine features with weights. The same operation can compare vectors to one another.

For three token-feature vectors in one prompt, multiply the token matrix by its transpose:

$$[T, D] \; @ \; [D, T] \rightarrow [T, T]$$

Every entry in the output is one dot product between two token positions.

These are raw alignment scores, not automatically cosine similarities. A dot product changes when vector magnitude changes; cosine similarity first divides by vector lengths.

Later, attention will create separate query and key vectors, scale the Q @ K.T score grid, and convert each row into weights that sum to one.^{[3]Reference 3Attention Is All You Need.https://arxiv.org/abs/1706.03762} You don't need attention mechanics yet. For now, recognize its shape as many dot products arranged in a matrix.

Build it: trace one tiny prompt projection

The lab below packages the shape reasoning into a guardrail. It accepts a batch of token-feature sequences and a projection matrix. Before multiplying, it checks that the hidden states have three named axes, the projection is a matrix, and the feature size agrees with the matrix input size.

These handcrafted sequences have three meaningful token positions each. Real padded text batches need masked pooling: plain .mean(axis=1) would let padding positions change the summaries.

Real token vectors won't be hand written, and the model's learned projections will be much wider. The invariant stays small enough to say aloud: preserve batch and sequence; match and transform feature size.

PyTorch keeps the same shape reasoning

NumPy is excellent for seeing the algebra. PyTorch carries the same tensor operations into trainable models and automatic differentiation.^{[4]Reference 4PyTorch: An Imperative Style, High-Performance Deep Learning Library.https://arxiv.org/abs/1912.01703} Confirm that the same batch projection gives the same values in both libraries:

This is enough PyTorch for now. Device placement, mixed precision, training modules, and full attention blocks each need their own careful lesson.

Practice with shapes before code

Use the same conventions: rows represent requests or token positions; the final dimension represents features.

1. A latency feature vector has shape [3], and a weight vector has shape [3]. What does x @ w return?
2. A training table X has shape [20, 3], and W has shape [3, 4]. What does X @ W mean and what shape does it produce?
3. Ticket hidden states have shape [8, 12, 64], and a projection has shape [64, 16]. Which axes remain unchanged?
4. Scores have shape [20, 4]. Why can adding a value of shape [20, 1] be a silent bug when you meant a four-feature bias?
5. Token vectors have shape [12, 64]. What transpose is needed to create a pairwise score grid?

Solution sketches

1. It returns a scalar because matching feature positions are multiplied and summed.
2. It applies one four-output transformation to each of 20 rows, producing [20, 4].
3. Batch size 8 and sequence length 12 stay unchanged; feature size changes from 64 to 16, giving [8, 12, 16].
4. [20, 1] legally broadcasts one different value across all four output columns in each row. The output shape looks right even though the adjustment is attached to requests instead of features.
5. Use the transpose [64, 12]; then [12, 64] @ [64, 12] produces [12, 12].

Shape contracts to carry forward

A model operation now reads as a contract:

Predict a multiplication output before running it, catch an invalid meeting dimension, and explain why a legal broadcast might still corrupt meaning.