Vectors, Matrices & Tensors

In the previous chapter, ShopFlow's delivery model produced a gradient [4, 2]: one correction for its distance weight and one for its warehouse-backlog weight. That list already has a mathematical name. It's a vector.

Now you will learn how one list becomes a row in a dataset, how learned weights change its features, and how batches of token sequences need extra axes. These shapes are the notation behind training code, similarity search, and later Transformer chapters.

Start with the vector you already computed

A scalar is one number, such as a loss of 1.0. A vector is an ordered list of numbers. Its order matters because each position has a meaning.

Reuse the delivery-time example from the gradient calculation:

Every vector above has shape [2], but they play different roles. Shape tells you how values are arranged. Names tell you what those values mean.

A vector isn't automatically an embedding. Here it holds two features or two parameters. Much later, a token embedding will be a longer vector whose positions are learned rather than named by hand.

A dot product makes one weighted prediction

The delivery model uses a dot product to combine two features into one prediction:

$$\hat{y} = x \cdot w = x_1w_1 + x_2w_2$$

Here x is the feature vector, w is the weight vector, and \hat{y} is the predicted delivery time. To isolate the operation, use a calibrated rule with x = [2, 1] and w = [2, 1]:

$$[2, 1] \cdot [2, 1] = (2 \times 2) + (1 \times 1) = 5$$

The dot product multiplies matching positions, then adds the contributions. Two vectors of length two produce one scalar prediction.

The output shape is (): NumPy's representation for a scalar. Multiplying element by element produced a vector of contributions; applying @ summed them into one number.

Several orders make a matrix

Training from one delivery would be fragile. Put three observed deliveries into a matrix: a rectangular array whose rows are orders and whose columns are features.

$$X = \begin{bmatrix} 1 & 0 \\ 2 & 1 \\ 3 & 1 \end{bmatrix} \quad w = \begin{bmatrix} 2 \\ 1 \end{bmatrix}$$

Read conceptual shape [3, 2] as three orders by two features. NumPy prints the same shape as tuple (3, 2). Multiplying every row by the same weight vector returns one prediction per order:

$$[3, 2] \; @ \; [2] \rightarrow [3]$$

This is the same computation you did for one order, repeated efficiently for every row. In this matrix-vector case, the feature dimensions must agree and the row axis survives in the result.^[1]

A matrix can transform features into new features

The weight vector above produces one output. A model often wants several outputs from the same input. Suppose ShopFlow needs a predicted number of hours and an internal handling-priority score.

Use one column of a weight matrix for each output:

$$W = \begin{bmatrix} 2.0 & 0.2 \\ 1.0 & 1.5 \end{bmatrix}$$

The first column computes delivery hours; the second computes priority. For x = [2, 1]:

$$xW = [2, 1] \begin{bmatrix} 2.0 & 0.2 \\ 1.0 & 1.5 \end{bmatrix} = [5.0, 1.9]$$

The shape contract is:

$$[2] \; @ \; [2, 2] \rightarrow [2]$$

For the whole three-row matrix:

$$[3, 2] \; @ \; [2, 2] \rightarrow [3, 2]$$

This is why matrices matter in neural networks: one learned transform can mix input features into many output features for every row at once.

Meeting dimensions are a contract, not a suggestion

Suppose a teammate replaces W with a matrix expecting three input features:

$$[3, 2] \; @ \; [3, 2]$$

The first object ends with feature count 2; the second expects feature count 3. No computation can match the missing feature, so NumPy raises an error.

When you see a multiplication error, don't randomly transpose arrays until it disappears. Name the axes first: order rows, input features, and output features. Then fix the object whose meaning is wrong.

Framework tensors add axes for sequences and batches

In NumPy and PyTorch code, a tensor is an array with any number of axes. This chapter uses that framework meaning: a vector is a one-axis tensor, and a matrix is a two-axis tensor. Text models commonly need three named axes:

Use tiny handcrafted token features so every number stays readable. This isn't a trained embedding table; it's a shape exercise:

Two tickets, each containing three token-feature vectors, have shape [2, 3, 2].

The last axis still contains features. Extra axes organize which ticket and which token each vector belongs to.

A token projection keeps leading tensor axes

Once a ticket tensor has token features, a learned projection can change only its final feature dimension. Let P map two input features into three output features:

$$[B, T, 2] \; @ \; [2, 3] \rightarrow [B, T, 3]$$

Batch and sequence remain intact. Only the token-feature size changes.

For this [B, T, D] @ [D, P] case, NumPy applies the same two-dimensional projection at every leading [B, T] position, preserving the batch and sequence axes. More generally, higher-rank @ treats operands as stacks of matrices on their last two axes and broadcasts compatible leading axes.^[1]

Broadcasting: valid arithmetic can still be the wrong meaning

Broadcasting lets a smaller array participate in element-wise operations against a larger one. Adding an output bias [hours_offset, priority_offset] to every order row is useful:

$$[3, 2] + [2] \rightarrow [3, 2]$$

NumPy compares dimensions from the right: equal dimensions match, a dimension of size 1 can expand, and missing leading dimensions act like size 1. This is powerful, but it can hide an axis mistake.^[2]

Both additions return shape [3, 2], yet only one uses the intended axis. A failing operation is easy to debug. A legal broadcast on the wrong axis is more dangerous, so assertions should state meaning, not only final size.

A score grid is repeated dot products

The delivery predictor used a dot product to combine features with weights. The same operation can compare vectors to one another.

For three token-feature vectors in one ticket, multiply the token matrix by its transpose:

$$[T, D] \; @ \; [D, T] \rightarrow [T, T]$$

Every entry in the output is one dot product between two token positions.

These are raw alignment scores, not automatically cosine similarities. A dot product changes when vector magnitude changes; cosine similarity first divides by vector lengths.

Later, attention will create separate query and key vectors, scale the Q @ K.T score grid, and convert each row into weights that sum to one.^[3] You don't need attention mechanics yet. For now, recognize its shape as many dot products arranged in a matrix.

Build it: trace one tiny ticket projection

The lab below packages the shape reasoning into a guardrail. It accepts a batch of token-feature sequences and a projection matrix. Before multiplying, it checks that the hidden states have three named axes, the projection is a matrix, and the feature size agrees with the matrix input size.

These handcrafted sequences have three meaningful token positions each. Real padded text batches need masked pooling: plain .mean(axis=1) would let padding positions change the summaries.

Real token vectors won't be hand written, and the model's learned projections will be much wider. The invariant stays small enough to say aloud: preserve batch and sequence; match and transform feature size.

PyTorch keeps the same shape reasoning

NumPy is excellent for seeing the algebra. PyTorch carries the same tensor operations into trainable models and automatic differentiation.^[4] Confirm that the same batch projection gives the same values in both libraries:

This is enough PyTorch for this chapter. Device placement, mixed precision, training modules, and full attention blocks each need their own careful lesson.

Practice with shapes before code

Use the conventions from this chapter: rows represent orders or token positions; the final dimension represents features.

1. A delivery feature vector has shape [3], and a weight vector has shape [3]. What does x @ w return?
2. A training table X has shape [20, 3], and W has shape [3, 4]. What does X @ W mean and what shape does it produce?
3. Ticket hidden states have shape [8, 12, 64], and a projection has shape [64, 16]. Which axes remain unchanged?
4. Scores have shape [20, 4]. Why can adding a value of shape [20, 1] be a silent bug when you meant a four-feature bias?
5. Token vectors have shape [12, 64]. What transpose is needed to create a pairwise score grid?

Solution sketches

1. It returns a scalar because matching feature positions are multiplied and summed.
2. It applies one four-output transformation to each of 20 rows, producing [20, 4].
3. Batch size 8 and sequence length 12 stay unchanged; feature size changes from 64 to 16, giving [8, 12, 16].
4. [20, 1] legally broadcasts one different value across all four output columns in each row. The output shape looks right even though the adjustment is attached to orders instead of features.
5. Use the transpose [64, 12]; then [12, 64] @ [64, 12] produces [12, 12].

What you are ready for next

You can now read a model operation as a contract:

You should be able to predict a multiplication output before running it, catch an invalid meeting dimension, and explain why a legal broadcast might still corrupt meaning.

Mastery check

Key concepts

• A vector is an ordered list whose positions carry agreed meaning.
• A dot product turns two matching vectors into one weighted prediction or similarity score.
• A matrix can hold many rows or map input features into new output features.
• Matrix multiplication succeeds only when the meeting feature dimensions agree.
• A tensor adds named axes such as batch and sequence while preserving feature-vector reasoning.
• Broadcasting needs semantic axis checks because incorrect operations can still run.
• PyTorch uses the same shape reasoning that you practiced in NumPy.

Evaluation rubric

• Foundational: Calculates the two-feature delivery prediction by hand and identifies vector, scalar, and matrix shapes.
• Intermediate: Traces X @ W and [B, T, D] @ [D, P] contracts before executing code.
• Advanced: Diagnoses both a hard dimension mismatch and a silent broadcasting bug, then implements guarded ticket projection in NumPy and confirms it with PyTorch.

Follow-up questions

Common pitfalls

Swapping feature order

Symptom: A prediction is numerically plausible but wrong for known examples. Cause: Feature and weight vectors have the same length but disagree about position meanings. Fix: Name feature order once and assert it at data boundaries.

Transposing until code runs

Symptom: A shape error disappears, but outputs no longer mean what the model intended. Cause: A transpose was used as a guess rather than from the row/column contract. Fix: Label each axis, then transpose only when the operation requires pairwise rows, such as a similarity grid.

Trusting a legal broadcast

Symptom: Results are shifted across entire orders or token positions. Cause: A singleton axis expanded in a valid but unintended direction. Fix: Assert the bias or mask shape before arithmetic and test a row with hand-computed expected values.

Losing the feature axis

Symptom: A token projection fails after batching or reshaping. Cause: The last axis no longer represents the feature size expected by the projection. Fix: Print or assert [batch, sequence, feature] immediately before multiplication.

Averaging padding positions

Symptom: Ticket summaries change when shorter tickets receive more padding. Cause: A plain sequence-axis mean treats padding positions like meaningful tokens. Fix: Use a padding mask and divide by the number of real tokens.