Neural Networks from Scratch

The previous lesson asked whether an observed model improvement was convincing. This lesson moves one step earlier in the pipeline: where does a model score come from?

Imagine a delivery operations team. Orders arrive with numerical signals such as route disruption and warehouse backlog. Before anyone can measure accuracy or compare models, a network must transform those signals into a delay-risk score that can trigger manual review.

Earlier math lessons gave you the notation:

We will build only the forward pass: input to score. Training will eventually choose the weights from labeled examples. Evaluation will tell you whether trained scores deserve trust.

Core idea: A dense neural network alternates affine maps, W @ x + b, with non-linear activations. The matrices move information; the activations keep many layers from collapsing into one linear rule.

One neuron produces one score

A neuron is the smallest useful scoring unit. For feature vector $x$, weights $w$, and bias $b$, it computes:

$$z = w^\top x + b$$

The intermediate value $z$ is a logit or pre-activation. Apply a function such as the sigmoid to map it into the open interval (0, 1):

$$\sigma(z) = \frac{1}{1 + e^{-z}}$$

The resulting number is a convenient score. It becomes a probability claim only after the model has been trained for that purpose and checked on held-out data.

Consider two inputs on a zero-to-ten operational scale:

With bias -4.0, the logit is 4.2 + 2.4 - 4.0 = 2.6; sigmoid maps it to 0.931. These are fixed demonstration parameters, not learned evidence that the order truly has a 93.1% delay probability.

A bias shifts the intervention threshold without changing feature values. Keep the same weights and compare a quieter order with the high-risk order:

The weighted-sum pattern reaches back to early perceptron models.^[1] Modern networks replace a single thresholding unit with many differentiable layers, but the local calculation remains recognizable.

Non-linearity is what makes depth useful

Suppose two layers contain only weighted sums and biases:

$$h = W_1x + b_1$$ $$y = W_2h + b_2$$

Substitute the first equation into the second:

$$y = (W_2W_1)x + (W_2b_1 + b_2)$$

Two affine layers have collapsed into one affine layer. The same is true for twenty layers. A non-linear activation placed between layers prevents that collapse, letting a network represent curved or piecewise decision boundaries.^[2]

Three functions matter here:

ReLU became a practical hidden-layer choice because its positive side doesn't saturate like sigmoid; its negative side can still produce inactive units.^[3] GELU is a smooth alternative that weights inputs by their values rather than gating only by sign.^[4]

Compute the bends rather than memorizing their names:

Here is the collapse as executable evidence. Without ReLU, two layers match one merged affine map exactly. With ReLU inserted, they no longer match.

The first hidden pre-activation is negative in this trace. ReLU clips that value to zero, so the activated network no longer matches the merged affine map.

A network is many neurons arranged as layers

One neuron emits one score. A dense layer emits a vector of scores by giving each output unit its own row of weights:

$$z = Wx + b$$

For three input features and four hidden units, W has shape (4, 3), x has shape (3,), and z has shape (4,). Matrix multiplication computes all four neurons at once.

Trace a tiny forward pass

Now use a two-feature, two-hidden-unit network. Its parameters are deliberately small enough to calculate by hand:

$$W_1 = \begin{bmatrix} 0.3 & -0.1 \\ 0.1 & 0.6 \end{bmatrix}, \quad b_1 = \begin{bmatrix} 0.0 \\ 0.0 \end{bmatrix}$$

For x = [2.0, 5.0], the hidden pre-activation is:

$$z_1 = W_1x + b_1 = [0.1, 3.2]$$

Both entries are positive, so ReLU leaves them unchanged: h = [0.1, 3.2]. The output parameters are w_2 = [0.4, 0.1] and b_2 = -0.5, giving:

$$z_2 = 0.4(0.1) + 0.1(3.2) - 0.5 = -0.14$$ $$\text{score} = \sigma(-0.14) \approx 0.465$$

Batch several orders through the same network

Production code rarely scores one order at a time. Put one order in each row of X. With row-major batches, compute X @ W1.T because each row of W1 describes one hidden unit.

Shape checks aren't busywork. An accidental transpose can either fail loudly or silently attach the wrong meaning to a dimension.

Parameters are stored decisions

Each weight and bias is one parameter. A dense layer with n_in inputs and n_out outputs has:

$$n_{\text{out}} \times n_{\text{in}} + n_{\text{out}}$$

parameters: one weight per connection plus one bias per output unit.

For our 2 -> 2 -> 1 network:

During training, labels and an optimization procedure adjust these parameters. At inference time, the network applies the stored parameters to new input values. New prompt tokens, retrieved context, or sensor readings are inputs; they aren't new weights.

Failure case: scales can overpower meaning

Suppose you replace route disruption with order value in dollars and feed raw values into a neuron:

$$z = 0.3 \cdot \text{backlog} + 0.3 \cdot \text{order value} - 2$$

Equal coefficients don't mean equal influence when one feature ranges from 0 to 10 and another ranges from 10 to 1000. A large measurement scale can dominate an initial weighted sum and make optimization harder.

This doesn't prove standardization is always required. It gives you a diagnostic question: do input magnitudes reflect meaning, or merely units such as dollars, milliseconds, and counts?

Failure case: naive sigmoid can overflow

Forward-pass code also needs numerical care. The formula 1 / (1 + exp(-z)) is harmless for z = 2.6, but for a very negative logit, exp(-z) can exceed floating-point range.

Use an algebraically equivalent branch:

$$\sigma(z) = \begin{cases} 1/(1+e^{-z}) & z \ge 0 \\ e^z/(1+e^z) & z < 0 \end{cases}$$

Build an inspectable delay-risk report

This final forward pass combines five engineering habits:

1. Standardize heterogeneous inputs with saved statistics.
2. Assert shapes before multiplying.
3. Apply a hidden non-linearity.
4. Use a numerically stable output activation.
5. Present scores as scores until validation supports probability language.

The parameters below are still illustrative. The point is an auditable computation path, not a trained delivery model.

What approximation theorems leave open

A classic result shows that a feed-forward network with one sufficiently wide hidden layer and suitable non-linear activation can approximate broad classes of target functions arbitrarily well.^[5][6]

That is a statement about representation: useful parameters can exist. It doesn't say:

• gradient-based training will find them;
• a finite dataset identifies the right behavior;
• a model is calibrated or safe to deploy;
• one architecture is efficient for images, sequences, or language.

Those remaining questions motivate the rest of the curriculum: convolution adds structure for spatial data, training turns parameters into learned behavior, and evaluation tests whether behavior generalizes.

Practice

1. In one-neuron-score.py, change the bias from -4.0 to -6.0. Explain why both orders receive lower scores even though their inputs don't change.
2. Extend batch-forward-pass.py with a third feature and update W1. Write each expected shape before executing the code.
3. Remove np.maximum from the hidden layer in delay-risk-report.py. Derive the single merged affine map and verify its logits match the two-layer version without ReLU.
4. Add a second output for lost package risk. Decide whether independent sigmoid scores or a softmax category fits the operational question, and state the assumption.

Mastery check

Key concepts

• weighted sum, bias, logit, and sigmoid score
• affine maps and dense-layer shapes
• ReLU and GELU non-linearities
• single-example and batched forward passes
• parameter counting
• feature scaling and stable sigmoid implementation
• representation versus training and calibration

Evaluation rubric

• Foundational: Computes one neuron's logit and sigmoid score, then explains why the score isn't automatically a probability.
• Intermediate: Traces a dense ReLU forward pass for one example and a batch while stating each tensor shape.
• Advanced: Demonstrates linear-layer collapse, counts parameters, and diagnoses scaling or numerical-stability failures before deployment.

Common pitfalls

• Symptom: A hand-selected sigmoid output is reported as a delay probability. Cause: output range was confused with calibration. Fix: call it a score until held-out calibration supports probability language.
• Symptom: Adding more affine layers doesn't improve representational power. Cause: no hidden non-linearity was applied. Fix: insert and trace an activation such as ReLU or GELU.
• Symptom: Dollar-valued features swamp smaller operational scores. Cause: coefficients were compared without checking feature units. Fix: inspect distributions and scale features when the model contract requires it.
• Symptom: Scoring extreme logits crashes or produces warnings. Cause: naive sigmoid exponentiation overflowed. Fix: use a numerically stable branch or library primitive.

Follow-up questions