LearnPreparation & PrerequisitesNeural Networks from Scratch

📝EasyNLP Fundamentals

Neural Networks from Scratch

Trace an incident-risk network from one neuron to a batched NumPy forward pass, then diagnose activation, shape, scale, and numerical-stability failures.

12 min read

Learning path

Step 18 of 177 in the full curriculum

Hypothesis Tests, Intervals, and pass@k CNNs from Scratch

Personalize this lesson

Adapt explanations and teaching visuals to your background and preferred voice.

A model improvement is only meaningful after there is a model score to measure. Start one step earlier in the pipeline: where does a model score come from?

An incident-routing team receives alerts with numerical signals such as auth failure rate and queue backlog. Before anyone can measure accuracy or compare models, a network must transform those signals into an escalation-risk score that can trigger manual review.

Earlier math lessons gave you the notation:

Object	Meaning here
vector `x`	features for one incident
matrix `W`	weights connecting all inputs to all units in a layer
batch matrix `X`	several incidents sent through the same network at once

You'll build only the forward pass: input to score. Training will eventually choose the weights from labeled examples. Evaluation will tell you whether trained scores deserve trust.

Core idea: A dense neural network alternates affine maps, W @ x + b, with non-linear activations. The matrices move information; the activations keep many layers from collapsing into one linear rule.

One neuron produces one score

A neuron is the smallest useful scoring unit. For feature vector $x$ , weights $w$ , and bias $b$ , it computes:

z = w^\top x + b

The intermediate value $z$ is a logit or pre-activation. Apply a function such as the sigmoid to map it into the open interval (0, 1):

\sigma(z) = \frac{1}{1 + e^{-z}}

The resulting number is a convenient score. It becomes a probability claim only after the model has been trained for that purpose and checked on held-out data.

Consider two inputs on a zero-to-ten operational scale:

Feature	Value	Chosen weight	Contribution
Route disruption score	`6`	`0.7`	`4.2`
Queue backlog score	`8`	`0.3`	`2.4`

With bias -4.0, the logit is 4.2 + 2.4 - 4.0 = 2.6; sigmoid maps it to 0.931. These are fixed demonstration parameters, not learned evidence that the run truly has a 93.1% breach probability.

one-neuron-score.py

import numpy as np

x = np.array([6.0, 8.0])           # disruption, backlog
w = np.array([0.7, 0.3])
b = -4.0

z = float(x @ w + b)
score = 1.0 / (1.0 + np.exp(-z))

print(f"logit:         {z:.3f}")
print(f"sigmoid score: {score:.3f}")

Output

logit:         2.600
sigmoid score: 0.931

Single escalation-risk neuron showing disruption six times weight zero point seven contributes four point two, queue backlog eight times weight zero point three contributes two point four, bias subtracts four, logit is two point six, and sigmoid score is zero point nine three one. — The forward pass is explicit: feature times weight contributions plus bias create the logit, then sigmoid maps that logit to a score. The score comes from chosen parameters, not calibration evidence.

A bias shifts the intervention threshold without changing feature values. Keep the same weights and compare a quieter run with the high-risk run:

bias-shifts-threshold.py

import numpy as np

w = np.array([0.7, 0.3])
b = -4.0
runs = {
    "quiet route": np.array([2.0, 3.0]),
    "disrupted route": np.array([6.0, 8.0]),
}

for name, x in runs.items():
    z = float(x @ w + b)
    score = 1.0 / (1.0 + np.exp(-z))
    print(f"{name:15s} logit={z:5.2f} score={score:.3f}")

Output

quiet route     logit=-1.70 score=0.154
disrupted route logit= 2.60 score=0.931

The weighted-sum pattern reaches back to early perceptron models.^{[1]Reference 1The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain.https://doi.org/10.1037/h0042519} Modern networks replace a single thresholding unit with many differentiable layers, but the local calculation remains recognizable.

Non-linearity is what makes depth useful

Suppose two layers contain only weighted sums and biases:

h = W_1x + b_1

y = W_2h + b_2

Substitute the first equation into the second:

y = (W_2W_1)x + (W_2b_1 + b_2)

Two affine layers have collapsed into one affine layer. The same is true for twenty layers. A non-linear activation placed between layers prevents that collapse, letting a network represent curved or piecewise decision boundaries.^{[2]Reference 2Deep Learning.https://www.deeplearningbook.org/}

Three functions matter here:

Activation	Formula	Useful intuition
Sigmoid	$\sigma(z)=1/(1+e^{-z})$	Maps a final binary-classification logit to a score in `(0, 1)`
ReLU	$\max(0,z)$	Passes positive evidence and clips negative pre-activations to zero
GELU	$z\Phi(z)$	Smoothly gates values using a Gaussian cumulative distribution

ReLU became a practical hidden-layer choice because its positive side doesn't saturate like sigmoid; its negative side can still produce inactive units.^{[3]Reference 3Rectified Linear Units Improve Restricted Boltzmann Machines.https://www.cs.toronto.edu/~fritz/absps/reluICML.pdf} GELU is a smooth alternative that weights inputs by their values rather than gating only by sign.^{[4]Reference 4Gaussian Error Linear Units (GELUs).https://arxiv.org/abs/1606.08415}

Comparison of linear, ReLU, sigmoid, and GELU functions from z equals minus three to plus three. Linear stays straight, ReLU creates a corner at zero, sigmoid bends into a bounded S curve, and GELU smoothly suppresses negative inputs. — Shape determines the role: linear leaves stacked layers collapsible, ReLU creates a hard corner, sigmoid maps to a bounded score, and GELU smoothly gates negative inputs without an upper bound.

Compute the bends rather than memorizing their names:

activation-values.py

from math import erf, sqrt
import numpy as np

z = np.array([-2.0, -0.5, 0.0, 1.0, 2.0])
relu = np.maximum(0.0, z)
sigmoid = 1.0 / (1.0 + np.exp(-z))
gelu = np.array([value * 0.5 * (1.0 + erf(value / sqrt(2.0))) for value in z])

for value, r, s, g in zip(z, relu, sigmoid, gelu):
    print(f"z={value:4.1f}  relu={r:5.3f}  sigmoid={s:5.3f}  gelu={g:6.3f}")

Output

z=-2.0  relu=0.000  sigmoid=0.119  gelu=-0.046
z=-0.5  relu=0.000  sigmoid=0.378  gelu=-0.154
z= 0.0  relu=0.000  sigmoid=0.500  gelu= 0.000
z= 1.0  relu=1.000  sigmoid=0.731  gelu= 0.841
z= 2.0  relu=2.000  sigmoid=0.881  gelu= 1.954

The collapse is executable evidence. Without ReLU, two layers match one merged affine map exactly. With ReLU inserted, they no longer match.

activation-prevents-collapse.py

import numpy as np

x = np.array([2.0, 1.0])
w1 = np.array([[1.0, -2.0], [0.5, 1.0]])
b1 = np.array([-0.5, 0.25])
w2 = np.array([[0.4, -0.8]])
b2 = np.array([0.1])

two_affine = w2 @ (w1 @ x + b1) + b2
merged_affine = (w2 @ w1) @ x + (w2 @ b1 + b2)
with_relu = w2 @ np.maximum(0.0, w1 @ x + b1) + b2

print("affine equals merged:", np.allclose(two_affine, merged_affine))
print(f"two affine output:    {float(two_affine[0]):.3f}")
print(f"with ReLU output:     {float(with_relu[0]):.3f}")

Output

affine equals merged: True
two affine output:    -1.900
with ReLU output:     -1.700

The first hidden pre-activation is negative in this trace. ReLU clips that value to zero, so the activated network no longer matches the merged affine map.

A network is many neurons arranged as layers

One neuron emits one score. A dense layer emits a vector of scores by giving each output unit its own row of weights:

z = Wx + b

For three input features and four hidden units, W has shape (4, 3), x has shape (3,), and z has shape (4,). Matrix multiplication computes all four neurons at once.

This layout matches PyTorch nn.Linear: weight is stored as (n_out, n_in), so a single example uses $z = Wx + b$ . The vectors and matrices lesson used the other common convention, with $W$ as (n_in, n_out) and batch multiply X @ W (no transpose). Both are correct; mix them carelessly and you get wrong units even when shapes still multiply.

Layout	$W$ shape	Single example	Row-major batch
Matrices lesson	`(n_in, n_out)`	row vector times $W$	`X @ W`
This lesson / `nn.Linear`	`(n_out, n_in)`	$Wx$	`X @ W.T`

This lesson stays on the PyTorch layout. When you see .T on weights, check which convention the surrounding code uses before copying a formula.

Grouped dense-network diagram with three input values entering one four-by-three affine layer, then four hidden units showing both pre-activation and ReLU values. The negative second unit is blocked; the three active hidden paths combine into output score 0.68. — Grouping the dense matrix operation removes crossing-wire clutter without hiding the network: three inputs produce four hidden units, ReLU blocks only the negative second unit, and three active paths reach the output.

Trace a tiny forward pass

Now use a two-feature, two-hidden-unit network. Its parameters are deliberately small enough to calculate by hand:

W_1 = \begin{bmatrix} 0.3 & -0.1 \\ 0.1 & 0.6 \end{bmatrix}, \quad b_1 = \begin{bmatrix} 0.0 \\ 0.0 \end{bmatrix}

For x = [2.0, 5.0], the hidden pre-activation is:

z_1 = W_1x + b_1 = [0.1, 3.2]

Both entries are positive, so ReLU leaves them unchanged: h = [0.1, 3.2]. The output parameters are w_2 = [0.4, 0.1] and b_2 = -0.5, giving:

z_2 = 0.4(0.1) + 0.1(3.2) - 0.5 = -0.14

\text{score} = \sigma(-0.14) \approx 0.465

Left-to-right trace of a two-input, two-hidden-unit neural network. Inputs 2 and 5 produce hidden logits 0.10 and 3.20, ReLU keeps both values, output weights and bias produce logit minus 0.14, and sigmoid produces score 0.465. — Each boundary records its operation, result, and shape. ReLU leaves both positive hidden logits unchanged; the output bias then turns the positive contributions into a negative logit and a score below one-half.

trace-one-forward-pass.py

import numpy as np

x = np.array([2.0, 5.0])
w1 = np.array([[0.3, -0.1], [0.1, 0.6]])
b1 = np.array([0.0, 0.0])
w2 = np.array([0.4, 0.1])
b2 = -0.5

z1 = w1 @ x + b1
h = np.maximum(0.0, z1)
z2 = float(w2 @ h + b2)
score = 1.0 / (1.0 + np.exp(-z2))

print("hidden pre-activation:", z1)
print("hidden after ReLU:    ", h)
print(f"output logit:          {z2:.3f}")
print(f"sigmoid score:         {score:.3f}")

Output

hidden pre-activation: [0.1 3.2]
hidden after ReLU:     [0.1 3.2]
output logit:          -0.140
sigmoid score:         0.465

Batch several runs through the same network

Production code rarely scores one run at a time. Put one run in each row of X. With row-major batches, compute X @ W1.T because each row of W1 describes one hidden unit.

batch-forward-pass.py

import numpy as np

x = np.array([
    [2.0, 5.0],
    [6.0, 8.0],
    [1.0, 1.0],
    [8.0, 2.0],
])
w1 = np.array([[0.3, -0.1], [0.1, 0.6]])
b1 = np.array([0.0, 0.0])
w2 = np.array([0.4, 0.1])
b2 = -0.5

h = np.maximum(0.0, x @ w1.T + b1)
logits = h @ w2 + b2
scores = 1.0 / (1.0 + np.exp(-logits))

print("input shape: ", x.shape)
print("hidden shape:", h.shape)
print("score shape: ", scores.shape)
print("scores:", np.round(scores, 3))

Output

input shape:  (4, 2)
hidden shape: (4, 2)
score shape:  (4,)
scores: [0.465 0.608 0.413 0.641]

Shape checks aren't busywork. An accidental transpose can either fail loudly or silently attach the wrong meaning to a dimension.

Parameters are stored decisions

Each weight and bias is one parameter. A dense layer with n_in inputs and n_out outputs has:

n_{\text{out}} \times n_{\text{in}} + n_{\text{out}}

parameters: one weight per connection plus one bias per output unit.

For our 2 -> 2 -> 1 network:

Layer	Weights	Biases	Parameters
Input to hidden	`2 x 2 = 4`	`2`	`6`
Hidden to output	`2 x 1 = 2`	`1`	`3`
Total			`9`

count-dense-parameters.py

def dense_parameters(n_in: int, n_out: int) -> int:
    if n_in <= 0 or n_out <= 0:
        raise ValueError("dense dimensions must be positive")
    return n_in * n_out + n_out

tiny = dense_parameters(2, 2) + dense_parameters(2, 1)
image_classifier = (
    dense_parameters(784, 256)
    + dense_parameters(256, 128)
    + dense_parameters(128, 10)
)

print("2 -> 2 -> 1 parameters:", tiny)
print("784 -> 256 -> 128 -> 10 parameters:", f"{image_classifier:,}")
try:
    dense_parameters(0, 2)
except ValueError as error:
    print(error)

Output

2 -> 2 -> 1 parameters: 9
784 -> 256 -> 128 -> 10 parameters: 235,146
dense dimensions must be positive

During training, labels and an optimization procedure adjust these parameters. At inference time, the network applies the stored parameters to new input values. New prompt tokens, retrieved context, or sensor readings are inputs; they aren't new weights.

Failure case: scales can overpower meaning

Suppose you replace auth failure rate with daily log volume in megabytes and feed raw values into a neuron:

z = 0.3 \cdot \text{backlog} + 0.3 \cdot \text{log volume} - 2

Equal coefficients don't mean equal influence when one feature ranges from 0 to 10 and another ranges from 10 to 1000. A large measurement scale can dominate an initial weighted sum and make optimization harder.

feature-scale-failure.py

import numpy as np

raw_run = np.array([8.0, 900.0])       # backlog score, log volume in MB
raw_contributions = 0.3 * raw_run

mean = np.array([5.0, 500.0])
scale = np.array([2.0, 200.0])
standardized_run = (raw_run - mean) / scale
scaled_contributions = 0.3 * standardized_run

print("raw contributions:       ", raw_contributions)
print("standardized features:    ", standardized_run)
print("scaled contributions:     ", np.round(scaled_contributions, 2))

Output

raw contributions:        [  2.4 270. ]
standardized features:     [1.5 2. ]
scaled contributions:      [0.45 0.6 ]

This doesn't prove standardization is always required. It leaves a diagnostic question: do input magnitudes reflect meaning, or merely units such as megabytes, milliseconds, and counts?

Failure case: naive sigmoid can overflow

Forward-pass code also needs numerical care. The formula 1 / (1 + exp(-z)) is harmless for z = 2.6, but for a very negative logit, exp(-z) can exceed floating-point range.

Use an algebraically equivalent branch:

\sigma(z) = \begin{cases} 1/(1+e^{-z}) & z \ge 0 \\ e^z/(1+e^z) & z < 0 \end{cases}

stable-sigmoid.py

from math import exp

def stable_sigmoid(z: float) -> float:
    if z >= 0:
        return 1.0 / (1.0 + exp(-z))
    ez = exp(z)
    return ez / (1.0 + ez)

for logit in [-1000.0, -2.0, 0.0, 2.0, 1000.0]:
    print(f"logit={logit:7.1f} score={stable_sigmoid(logit):.6f}")

Output

logit=-1000.0 score=0.000000
logit=   -2.0 score=0.119203
logit=    0.0 score=0.500000
logit=    2.0 score=0.880797
logit= 1000.0 score=1.000000

Build an inspectable escalation-risk report

This final forward pass combines five engineering habits:

Standardize heterogeneous inputs with saved statistics.
Assert shapes before multiplying.
Apply a hidden non-linearity.
Use a numerically stable output activation.
Present scores as scores until validation supports probability language.

The parameters below are still illustrative. What matters here is an auditable computation path, not a trained incident-risk model.

escalation-risk-report.py

import numpy as np

def stable_sigmoid(values: np.ndarray) -> np.ndarray:
    scores = np.empty_like(values, dtype=float)
    nonnegative = values >= 0
    scores[nonnegative] = 1.0 / (1.0 + np.exp(-values[nonnegative]))
    exp_values = np.exp(values[~nonnegative])
    scores[~nonnegative] = exp_values / (1.0 + exp_values)
    return scores

feature_names = ["auth failure rate", "queue age", "retry rate"]
run_ids = ["A-104", "B-208", "C-311"]
raw = np.array([
    [2.0, 3.0, 1.0],
    [8.0, 7.0, 9.0],
    [5.0, 9.0, 4.0],
])
mean = np.array([5.0, 5.0, 5.0])
scale = np.array([2.0, 2.0, 2.0])
x = (raw - mean) / scale

w1 = np.array([
    [0.8, 0.3, 0.4],
    [-0.2, 0.9, 0.5],
    [0.4, -0.1, 0.7],
])
b1 = np.array([0.0, -0.2, 0.1])
w2 = np.array([0.9, 0.7, 0.6])
b2 = -0.3

assert x.shape[1] == len(feature_names) == w1.shape[1]
assert w1.shape[0] == b1.shape[0] == w2.shape[0]

hidden = np.maximum(0.0, x @ w1.T + b1)
logits = hidden @ w2 + b2
scores = stable_sigmoid(logits)

print("Escalation-risk scoring report")
for run_id, score in zip(run_ids, scores):
    queue = "review" if score >= 0.60 else "monitor"
    print(f"{run_id}: score={score:.3f} -> {queue}")

Output

Escalation-risk scoring report
A-104: score=0.426 -> monitor
B-208: score=0.981 -> review
C-311: score=0.732 -> review

What approximation theorems leave open

A classic result shows that a feed-forward network with one sufficiently wide hidden layer and suitable non-linear activation can approximate broad classes of target functions arbitrarily well.^{[5]Reference 5Multilayer Feedforward Networks are Universal Approximators.https://doi.org/10.1016/0893-6080(89)90020-8}^{[6]Reference 6Approximation by Superpositions of a Sigmoidal Function.https://doi.org/10.1007/BF02551274}

That's a statement about representation: useful parameters can exist. It doesn't say:

gradient-based training will find them;
a finite dataset identifies the right behavior;
a model is calibrated or safe to deploy;
one architecture is efficient for images, sequences, or language.

Those remaining questions motivate the rest of the curriculum: convolution adds structure for spatial data, training turns parameters into learned behavior, and evaluation tests whether behavior generalizes.

Practice

In one-neuron-score.py, change the bias from -4.0 to -6.0. Explain why both runs receive lower scores even though their inputs don't change.
Extend batch-forward-pass.py with a third feature and update W1. Write each expected shape before executing the code.
Remove np.maximum from the hidden layer in escalation-risk-report.py. Derive the single merged affine map and verify its logits match the two-layer version without ReLU.
Add a second output for timeout risk. Decide whether independent sigmoid scores or a softmax category fits the operational question, and state the assumption.

Mastery check

Key concepts

weighted sum, bias, logit, and sigmoid score
affine maps and dense-layer shapes
ReLU and GELU non-linearities
single-example and batched forward passes
parameter counting
feature scaling and stable sigmoid implementation
representation versus training and calibration

Evaluation rubric

Foundational: Computes one neuron's logit and sigmoid score, then explains why the score isn't automatically a probability.
Intermediate: Traces a dense ReLU forward pass for one example and a batch while stating each tensor shape.
Advanced: Demonstrates linear-layer collapse, counts parameters, and diagnoses scaling or numerical-stability failures before deployment.

Common pitfalls

Symptom: A hand-selected sigmoid output is reported as an escalation probability. Cause: output range was confused with calibration. Fix: call it a score until held-out calibration supports probability language.
Symptom: Adding more affine layers doesn't improve representational power. Cause: no hidden non-linearity was applied. Fix: insert and trace an activation such as ReLU or GELU.
Symptom: Byte-volume features swamp smaller operational scores. Cause: coefficients were compared without checking feature units. Fix: inspect distributions and scale features when the model contract requires it.
Symptom: Scoring extreme logits crashes or produces warnings. Cause: naive sigmoid exponentiation overflowed. Fix: use a numerically stable branch or library primitive.

Follow-up questions

Next Step

Continue to CNNs from Scratch

Dense forward-pass intuition now becomes a spatial model for images, where shared kernels preserve local structure.

PreviousHypothesis Tests, Intervals, and pass@k

Share this article

X Facebook LinkedIn Bluesky Reddit Hacker News Email

References

The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain.

Rosenblatt, F. · 1958 · Psychological Review, 65(6)

https://doi.org/10.1037/h0042519

Deep Learning.

Goodfellow, I., Bengio, Y., Courville, A. · 2016

https://www.deeplearningbook.org/

Rectified Linear Units Improve Restricted Boltzmann Machines.

Nair, V., Hinton, G. E. · 2010 · ICML 2010

https://www.cs.toronto.edu/~fritz/absps/reluICML.pdf

Gaussian Error Linear Units (GELUs).

Hendrycks, D., Gimpel, K. · 2016

https://arxiv.org/abs/1606.08415

Multilayer Feedforward Networks are Universal Approximators.

Hornik, K., Stinchcombe, M., White, H. · 1989 · Neural Networks, 2(5)

https://doi.org/10.1016/0893-6080(89)90020-8

Approximation by Superpositions of a Sigmoidal Function.

Cybenko, G. · 1989 · Mathematics of Control, Signals and Systems, 2(4)

https://doi.org/10.1007/BF02551274

Discussion

Questions and insights from fellow learners.

Discussion loads when you reach this section.

Back to Topics

LearnPreparation & PrerequisitesNeural Networks from Scratch

📝EasyNLP Fundamentals

Neural Networks from Scratch

Trace an incident-risk network from one neuron to a batched NumPy forward pass, then diagnose activation, shape, scale, and numerical-stability failures.

12 min read

Learning path

Step 18 of 177 in the full curriculum

Hypothesis Tests, Intervals, and pass@k CNNs from Scratch

Personalize this lesson

Adapt explanations and teaching visuals to your background and preferred voice.

A model improvement is only meaningful after there is a model score to measure. Start one step earlier in the pipeline: where does a model score come from?

Earlier math lessons gave you the notation:

Object	Meaning here
vector `x`	features for one incident
matrix `W`	weights connecting all inputs to all units in a layer
batch matrix `X`	several incidents sent through the same network at once

You'll build only the forward pass: input to score. Training will eventually choose the weights from labeled examples. Evaluation will tell you whether trained scores deserve trust.

Core idea: A dense neural network alternates affine maps, W @ x + b, with non-linear activations. The matrices move information; the activations keep many layers from collapsing into one linear rule.

One neuron produces one score

A neuron is the smallest useful scoring unit. For feature vector $x$ , weights $w$ , and bias $b$ , it computes:

z = w^\top x + b

The intermediate value $z$ is a logit or pre-activation. Apply a function such as the sigmoid to map it into the open interval (0, 1):

\sigma(z) = \frac{1}{1 + e^{-z}}

The resulting number is a convenient score. It becomes a probability claim only after the model has been trained for that purpose and checked on held-out data.

Consider two inputs on a zero-to-ten operational scale:

Feature	Value	Chosen weight	Contribution
Route disruption score	`6`	`0.7`	`4.2`
Queue backlog score	`8`	`0.3`	`2.4`

With bias -4.0, the logit is 4.2 + 2.4 - 4.0 = 2.6; sigmoid maps it to 0.931. These are fixed demonstration parameters, not learned evidence that the run truly has a 93.1% breach probability.

one-neuron-score.py

import numpy as np

x = np.array([6.0, 8.0])           # disruption, backlog
w = np.array([0.7, 0.3])
b = -4.0

z = float(x @ w + b)
score = 1.0 / (1.0 + np.exp(-z))

print(f"logit:         {z:.3f}")
print(f"sigmoid score: {score:.3f}")

Output

logit:         2.600
sigmoid score: 0.931

A bias shifts the intervention threshold without changing feature values. Keep the same weights and compare a quieter run with the high-risk run:

bias-shifts-threshold.py

import numpy as np

w = np.array([0.7, 0.3])
b = -4.0
runs = {
    "quiet route": np.array([2.0, 3.0]),
    "disrupted route": np.array([6.0, 8.0]),
}

for name, x in runs.items():
    z = float(x @ w + b)
    score = 1.0 / (1.0 + np.exp(-z))
    print(f"{name:15s} logit={z:5.2f} score={score:.3f}")

Output

quiet route     logit=-1.70 score=0.154
disrupted route logit= 2.60 score=0.931

Non-linearity is what makes depth useful

Suppose two layers contain only weighted sums and biases:

h = W_1x + b_1

y = W_2h + b_2

Substitute the first equation into the second:

y = (W_2W_1)x + (W_2b_1 + b_2)

Three functions matter here:

Activation	Formula	Useful intuition
Sigmoid	$\sigma(z)=1/(1+e^{-z})$	Maps a final binary-classification logit to a score in `(0, 1)`
ReLU	$\max(0,z)$	Passes positive evidence and clips negative pre-activations to zero
GELU	$z\Phi(z)$	Smoothly gates values using a Gaussian cumulative distribution

Compute the bends rather than memorizing their names:

activation-values.py

from math import erf, sqrt
import numpy as np

z = np.array([-2.0, -0.5, 0.0, 1.0, 2.0])
relu = np.maximum(0.0, z)
sigmoid = 1.0 / (1.0 + np.exp(-z))
gelu = np.array([value * 0.5 * (1.0 + erf(value / sqrt(2.0))) for value in z])

for value, r, s, g in zip(z, relu, sigmoid, gelu):
    print(f"z={value:4.1f}  relu={r:5.3f}  sigmoid={s:5.3f}  gelu={g:6.3f}")

Output

z=-2.0  relu=0.000  sigmoid=0.119  gelu=-0.046
z=-0.5  relu=0.000  sigmoid=0.378  gelu=-0.154
z= 0.0  relu=0.000  sigmoid=0.500  gelu= 0.000
z= 1.0  relu=1.000  sigmoid=0.731  gelu= 0.841
z= 2.0  relu=2.000  sigmoid=0.881  gelu= 1.954

The collapse is executable evidence. Without ReLU, two layers match one merged affine map exactly. With ReLU inserted, they no longer match.

activation-prevents-collapse.py

import numpy as np

x = np.array([2.0, 1.0])
w1 = np.array([[1.0, -2.0], [0.5, 1.0]])
b1 = np.array([-0.5, 0.25])
w2 = np.array([[0.4, -0.8]])
b2 = np.array([0.1])

two_affine = w2 @ (w1 @ x + b1) + b2
merged_affine = (w2 @ w1) @ x + (w2 @ b1 + b2)
with_relu = w2 @ np.maximum(0.0, w1 @ x + b1) + b2

print("affine equals merged:", np.allclose(two_affine, merged_affine))
print(f"two affine output:    {float(two_affine[0]):.3f}")
print(f"with ReLU output:     {float(with_relu[0]):.3f}")

Output

affine equals merged: True
two affine output:    -1.900
with ReLU output:     -1.700

The first hidden pre-activation is negative in this trace. ReLU clips that value to zero, so the activated network no longer matches the merged affine map.

A network is many neurons arranged as layers

One neuron emits one score. A dense layer emits a vector of scores by giving each output unit its own row of weights:

z = Wx + b

For three input features and four hidden units, W has shape (4, 3), x has shape (3,), and z has shape (4,). Matrix multiplication computes all four neurons at once.

Layout	$W$ shape	Single example	Row-major batch
Matrices lesson	`(n_in, n_out)`	row vector times $W$	`X @ W`
This lesson / `nn.Linear`	`(n_out, n_in)`	$Wx$	`X @ W.T`

This lesson stays on the PyTorch layout. When you see .T on weights, check which convention the surrounding code uses before copying a formula.

Trace a tiny forward pass

Now use a two-feature, two-hidden-unit network. Its parameters are deliberately small enough to calculate by hand:

W_1 = \begin{bmatrix} 0.3 & -0.1 \\ 0.1 & 0.6 \end{bmatrix}, \quad b_1 = \begin{bmatrix} 0.0 \\ 0.0 \end{bmatrix}

For x = [2.0, 5.0], the hidden pre-activation is:

z_1 = W_1x + b_1 = [0.1, 3.2]

Both entries are positive, so ReLU leaves them unchanged: h = [0.1, 3.2]. The output parameters are w_2 = [0.4, 0.1] and b_2 = -0.5, giving:

z_2 = 0.4(0.1) + 0.1(3.2) - 0.5 = -0.14

\text{score} = \sigma(-0.14) \approx 0.465

trace-one-forward-pass.py

import numpy as np

x = np.array([2.0, 5.0])
w1 = np.array([[0.3, -0.1], [0.1, 0.6]])
b1 = np.array([0.0, 0.0])
w2 = np.array([0.4, 0.1])
b2 = -0.5

z1 = w1 @ x + b1
h = np.maximum(0.0, z1)
z2 = float(w2 @ h + b2)
score = 1.0 / (1.0 + np.exp(-z2))

print("hidden pre-activation:", z1)
print("hidden after ReLU:    ", h)
print(f"output logit:          {z2:.3f}")
print(f"sigmoid score:         {score:.3f}")

Output

hidden pre-activation: [0.1 3.2]
hidden after ReLU:     [0.1 3.2]
output logit:          -0.140
sigmoid score:         0.465

Batch several runs through the same network

Production code rarely scores one run at a time. Put one run in each row of X. With row-major batches, compute X @ W1.T because each row of W1 describes one hidden unit.

batch-forward-pass.py

import numpy as np

x = np.array([
    [2.0, 5.0],
    [6.0, 8.0],
    [1.0, 1.0],
    [8.0, 2.0],
])
w1 = np.array([[0.3, -0.1], [0.1, 0.6]])
b1 = np.array([0.0, 0.0])
w2 = np.array([0.4, 0.1])
b2 = -0.5

h = np.maximum(0.0, x @ w1.T + b1)
logits = h @ w2 + b2
scores = 1.0 / (1.0 + np.exp(-logits))

print("input shape: ", x.shape)
print("hidden shape:", h.shape)
print("score shape: ", scores.shape)
print("scores:", np.round(scores, 3))

Output

input shape:  (4, 2)
hidden shape: (4, 2)
score shape:  (4,)
scores: [0.465 0.608 0.413 0.641]

Shape checks aren't busywork. An accidental transpose can either fail loudly or silently attach the wrong meaning to a dimension.

Parameters are stored decisions

Each weight and bias is one parameter. A dense layer with n_in inputs and n_out outputs has:

n_{\text{out}} \times n_{\text{in}} + n_{\text{out}}

parameters: one weight per connection plus one bias per output unit.

For our 2 -> 2 -> 1 network:

Layer	Weights	Biases	Parameters
Input to hidden	`2 x 2 = 4`	`2`	`6`
Hidden to output	`2 x 1 = 2`	`1`	`3`
Total			`9`

count-dense-parameters.py

def dense_parameters(n_in: int, n_out: int) -> int:
    if n_in <= 0 or n_out <= 0:
        raise ValueError("dense dimensions must be positive")
    return n_in * n_out + n_out

tiny = dense_parameters(2, 2) + dense_parameters(2, 1)
image_classifier = (
    dense_parameters(784, 256)
    + dense_parameters(256, 128)
    + dense_parameters(128, 10)
)

print("2 -> 2 -> 1 parameters:", tiny)
print("784 -> 256 -> 128 -> 10 parameters:", f"{image_classifier:,}")
try:
    dense_parameters(0, 2)
except ValueError as error:
    print(error)

Output

2 -> 2 -> 1 parameters: 9
784 -> 256 -> 128 -> 10 parameters: 235,146
dense dimensions must be positive

Failure case: scales can overpower meaning

Suppose you replace auth failure rate with daily log volume in megabytes and feed raw values into a neuron:

z = 0.3 \cdot \text{backlog} + 0.3 \cdot \text{log volume} - 2

feature-scale-failure.py

import numpy as np

raw_run = np.array([8.0, 900.0])       # backlog score, log volume in MB
raw_contributions = 0.3 * raw_run

mean = np.array([5.0, 500.0])
scale = np.array([2.0, 200.0])
standardized_run = (raw_run - mean) / scale
scaled_contributions = 0.3 * standardized_run

print("raw contributions:       ", raw_contributions)
print("standardized features:    ", standardized_run)
print("scaled contributions:     ", np.round(scaled_contributions, 2))

Output

raw contributions:        [  2.4 270. ]
standardized features:     [1.5 2. ]
scaled contributions:      [0.45 0.6 ]

This doesn't prove standardization is always required. It leaves a diagnostic question: do input magnitudes reflect meaning, or merely units such as megabytes, milliseconds, and counts?

Failure case: naive sigmoid can overflow

Forward-pass code also needs numerical care. The formula 1 / (1 + exp(-z)) is harmless for z = 2.6, but for a very negative logit, exp(-z) can exceed floating-point range.

Use an algebraically equivalent branch:

\sigma(z) = \begin{cases} 1/(1+e^{-z}) & z \ge 0 \\ e^z/(1+e^z) & z < 0 \end{cases}

stable-sigmoid.py

from math import exp

def stable_sigmoid(z: float) -> float:
    if z >= 0:
        return 1.0 / (1.0 + exp(-z))
    ez = exp(z)
    return ez / (1.0 + ez)

for logit in [-1000.0, -2.0, 0.0, 2.0, 1000.0]:
    print(f"logit={logit:7.1f} score={stable_sigmoid(logit):.6f}")

Output

logit=-1000.0 score=0.000000
logit=   -2.0 score=0.119203
logit=    0.0 score=0.500000
logit=    2.0 score=0.880797
logit= 1000.0 score=1.000000

Build an inspectable escalation-risk report

This final forward pass combines five engineering habits:

Standardize heterogeneous inputs with saved statistics.
Assert shapes before multiplying.
Apply a hidden non-linearity.
Use a numerically stable output activation.
Present scores as scores until validation supports probability language.

The parameters below are still illustrative. What matters here is an auditable computation path, not a trained incident-risk model.

escalation-risk-report.py

import numpy as np

def stable_sigmoid(values: np.ndarray) -> np.ndarray:
    scores = np.empty_like(values, dtype=float)
    nonnegative = values >= 0
    scores[nonnegative] = 1.0 / (1.0 + np.exp(-values[nonnegative]))
    exp_values = np.exp(values[~nonnegative])
    scores[~nonnegative] = exp_values / (1.0 + exp_values)
    return scores

feature_names = ["auth failure rate", "queue age", "retry rate"]
run_ids = ["A-104", "B-208", "C-311"]
raw = np.array([
    [2.0, 3.0, 1.0],
    [8.0, 7.0, 9.0],
    [5.0, 9.0, 4.0],
])
mean = np.array([5.0, 5.0, 5.0])
scale = np.array([2.0, 2.0, 2.0])
x = (raw - mean) / scale

w1 = np.array([
    [0.8, 0.3, 0.4],
    [-0.2, 0.9, 0.5],
    [0.4, -0.1, 0.7],
])
b1 = np.array([0.0, -0.2, 0.1])
w2 = np.array([0.9, 0.7, 0.6])
b2 = -0.3

assert x.shape[1] == len(feature_names) == w1.shape[1]
assert w1.shape[0] == b1.shape[0] == w2.shape[0]

hidden = np.maximum(0.0, x @ w1.T + b1)
logits = hidden @ w2 + b2
scores = stable_sigmoid(logits)

print("Escalation-risk scoring report")
for run_id, score in zip(run_ids, scores):
    queue = "review" if score >= 0.60 else "monitor"
    print(f"{run_id}: score={score:.3f} -> {queue}")

Output

Escalation-risk scoring report
A-104: score=0.426 -> monitor
B-208: score=0.981 -> review
C-311: score=0.732 -> review

What approximation theorems leave open

That's a statement about representation: useful parameters can exist. It doesn't say:

gradient-based training will find them;
a finite dataset identifies the right behavior;
a model is calibrated or safe to deploy;
one architecture is efficient for images, sequences, or language.

Practice

In one-neuron-score.py, change the bias from -4.0 to -6.0. Explain why both runs receive lower scores even though their inputs don't change.
Extend batch-forward-pass.py with a third feature and update W1. Write each expected shape before executing the code.
Remove np.maximum from the hidden layer in escalation-risk-report.py. Derive the single merged affine map and verify its logits match the two-layer version without ReLU.
Add a second output for timeout risk. Decide whether independent sigmoid scores or a softmax category fits the operational question, and state the assumption.

Mastery check

Key concepts

weighted sum, bias, logit, and sigmoid score
affine maps and dense-layer shapes
ReLU and GELU non-linearities
single-example and batched forward passes
parameter counting
feature scaling and stable sigmoid implementation
representation versus training and calibration

Evaluation rubric

Foundational: Computes one neuron's logit and sigmoid score, then explains why the score isn't automatically a probability.
Intermediate: Traces a dense ReLU forward pass for one example and a batch while stating each tensor shape.
Advanced: Demonstrates linear-layer collapse, counts parameters, and diagnoses scaling or numerical-stability failures before deployment.

Common pitfalls

Symptom: A hand-selected sigmoid output is reported as an escalation probability. Cause: output range was confused with calibration. Fix: call it a score until held-out calibration supports probability language.
Symptom: Adding more affine layers doesn't improve representational power. Cause: no hidden non-linearity was applied. Fix: insert and trace an activation such as ReLU or GELU.
Symptom: Byte-volume features swamp smaller operational scores. Cause: coefficients were compared without checking feature units. Fix: inspect distributions and scale features when the model contract requires it.
Symptom: Scoring extreme logits crashes or produces warnings. Cause: naive sigmoid exponentiation overflowed. Fix: use a numerically stable branch or library primitive.

Follow-up questions

Next Step

Continue to CNNs from Scratch

Dense forward-pass intuition now becomes a spatial model for images, where shared kernels preserve local structure.

PreviousHypothesis Tests, Intervals, and pass@k

Share this article

X Facebook LinkedIn Bluesky Reddit Hacker News Email

References

The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain.

Rosenblatt, F. · 1958 · Psychological Review, 65(6)

https://doi.org/10.1037/h0042519

Deep Learning.

Goodfellow, I., Bengio, Y., Courville, A. · 2016

https://www.deeplearningbook.org/

Rectified Linear Units Improve Restricted Boltzmann Machines.

Nair, V., Hinton, G. E. · 2010 · ICML 2010

https://www.cs.toronto.edu/~fritz/absps/reluICML.pdf

Gaussian Error Linear Units (GELUs).

Hendrycks, D., Gimpel, K. · 2016

https://arxiv.org/abs/1606.08415

Multilayer Feedforward Networks are Universal Approximators.

Hornik, K., Stinchcombe, M., White, H. · 1989 · Neural Networks, 2(5)

https://doi.org/10.1016/0893-6080(89)90020-8

Approximation by Superpositions of a Sigmoidal Function.

Cybenko, G. · 1989 · Mathematics of Control, Signals and Systems, 2(4)

https://doi.org/10.1007/BF02551274

Discussion

Questions and insights from fellow learners.

Discussion loads when you reach this section.

One neuron produces one score

Why should 0.931 above be called a score rather than a confirmed escalation probability?

Why should 0.931 above be called a score rather than a confirmed escalation probability?

Non-linearity is what makes depth useful

If every hidden activation is removed from a dense network, what is lost?

If every hidden activation is removed from a dense network, what is lost?

A network is many neurons arranged as layers

Trace a tiny forward pass

Batch several runs through the same network

Parameters are stored decisions

Failure case: scales can overpower meaning

Failure case: naive sigmoid can overflow

Build an inspectable escalation-risk report

Your forward pass returns a value between zero and one. What evidence is still missing before operations treats it as a probability?

Your forward pass returns a value between zero and one. What evidence is still missing before operations treats it as a probability?

What approximation theorems leave open

Practice

Mastery check

Key concepts

Evaluation rubric

Common pitfalls

Follow-up questions

Why can two affine layers without a hidden activation be replaced by one layer?

Why can two affine layers without a hidden activation be replaced by one layer?

For batch X shaped (4, 2) and first-layer weights W1 shaped (2, 2), why does the code use X @ W1.T?

For batch X shaped (4, 2) and first-layer weights W1 shaped (2, 2), why does the code use X @ W1.T?

What changes when a model's output is called a calibrated probability instead of a score?

What changes when a model's output is called a calibrated probability instead of a score?

Why can raw feature scale be a debugging problem even when every feature is meaningful?

Why can raw feature scale be a debugging problem even when every feature is meaningful?

Mastery Check

Discussion

One neuron produces one score

Why should 0.931 above be called a score rather than a confirmed escalation probability?

Why should 0.931 above be called a score rather than a confirmed escalation probability?

Non-linearity is what makes depth useful

If every hidden activation is removed from a dense network, what is lost?

If every hidden activation is removed from a dense network, what is lost?

A network is many neurons arranged as layers

Trace a tiny forward pass

Batch several runs through the same network

Parameters are stored decisions

Failure case: scales can overpower meaning

Failure case: naive sigmoid can overflow

Build an inspectable escalation-risk report

Your forward pass returns a value between zero and one. What evidence is still missing before operations treats it as a probability?

Your forward pass returns a value between zero and one. What evidence is still missing before operations treats it as a probability?

What approximation theorems leave open

Practice

Mastery check

Key concepts

Evaluation rubric

Common pitfalls

Follow-up questions

Why can two affine layers without a hidden activation be replaced by one layer?

Why can two affine layers without a hidden activation be replaced by one layer?

For batch X shaped (4, 2) and first-layer weights W1 shaped (2, 2), why does the code use X @ W1.T?

For batch X shaped (4, 2) and first-layer weights W1 shaped (2, 2), why does the code use X @ W1.T?

What changes when a model's output is called a calibrated probability instead of a score?

What changes when a model's output is called a calibrated probability instead of a score?

Why can raw feature scale be a debugging problem even when every feature is meaningful?

Why can raw feature scale be a debugging problem even when every feature is meaningful?

Mastery Check

Discussion

Why should `0.931` above be called a score rather than a confirmed escalation probability?

For batch `X` shaped `(4, 2)` and first-layer weights `W1` shaped `(2, 2)`, why does the code use `X @ W1.T`?

Why should `0.931` above be called a score rather than a confirmed escalation probability?

For batch `X` shaped `(4, 2)` and first-layer weights `W1` shaped `(2, 2)`, why does the code use `X @ W1.T`?