CNNs from Scratch

An engineer uploads a photo of a cracked equipment panel with a narrow fault line across its surface. In the previous lesson, a small neural network accepted already-extracted numbers and produced a score. For a photo, those numbers are still arranged in space: neighboring bright and dark pixels may form the crack, while the same pixels in a different arrangement may mean nothing.

A convolutional neural network (CNN) handles that structure by applying small filters across nearby pixels. The same filter is reused at every position, so it can respond to a useful local pattern wherever it appears in the image. This use of local receptive fields and shared weights is the central CNN idea.^{[1]Reference 1Gradient-Based Learning Applied to Document Recognitionhttps://ieeexplore.ieee.org/document/726791[2]Reference 2Deep Learning.https://www.deeplearningbook.org/}

Why a photo needs local weights

Suppose the upload is resized to 224 x 224 RGB pixels. Feeding every pixel into one dense hidden layer of 256 units creates 38,535,424 parameters before the model has even recognized an edge: 38,535,168 weights plus 256 biases. A convolution layer with 16 filters of size 3 x 3 across three color channels uses 448 parameters: 432 weights plus 16 biases.

This comparison is about parameter count, not identical capabilities: a dense layer connects every location independently, while a convolution deliberately assumes that the same local detector should be useful across the image.

Slide one chosen kernel by hand

Start with one grayscale crop from the equipment-panel photo. Bright values in the center might be reflective surface around a crack:

For now, choose a 3 x 3 filter that gives positive weight to a bright horizontal middle line and negative weight above and below it:

The first output value comes from overlaying the kernel on the upper-left 3 x 3 region, multiplying aligned values, and summing:

$$\begin{aligned} y_{0,0} ={}& (-0.6)(0.05 + 0.08 + 0.06) \\ &+ (1.1)(0.12 + 0.92 + 0.88) \\ &+ (-0.6)(0.08 + 0.90 + 0.93) \\ ={}& 0.852 \end{aligned}$$

Slide exactly the same kernel one column or row at a time. The output is called a feature map: it records where this chosen pattern responds strongly.

Deep-learning libraries usually call this sliding multiply-and-sum a convolution even though the kernel isn't flipped. In signal-processing terminology, this exact operation is cross-correlation.^{[2]Reference 2Deep Learning.https://www.deeplearningbook.org/}

This detector is fixed for inspection. During training, kernel weights are adjusted from prediction error rather than supplied by a person. Also note the careful claim: shifting an input pattern tends to shift its convolution response, which is translation equivariance; it isn't automatic invariance to arbitrary movement or deformation.^{[2]Reference 2Deep Learning.https://www.deeplearningbook.org/}

Shapes and channels are part of the contract

For one spatial dimension of a standard convolution with dilation 1, output size is:

$$\left\lfloor \frac{N + 2P - K}{S} \right\rfloor + 1$$

where N is input size, K is kernel size, P is padding on each side, and S is stride. The worked example uses N = 4, K = 3, P = 0, and S = 1, giving output width and height 2.

A color image tensor has channels as well as height and width. Filters looking at RGB input must span all three channels, so one 3 x 3 filter has shape 3 x 3 x 3. A layer with 16 such filters creates 16 output feature maps.

Receptive fields grow one layer at a time

One response in the first feature map sees only a 3 x 3 patch. Later layers combine nearby responses, so one later value depends on a larger region of the original image. That dependency region is the receptive field.

Keep two numbers while tracing a stack:

• field: how many original pixels influence one current value.
• jump: how far apart neighboring current values are in original-pixel coordinates.

For a layer with kernel size K and stride S:

$$\text{new field} = \text{field} + (K - 1)\text{jump}, \qquad \text{new jump} = \text{jump} \cdot S$$

A larger receptive field only says what input can affect a value. It doesn't prove that the model understands cracks, panels, or any other object. That depends on learned weights and training data.

Pooling reduces resolution and records winners

A common CNN step is 2 x 2 max pooling: in each small block, keep only the largest activation. It reduces spatial resolution and gives limited tolerance to small local shifts in a strong response.

Consider a larger response map:

Why save winner coordinates? When you train, an error signal that arrives at a pooled value has to travel back through max pooling to the input cell that won the forward comparison. Non-winners get zero gradient for that path. The next lesson walks through that routing (and the dense-layer / ReLU rules that sit next to it). For now, the forward-pass rule is enough: pool values and retain their locations.

Failure case: border evidence gets fewer chances to fire

Valid convolution, with no padding, never centers a 3 x 3 filter on an outermost pixel. Evidence near an image border participates in fewer windows than identical evidence near the center. That matters in equipment-panel photos: a crack reaching the edge of a crop can be weakened by preprocessing choices.

Padding doesn't make border context identical to interior context: outside-image pixels still have to be filled somehow. It still keeps more filter placements available near the border, which is why padding policy is part of model debugging rather than a cosmetic setting.

Build it: an inspectable CNN forward pass

Now assemble the pieces into a tiny forward pass. This model has one chosen convolution filter, a ReLU nonlinearity, max pooling, and a two-score linear classifier. The scores are logits, not probabilities; converting logits into probabilities and training weights comes later.

PyTorch performs the same operations with standard layers. Here the filter and classifier weights are copied from the NumPy example only so the two implementations can be checked against one another.

CNNs and vision transformers

CNNs became practical image models in part because local connections and shared weights made them efficient enough to train for tasks such as handwritten-document recognition in LeNet-style systems.^{[1]Reference 1Gradient-Based Learning Applied to Document Recognitionhttps://ieeexplore.ieee.org/document/726791} Newer vision architectures may use different starting assumptions: a Vision Transformer, for example, treats fixed-size image patches as a sequence of embedded tokens rather than beginning with sliding convolution filters.^{[3]Reference 3An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.https://arxiv.org/abs/2010.11929}

That contrast is orientation, not a model-selection rule. The transferable skill is being able to trace what spatial evidence reaches a score, which assumptions the model makes, and which failure mode a shape or padding choice can introduce.

Mastery check

Key concepts

• A convolution applies one local filter at many positions, producing a feature map.
• Weight sharing reduces parameter count and supports translation-equivariant responses.
• Output shapes depend on kernel size, stride, and padding; channel counts must also agree.
• Receptive fields grow as layers combine neighboring responses.
• Max pooling retains strong local activations and must remember winning positions for later training.
• Padding policy can change how strongly edge-of-image evidence is represented.

Evaluation rubric

• Foundational: Calculate one convolution response and state why the same kernel is reused across positions.
• Intermediate: Derive output shapes and parameter counts for grayscale or RGB convolution layers, then trace pooling output.
• Advanced: Diagnose a border-evidence or receptive-field failure and implement an equivalent NumPy and PyTorch forward pass.

Common pitfalls

• Symptom: A hand-entered detector is described as learned. Cause: Forward-pass inspection is confused with training. Fix: Call it a chosen kernel until gradients have updated it from data.
• Symptom: Feature-map dimensions differ from expected dimensions. Cause: Padding or stride was omitted from the shape calculation. Fix: Compute each spatial output with the convolution formula before wiring the next layer.
• Symptom: An RGB image fails to enter a convolution layer. Cause: Each filter doesn't span all input channels. Fix: Match filter input-channel dimension to the tensor channel dimension.
• Symptom: A defect near a crop boundary produces a weaker response than the same defect at the center. Cause: Valid convolution gives border pixels fewer filter placements. Fix: Audit crop and padding policy, then compare edge and center responses explicitly.

Follow-up questions

Using the kernel in this lesson, calculate the first response when its upper-left weight is changed from -0.6 to 0.0. Which input pixel causes the difference?

Answer

Only the upper-left aligned input value, 0.05, changes its contribution. Removing its -0.6 weight increases the response by 0.03, from 0.852 to 0.882, because 0.05 times -0.6 had contributed -0.03.

An RGB image of shape 64 by 64 enters eight 5 by 5 filters with stride 2 and padding 2. What is the output spatial shape, and how many parameters, including one bias per filter, does the layer have?

Answer

Each output dimension is floor((64 + 2 times 2 - 5) / 2) + 1 = 32, so output shape is 8 by 32 by 32. Parameters are 8 times 3 times 5 times 5 weights plus 8 biases, or 608 total.

Trace receptive field and jump for two 3 by 3 stride-one convolutions followed by one 3 by 3 stride-two convolution.

Answer

Start with field 1 and jump 1. First convolution gives field 3 and jump 1. Second gives field 5 and jump 1. Stride-two convolution gives field 7 and jump 2, so one final activation can use a 7 by 7 input region and adjacent final activations move two input pixels apart.

Why might valid convolution weaken defect evidence close to a crop boundary, and what does padding fix?

Answer

A border pixel belongs to fewer valid filter windows than an interior pixel, so comparable evidence is counted fewer times. Padding supplies more windows near the edge and can retain more response opportunities. It doesn't create real context outside the image, so boundary uncertainty remains.