CNNs from Scratch

A bright hairline mark cuts diagonally across an equipment panel photo. Shift that mark two pixels right, and any capable inspector still recognizes the scratch. A model that memorizes absolute pixel coordinates breaks on that tiny shift because each coordinate feeds a separate learned weight.

The previous lesson fed named tabular features into dense layers. Their order had to match the weight columns. Images add a useful structure: nearby pixels form edges, corners, and textures. Flattening preserves every pixel value and can be reversed if you know the shape; a dense layer simply doesn't impose local neighborhoods or shared detectors on that vector.

Convolutional neural networks tackle this challenge through two ideas: local receptive fields and weight sharing.^{[1]Reference 1Gradient-Based Learning Applied to Document Recognitionhttps://ieeexplore.ieee.org/document/726791[2]Reference 2Deep Learning.https://www.deeplearningbook.org/} We trace one 4 x 4 panel crop through hand calculations, tensor dimensions, pooling layers, and a matching PyTorch forward pass to see how raw pixels turn into class scores.

Make reuse pay for itself

Suppose the inspection camera resizes images to 224 x 224 RGB pixels. Each pixel holds red, green, and blue intensities, giving three channels. Connecting that input to a modest hidden layer of 256 dense units gives every single unit a distinct weight for every pixel intensity.

A convolutional layer replaces that global wiring with localized detectors. Instead of observing all 150,528 numbers at once, a 3 x 3 RGB detector inspects a small window of 3 x 3 x 3 = 27 weights plus one bias scalar. That exact detector slides across every valid position in the image.

The dense layer has over 38.5 million parameters; the convolutional layer uses 448. The ratio is about 86,000, but the outputs and connections differ: 256 global dense values versus sixteen 222 x 222 local feature maps. This isn't the same function implemented with fewer parameters or a measured speedup.

Two architectural choices drive this reduction: local connectivity and weight sharing. Local connectivity restricts each output neuron to a tiny spatial neighborhood. Weight sharing reuses the identical set of weights across every spatial position.

To isolate the contribution of weight sharing, consider a locally connected layer with sixteen 3 x 3 RGB filters that doesn't share weights across positions. Without padding, the output grid has 222 x 222 positions. Giving each position its own private weights balloons the count to 222 * 222 * 448 = 22,079,232 parameters. Weight sharing compresses those 22 million parameters down to 448 without altering the output dimensions.

Beyond parameter savings, weight sharing introduces a mathematical property called translation equivariance. Formally, an operation $f$ is equivariant to translation operator $T_{\Delta}$ when shifting the input produces an identically shifted feature map:

$$f(T_{\Delta}(x)) = T_{\Delta}(f(x))$$

For stride-one convolution away from boundary effects, shifting a pattern three pixels right shifts its response three pixels right. The same detector is reused at each position. On a finite crop, padding and a pattern entering or leaving the image can break that exact equality. With stride greater than one, arbitrary pixel shifts also change alignment with the sampling grid.

Equivariance differs from translation invariance, where $g(T_{\Delta}(x)) = g(x)$ produces an unchanged output. Local pooling can provide some shift tolerance, but a classification head isn't automatically invariant. Global spatial averaging is unchanged by a rearrangement of its input feature-map values; a translated finite image needn't produce such a rearrangement because of boundaries and downsampling.

Let one detector find one pattern

Examine a concrete grayscale crop representing a small defect on a dark metallic surface. Values range between 0.0 (dark surface) and 1.0 (bright scratch). Four bright values form a central 2 x 2 mark surrounded by background:

Select a 3 x 3 filter, also called a kernel. This kernel rewards bright middle rows while penalizing brightness in adjacent rows, acting as a horizontal line and ridge detector:

Placing a 3 x 3 window over a 4 x 4 grid yields 4 - 3 + 1 = 2 valid starting rows and 2 valid starting columns, producing a 2 x 2 output feature map.

Assuming zero bias for this trace, overlay the kernel on the upper-left 3 x 3 patch covering rows 0 to 2 and columns 0 to 2. Multiply aligned values elementwise and sum the products:

$$\begin{aligned} y_{0,0} ={}& (-0.6)(0.05 + 0.08 + 0.06) \\ &+ (1.1)(0.12 + 0.92 + 0.88) \\ &+ (-0.6)(0.08 + 0.90 + 0.93) \\ ={}& -0.114 + 2.112 - 1.146 \\ ={}& 0.852 \end{aligned}$$

The middle row contributes +2.112, exceeding the combined negative contributions from the neighboring rows (-0.114 and -1.146). The response before any activation function is 0.852.

Slide the identical kernel across the remaining three positions. Each placement computes an inner product between kernel weights and the aligned image window:

The bottom-right window yields 0.874, the highest activation on the grid. Its window aligns most favorably with the bright pixels in rows 2 and 3.

Mathematical convolution flips the kernel horizontally and vertically before computing these sliding sums. PyTorch uses cross-correlation, with no flip.^{[2]Reference 2Deep Learning.https://www.deeplearningbook.org/ [3]Reference 3Conv2dhttps://docs.pytorch.org/docs/stable/generated/torch.nn.Conv2d.html} For freely learned filters, flipping changes the parameter convention rather than the set of representable operations. For a fixed or imported asymmetric kernel, the distinction changes its responses. Our vertically symmetric kernel happens to give the same result either way.

Shapes, channels, and padding contracts

Moving beyond single-channel crops requires tracking dimensions across stride, padding, dilation, and channel depth.

For an input spatial dimension $W$, kernel size $K$, padding $P$, stride $S$, and dilation rate $D$, the output spatial dimension is:

$$W_{\text{out}} = \left\lfloor \frac{W + 2P - D(K - 1) - 1}{S} \right\rfloor + 1$$

When dilation $D = 1$ (standard dense sampling), the effective kernel size equals $K$, simplifying the formula to:

$$W_{\text{out}} = \left\lfloor \frac{W + 2P - K}{S} \right\rfloor + 1$$

The floor operation rounds down, discarding incomplete windows that hang off the edge of the padded input tensor.

Padding controls border behavior:

1. Valid padding ($P = 0$): No padding added. At stride one, the spatial dimension shrinks by $D(K - 1)$, or just $K - 1$ when dilation is one.
2. Same padding at stride one: Choose padding so $W_{\text{out}} = W$. With dilation one and an odd kernel, symmetric padding $P = (K - 1)/2$ works: one pixel for a 3 x 3 kernel, two for 5 x 5. Dilation changes the required total padding to $D(K - 1)$; an odd total needs unequal padding on the two sides. PyTorch's padding='same' supports only stride one.^{[3]Reference 3Conv2dhttps://docs.pytorch.org/docs/stable/generated/torch.nn.Conv2d.html}

Dilation ($D > 1$) inserts $(D - 1)$ spaces between kernel elements without adding learned parameters. A $3 \times 3$ kernel with dilation $D = 2$ spans a $5 \times 5$ receptive footprint while retaining only 9 parameter weights.

Channels introduce a third dimension. In the ordinary groups=1 convolution used here, one filter has shape $(C_{\text{in}}, K, K)$ and spans all three RGB channels.

At each spatial stop, the filter computes a three-dimensional inner product across all channels and sums them into a single scalar response, adding one bias term. To generate $F$ distinct output feature maps, the layer uses $F$ separate filters, yielding a parameter tensor of shape $(F, C_{\text{in}}, K, K)$ plus $F$ biases.

For a PyTorch batch with dimensions ordered NCHW (batch, channels, height, width), $F$ filters map $(N, C_{\text{in}}, H_{\text{in}}, W_{\text{in}})$ to $(N, F, H_{\text{out}}, W_{\text{out}})$. Input and output channel counts can differ. Grouped convolution changes connectivity: each filter sees $C_{\text{in}}/G$ channels, and both input and output counts must be divisible by the group count $G$.^{[3]Reference 3Conv2dhttps://docs.pytorch.org/docs/stable/generated/torch.nn.Conv2d.html} NCHW describes axis order here, not a required physical memory layout.

Receptive fields and channel expansion

An activation in the first feature map sees a small $3 \times 3$ input window. Stacking a second $3 \times 3$ convolution layer enables each second-layer activation to view a $3 \times 3$ patch of the first feature map. That patch depends on a $5 \times 5$ region of the original input.

The input region that influences a specific unit's activation is its receptive field. Tracking receptive field expansion across layers requires tracking two state variables:^{[4]Reference 4Computing Receptive Fields of Convolutional Neural Networkshttps://distill.pub/2019/computing-receptive-fields/}

1. field: the side length of the receptive field in raw input pixel units.
2. jump: the distance between adjacent activation centers in raw input pixel units.

For a layer with kernel size $K$, stride $S$, and dilation $D$, its effective kernel span is $D(K - 1) + 1$:

$$\text{field}_{\text{new}} = \text{field} + D(K - 1) \cdot \text{jump}, \qquad \text{jump}_{\text{new}} = \text{jump} \cdot S$$

All layers in this trace have dilation one. field describes the theoretical footprint, not a guarantee that every pixel contributes equally or even has a nonzero local gradient:

Notice how pooling accelerates receptive field growth. A stride of 2 doubles the jump to 2. The subsequent $3 \times 3$ convolution expands the receptive field by $(3 - 1) \times 2 = 4$ input pixels rather than 2, reaching a $10 \times 10$ context window.

Many classification backbones reduce spatial resolution and increase channel count across stages, though the exact progression varies by architecture. More channels can hold more distinct feature responses; they aren't a necessary consequence of downsampling. The figure below uses an illustrative all-valid sequence matching our receptive-field calculation: convolution, convolution, pooling, then convolution.

Early activations depend on small neighborhoods; later activations can combine evidence across larger regions. A trained model may learn edge-like early filters and more complex combinations later, but the shapes alone don't establish what features it learned. Downsampling also loses detail, which matters when the defect is only a few pixels wide.

Pooling mechanisms and gradient routing

Downsampling feature maps reduces computation while expanding receptive fields. Pooling achieves this downsampling without introducing learnable weights.

Two primary pooling strategies dominate convolutional networks:

1. Max pooling: Selects the maximum value within each window. It acts as a feature detector: if a crack feature fires strongly anywhere in a $2 \times 2$ window, max pooling preserves that peak response, offering localized translation tolerance.
2. Average pooling: Computes the arithmetic mean within each window. It summarizes feature-map responses, which may represent learned features rather than raw background intensity. A single large response contributes less to the mean than to the maximum.

Global Average Pooling (GAP) averages the spatial values separately for each channel, giving one value per channel and example. Starting with $(N, C, H, W)$, the result can retain shape $(N, C, 1, 1)$ or be flattened to $(N, C)`. A following$Q$-class linear head needs$CQ + Q$parameters instead of$CHWQ + Q$. This saves parameters when$HW > 1$, while removing spatial information from the head; lower test error isn't guaranteed.

Evaluate standard $2 \times 2$ max pooling with stride 2 over this $4 \times 4$ response grid:

The output picks 0.91, 0.44, 0.21, and 0.77, one winner per quadrant.

During backpropagation, error gradients $\delta = \frac{\partial \mathcal{L}}{\partial y}$ propagate backwards through pooling operations differently:

• In max pooling, the selected input activation receives the window's gradient: $\frac{\partial \mathcal{L}}{\partial x_{\text{winner}}} = \delta$. Other cells receive zero from that window. These aren't necessarily raw pixels. Ties require a backward convention; our NumPy example selects the first maximum in flattened order.
• In average pooling over an unpadded $K \times K$ window, each cell receives $\delta/K^2$ from that window. Backpropagation computes gradients, not parameter updates. If pooling windows overlap, contributions to a shared cell add.

Failure modes: border evidence and padding artifacts

Boundary handling introduces subtle failure modes that impact model predictions.

For a stride-one valid 3 x 3 operation on an image at least 5 x 5, a corner pixel participates in one window and an interior pixel can participate in nine. On our smaller 4 x 4 crop, even the most central pixels participate in only four. Fewer placements can reduce aggregate evidence in some models; they don't guarantee a smaller response for every kernel and classification head.

Padding helps by creating extra placements where the corner pixel lands inside valid windows, raising the count from 1 to 4.

Padding can also introduce responses to synthetic structure. Zero padding places zero-valued pixels around the image. On a uniform gray surface of brightness 0.6, the transition to 0.0 creates an artificial intensity step. An edge filter can respond strongly to that step; a classifier might mistake the response for a defect.

Reflection padding (mode='reflect') mirrors interior values across the boundary.^{[5]Reference 5numpy.padhttps://numpy.org/doc/stable/reference/generated/numpy.pad.html} It avoids this gray-to-black transition on a uniform image, but it imposes its own artificial continuation. It doesn't guarantee smooth gradients or remove all border artifacts.

Constant zero padding yields an artificial response of 1.8 on an untextured, uniform surface. Reflection padding outputs 0.0. When edge artifacts appear near image borders, evaluate padding modes and keep training and inference configurations synchronized.

Build it: an inspectable forward pass

Assemble a complete toy forward pipeline: the unpadded 4 x 4 crop passes through convolution, ReLU, max pooling, and a two-class linear head. All weights are hand-picked, so its logits illustrate class-score arithmetic, not a validated crack detector.

A stack of convolutions without non-linear operations remains affine, or linear if biases are absent. It can't create a non-linear dependence on the input. The merged operation isn't necessarily one ordinary shared-kernel convolution when finite boundaries and strides are involved.

The NumPy pipeline produces logits [1.024, -0.687].

Now construct the same architecture in PyTorch. Conv2d accepts a batch with shape (N, C, H, W) or an unbatched (C, H, W) tensor. Here unsqueeze(0).unsqueeze(0) adds the batch and channel axes, turning (4, 4) into (1, 1, 4, 4). Copy the manual weights to check numerical agreement.

The float32 PyTorch results agree with NumPy's float64 calculation within the asserted tolerance. Optimized implementations can use different arithmetic orders, so mathematical equivalence doesn't promise bitwise equality.

Modern vision architectures: ResNet and vision transformers

LeNet-5 and AlexNet demonstrated learned convolutional features on document recognition and large-scale image classification, respectively.^{[1]Reference 1Gradient-Based Learning Applied to Document Recognitionhttps://ieeexplore.ieee.org/document/726791 [6]Reference 6ImageNet Classification with Deep Convolutional Neural Networkshttps://www.cs.toronto.edu/~kriz/imagenet_classification_with_deep_convolutional.pdf} As researchers increased depth, they encountered the degradation problem: some deeper plain networks had higher training error than shallower counterparts.

Higher training error points to an optimization difficulty, rather than an explanation based solely on overfitting. The ResNet paper distinguishes this degradation from vanishing gradients: its plain networks used batch normalization, and the authors reported healthy backward gradient norms.^{[7]Reference 7Deep Residual Learning for Image Recognition.https://arxiv.org/abs/1512.03385} More depth can make optimization harder even when gradient magnitudes haven't vanished.

He et al.'s ResNet makes the learned branch represent a residual, $\mathcal{F}(\mathbf{x})$, rather than the whole desired mapping.^{[7]Reference 7Deep Residual Learning for Image Recognition.https://arxiv.org/abs/1512.03385} With matching shapes, an identity shortcut adds the original input to that branch:

$$\mathbf{y} = \mathcal{F}(\mathbf{x}) + \mathbf{x}$$

The following block applies ReLU after this sum, as the original basic ResNet block does. Changing spatial size or channel count would also require a compatible shortcut, such as a projection; this example keeps both unchanged.

BatchNorm2d normalizes each channel using statistics across batch and spatial positions, then applies learned scale and offset parameters. With its default settings, training updates running statistics that evaluation reuses.^{[8]Reference 8BatchNorm2dhttps://docs.pytorch.org/docs/2.14/generated/torch.nn.BatchNorm2d.html} The newly created block below is in training mode; this example checks shapes, not classifier accuracy.

The derivative of the residual connection during backpropagation reveals why this works:

$$\frac{\partial \mathcal{L}}{\partial \mathbf{x}} = \frac{\partial \mathcal{L}}{\partial \mathbf{y}} \left( \frac{\partial \mathcal{F}}{\partial \mathbf{x}} + \mathbf{I} \right)$$

For the residual sum, the identity term provides a direct additive gradient path. If the branch Jacobian is small, this path can help avoid repeatedly shrinking gradients through that branch. It isn't an unconditional lower bound on the total gradient: a branch Jacobian of $-\mathbf{I}$ cancels it exactly. The block's ReLU after addition introduces another gate, so the displayed derivative is for the sum before that ReLU. The original paper demonstrates successful training of residual networks with 50, 101, and 152 layers.^{[7]Reference 7Deep Residual Learning for Image Recognition.https://arxiv.org/abs/1512.03385}

A different connection pattern powers Vision Transformers (ViT).^{[9]Reference 9An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.https://arxiv.org/abs/2010.11929} Instead of sliding small local filters, ViT slices an image into non-overlapping patches (typically $16 \times 16$ pixels). Linearly projecting each flattened patch into an embedding vector of dimension $D$ is mathematically equivalent to a standard 2D convolution with kernel_size=16, stride=16, and out_channels=D.

The architectural contrast lies in how subsequent layers mix information:

• Convolutional networks share local filters across space, encouraging a detector learned in one location to be useful elsewhere. This inductive bias, an architectural assumption about useful functions, can help with limited labeled data. Strides, boundaries, and later heads still affect shift behavior.
• The original Vision Transformer retains spatial structure through patch extraction and positional embeddings, but its self-attention can combine information from any pair of patches in a layer. It has less built-in image locality than a CNN, rather than no spatial assumptions. The original paper found large-scale pretraining useful for its comparisons.^{[9]Reference 9An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.https://arxiv.org/abs/2010.11929}

Architecture alone doesn't determine data efficiency or a universal winner. DeiT showed competitive image transformers trained on ImageNet without external training images, using a stronger training recipe and teacher distillation.^{[10]Reference 10Training Data-Efficient Image Transformers & Distillation through Attentionhttps://arxiv.org/abs/2012.12877} ConvNeXt modernized convolutional design and competed with the Transformers studied in its experiments.^{[11]Reference 11A ConvNet for the 2020shttps://arxiv.org/abs/2201.03545} For an inspection task, compare suitable pretrained backbones on held-out panel photos, including tiny defects and image boundaries; account for inference cost as well as accuracy.

Change the crop, predict the response

Test your understanding of spatial shapes, kernel weights, and receptive fields before revealing the answers.