CNNs from Scratch

A bright hairline mark cuts diagonally across an equipment panel photo. Shift that mark two pixels right, and any capable inspector still recognizes the scratch. A model that memorizes absolute pixel coordinates breaks on that tiny shift because each coordinate feeds a separate learned weight.

The previous lesson fed named tabular features into dense layers. Their order had to match the weight columns. Images add a useful structure: nearby pixels form edges, corners, and textures. Flattening preserves every pixel value and can be reversed if you know the shape. A dense layer connects coordinates to weights; local neighborhoods and shared detectors must come from the architecture or be learned from examples.

Convolutional neural networks tackle this challenge through two ideas: local receptive fields and weight sharing.^{[1]Reference 1Gradient-Based Learning Applied to Document Recognitionhttps://ieeexplore.ieee.org/document/726791[2]Reference 2Deep Learninghttps://www.deeplearningbook.org/} We trace one 4 x 4 panel crop through hand calculations, tensor dimensions, pooling layers, and a matching PyTorch forward pass to see how raw pixels turn into class scores.

Make reuse pay for itself

Suppose the inspection camera resizes images to 224 x 224 RGB pixels. Each pixel holds red, green, and blue intensities, giving three channels. Connecting that input to a modest hidden layer of 256 dense units gives every single unit a distinct weight for every pixel intensity.

A convolutional layer replaces that global wiring with localized detectors. Instead of observing all 150,528 numbers at once, a 3 x 3 RGB detector inspects a small window of 3 x 3 x 3 = 27 weights plus one bias scalar. That exact detector slides across every valid position in the image.

height, width, channels, hidden = 224, 224, 3, 256
dense_weights = height * width * channels * hidden
dense_biases = hidden
dense_parameters = dense_weights + dense_biases

filters, kernel = 16, 3
conv_weights = filters * channels * kernel * kernel
conv_biases = filters
conv_parameters = conv_weights + conv_biases

print(f"dense parameters: {dense_parameters:,} ({dense_weights:,} weights + {dense_biases} biases)")
print(f"conv parameters: {conv_parameters:,} ({conv_weights:,} weights + {conv_biases} biases)")
print(f"parameter-count ratio: {dense_parameters / conv_parameters:,.0f}x")

dense parameters: 38,535,424 (38,535,168 weights + 256 biases)
conv parameters: 448 (432 weights + 16 biases)
parameter-count ratio: 86,017x

The dense layer has over 38.5 million parameters; the convolutional layer uses 448. The ratio is about 86,000, but the outputs and connections differ: 256 global dense values versus sixteen 222 x 222 local feature maps. Compare the task outputs and measured speed separately.

Two architectural choices drive this reduction: local connectivity and weight sharing. Local connectivity restricts each output neuron to a tiny spatial neighborhood. Weight sharing reuses the identical set of weights across every spatial position.

To isolate the contribution of weight sharing, consider a locally connected layer with sixteen 3 x 3 RGB filters and independent weights at every position. Without padding, the output grid has 222 x 222 positions. Giving each position its own private weights balloons the count to 222 * 222 * 448 = 22,079,232 parameters. Weight sharing compresses those 22 million parameters down to 448 without altering the output dimensions.

View larger

Beyond parameter savings, weight sharing introduces a mathematical property called translation equivariance. Formally, an operation $f$ is equivariant to translation operator $T_{\Delta}$ when shifting the input produces an identically shifted feature map:

For stride-one convolution away from boundary effects, shifting a pattern three pixels right shifts its response three pixels right. The same detector is reused at each position. On a finite crop, padding and a pattern entering or leaving the image can break that exact equality. With stride greater than one, arbitrary pixel shifts also change alignment with the sampling grid.

Equivariance differs from translation invariance, where $g(T_{\Delta}(x)) = g(x)$ produces an unchanged output. Local pooling can provide some shift tolerance. A classification head is invariant only when its features and pooling preserve the relevant shifts. Global spatial averaging is unchanged by rearranging feature-map values; boundaries and downsampling can make a translated finite image produce different features.

Let one detector find one pattern

Examine a concrete grayscale crop representing a small defect on a dark metallic surface. Values range between 0.0 (dark surface) and 1.0 (bright scratch). Four bright values form a central 2 x 2 mark surrounded by background:

Select a 3 x 3 filter, also called a kernel. This kernel rewards bright middle rows while penalizing brightness in adjacent rows, acting as a horizontal line and ridge detector:

Placing a 3 x 3 window over a 4 x 4 grid yields 4 - 3 + 1 = 2 valid starting rows and 2 valid starting columns, producing a 2 x 2 output feature map.

Assuming zero bias for this trace, overlay the kernel on the upper-left 3 x 3 patch covering rows 0 to 2 and columns 0 to 2. Multiply aligned values elementwise and sum the products:

The middle row contributes +2.112, exceeding the combined negative contributions from the neighboring rows (-0.114 and -1.146). The response before any activation function is 0.852.

patch = (
    (0.05, 0.08, 0.06),
    (0.12, 0.92, 0.88),
    (0.08, 0.90, 0.93),
)
kernel = (
    (-0.6, -0.6, -0.6),
    (1.1, 1.1, 1.1),
    (-0.6, -0.6, -0.6),
)

row_contributions = [
    round(sum(pixel * weight for pixel, weight in zip(row, kernel_row)), 3)
    for row, kernel_row in zip(patch, kernel)
]
response = sum(
    pixel * weight
    for row, kernel_row in zip(patch, kernel)
    for pixel, weight in zip(row, kernel_row)
)

print(f"row contributions: {row_contributions}")
print(f"first response: {response:.3f}")

row contributions: [-0.114, 2.112, -1.146]
first response: 0.852

Slide the identical kernel across the remaining three positions. Each placement computes an inner product between kernel weights and the aligned image window:

The bottom-right window yields 0.874, the highest activation on the grid. Its window aligns most favorably with the bright pixels in rows 2 and 3.

import numpy as np

image = np.array([
    [0.05, 0.08, 0.06, 0.07],
    [0.12, 0.92, 0.88, 0.09],
    [0.08, 0.90, 0.93, 0.11],
    [0.06, 0.07, 0.09, 0.05],
])
kernel = np.array([
    [-0.6, -0.6, -0.6],
    [1.1, 1.1, 1.1],
    [-0.6, -0.6, -0.6],
])

def convolve_valid(image: np.ndarray, kernel: np.ndarray) -> np.ndarray:
    out_height = image.shape[0] - kernel.shape[0] + 1
    out_width = image.shape[1] - kernel.shape[1] + 1
    output = np.empty((out_height, out_width))
    for row in range(out_height):
        for col in range(out_width):
            window = image[row:row + kernel.shape[0], col:col + kernel.shape[1]]
            output[row, col] = np.sum(window * kernel)
    return output

feature_map = convolve_valid(image, kernel)
strongest = tuple(int(pos) for pos in np.unravel_index(np.argmax(feature_map), feature_map.shape))
print(np.round(feature_map, 3))
print("strongest response:", strongest)

[[0.852 0.789]
 [0.817 0.874]]
strongest response: (1, 1)

View larger

Mathematical convolution flips the kernel horizontally and vertically before computing these sliding sums. PyTorch uses cross-correlation, with no flip.^{[2]Reference 2Deep Learninghttps://www.deeplearningbook.org/ [3]Reference 3Conv2dhttps://docs.pytorch.org/docs/stable/generated/torch.nn.Conv2d.html} With freely learned filters, either kernel orientation can be absorbed into the learned parameters, so both conventions represent the same operations. For a fixed or imported asymmetric kernel, the orientation changes its responses. Our vertically symmetric kernel happens to give the same result either way. That trace used one grayscale channel and windows that fit inside the crop. What happens to the output shape when the photo has color, or when the kernel has to see the border?

Shapes, channels, and padding contracts

The crop so far has one channel. Real images bring color channels, and larger models add stride, padding, dilation, and more filters. Tracking shapes keeps each dimension in view.

For an input spatial dimension $W$, kernel size $K$, padding $P$, stride $S$, and dilation rate $D$, the output spatial dimension is:

When dilation $D = 1$ (standard dense sampling), the effective kernel size equals $K$, simplifying the formula to:

The floor operation rounds down, discarding incomplete windows that hang off the edge of the padded input tensor.

Padding controls border behavior:

1. Valid padding ($P = 0$): No padding added. At stride one, the spatial dimension shrinks by $D(K - 1)$, or by $K - 1$ when dilation is one.
2. Same padding at stride one: Choose padding so $W_{\text{out}} = W$. With dilation one and an odd kernel, symmetric padding $P = (K - 1)/2$ works: one pixel for a 3 x 3 kernel, two for 5 x 5. Dilation changes the required total padding to $D(K - 1)$; an odd total needs unequal padding on the two sides. PyTorch's padding='same' supports only stride one.^{[3]Reference 3Conv2dhttps://docs.pytorch.org/docs/stable/generated/torch.nn.Conv2d.html}

Dilation ($D > 1$) inserts $(D - 1)$ spaces between kernel elements without adding learned parameters. A $3 \times 3$ kernel with dilation $D = 2$ spans a $5 \times 5$ receptive footprint while retaining only 9 parameter weights.

def conv_size(size: int, kernel: int, stride: int = 1, padding: int = 0, dilation: int = 1) -> int:
    if size <= 0 or kernel <= 0 or stride <= 0 or padding < 0 or dilation <= 0:
        raise ValueError("size, kernel, stride, and dilation must be positive; padding cannot be negative")
    effective_kernel = dilation * (kernel - 1) + 1
    if effective_kernel > size + 2 * padding:
        raise ValueError("effective kernel cannot exceed padded input size")
    return (size + 2 * padding - effective_kernel) // stride + 1

configurations = [
    ("worked crop", 4, 3, 1, 0, 1),
    ("same-size crop", 4, 3, 1, 1, 1),
    ("strided screenshot", 224, 3, 2, 1, 1),
    ("dilated feature map", 28, 3, 1, 0, 2),
]

for name, size, kernel, stride, padding, dilation in configurations:
    output = conv_size(size, kernel, stride, padding, dilation)
    print(f"{name}: {size} -> {output}")

worked crop: 4 -> 2
same-size crop: 4 -> 4
strided screenshot: 224 -> 112
dilated feature map: 28 -> 24

Channels introduce a third dimension. In the ordinary groups=1 convolution used here, one filter has shape $(C_{\text{in}}, K, K)$ and spans all three RGB channels.

At each spatial stop, the filter computes a three-dimensional inner product across all channels and sums them into a single scalar response, adding one bias term. To generate $F$ distinct output feature maps, the layer uses $F$ separate filters, yielding a parameter tensor of shape $(F, C_{\text{in}}, K, K)$ plus $F$ biases.

import numpy as np

image = np.zeros((3, 224, 224))
filters = np.zeros((16, 3, 3, 3))
biases = np.zeros(16)

assert filters.shape[1] == image.shape[0]
print("input shape:", image.shape)
print("one filter shape:", filters[0].shape)
print("output channels:", filters.shape[0])
print("parameters including biases:", filters.size + biases.size)

input shape: (3, 224, 224)
one filter shape: (3, 3, 3)
output channels: 16
parameters including biases: 448

For a PyTorch batch with dimensions ordered NCHW (batch, channels, height, width), $F$ filters map $(N, C_{\text{in}}, H_{\text{in}}, W_{\text{in}})$ to $(N, F, H_{\text{out}}, W_{\text{out}})$. Input and output channel counts can differ. Grouped convolution changes connectivity: each filter sees $C_{\text{in}}/G$ channels, and both input and output counts must be divisible by the group count $G$.^{[3]Reference 3Conv2dhttps://docs.pytorch.org/docs/stable/generated/torch.nn.Conv2d.html} NCHW describes axis order here, not a required physical memory layout.

Receptive fields and channel expansion

An activation in the first feature map sees a small $3 \times 3$ input window. Stacking a second $3 \times 3$ convolution layer enables each second-layer activation to view a $3 \times 3$ patch of the first feature map. That patch depends on a $5 \times 5$ region of the original input.

The input region that influences a specific unit's activation is its receptive field. Tracking receptive field expansion across layers requires tracking two state variables:^{[4]Reference 4Computing Receptive Fields of Convolutional Neural Networkshttps://distill.pub/2019/computing-receptive-fields/}

1. field: the side length of the receptive field in raw input pixel units.
2. jump: the distance between adjacent activation centers in raw input pixel units.

For a layer with kernel size $K$, stride $S$, and dilation $D$, its effective kernel span is $D(K - 1) + 1$:

All layers in this trace have dilation one. field describes the theoretical footprint. Learned weights, activations, padding, and input determine each pixel's actual contribution; some pixels can have zero local gradient:

layers = [
    ("conv 3x3", 3, 1),
    ("conv 3x3", 3, 1),
    ("pool 2x2", 2, 2),
    ("conv 3x3", 3, 1),
]

field, jump = 1, 1
for name, kernel, stride in layers:
    field = field + (kernel - 1) * jump
    jump = jump * stride
    print(f"{name:9s} receptive field={field:2d}, jump={jump}")

conv 3x3  receptive field= 3, jump=1
conv 3x3  receptive field= 5, jump=1
pool 2x2  receptive field= 6, jump=2
conv 3x3  receptive field=10, jump=2

Notice how pooling accelerates receptive field growth. A stride of 2 doubles the jump to 2. The subsequent $3 \times 3$ convolution adds $(3 - 1) \times 2 = 4$ input pixels to the receptive field, reaching a $10 \times 10$ context window.

Many classification backbones reduce spatial resolution and increase channel count across stages, though the exact progression varies by architecture. More channels can hold more distinct feature responses; they aren't a necessary consequence of downsampling. The figure below uses an illustrative all-valid sequence matching our receptive-field calculation: convolution, convolution, pooling, then convolution.

Early activations depend on small neighborhoods; later activations can combine evidence across larger regions. A trained model may learn edge-like early filters and more complex combinations later. Tensor shapes describe geometry; inspect filter activations on inputs to identify learned features. Downsampling also loses detail, which matters when the defect is only a few pixels wide.

View larger

Pooling mechanisms and gradient routing

Downsampling feature maps reduces the number of spatial positions processed by later layers, while pooling combines evidence over a larger footprint. It also discards detail. Pooling achieves this downsampling without introducing learnable weights.

Two pooling strategies you'll encounter often are:

1. Max pooling: Selects the maximum value within each window. It acts as a feature detector: if a crack feature fires strongly anywhere in a $2 \times 2$ window, max pooling preserves that peak response, offering localized translation tolerance.
2. Average pooling: Computes the arithmetic mean within each window. Feature-map responses may represent learned patterns, including background intensity. A single large response contributes less to the mean than to the maximum.

Global Average Pooling (GAP) averages the spatial values separately for each channel, giving one value per channel and example. Starting with $(N, C, H, W)$, the result can retain shape $(N, C, 1, 1)$ or be flattened to $(N, C)$. A following $Q$-class linear head needs $CQ + Q$ parameters instead of $CHWQ + Q$. This saves parameters when $HW > 1$ and removes spatial information from the head; measure the effect on held-out accuracy.

Evaluate standard $2 \times 2$ max pooling with stride 2 over this $4 \times 4$ response grid:

import numpy as np

feature_map = np.array([
    [0.10, 0.42, 0.18, 0.30],
    [0.25, 0.91, 0.44, 0.12],
    [0.08, 0.21, 0.77, 0.55],
    [0.16, 0.14, 0.32, 0.49],
])

pooled = np.empty((2, 2))
winners = []
for row in range(2):
    for col in range(2):
        block = feature_map[row * 2:row * 2 + 2, col * 2:col * 2 + 2]
        local_row, local_col = np.unravel_index(np.argmax(block), block.shape)
        pooled[row, col] = block[local_row, local_col]
        winners.append((int(row * 2 + local_row), int(col * 2 + local_col)))

print(np.round(pooled, 2))
print("winner coordinates:", winners)

[[0.91 0.44]
 [0.21 0.77]]
winner coordinates: [(1, 1), (1, 2), (2, 1), (2, 2)]

The output picks 0.91, 0.44, 0.21, and 0.77, one winner per quadrant.

During backpropagation, error gradients $\delta = \frac{\partial \mathcal{L}}{\partial y}$ propagate backwards through pooling operations differently:

• In max pooling, the selected input activation receives the window's gradient: $\frac{\partial \mathcal{L}}{\partial x_{\text{winner}}} = \delta$. Other cells receive zero from that window. These aren't necessarily raw pixels. Ties require a backward convention; our NumPy example selects the first maximum in flattened order.
• In average pooling over an unpadded $K \times K$ window, each cell receives $\delta/K^2$ from that window. Backpropagation computes gradients, not parameter updates. If pooling windows overlap, contributions to a shared cell add.

Now reduce the stride to one on a 3 x 3 map whose center is the largest value. Its center wins all four overlapping windows. Predict whether it receives the largest incoming gradient or their sum, then check PyTorch's forward indices and backward result:

import torch

values = torch.tensor([[1., 2., 3.], [4., 9., 5.], [6., 7., 8.]],
                      dtype=torch.float64, requires_grad=True)
pool = torch.nn.MaxPool2d(kernel_size=2, stride=1, return_indices=True)
pooled, indices = pool(values[None, None])
incoming = torch.tensor([[[[2., -1.], [0.5, 3.]]]], dtype=torch.float64)
pooled.backward(incoming)

print("pooled:", pooled.detach()[0, 0].tolist())
print("flat winner indices:", indices[0, 0].tolist())
print("input gradients:", values.grad.tolist())
assert values.grad[1, 1].item() == 4.5
assert torch.count_nonzero(values.grad).item() == 1

pooled: [[9.0, 9.0], [9.0, 9.0]]
flat winner indices: [[4, 4], [4, 4]]
input gradients: [[0.0, 0.0, 0.0], [0.0, 4.5, 0.0], [0.0, 0.0, 0.0]]

The center's flattened index is 4 in the 3 x 3 input plane. All four outputs return that index, and backward adds their signed contributions: $2-1+0.5+3=4.5$.^{[5]Reference 5MaxPool2d (PyTorch 2.14)https://docs.pytorch.org/docs/2.14/generated/torch.nn.MaxPool2d.html} A manual implementation must accumulate with +=; assigning with = would overwrite earlier contributions. Pooling selects the winner from forward activations, then routes the signed incoming gradient to that location.

View larger

Failure modes: border evidence and padding artifacts

Our worked windows so far fit neatly inside the crop. At an image edge, padding supplies values beyond the picture, and those synthetic values can contribute to the filter response.

For a stride-one valid 3 x 3 operation on an image at least 5 x 5, a corner pixel participates in one window and an interior pixel can participate in nine. On our smaller 4 x 4 crop, even the most central pixels participate in only four. Fewer placements can reduce aggregate evidence in some models; the response also depends on the kernel weights and classification head.

import numpy as np

def total_window_response(image: np.ndarray, padding: int) -> float:
    padded = np.pad(image, padding)
    kernel = np.ones((3, 3))
    total = 0.0
    for row in range(padded.shape[0] - 2):
        for col in range(padded.shape[1] - 2):
            total += np.sum(padded[row:row + 3, col:col + 3] * kernel)
    return total

edge_signal = np.zeros((5, 5))
edge_signal[0, 0] = 1.0
center_signal = np.zeros((5, 5))
center_signal[2, 2] = 1.0

print(f"valid edge total: {total_window_response(edge_signal, padding=0):.0f}")
print(f"valid center total: {total_window_response(center_signal, padding=0):.0f}")
print(f"padded edge total: {total_window_response(edge_signal, padding=1):.0f}")

valid edge total: 1
valid center total: 9
padded edge total: 4

Padding helps by creating extra placements where the corner pixel lands inside valid windows, raising the count from 1 to 4.

Padding can also introduce responses to synthetic structure. Zero padding places zero-valued pixels around the image. On a uniform gray surface of brightness 0.6, the transition to 0.0 creates an artificial intensity step. An edge filter can respond strongly to that step; a classifier might mistake the response for a defect.

Reflection padding (mode='reflect') mirrors interior values across the boundary.^{[6]Reference 6numpy.padhttps://numpy.org/doc/stable/reference/generated/numpy.pad.html} It avoids this gray-to-black transition on a uniform image, while still imposing an artificial continuation that can create other gradients or border artifacts.

import numpy as np

card = np.full((5, 5), 0.6)
horizontal_edge = np.array([
    [1.0, 1.0, 1.0],
    [0.0, 0.0, 0.0],
    [-1.0, -1.0, -1.0],
])

def response_map(padded: np.ndarray, kernel: np.ndarray) -> np.ndarray:
    output = np.empty((padded.shape[0] - 2, padded.shape[1] - 2))
    for row in range(output.shape[0]):
        for col in range(output.shape[1]):
            output[row, col] = np.sum(padded[row:row + 3, col:col + 3] * kernel)
    return output

for mode in ("constant", "reflect"):
    padded = np.pad(card, 1, mode=mode)
    responses = response_map(padded, horizontal_edge)
    print(f"{mode} max absolute response: {np.abs(responses).max():.1f}")

constant max absolute response: 1.8
reflect max absolute response: 0.0

Constant zero padding yields an artificial response of 1.8 on an untextured, uniform surface. Reflection padding outputs 0.0. When edge artifacts appear near image borders, evaluate padding modes and keep training and inference configurations synchronized.

Build it: an inspectable forward pass

Now follow the unpadded 4 x 4 crop through one complete forward pass: convolution, ReLU, max pooling, and a two-class linear head. The weights are hand-picked, so the logits illustrate class-score arithmetic. This network has no validation evidence as a crack detector.

A stack of convolutions without non-linear operations composes into an affine map, or a linear map if biases are absent. Finite boundaries and strides can make the merged operation differ from an ordinary shared-kernel convolution.

View larger

import numpy as np

image = np.array([
    [0.05, 0.08, 0.06, 0.07],
    [0.12, 0.92, 0.88, 0.09],
    [0.08, 0.90, 0.93, 0.11],
    [0.06, 0.07, 0.09, 0.05],
])
kernel = np.array([
    [-0.6, -0.6, -0.6],
    [1.1, 1.1, 1.1],
    [-0.6, -0.6, -0.6],
])

def conv_valid(image: np.ndarray, kernel: np.ndarray) -> np.ndarray:
    output = np.empty((2, 2))
    for row in range(2):
        for col in range(2):
            output[row, col] = np.sum(image[row:row + 3, col:col + 3] * kernel)
    return output

feature_map = conv_valid(image, kernel)
activated = np.maximum(feature_map, 0.0)
pooled = np.array([activated.max()])
classifier_weights = np.array([[1.4], [-0.9]])
classifier_bias = np.array([-0.2, 0.1])
numpy_logits = classifier_weights @ pooled + classifier_bias

assert feature_map.shape == (2, 2)
assert pooled.shape == (1,)
print("feature map:")
print(np.round(feature_map, 3))
print("pooled activation:", np.round(pooled, 3))
print("logits:", np.round(numpy_logits, 3))

feature map:
[[0.852 0.789]
 [0.817 0.874]]
pooled activation: [0.874]
logits: [ 1.024 -0.687]

The NumPy pipeline produces logits [1.024, -0.687].

Now construct the same architecture in PyTorch. Conv2d accepts a batch with shape (N, C, H, W) or an unbatched (C, H, W) tensor. Here unsqueeze(0).unsqueeze(0) adds the batch and channel axes, turning (4, 4) into (1, 1, 4, 4). Copy the manual weights to check numerical agreement.

import torch
from torch import nn

image_bchw = torch.tensor(image, dtype=torch.float32).unsqueeze(0).unsqueeze(0)

model = nn.Sequential(
    nn.Conv2d(1, 1, kernel_size=3, bias=False),
    nn.ReLU(),
    nn.MaxPool2d(kernel_size=2),
    nn.Flatten(),
    nn.Linear(1, 2),
)

with torch.no_grad():
    model[0].weight.copy_(torch.tensor(kernel, dtype=torch.float32).view(1, 1, 3, 3))
    model[4].weight.copy_(torch.tensor(classifier_weights, dtype=torch.float32))
    model[4].bias.copy_(torch.tensor(classifier_bias, dtype=torch.float32))

pytorch_logits = model(image_bchw).detach().numpy()[0]
assert np.allclose(pytorch_logits, numpy_logits, rtol=0.0, atol=1e-6)
print("pytorch logits:", np.round(pytorch_logits, 3))
print("matches NumPy:", True)

pytorch logits: [ 1.024 -0.687]
matches NumPy: True

The float32 PyTorch results agree with NumPy's float64 calculation within the asserted tolerance. Optimized implementations can use different arithmetic orders and produce different bitwise results through floating-point rounding.

Modern vision architectures: ResNet and vision transformers

The forward pass is now a score you can recompute with fixed kernels. The tempting next move is more layers. LeNet-5 and AlexNet showed that learned convolutional features work for document recognition and for large-scale image classification.^{[1]Reference 1Gradient-Based Learning Applied to Document Recognitionhttps://ieeexplore.ieee.org/document/726791 [7]Reference 7ImageNet Classification with Deep Convolutional Neural Networkshttps://www.cs.toronto.edu/~kriz/imagenet_classification_with_deep_convolutional.pdf} Adding depth then produced a surprise. Some deeper plain networks had higher training error than shallower ones. That is the degradation problem.

Higher training error points to an optimization difficulty. The ResNet paper distinguishes this degradation from overfitting and vanishing gradients: its plain networks used batch normalization, and the authors reported healthy backward gradient norms.^{[8]Reference 8Deep Residual Learning for Image Recognition.https://arxiv.org/abs/1512.03385} More depth can make optimization harder even when gradient magnitudes remain healthy.

He et al.'s ResNet makes the learned branch represent a residual, $\mathcal{F}(\mathbf{x})$, which is added to the original input when their shapes match.^{[8]Reference 8Deep Residual Learning for Image Recognition.https://arxiv.org/abs/1512.03385} An identity shortcut computes:

The following block applies ReLU after this sum, as the original basic ResNet block does. Changing spatial size or channel count would also require a compatible shortcut, such as a projection; this example keeps both unchanged.

BatchNorm2d normalizes each channel using statistics across batch and spatial positions, then applies learned scale and offset parameters. With its default settings, training updates running statistics that evaluation reuses.^{[9]Reference 9BatchNorm2dhttps://docs.pytorch.org/docs/2.14/generated/torch.nn.BatchNorm2d.html} The newly created block below is in training mode. This example checks tensor shapes; evaluate classifier accuracy on held-out data.

import torch
from torch import nn

class ResidualBlock(nn.Module):
    def __init__(self, channels: int):
        super().__init__()
        self.conv1 = nn.Conv2d(channels, channels, kernel_size=3, padding=1, bias=False)
        self.bn1 = nn.BatchNorm2d(channels)
        self.relu = nn.ReLU(inplace=True)
        self.conv2 = nn.Conv2d(channels, channels, kernel_size=3, padding=1, bias=False)
        self.bn2 = nn.BatchNorm2d(channels)

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        identity = x
        out = self.relu(self.bn1(self.conv1(x)))
        out = self.bn2(self.conv2(out))
        out += identity
        return self.relu(out)

block = ResidualBlock(channels=16)
dummy_input = torch.randn(1, 16, 28, 28)
output = block(dummy_input)
print("input shape:", list(dummy_input.shape))
print("output shape:", list(output.shape))
print("shape preserved:", dummy_input.shape == output.shape)

input shape: [1, 16, 28, 28]
output shape: [1, 16, 28, 28]
shape preserved: True

The derivative of the residual connection during backpropagation reveals why this works:

For the residual sum, the identity term provides a direct additive gradient path. If the branch Jacobian is small, this path can help avoid repeatedly shrinking gradients through that branch. A branch Jacobian of $-\mathbf{I}$ cancels the identity term exactly, so total gradient size depends on both paths. The block's ReLU after addition introduces another gate, so the displayed derivative is for the sum before that ReLU. The original paper demonstrates successful training of residual networks with 50, 101, and 152 layers.^{[8]Reference 8Deep Residual Learning for Image Recognition.https://arxiv.org/abs/1512.03385}

A different connection pattern powers Vision Transformers (ViT).^{[10]Reference 10An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.https://arxiv.org/abs/2010.11929} Instead of sliding small local filters, ViT slices an image into non-overlapping patches (typically $16 \times 16$ pixels). Linearly projecting each flattened patch into an embedding vector of dimension $D$ is mathematically equivalent to a standard 2D convolution with kernel_size=16, stride=16, and out_channels=D.

The architectural contrast lies in how subsequent layers mix information:

• Convolutional networks share local filters across space, encouraging a detector learned in one location to be useful elsewhere. This inductive bias, an architectural assumption about useful functions, can help with limited labeled data. Strides, boundaries, and later heads still affect shift behavior.
• The original Vision Transformer retains spatial structure through patch extraction and positional embeddings, while its self-attention can combine information from any pair of patches in a layer. It has less built-in image locality than a CNN; the original paper found large-scale pretraining useful for its comparisons.^{[10]Reference 10An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.https://arxiv.org/abs/2010.11929}

Data efficiency and model rankings depend on task and training recipe. DeiT showed competitive image transformers trained on ImageNet without external training images, using a stronger training recipe and teacher distillation.^{[11]Reference 11Training Data-Efficient Image Transformers & Distillation through Attentionhttps://arxiv.org/abs/2012.12877} ConvNeXt modernized convolutional design and competed with the Transformers studied in its experiments.^{[12]Reference 12A ConvNet for the 2020shttps://arxiv.org/abs/2201.03545} For an inspection task, compare suitable pretrained backbones on held-out panel photos, including tiny defects and image boundaries. Account for inference cost as well as accuracy. A score traces back to a kernel window, a pooling winner, and a border the padding invented. This chapter uses hand-chosen weights to trace predictions; training would update them using prediction error.

Change the crop, predict the response

Test your understanding of spatial shapes, kernel weights, and receptive fields before revealing the answers.