Convolutional Neural Networks

Why convolutional networks matter

Convolutional neural networks (CNNs) are the architecture that made deep learning work for images. They power photo search, medical image triage, defect detection on factory lines, document scanning, and the perception stacks of robots and cars. Even in a period dominated by transformers, convolutions remain the default when you need a fast, data-efficient vision model, and they live on inside many hybrid and generative architectures. This lesson takes you from the convolution operation itself to fine-tuning a pretrained backbone, and closes with a careful comparison against vision transformers.

Intuition: a small stencil slid across the image

Picture a 3×3 cardboard stencil with nine numbers written on it. You lay it on the top-left corner of a grayscale image, multiply each stencil number by the pixel underneath, add the nine products, and write the sum in a new grid. Then you slide the stencil one pixel right and repeat, row by row, until you have covered the image. The new grid is a feature map: it is bright wherever the image locally "looks like" the stencil's pattern. A stencil with positive numbers on its left column and negative numbers on its right lights up at vertical edges.

A CNN learns the numbers on hundreds of such stencils (called kernels or filters) by gradient descent, stacks them in layers, and lets deeper layers combine edges into textures, textures into parts, and parts into objects. Two built-in assumptions make this efficient:

• Locality: useful patterns are built from nearby pixels, so each output only looks at a small neighbourhood.
• Weight sharing / translation equivariance: the same detector is useful everywhere, so one kernel is reused at every position. If the input shifts, the feature map shifts with it.

These assumptions are called the CNN's inductive bias. They are why a CNN with a few million parameters can learn from tens of thousands of images, where a fully connected network would need billions of weights.

The convolution operation, formally

Let the input be a tensor $X \in \mathbb{R}^{C_{in} \times H \times W}$ (channels, height, width) and let the layer have $C_{out}$ kernels, each of shape $C_{in} \times k \times k$. The output feature map for kernel $o$ at position $(i, j)$ is

where $K$ is the kernel weight tensor, $b_o$ is a per-output-channel bias, and $S$ is the stride (how many pixels the kernel jumps between positions). Note that each kernel spans all input channels: a 3×3 kernel on an RGB image really has $3 \times 3 \times 3 = 27$ weights.

Strictly, this is cross-correlation; true mathematical convolution flips the kernel first. Deep learning libraries (PyTorch's conv2d included) implement cross-correlation and call it convolution. Because the kernel is learned, the flip does not matter.

Worked example: one kernel, every step

Input $X$ (4×4, one channel) and a vertical-edge kernel $K$ (3×3), stride 1, no padding:

Because $K$ has $+1$ in its left column and $-1$ in its right column, each output equals (sum of the left column of the patch) minus (sum of the right column).

1. Top-left, rows 0–2, columns 0–2: left column $1+0+2 = 3$, right column $0+3+0 = 3$, output $3-3 = 0$.
2. Top-right, rows 0–2, columns 1–3: left column $2+1+1 = 4$, right column $1+1+0 = 2$, output $4-2 = 2$.
3. Bottom-left, rows 1–3, columns 0–2: left $0+2+1 = 3$, right $3+0+1 = 4$, output $-1$.
4. Bottom-right, rows 1–3, columns 1–3: left $1+1+0 = 2$, right $1+0+2 = 3$, output $-1$.

A 4×4 input and 3×3 kernel give a 2×2 output: the kernel fits in only two positions along each axis.

Output size, padding, stride, dilation

Padding $P$ adds $P$ rows/columns of (usually) zeros around the border so the kernel can be centred on edge pixels. For input width $W$, kernel $k$, padding $P$, stride $S$:

The same formula applies to height. Two cases to memorise: with stride 1, "same" padding $P = (k-1)/2$ for odd $k$ keeps the size unchanged (3×3 with $P=1$, 5×5 with $P=2$); with $k = S$ and $P = 0$ (patchify), the size divides by $S$.

Worked example — the classic ResNet "stem" on a 224×224 image: a 7×7 convolution with stride 2 and padding 3 gives $\lfloor (224 - 7 + 6)/2 \rfloor + 1 = \lfloor 111.5 \rfloor + 1 = 112$. A following 3×3 max-pool with stride 2 and padding 1 gives $\lfloor (112 - 3 + 2)/2 \rfloor + 1 = 55 + 1 = 56$.

Dilation $D$ spaces the kernel taps $D$ pixels apart, giving an effective kernel size $k_{eff} = D(k-1) + 1$ with no extra weights; substitute $k_{eff}$ for $k$ in the formula. Dilated convolutions are common in segmentation and audio models where a large context is needed at full resolution.

Pooling

Pooling downsamples each channel independently with a fixed (non-learned) function. Max pooling with a 2×2 window and stride 2 keeps the largest value in each window, halving height and width and giving a little local translation invariance. Average pooling takes the mean. Global average pooling averages each entire channel to a single number, turning a $C \times H \times W$ map into a length-$C$ vector; modern classifiers use it in place of large fully connected layers, which removes most of their parameters and lets the network accept varied input sizes. Many recent designs replace max pooling with strided convolutions, letting the network learn how to downsample.

Parameter and compute counting

A convolutional layer has

and costs about $k_h k_w C_{in} C_{out} H_{out} W_{out}$ multiply-accumulates (MACs). Crucially, the parameter count does not depend on image size.

Worked example: a 3×3 convolution from 64 to 128 channels has $3 \cdot 3 \cdot 64 \cdot 128 + 128 = 73{,}728 + 128 = 73{,}856$ parameters. At a 56×56 output it costs $73{,}728 \times 3{,}136 \approx 2.3 \times 10^8$ MACs. A fully connected layer mapping the same $56 \cdot 56 \cdot 64 = 200{,}704$ inputs to $56 \cdot 56 \cdot 128 = 401{,}408$ outputs would need about $8.1 \times 10^{10}$ weights — roughly a million times more. That gap is the practical payoff of locality and weight sharing.

Receptive field

The receptive field of a unit is the region of the input that can influence it. For a stack of layers with kernel sizes $k_l$ and strides $s_l$, the receptive field after layer $L$ is

The product of earlier strides (the "jump") is how many input pixels one step in layer $l$'s input corresponds to. Continuing the ResNet stem: the 7×7 stride-2 conv gives $r = 7$; the 3×3 stride-2 pool adds $(3-1)\times 2 = 4$, giving $r = 11$; a following 3×3 stride-1 conv adds $(3-1) \times 4 = 8$, giving $r = 19$.

Two consequences matter in practice. First, stacking two 3×3 convolutions gives a 5×5 receptive field with $2 \cdot 9 C^2 = 18C^2$ weights versus $25C^2$ for one 5×5 layer (with $C$ channels in and out), plus an extra nonlinearity; three 3×3 layers match a 7×7 at $27C^2$ versus $49C^2$. This is the VGG design argument. Second, the effective receptive field — where gradients are actually large — is concentrated near the centre and smaller than the theoretical one, which is why downsampling and global pooling are needed to see whole objects.

Building blocks of a modern CNN

A typical stage is: convolution → batch normalization (normalizes each channel using batch statistics, then applies a learned scale and shift) → ReLU, repeated, followed by downsampling that halves resolution while doubling channels. The network ends with global average pooling and a linear classifier.

The code below checks the worked example against PyTorch and builds a small CIFAR-sized CNN.

import numpy as np, torch, torch.nn as nn, torch.nn.functional as F

X = np.array([[1,2,0,1],[0,1,3,1],[2,1,0,0],[1,0,1,2]], dtype=np.float32)
K = np.array([[1,0,-1],[1,0,-1],[1,0,-1]], dtype=np.float32)

def conv2d_valid(x, k, stride=1):            # single channel, no padding
    kh, kw = k.shape
    oh, ow = (x.shape[0]-kh)//stride + 1, (x.shape[1]-kw)//stride + 1
    out = np.zeros((oh, ow), dtype=x.dtype)
    for i in range(oh):
        for j in range(ow):
            patch = x[i*stride:i*stride+kh, j*stride:j*stride+kw]
            out[i, j] = np.sum(patch * k)    # cross-correlation (no flip)
    return out

print(conv2d_valid(X, K))                                   # [[0,2],[-1,-1]]
print(F.conv2d(torch.from_numpy(X)[None, None], torch.from_numpy(K)[None, None])[0, 0])

class SmallCNN(nn.Module):
    def __init__(self, num_classes=10):
        super().__init__()
        self.features = nn.Sequential(
            nn.Conv2d(3, 32, 3, padding=1), nn.BatchNorm2d(32), nn.ReLU(),
            nn.Conv2d(32, 32, 3, padding=1), nn.BatchNorm2d(32), nn.ReLU(),
            nn.MaxPool2d(2),                                    # 32x32 -> 16x16
            nn.Conv2d(32, 64, 3, padding=1), nn.BatchNorm2d(64), nn.ReLU(),
            nn.MaxPool2d(2))                                    # 16x16 -> 8x8
        self.head = nn.Sequential(nn.AdaptiveAvgPool2d(1), nn.Flatten(),
                                  nn.Linear(64, num_classes))
    def forward(self, x):
        return self.head(self.features(x))

model = SmallCNN()
print(sum(p.numel() for p in model.parameters()))           # 29,546
print(model(torch.randn(8, 3, 32, 32)).shape)               # torch.Size([8, 10])

PyTorch tensors for images are laid out as (batch, channels, height, width), and convolution weights as (out_channels, in_channels, kH, kW) — hence the [None, None] to add batch and channel axes. The 29,546 parameters break down as 896 + 9,248 + 18,496 for the three convolutions, 64 + 64 + 128 for the batch-norm scales and shifts, and 650 for the classifier.

Classic architectures and what each one taught

Why residuals work. In a residual block the layers learn a correction $F(x)$ rather than a full mapping, and the gradient has a direct path: $\partial y / \partial x = I + \partial F / \partial x$. Even if $\partial F/\partial x$ is small, the identity term carries signal backward through dozens of blocks. A bottleneck block (1×1 conv to reduce channels, 3×3 conv, 1×1 conv to expand) keeps the cost of deep networks manageable; ResNet-50 and deeper use it.

1×1 convolutions mix channels at each pixel without looking at neighbours. They are cheap channel-wise linear layers used to shrink or expand channel counts. Depthwise-separable convolutions (a per-channel $k \times k$ conv followed by a 1×1 conv) cut parameters from $k^2 C_{in} C_{out}$ to $k^2 C_{in} + C_{in} C_{out}$ and are the basis of mobile architectures, including EfficientNet's building blocks.

Transfer learning with a pretrained backbone

Training a CNN from scratch needs a large labelled dataset. In practice you almost always start from a backbone pretrained on a large dataset such as ImageNet, because its early layers already detect generic edges, colours and textures. There are two standard regimes:

• Feature extraction (linear probing): freeze the backbone, replace the final classifier, train only the new head. Fast, needs little data, hard to overfit.
• Fine-tuning: unfreeze some or all of the backbone and train with a small learning rate (often 10× smaller than for the head), so the pretrained features adapt without being destroyed.

import torch, torch.nn as nn
from torchvision import models, transforms

weights = models.ResNet18_Weights.DEFAULT
model = models.resnet18(weights=weights)
for p in model.parameters():
    p.requires_grad = False                         # stage 1: freeze backbone
model.fc = nn.Linear(model.fc.in_features, 5)       # new trainable head, 5 classes

train_tf = transforms.Compose([
    transforms.RandomResizedCrop(224, scale=(0.6, 1.0)),
    transforms.RandomHorizontalFlip(),
    transforms.ColorJitter(0.2, 0.2, 0.2),
    transforms.ToTensor(),
    transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),
])
eval_tf = weights.transforms()                      # preprocessing matching pretraining
head_opt = torch.optim.AdamW(model.fc.parameters(), lr=1e-3)

# stage 2: unfreeze the last stage and fine-tune with discriminative learning rates
for p in model.layer4.parameters():
    p.requires_grad = True
ft_opt = torch.optim.AdamW([
    {"params": model.layer4.parameters(), "lr": 1e-4},
    {"params": model.fc.parameters(), "lr": 1e-3},
], weight_decay=0.01)

The Normalize statistics are the ImageNet channel means and standard deviations the backbone was trained with; using different preprocessing at inference than at pretraining silently costs accuracy. The weights enum API shown here is the current torchvision interface (older code used pretrained=True).

Data augmentation

Data augmentation applies random, label-preserving transformations to training images so the model sees a different version of each image every epoch. It is a regularizer that encodes invariances you believe in. Standard choices: random resized crops, horizontal flips, small rotations, colour jitter; stronger recipes add RandAugment-style random policies, Mixup (blend two images and their labels) and CutMix (paste a patch from one image into another and mix labels in proportion to area). Augment only the training set, and choose transformations that truly preserve the label: horizontal flips are fine for cats, wrong for reading text, and vertical flips are usually wrong for street scenes but fine for satellite imagery.

CNNs versus vision transformers

A vision transformer (ViT) (Dosovitskiy et al., 2020) cuts an image into fixed patches — 16×16 pixels for ViT-B/16, so a 224×224 image becomes $14 \times 14 = 196$ tokens — linearly embeds each patch (equivalent to a convolution with kernel size and stride both 16), adds position embeddings, and runs a standard transformer encoder (see the transformers lesson). Every patch can attend to every other patch from the first layer, so the receptive field is global immediately.

In practice the gap has narrowed from both sides: transformers borrowed hierarchical, local-window designs, and modernized CNNs adopted transformer training recipes. As a rule of thumb: for small data, edge devices or high resolution, start with a CNN or a hybrid; when a large pretrained ViT backbone is available for your domain, fine-tuning it is often the strongest option.

Common pitfalls

• Wrong preprocessing for a pretrained model: mismatched resize, crop or normalization statistics; or feeding BGR images (OpenCV's default) to an RGB-trained model.
• Forgetting model.eval() at inference: batch-norm then uses the current batch's statistics and dropout stays active, giving noisy, batch-dependent predictions.
• Fine-tuning with a high learning rate: wipes out pretrained features in a few hundred steps. Use a low rate for the backbone and warm up.
• Tiny batch sizes with batch norm: statistics become noisy. Freeze BN statistics or use GroupNorm.
• Augmentations that change the label: flipping digits, rotating text, colour-jittering tasks where colour is the signal.
• Shape bugs: channels-last arrays fed to a channels-first API, or miscomputed output sizes. Compute sizes with the formula, or print them after each layer.
• Leakage across splits: near-duplicate images (frames from the same video, the same patient) in both train and test make results look far better than they are.

In practice

• Start with a pretrained backbone from a maintained library (torchvision, timm, Hugging Face) rather than a hand-built architecture. ResNet-50 and EfficientNet-class models remain strong, cheap baselines; try a pretrained ViT or modern hybrid if latency allows.
• Train in two stages — head only, then partial or full unfreezing with discriminative learning rates — and track validation metrics per stage.
• Use mixed precision, channels-last memory format on GPUs, and a learning-rate schedule with warmup and cosine decay.
• Budget compute with the MAC formula; resolution is usually the most expensive knob because cost scales with $H \times W$.
• For deployment, export to ONNX or a runtime such as TensorRT or Core ML, quantize to INT8 where accuracy allows, and benchmark latency on the target device, not your workstation.
• The capstone capstone-image-classifier walks through this pipeline end to end.