Computer Vision: Tasks, Metrics, Models and Deployment

Computer vision (CV) is the field that turns pixels into decisions: reading a shipping label, finding a pedestrian in front of a car, measuring a tumor on a scan, flagging a defective weld on a production line. It is one of the most mature areas of applied AI, which means two things for you as a practitioner: there is almost always a strong pretrained model to start from, and the hard part of a real project is usually the data, the metric and the deployment target rather than the architecture.

The picture in your head

Think of an image as a spreadsheet of numbers: a color photo of 224 × 224 pixels is three stacked grids (red, green, blue) of 224 × 224 values, about 150,000 numbers. A vision model is a function that compresses those numbers into something useful. Early layers act like a team of very junior inspectors, each one only looking at a tiny patch and shouting when it sees an edge or a color blob. Later layers are managers who listen to many inspectors at once and say "those edges form a wheel" and then "those wheels and windows form a car." The tasks below differ mainly in what the final manager must report: one label, a list of boxes, or a label for every pixel.

The core tasks

Some vocabulary used throughout: a backbone is the feature-extracting network (for example a ResNet) that turns an image into a grid of feature vectors; a head is the small task-specific network on top (a classifier, a box regressor, a mask decoder). Most CV systems are "pretrained backbone + task head."

Classification

The model outputs a vector of logits $z \in \mathbb{R}^C$ for $C$ classes; softmax turns it into probabilities $p_c = e^{z_c} / \sum_{j} e^{z_j}$ and training minimizes cross-entropy $-\log p_{y}$ for the true class $y$. Top-k accuracy counts a prediction as correct if the true label is among the $k$ highest-probability classes. Top-5 accuracy was the headline metric on ImageNet because many images legitimately contain several objects and some classes (dozens of dog breeds) are hard even for people.

Detection

A detector must answer "what and where" for an unknown number of objects. A box is usually $(x_1, y_1, x_2, y_2)$, the top-left and bottom-right corners in pixels. Two families dominate:

• Two-stage detectors (the R-CNN line, e.g. Faster R-CNN) first propose candidate regions, then classify and refine each one. Accurate, slower.
• One-stage detectors (YOLO, SSD, RetinaNet) predict boxes and classes densely over a grid in a single pass. The original YOLO paper (Redmon et al., 2015) framed detection as a single regression problem from image to boxes, which is why the family became the default for real-time use.

Dense detectors predict many overlapping boxes for the same object. Classic pipelines clean this up with non-maximum suppression (NMS): sort boxes by confidence, keep the best, delete any remaining box whose IoU with it exceeds a threshold (often 0.5–0.7), repeat. NMS is a hand-tuned post-processing step that is awkward to run on some accelerators, so newer designs remove it. DETR (Carion et al., 2020) treated detection as set prediction with a transformer and a one-to-one matching loss, and Ultralytics' YOLO26, launched in January 2026, also advertises end-to-end NMS-free inference aimed at edge devices (as of November 2026 the YOLO family is maintained by several independent groups, so "YOLO" names an approach and a brand more than one lineage).

Segmentation

Segmentation is per-pixel classification. U-Net (Ronneberger, Fischer and Brox, 2015), designed for biomedical images, is still the mental model: an encoder downsamples the image into coarse, semantically rich features; a decoder upsamples back to full resolution; skip connections copy high-resolution encoder features into the decoder so boundaries stay sharp. The same encoder–decoder-with-skips shape reappears in diffusion models. Instance segmentation adds "which object" on top; Mask R-CNN (He et al., 2017) did it by adding a mask head to Faster R-CNN.

Pose, OCR and tracking

Pose models usually predict one heatmap per keypoint and take the peak. OCR is traditionally two stages, text detection then text recognition (a CNN or transformer reading a cropped line into characters, often trained with CTC loss, which you will meet in the speech lesson); vision-language models now read whole documents end to end, but dedicated OCR remains cheaper and more predictable at high volume. Tracking is usually "tracking by detection": run a detector on each frame, then associate boxes across frames using motion prediction (a Kalman filter), IoU overlap and appearance embeddings. Most identity switches happen at occlusions.

Metrics, worked through

Intersection over union

IoU measures how well a predicted region matches a ground-truth region:

where $|\cdot|$ is area (number of pixels). Worked example. Ground truth $A = (0, 0, 10, 10)$, prediction $B = (5, 0, 15, 10)$. Each box has area $10 \times 10 = 100$. The intersection spans $x \in [5, 10]$, $y \in [0, 10]$: area $5 \times 10 = 50$. Union $= 100 + 100 - 50 = 150$. IoU $= 50 / 150 = 0.33$. Note how harsh this is: the box is shifted by half its width and still "only" scores 0.33, so it would count as a miss at the common 0.5 threshold.

For segmentation, mIoU computes IoU per class over all pixels in the dataset and averages across classes, so rare classes count as much as common ones.

Average precision and mAP

A detection is a true positive (TP) if its IoU with an unmatched ground-truth box of the same class is at least a threshold $t$ (each ground-truth box can be matched once; extra detections of the same object are false positives). Sort all detections of one class by confidence and walk down the list, computing precision $= \text{TP} / (\text{TP} + \text{FP})$ and recall $= \text{TP} / N_{\text{gt}}$ after each one. Average precision (AP) is the area under the resulting precision–recall curve, after making precision monotonically non-increasing (each precision value is replaced by the maximum precision at any equal or higher recall). mAP averages AP over classes.

Worked example. One class, $N_{\text{gt}} = 4$ objects in the test set. The model emits five detections; sorted by confidence, they are TP, TP, FP, TP, FP.

Interpolated precision: at recall 0.25 and 0.50 the best precision at that recall or beyond is 1.00; at recall 0.75 it is 0.75; recall never reaches 1.0 (one object was never found), so that last segment contributes 0. AP $= 0.25(1.00) + 0.25(1.00) + 0.25(0.75) + 0.25(0) = 0.6875$.

Conventions differ, and you must state which one you report. Pascal VOC used IoU 0.5 ("AP50"). COCO's primary metric averages AP over ten IoU thresholds $t \in \{0.50, 0.55, \dots, 0.95\}$ and samples precision at 101 recall points, which rewards tight boxes. A model can rank first on AP50 and lower on COCO AP.

The code below implements both computations; it reproduces the numbers above.

import numpy as np

def iou(a, b):
    """Boxes as (x1, y1, x2, y2) with x2 > x1 and y2 > y1."""
    ix1, iy1 = max(a[0], b[0]), max(a[1], b[1])
    ix2, iy2 = min(a[2], b[2]), min(a[3], b[3])
    inter = max(0.0, ix2 - ix1) * max(0.0, iy2 - iy1)
    area_a = (a[2] - a[0]) * (a[3] - a[1])
    area_b = (b[2] - b[0]) * (b[3] - b[1])
    return inter / (area_a + area_b - inter)

def average_precision(tp_flags, n_gt):
    """tp_flags: 1 (TP) or 0 (FP) per detection, sorted by descending score."""
    tp = np.cumsum(tp_flags)
    fp = np.cumsum(1 - np.asarray(tp_flags))
    recall, precision = tp / n_gt, tp / (tp + fp)
    r = np.concatenate([[0.0], recall, [1.0]])
    p = np.concatenate([[0.0], precision, [0.0]])
    for i in range(len(p) - 2, -1, -1):          # make precision non-increasing
        p[i] = max(p[i], p[i + 1])
    steps = np.where(r[1:] != r[:-1])[0]          # where recall changes
    return float(np.sum((r[steps + 1] - r[steps]) * p[steps + 1]))

print(iou((0, 0, 10, 10), (5, 0, 15, 10)))        # 0.333...
print(average_precision([1, 1, 0, 1, 0], n_gt=4)) # 0.6875

Architectures you should recognize

CNN backbones. Convolutional networks (see the deep-learning course) slide small learned filters over the image, sharing weights across positions, which builds in translation equivariance and locality. ResNet (He, Zhang, Ren and Sun, 2015) added residual connections, $y = x + F(x)$, which made networks of 50–150+ layers trainable and is still a default baseline backbone. Later families (EfficientNet, ConvNeXt) tuned the recipe for accuracy per FLOP.

Vision Transformer (ViT). Dosovitskiy et al. (arXiv 2020, ICLR 2021) cut the image into fixed patches, for example $16 \times 16$ pixels, flattened and linearly projected each patch into a token, added position embeddings, and fed the sequence to a standard transformer encoder. A $224 \times 224$ image becomes $(224/16)^2 = 196$ patch tokens. ViTs have weaker built-in assumptions than CNNs, so they need more data or strong pretraining, but they scale well and share infrastructure with language models. Attention cost grows with the square of the token count, which is why high-resolution inputs are expensive.

CLIP-style zero-shot models. CLIP (Radford et al., 2021) trained an image encoder $f$ and a text encoder $g$ on roughly 400 million image–caption pairs with a contrastive objective: in a batch of $N$ pairs, push each image embedding toward its own caption and away from the other $N - 1$. At inference you classify without any task-specific training by writing class names as prompts ("a photo of a forklift") and picking the most similar:

where $x$ is the image, $t_c$ the prompt text for class $c$, and $s$ a learned temperature (scale). The same embeddings power image search and are the visual front end of many vision-language models.

Promptable segmentation (Segment Anything). Meta's SAM (Kirillov et al., 2023) introduced a model that takes an image plus a prompt (a click, a box or a rough mask) and returns masks for "the thing you pointed at," trained on the SA-1B dataset of more than a billion masks. It splits the work into a heavy image encoder run once per image and a light prompt-conditioned mask decoder run per click, which is what makes interactive use feel instant. SAM 2 (2024) extended this to video with a memory of past frames. SAM 3 (Meta, paper submitted November 2025) adds promptable concept segmentation: given a short noun phrase such as "yellow school bus", image exemplars, or both, it returns masks and identities for every matching instance in images and videos. As of November 2026 this is the clearest example of segmentation moving from fixed class lists to open-vocabulary prompts.

Datasets you will hear about

• ImageNet (ILSVRC subset): about 1.28 million training images in 1,000 classes; the classic pretraining and classification benchmark.
• COCO: everyday scenes with boxes and instance masks for 80 object categories, plus body keypoints for people; the standard detection and instance-segmentation benchmark.
• Pascal VOC: an older, smaller detection and segmentation benchmark (20 classes).
• Cityscapes and ADE20K: semantic segmentation of street and general scenes.
• MOT Challenge sequences for multi-object tracking.

Benchmark scores tell you a model learned those distributions. Your factory floor, scanner or camera will differ, so plan on a labeled evaluation set of your own.

Using a pretrained model

Torchvision ships ImageNet classifiers whose weights objects carry their own preprocessing and class names, which removes the most common bug (preprocessing that does not match training).

import torch
from torchvision.io import read_image, ImageReadMode
from torchvision.models import resnet50, ResNet50_Weights

weights = ResNet50_Weights.DEFAULT            # best available ImageNet weights
model = resnet50(weights=weights).eval()      # eval(): freeze batch-norm stats, no dropout
preprocess = weights.transforms()             # the resize/crop/normalize used in training

img = read_image("dog.jpg", mode=ImageReadMode.RGB)   # uint8 tensor (3, H, W)
batch = preprocess(img).unsqueeze(0)                  # float (1, 3, 224, 224)

with torch.inference_mode():
    probs = model(batch).softmax(dim=1)[0]

top5 = probs.topk(5)
for p, idx in zip(top5.values, top5.indices):
    print(f"{weights.meta['categories'][idx]:>25s}  {p.item():.3f}")

Detection works the same way; detectors take a list of images (they can differ in size) and return a list of dictionaries:

from torchvision.models.detection import (
    fasterrcnn_resnet50_fpn_v2, FasterRCNN_ResNet50_FPN_V2_Weights)

weights = FasterRCNN_ResNet50_FPN_V2_Weights.DEFAULT
model = fasterrcnn_resnet50_fpn_v2(weights=weights, box_score_thresh=0.5).eval()
img = read_image("street.jpg", mode=ImageReadMode.RGB)
with torch.inference_mode():
    out = model([weights.transforms()(img)])[0]
for box, label, score in zip(out["boxes"], out["labels"], out["scores"]):
    name = weights.meta["categories"][label]
    print(name, [round(v) for v in box.tolist()], f"{score.item():.2f}")

For open-vocabulary classification, Hugging Face's pipeline wraps a CLIP checkpoint; you supply the labels at call time:

from transformers import pipeline

clf = pipeline("zero-shot-image-classification", model="openai/clip-vit-base-patch32")
labels = ["a photo of a forklift", "a photo of a pallet", "a photo of a person"]
for r in clf("warehouse.jpg", candidate_labels=labels):
    print(f"{r['label']:>25s}  {r['score']:.3f}")

When zero-shot accuracy is not enough, the next step is transfer learning: replace the classifier head (model.fc on a ResNet) with a new nn.Linear for your classes, train only the head first, then optionally unfreeze the backbone with a small learning rate. A few hundred well-labeled images per class often beat a zero-shot model on a narrow domain.

Deployment considerations

• Latency budget. At 30 frames per second you have about 33 ms per frame for decode, preprocess, inference and post-processing combined. Measure end to end, not just the model.
• Resolution is a dial. Halving input width and height cuts CNN compute roughly 4×, but small objects vanish. Tiling (running on overlapping crops) recovers small objects at extra cost.
• Export and quantize. Production vision models are commonly exported to ONNX and run with an optimized runtime (TensorRT on NVIDIA, OpenVINO on Intel, Core ML on Apple, TFLite/LiteRT on mobile). INT8 quantization usually costs little accuracy for CNNs but must be validated on your own data with a calibration set.
• Post-processing lives on the CPU unless you design otherwise; NMS over thousands of boxes can dominate latency on small devices, one reason NMS-free detectors appeal at the edge.
• Privacy and law. Faces, license plates and bodies are personal data in many jurisdictions, and some biometric uses are restricted or prohibited (the EU AI Act, for example, singles out certain facial-recognition practices). Blur or crop at the edge when you can.

Common pitfalls

• Preprocessing mismatch. OpenCV loads images as BGR; torchvision and PIL use RGB. Forgetting to convert, or normalizing with the wrong mean and standard deviation, silently drops accuracy rather than crashing.
• EXIF orientation. Phone photos store rotation in metadata; some loaders ignore it, so your model sees sideways images in production that were upright in your labeling tool.
• Leakage across splits. Frames from the same video, or images of the same patient, in both train and test inflate scores dramatically. Split by video, patient, site or time.
• Reporting an unnamed mAP. "mAP 0.62" is meaningless without the IoU threshold(s), the dataset split and the image size.
• Ignoring the confidence threshold. mAP integrates over all thresholds, but production picks one. Choose it from the precision–recall trade-off your application needs.
• Trusting zero-shot labels blindly. CLIP-style models inherit web biases and are sensitive to prompt wording; evaluate the prompt set like any other hyperparameter.
• Class imbalance in segmentation. Background pixels dominate; plain cross-entropy can learn to predict "background" everywhere. Use per-class IoU, Dice or focal losses, and look at rare classes separately.

In practice

Professionals start from the strongest pretrained model that fits their latency and licensing constraints, measure it zero-shot or with a linear head on a few hundred in-domain labeled examples, and only then decide whether to fine-tune. Labeling is the main cost, so they use promptable segmenters and existing detectors to pre-label and have humans correct, which is often several times faster than drawing from scratch. They build an evaluation set that mirrors deployment (same cameras, lighting, angles), report the exact metric convention, track per-class and per-condition (night, rain, occlusion) slices, and monitor input drift after launch because a new camera or a dirty lens can break a model that never changed. Check model licenses early: several popular detector code bases are under copyleft licenses (AGPL-3.0 in Ultralytics' case), which matters for commercial products. Finally, remember that general vision-language models can now answer many one-off visual questions, but a small specialized detector is still far cheaper, faster and easier to validate when the same question is asked millions of times a day.