Vision

Everything we've built so far — the whole transformer story — was about sequences of words. Now we point the same machinery at the visual world, and a surprising amount of it carries straight over: the same vanishing-gradient villain, the same residual-connection hero, the same "tokenize everything and let attention sort it out" punchline. But vision also has its own beautiful set of ideas, built around one operation — the convolution — that was purpose-designed for images.

Here's the road we'll travel, and like last time, every step is a fix for the thing before it. We start with what an image even is to a computer. Then we ask the obvious question — "can't I just feed pixels to a regular neural network?" — watch it fail, and that failure gives us the CNN. We stack CNNs into a feature hierarchy, then put them to work: first detecting objects with YOLO, then labeling every pixel with segmentation (FCN → U-Net → Mask R-CNN), then the foundation model that segments anything (SAM). Finally we come full circle: drop the convolution entirely, chop the image into patches, and feed them to a transformer — the Vision Transformer — which leads straight to vision-language models, where images and text live in the same stream of tokens.

I'm assuming you know what a neural network is and roughly how a GPU works, and that you've seen the transformer (we'll lean on it once we reach ViTs). Everything else, we build from the ground up — and since this is the visual chapter, we'll lean hard on diagrams. Let's go.

Before any neural network can touch an image, we have to agree on what an image is to a computer. Spoiler: it's just a big grid of numbers. Once you see that clearly, everything else clicks into place.

A digital image is a grid of pixels. Each pixel is a tiny block of color described by numbers — usually three of them: the red, green, and blue (RGB) intensities. Each one is an integer from 0 to 255 (that's 8 bits per channel). So a single pixel is a 3-number vector like (200, 50, 80) — a pinkish red.

Stack that grid up and an image becomes a 3D tensor with three axes:

Here's the thing that trips people up, and it's worth getting straight right now: the channel dimension is conceptually way more important than just "RGB." For the input image, sure, channels are colors. But after the very first convolutional layer, channels stop meaning colors and start meaning learned features — maybe one channel lights up on horizontal edges, another on red blobs, another on diagonal stripes. By the time you're deep in a CNN, each channel stands for some abstract pattern the network taught itself to look for, and the channel count typically grows from 3 at the input to hundreds or thousands deep in. Hold onto that — it's the heart of how CNNs work.

A concrete example: a 1024×768 RGB image is a tensor of shape (3, 768, 1024) in PyTorch's convention (channels first) or (768, 1024, 3) in TensorFlow/NumPy's convention (channels last). Same data, different bookkeeping.

When you stack multiple images into a batch for training, you add a fourth axis:

(N, C, H, W) — N images, C channels, height H, width W.

That's the input tensor shape your CNN expects, and basically every other shape in this chapter eventually traces back to it.

Let's see all of that in one picture:

Raw pixel values in [0, 255] make terrible inputs to a neural network. They're too large and not centered around zero, which makes optimization jumpy and unstable. So we always preprocess, in two steps:

1. Divide by 255 to squeeze every value into [0, 1].

2. Subtract the dataset mean and divide by the dataset standard deviation, computed per channel. This re-centers the data around zero with a consistent spread.

For models trained on ImageNet, the standard normalization uses mean = (0.485, 0.456, 0.406) and std = (0.229, 0.224, 0.225) — one number per RGB channel. The golden rule: normalize the exact same way at inference as you did at training. If you trained on normalized inputs and then feed raw pixels at test time, the model sees garbage.

If that felt a little abstract — why these numbers matter, why zero-centering helps — hang tight. The next sections wire it all back into the network, and it should click.

So: how do we actually feed images to a neural network? Your first instinct is probably the natural one — an image is just numbers, so why not flatten it and pour it into a plain fully-connected network (an MLP), like any other input? Let's try exactly that and watch it fall apart, because the way it fails tells us precisely what to build instead.

Take a 224×224×3 image. Flatten it into one long vector and you get 150,528 numbers. Feed that to a fully-connected network. Two problems make this a disaster.

Parameter explosion. The first layer's weight matrix would be 150,528 × hidden_dim. For even a modest hidden_dim of 1,000, that's 150 million parameters in the first layer alone. Networks that big do train, but you'd be burning your entire parameter budget just learning to look at individual pixels.

No translation invariance. A dog in the top-left corner and the same dog in the bottom-right corner would activate completely different input neurons. The network would have to learn "what a dog looks like" separately for every possible position in the image. With any finite amount of training data, that's hopeless.

The fix is a specialized architecture: the Convolutional Neural Network. CNNs are built around two facts about images that the MLP ignored:

The convolution operation is engineered around exactly these two facts. Let's build it.

A convolution slides a small filter (also called a kernel) across the image, computing a dot product at each position. A filter is a small tensor of weights — typically 3×3 or 5×5 in spatial size — with the same number of channels as its input. At each spatial position you run this little algorithm:

1. Multiply the filter element-wise with the patch of input it's sitting on.

2. Sum all those products.

3. Add a bias.

4. Write the result to that position in the output.

5. Slide the filter one step over and repeat, until you've covered the whole input.

The output is a 2D feature map showing how strongly that filter's pattern is detected at each location. Mathematically, for an input $I$ and filter $K$:

Reading the symbols:

That triple sum is the whole operation. Here's the slide-and-multiply in motion:

Two quick notes for completeness. In full generality, with explicit kernel size $k$ and input channels $C_{in}$:

where $y(i,j)$ is the output, $x$ the input, $w$ the filter (kernel size $k \times k$ with the same channel count $C_{in}$ as the input), and $b$ a single learned bias added to every output position.

And a technical aside worth knowing: what we just described is technically cross-correlation, not true convolution. A real mathematical convolution would flip the filter before applying it. In deep learning everyone calls it convolution anyway, because the filter is learned — flipping or not flipping makes no difference, the network just learns whatever weights work either way.

Let's make this concrete with real numbers. Take a tiny 5×5 grayscale input:

And this 3×3 filter — roughly a horizontal-edge detector:

To get the output at position (0,0) — the top-left output cell — overlay the filter on the top-left 3×3 patch:

Multiply element-wise and sum:

So output(0,0) = 5. Now slide one step right:

So output(0,1) = 10. Keep sliding across and down. A 5×5 input convolved with a 3×3 filter gives a 3×3 output by default — we'll see exactly why in a moment.

What is this filter actually doing? It returns a big positive value wherever the bottom row of the patch is brighter than the top row — that is, wherever there's a horizontal edge running from dark-on-top to light-on-bottom. The value 5 at (0,0) means a moderate horizontal edge there; the 10 at (0,1) means an even stronger one just to the right. That's what "detecting a pattern" means: the output is highest where the input matches what the filter is looking for.

One filter produces one feature map — one kind of pattern detected across the image. But the world has many kinds of patterns: horizontal edges, vertical edges, diagonals, color blobs, textures. So a conv layer uses many filters in parallel, each hunting for a different thing.

If a layer has $K$ filters and the input has $C_{in}$ channels, then:

So for an input of shape (3, 224, 224) — RGB, 224 pixels a side — a conv layer with 64 filters of size 3×3 produces an output of shape (64, 224, 224). Each of those 64 output channels is one filter's response across the whole image.

This is the key reason CNNs scale so gracefully: the input has 3 channels (R, G, B), but after one layer you have 64 abstract feature channels. After another, 128. After another, 256. The network keeps building richer and richer descriptions of the image. And — exactly as promised earlier — after that first layer "channels" no longer mean colors. Maybe channel 7 fires on vertical edges, channel 23 on red blobs, channel 41 on diagonal textures. Deeper still and channels mean eyes, faces, wheels, paws. The network discovers all of this from data; nobody assigns the meanings.

Let's actually count, because the savings are staggering. Input (C_in, H, W) = (3, 224, 224), filter size 3×3, output channels K = 64.

Now compare to a fully-connected layer mapping the flattened input to just 1,000 outputs: $150{,}528 \times 1{,}000 = 150$ million parameters. That's an 80,000× reduction. And here's the kicker: those 1,792 parameters are reused at every one of the $224 \times 224 = 50{,}176$ spatial positions. The CNN gets translation invariance for free out of this exact design — the same filter looks for the same pattern everywhere.

Back in the worked example, a 5×5 input gave a 3×3 output. Why did it shrink? Because we only placed the filter where it fully fit inside the input — and a 3×3 filter only has 3 valid starting positions along a 5-wide axis. In a deep network that shrinking is a real problem: after a few layers your image would dwindle to nothing.

Padding fixes it. Add a border of zeros around the input before convolving. With padding $p$, the input effectively becomes $(H + 2p) \times (W + 2p)$. With $p = 1$ for a 3×3 filter, the output comes out the same spatial size as the input — this is called "same" padding, and nearly every modern CNN uses it.

Stride is how far the filter jumps between positions. Stride 1 means "slide one pixel at a time." Stride 2 means "skip every other position," which halves the output's spatial size and is a common way to downsample.

The output spatial size formula ties it all together:

where $p$ is padding, $k$ is kernel size, $s$ is stride, and $\lfloor \cdot \rfloor$ is the floor (round down). Let's plug in a few cases:

A conv layer is a linear function of its input. And stacking linear layers just gives you another linear function — no matter how many you pile up, the whole thing collapses into one single linear transformation. That's useless for the rich, nonlinear patterns in images. (If this argument feels familiar, it's the same reason the transformer's feed-forward block needed a nonlinearity — the villain and the fix recur across architectures.)

So after every conv layer we apply a nonlinear activation function, almost always ReLU (Rectified Linear Unit):

That's the whole thing: any negative value becomes 0, positives pass through unchanged. ReLU is dirt cheap to compute, it doesn't saturate (its gradient is 1 everywhere positive, so signal flows nicely during backprop), and empirically it trains deep networks far better than the older sigmoid and tanh. Modern variants include Leaky ReLU (a small negative slope instead of a hard zero), GELU (a smooth version used in transformers), and SiLU/Swish (smooth, used in some recent CNNs) — but for classic CNN work, plain ReLU is the default.

The standard conv-layer pattern, end to end, is Conv → BatchNorm → ReLU. That batch normalization step (introduced in 2015) normalizes activations to zero mean and unit variance across the batch, which dramatically stabilizes training and is nearly universal in modern CNNs.

After several conv layers you usually want to downsample — shrink the spatial resolution to focus on what matters and cut the compute. There are two ways:

Strided convolution — use stride 2 in a conv layer to halve the spatial size while learning the downsampling (the filter weights decide what to keep).

Pooling — a fixed, non-learned downsampling. The most common is max pooling: slide a small window (typically 2×2 with stride 2) across the feature map and keep only the maximum value in each window. For a 2×2 max pool with stride 2, every 2×2 patch becomes one output value — the max of those four numbers — so both spatial dimensions halve.

Why max instead of average? Max pooling preserves the strongest response — if some pixel in the window fires hard for "horizontal edge," that signal survives. Average pooling would water it down. Max pooling also hands you a little translation invariance for free: nudge a feature by a pixel within a 2×2 window and the max is unchanged.

There's also global average pooling, used at the very end of a network: average each entire feature map down to a single scalar, collapsing a (C, H, W) tensor to a length-C vector. This replaced the heavy fully-connected layers of older CNNs and was popularized by ResNet.

Here's the deepest idea about CNNs — the reason they work as well as they do. By stacking conv + pool blocks, the network builds a hierarchy of features of increasing complexity. Visualize what the filters in a trained CNN respond to and you see a clear progression:

Nobody programs this hierarchy in. It emerges from training. The network discovers, by itself, that the way to recognize a dog is to first find edges, combine edges into textures and shapes, combine shapes into dog-parts, and combine parts into a whole dog. That compositional structure is exactly how vision researchers — and probably biological visual systems — think about the problem. And the reason deeper layers can represent bigger concepts comes down to one idea: their receptive fields are larger. Let's unpack that.

The receptive field of a neuron is the region of the input image that affects its value. Early-layer neurons see only a small patch — each looked at just a 3×3 region of the input. But deeper neurons combine outputs from many earlier neurons, so they end up seeing much more of the original image.

Here's how it grows for stacked 3×3 conv layers:

Pooling and strided convolutions speed this up dramatically — a stride-2 layer effectively doubles the receptive field of everything after it. By the end of a deep CNN like ResNet-50, individual neurons can have receptive fields covering the entire input image. That's how deep neurons can "see" whole objects: their window is finally big enough to contain them.

Let's trace a full CNN — a simplified ResNet-style classifier — from input to output, watching the tensor shapes change at every step. Keep one eye on the pattern: spatial size shrinks while channel count grows.

Input: a single 224×224 RGB image, shape (3, 224, 224).

Block 1 — Stem.

Block 2.

Block 3.

Block 4.

Block 5.

Head.

Notice the pattern: as we go deeper, spatial dimensions shrink (224 → 7) while channel count grows (3 → 512). The network is trading spatial resolution for semantic depth. Early on you have lots of pixels each described by 3 numbers (R, G, B). At the end you have just 7×7 = 49 spatial positions, but each is described by 512 abstract feature dimensions. Then global pooling throws away spatial info entirely and the fully-connected layer produces a class score. This is the canonical CNN recipe — every classifier from AlexNet to EfficientNet follows some variant of it.

Training is exactly what you saw in earlier chapters: forward pass, compute loss, backprop, update weights. The only CNN-specific wrinkle is how gradients flow through the convolution. For each training example you (1) run the image forward to get logits, apply softmax, and compute cross-entropy loss against the true label; (2) backpropagate the gradient of the loss to every parameter — for conv layers, how each filter weight should change; and (3) update with an optimizer like SGD-with-momentum or Adam. Neat fact: the gradient of a conv layer is itself computed via a convolution (with flipped filters), which is why frameworks like PyTorch implement conv backprop efficiently on GPU.

The training tricks that matter most for CNNs: data augmentation (random crops, horizontal flips, color jitter, mixup — essential; without it CNNs overfit most datasets), batch normalization (normalize activations within each batch — stabilizes training, allows higher learning rates), learning-rate schedules (start high, decay via cosine annealing or step decay — critical for converging well), and pretraining (train on ImageNet first, then fine-tune on your task — this transfer-learning approach is overwhelmingly the default; training from scratch on a small dataset is rarely the right move).

To wrap up CNNs, here are the three ideas — beyond convolution itself — that turned them from a 1990s curiosity into the dominant vision architecture for a decade:

1. ReLU activations (AlexNet, 2012). Replaced saturating sigmoid/tanh with $\max(0, x)$. Gradients flow much better; training is faster.

2. Batch normalization (2015). Normalizes activations within each mini-batch. Enables deeper networks, higher learning rates, and far less sensitivity to weight initialization.

3. Residual connections (ResNet, 2015). Skip connections that add a block's input back to its output. They let gradients flow back through arbitrarily deep networks — and this is, no exaggeration, the single most important architectural idea since convolution itself. It's the exact same idea used in transformers today. (Remember the $+x$ skip path from the transformer chapter? Same hero, different architecture.)

Together with the convolution operation, these are the load-bearing ideas in every modern vision CNN. Understand them and you understand why every detection model (YOLO, R-CNN), every segmentation model (U-Net, Mask R-CNN), and even diffusion models (which run on U-Nets) share the same conceptual DNA.

Funny name, huh? YOLO became one of the most popular computer-vision architectures around, and it's worth understanding why the design is so good. But first we need to know what came before it — because, just like in the transformer story, YOLO is best understood as a reaction to the slow, clunky thing it replaced.

A quick framing of the task. Classification answers "what's in this image?" with one label. Detection is harder: it answers "what objects are here, and where?" — drawing a bounding box around each object and labeling it. That "where" is the whole challenge.

The dominant detection method before deep learning was DPM (Deformable Parts Model). It worked by sliding window: take a classifier for your target object, slide it across the image at evenly spaced locations and at multiple scales, and at each stop ask "is the object here?" For each window the model would extract hand-crafted features (typically HOG — Histogram of Oriented Gradients), score the window against a learned template for the object's overall shape, score against templates for the object's parts (legs, head, wheels), and combine the scores while allowing some deformation between parts.

The problems were severe: the pipeline was complex (feature extraction, root filter, part filters, deformation cost, post-processing — each piece designed or trained separately, no joint optimization), it was slow (even the fastest variant couldn't really do real-time general detection), and the hand-crafted HOG features worked for some objects like pedestrians but couldn't capture the diversity of natural images the way learned features could.

The first deep-learning detectors — R-CNN (2014), then Fast R-CNN, then Faster R-CNN — replaced hand-crafted features with CNNs but kept a two-stage structure:

Then, on top of that: a separate linear model to refine boxes, NMS to remove duplicates, and rescoring based on context. The numbers tell the story — original R-CNN took more than 40 seconds per image at test time; even Fast R-CNN at 0.5 frames per second was nowhere near real-time. Too many parts, each trained separately, each adding its own cost.

Both DPM and R-CNN share one deep flaw: they repurpose a classifier to do detection. They take a classifier, run it at many locations, and post-process the results. That means each component is trained and tuned separately; the classifier never sees the whole image (only local patches or proposals), so it can't reason about context; and the pipeline is slow because so many separate operations have to run.

The YOLO authors' core insight: detection should be one regression problem, optimized end-to-end on the actual goal — good bounding boxes and class predictions — not a stack of separately trained classifiers.

The YOLO inference recipe is only three steps:

1. Resize the input image to 448×448.

2. Run a single CNN on the image.

3. Threshold the resulting detections by confidence (and apply NMS).

One image goes in, a list of detection boxes comes out — a genuine one-shot pipeline. Unlike R-CNN, which effectively looks at the image thousands of times, with YOLO you only look once. (NMS, by the way, is Non-Maximum Suppression — a post-processing step that removes duplicate boxes and keeps the best detections. We'll cover it properly below.)

Now the architecture at a high level — it's organized around a grid:

Step 1 — divide the image into an S × S grid. For PASCAL VOC, S = 7, so the 448×448 image is conceptually broken into a 7×7 grid, each cell covering a 64×64-pixel region.

Step 2 — each cell predicts B bounding boxes plus C class probabilities. For VOC, B = 2 (each cell proposes two candidate boxes) and C = 20 (twenty object classes).

Step 3 — the "responsible" cell. If the center of an object falls inside a grid cell, that cell is responsible for detecting it. This is the critical rule: an object belongs to exactly one cell, regardless of how big it is. A dog whose center lands in cell (3, 4) is detected by cell (3, 4), even if its body sprawls across many cells.

So the model's prediction is entirely spatially organized. The output isn't just a flat list of boxes — it's a 3D tensor laid out spatially, where each (row, column) slot predicts what's centered in the corresponding region of the image.

The network has 24 convolutional layers followed by 2 fully-connected layers (the paper's Figure 3). It's inspired by GoogLeNet but simpler — instead of inception modules, YOLO alternates 1×1 reduction layers with 3×3 convolutions.

What's a 1×1 convolution? It mixes channels at a single spatial position without looking at neighbors — it's used for dimensionality reduction. If a feature map has 512 channels and you want to cut it to 256 before an expensive 3×3 conv, a 1×1 conv is the cheapest way to do it. The 1×1 → 3×3 pattern became standard after this paper.

Reading the shape progression off the paper's Figure 3:

The final output is reshaped to a 7 × 7 × 30 tensor — that's the entire detection prediction for the image. Where does 30 come from? For VOC, S = 7, B = 2, C = 20, and each cell predicts: B = 2 boxes × 5 numbers each (x, y, w, h, confidence) = 10, plus C = 20 class probabilities, for a total of 30 channels per cell. The 7 × 7 spatial layout corresponds directly to the 7×7 grid over the image, so cell (i, j) of the output describes grid cell (i, j) of the input.

The network's convolutional layers are pretrained on ImageNet classification first. (ImageNet is a massive set of over 14 million images across more than 20,000 categories.) Then those pretrained conv layers are transferred to detection. In the original paper, the authors: take the first 20 conv layers, append an average-pooling and a fully-connected layer, train on ImageNet at 224×224 for about a week, and hit 88% top-5 accuracy (comparable to GoogLeNet). Then they convert to detection by adding 4 more conv layers and 2 fully-connected layers (randomly initialized) and doubling the input resolution to 448×448, since detection needs finer detail than classification. This pretrain-then-fine-tune approach is now standard for nearly every detection model.

(Side note — Fast YOLO uses the same training setup but only 9 conv layers instead of 24, with fewer filters. It runs at 155 FPS on a Titan X GPU, trading some accuracy for speed while staying more than 2× as accurate as any prior real-time detector.)

Every layer except the final one uses Leaky ReLU:

Standard ReLU outputs zero for any negative input, which can cause dead neurons — neurons stuck outputting zero that never recover, because zero output means zero gradient. Leaky ReLU keeps a small slope (0.1) for negative inputs, so a little gradient still flows even when the neuron isn't firing. The final layer uses a linear activation, because it predicts coordinates and probabilities that need to range over the real numbers.

To reiterate, YOLO produces bounding boxes with coordinates x, y, w, h. Each of the B = 2 boxes per cell is described by four spatial numbers plus a confidence:

x, y — the box center, relative to the grid cell, normalized to [0, 1]. So x = 0.5, y = 0.5 puts the center smack in the middle of the cell; x = 0 is the left edge, y = 1 the bottom edge. By construction, the box center cannot leave its responsible cell.

w, h — width and height, relative to the whole image, normalized to [0, 1]. So w = 0.5 means the box spans half the image's width. Note these are not relative to the cell — they're relative to the entire image, because objects can be far larger than one cell.

This is a careful, deliberate design. It means (x, y) are tightly bound to [0, 1] and the network just learns small offsets from a known cell center, while (w, h) can express any object size from tiny to image-spanning.

The fifth number per box is the confidence C, which the paper defines as:

In words: confidence is the probability that an object exists in this box, multiplied by how good the box is — measured as the IOU (Intersection over Union) between the predicted box and the ground-truth box. So:

A single number thus encodes both existence and accuracy — a high-confidence box is more likely both to contain an object and to localize it well. Let's pin down IOU, since it shows up everywhere in detection:

IOU = 1 means the boxes are identical; IOU = 0 means they don't overlap at all; IOU > 0.5 usually means "significant overlap, probably the same object."

The remaining C = 20 numbers in each cell are conditional class probabilities:

The "conditioned on object" part is the key: these answer "given that an object is in this cell, what class is it?" They don't need to be zero when there's no object — they're only meaningful when an object is present. And importantly, YOLO predicts one class-probability vector per cell, regardless of how many boxes B that cell predicts. Both of a cell's boxes share the same class vector — which leads to a limitation we'll discuss shortly.

At inference, the network hands you confidence and class probabilities separately. To get the final class-specific score for a box, just multiply:

This single score per (box, class) pair combines the probability of that class, and how well the box fits. Now you have, for every box in every cell, a confidence score for every class — you filter out the low-confidence ones, apply NMS, and you're done.

For a 7 × 7 grid with B = 2 boxes per cell, the network outputs 7 × 7 × 2 = 98 bounding boxes per image. Compare that to R-CNN's ~2,000 region proposals! Far fewer candidates, all generated in parallel by one network. Most of those 98 will have very low confidence (no object in that cell) and get filtered out at the thresholding step. Cool, right?

The grid design already prevents most duplicate detections — most objects fall cleanly into one cell. But some objects, especially large ones or those near cell borders, get detected by multiple cells. NMS cleans these up. The algorithm, run for each class separately:

1. Take all boxes predicted for that class with class-specific confidence above some threshold.

2. Sort them by confidence, highest first.

3. Take the top box and add it to the final output.

4. Compute IOU between this top box and all remaining boxes.

5. Discard any box whose IOU with the top box exceeds a threshold (e.g. 0.5) — those are duplicates of the same object.

6. Repeat from step 3 with the next-highest remaining box, until none are left.

How important is NMS to YOLO? The paper makes a subtle point: NMS adds only 2–3% mAP to YOLO — far less than it adds to R-CNN or DPM. Why? Because YOLO's grid already enforces spatial diversity (each cell can only predict objects centered there), so most duplicates never arise in the first place. R-CNN's region proposals can overlap freely, so it leans heavily on NMS. YOLO's structure has NMS partially built in via the grid; NMS just polishes the edges.

The total loss is $L = L_{\text{cls}} + L_{\text{loc}}$ — a classification part and a localization part — and the trick that makes the whole thing trainable is that everything is squared error, which turns detection into a regression problem. The full loss has five terms:

A couple of those weights deserve a word. $\lambda_{\text{coord}} = 5$ up-weights localization so getting boxes right matters more than the raw class terms. $\lambda_{\text{noobj}} = 0.5$ down-weights the flood of empty cells so they don't drown out the signal from the few cells that actually contain objects. And the $\sqrt{w}, \sqrt{h}$ trick: a small absolute error in a small box hurts IOU far more than the same error in a large box; taking the square root compresses large values so that equal relative errors are penalized more equally across box sizes.

Each cell predicts B = 2 boxes, but at training time we want exactly one of them responsible for any given object. The rule (paper section 2.2): assign the predictor whose box currently has the highest IOU with the ground truth. So if cell (3, 4) holds a dog, both its box predictors make predictions, and whichever has the higher IOU with the true dog box is declared responsible. Only that responsible predictor incurs the coordinate and box-confidence losses for the object; the other gets a no-object signal.

This produces specialization: the two predictors in each cell learn to handle different kinds of boxes — one might gravitate toward tall, narrow boxes (people), the other toward wide, short ones (cars). The authors note this "improves overall recall," since different shapes get handled by different predictors.

Training hyperparameters, from the paper, for completeness: 135 epochs on PASCAL VOC 2007 + 2012; batch size 64, momentum 0.9, weight decay 0.0005; a learning-rate schedule that warms up from $10^{-3}$ to $10^{-2}$ over the first epochs (jumping straight to $10^{-2}$ would make training diverge), holds $10^{-2}$ for 75 epochs, drops to $10^{-3}$ for 30, then $10^{-4}$ for the final 30; dropout 0.5 after the first FC layer; and data augmentation with random scaling/translation up to 20% of image size plus random exposure and saturation jitter up to 1.5× in HSV space. The warm-up-then-decay schedule is a familiar pattern in modern training, and the augmentation — modest by today's standards — is still critical to avoid overfitting VOC's smallish training set.

The paper is refreshingly honest about its model's weaknesses (section 2.4). Worth knowing, because every later YOLO version is largely about fixing these.

Strong spatial constraints. Each cell predicts only B = 2 boxes and only one class. So a cell containing two objects of different classes is in trouble — it can predict two boxes but only one class vector, so a person standing next to a bird in the same cell forces a choice. And groups of small objects (a flock of birds, many centers in one cell) simply can't all be predicted. This is structural; v2 onward loosened it dramatically with anchor boxes and higher S.

Poor generalization to unusual aspect ratios. YOLO learns box shapes from data, so it struggles with objects in configurations or aspect ratios it didn't see in training. There's no principled handling of out-of-distribution box shapes — it just relies on having seen enough examples.

Coarse features. The 448×448 input is downsampled hard before prediction, so each output position corresponds to a 64×64 image patch (448/7). Small objects may not have enough features left at that resolution to be detected well.

The loss doesn't match the goal. Sum-squared error treats localization and classification errors as comparable (even the $\lambda_{\text{coord}} = 5$ trick only partly fixes this), and treats errors in large and small boxes similarly (the $\sqrt{\cdot}$ trick helps but doesn't fully fix it). The actual goal is mean Average Precision (mAP), a complex ranking metric; squared error is a convenient proxy that usually does the right thing but not always.

Detection draws a box around each object. Segmentation goes finer: it figures out what each pixel belongs to. The output of a segmentation model is a mask the same spatial size as the input — for a 512×512 input, that's a 512×512 grid of labels, 262,144 predictions per image. Three flavors are worth knowing:

There's a tension here that classification simply doesn't have. A classification CNN aggressively downsamples — from 224×224 down to a 7×7 final feature map, a 32× reduction. That downsampling is good for classification: the deepest features have huge receptive fields and capture the whole object, and you don't need spatial precision because you only output one label.

Segmentation needs both deep semantics and full resolution at once. You need the deepest features to know what you're looking at, but you also need to output a prediction at every original pixel. A 7×7 mask for a 224×224 input is useless — that's one label per 32×32 patch. The classical CNN treats spatial resolution and semantic depth as a tradeoff: you get one or the other, not both. To segment, you need a way to recover the lost resolution while keeping the deep semantic information. Every modern segmentation architecture exists to solve exactly this.

The first deep approach to segmentation was the Fully Convolutional Network. The idea: take a classification CNN and turn it into a segmentation model by removing the fully-connected layers and replacing them with convolutional ones that output a per-pixel prediction.

The key realization (from the FCN paper) is that a fully-connected layer is mathematically the same as a convolution with a kernel the size of its input — same weights, same arithmetic. So you can "convolutionalize" the classifier. Now its output goes from one vector per image to a grid of class scores — a small spatial map. Run a 224×224 image through a converted VGG-16 and you get roughly a 7×7 map where each spatial cell holds 21 class scores (for PASCAL VOC's 21 classes).

But that's still a 7×7 grid, and you need 224×224 for a per-pixel mask. The fix is upsampling: blow the small score grid back up to the input size with a learnable upsampling operation (originally transposed convolution, which we'll detail soon). The full FCN pipeline: run the input through a CNN backbone to get a 7×7×21 score grid, upsample 32× back to 224×224×21, then take the argmax along the class dimension to get a 224×224 mask.

This worked — it was the first end-to-end neural network for semantic segmentation, and it crushed the hand-engineered approaches. But it had a problem: upsampling from 7×7 straight to 224×224 produces blurry, low-detail masks. The model knows what's in the image but loses precision about where the boundaries are. The reason is fundamental — by the time you reach the 7×7 map, the spatial detail is gone. You can upsample the resolution arithmetically, but you can't conjure back information that was thrown away. You'd need to combine the deep features (which know what) with shallow features (which still hold the spatial where). FCN partially patched this with skip connections from earlier layers (FCN-16s, FCN-8s) — small but real gains — but the architecture wasn't designed around the idea; it was bolted on. The model that was designed around it: U-Net.

U-Net (Ronneberger, Fischer, Brox, 2015) came out the same year as FCN, originally for medical image segmentation, and it's now the most influential segmentation architecture there is — the foundation of countless models, including modern diffusion models. The reason it took over: it solved the resolution-versus-semantics problem with a clean, symmetric design.

As the name says, the architecture is shaped like a U. The left side is the encoder, the bottom is the deepest, most compressed point, the right side is the decoder, and arcing across the U are the skip connections — the special ingredient.

Why do the skip connections matter so much? Walk through the problem. The encoder loses spatial detail as it downsamples — each pooling step throws away exact pixel positions. By the bottom of the U, you have rich semantics ("there's a dog here") but no precise edges ("the outline is at exactly these pixels"). The decoder upsamples back to full resolution but, working only from the coarse bottom, lacks that original detail. The skip connections deliver the original spatial detail directly across: when the decoder is at the 64×64 stage, it receives the encoder's 64×64 features — features that never went through the downsample-and-upsample cycle and still know exactly where the edges are. Mathematically, at each decoder level:

The concatenation stacks the channels — if the upsampled features have 256 channels and the encoder features have 256, you get 512 after concatenation — and the following conv layer learns to merge the two sources: deep semantics from the upsampled path, spatial detail from the skip path. This is the architectural realization of "both at the same time": semantics flow up through the bottom and decoder, spatial detail flows across through the skips, and the decoder's conv layers figure out how to combine them.

Encoder block: Conv 3×3 → BatchNorm → ReLU, then Conv 3×3 → BatchNorm → ReLU, then MaxPool 2×2 (downsample to the next level).

Decoder block: Upsample 2× (transposed conv, or interpolation + conv), then Concatenate with the encoder skip features, then Conv 3×3 → BatchNorm → ReLU, then Conv 3×3 → BatchNorm → ReLU.

(The original U-Net used Conv → ReLU without BatchNorm, since BN was published the same year. Modern U-Nets always include BatchNorm or GroupNorm.)

The decoder needs to double spatial resolution at each step. Two main ways:

Transposed convolution (also called "deconv"). A learned upsampling — mathematically it's the gradient operation of a strided convolution. Each input pixel "spreads out" into a small patch of the output, with learnable weights deciding the spread pattern. Pro: learnable, adapts to data. Con: can produce checkerboard artifacts if the stride and kernel size are chosen badly.

Interpolation + convolution. Use a fixed upsampling (nearest-neighbor or bilinear) to double the size, then apply a regular conv to refine. Pro: no checkerboard artifacts, simpler. Con: slightly less expressive. Most modern segmentation models prefer this.

The decoder ends at the original input resolution with some feature channels (typically 64 at the first level). The final operation is a 1×1 convolution mapping those features to C output channels, one per class — output shape (H, W, C). A 1×1 conv just remixes channels at each pixel without combining neighbors, so it maps each pixel's feature vector to a per-pixel class-score vector (the per-pixel logit map). Apply softmax along the channel dimension to get per-pixel class probabilities:

where $z_{i,j,c}$ is the logit for class $c$ at pixel $(i,j)$, and the denominator sums over all classes $k$. To produce the final mask, take the argmax over channels at each pixel.

Training a U-Net is just like training a classifier, but per-pixel. The loss is per-pixel cross-entropy — for each pixel, the cross-entropy between the predicted class distribution and the true label:

where $(i, j)$ runs over pixels, $c$ over classes, $y_{i,j,c}$ is the one-hot true label, and $\hat{p}_{i,j,c}$ is the predicted probability. For binary segmentation this becomes binary cross-entropy.

The big practical wrinkle is class imbalance. In medical imaging especially, you often have a tiny object (a tumor a few hundred pixels wide) in a huge image. A lazy model that predicts "background" everywhere scores 99% pixel accuracy and is completely useless. Common fixes:

where $\hat{p}_i$ is the predicted probability and $y_i$ the true label at pixel $i$. Because this is a ratio of overlap to combined area, the size of the background doesn't drown it out — it intrinsically handles imbalance. Often combined: total loss = Cross-Entropy + Dice.

A few reasons it's still the default segmentation architecture a decade on: it works with limited data (originally trained on ~30 labeled cell images — sample-efficient thanks to strong inductive biases: locality, hierarchy, skip connections); it's architecturally simple (conv blocks, max pools, transposed convs, skips — easy to implement and modify); it produces sharp masks (skip connections give precise boundaries, critical for medical imaging); it's modality-agnostic (2D images, 3D volumes via 3D U-Net, medical scans, satellite imagery, microscopy); and it's the backbone of diffusion models — the same U-Net that segments cells is the architecture inside Stable Diffusion. Every modern image generator runs on a U-Net. Its longevity is striking: it's older than the transformer, and still everywhere.

U-Net gives you semantic segmentation — every pixel labeled by class. But how do you tell three dogs apart? For instance segmentation, the dominant architecture is Mask R-CNN (He, Gkioxari, Dollár, Girshick, 2017). It extends Faster R-CNN (a two-stage detector) with a mask-predicting branch.

The core stance: instance segmentation is detection plus segmentation. The high-level steps: (1) detect each instance and produce a bounding box around it; (2) for each detected box, produce a binary mask of that single object inside the box. This decomposition is elegant — the detector handles "different instances are different objects," and the mask head handles "which exact pixels belong to this instance." Each is a well-studied subproblem, and combining them yields instance segmentation.

So Mask R-CNN's output for each detected region has three branches: a class label (one of C classes or background), a bounding-box refinement (fine-tuned coordinates), and a binary mask (a small mask within the box, one channel per class). The first two come straight from Faster R-CNN; the third is the new addition — a small fully-convolutional network that takes the region's feature map and outputs an m × m binary mask (typically m = 28).

This is Mask R-CNN's most important technical contribution. A region proposal might land at fractional coordinates like (137.3, 248.7, 282.5, 451.1). To extract a fixed-size feature map for that region (say 7×7), you have to map those real-valued coordinates onto the discrete grid of the CNN's feature map.

RoI Pool (the old way, from Fast/Faster R-CNN): round the box coordinates to integer pixels, divide the rounded box into a 7×7 grid of sub-regions, and max-pool each. This works for classification — small misalignments don't matter when you only predict one label per region. But for mask prediction every pixel matters, and those rounding errors compound, leaving the final mask misaligned by a few pixels.

RoI Align (the Mask R-CNN way): don't round. Use bilinear interpolation to sample feature values at exact real-valued coordinates within the box. The 7×7 output grid has its corners at exactly the right floating-point positions, and each output cell pools values interpolated from neighboring feature-map cells. The math, for a sample point at floating-point $(x, y)$, looks at the 4 nearest integer feature positions and takes a distance-weighted sum:

where $f(i,j)$ are the feature values at the four surrounding integer positions and the $\max(0, 1 - |\cdot|)$ terms are the bilinear weights (closer positions count more). That's just standard bilinear interpolation — the insight is using it instead of rounding. The result is sub-pixel-accurate feature extraction, and mask quality jumps: the paper reported a ~10% improvement in mask average precision from this single change.

Inside each detected region, the mask head is a small fully-convolutional network: take the RoI-aligned feature map (14×14 or 7×7), apply a few conv layers, upsample with a transposed conv to 28×28, then a 1×1 conv to produce C output channels — one per class, each a sigmoid binary mask. Critically, the model outputs K binary masks per region, one for each possible class, and at inference you simply take the mask for the class the classification head predicted. This decouples classification from mask prediction: the mask head doesn't have to decide "is this a dog or a cat," it just draws the right pixels.

The total loss combines all three branches:

The mask loss is per-pixel binary cross-entropy, applied only to the channel for the ground-truth class:

where $m^2$ is the number of mask pixels (784 for a 28×28 mask), $y_{i,j}$ is the true 0/1 mask value, and $\hat{y}_{i,j}$ the predicted probability. Notice: sigmoid per pixel, not softmax. Each pixel independently answers "am I part of this object or not," with no competition between classes — which is exactly what allows the decoupling.

That decoupling is one of Mask R-CNN's most elegant ideas. In semantic segmentation (U-Net), you ask each pixel "which of the C classes are you?" — softmax forces classes to compete. In Mask R-CNN you ask each pixel "are you part of this object?" — binary, no competition. So the mask head's job is much simpler: it doesn't have to learn what a dog looks like versus a cat and distinguish them at the pixel level; it just learns to draw the boundary of whatever object is in this region, while the class head answers "what" independently. The result: cleaner gradient signal, better masks, lower data requirements.

In 2023, Meta released the Segment Anything Model (SAM), and segmentation got its foundation model. Before SAM, every segmentation task needed its own model trained on its own labeled dataset — one for tumors, one for roads, one for satellite imagery. After SAM, a single model could segment essentially anything in any image with a click, a box, or a rough mask — including objects it had never been explicitly trained on. It was trained on 11 million images and 1.1 billion masks (over 400× larger than the previous biggest segmentation dataset), and its architecture has three parts: a heavy image encoder, a lightweight prompt encoder, and a small mask decoder.

The single most important idea in SAM is the shift from task-specific to promptable segmentation. A traditional model is trained for a fixed output — a U-Net trained on medical scans outputs tumor masks, a model trained on COCO outputs masks of 80 categories. The set of possible outputs is locked at training time. SAM flips this: it's trained to take a prompt — a point click, a bounding box, or a rough mask — and produce the segmentation that the prompt indicates. Whatever you point at, SAM segments. Which means you don't need labeled data for your specific task (SAM works zero-shot), you can segment things the model never saw in training, and one model serves countless downstream tasks just by changing the prompt.

If that sounds familiar, it should: it's the exact same conceptual move that GPT made in language. Instead of training a separate model for translation, summarization, and classification, you train one model that responds to prompts and let the prompt encode the task. SAM is the segmentation version of that shift.

SAM has three components, designed around one constraint: interactive use must feel almost instant. When you click, the mask should appear in milliseconds.

The asymmetry is the entire point. Encoding the image is expensive (the ViT-H takes hundreds of milliseconds on a modern GPU), but decoding from a prompt is cheap. So you encode an image once, then click on it many times, getting a fresh mask in real time for each click without ever recomputing the image embedding.

The image encoder is SAM's eyes. It turns a 1024×1024 RGB image into a 64×64 grid of 256-dimensional feature vectors. It's a Vision Transformer — ViT-H in the largest variant — pretrained with Masked Autoencoding (MAE) before being adapted for segmentation. (We'll cover ViTs in full in the next chapter; here's just enough to follow SAM.)

Why a ViT and not a CNN? CNNs are excellent at local features through convolution, but they struggle with long-range dependencies because receptive fields grow only slowly with depth. Segmentation often needs global reasoning — picture segmenting a person partially hidden behind a tree. A CNN might segment the visible body parts separately, because the disconnected regions can't "talk" until very deep layers. A ViT's self-attention gives every patch immediate access to every other patch in a single layer — exactly what global reasoning needs.

The first step turns the 2D image into discrete tokens the transformer can process. SAM uses 16×16 non-overlapping patches. Operationally that's a single convolution with kernel size 16 and stride 16 — each output position corresponds to one 16×16 input patch. The shape transformation:

So we go from 1024×1024 raw pixels to a 64×64 grid of patch tokens, each a rich learned feature vector — a 256× reduction in spatial resolution, packed into much richer per-location features.

Patches alone have no order — the transformer can't tell where each came from. SAM adds learnable absolute positional embeddings: each of the 64×64 = 4096 patch positions gets its own learnable vector, added to the patch embedding. Unlike the sinusoidal encoding of the original transformer, these are learned — the network discovers whatever positional representation works best. (And unlike relative encodings, which we'll see in a moment, absolute embeddings tell each patch its own coordinate, not its relationship to others.)

Here SAM does something clever. A standard ViT does full self-attention at every layer — every patch attends to every other. For 4096 tokens (the flattened 64×64 grid), that's $4096^2 \approx 16.8$ million attention operations per head per layer, multiplied across many heads and layers. Expensive. So SAM uses mostly windowed attention with periodic global attention:

This is the same idea as Swin Transformer's hierarchical attention (which we'll meet later): cheap windowed layers capture local patterns, while rare, expensive global layers mix information globally. Far more efficient than full attention everywhere, with no real loss of global reasoning.

On top of the absolute positions, SAM adds relative positional embeddings inside each attention block. The framing is clean: absolute positions tell each patch "where I am," while relative positions tell pairs of patches "how I relate to you spatially." Both are useful — absolute lets the network reason about specific locations ("top-right corner"), relative lets it reason about relationships ("these two patches are vertically adjacent"). For segmentation, relative positions are arguably more useful, since what matters is which patches belong to the same object — a relationship, not an absolute spot.

Recall the standard attention computation:

SAM adds position-dependent biases to the scores before the softmax:

where $\Delta h_{ij}$ and $\Delta w_{ij}$ are the vertical and horizontal distances between query position $i$ and key position $j$, and $\text{rel\_h}, \text{rel\_w}$ are learned biases for those distances. The naive approach would learn a separate bias for every $(\Delta h, \Delta w)$ pair — for a 64×64 map that's $(2 \cdot 64 - 1)^2 = 127^2 \approx 16{,}000$ parameters per head. Expensive. SAM instead decomposes the bias into separate height and width components:

which needs only $2 \cdot (2 \cdot 64 - 1) = 254$ parameters per head — a 98% reduction in positional parameters. The assumption is that horizontal and vertical relationships can be encoded independently, which is reasonable for natural images. Intuitively, the model can learn things like "patches in the same row are highly related" (small horizontal bias at $\Delta w = 0$) and "patches close vertically are more related than distant ones" (vertical bias decaying with $|\Delta h|$); the two combine into a position-aware attention adjustment.