Machine Learning & Signals Learning
23 Image Classification
Notation Throughout this and the following computer-vision chapters, an input image \(\bX \in \mathbb {R}^{H\times W\times C_{\text {in}}}\) has spatial dimensions:
-
• \(H\) (height, in pixels) and
-
• \(W\) (width, in pixels), with
-
• \(C_{\text {in}}\in \{1,3\}\) channels (grayscale or RGB).
\(C\) denotes the number of target classes (excluding the background class \(0\) when present).
This chapter and the next two cover three computer-vision tasks: image classification (one class label for the whole image), image segmentation (a class label for every pixel) and object detection (a variable number of objects, each located by a bounding box and classified). All three assume a class list fixed at training time.
Image classification is the canonical computer-vision task. The model \(f_\theta (\bX )\) returns a \(C\)-length probabilistic vector on the simplex \(\Delta ^{C-1}\) (Sec. 21.3); the prediction is \(\widehat {y}=\argmax _c [f_\theta (\bX )]\).
Two sub-flavors are worth distinguishing:
-
• Single-label (multi-class) classification: exactly one of \(C\) classes per image. The output layer is a softmax (Sec. 21.3) of size \(C\) and the loss is categorical cross-entropy. This is the focus of the present chapter.
-
• Multi-label classification: any subset of \(C\) tags per image (e.g. scene attributes, chest-X-ray findings). The softmax is replaced by \(C\) independent sigmoids and the loss is the sum of binary cross-entropies, one per class. The architectural pattern is otherwise identical.
23.1 Performance metrics
Classification reuses the binary/multi-class machinery of Chapter 17: the confusion matrix and everything derived from it. The image-level specifics are top-\(n\) accuracy/error (Sec. 23.1.1) and the class-averaged metrics introduced below.
23.1.1 Top-\(n\) accuracy and error
A probabilistic multi-class classifier outputs a vector of class probabilities \(\hat {\bp }_i=(\hat {p}_{i,1},\ldots ,\hat {p}_{i,C})\). Ordinary accuracy uses only the top-1 prediction \(\arg \max _c \hat {p}_{i,c}\). With many classes, or with semantically similar classes (e.g. ImageNet has 1000 classes including several breeds of dog), the top-1 prediction is often a near-miss while the true class still appears among the most probable few. Allowing the classifier to return a short list of candidates and counting it correct whenever the true class lies in the list gives the top-\(n\) family of metrics.
Top-\(n\) accuracy: A prediction is counted as correct if the true class is among the \(n\) classes with the highest predicted probability. Let \(\mathcal {T}_n(\hat {\bp }_i)\) denote the set of indices of the \(n\) largest entries of \(\hat {\bp }_i\). Then
\(\seteqnumber{0}{23.}{0}\)\begin{equation} \text {Top-}n\text { accuracy} = \frac {1}{M}\sum _{i=1}^{M}\indFunc \!\left [y_i \in \mathcal {T}_n(\hat {\bp }_i)\right ], \end{equation}
where \(\indFunc [\cdot ]\) is the indicator function (Sec. B.2). Top-\(1\) accuracy coincides with ordinary accuracy. Top-\(n\) accuracy is non-decreasing in \(n\) and reaches \(100\%\) at \(n=C\). Top-\(5\) is the conventional companion to top-\(1\) on ImageNet-scale benchmarks.
Top-\(n\) error: The complement of top-\(n\) accuracy: the fraction of samples for which the true class is not among the \(n\) most probable predictions,
\(\seteqnumber{0}{23.}{1}\)\begin{equation} \text {Top-}n\text { error} = 1 - \text {Top-}n\text { accuracy} = \frac {1}{M}\sum _{i=1}^{M}\indFunc \!\left [y_i \notin \mathcal {T}_n(\hat {\bp }_i)\right ]. \end{equation}
ImageNet leaderboards (Sec. 22.2.3) conventionally report top-\(5\) error, because once accuracy reaches the high \(90\,\%\) range, differences are easier to read on the error scale (e.g. \(5.1\,\%\) vs. \(3.6\,\%\) rather than \(94.9\,\%\) vs. \(96.4\,\%\)).
-
Example 23.1: A 5-class image classifier produces the following probability vectors on four test images (true-class probability in bold):
\(i\) true \(\hat p_1\) \(\hat p_2\) \(\hat p_3\) \(\hat p_4\) \(\hat p_5\) top-1 top-3 1 2 0.10 0.55 0.20 0.10 0.05 yes yes 2 4 0.40 0.30 0.15 0.10 0.05 no no 3 3 0.05 0.10 0.30 0.45 0.10 no yes 4 1 0.50 0.20 0.15 0.10 0.05 yes yes For sample 3, the top-3 predicted classes are \(\{4,3,2\}\), which contains the true class 3, so the prediction counts as correct under top-\(3\). Aggregating:
\(\seteqnumber{0}{23.}{2}\)\begin{equation*} \text {Top-1 accuracy} = \frac {2}{4} = 50\%, \qquad \text {Top-3 accuracy} = \frac {3}{4} = 75\%. \end{equation*}
The gap reveals that the classifier often ranks the correct class highly even when its top-\(1\) pick is wrong, suggesting that the model has captured useful structure that a stricter accuracy measure obscures.
23.1.2 Class-averaged metrics
Macro-\(F_1\) and balanced accuracy (Sec. 17.6.2): per-class \(F_1\) or recall averaged uniformly across classes. Mandatory on long-tailed datasets where top-1 is dominated by the majority classes.
23.2 Practical recipe
Table 23.1 summarises a representative set of available models. Three families recur:
-
• Mobile/edge (MobileNetV3, EfficientNet-B0): \(\le 10\) M parameters, optimised for latency on CPU/mobile.
-
• General-purpose (ResNet-50, EfficientNet-B4, ConvNeXt-Tiny, ViT-B/16): \(20\)–\(90\) M parameters, the default starting point when no hard latency budget applies.
-
• High-capacity (ConvNeXt-Large, ViT-L/16 and bigger): hundreds of millions of parameters.
To quantify model size and computational complexity, we use parameter counts (associated with memory footprint) and multiply-accumulate operations (MACs, associated with latency and computational cost).
Multiply-accumulate operation (MAC): A multiply-accumulate operation (MAC) computes the product of two numbers and adds that product to an accumulator:
\(\seteqnumber{0}{23.}{2}\)\begin{equation} a \;\leftarrow \; a \;+\; (b \;\times \; c) . \end{equation}
In the context of deep learning, a MAC operation (consisting of one multiplication and one addition) represents the fundamental computation in dot products, matrix multiplications, and convolutions. A single MAC is approximately equivalent to two floating-point operations (FLOPs):
\(\seteqnumber{0}{23.}{3}\)\begin{equation} \text {FLOPs} \;\approx \; 2 \;\times \; \text {MACs} . \end{equation}
| Backbone | Params [M] | MACs [G] | Top-1 [%] |
| MobileNetV3-Large | 5.5 | 0.22 | 75.2 |
| EfficientNet-B0 | 5.3 | 0.39 | 77.1 |
| ResNet-18 | 11.7 | 1.8 | 69.8 |
| ResNet-50 | 25.6 | 4.1 | 76.1 |
| EfficientNet-B4 | 19 | 4.2 | 82.9 |
| ConvNeXt-Tiny | 28 | 4.5 | 82.1 |
| ViT-B/16 | 86 | 17.6 | 81.8 |
| ConvNeXt-Base | 89 | 15.4 | 83.8 |
| ConvNeXt-Large | 198 | 34.4 | 84.3 |
Doubling parameters buys a few accuracy points
An accuracy point here means one absolute percentage point on the Top-1 column of Table 23.1 (e.g. moving from \(76.1\%\) to \(77.1\%\) is a gain of one point). The gap between ResNet-50 and ConvNeXt-Large is roughly \(8\) points despite an \(8\times \) parameter increase. Pick the smallest model whose accuracy clears the application threshold.
23.3 Loss (*)
For single-label classification with one-hot target \(\by \in \{0,1\}^C\) (Sec. 14.1.3) and softmax probabilities \(\bp =g_\phi (\bz )\in \Delta ^{C-1}\) (the probability simplex; Sec. 21.3), the categorical cross-entropy is
\(\seteqnumber{0}{23.}{4}\)\begin{equation} \mathcal {L}_{\text {CE}} \;=\; -\sum _{c=1}^{C} y_c\,\log p_c \;=\; -\log p_{y} , \label {eq-dlarch-cls-ce} \end{equation}
where \(y\) denotes the index of the true class. Two refinements are routine:
23.3.1 Focal loss
The cross-entropy contribution of an example with predicted true-class probability \(p_y\) is \(-\log p_y\), which remains non-negligible even when \(p_y\) is already large. With many easy examples (the typical regime when one class dominates), their summed gradient swamps the contribution of the few hard ones. Focal loss multiplies CE by a modulating factor that vanishes as \(p_y\to 1\),
\(\seteqnumber{0}{23.}{5}\)\begin{equation} \mathcal {L}_{\text {focal}} \;=\; -(1-p_y)^\gamma \,\log p_y , \label {eq-dlarch-focal} \end{equation}
where \(\gamma \ge 0\) controls the steepness: \(\gamma =0\) recovers CE and larger \(\gamma \) flattens the loss in the well-classified region (Fig. 23.1). Concretely, at \(p_y=0.9\) (an easy example), CE is \(-\log 0.9\approx 0.105\) while focal with \(\gamma =2\) scales it by \((1-0.9)^2=0.01\) down to \(\approx 0.00105\), a \(100\times \) down-weighting. At \(p_y=0.1\) (a hard example), the factor \((1-0.1)^2=0.81\) leaves the loss almost untouched, so the gradient is redirected toward the hard examples.
23.4 Data Augmentation
A classifier whose training accuracy stays well above its validation accuracy is overfitting (Sec. 9.4.2). Changing the model is not always the fix: the training images may simply not show enough variety. A cat photographed from one side, in one light, teaches the network that one view. More real images are the best remedy (Sec. 9.8), but they are often expensive to collect and label. Images offer a cheap substitute: many edits that a photo editor applies (mirroring, rotating, cropping, brightening) produce a new image that a human still labels the same way.
Data augmentation: A label-preserving transformation \(\mathcal {A}\) maps an image to a new image of the same size whose class is unchanged,
\(\seteqnumber{0}{23.}{6}\)\begin{equation} (\bX , y) \;\to \; \bigl (\mathcal {A}(\bX ),\, y\bigr ). \end{equation}
Data augmentation draws a new random \(\mathcal {A}\) (which transformations, combined how, with which parameters) every time an image enters a mini-batch, so the network rarely sees exactly the same input twice.
23.4.1 Common transformations
Imagine the image printed on a sheet of paper. Geometric transformations move the sheet, and photometric ones change its colors, imitating lighting that is rarely constant outside a studio and cameras that differ in color response. Figure 23.2 applies each of them to one image.
-
• Flip: turn the sheet over, left to right (horizontal flip) or top to bottom (vertical flip).
-
• Rotation: turn the sheet about its center by a random angle.
-
• Zoom: bring the sheet closer, so the object fills more of the frame. Only zooming in is common: zooming out leaves empty borders that must be filled with invented pixels.
-
• Width and height shift: slide the sheet left/right or up/down. It is usually combined with a zoom, so that the pixels shifted into view are real ones and not padding.
-
• Homography: tilt the sheet, as if the camera looked at it from a different position and angle. The four transformations above are its special cases.
-
• Brightness: multiply all pixel values by a random factor, making the image darker or lighter.
-
• Channel shift: add a random offset, drawn separately for each color channel (R, G, B), which tints the image.
Pixels that a geometric transformation brings in from outside the original image are filled by padding (zeros, or a reflection of the border, as in Fig. 23.2). In the figure, brightness is scaled by \(1.5\) and the channel offsets are \((+50,-20,-50)\) on the \(0\) to \(255\) scale. Too large a factor or offset saturates the image to plain white or a single color.
The transformations are not alternatives: one image can be augmented by several of them at once. In practice \(\mathcal {A}\) is a chain, each transformation switched on at random with its own random parameters, e.g. a color change followed by a resize (zoom); the “Zoom + shift” panel of Fig. 23.2 is such a chain. Chaining multiplies the number of distinct variants obtainable from one image: with five transformations, each either on or off, there are already \(2^5=32\) combinations before any parameter is varied.
23.4.2 Choosing the transformations
A transformation is label-preserving only for a given task. The set of allowed transformations encodes what the classifier should be invariant to, and that is a property of the task, not of the images:
-
• Digits: rotating a “6” by \(180^\circ \) (a horizontal plus a vertical flip) produces a “9” (Fig. 23.3). Small rotations are safe; flips are not.
-
• Sign language: most signs are mirrored between left- and right-handed signers, so a horizontal flip is safe. Nobody signs upside down, so a vertical flip only adds images that never occur.
-
• Flower species: color is often the distinguishing feature, so channel shifts and hue changes destroy the label, while flips and rotations are safe.
-
• Text and road signs: a horizontal flip mirrors letters and turns a “left turn” sign into a “right turn” sign.
Augment after the split
Augmented copies of one image are strongly correlated; they are not new independent samples. Augmenting the whole dataset before splitting it puts near-copies of the same image in both the training and the test set, which is group leakage (Sec. 18.1.2) and inflates the test score. Augment the training split only, inside the training loop, and evaluate on unmodified images.
Spatial targets move with the image
A geometric transformation moves the objects in the image. When the target is itself spatial, a pixel mask or a set of bounding boxes, as in the next two chapters, the same geometric transformation must be applied to the target, while photometric transformations leave it unchanged.
Practical recipe
-
• Start from the invariances of the task and enable only the transformations that respect them, with moderate ranges (e.g. rotations of a few degrees, zoom up to \(\times 1.2\), brightness within \(\pm 20\%\)).
-
• Look at a random sample of augmented training images before training. If a human cannot tell the class, the network cannot either; shrink the ranges.
-
• Deep-learning frameworks implement all the transformations of this section as random layers or data-loader stages, applied on the fly, so no augmented image is ever stored.