Machine Learning & Signals Learning

18 Computer Vision Metrics

Notation Throughout this chapter, an input image \(\bX \in \mathbb {R}^{H\times W\times C_{\text {in}}}\) has spatial dimensions:

• • \(H\) (height, in pixels) and

• • \(W\) (width, in pixels), with

• • \(C_{\text {in}}\in \{1,3\}\) channels (grayscale or RGB).

\(C\) denotes the number of target classes (excluding the background class \(0\) when present).

The chapter covers four computer-vision tasks:

• • Section 18.2 (Image Classification): assign a single class label to the whole image.

• • Section 18.3 (Image Segmentation): assign a class label to every pixel.

• • Section 18.4 (Object Detection): locate and classify a variable number of objects with bounding boxes.

• • Section 18.5 (Siamese Networks): learn a numeric “fingerprint” for each image so that similar images get nearby fingerprints; useful when new classes appear at test time or only a handful of examples per class are available.

18.1 Benchmark datasets

Throughout this chapter, metric conventions are tied to the public benchmarks that introduced them. The three datasets below cover detection, instance/panoptic segmentation, and urban-scene semantic segmentation; ImageNet, used in classification, is covered separately in Sec. 17.2.2.

Pascal VOC The PASCAL Visual Object Classes challenge ran from 2005 to 2012 and was the first widely-adopted detection/segmentation benchmark. The standard VOC2007 / VOC2012 splits contain \(\sim 11{,}000\) images annotated with \(20\) object classes (person, vehicles, animals, indoor objects) plus a background class. VOC introduced two conventions that this chapter inherits: the lenient IoU threshold \(\tau =0.5\) for box matching (Sec. 18.4.1) and 11-point interpolated AP for the pre-2010 editions, replaced by all-point interpolation from 2010 onward. VOC numbers are not directly comparable to COCO numbers because the threshold and interpolation conventions differ.

COCO Common Objects in Context (Lin et al., 2014) is the de-facto modern benchmark for detection, instance segmentation and panoptic segmentation. It contains \(\sim 330{,}000\) images with \(80\) “thing” classes (animals, vehicles, people, household objects) for detection / instance segmentation and a further \(91\) “stuff” classes (sky, road, grass) used by the panoptic split. COCO introduced the stricter mAP@[.5:.95] convention (averaging over ten IoU thresholds, Eq. 18.24), the \(\text {AP}_{\text {small/medium/large}}\) size breakdown that exposes the small-object weakness of IoU, and the mask-AP variant that reuses the detection recipe with mask IoU. Most pretrained detection/segmentation checkpoints in this chapter are trained on COCO.

Cityscapes Cityscapes (Cordts et al., 2016) is the standard semantic-segmentation benchmark for urban driving: \(5{,}000\) images with fine pixel-level annotations and a further \(\sim 20{,}000\) with coarse annotations, drawn from 50 European cities, evaluated on \(19\) classes (road, sidewalk, car, pedestrian, sign, \(\ldots \)) plus a void / ignore label for ambiguous or out-of-vocabulary pixels. The void label is the convention to remember: it must be excluded from both numerator and denominator of every metric, otherwise mIoU is silently inflated.

18.2 Image Classification

Image classification is the canonical computer-vision task. The model \(f_\theta (\bX )\) returns a \(C\)-length probabilistic vector on the simplex \(\Delta ^{C-1}\) (Sec. 16.3); the prediction is \(\widehat {y}=\argmax _c [f_\theta (\bX )]\).

Two sub-flavors are worth distinguishing:

• • Single-label (multi-class) classification: exactly one of \(C\) classes per image. The output layer is a softmax (Sec. 16.3) of size \(C\) and the loss is categorical cross-entropy. This is the focus of the present section.

• • Multi-label classification: any subset of \(C\) tags per image (e.g. scene attributes, chest-X-ray findings). The softmax is replaced by \(C\) independent sigmoids and the loss is the sum of binary cross-entropies, one per class. The architectural pattern is otherwise identical.

18.2.1 Performance metrics

Classification reuses the binary/multi-class machinery of Chapter 12: the confusion matrix and everything derived from it. The image-level specifics are top-\(n\) accuracy/error (Sec. 18.2.1) and the class-averaged metrics introduced below.

Top-\(n\) accuracy and error

A probabilistic multi-class classifier outputs a vector of class probabilities \(\hat {\bp }_i=(\hat {p}_{i,1},\ldots ,\hat {p}_{i,C})\). Ordinary accuracy uses only the top-1 prediction \(\arg \max _c \hat {p}_{i,c}\). With many classes, or with semantically similar classes (e.g. ImageNet has 1000 classes including several breeds of dog), the top-1 prediction is often a near-miss while the true class still appears among the most probable few. Allowing the classifier to return a short list of candidates and counting it correct whenever the true class lies in the list gives the top-\(n\) family of metrics.

Top-\(n\) accuracy: A prediction is counted as correct if the true class is among the \(n\) classes with the highest predicted probability. Let \(\mathcal {T}_n(\hat {\bp }_i)\) denote the set of indices of the \(n\) largest entries of \(\hat {\bp }_i\). Then

\begin{equation} \text {Top-}n\text { accuracy} = \frac {1}{M}\sum _{i=1}^{M}\indFunc \!\left [y_i \in \mathcal {T}_n(\hat {\bp }_i)\right ], \end{equation}

where \(\indFunc [\cdot ]\) is the indicator function (Sec. B.2). Top-\(1\) accuracy coincides with ordinary accuracy. Top-\(n\) accuracy is non-decreasing in \(n\) and reaches \(100\%\) at \(n=C\). Top-\(5\) is the conventional companion to top-\(1\) on ImageNet-scale benchmarks.

Top-\(n\) error: The complement of top-\(n\) accuracy: the fraction of samples for which the true class is not among the \(n\) most probable predictions,

\begin{equation} \text {Top-}n\text { error} = 1 - \text {Top-}n\text { accuracy} = \frac {1}{M}\sum _{i=1}^{M}\indFunc \!\left [y_i \notin \mathcal {T}_n(\hat {\bp }_i)\right ]. \end{equation}

ImageNet leaderboards (Sec. 17.2.2) conventionally report top-\(5\) error, because once accuracy reaches the high \(90\,\%\) range, differences are easier to read on the error scale (e.g. \(5.1\,\%\) vs. \(3.6\,\%\) rather than \(94.9\,\%\) vs. \(96.4\,\%\)).

Class-averaged metrics

Macro-\(F_1\) and balanced accuracy (Sec. 12.6.2): per-class \(F_1\) or recall averaged uniformly across classes. Mandatory on long-tailed datasets where top-1 is dominated by the majority classes.

18.2.2 Practical recipe

Table 18.1 summarises a representative set of available models. Three families recur:

• • Mobile/edge (MobileNetV3, EfficientNet-B0): \(\le 10\) M parameters, optimised for latency on CPU/mobile.

• • General-purpose (ResNet-50, EfficientNet-B4, ConvNeXt-Tiny, ViT-B/16): \(20\)–\(90\) M parameters, the default starting point when no hard latency budget applies.

• • High-capacity (ConvNeXt-Large, ViT-L/16 and bigger): hundreds of millions of parameters.

To quantify model size and computational complexity, we use parameter counts (associated with memory footprint) and multiply-accumulate operations (MACs, associated with latency and computational cost).

Multiply-accumulate operation (MAC): A multiply-accumulate operation (MAC) computes the product of two numbers and adds that product to an accumulator:

\begin{equation} a \;\leftarrow \; a \;+\; (b \;\times \; c) . \end{equation}

In the context of deep learning, a MAC operation (consisting of one multiplication and one addition) represents the fundamental computation in dot products, matrix multiplications, and convolutions. A single MAC is approximately equivalent to two floating-point operations (FLOPs):

\begin{equation} \text {FLOPs} \;\approx \; 2 \;\times \; \text {MACs} . \end{equation}

18.2.3 Loss (*)

For single-label classification with one-hot target \(\by \in \{0,1\}^C\) (Sec. 9.1.3) and softmax probabilities \(\bp =g_\phi (\bz )\in \Delta ^{C-1}\) (the probability simplex; Sec. 16.3), the categorical cross-entropy is

\begin{equation} \mathcal {L}_{\text {CE}} \;=\; -\sum _{c=1}^{C} y_c\,\log p_c \;=\; -\log p_{y} , \label {eq-dlarch-cls-ce} \end{equation}

where \(y\) denotes the index of the true class. Two refinements are routine:

Focal loss

The cross-entropy contribution of an example with predicted true-class probability \(p_y\) is \(-\log p_y\), which remains non-negligible even when \(p_y\) is already large. With many easy examples (the typical regime when one class dominates), their summed gradient swamps the contribution of the few hard ones. Focal loss multiplies CE by a modulating factor that vanishes as \(p_y\to 1\),

\begin{equation} \mathcal {L}_{\text {focal}} \;=\; -(1-p_y)^\gamma \,\log p_y , \label {eq-dlarch-focal} \end{equation}

where \(\gamma \ge 0\) controls the steepness: \(\gamma =0\) recovers CE and larger \(\gamma \) flattens the loss in the well-classified region (Fig. 18.1). Concretely, at \(p_y=0.9\) (an easy example), CE is \(-\log 0.9\approx 0.105\) while focal with \(\gamma =2\) scales it by \((1-0.9)^2=0.01\) down to \(\approx 0.00105\), a \(100\times \) down-weighting. At \(p_y=0.1\) (a hard example), the factor \((1-0.1)^2=0.81\) leaves the loss almost untouched, so the gradient is redirected toward the hard examples.

18.3 Image Segmentation

Throughout this subsection:

• • \(\bM \in \{0,1,\ldots ,C\}^{H\times W}\) is the ground-truth label map,

• • \(\hat \bM = f_\theta (\bX )\) the predicted label map, and

• • \(|\cdot |\) denotes pixel count.

Background and foreground

One class index is reserved for the background (\(c=0\)): pixels that do not belong to any object of interest (sky, empty tissue, unlabelled regions). All other classes (\(c=1,\dots ,C\)) are foreground classes (tumour, road, vehicle, sheep), so \(C\) counts the foreground classes only and the total number of class indices is \(C+1\).

In a binary problem (\(C=1\)) there is a single foreground class and \(y_i\in \{0,1\}\) indicates whether pixel \(i\) belongs to it. With multiple foreground classes, each is treated separately when computing per-class losses and metrics: class \(c\) is its own foreground (\(y_i^{(c)}=1\)) and every other label, including the background (one-vs-rest), is its background (\(y_i^{(c)}=0\)). Foreground classes are typically the minority, and the pixel fraction each one occupies drives most of the design choices below (loss, metrics, sampling).

Objectives

Image segmentation comes in three flavors, distinguished by what each pixel of the output carries. It is useful to split classes into

• • stuff (uncountable regions: sky, road, grass) and

• • things (countable objects: sheep, car, person).

The corresponding objectives are:

• • Semantic segmentation: every pixel gets a single class label \(c\in \{0,\dots ,C\}\). Two sheep standing side by side are merged into one connected sheep region; the model cannot tell them apart.

• • Instance segmentation: every thing pixel gets a pair (class, instance id), so the two sheep receive different ids; stuff pixels (sky, road) are left unlabelled.

• • Panoptic segmentation: every pixel gets a pair (class, instance id). Things receive a unique id per object; each stuff class shares one id. It is a strict superset of the two above.

Counter-indications for segmentation:

• • If a single image-level label suffices, use a classifier that cheaper to label, train and run (Sec. 18.2).

• • If approximate locations of multiple objects are enough, use detection (Sec. 18.4).

18.3.1 Pixel Accuracy

Pixel-level confusion matrix A multi-class confusion matrix \(N\in \mathbb {N}^{(C+1)\times (C+1)}\) (the per-pixel analogue of the per-sample matrix of Sec. 12.6.1),

\begin{equation} N_{cc'} \;=\; \sum _{i,j} \indFunc \!\left [M_{ij}=c\right ]\,\indFunc \!\left [\hat M_{ij}=c'\right ] , \end{equation}

counts how many pixels of true class \(c\) are predicted as \(c'\). Every per-pixel metric below is a simple function of \(N\).

Pixel accuracy Pixel accuracy is the diagonal sum normalized by total pixels,

\begin{equation} \text {PA} \;=\; \frac {\sum _c N_{cc}}{\sum _{c,c'} N_{cc'}} , \label {eq-dlarch-pa} \end{equation}

treating each pixel as an equal-weight classification example.

Per-class pixel accuracy The per-class pixel accuracy is the fraction of class-\(c\) pixels recovered,

\begin{equation} \text {PA}_c \;=\; \frac {N_{cc}}{\sum _{c'} N_{cc'}} \;=\; \frac {\text {TP}_c}{\text {TP}_c+\text {FN}_c} , \end{equation}

which is the recall on class \(c\). The denominator \(\sum _{c'} N_{cc'}\) is the row-\(c\) sum of the confusion matrix, i.e. the number of pixels whose true label is \(c\), counted across every predicted class \(c'\).

Mean pixel accuracy

\begin{equation} \text {mPA} \;=\; \frac {1}{C+1}\sum _c \text {PA}_c \;=\; \frac {1}{C+1}\sum _c \frac {N_{cc}}{\sum _{c'} N_{cc'}} \end{equation}

averages \(\text {PA}_c\) equally over the \(C+1\) classes (\(+1\) stands for the background class). It is the macro-averaged per-class recall (Sec. 12.6.2), less biased by class frequency than PA.

18.3.2 Intersection over Union

Definition

For two regions \(A\) and \(B\) (pixel sets, or, in Sec. 18.4, bounding boxes) the Intersection over Union (IoU, also called the Jaccard index) is

\begin{equation} \text {IoU}(A,B) \;=\; \frac {|A\cap B|}{|A\cup B|} \;=\; \frac {|A\cap B|}{|A|+|B|-|A\cap B|} , \label {eq-dlarch-iou} \end{equation}

with \(\text {IoU}\in [0,1]\) (\(1\) for perfect overlap, \(0\) for disjoint regions).

IoU per class

For class \(c\), treating “class \(c\)” vs. “not class \(c\)” as a binary problem, apply the IoU definition (Eq. 18.11) with \(B=\{\,\text {pixels with } M_{ij}=c\,\}\) (the actual class-\(c\) region) and \(A=\{\,\text {pixels with } \hat M_{ij}=c\,\}\) (the predicted class-\(c\) region). The matrix entries read out the three set cardinalities directly:

• • \(|A\cap B| = N_{cc} = \text {TP}_c\) — pixels that are class \(c\) in both the truth and the prediction (diagonal entry);

• • \(|B| = \sum _{c'} N_{cc'} = \text {TP}_c + \text {FN}_c\) — row-\(c\) sum, all actual class-\(c\) pixels;

• • \(|A| = \sum _{c'} N_{c'c} = \text {TP}_c + \text {FP}_c\) — column-\(c\) sum, all predicted class-\(c\) pixels.

• • \(|A\cup B|=|A|+|B|-|A\cap B|\) - inclusion-exclusion formula.

Substituting into Eq. 18.11 gives

\begin{equation} \text {IoU}_c \;=\; \frac {N_{cc}}{\sum _{c'} N_{cc'} + \sum _{c'} N_{c'c} - N_{cc}} \;=\; \frac {\text {TP}_c}{\text {TP}_c + \text {FP}_c + \text {FN}_c} . \end{equation}

Mean IoU

The mean IoU averages over classes,

\begin{equation} \text {mIoU} \;=\; \frac {1}{C+1}\sum _{c=0}^{C} \text {IoU}_c , \label {eq-dlarch-miou} \end{equation}

and is the de-facto reporting metric. A frequency-weighted variant is

\begin{equation} \text {fwIoU}=\sum _c f_c\,\text {IoU}_c \end{equation}

with \(f_c\) the pixel frequency of class \(c\) (the fraction of all pixels whose true label is \(c\)).

Median IoU

The median IoU is a robust alternative,

\begin{equation} \text {medIoU} \;=\; \operatorname *{median}_{c=0,\dots ,C}\,\text {IoU}_c , \label {eq-dlarch-mediou} \end{equation}

useful on long-tailed benchmarks: a single rare class whose IoU collapses to near zero can drag mIoU down disproportionately, while medIoU is unaffected by such per-class outliers. It is sometimes reported alongside mIoU rather than instead of it.

18.3.3 Dice coefficient

The Dice coefficient (the \(F_1\) score in its binary form) is

\begin{equation} \text {Dice}(A,B) \;=\; \frac {2|A\cap B|}{|A|+|B|} \;=\; \frac {2\,\text {TP}}{2\,\text {TP}+\text {FP}+\text {FN}} . \label {eq-dlarch-dice} \end{equation}

Dice and IoU are monotone transforms of each other,

\begin{equation} \label {eq-dice-iou} \text {Dice} \;=\; \frac {2\,\text {IoU}}{1+\text {IoU}} , \qquad \text {IoU} \;=\; \frac {\text {Dice}}{2-\text {Dice}} , \end{equation}

so they rank methods identically; Dice gives a higher score for the same overlap (Fig. 18.6).

18.3.4 Hausdorff distance

IoU and Dice score area overlap and are dominated by the interior of large objects, so a prediction can score well even while its boundary wanders far from the truth in a few places. The Hausdorff distance measures exactly that: the largest gap between the two boundaries (Fig. 18.7). It is the distance-based complement to the overlap metrics above and the standard companion to Dice in medical imaging, where contour carries clinical information (cf. the boundary metrics in Sec. 18.3.3). This distance exists in multiple variants.

18.3.5 Panoptic quality

Panoptic quality (PQ) grades a result at the level of whole segments rather than individual pixels. It is built in two stages, as follows.

Step 1: match segments Pair a predicted segment \(p\) with a ground-truth segment \(g\) only when their mask overlap clears \(\text {IoU}(p,g)>0.5\).

The threshold of 0.5 is what keeps the matching clean: no two predictions can both exceed \(0.5\) IoU with the same ground truth, so each ground truth matches at most one prediction and the pairing is unique. The matching sorts every segment into three counts (segment-level, not pixels):

• • \(\text {TP}\) – matched pairs \((p,g)\);

• • \(\text {FP}\) – predicted segments with no matching ground truth (spurious detections);

• • \(\text {FN}\) – ground-truth segments with no matching prediction (misses).

Step 2: grade with two questions A good panoptic result must answer two independent questions, and PQ measures each with its own factor.

• • Did the model find the right segments? This is recognition quality (RQ), the \(F_1\) score over segments, treating each segment as one classification example:

  \(\seteqnumber{0}{}{17}\)

  \begin{equation*} \text {RQ} \;=\; \frac {|\text {TP}|}{|\text {TP}|+\tfrac 12|\text {FP}|+\tfrac 12|\text {FN}|} . \end{equation*}

  The \(\tfrac 12\) weights split each unmatched segment evenly between FP and FN, the standard \(F_1\) form.

• • When it does find a segment, how tightly does the mask fit? This is segmentation quality (SQ), the mean IoU over the matched pairs only (unmatched segments do not enter):

  \(\seteqnumber{0}{}{17}\)

  \begin{equation*} \text {SQ} \;=\; \frac {\sum _{(p,g)\in \text {TP}}\text {IoU}(p,g)}{|\text {TP}|} . \end{equation*}

Step 3: combine Panoptic quality is the product of the two factors,

\begin{equation} \text {PQ} \;=\; \underbrace {\frac {\sum _{(p,g)\in \text {TP}}\text {IoU}(p,g)}{|\text {TP}|}}_{\text {SQ}} \;\times \; \underbrace {\frac {|\text {TP}|}{|\text {TP}|+\tfrac 12|\text {FP}|+\tfrac 12|\text {FN}|}}_{\text {RQ}} . \label {eq-dlarch-pq} \end{equation}

Because it is a product, a high PQ demands both: detect the right segments (high RQ) and outline them tightly (high SQ). A model that detects everything with sloppy masks, or produces a few perfect masks while missing most objects, scores poorly.

18.3.6 Instance segmentation

Evaluation reuses the detection recipe of Sec. 18.4.1 below, with mask IoU (Eq. 18.11 applied to per-instance binary masks) replacing box IoU. The standard mask-AP metric is the COCO (Sec. 18.1) mAP averaged over IoU thresholds \(\{0.5,0.55,\ldots ,0.95\}\) (mAP@[.5:.95], Eq. 18.24) applied to masks instead of boxes.

Inference hyperparameters Two deployment-time knobs sit between the network output and the reported metric:

• • Score threshold \(\tau _{\text {score}}\): the confidence below which predicted segments are dropped before any TP/FP/FN counting. For computing mAP, keep \(\tau _{\text {score}}\) low (e.g. \(0.05\)) so the precision–recall curve is sampled across the full operating range (Sec. 18.4.1). For visualisation or deployment, raise it (e.g. \(0.5\)–\(0.7\)) so only confident segments are shown.

• • Mask threshold \(\tau _{\text {mask}}\): the per-pixel probability cutoff that binarises the soft mask into \(\{0,1\}\) at inference, \(\hat M_{ij} = \indFunc [p_{ij} > \tau _{\text {mask}}]\) (the same \(0.5\) choice that defines the hard-mask Dice loss in Sec. 18.3.3 below). Lower it to grow masks (boosts recall, may merge instances); raise it to shrink them (boosts precision, may fragment).

18.3.7 Dice Loss

Definition

The baseline is per-pixel cross-entropy summed over the \(H\times W\) grid, identical to a classification CE applied independently at every pixel. On imbalanced foreground (small lesions on large background, thin roads on large satellite tiles), CE alone underweights the minority class and the model collapses toward the background.

The Dice loss addresses this directly. In the binary case (background and foreground, \(C=1\)),

\begin{equation} \mathcal {L}_{\text {Dice}} \;=\; 1 \;-\; \frac {2\sum _i p_i\,y_i}{\sum _i p_i + \sum _i y_i} , \end{equation}

where \(p_i\in [0,1]\) is the predicted foreground probability and \(y_i\in \{0,1\}\) the ground-truth indicator at pixel \(i\) (with \(i\) ranging over all \(HW\) pixels). Each of the three sums has a direct interpretation:

• • \(\sum _i y_i = |B|\) counts the foreground pixels in the ground-truth mask,

• • \(\sum _i p_i\) is the soft analogue of \(|A|\): the expected number of foreground pixels predicted by the model, where each pixel contributes its probability rather than a \(\{0,1\}\) vote, and

• • \(\sum _i p_i\,y_i\) measures their soft overlap (the predicted probability summed over the true foreground pixels).

Two variants share this formula and differ only in the input.

• • The soft Dice loss is \(\mathcal {L}_{\text {Dice}}\) as written above: the inputs are the continuous probabilities \(p_i\in [0,1]\), so the loss is differentiable in the model parameters and can drive training.

• • The hard-mask Dice loss replaces \(p_i\) with the thresholded mask \(\hat p_i = \indFunc [p_i > 0.5]\in \{0,1\}\) before the same formula is evaluated; the three sums then become the cardinalities \(|B|\), \(|A|\) and \(|A\cap B|\), recovering the set-based Dice coefficient of Eq. 18.16 (Sec. 18.3.3). Training therefore uses the soft form (gradient everywhere); evaluation reports the hard form (matches the set-based coefficient that humans interpret).

Motivation (*)

Because Dice definition in 18.17 makes it a strictly increasing function of IoU, minimising \(\mathcal {L}_{\text {Dice}}\) is equivalent to maximising IoU: any drop in the loss is a guaranteed gain in IoU on the same prediction, so the loss and the evaluation metric rank predictions in the same order.

The second advantage is robustness to class imbalance. Let \(f = \tfrac {1}{HW}\sum _i y_i\) denote the foreground fraction (the share of pixels of foreground). For per-pixel cross-entropy with a constant-probability predictor \(p_i\equiv p\) (every pixel’s predicted foreground probability \(p_i\) is the same scalar \(p\), so the loss is a function of one number instead of \(HW\)), the optimum is \(p^*=f\): as \(f\to 0\), CE rewards the model for predicting all-background and the foreground gradient vanishes. Dice loss has no such collapse, because its ratio is scale-invariant in the foreground set: doubling \(\sum _i y_i\) and \(\sum _i p_i y_i\) together leaves \(\mathcal {L}_{\text {Dice}}\) unchanged, so a tiny foreground is not automatically a small loss. Figure 18.9 shows both behaviours side by side at three foreground fractions:

• • \(f=0.5\) – balanced (a roughly 50/50 split, e.g. generic binary classification). The CE optimum sits at \(p^{*}=0.5\) (middle dotted line in panel a); CE and soft Dice agree on direction, both descending together as \(p\to 1\). The hard-mask Dice loss in panel b drops from \(1\) (all-background, \(p\le 0.5\)) to \((1-f)/(1+f)=1/3\) (all-foreground, \(p>0.5\)) – a clear improvement, though Dice\(=2/3\) is still far from a good prediction.

• • \(f=0.1\) – mildly imbalanced (typical foreground objects in natural images). The CE optimum has slid left to \(p^{*}=0.1\): a model that admits only \(10\%\) foreground confidence is already CE-optimal, while soft Dice keeps decreasing all the way to \(p=1\) and its gradient still asks for more foreground confidence. Hard-mask Dice plateaus at \((1-f)/(1+f)\approx 0.82\) on \(p>0.5\).

• • \(f=0.01\) – severely imbalanced (small lesions in medical scans, thin roads in satellite tiles). The CE optimum collapses against the left axis: predicting all-background is essentially optimal and the foreground gradient is negligible. Soft Dice is still monotone-decreasing in \(p\), but the hard-mask Dice loss only drops to \(\approx 0.98\) even after thresholding everything to foreground – which is why training uses the soft form rather than thresholding inside the loss.

Multi-class case. With \(C\) foreground classes and a softmax probability vector output \(\bp _i\in [0,1]^{C+1}\) at every pixel and the ground truth is the one-hot vector \(\by _i\). The Dice loss is computed per class against its one-vs-rest indicator and then averaged,

\begin{equation} \mathcal {L}_{\text {Dice}}^{\text {multi}} \;=\; 1 \;-\; \frac {1}{C}\sum _{c=1}^{C}\frac {2\sum _i p_i^{(c)} y_i^{(c)}}{\sum _i p_i^{(c)} + \sum _i y_i^{(c)} + \varepsilon } , \end{equation}

where \(p_i^{(c)}\) is the softmax probability of class \(c\) at pixel \(i\), \(y_i^{(c)}\in \{0,1\}\) is the one-hot ground-truth indicator, and \(\varepsilon \) is a small constant (\(\sim 1\)) that prevents division by zero on images where a class is absent (\(\sum _i y_i^{(c)}=0\)). The background class \(c=0\) is excluded from the sum.

Practical recipe. The standard combined loss is \(\mathcal {L} = \mathcal {L}_{\text {CE}} + \mathcal {L}_{\text {Dice}}\) (binary or multi-class as appropriate), which keeps the per-pixel calibration of CE and the imbalance robustness of Dice.

Summary

18.4 Object Detection

Detection sits at the end of a natural progression of image-level tasks, each adding one more degree of output complexity:

• • Image classification (Sec. 18.2)

  • – Dataset provides one label per image, \(1\,\text {image}\to 1\,\text {class}\).

  • – Model output is a probability simplex \(\Delta ^{C}\) over the \(C\) classes.

• • Classification with localization (currently not discussed in-depth in this chapter):

  • – Dataset provides one label and one bounding box for a single dominant object, \(1\,\text {image}\to 1\,\text {class}+1\,\text {box}\).

  • – Model output is a probability simplex \(\Delta ^{C}\) and four box coordinates. The output shape is still fixed, so a single regression head bolted onto a classifier suffices.

• • Object detection (this section)

  • – Dataset provides a variable-length list of (class, box) annotations per image, \(1\,\text {image}\to N\,(\text {class}+\text {box})\) with \(N\) unknown in advance.

  • – Model output is a set of (class, box, score) tuples, where the score \(s\in [0,1]\) is a per-(box, class) confidence used to rank the prediction against others on the precision-recall curve (Sec. 18.4.1).

  • – Makes the output variable-length (an image may contain zero, one or many objects).

  • – Couples a regression (box coordinates) and a classification (class + objectness), evaluated jointly.

Bounding-box principle

A bounding box is the smallest axis-aligned rectangle that fully encloses an object in the image plane. Two equivalent four-number parameterizations are used:

• • Corner form \(b=(x_1,y_1,x_2,y_2)\): top-left and bottom-right pixel coordinates, with \(x_1<x_2\) and \(y_1<y_2\). Convenient for IoU and matching (Sec. 18.4.1).

• • Center-size form \(b=(x_c,y_c,w,h)\): box center and width/height, with each coordinate normalized to \([0,1]\) by dividing \(x_c,w\) by the image width and \(y_c,h\) by the image height. Normalization makes the four numbers resolution-independent: the same box parameters describe the object before and after resizing a \(1920\times 1080\) image to the \(640\times 640\) training crop, so one regression target stays valid across the multi-scale training augmentations and the variable camera resolutions seen at inference. The parameterization also decouples location (\(x_c,y_c\)) from scale (\(w,h\)).

The box carries no orientation (no rotation) and no shape information beyond the rectangle; tighter outlines require instance segmentation (Sec. 18.3). A box is considered correct, against a ground-truth box of the same class, when their overlap exceeds an IoU threshold (Sec. 18.4.1).

When to use it Detection is the right tool when you must locate and count multiple objects of possibly overlapping categories in a scene, and a bounding box is precise enough (otherwise use instance segmentation). Counter-indications:

• • A single dominant object per image \(\to \) classifier (Sec. 18.2).

• • Pixel-precise contours required \(\to \) segmentation (Sec. 18.3).

• • Open-set identification (who is this person?) \(\to \) Siamese / embedding approach (Sec. 18.5).

18.4.1 Performance metrics

An object detector (the model being evaluated) outputs a set of scored boxes per image, and the metric has to collapse that set into a single comparable number. It does so through a four-stage pipeline, each stage feeding the next:

• 1. Match and label: pair each predicted box with a ground-truth box by bounding-box IoU, label it TP or FP, and count the missed ground truths as FN.

• 2. Sweep: vary the confidence cut-off to trace a precision–recall curve.

• 3. Integrate: collapse that curve into one average precision (AP) at a fixed IoU threshold.

• 4. Average: average AP over classes (mAP) and over IoU thresholds (mAP@[.5:.95]).

The subsections below walk these stages in order; non-maximum suppression is an inference-time aside (it reuses the same IoU test) inserted between Stages 1 and 2.

Box matching and TP/FP/FN labeling

Matching is done one class at a time and repeated identically for each class (Stage 4 then averages across classes).

The detector’s class-\(c\) predictions are a set of boxes \(\{\hat b_i\}\) with confidence scores \(\{s_i\}\), which must be scored against the class-\(c\) ground-truth boxes \(\{b_j\}\). Matching turns these two box sets into TP, FP, and FN counts.

When does a box count as correct? Overlap is measured by the bounding-box IoU (Eq. 18.11, Fig. 18.11): a prediction can match a ground-truth box only if their IoU clears a threshold \(\tau \in [0,1]\) (Pascal VOC uses \(\tau =0.5\); a stricter \(\tau \) demands tighter boxes).

The matching rule Several predictions may overlap the same object, so to keep the count unambiguous they are matched greedily in order of confidence:

• 1. Sort the predictions by descending confidence scores \(\{s_i\}\) and walk down the list.

• 2. Each \(\hat b_i\) claims the still-unclaimed ground-truth box \(b_j\) with which its \(\text {IoU}(\hat b_i, b_j)\ge \tau \) is highest.

This sorts every box into one of three outcomes:

• • True positive (TP): the claimed box clears \(\text {IoU}(\hat b_i, b_j)\ge \tau \), and that ground truth leaves the pool.

• • False positive (FP): the prediction wins no valid claim, either because it overlaps no ground truth above \(\tau \) (a misplaced box) or because the ground truth it overlaps was already taken by a higher-scoring prediction (a duplicate detection).

• • False negative (FN): a ground-truth box left unclaimed once every prediction has been processed.

Why there is no TN This reduces single-image detection to TP, FP, FN counts per class, almost the binary classification setting of Chapter 12. The missing fourth cell is TN: a true negative would be a correctly emitted “no object here” decision, but the negative space is every possible box in the image plane (infinitely many), so TN is undefined and not counted. This is why detection metrics use precision and recall (which need only TP, FP, FN) and never accuracy or specificity (which need TN).

Non-Maximum Suppression (NMS)

Table 18.2 contrasts box matching and NMS.

At inference time the detector must return a clean box list rather than a labeled one, so it deletes the duplicates outright. Non-maximum suppression (NMS, used by one-stage detectors such as YOLO, Sec. 18.4.2) does this with the same greedy descending-confidence sweep as the matching rule, except each prediction is compared against the boxes already kept rather than against the ground truth:

• 1. Sort the class-\(c\) predictions by descending confidence scores \(\{s_i\}\), as in Stage 1.

• 2. Pop the highest-scoring box \(\hat b^{*}\) and keep it; remove every remaining prediction \(\hat b_i\) with \(\text {IoU}(\hat b_i,\hat b^{*}) > \tau _{\text {NMS}}\). Predictions with \(\text {IoU}(\hat b_i,\hat b^{*}) \le \tau _{\text {NMS}}\) overlap \(\hat b^{*}\) too little to be duplicates and stay in the pool as candidates for separate objects.

• 3. Repeat on what is left.

The NMS threshold \(\tau _{\text {NMS}}\) (commonly \(0.5\)–\(0.7\)) trades the two failure modes: too low merges genuinely adjacent objects (two pedestrians become one box), too high leaves duplicates. Figure 18.13 illustrates the effect on a two-sheep scene.

Precision–recall curve

With every prediction labeled TP or FP and the FN count fixed (Stage 1), the only remaining freedom is how many predictions to admit. Each predicted box has a confidence \(s\). Sweeping a cut-off \(s_0\) from \(1\) down to \(0\) and admitting only predictions with \(s\ge s_0\) produces a sequence of (precision, recall) pairs:

\begin{equation} \text {Pr}(s_0) = \frac {\text {TP}(s_0)}{\text {TP}(s_0)+\text {FP}(s_0)} , \qquad \text {Re}(s_0) = \frac {\text {TP}(s_0)}{\text {TP}(s_0)+\text {FN}} . \end{equation}

Plotting Pr vs. Re gives the precision–recall curve at IoU threshold \(\tau \).

Average precision

A whole curve is awkward to compare across models, so Stage 3 reduces it to a single scalar. Average Precision (AP) at IoU threshold \(\tau \) is the area under the (interpolated) precision–recall curve,

\begin{equation} \text {AP}_\tau \;=\; \int _0^1 \text {Pr}_{\text {int}}(r)\,\mathrm {d}r , \qquad \text {Pr}_{\text {int}}(r) \;=\; \max _{r'\ge r}\text {Pr}(r') , \label {eq-dlarch-ap} \end{equation}

where the interpolation \(\text {Pr}_{\text {int}}\) replaces precision at recall \(r\) by the highest precision attained at any recall \(r'\ge r\), removing the zig-zag of the raw curve.

Two integration conventions coexist:

• • 11-point interpolation (Pascal VOC, Sec. 18.1, pre-2010): average \(\text {Pr}_{\text {int}}\) over \(r\in \{0,0.1,\ldots ,1.0\}\).

• • All-point interpolation (VOC post-2010, COCO, Sec. 18.1): integrate over all distinct recall values.

The two yield slightly different numbers and are not directly comparable.

mAP and COCO-style mAP

Stage 3 produces one AP per class at one IoU threshold; the final stage averages those into the single number reported on a benchmark. Mean Average Precision (mAP) at IoU threshold \(\tau \) is the arithmetic mean of per-class \(\text {AP}_{\tau ,c}\) over the \(C\) classes:

\begin{equation} \text {mAP}_\tau \;=\; \frac {1}{C}\sum _{c=1}^C \text {AP}_{\tau ,c} , \label {eq-dlarch-map} \end{equation}

where \(\text {AP}_{\tau ,c}\) is computed exactly as in Eq. 18.22 but restricting predictions and ground-truth boxes to class \(c\). Averaging over classes treats every class as equally important regardless of its frequency in the dataset: a model that nails the dominant class but misses every rare class will still score poorly. (To compute one \(\text {AP}_{\tau ,c}\), sweep the per-class predictions sorted by score, accumulate per-class TP/FP/FN, and integrate the resulting per-class PR curve.) COCO introduced a stricter metric that averages over IoU thresholds as well,

\begin{equation} \text {mAP}_{[.5:.95]} \;=\; \frac {1}{10}\sum _{\tau \in \{0.50,0.55,\ldots ,0.95\}}\text {mAP}_\tau , \label {eq-dlarch-map-coco} \end{equation}

which rewards tight localization: a model that overlaps ground-truth by \(0.5\) scores well at \(\tau =0.5\) but collapses at \(\tau =0.95\).

18.4.2 Architecture

All detection networks share the same skeleton: a CNN backbone produces a dense feature map, i.e. a low-resolution \(H'\times W'\times C\) tensor whose \((h,w)\) cell carries a learned \(C\)-dimensional descriptor of the input-image patch that maps to that location; one or more prediction heads turn each feature-map cell into candidate boxes with an objectness (the class-agnostic probability that the box contains any object versus background) and a class distribution; NMS (Sec. 18.4.1) prunes duplicates. The per-(box, class) score \(s\) that the evaluator sees is built from these two: in one-stage detectors typically \(s = \text {objectness}\times \text {class probability}\), in two-stage detectors objectness is the score the first stage uses to keep proposals and \(s\) is the class probability output by the second-stage classifier. Families differ in how many passes it takes to get from the feature map to the final box list.

The two practically relevant families are then:

• • Two-stage detectors (R-CNN family: Faster R-CNN, Mask R-CNN): stage 1 emits region proposals, stage 2 classifies and refines each.

• • One-stage detectors (YOLO, SSD, RetinaNet): a single forward pass predicts boxes and classes densely over the feature grid.

A third family, transformer-based detectors (DETR and successors), predicts the whole set of boxes in one forward pass without anchors and without NMS, trading longer training for a simpler inference pipeline; it is increasingly competitive but not yet the practitioner default.

R-CNN family The modern reference is Faster R-CNN: the backbone produces a feature map once per image; a small sub-network slides over that feature map and emits a fixed pool of anchor-based region proposals with objectness scores; the top-scoring proposals are passed to a second-stage head that outputs a class label and a refined box per proposal. Mask R-CNN adds a parallel head that produces a binary mask per proposal, making the same network usable for instance segmentation (Sec. 18.3). Practical profile: highest accuracy on COCO-style benchmarks, slowest per image, the default when latency is not the bottleneck or when masks or keypoints are also required.

YOLO YOLO (“You Only Look Once”) treats detection as a single dense regression problem. A CNN backbone reduces the input to a coarse feature grid (e.g. \(13\times 13\)); each grid cell predicts a small fixed number of candidate boxes together with an objectness score and a class distribution, all in one forward pass. Low-confidence boxes are discarded and overlapping survivors are pruned per class by NMS (Sec. 18.4.1). Modern variants predict at multiple feature-map resolutions so that small, medium and large objects share the same backbone, and recent open-source releases (Ultralytics YOLOv8/v11) ship anchor-free heads and one-line fine-tuning from COCO weights. Practical profile: real-time on commodity GPUs and edge devices, the practitioner default for deployment, accuracy now close to two-stage on modern variants.

18.4.3 Notes

18.5 Siamese Networks

A Siamese network is not a new layer type but a training configuration: the same backbone (CNN for images, RNN or Transformer for sequences, fully-connected for tabular data) is applied to each input in a pair (or triplet), and the loss is defined on distances in the resulting embedding space rather than on a classification output.

18.5.1 Definitions

Input space

The input space \(\mathcal {X}\) is the set of all admissible raw inputs to the model. Each \(\bx \in \mathcal {X}\) is a single example (e.g. an image, a sentence, a signal segment, or a tabular row). The dimensionality of \(\mathcal {X}\) is typically large and structured: a \(224\times 224\) RGB image lives in \(\mathbb {R}^{224\times 224\times 3}\), a \(T\)-sample audio clip in \(\mathbb {R}^{T}\).

Learned Features

In modern representation learning, learned features refer to any internal data representations generated by a machine learning model. Instead of relying on manual, hand-crafted feature engineering, models optimize parameters \(\theta \) to automatically extract useful characteristics from a raw input space \(\mathcal {X}\). These representations can take a variety of structural forms, including high-dimensional spatial grids in convolutional neural networks, categorical logit scores, or attention weight matrices in transformers.

Embedding

An embedding is a specific, highly structured subset of learned features. It is defined as a learned mapping \(f_\theta :\mathcal {X}\to \mathbb {R}^d\) from an input space \(\mathcal {X}\) (such as images, text, audio, or signals) to a low-dimensional Euclidean space \(\mathbb {R}^d\), called the embedding space or latent space. The continuous vector \(f_\theta (\bx )\in \mathbb {R}^d\) is the embedding of input \(\bx \). An effective embedding satisfies two core properties:

• • Geometric: Euclidean (or cosine) distance in \(\mathbb {R}^d\) reflects task-relevant semantic similarity.

• • Compact: \(d \ll \dim (\mathcal {X})\), so downstream tasks (classification, retrieval, clustering) operate on a small fixed-size vector rather than on the raw input.

One-shot and few-shot learning

A classification setting in which the classes encountered at inference time are not those seen during training, and only a very small number of labelled examples per class is available for the new classes:

• • One-shot: exactly one labelled example per class.

• • Few-shot (\(K\)-shot, \(N\)-way): \(K\) labelled examples (typically \(K\le 5\)) for each of \(N\) classes; a query is classified among those \(N\).

The small set of labelled examples for the new classes is called the support set; the unlabelled inputs to be classified form the query set.

A standard solution is to learn a similarity-preserving embedding \(f_\theta \) on a separate large dataset and, at inference, classify each query by some classifier.

18.5.2 When to use it

Choose a Siamese embedding when any of the following hold:

• • The class set is open: new identities or categories appear at test time without retraining (as opposed to a closed set, fixed at training time).

• • Only a handful of labelled examples per class is available (one-shot or few-shot, Sec. 18.5.1).

• • The deployment task is verification or retrieval rather than closed-set classification.

Typical settings:

• • Face verification and recognition (FaceNet, DeepFace): one model handles open-set identification, where the set of identities at test time is not known at training time.

• • Signature and handwriting verification.

• • One-shot and few-shot learning (Sec. 18.5.1): a handful of labelled examples per class are enough at inference, since classification reduces to nearest-neighbor search in embedding space.

• • Signal similarity, e.g. matching ECG segments or audio clips by a learned distance rather than hand-crafted features.

Counter-indications. If the class set is closed and labelled data is plentiful per class, a plain softmax classifier (Sec. 18.2) is simpler, faster to train, and usually as accurate. If pixel-precise localization or per-pixel labels are needed, use segmentation (Sec. 18.3) or detection (Sec. 18.4) instead; an embedding alone does not localize.

18.5.3 Architecture

Two identical sub-networks share the same parameters \(\theta \) (tied weights, Fig. 18.16). Each branch maps an input \(\bx \) to an embedding \(f_\theta (\bx )\in \mathbb {R}^d\), and a pair is compared by the embeddings distance \(d(\bx _1,\bx _2)\) (Sec. 10.2). The most common embedding distances are \(L_2\),

\begin{equation} d(\bx _1,\bx _2) \;=\; \bigl \|f_\theta (\bx _1)-f_\theta (\bx _2)\bigr \|_2 . \end{equation}

and cosine distance.

Properties of the architecture:

• • Weight sharing guarantees the comparison is symmetric: \(d(\bx _1,\bx _2)=d(\bx _2,\bx _1)\).

• • Halves the number of trainable parameters relative to two independent encoders.

• • Both inputs are guaranteed to be processed by the same feature extractor, so distances are meaningful.

• • The embedding is normalised (typically to the unit sphere, \(\|f_\theta (\bx )\|_2=1\)) to bound the loss and decouple it from feature magnitude.

18.5.4 Loss

The Siamese architecture (Fig. 18.16) produces embeddings, not class probabilities, so cross-entropy is not directly applicable. The supervision is structural: rather than a class label per input, the training set provides pairs \((\bx _1,\bx _2,y)\) with a binary similarity label \(y\in \{0,1\}\), or triplets \((\bx _a,\bx _p,\bx _n)\) where \(\bx _p\) is known to share the anchor’s class and \(\bx _n\) does not. The loss is therefore a function of the embedding distance \(d(\bx _1,\bx _2) = \|f_\theta (\bx _1) - f_\theta (\bx _2)\|_2\) and of the similarity labels.

Two design choices recur in all Siamese losses:

• • Distance pulls similar pairs together and pushes dissimilar pairs apart. Without an asymmetry between the two, the trivial solution \(f_\theta \equiv \text {const}\) minimises every distance and the network learns nothing.

• • A margin \(m>0\) caps how hard a dissimilar pair is pushed. Once the pair is far enough apart, the loss contributes no gradient; training effort is spent only on pairs that still violate the geometric goal.

The next two paragraphs introduce the two standard instantiations: contrastive loss on pairs, and triplet loss on (anchor, positive, negative) triplets.

Contrastive loss Pair-based supervision with a binary similarity label \(y\in \{0,1\}\) (\(y=1\) similar, \(y=0\) dissimilar). The idea is to enforce one geometric goal per pair:

\begin{equation} d(\bx _1,\bx _2)\;\to \; 0 \text { when } y=1, \qquad d(\bx _1,\bx _2)\;\ge \; m \text { when } y=0, \end{equation}

i.e. similar pairs should collapse to the same embedding, while dissimilar pairs should be at least \(m\) apart. The two cases are turned into a single loss by penalising the squared distance for similar pairs and the hinged margin violation \(\max (0,\,m-d)\) for dissimilar ones, then selecting between them with the label \(y\):

\begin{equation} \mathcal {L}_{\text {contrastive}} \;=\; y\,d^2 \;+\; (1-y)\,\max (0,\,m-d)^2, \end{equation}

where \(m>0\) is a margin. Similar pairs are pulled together (the first term shrinks \(d\)), while dissimilar pairs are pushed apart only until their distance reaches \(m\); beyond that, no gradient flows. The margin prevents the trivial collapse \(f_\theta \equiv \text {const}\).

Triplet loss Triplet-based supervision is usually preferred in practice. Each example is a triplet \((\bx _a,\bx _p,\bx _n)\) with:

• • anchor \(\bx _a\),

• • positive \(\bx _p\) (same class as the anchor) and

• • negative \(\bx _n\) (different class).

The idea is to enforce one geometric inequality per triplet:

\begin{equation} d(\bx _a,\bx _p) + m \;\le \; d(\bx _a,\bx _n) , \end{equation}

i.e. the anchor must be at least \(m\) closer to the positive than to the negative. Rearranging gives a non-negative violation \(v = d(\bx _a,\bx _p)^2 - d(\bx _a,\bx _n)^2 + m\), positive exactly when the constraint is broken. Hinging it at \(0\) (so satisfied triplets contribute no loss) and using squared distances for a smoother gradient yields

\begin{equation} \mathcal {L}_{\text {triplet}} \;=\; \max \!\Bigl (0,\; \|f(\bx _a)-f(\bx _p)\|_2^2 \;-\; \|f(\bx _a)-f(\bx _n)\|_2^2 \;+\; m\Bigr ) \end{equation}

forces the anchor-to-positive distance to be smaller than the anchor-to-negative distance by at least the margin \(m\). Compared with contrastive loss, the triplet form encodes a relative ranking: it does not require an absolute distance threshold for similar pairs, only that positives are closer than negatives.

Supervised contrastive (SupCon) loss Triplet loss uses one positive and one negative per anchor, so each gradient step depends critically on the quality of the mined triplet. Supervised contrastive (SupCon) loss generalises this: every same-class example in the mini-batch is treated as a positive, and every other example as a negative, so each anchor contributes a softmax over many pairs at once. SupCon pairs naturally with the \(P\times K\) sampling described in Batch construction below and removes the need for explicit triplet mining.

18.5.5 Training

Hard negative mining The number of possible triplets grows cubically with the dataset, but most are uninformative.

• • Easy triplets already satisfy the margin and contribute zero gradient; including them slows training without improving the model.

• • Hard triplets violate the margin by a large amount and dominate the loss; using only the hardest ones can destabilise training (especially with label noise).

• • Semi-hard triplets: negatives that are farther than the positive but still within the margin. This is the standard compromise.

• • Batch-hard mining: for each anchor in a mini-batch, form a triplet from the hardest positive (the same-class example that is currently farthest in embedding space, i.e. the one the model gets most wrong) and the hardest negative (the different-class example that is currently closest). The two extrema are taken within the batch only, so the cost reduces to one \(O(B^2)\) pairwise-distance matrix on the \(B\) batch embeddings instead of a nearest-neighbour search across the whole training set, and the gradient at every step is concentrated on the most informative triplet per anchor. It pairs naturally with \(P\times K\) batch construction, which guarantees that each batch contains enough same-class and different-class candidates.

Batch construction Triplet and contrastive losses are only as good as the pairs the mini-batch actually contains: a batch of mostly singletons cannot supply positives, and a batch of mostly one class cannot supply informative negatives. The standard recipe is \(P\times K\) sampling: each mini-batch is built from \(P\) classes drawn uniformly, with \(K\) examples per class.

• • Guarantees \(K-1\) in-batch positives and \((P-1)\,K\) in-batch negatives for every anchor, so batch-hard mining always has candidates.

• • Typical values: \(P\in [8,32]\), \(K\in [4,8]\), giving batches of \(64\)–\(256\) samples; small \(K\) wastes positives, large \(K\) wastes negatives.

• • Classes are usually sampled uniformly rather than proportionally to frequency, so head classes do not dominate the loss.

• • Combined naturally with batch-hard mining: for each anchor, pick the farthest same-class example and the closest different-class example within the batch.

18.5.6 Classifier

The trained Siamese network produces embeddings, not class probabilities; classification is built on top of them in one of four ways.

• • Verification (open-set, “are these two inputs from the same class?”; e.g. face unlock, signature check): map a pair to a binary accept/reject decision \(\widehat {y} = \bOne [\,d(\bx _1,\bx _2) \le \tau \,]\in \{0,1\}\), accepting the pair as same-class when their embedding distance falls below a threshold \(\tau \) tuned on a held-out validation set of labelled same/different pairs. Because the decision depends only on the distance, identities unseen during training can be verified at test time without retraining.

• • Identification / retrieval (open-set, “which class is this?”): single input \(\to \) class label. Precompute a gallery of embeddings \(\{f_\theta (\bx _i)\}\) with known labels \(y_i\in \{1,\dots ,C\}\) and classify a query \(\bx \) by \(k\)-NN in embedding space, \(\widehat {y}(\bx )\in \{1,\dots ,C\}\). Adding a new class only requires inserting its embeddings into the gallery.

• • Embeddings as features for an ML classifier (closed-set): freeze \(f_\theta \) and treat \(\bz =f_\theta (\bx )\in \mathbb {R}^d\) as a fixed-length feature vector. Fit a standard supervised classifier, e.g. multinomial logistic regression or a (kernel) SVM, on the labelled set \(\{(\bz _i,y_i)\}\) for the \(C\) target classes. Because the embedding is low-dimensional and (typically) \(L_2\)-normalised, even a linear classifier is usually sufficient and trains in seconds.

• • Closed-set classification (Siamese as pre-training): once \(f_\theta \) has learned a good representation, freeze it and fit a small classifier head \(g_\phi :\mathbb {R}^d \to \Delta ^{C-1}\) (typically a single linear layer with softmax) on labelled data for the \(C\) target classes. This decouples representation learning from classification and is the standard recipe for using a Siamese / contrastive backbone in supervised downstream tasks.

18.5.7 Performance metrics

For the verification deployment mode, a binary same/different decision is driven by a threshold \(\tau \) on the embedding distance \(d(\bx _1,\bx _2)\). Writing

\begin{equation} \text {TAR}(\tau ) = \Pr (d(\bx _1,\bx _2)\le \tau \,\big |\, y=1) , \qquad \text {FAR}(\tau ) = \Pr (d(\bx _1,\bx _2)\le \tau \,\big |\, y=0) , \end{equation}

the true accept rate (TAR \(=\) TPR) and false accept rate (FAR \(=\) FPR) trace a ROC curve as \(\tau \) varies, summarised by AUC (Sec. 12.5 of Chapter 12). Two operating-point summaries dominate biometric reporting:

• • Equal Error Rate (EER): the point where \(\text {FAR}=\text {FRR}\) (false reject rate \(=1-\text {TAR}\)), the intersection of the ROC with the anti-diagonal \(\text {TPR}=1-\text {FPR}\). Single-number summary.

• • TAR@FAR\(=10^{-k}\): true-accept rate at a fixed, very small FAR (typically \(10^{-3}\), \(10^{-4}\) or \(10^{-6}\)). The standard reporting on face-verification benchmarks (LFW, MegaFace, IJB-C) because deployments target a low false-accept budget.

Retrieval and frozen-embedding deployment modes are evaluated by Rank-\(k\) / CMC and Recall@\(k\) for retrieval, and by a linear probe or \(k\)-NN accuracy for downstream classification on frozen embeddings.

18.5.8 Summary

18.6 Comparison of CV tasks

Table 18.5 summarizes the four computer-vision tasks covered in this chapter: image classification (Sec. 18.2), image segmentation (Sec. 18.3), object detection (Sec. 18.4), and Siamese embeddings (Sec. 18.5). The right column captures the practical question each task is designed to answer.

Table 18.6 consolidates the evaluation metrics introduced in this chapter, grouped by the task that produces them. Classification primitives (accuracy, \(F_1\), AUC) are deferred to Chapter 12 and reused by several modes here.

18.7 Image-Specific Sanity Checks (*)

The sanity checks in this section (beyond these in Ch. 13) are specific to classifiers operating on image data (raw pixels or spatial feature maps). They test whether the model exploits spatial structure—edges, textures, shapes—or relies on superficial statistics that happen to correlate with the labels.

18.7.1 Visual Inspection of Worst-Performing Examples

Worst-case visual inspection: Rank the held-out examples by a per-example performance score (loss, or \(1-p_y\) for classification, \(1-\text {IoU}\) for segmentation, \(1-\text {IoU}\) between best-matched boxes for detection), select the worst \(k\) (typically \(k=20\) to \(50\)), and display them in a grid annotated with the true label, the predicted label, and the model’s confidence.

Aggregate metrics (Sec. 18.2.1 for classification, Sec. 18.3.1 for segmentation, Sec. 18.4.1 for detection) summarize performance into a single number and hide which images the model gets wrong. Manually inspecting a few dozen worst cases regularly surfaces failure modes that no single-number metric can reveal, and it is among the simplest diagnostics available: it requires no retraining and no additional labels.

Patterns to look for in the worst-case grid:

• • Label noise: the ground-truth annotation is wrong or ambiguous, and the model’s prediction is in fact reasonable. Common on crowd-sourced or hastily relabeled datasets.

• • Out-of-distribution inputs: corrupt files, extreme rotation or cropping, drastically different lighting or resolution from the training set.

• • Confounding artifacts: timestamps, watermarks, scanner markers, ruler overlays, or hospital identifiers that correlate with the label in training but not at deployment.

• • Systematic class confusion: one class is consistently mistaken for another (e.g. visually similar dog breeds, two adjacent organ classes), pointing to a missing distinguishing feature or insufficient training examples for the confused pair.

18.7.2 Pixel Shuffle Test

Pixel shuffle test: Given a dataset of \(N\) images, randomly permute all pixel locations within each image independently, producing a shuffled dataset \(\bX _{\text {shuf}}\). Re-extract features from \(\bX _{\text {shuf}}\) and repeat the cross-validation procedure.

Shuffling destroys all spatial structure (edges, textures, object shapes) while preserving the per-image pixel histogram (mean intensity, variance, and higher-order marginal statistics remain identical). A classifier that genuinely relies on spatial patterns will see its accuracy drop to chance level on \(\bX _{\text {shuf}}\). Conversely, if the accuracy remains high after shuffling, the classifier exploits per-pixel statistics rather than spatial structure, and the original cross-validation result is unreliable.

18.7.3 Black-Patch Test

Black-patch test: Occlude a randomly positioned square region of each image with a black (zero or other constant-valued) patch of side length \(s\) pixels. Re-extract features and repeat cross-validation. Repeat for progressively larger patch sizes (e.g., \(s \in \{16, 32, 56, 112\}\)) to obtain an accuracy-versus-occlusion curve.

The black-patch test probes whether the classifier uses distributed spatial information or relies on a localized region. Three representative outcomes:

• • Gradual, monotonic decline in accuracy as \(s\) increases—the classifier uses information spread across the image, which is the expected behavior for a well-trained model.

• • Sharp drop at a specific patch size—the classifier depends on a localized region. This may indicate a genuine region of interest, but could also signal reliance on a confounding artifact (e.g., a timestamp, label overlay, or acquisition marker embedded in a fixed image location).

• • Stable accuracy across all patch sizes—the classifier is largely insensitive to spatial content, raising concern that it exploits non-image-based confounders (e.g., differences in file encoding or image dimensions between classes).

Further Reading

Convolutional Neural Networks (Course 4 of the Deep Learning Specialization), especially C4W3 (object detection) and C4W4 (Siamese Network)

• • Intersection Over Union (IoU): From Theory to Practice

• • Intersection over Union (IoU) in Object Detection & Segmentation

Examples and Demos:

• • Segment Anything and Segment Anything Model by Meta AI Demos (accessed 25-May-2026)

• • Segment Anything by Hugging Face (accessed 25-May-2026)

• • WPI Demo

• • Web AI Demo of the library that allows you to run modern deep learning models directly in your web browser (accessed 25-May-2026)

• • Conversational image segmentation with Gemini 2.5 by Google (accessed 25-May-2026)