Machine Learning & Signals Learning

24 Image Segmentation

The image notation of Ch. 23 carries over: an input image \(\bX \in \mathbb {R}^{H\times W\times C_{\text {in}}}\), and \(C\) foreground classes.

Throughout this chapter:

• • \(\bM \in \{0,1,\ldots ,C\}^{H\times W}\) is the ground-truth label map,

• • \(\hat \bM = f_\theta (\bX )\) the predicted label map, and

• • \(|\cdot |\) denotes pixel count.

Data augmentation (Sec. 23.4) carries over with one change: every geometric transformation applied to \(\bX \) is applied identically to the label map \(\bM \), so that each pixel keeps its label; photometric transformations leave \(\bM \) unchanged.

24.1 Background and foreground

One class index is reserved for the background (\(c=0\)): pixels that do not belong to any object of interest (sky, empty tissue, unlabelled regions). All other classes (\(c=1,\dots ,C\)) are foreground classes (tumour, road, vehicle, sheep), so \(C\) counts the foreground classes only and the total number of class indices is \(C+1\).

In a binary problem (\(C=1\)) there is a single foreground class and \(y_i\in \{0,1\}\) indicates whether pixel \(i\) belongs to it. With multiple foreground classes, each is treated separately when computing per-class losses and metrics: class \(c\) is its own foreground (\(y_i^{(c)}=1\)) and every other label, including the background (one-vs-rest), is its background (\(y_i^{(c)}=0\)). Foreground classes are typically the minority, and the pixel fraction each one occupies drives most of the design choices below (loss, metrics, sampling).

24.2 Objectives

Image segmentation comes in three flavors, distinguished by what each pixel of the output carries. It is useful to split classes into

• • stuff (uncountable regions: sky, road, grass) and

• • things (countable objects: sheep, car, person).

The corresponding objectives are:

• • Semantic segmentation: every pixel gets a single class label \(c\in \{0,\dots ,C\}\). Two sheep standing side by side are merged into one connected sheep region; the model cannot tell them apart.

• • Instance segmentation: every thing pixel gets a pair (class, instance id), so the two sheep receive different ids; stuff pixels (sky, road) are left unlabelled.

• • Panoptic segmentation: every pixel gets a pair (class, instance id). Things receive a unique id per object; each stuff class shares one id. It is a strict superset of the two above.

Counter-indications for segmentation:

• • If a single image-level label suffices, use a classifier that cheaper to label, train and run (Ch. 23).

• • If approximate locations of multiple objects are enough, use detection (Ch. 25).

24.3 Benchmark datasets

Throughout this chapter and the next, metric conventions are tied to the public benchmarks that introduced them. The three datasets below cover detection, instance/panoptic segmentation, and urban-scene semantic segmentation; ImageNet, used in classification, is covered separately in Sec. 22.2.3.

Pascal VOC The PASCAL Visual Object Classes challenge ran from 2005 to 2012 and was the first widely-adopted detection/segmentation benchmark. The standard VOC2007 / VOC2012 splits contain \(\sim 11{,}000\) images annotated with \(20\) object classes (person, vehicles, animals, indoor objects) plus a background class. VOC introduced two conventions that this chapter inherits: the lenient IoU threshold \(\tau =0.5\) for box matching (Sec. 25.2) and 11-point interpolated AP for the pre-2010 editions, replaced by all-point interpolation from 2010 onward. VOC numbers are not directly comparable to COCO numbers because the threshold and interpolation conventions differ.

COCO Common Objects in Context (Lin et al., 2014) is the de-facto modern benchmark for detection, instance segmentation and panoptic segmentation. It contains \(\sim 330{,}000\) images with \(80\) “thing” classes (animals, vehicles, people, household objects) for detection / instance segmentation and a further \(91\) “stuff” classes (sky, road, grass) used by the panoptic split. COCO introduced the stricter mAP@[.5:.95] convention (averaging over ten IoU thresholds, Eq. 25.4), the \(\text {AP}_{\text {small/medium/large}}\) size breakdown that exposes the small-object weakness of IoU, and the mask-AP variant that reuses the detection recipe with mask IoU. Most pretrained detection/segmentation checkpoints in this chapter are trained on COCO.

Cityscapes Cityscapes (Cordts et al., 2016) is the standard semantic-segmentation benchmark for urban driving: \(5{,}000\) images with fine pixel-level annotations and a further \(\sim 20{,}000\) with coarse annotations, drawn from 50 European cities, evaluated on \(19\) classes (road, sidewalk, car, pedestrian, sign, \(\ldots \)) plus a void / ignore label for ambiguous or out-of-vocabulary pixels. The void label is the convention to remember: it must be excluded from both numerator and denominator of every metric, otherwise mIoU is silently inflated.

24.4 Pixel Accuracy

Pixel-level confusion matrix A multi-class confusion matrix \(N\in \mathbb {N}^{(C+1)\times (C+1)}\) (the per-pixel analogue of the per-sample matrix of Sec. 17.6.1),

\begin{equation} N_{cc'} \;=\; \sum _{i,j} \indFunc \!\left [M_{ij}=c\right ]\,\indFunc \!\left [\hat M_{ij}=c'\right ] , \end{equation}

counts how many pixels of true class \(c\) are predicted as \(c'\). Every per-pixel metric below is a simple function of \(N\).

Pixel accuracy Pixel accuracy is the diagonal sum normalized by total pixels,

\begin{equation} \text {PA} \;=\; \frac {\sum _c N_{cc}}{\sum _{c,c'} N_{cc'}} , \label {eq-dlarch-pa} \end{equation}

treating each pixel as an equal-weight classification example.

Per-class pixel accuracy The per-class pixel accuracy is the fraction of class-\(c\) pixels recovered,

\begin{equation} \text {PA}_c \;=\; \frac {N_{cc}}{\sum _{c'} N_{cc'}} \;=\; \frac {\text {TP}_c}{\text {TP}_c+\text {FN}_c} , \end{equation}

which is the recall on class \(c\). The denominator \(\sum _{c'} N_{cc'}\) is the row-\(c\) sum of the confusion matrix, i.e. the number of pixels whose true label is \(c\), counted across every predicted class \(c'\).

Mean pixel accuracy

\begin{equation} \text {mPA} \;=\; \frac {1}{C+1}\sum _c \text {PA}_c \;=\; \frac {1}{C+1}\sum _c \frac {N_{cc}}{\sum _{c'} N_{cc'}} \end{equation}

averages \(\text {PA}_c\) equally over the \(C+1\) classes (\(+1\) stands for the background class). It is the macro-averaged per-class recall (Sec. 17.6.2), less biased by class frequency than PA.

24.5 Intersection over Union

24.5.1 Definition

For two regions \(A\) and \(B\) (pixel sets, or, in Ch. 25, bounding boxes) the Intersection over Union (IoU, also called the Jaccard index) is

\begin{equation} \text {IoU}(A,B) \;=\; \frac {|A\cap B|}{|A\cup B|} \;=\; \frac {|A\cap B|}{|A|+|B|-|A\cap B|} , \label {eq-dlarch-iou} \end{equation}

with \(\text {IoU}\in [0,1]\) (\(1\) for perfect overlap, \(0\) for disjoint regions).

24.5.2 IoU per class

For class \(c\), treating “class \(c\)” vs. “not class \(c\)” as a binary problem, apply the IoU definition (Eq. 24.5) with \(B=\{\,\text {pixels with } M_{ij}=c\,\}\) (the actual class-\(c\) region) and \(A=\{\,\text {pixels with } \hat M_{ij}=c\,\}\) (the predicted class-\(c\) region). The matrix entries read out the three set cardinalities directly:

• • \(|A\cap B| = N_{cc} = \text {TP}_c\) — pixels that are class \(c\) in both the truth and the prediction (diagonal entry);

• • \(|B| = \sum _{c'} N_{cc'} = \text {TP}_c + \text {FN}_c\) — row-\(c\) sum, all actual class-\(c\) pixels;

• • \(|A| = \sum _{c'} N_{c'c} = \text {TP}_c + \text {FP}_c\) — column-\(c\) sum, all predicted class-\(c\) pixels.

• • \(|A\cup B|=|A|+|B|-|A\cap B|\) - inclusion-exclusion formula.

Substituting into Eq. 24.5 gives

\begin{equation} \text {IoU}_c \;=\; \frac {N_{cc}}{\sum _{c'} N_{cc'} + \sum _{c'} N_{c'c} - N_{cc}} \;=\; \frac {\text {TP}_c}{\text {TP}_c + \text {FP}_c + \text {FN}_c} . \end{equation}

24.5.3 Mean IoU

The mean IoU averages over classes,

\begin{equation} \text {mIoU} \;=\; \frac {1}{C+1}\sum _{c=0}^{C} \text {IoU}_c , \label {eq-dlarch-miou} \end{equation}

and is the de-facto reporting metric. A frequency-weighted variant is

\begin{equation} \text {fwIoU}=\sum _c f_c\,\text {IoU}_c \end{equation}

with \(f_c\) the pixel frequency of class \(c\) (the fraction of all pixels whose true label is \(c\)).

24.5.4 Median IoU

The median IoU is a robust alternative,

\begin{equation} \text {medIoU} \;=\; \operatorname *{median}_{c=0,\dots ,C}\,\text {IoU}_c , \label {eq-dlarch-mediou} \end{equation}

useful on long-tailed benchmarks: a single rare class whose IoU collapses to near zero can drag mIoU down disproportionately, while medIoU is unaffected by such per-class outliers. It is sometimes reported alongside mIoU rather than instead of it.

24.6 Dice coefficient

The Dice coefficient (the \(F_1\) score in its binary form) is

\begin{equation} \text {Dice}(A,B) \;=\; \frac {2|A\cap B|}{|A|+|B|} \;=\; \frac {2\,\text {TP}}{2\,\text {TP}+\text {FP}+\text {FN}} . \label {eq-dlarch-dice} \end{equation}

Dice and IoU are monotone transforms of each other,

\begin{equation} \label {eq-dice-iou} \text {Dice} \;=\; \frac {2\,\text {IoU}}{1+\text {IoU}} , \qquad \text {IoU} \;=\; \frac {\text {Dice}}{2-\text {Dice}} , \end{equation}

so they rank methods identically; Dice gives a higher score for the same overlap (Fig. 24.5).

24.7 Hausdorff distance

IoU and Dice score area overlap and are dominated by the interior of large objects, so a prediction can score well even while its boundary wanders far from the truth in a few places. The Hausdorff distance measures exactly that: the largest gap between the two boundaries (Fig. 24.6). It is the distance-based complement to the overlap metrics above and the standard companion to Dice in medical imaging, where contour carries clinical information (cf. the boundary metrics in Sec. 24.6). This distance exists in multiple variants.

24.8 Panoptic quality

Panoptic quality (PQ) grades a result at the level of whole segments rather than individual pixels. It is built in two stages, as follows.

Step 1: match segments Pair a predicted segment \(p\) with a ground-truth segment \(g\) only when their mask overlap clears \(\text {IoU}(p,g)>0.5\).

The threshold of 0.5 is what keeps the matching clean: no two predictions can both exceed \(0.5\) IoU with the same ground truth, so each ground truth matches at most one prediction and the pairing is unique. The matching sorts every segment into three counts (segment-level, not pixels):

• • \(\text {TP}\) – matched pairs \((p,g)\);

• • \(\text {FP}\) – predicted segments with no matching ground truth (spurious detections);

• • \(\text {FN}\) – ground-truth segments with no matching prediction (misses).

Step 2: grade with two questions A good panoptic result must answer two independent questions, and PQ measures each with its own factor.

• • Did the model find the right segments? This is recognition quality (RQ), the \(F_1\) score over segments, treating each segment as one classification example:

  \(\seteqnumber{0}{24.}{11}\)

  \begin{equation*} \text {RQ} \;=\; \frac {|\text {TP}|}{|\text {TP}|+\tfrac 12|\text {FP}|+\tfrac 12|\text {FN}|} . \end{equation*}

  The \(\tfrac 12\) weights split each unmatched segment evenly between FP and FN, the standard \(F_1\) form.

• • When it does find a segment, how tightly does the mask fit? This is segmentation quality (SQ), the mean IoU over the matched pairs only (unmatched segments do not enter):

  \(\seteqnumber{0}{24.}{11}\)

  \begin{equation*} \text {SQ} \;=\; \frac {\sum _{(p,g)\in \text {TP}}\text {IoU}(p,g)}{|\text {TP}|} . \end{equation*}

Step 3: combine Panoptic quality is the product of the two factors,

\begin{equation} \text {PQ} \;=\; \underbrace {\frac {\sum _{(p,g)\in \text {TP}}\text {IoU}(p,g)}{|\text {TP}|}}_{\text {SQ}} \;\times \; \underbrace {\frac {|\text {TP}|}{|\text {TP}|+\tfrac 12|\text {FP}|+\tfrac 12|\text {FN}|}}_{\text {RQ}} . \label {eq-dlarch-pq} \end{equation}

Because it is a product, a high PQ demands both: detect the right segments (high RQ) and outline them tightly (high SQ). A model that detects everything with sloppy masks, or produces a few perfect masks while missing most objects, scores poorly.

24.9 Instance segmentation

Evaluation reuses the detection recipe of Sec. 25.2 below, with mask IoU (Eq. 24.5 applied to per-instance binary masks) replacing box IoU. The standard mask-AP metric is the COCO (Sec. 24.3) mAP averaged over IoU thresholds \(\{0.5,0.55,\ldots ,0.95\}\) (mAP@[.5:.95], Eq. 25.4) applied to masks instead of boxes.

Inference hyperparameters Two deployment-time knobs sit between the network output and the reported metric:

• • Score threshold \(\tau _{\text {score}}\): the confidence below which predicted segments are dropped before any TP/FP/FN counting. For computing mAP, keep \(\tau _{\text {score}}\) low (e.g. \(0.05\)) so the precision–recall curve is sampled across the full operating range (Sec. 25.2). For visualisation or deployment, raise it (e.g. \(0.5\)–\(0.7\)) so only confident segments are shown.

• • Mask threshold \(\tau _{\text {mask}}\): the per-pixel probability cutoff that binarises the soft mask into \(\{0,1\}\) at inference, \(\hat M_{ij} = \indFunc [p_{ij} > \tau _{\text {mask}}]\) (the same \(0.5\) choice that defines the hard-mask Dice loss in Sec. 24.6 below). Lower it to grow masks (boosts recall, may merge instances); raise it to shrink them (boosts precision, may fragment).

24.10 Dice Loss

24.10.1 Definition

The baseline is per-pixel cross-entropy summed over the \(H\times W\) grid, identical to a classification CE applied independently at every pixel. On imbalanced foreground (small lesions on large background, thin roads on large satellite tiles), CE alone underweights the minority class and the model collapses toward the background.

The Dice loss addresses this directly. In the binary case (background and foreground, \(C=1\)),

\begin{equation} \mathcal {L}_{\text {Dice}} \;=\; 1 \;-\; \frac {2\sum _i p_i\,y_i}{\sum _i p_i + \sum _i y_i} , \end{equation}

where \(p_i\in [0,1]\) is the predicted foreground probability and \(y_i\in \{0,1\}\) the ground-truth indicator at pixel \(i\) (with \(i\) ranging over all \(HW\) pixels). Each of the three sums has a direct interpretation:

• • \(\sum _i y_i = |B|\) counts the foreground pixels in the ground-truth mask,

• • \(\sum _i p_i\) is the soft analogue of \(|A|\): the expected number of foreground pixels predicted by the model, where each pixel contributes its probability rather than a \(\{0,1\}\) vote, and

• • \(\sum _i p_i\,y_i\) measures their soft overlap (the predicted probability summed over the true foreground pixels).

Two variants share this formula and differ only in the input.

• • The soft Dice loss is \(\mathcal {L}_{\text {Dice}}\) as written above: the inputs are the continuous probabilities \(p_i\in [0,1]\), so the loss is differentiable in the model parameters and can drive training.

• • The hard-mask Dice loss replaces \(p_i\) with the thresholded mask \(\hat p_i = \indFunc [p_i > 0.5]\in \{0,1\}\) before the same formula is evaluated; the three sums then become the cardinalities \(|B|\), \(|A|\) and \(|A\cap B|\), recovering the set-based Dice coefficient of Eq. 24.10 (Sec. 24.6). Training therefore uses the soft form (gradient everywhere); evaluation reports the hard form (matches the set-based coefficient that humans interpret).

24.10.2 Motivation (*)

Because Dice definition in 24.11 makes it a strictly increasing function of IoU, minimising \(\mathcal {L}_{\text {Dice}}\) is equivalent to maximising IoU: any drop in the loss is a guaranteed gain in IoU on the same prediction, so the loss and the evaluation metric rank predictions in the same order.

The second advantage is robustness to class imbalance. Let \(f = \tfrac {1}{HW}\sum _i y_i\) denote the foreground fraction (the share of pixels of foreground). For per-pixel cross-entropy with a constant-probability predictor \(p_i\equiv p\) (every pixel’s predicted foreground probability \(p_i\) is the same scalar \(p\), so the loss is a function of one number instead of \(HW\)), the optimum is \(p^*=f\): as \(f\to 0\), CE rewards the model for predicting all-background and the foreground gradient vanishes. Dice loss has no such collapse, because its ratio is scale-invariant in the foreground set: doubling \(\sum _i y_i\) and \(\sum _i p_i y_i\) together leaves \(\mathcal {L}_{\text {Dice}}\) unchanged, so a tiny foreground is not automatically a small loss. Figure 24.8 shows both behaviours side by side at three foreground fractions:

• • \(f=0.5\) – balanced (a roughly 50/50 split, e.g. generic binary classification). The CE optimum sits at \(p^{*}=0.5\) (middle dotted line in panel a); CE and soft Dice agree on direction, both descending together as \(p\to 1\). The hard-mask Dice loss in panel b drops from \(1\) (all-background, \(p\le 0.5\)) to \((1-f)/(1+f)=1/3\) (all-foreground, \(p>0.5\)) – a clear improvement, though Dice\(=2/3\) is still far from a good prediction.

• • \(f=0.1\) – mildly imbalanced (typical foreground objects in natural images). The CE optimum has slid left to \(p^{*}=0.1\): a model that admits only \(10\%\) foreground confidence is already CE-optimal, while soft Dice keeps decreasing all the way to \(p=1\) and its gradient still asks for more foreground confidence. Hard-mask Dice plateaus at \((1-f)/(1+f)\approx 0.82\) on \(p>0.5\).

• • \(f=0.01\) – severely imbalanced (small lesions in medical scans, thin roads in satellite tiles). The CE optimum collapses against the left axis: predicting all-background is essentially optimal and the foreground gradient is negligible. Soft Dice is still monotone-decreasing in \(p\), but the hard-mask Dice loss only drops to \(\approx 0.98\) even after thresholding everything to foreground – which is why training uses the soft form rather than thresholding inside the loss.

Multi-class case. With \(C\) foreground classes and a softmax probability vector output \(\bp _i\in [0,1]^{C+1}\) at every pixel and the ground truth is the one-hot vector \(\by _i\). The Dice loss is computed per class against its one-vs-rest indicator and then averaged,

\begin{equation} \mathcal {L}_{\text {Dice}}^{\text {multi}} \;=\; 1 \;-\; \frac {1}{C}\sum _{c=1}^{C}\frac {2\sum _i p_i^{(c)} y_i^{(c)}}{\sum _i p_i^{(c)} + \sum _i y_i^{(c)} + \varepsilon } , \end{equation}

where \(p_i^{(c)}\) is the softmax probability of class \(c\) at pixel \(i\), \(y_i^{(c)}\in \{0,1\}\) is the one-hot ground-truth indicator, and \(\varepsilon \) is a small constant (\(\sim 1\)) that prevents division by zero on images where a class is absent (\(\sum _i y_i^{(c)}=0\)). The background class \(c=0\) is excluded from the sum.

Practical recipe. The standard combined loss is \(\mathcal {L} = \mathcal {L}_{\text {CE}} + \mathcal {L}_{\text {Dice}}\) (binary or multi-class as appropriate), which keeps the per-pixel calibration of CE and the imbalance robustness of Dice.

24.10.3 Summary

Further Reading

• • Intersection Over Union (IoU): From Theory to Practice

• • Intersection over Union (IoU) in Object Detection & Segmentation

Examples and Demos:

• • Segment Anything and Segment Anything Model by Meta AI Demos (accessed 25-May-2026)

• • Segment Anything by Hugging Face (accessed 25-May-2026)

• • Conversational image segmentation with Gemini 2.5 by Google (accessed 25-May-2026)