27 Computer Vision: Task Comparison and Diagnostics
Goal: Compare the computer-vision tasks of the previous chapters side by side, and check that a trained vision model relies on real image structure.
The chapter covers:
• Section 27.1(Comparison): the tasks of Chapters 23 to 26 side by side, with their losses and headline metrics.
• Sections 27.2 and 27.3(Diagnostics): image-specific sanity
checks, and the interpretation of models that fuse an image with another modality.
27.1 Comparison of CV tasks
Table 27.1 summarizes the computer-vision tasks covered in the previous four chapters: image classification (Ch. 23), image segmentation (Ch. 24), object detection (Ch. 25),
Siamese embeddings (Sec. 26.1), and masked-autoencoder pretraining (Sec. 26.2). The right column captures the
practical question each one is designed to answer.
Table 27.1: Computer-vision tasks side by side.
.
Task
Output
Loss family
Headline metric
When to use
Image classification
one class label per image
cross-entropy (optionally focal)
Top-1 / Top-\(k\), macro-\(F_1\)
one dominant object, closed set
Image segmentation
per-pixel class map \(\in \{0,\dots ,C\}^{H\times W}\)
CE \(+\) Dice
mIoU, Dice
masks needed, boundaries matter
Object detection
list of \((b, y, s)\) tuples
box regression \(+\) objectness/class CE
mAP@[.5:.95]
locate and count objects, real-time
Siamese embedding
embedding \(\bz \in \mathbb {R}^d\)
contrastive / triplet / SupCon
verification AUC, top-\(k\) retrieval
open set, few-shot, similarity search
Masked autoencoder
reusable backbone \(f_\theta \)
masked-patch MSE
fine-tuned / linear-probe accuracy
unlabeled images plentiful, labels scarce
Table 27.2 consolidates the evaluation metrics of these chapters, grouped by the task that produces them. Classification primitives (accuracy, \(F_1\), AUC) are deferred to
Chapter 17 and reused by several modes here.
Table 27.2: Headline performance metrics of the computer-vision chapters, grouped by task.
Goal: Validate that image-based classifiers rely on meaningful spatial or global structure rather than localized artifacts or per-pixel statistics.
The sanity checks in this section (beyond these in Ch. 18) are specific to classifiers operating on image data (raw pixels or spatial feature maps). They test whether the model exploits
spatial structure—edges, textures, shapes—or relies on superficial statistics that happen to correlate with the labels.
27.2.1 Visual Inspection of Worst-Performing Examples
Worst-case visual inspection: Rank the held-out examples by a per-example performance score (loss, or \(1-p_y\) for classification, \(1-\text {IoU}\) for segmentation, \(1-\text {IoU}\) between best-matched boxes for
detection), select the worst \(k\) (typically \(k=20\) to \(50\)), and display them in a grid annotated with the true label, the predicted label, and the model’s confidence.
Aggregate metrics (Sec. 23.1 for classification, Sec. 24.4 for segmentation, Sec. 25.2 for detection) summarize performance into a single number and hide which images the model gets wrong. Manually inspecting a few dozen worst cases regularly
surfaces failure modes that no single-number metric can reveal, and it is among the simplest diagnostics available: it requires no retraining and no additional labels.
Patterns to look for in the worst-case grid:
•Label noise: the ground-truth annotation is wrong or ambiguous, and the model’s prediction is in fact reasonable. Common on crowd-sourced or hastily relabeled datasets.
•Out-of-distribution inputs: corrupt files, extreme rotation or cropping, drastically different lighting or resolution from the training set.
•Confounding artifacts: timestamps, watermarks, scanner markers, ruler overlays, or hospital identifiers that correlate with the label in training but not at deployment.
•Systematic class confusion: one class is consistently mistaken for another (e.g. visually similar dog breeds, two adjacent organ classes), pointing to a missing distinguishing feature or
insufficient training examples for the confused pair.
Model vs annotation
The worst-case set is usually a mix of:
• model failure and
• annotation failure.
Separate the two before reacting: relabeling or removing mislabeled examples often improves more than another round of hyperparameter tuning.
27.2.2 Pixel Shuffle Test
Pixel shuffle test: Given a dataset of \(N\) images, randomly permute all pixel locations within each image independently, producing a shuffled dataset \(\bX _{\text {shuf}}\). Re-extract features from \(\bX _{\text {shuf}}\)
and repeat the cross-validation procedure.
Shuffling destroys all spatial structure (edges, textures, object shapes) while preserving the per-image pixel histogram (mean intensity, variance, and higher-order marginal statistics remain identical). A classifier that genuinely relies on spatial
patterns will see its accuracy drop to chance level on \(\bX _{\text {shuf}}\). Conversely, if the accuracy remains high after shuffling, the classifier exploits per-pixel statistics rather than spatial structure, and the original cross-validation
result is unreliable.
The pixel shuffle test is specific to classifiers that operate on raw pixel features or spatial feature maps (e.g., convolutional networks). It does not apply when hand-crafted, non-spatial features are extracted before classification.
27.2.3 Black-Patch Test
Black-patch test: Occlude a randomly positioned square region of each image with a black (zero or other constant-valued) patch of side length \(s\) pixels. Re-extract features and repeat cross-validation. Repeat for progressively larger
patch sizes (e.g., \(s \in \{16, 32, 56, 112\}\)) to obtain an accuracy-versus-occlusion curve.
The black-patch test probes whether the classifier uses distributed spatial information or relies on a localized region. Three representative outcomes:
• Gradual, monotonic decline in accuracy as \(s\) increases—the classifier uses information spread across the image, which is the expected behavior for a well-trained model.
• Sharp drop at a specific patch size—the classifier depends on a localized region. This may indicate a genuine region of interest, but could also signal reliance on a confounding artifact (e.g., a timestamp,
label overlay, or acquisition marker embedded in a fixed image location).
• Stable accuracy across all patch sizes—the classifier is largely insensitive to spatial content, raising concern that it exploits non-image-based confounders (e.g., differences in file encoding or image
dimensions between classes).
When interpreting the black-patch test, consider the patch area relative to the total image area. A \(112 \times 112\) patch occludes roughly \(44\%\) of a \(168 \times 168\) image but only \(5\%\) of a \(512 \times 512\) image. Report
patch sizes as fractions of the image dimensions to allow meaningful comparison across datasets.
Goal: Explain a prediction from a model that fuses several modalities (e.g., an image and a text question) by attributing it to the contributing inputs, their interactions, and the internal representations.
The diagnostics above probe a classifier that sees a single image. Many modern vision models are instead multimodal: they fuse an image with another modality, such as the text of a question in visual question answering (VQA), or
accompanying tabular and clinical data. For such a model the interpretation question is richer: not only where in one image the model looks, but which modality drives the answer and how the modalities
interact. MultiViz is a framework that organizes this analysis into four stages.
It is worth separating this task from the Classifier Comparison chapter (Ch. 19). Those methods compare two models from their outputs alone (hard labels or
predicted probabilities on a shared test set) to decide which is better or whether to fuse them. MultiViz instead interprets a single fused model, one prediction at a time, by opening its inputs and internal features. The two live on
orthogonal axes; the closest relatives of MultiViz are the single-model, input-level diagnostics of this chapter, such as the worst-case inspection and the black-patch/occlusion test (Sec. 27.2.3).
The four MultiViz stages are:
•Unimodal importance: attribute the prediction to individual inputs within each modality, using per-modality saliency (gradient-based or LIME). The single-image analogue is the
black-patch/occlusion test (Sec. 27.2.3), which localizes the image region a classifier depends on.
•Cross-modal interactions: identify which pairs of inputs across modalities jointly drive the prediction, for example a specific question word tied to a specific image region. This stage
has no single-modality counterpart, and it is often where a multimodal model succeeds or fails.
•Multimodal representations: inspect the fused internal features, in the same spirit as the embedding analysis used for Siamese networks (Sec. 26.1) and the low-dimensional projections of Ch. 16 (t-SNE, UMAP), to see what the representation has encoded.
•Prediction analysis: fit a sparse, local surrogate that relates a few representation units to the final output, closing the loop from inputs, through interactions and representations, to the prediction.
The visualizations produced at each stage are hypotheses about the model’s behavior, not proofs. A highlighted region or a clean-looking embedding cluster can still be a spurious artifact, so the same cautions as the sanity checks above apply:
confirm an interpretation with a controlled intervention (e.g., occluding the highlighted region, or perturbing the implicated modality) before trusting it.