Machine Learning & Signals Learning

19 Visual Representation Learning and Diagnostics

The previous chapter (Ch. 18) covered three tasks whose output is tied to a fixed class list: classification, segmentation and detection. This chapter covers what to do when that list is not fixed, or not available at all, and how to audit the result.

Notation As in Ch. 18, an input image is \(\bX \in \mathbb {R}^{H\times W\times C_{\text {in}}}\) with height \(H\), width \(W\) and \(C_{\text {in}}\in \{1,3\}\) channels, and \(C\) denotes the number of target classes. The new object of this chapter is the embedding \(\bz =f_\theta (\bX )\in \mathbb {R}^d\), a fixed-length vector that stands in for the image in every downstream computation.

The chapter covers:

• • Section 19.1 (Siamese Networks): learn a numeric “fingerprint” for each image so that similar images get nearby fingerprints; useful when new classes appear at test time or only a handful of examples per class are available.

• • Section 19.2 (Masked Autoencoders): learn the same kind of backbone from unlabeled images alone, by hiding most of each image and reconstructing it.

• • Section 19.3 (Comparison): the tasks of both chapters side by side, with their losses and headline metrics.

• • Sections 19.4 and 19.5 (Diagnostics): image-specific sanity checks, and the interpretation of models that fuse an image with another modality.

19.1 Siamese Networks

A Siamese network is not a new layer type but a training configuration: the same backbone (CNN for images, RNN or Transformer for sequences, fully-connected for tabular data) is applied to each input in a pair (or triplet), and the loss is defined on distances in the resulting embedding space rather than on a classification output.

19.1.1 Definitions

Input space

The input space \(\mathcal {X}\) is the set of all admissible raw inputs to the model. Each \(\bx \in \mathcal {X}\) is a single example (e.g. an image, a sentence, a signal segment, or a tabular row). The dimensionality of \(\mathcal {X}\) is typically large and structured: a \(224\times 224\) RGB image lives in \(\mathbb {R}^{224\times 224\times 3}\), a \(T\)-sample audio clip in \(\mathbb {R}^{T}\).

Learned Features

In modern representation learning, learned features refer to any internal data representations generated by a machine learning model. Instead of relying on manual, hand-crafted feature engineering, models optimize parameters \(\theta \) to automatically extract useful characteristics from a raw input space \(\mathcal {X}\). These representations can take a variety of structural forms, including high-dimensional spatial grids in convolutional neural networks, categorical logit scores, or attention weight matrices in transformers.

Embedding

An embedding is a specific, highly structured subset of learned features. It is defined as a learned mapping \(f_\theta :\mathcal {X}\to \mathbb {R}^d\) from an input space \(\mathcal {X}\) (such as images, text, audio, or signals) to a low-dimensional Euclidean space \(\mathbb {R}^d\), called the embedding space or latent space. The continuous vector \(f_\theta (\bx )\in \mathbb {R}^d\) is the embedding of input \(\bx \). An effective embedding satisfies two core properties:

• • Geometric: Euclidean (or cosine) distance in \(\mathbb {R}^d\) reflects task-relevant semantic similarity.

• • Compact: \(d \ll \dim (\mathcal {X})\), so downstream tasks (classification, retrieval, clustering) operate on a small fixed-size vector rather than on the raw input.

One-shot and few-shot learning

A classification setting in which the classes encountered at inference time are not those seen during training, and only a very small number of labelled examples per class is available for the new classes:

• • One-shot: exactly one labelled example per class.

• • Few-shot (\(K\)-shot, \(N\)-way): \(K\) labelled examples (typically \(K\le 5\)) for each of \(N\) classes; a query is classified among those \(N\).

The small set of labelled examples for the new classes is called the support set; the unlabelled inputs to be classified form the query set.

A standard solution is to learn a similarity-preserving embedding \(f_\theta \) on a separate large dataset and, at inference, classify each query by some classifier.

19.1.2 When to use it

Choose a Siamese embedding when any of the following hold:

• • The class set is open: new identities or categories appear at test time without retraining (as opposed to a closed set, fixed at training time).

• • Only a handful of labelled examples per class is available (one-shot or few-shot, Sec. 19.1.1).

• • The deployment task is verification or retrieval rather than closed-set classification.

Typical settings:

• • Face verification and recognition (FaceNet, DeepFace): one model handles open-set identification, where the set of identities at test time is not known at training time.

• • Signature and handwriting verification.

• • One-shot and few-shot learning (Sec. 19.1.1): a handful of labelled examples per class are enough at inference, since classification reduces to nearest-neighbor search in embedding space.

• • Signal similarity, e.g. matching ECG segments or audio clips by a learned distance rather than hand-crafted features.

Counter-indications. If the class set is closed and labelled data is plentiful per class, a plain softmax classifier (Sec. 18.2) is simpler, faster to train, and usually as accurate. If pixel-precise localization or per-pixel labels are needed, use segmentation (Sec. 18.3) or detection (Sec. 18.4) instead; an embedding alone does not localize.

19.1.3 Architecture

Two identical sub-networks share the same parameters \(\theta \) (tied weights, Fig. 19.1). Each branch maps an input \(\bx \) to an embedding \(f_\theta (\bx )\in \mathbb {R}^d\), and a pair is compared by the embeddings distance \(d(\bx _1,\bx _2)\) (Sec. 10.2). The most common embedding distances are \(L_2\),

\begin{equation} d(\bx _1,\bx _2) \;=\; \bigl \|f_\theta (\bx _1)-f_\theta (\bx _2)\bigr \|_2 . \end{equation}

and cosine distance.

Properties of the architecture:

• • Weight sharing guarantees the comparison is symmetric: \(d(\bx _1,\bx _2)=d(\bx _2,\bx _1)\).

• • Halves the number of trainable parameters relative to two independent encoders.

• • Both inputs are guaranteed to be processed by the same feature extractor, so distances are meaningful.

• • The embedding is normalised (typically to the unit sphere, \(\|f_\theta (\bx )\|_2=1\)) to bound the loss and decouple it from feature magnitude.

19.1.4 Loss

The Siamese architecture (Fig. 19.1) produces embeddings, not class probabilities, so cross-entropy is not directly applicable. The supervision is structural: rather than a class label per input, the training set provides pairs \((\bx _1,\bx _2,y)\) with a binary similarity label \(y\in \{0,1\}\), or triplets \((\bx _a,\bx _p,\bx _n)\) where \(\bx _p\) is known to share the anchor’s class and \(\bx _n\) does not. The loss is therefore a function of the embedding distance \(d(\bx _1,\bx _2) = \|f_\theta (\bx _1) - f_\theta (\bx _2)\|_2\) and of the similarity labels.

Two design choices recur in all Siamese losses:

• • Distance pulls similar pairs together and pushes dissimilar pairs apart. Without an asymmetry between the two, the trivial solution \(f_\theta \equiv \text {const}\) minimises every distance and the network learns nothing.

• • A margin \(m>0\) caps how hard a dissimilar pair is pushed. Once the pair is far enough apart, the loss contributes no gradient; training effort is spent only on pairs that still violate the geometric goal.

The next two paragraphs introduce the two standard instantiations: contrastive loss on pairs, and triplet loss on (anchor, positive, negative) triplets.

Contrastive loss Pair-based supervision with a binary similarity label \(y\in \{0,1\}\) (\(y=1\) similar, \(y=0\) dissimilar). The idea is to enforce one geometric goal per pair:

\begin{equation} d(\bx _1,\bx _2)\;\to \; 0 \text { when } y=1, \qquad d(\bx _1,\bx _2)\;\ge \; m \text { when } y=0, \end{equation}

i.e. similar pairs should collapse to the same embedding, while dissimilar pairs should be at least \(m\) apart. The two cases are turned into a single loss by penalising the squared distance for similar pairs and the hinged margin violation \(\max (0,\,m-d)\) for dissimilar ones, then selecting between them with the label \(y\):

\begin{equation} \mathcal {L}_{\text {contrastive}} \;=\; y\,d^2 \;+\; (1-y)\,\max (0,\,m-d)^2, \end{equation}

where \(m>0\) is a margin. Similar pairs are pulled together (the first term shrinks \(d\)), while dissimilar pairs are pushed apart only until their distance reaches \(m\); beyond that, no gradient flows. The margin prevents the trivial collapse \(f_\theta \equiv \text {const}\).

Triplet loss Triplet-based supervision is usually preferred in practice. Each example is a triplet \((\bx _a,\bx _p,\bx _n)\) with:

• • anchor \(\bx _a\),

• • positive \(\bx _p\) (same class as the anchor) and

• • negative \(\bx _n\) (different class).

The idea is to enforce one geometric inequality per triplet:

\begin{equation} d(\bx _a,\bx _p) + m \;\le \; d(\bx _a,\bx _n) , \end{equation}

i.e. the anchor must be at least \(m\) closer to the positive than to the negative. Rearranging gives a non-negative violation \(v = d(\bx _a,\bx _p)^2 - d(\bx _a,\bx _n)^2 + m\), positive exactly when the constraint is broken. Hinging it at \(0\) (so satisfied triplets contribute no loss) and using squared distances for a smoother gradient yields

\begin{equation} \mathcal {L}_{\text {triplet}} \;=\; \max \!\Bigl (0,\; \|f(\bx _a)-f(\bx _p)\|_2^2 \;-\; \|f(\bx _a)-f(\bx _n)\|_2^2 \;+\; m\Bigr ) \end{equation}

forces the anchor-to-positive distance to be smaller than the anchor-to-negative distance by at least the margin \(m\). Compared with contrastive loss, the triplet form encodes a relative ranking: it does not require an absolute distance threshold for similar pairs, only that positives are closer than negatives.

Supervised contrastive (SupCon) loss Triplet loss uses one positive and one negative per anchor, so each gradient step depends critically on the quality of the mined triplet. Supervised contrastive (SupCon) loss generalises this: every same-class example in the mini-batch is treated as a positive, and every other example as a negative, so each anchor contributes a softmax over many pairs at once. SupCon pairs naturally with the \(P\times K\) sampling described in Batch construction below and removes the need for explicit triplet mining.

19.1.5 Training

Hard negative mining The number of possible triplets grows cubically with the dataset, but most are uninformative.

• • Easy triplets already satisfy the margin and contribute zero gradient; including them slows training without improving the model.

• • Hard triplets violate the margin by a large amount and dominate the loss; using only the hardest ones can destabilise training (especially with label noise).

• • Semi-hard triplets: negatives that are farther than the positive but still within the margin. This is the standard compromise.

• • Batch-hard mining: for each anchor in a mini-batch, form a triplet from the hardest positive (the same-class example that is currently farthest in embedding space, i.e. the one the model gets most wrong) and the hardest negative (the different-class example that is currently closest). The two extrema are taken within the batch only, so the cost reduces to one \(O(B^2)\) pairwise-distance matrix on the \(B\) batch embeddings instead of a nearest-neighbour search across the whole training set, and the gradient at every step is concentrated on the most informative triplet per anchor. It pairs naturally with \(P\times K\) batch construction, which guarantees that each batch contains enough same-class and different-class candidates.

Batch construction Triplet and contrastive losses are only as good as the pairs the mini-batch actually contains: a batch of mostly singletons cannot supply positives, and a batch of mostly one class cannot supply informative negatives. The standard recipe is \(P\times K\) sampling: each mini-batch is built from \(P\) classes drawn uniformly, with \(K\) examples per class.

• • Guarantees \(K-1\) in-batch positives and \((P-1)\,K\) in-batch negatives for every anchor, so batch-hard mining always has candidates.

• • Typical values: \(P\in [8,32]\), \(K\in [4,8]\), giving batches of \(64\)–\(256\) samples; small \(K\) wastes positives, large \(K\) wastes negatives.

• • Classes are usually sampled uniformly rather than proportionally to frequency, so head classes do not dominate the loss.

• • Combined naturally with batch-hard mining: for each anchor, pick the farthest same-class example and the closest different-class example within the batch.

19.1.6 Classifier

The trained Siamese network produces embeddings, not class probabilities; classification is built on top of them in one of four ways.

• • Verification (open-set, “are these two inputs from the same class?”; e.g. face unlock, signature check): map a pair to a binary accept/reject decision \(\widehat {y} = \bOne [\,d(\bx _1,\bx _2) \le \tau \,]\in \{0,1\}\), accepting the pair as same-class when their embedding distance falls below a threshold \(\tau \) tuned on a held-out validation set of labelled same/different pairs. Because the decision depends only on the distance, identities unseen during training can be verified at test time without retraining.

• • Identification / retrieval (open-set, “which class is this?”): single input \(\to \) class label. Precompute a gallery of embeddings \(\{f_\theta (\bx _i)\}\) with known labels \(y_i\in \{1,\dots ,C\}\) and classify a query \(\bx \) by \(k\)-NN in embedding space, \(\widehat {y}(\bx )\in \{1,\dots ,C\}\). Adding a new class only requires inserting its embeddings into the gallery.

• • Embeddings as features for an ML classifier (closed-set): freeze \(f_\theta \) and treat \(\bz =f_\theta (\bx )\in \mathbb {R}^d\) as a fixed-length feature vector. Fit a standard supervised classifier, e.g. multinomial logistic regression or a (kernel) SVM, on the labelled set \(\{(\bz _i,y_i)\}\) for the \(C\) target classes. Because the embedding is low-dimensional and (typically) \(L_2\)-normalised, even a linear classifier is usually sufficient and trains in seconds.

• • Closed-set classification (Siamese as pre-training): once \(f_\theta \) has learned a good representation, freeze it and fit a small classifier head \(g_\phi :\mathbb {R}^d \to \Delta ^{C-1}\) (typically a single linear layer with softmax) on labelled data for the \(C\) target classes. This decouples representation learning from classification and is the standard recipe for using a Siamese / contrastive backbone in supervised downstream tasks.

19.1.7 Evaluation pitfall: leakage through pretraining

Every deployment mode above is built in two phases: pretrain \(f_\theta \), then freeze it and fit something on the embeddings. The train/test split is normally applied in the second phase, when the classifier is fitted. Phase 1 is left to run over whatever data is at hand, which is usually the whole dataset. When that happens the test examples are inside the embedding’s own training set, and the score reported at the end is optimistic (Fig. 19.2).

19.1.8 Performance metrics

For the verification deployment mode, a binary same/different decision is driven by a threshold \(\tau \) on the embedding distance \(d(\bx _1,\bx _2)\). Writing

\begin{equation} \text {TAR}(\tau ) = \Pr (d(\bx _1,\bx _2)\le \tau \,\big |\, y=1) , \qquad \text {FAR}(\tau ) = \Pr (d(\bx _1,\bx _2)\le \tau \,\big |\, y=0) , \end{equation}

the true accept rate (TAR \(=\) TPR) and false accept rate (FAR \(=\) FPR) trace a ROC curve as \(\tau \) varies, summarised by AUC (Sec. 12.5 of Chapter 12). Two operating-point summaries dominate biometric reporting:

• • Equal Error Rate (EER): the point where \(\text {FAR}=\text {FRR}\) (false reject rate \(=1-\text {TAR}\)), the intersection of the ROC with the anti-diagonal \(\text {TPR}=1-\text {FPR}\). Single-number summary.

• • TAR@FAR\(=10^{-k}\): true-accept rate at a fixed, very small FAR (typically \(10^{-3}\), \(10^{-4}\) or \(10^{-6}\)). The standard reporting on face-verification benchmarks (LFW, MegaFace, IJB-C) because deployments target a low false-accept budget.

Retrieval and frozen-embedding deployment modes are evaluated by Rank-\(k\) / CMC and Recall@\(k\) for retrieval, and by a linear probe or \(k\)-NN accuracy for downstream classification on frozen embeddings.

19.1.9 Summary

19.2 Self-Supervised Pretraining with Masked Autoencoders

Every recipe so far in this chapter reads labels. The contrastive, triplet and SupCon losses of Sec. 19.1 all have to know which pairs share a class, which is exactly why the leakage box of Sec. 19.1.7 classifies contrastive pretraining as supervised. In practice that requirement, and not the network, is the bottleneck: unlabeled images are abundant, annotated ones are not.

Self-supervised learning (SSL) removes the requirement by manufacturing the supervision from the input itself. The training signal comes from a pretext task: an artificial prediction problem whose target is produced by a deterministic corruption of the input, so that every unlabeled image supplies its own target for free. The pretext task is never the goal; it is a device for forcing the encoder to learn something transferable, after which it is thrown away and the encoder is reused. The masked autoencoder (MAE) is the pretext task that dominates image pretraining, and its rule is one line: hide most of the image, reconstruct it.

19.2.1 The pretext task

Masked autoencoder (MAE): Cut the input image \(\bX \in \mathbb {R}^{H\times W\times C_{\text {in}}}\) into \(N=(H/\ell )(W/\ell )\) non-overlapping square patches \(\bx _1,\dots ,\bx _N\) of side \(\ell \) pixels, each flattened to \(\bx _n\in \mathbb {R}^{D}\) with \(D=\ell ^2 C_{\text {in}}\). Draw a random subset \(\mathcal {M}\subset \{1,\dots ,N\}\) of size \(|\mathcal {M}|=\rho N\), where \(\rho \in (0,1)\) is the mask ratio, and hide those patches. Then

• • the encoder \(f_\theta \) processes the \((1-\rho )N\) visible patches only, and never sees a masked one;

• • the decoder \(g_\phi \) receives the encoded visible patches together with a shared learned mask token placed at each masked position, and outputs a reconstruction \(\hat \bx _n\in \mathbb {R}^{D}\) for every \(n\in \mathcal {M}\).

Figure 19.3 shows the pipeline. The patch decomposition is not an extra assumption: it is how a vision transformer (the ViT rows of Table 18.1) already reads an image, one token per patch, so masking a patch simply means dropping a token.

19.2.2 Loss

The loss is the mean squared error over the masked patches only,

\begin{equation} \mathcal {L}_{\text {MAE}} \;=\; \frac {1}{|\mathcal {M}|\,D}\sum _{n\in \mathcal {M}} \bigl \|\hat \bx _n - \bx _n\bigr \|_2^2 , \label {eq-dlarch-mae} \end{equation}

that is, the average squared pixel error per masked patch, normalized per pixel by \(D\) so the number is comparable across patch sizes. Two conventions inside (19.7) carry most of its behavior:

• • The visible patches are excluded from the sum. Were they included, the network could drive the loss down by copying the patches it can already see, and that term, being trivially solvable, would dominate the gradient while teaching nothing about the hidden content.

• • The target patch is normalized before the comparison. Each \(\bx _n\) has its own mean subtracted and is divided by its own standard deviation (biased, over the \(D\) values of the patch). This removes patch brightness and contrast, which are predictable from the surrounding patches and would otherwise consume most of the loss budget, and leaves the local structure as the thing being scored.

19.2.3 Why the mask ratio is so high

The standard setting is \(\rho =0.75\): three quarters of the patches are removed. This is far more aggressive than the equivalent recipe for text, where masking roughly \(15\,\%\) of the tokens is enough, and the reason is redundancy. Neighboring pixels are strongly correlated, so at a low mask ratio a hidden patch is recoverable by interpolating from the patches that touch it, a local operation that requires no notion of what the image contains. Raising \(\rho \) removes that shortcut: at \(\rho =0.75\) the surviving patches are spatially sparse, the nearest visible neighbor of a hole is typically several patches away, and filling the hole requires the global structure of the object.

The mask ratio \(\rho \) and the patch side \(\ell \) are the two hyperparameters of the pretext task, and they interact: a small \(\ell \) makes each patch more predictable from its neighbors and therefore calls for a larger \(\rho \), while a large \(\ell \) makes the reconstruction harder and the token sequence shorter, so pretraining is cheaper but coarser.

19.2.4 Asymmetric encoder and decoder

The design has one more property worth naming, because it is what makes the recipe affordable. The encoder runs on the visible patches only, so at \(\rho =0.75\) it processes a quarter of the tokens; the mask tokens enter at the decoder, not at the encoder. The decoder is deliberately shallow, a few blocks against the encoder’s dozens, because it is discarded after pretraining and any capacity spent there is capacity not spent on the part that is kept.

19.2.5 Using the pretrained encoder

When pretraining ends, \(g_\phi \) and the mask token are discarded and \(f_\theta \) is the backbone. From there the options are the ones already tabulated for the Siamese embedding (Table 19.2, Sec. 19.1.6), since both phases produce the same object, a frozen map from image to vector:

• • Fine-tuning: unfreeze \(f_\theta \) and train it together with a task head on the labelled set. This is the default for MAE and where its accuracy advantage appears.

• • Linear probe: freeze \(f_\theta \) and fit a single linear softmax layer on \(\bz =f_\theta (\bX )\).

• • \(k\)-NN on frozen embeddings: no fitting at all, used as a cheap diagnostic of representation quality.

Table 19.3 places the two pretraining families side by side.

19.2.6 Summary

19.3 Comparison of CV tasks

Table 19.4 summarizes the computer-vision tasks covered in this chapter and the previous one: image classification (Sec. 18.2), image segmentation (Sec. 18.3), object detection (Sec. 18.4), Siamese embeddings (Sec. 19.1), and masked-autoencoder pretraining (Sec. 19.2). The right column captures the practical question each one is designed to answer.

Table 19.5 consolidates the evaluation metrics of both chapters, grouped by the task that produces them. Classification primitives (accuracy, \(F_1\), AUC) are deferred to Chapter 12 and reused by several modes here.

19.4 Image-Specific Sanity Checks (*)

The sanity checks in this section (beyond these in Ch. 13) are specific to classifiers operating on image data (raw pixels or spatial feature maps). They test whether the model exploits spatial structure—edges, textures, shapes—or relies on superficial statistics that happen to correlate with the labels.

19.4.1 Visual Inspection of Worst-Performing Examples

Worst-case visual inspection: Rank the held-out examples by a per-example performance score (loss, or \(1-p_y\) for classification, \(1-\text {IoU}\) for segmentation, \(1-\text {IoU}\) between best-matched boxes for detection), select the worst \(k\) (typically \(k=20\) to \(50\)), and display them in a grid annotated with the true label, the predicted label, and the model’s confidence.

Aggregate metrics (Sec. 18.2.1 for classification, Sec. 18.3.1 for segmentation, Sec. 18.4.1 for detection) summarize performance into a single number and hide which images the model gets wrong. Manually inspecting a few dozen worst cases regularly surfaces failure modes that no single-number metric can reveal, and it is among the simplest diagnostics available: it requires no retraining and no additional labels.

Patterns to look for in the worst-case grid:

• • Label noise: the ground-truth annotation is wrong or ambiguous, and the model’s prediction is in fact reasonable. Common on crowd-sourced or hastily relabeled datasets.

• • Out-of-distribution inputs: corrupt files, extreme rotation or cropping, drastically different lighting or resolution from the training set.

• • Confounding artifacts: timestamps, watermarks, scanner markers, ruler overlays, or hospital identifiers that correlate with the label in training but not at deployment.

• • Systematic class confusion: one class is consistently mistaken for another (e.g. visually similar dog breeds, two adjacent organ classes), pointing to a missing distinguishing feature or insufficient training examples for the confused pair.

19.4.2 Pixel Shuffle Test

Pixel shuffle test: Given a dataset of \(N\) images, randomly permute all pixel locations within each image independently, producing a shuffled dataset \(\bX _{\text {shuf}}\). Re-extract features from \(\bX _{\text {shuf}}\) and repeat the cross-validation procedure.

Shuffling destroys all spatial structure (edges, textures, object shapes) while preserving the per-image pixel histogram (mean intensity, variance, and higher-order marginal statistics remain identical). A classifier that genuinely relies on spatial patterns will see its accuracy drop to chance level on \(\bX _{\text {shuf}}\). Conversely, if the accuracy remains high after shuffling, the classifier exploits per-pixel statistics rather than spatial structure, and the original cross-validation result is unreliable.

19.4.3 Black-Patch Test

Black-patch test: Occlude a randomly positioned square region of each image with a black (zero or other constant-valued) patch of side length \(s\) pixels. Re-extract features and repeat cross-validation. Repeat for progressively larger patch sizes (e.g., \(s \in \{16, 32, 56, 112\}\)) to obtain an accuracy-versus-occlusion curve.

The black-patch test probes whether the classifier uses distributed spatial information or relies on a localized region. Three representative outcomes:

• • Gradual, monotonic decline in accuracy as \(s\) increases—the classifier uses information spread across the image, which is the expected behavior for a well-trained model.

• • Sharp drop at a specific patch size—the classifier depends on a localized region. This may indicate a genuine region of interest, but could also signal reliance on a confounding artifact (e.g., a timestamp, label overlay, or acquisition marker embedded in a fixed image location).

• • Stable accuracy across all patch sizes—the classifier is largely insensitive to spatial content, raising concern that it exploits non-image-based confounders (e.g., differences in file encoding or image dimensions between classes).

19.5 Interpreting Multimodal Models (MultiViz) (*)

The diagnostics above probe a classifier that sees a single image. Many modern vision models are instead multimodal: they fuse an image with another modality, such as the text of a question in visual question answering (VQA), or accompanying tabular and clinical data. For such a model the interpretation question is richer: not only where in one image the model looks, but which modality drives the answer and how the modalities interact. MultiViz is a framework that organizes this analysis into four stages.

It is worth separating this task from the Classifier Comparison chapter (Ch. 14). Those methods compare two models from their outputs alone (hard labels or predicted probabilities on a shared test set) to decide which is better or whether to fuse them. MultiViz instead interprets a single fused model, one prediction at a time, by opening its inputs and internal features. The two live on orthogonal axes; the closest relatives of MultiViz are the single-model, input-level diagnostics of this chapter, such as the worst-case inspection and the black-patch/occlusion test (Sec. 19.4.3).

The four MultiViz stages are:

• • Unimodal importance: attribute the prediction to individual inputs within each modality, using per-modality saliency (gradient-based or LIME). The single-image analogue is the black-patch/occlusion test (Sec. 19.4.3), which localizes the image region a classifier depends on.

• • Cross-modal interactions: identify which pairs of inputs across modalities jointly drive the prediction, for example a specific question word tied to a specific image region. This stage has no single-modality counterpart, and it is often where a multimodal model succeeds or fails.

• • Multimodal representations: inspect the fused internal features, in the same spirit as the embedding analysis used for Siamese networks (Sec. 19.1) and the low-dimensional projections of Ch. 11 (t-SNE, UMAP), to see what the representation has encoded.

• • Prediction analysis: fit a sparse, local surrogate that relates a few representation units to the final output, closing the loop from inputs, through interactions and representations, to the prediction.

Further Reading

• • Convolutional Neural Networks (Course 4 of the Deep Learning Specialization), especially C4W4 (Siamese Network)

• • Kaiming He et al., Masked Autoencoders Are Scalable Vision Learners, CVPR 2022 (Sec. 19.2).

• • Paul Pu Liang et al., MultiViz: Towards Visualizing and Understanding Multimodal Models, ICLR 2023 (Sec. 19.5).