Machine Learning & Signals Learning

15 Classifier Fusion

When multiple data sources are available, there are several strategies for incorporating them into a classification pipeline. Figure 15.1 illustrates five common approaches.

• • Single source (a): a single dataset is processed through one feature-extraction and classification pipeline.

• • Data concatenation (b): raw data from multiple sources are concatenated before feature extraction, producing a single feature vector per sample.

• • Feature concatenation (c): each data source undergoes independent feature extraction; the resulting feature vectors are concatenated before classification.

• • Classifier fusion (d): each data source is processed by an independent feature-extraction and classification pipeline; the individual predictions are combined by a fusion rule (Sec. 15.3).

• • Classifiers ensemble (e): a single data source undergoes feature extraction, then multiple different classifiers are applied in parallel; their predictions are combined by an ensemble rule (e.g., majority voting or averaging) (Sec. 15.3).

Approaches (b) and (c) are collectively termed early fusion because they merge information before the classifier; approaches (d) and (e) are late fusion because independent classifiers produce separate outputs that are combined afterward.

15.1 Data Concatenation

Given \(K\) data sources producing raw vectors \(\bd _1, \bd _2, \ldots , \bd _K\) for the same sample, form the concatenated input:

\begin{equation} \bd = [\bd _1;\, \bd _2;\, \ldots ;\, \bd _K] \end{equation}

A single feature-extraction pipeline then maps \(\bd \) to a feature vector \(\bx \), which is passed to the classifier (Fig. 15.1b).

15.2 Feature Concatenation

Each source \(k\) is processed by its own feature-extraction pipeline, yielding \(\bx _k \in \mathbb {R}^{N_k}\). The concatenated feature vector is

\begin{equation} \bx = [\bx _1;\, \bx _2;\, \ldots ;\, \bx _K] \in \mathbb {R}^{N}, \quad N = \sum _{k=1}^{K} N_k \end{equation}

A single classifier operates on \(\bx \) (Fig. 15.1c).

Feature concatenation allows each source to use a specialized extraction pipeline suited to its modality. However, the concatenated dimension \(N\) grows with \(K\), increasing the risk of overfitting when \(M\) is small relative to \(N\).

15.3 Ensemble of Classifiers

15.3.1 Preface

Given \(K\) classifiers, each producing a prediction for the same input \(\bx \), fusion methods aggregate these predictions into a single output. The methods in this section apply to two settings (Fig. 15.1d,e):

• • Classifiers ensemble: \(K\) different classifiers operate on the same data source. The ensemble benefits from diversity among classifiers; if they make similar errors, combining them provides little gain. Classifier comparison methods (Sec. 14.1 and 14.2.8) can quantify this diversity by measuring:

  • – Yule’s Q (Sec. 14.1.6): measures the association between binary correct/incorrect outcomes. \(Q \approx 0\) indicates independent errors, while \(Q < 0\) suggests complementary performance.

  • – Error correlation (Sec. 14.2.8 and 14.2.9): the Pearson/Spearman correlation between per-sample probabilistic scores. \(r < 0\) and \(r_s<0\) indicate that the classifiers’ errors are negatively correlated, which indicates a high ensemble potential.

• • Classifier fusion: a single classifier is applied to \(K\) different data sources (e.g., multimodal sensors). When the sources originate from different domains, a domain consistency check (Sec. 13.2.2) should verify distributional compatibility before fusion. Classifier fusion methods are a subset of the ensemble methods; applicability is summarized in Sec. 15.3.9.

15.3.2 Majority Vote

Condorcet’s jury theorem

Condorcet’s jury theorem (1785) is a political science theorem about the relative probability of a given group of individuals arriving at a correct decision. The theorem suggests a decision-making scenario with two options, where one option is assumed to be correct. Given a group of \(n\) voters making independent choices with \(p>\frac {1}{2}\) of making the correct choice, the theorem states that the majority vote is most likely to select the correct option. Moreover, for \(n\rightarrow \infty \) the probability of the group reaching the correct decision by majority vote approaches 1.

Definition

Each classifier \(k\) produces a binary prediction \(\hat {y}_k\in \{0,1\}\). The fused prediction is

\begin{equation} \hat {y}_{\text {fused}} = \begin{cases} 1 & \displaystyle \sum _{k=1}^{K} \hat {y}_k > K/2 \\[6pt] 0 & \text {otherwise} \end {cases} \end{equation}

15.3.3 Soft Vote

Each probabilistic classifier \(k\) outputs \(p_k = f_k(\bx ) = \Pr (\hat {y}=1\mid \bx )\). The soft vote averages these probabilities:

\begin{equation} p_{\text {soft}} = \frac {1}{K}\sum _{k=1}^{K} p_k \end{equation}

The fused decision is

\begin{equation} \hat {y}_{\text {fused}} = \begin{cases} 1 & p_{\text {soft}} \ge 0.5\\ 0 & p_{\text {soft}} < 0.5 \end {cases} \end{equation}

For the special case of \(K=2\), if the classifiers disagree (\(\hat {y}_1 \neq \hat {y}_2\)), the fused prediction follows the more confident classifier:

\begin{equation} \hat {y}_{\text {fused}} = \begin{cases} \hat {y}_1 & |p_1 - 0.5| > |p_2 - 0.5| \\ \hat {y}_2 & |p_2 - 0.5| > |p_1 - 0.5| \end {cases} \end{equation}

If one classifier is very confident (e.g., \(p_k = 0.95\)) while the others are weakly opposed (e.g., \(p_j = 0.45\)), the confident vote contributes more to the average than a single hard vote would.

15.3.4 Linearly Weighted Combining

Each probabilistic classifier \(k\) outputs \(p_k = f_{k}(\bx )\). Assign non-negative weights \(w_k\ge 0\) with \(\sum _{k=1}^{K}w_k = 1\). The fused probability is

\begin{equation} p_{\text {fused}} = \sum _{k=1}^{K} w_k \, p_k \end{equation}

The fused decision is

\begin{equation} \hat {y}_{\text {fused}} = \begin{cases} 1 & p_{\text {fused}} \ge 0.5\\ 0 & p_{\text {fused}} < 0.5 \end {cases} \end{equation}

A common approach is to normalize validation accuracies [17]:

\begin{equation} w_k = \frac {a_k}{\sum _{j=1}^{K} a_j} \end{equation}

where \(a_k\) is the validation accuracy (or other performance metric, e.g., \(F_1\)) of classifier \(k\). Equal weights \(w_k = 1/K\) recover simple averaging.

15.3.5 Log-Likelihood Ratio (LLR)

Each probabilistic classifier \(k\) outputs \(p_k = f_{k}(\bx ) = \Pr (\hat {y}=1\mid \bx )\). Recall that the logit (log-odds) is given by

\begin{equation} \text {logit}_k = \ln \frac {p_k}{1 - p_k} \end{equation}

Under the assumption that the \(K\) classifiers (or classifications) are conditionally independent given the true label, the fused log-odds is the sum of individual logits:

\begin{equation} \text {logit}_{\text {fused}} = \sum _{k=1}^{K}\text {logit}_k = \sum _{k=1}^{K} \ln \frac {p_k}{1 - p_k} = \ln \prod _{k=1}^{K}\frac {p_k}{1 - p_k} \end{equation}

The fused probability is recovered by applying the sigmoid: \(p_{\text {fused}} = \sigma (\text {logit}_{\text {fused}})\).

When the classes are imbalanced, the log-prior odds should not be dropped. The general fusion formula is

\begin{equation} \text {logit}_{\text {fused}} = \ln \frac {M_1}{M_0} + \sum _{k=1}^{K}\text {logit}_k \end{equation}

where \(M_0\) and \(M_1\) are the class counts. The log-prior term acts as a constant bias that shifts the decision boundary toward the minority class. When the classes are balanced (\(M_0 = M_1\)), the bias vanishes and the standard formula is recovered.

The fused decision is

\begin{equation} \hat {y}_{\text {fused}} = \begin{cases} 1 & \text {logit}_{\text {fused}} \ge 0 \quad (p_{\text {fused}} \ge 0.5)\\ 0 & \text {logit}_{\text {fused}} < 0 \end {cases} \end{equation}

With imbalanced priors, the log-prior bias in \(\text {logit}_{\text {fused}}\) effectively shifts the decision boundary away from \(0.5\) toward the minority class.

15.3.6 Learned Combiner (Stacking)

The previous fusion methods use fixed rules. When the classifiers’ outputs are probabilistic, a stronger ensemble can be obtained by fitting a logistic regression model on the classifiers’ log-odds:

\begin{equation} \text {logit}(p_{\text {fused}}) = \beta _0 + \sum _{k=1}^{K} \beta _k\, \text {logit}_k \end{equation}

where \(\beta _0\) is a bias term and \(\beta _k\) are learned coefficients. The parameters \(\{\beta _0, \beta _1, \ldots , \beta _K\}\) are fit on a validation set, and the stacked model is evaluated on a separate test set to avoid overfitting.

The fused probability is recovered via the sigmoid: \(p_{\text {fused}} = \sigma \!\left (\beta _0 + \sum _{k} \beta _k\, \text {logit}_k\right )\).

15.3.7 General Non-Linear Combining

The stacking approach can be generalized by replacing the linear logistic regression model with any non-linear classifier (e.g., a neural network, random forest, or gradient boosting machine). The fused prediction is formulated as:

\begin{equation} p_{\text {fused}} = g_{\bphi }(p_1, \ldots , p_K) \end{equation}

where \(g_{\bphi }\) is the non-linear combiner parameterized by \(\bphi \). As with the linear learned combiner, the parameters \(\bphi \) are fit on a validation set, and the stacked model is evaluated on a separate test set.

15.3.8 Held-out average log-likelihood gain (*)

The ensemble gain can be quantified using the log-likelihood. Recall (Sec. 8.4) that the per-sample log-likelihood for a probabilistic classifier with output \(p_i = f_\bw (\bx _i)\) is

\begin{equation} \log p(y_i \mid \bx _i) = y_i \log p_i + (1-y_i)\log (1-p_i) \end{equation}

The held-out average log-likelihood gain of the ensemble over the best base model is

\begin{equation} \Delta \overline {\ell } = \frac {1}{M_{\text {test}}} \sum _{i=1}^{M_{\text {test}}} \left [\log p_{\text {fused}}(y_i \mid \bx _i) - \log p_{\text {best}}(y_i \mid \bx _i)\right ] \end{equation}

where \(p_{\text {fused}}\) and \(p_{\text {best}}\) are the predicted probabilities of the stacked model and the best individual classifier, respectively, evaluated on the test set.

Positive \(\Delta \overline {\ell }\) indicates that the ensemble provides better calibrated probabilities than the best base model. Equivalently, \(\Delta \overline {\ell }\) equals the reduction in BCE loss: \(\Delta \overline {\ell } = \loss _{\text {best}} - \loss _{\text {fused}}\).

15.3.9 Summary

The methods above apply to both classifiers ensemble of \(K\) classifiers and classifier fusion of interference from \(K\) different data sources (or noisy measurements \(\bx _1, \ldots , \bx _K\) of the same quantity). In the classifier-fusion setting, the independence assumption required for LLR fusion corresponds to independent measurement noise, and methods that assign different learned parameters per classifier (Linearly Weighted Combining, Learned Combiner) are not applicable, since there is only one classifier. The method summary is presented in Table 15.1.

Cross-validation of ensembles When evaluating an ensemble with cross-validation, the procedure depends on whether the fusion rule has learned parameters:

• • Fixed-rule methods (Majority vote, Soft vote, LLR fusion): no parameters are learned from data. A standard outer CV loop suffices. In each fold, train the \(K\) base classifiers on the training partition, apply the fixed fusion rule, and evaluate on the held-out test partition.

• • Learned-parameter methods (Weighted combining, Learned combiner): the combiner weights (\(w_k\) or \(\beta _k\)) must be fitted on data that is separate from both the base-classifier training data and the test data. Within each outer CV fold, further split the training partition into a base-training set (to train the \(K\) classifiers) and a validation set (to fit the combiner weights). The fused model is then evaluated on the outer test fold (Sec. 4.5).

15.4 Multi-Class Ensemble

Consider \(K\) base classifiers and \(C\ge 2\) classes. Each classifier \(k\) outputs either a hard label \(\hat {y}_k\in \{1,\ldots ,C\}\) or a probability vector \(\bp _k = (p_{k,1},\ldots ,p_{k,C})\) with \(p_{k,c}\ge 0\) and \(\sum _{c=1}^{C} p_{k,c} = 1\). The fused output is either a single label \(\hat {y}_{\text {fused}}\) or a probability vector \(\bp _{\text {fused}}\). The methods below are direct lifts of their binary counterparts; Table 15.2 summarizes them upfront.

15.4.1 Plurality Vote

Each classifier \(k\) produces a hard label \(\hat {y}_k\in \{1,\ldots ,C\}\). The fused prediction is

\begin{equation} \hat {y}_{\text {fused}} = \arg \max _{c\in \{1,\ldots ,C\}}\, \sum _{k=1}^{K} \indFunc [\hat {y}_k = c] \end{equation}

Condorcet’s theorem extends to multi-class with the threshold \(p > 1/C\): if each classifier is correct with probability \(p > 1/C\) and errors are independent, the plurality vote converges to the correct class as \(K\to \infty \).

15.4.2 Soft Vote

Each classifier \(k\) outputs \(\bp _k\). The fused probability vector is the elementwise mean:

\begin{equation} \bp _{\text {soft}} = \frac {1}{K}\sum _{k=1}^{K} \bp _k, \qquad \hat {y}_{\text {fused}} = \arg \max _{c} p_{\text {soft},c} \end{equation}

15.4.3 Linearly Weighted Combining

Assign non-negative weights \(w_k\) with \(\sum _k w_k = 1\). The fused probability vector and decision are

\begin{equation} \bp _{\text {fused}} = \sum _{k=1}^{K} w_k\, \bp _k, \qquad \hat {y}_{\text {fused}} = \arg \max _{c}\, p_{\text {fused},c} \end{equation}

A common choice is \(w_k = a_k / \sum _j a_j\) where \(a_k\) is the validation accuracy (or macro-\(F_1\)) of classifier \(k\).

15.4.4 Log-Probability Fusion

Under the assumption that the \(K\) classifiers are conditionally independent given the true class, the joint log-posterior factorizes as

\begin{equation} \log \Pr (y = c\mid \bx _1,\ldots ,\bx _K) \;\propto \; \log \pi _c + \sum _{k=1}^{K} \log p_{k,c} \end{equation}

where \(\pi _c = \Pr (y = c)\) is the class prior. Normalizing by the softmax across classes yields

\begin{equation} p_{\text {fused},c} = \frac {\pi _c \prod _{k=1}^{K} p_{k,c}}{\sum _{c'=1}^{C} \pi _{c'} \prod _{k=1}^{K} p_{k,c'}} \end{equation}

The fused decision is

\begin{equation} \hat {y}_{\text {fused}} = \arg \max _{c}\left [\log \pi _c + \sum _{k=1}^{K} \log p_{k,c}\right ] \end{equation}

When classes are balanced (\(\pi _c = 1/C\)), the prior term is a constant that drops out of the \(\arg \max \). The binary case (Sec. 15.3.5) is recovered by taking the log-ratio against class \(0\).

15.4.5 Learned Combiner (Multinomial Stacking)

The fused log-posterior is

\begin{equation} \log \Pr (y = c\mid \cdot ) \;\propto \; \beta _{0,c} + \sum _{k=1}^{K}\sum _{c'=1}^{C} \beta _{k,c,c'}\, \log p_{k,c'} \end{equation}

followed by a softmax across \(c\). Parameters \(\{\beta _{0,c}, \beta _{k,c,c'}\}\) are fitted on a held-out validation set and the stacked model is evaluated on a separate test set.

The log-probability fusion of Sec. 15.4.4 is the special case \(\beta _{0,c} = \log \pi _c\), \(\beta _{k,c,c} = 1\), and \(\beta _{k,c,c'} = 0\) for \(c'\ne c\). The learned coefficients allow the stacked model to down-weight redundant or poorly calibrated classifiers, and the off-diagonal terms (\(c'\ne c\)) capture systematic class confusions in individual base classifiers.

Non-linear combiners (Sec. 15.3.7) apply unchanged: replace the multinomial logistic head with any classifier that accepts the stacked probability vectors \((\bp _1,\ldots ,\bp _K)\) as input and produces a \(C\)-dimensional probability vector.

15.4.6 Hard versus Soft Outputs

The rules above produce a probability vector \(\bp _{\text {fused}}\), from which one can take \(\arg \max \) (top-1) or report the top-\(k\) predictions ranked by \(p_{\text {fused},c}\). If only hard labels \(\hat {y}_k\) are available from the base classifiers, only plurality vote (and a degenerate weighted vote where each classifier contributes a one-hot indicator) applies.

15.5 Sequential Combining

The ensemble methods in the previous section assume that all \(K\) classifier outputs (or measurements) are available simultaneously. In many practical settings, however, measurements arrive one at a time (streaming data), and a decision can/must be updated after each new observation.

15.5.1 Sequential Log-Odds Update

Consider a single classifier applied to a sequence of \(K\) independent noisy measurements \(\bx _1, \bx _2, \ldots , \bx _K\) of the same underlying quantity. After measurement \(\bx _t\), the classifier outputs the probability \(p_t = f_\bw (\bx _t)\).

In the batch LLR fusion, all measurements are combined at once:

\begin{equation} \text {logit}_{\text {fused}} = \sum _{k=1}^{K} \ln \frac {p_k}{1 - p_k} \end{equation}

The same result can be obtained sequentially. Define the accumulated log-odds after \(t\) measurements as

\begin{equation} L_t = L_{t-1} + \underbrace {\ln \frac {p_t}{1-p_t}}_{\text {logit}_t}, \quad L_0 = 0 \end{equation}

After each update, the fused probability is

\begin{equation} p_{\text {fused},t} = \sigma (L_t) = \frac {1}{1 + e^{-L_t}} \end{equation}

The sequential formulation produces identical results to the batch LLR fusion: \(L_K = \text {logit}_{\text {fused}}\). Its advantage is that the decision can be monitored and potentially made before all \(K\) measurements are collected.

A key advantage of the sequential formulation is that the decision can be made as soon as sufficient evidence has accumulated, without waiting for all \(K\) measurements.

Fixed-threshold decision At each step \(t\), compare the accumulated log-odds to a threshold:

\begin{equation} \hat {y}_t = \begin{cases} 1 & L_t \ge \tau _1\\ 0 & L_t \le \tau _0\\ \text {continue} & \tau _0 < L_t < \tau _1 \end {cases} \end{equation}

where \(\tau _1 > 0\) and \(\tau _0 < 0\) are decision thresholds. If \(L_t\) falls between the two thresholds, no decision is made and the next measurement is collected.

Confidence-based stopping Equivalently, the decision can be expressed in terms of the fused probability \(p_{\text {fused},t} = \sigma (L_t)\). Stop when the confidence exceeds a desired level \(\gamma \):

\begin{equation} \text {stop at time } t^* = \min \left \{t : p_{\text {fused},t} \ge \gamma \;\text { or }\; p_{\text {fused},t} \le 1-\gamma \right \} \end{equation}

where \(\gamma \in (0.5, 1)\) is the confidence threshold. For example, \(\gamma = 0.95\) means the decision is made when the fused probability exceeds \(95\%\) for either class.

15.5.2 Multiple Independent Sensors

The previous subsection considers repeated measurements from a single classifier. In many applications, \(N\) different independent sensors (or classifiers) observe the same underlying state, each producing its own probability estimate. In this case:

• 1. Since the sensors are independent, the log-odds combine by summation (LLR fusion).

• 2. Sequential update (Sec. 15.5.1) is performed with combined log-odds.

15.5.3 Connection to Bayesian Learning (*)

The sequential log-odds update is a special case of Bayesian belief updating.

Belief: A belief \(\pi (H_i)\) is a probability assigned to hypothesis \(H_i\), representing the agent’s confidence that \(H_i\) is the true state of the world. The belief vector satisfies \(\pi (H_i) \ge 0\) and \(\sum _i \pi (H_i) = 1\).

Consider a binary hypothesis set \(\{H_0, H_1\}\) (e.g., \(y=0\) vs. \(y=1\)) with prior beliefs \(\pi _{\text {old}}(H_0)\) and \(\pi _{\text {old}}(H_1)\). After observing new evidence \(x\), the posterior is updated using Bayes’ rule:

\begin{equation} \label {eq-bayes-belief-update} \pi _{\text {new}}(H_i) = \frac {\pi _{\text {old}}(H_i) \cdot L(H_i; x)}{\displaystyle \sum _{j} \pi _{\text {old}}(H_j) \cdot L(H_j; x)} \end{equation}

where \(L(H_i; x)\) is the likelihood of the observation \(x\) under hypothesis \(H_i\).

In the sequential setting, the posterior after observation \(x_t\) becomes the prior for observation \(x_{t+1}\). Taking the log-ratio of the two hypotheses after \(t\) observations yields

\begin{equation} \label {eq-log-odds-bayesian} \ln \frac {\pi (H_1)}{\pi (H_0)} = \underbrace {\ln \frac {\pi _0(H_1)}{\pi _0(H_0)}}_{\text {log-prior odds}} + \sum _{k=1}^{t} \underbrace {\ln \frac {L(H_1; x_k)}{L(H_0; x_k)}}_{\text {log-likelihood ratio}_k} \end{equation}

This is exactly the sequential log-odds update with \(L_0 = \ln \frac {\pi _0(H_1)}{\pi _0(H_0)}\) and \(\text {logit}_k = \ln \frac {L(H_1; x_k)}{L(H_0; x_k)}\).

When a probabilistic classifier outputs \(p_k = f_\bw (\bx _k)\), its logit \(\ln \frac {p_k}{1-p_k}\) serves as an estimate of the log-likelihood ratio. The sequential log-odds update therefore implements approximate Bayesian inference using classifier outputs as surrogate likelihood ratios.

15.6 Multiple Instance Learning (MIL)

In MIL, the \(i\)-th training example is a bag of \(n_i\) instances,

\begin{equation} B_i = \{\bx _{i,1},\, \bx _{i,2},\, \ldots ,\, \bx _{i,n_i}\}, \end{equation}

with a single observed bag label \(Y_i \in \{0,1\}\). Each instance has a latent label \(y_{i,j} \in \{0,1\}\) that is not available during training. The standard MIL assumption relates the two:

\begin{equation} \label {eq-mil-assumption} Y_i = \max _{j=1,\ldots ,n_i} y_{i,j}, \end{equation}

i.e., a bag is positive if and only if at least one of its instances is positive. This setting reduces annotation cost whenever labeling every instance is impractical (e.g., a whole-slide pathology image labeled only as “tumor/no tumor,” with the individual image patches left unlabeled).

Two learning strategies are common: (i) train an instance-level classifier \(f(\bx _{i,j}) = p_{i,j}\) and aggregate the per-instance probabilities into a bag probability \(P_i\); or (ii) train a bag-level classifier that maps the whole set \(B_i\) directly to \(P_i\). Both strategies ultimately depend on an aggregation rule

\begin{equation} P_i = g(p_{i,1},\, p_{i,2},\, \ldots ,\, p_{i,n_i}), \end{equation}

which plays the same role that the fusion rules of Sec. 15.3 play for ensembles of classifiers, except the index now runs over instances within one bag rather than classifiers on one sample. MIL is therefore ensemble aggregation applied along the instance axis.

The aggregated bag prediction \(P_i\) acts as the bridge between the micro-level instances and the macro-level bag. It represents the model’s overall confidence that the entire bag belongs to the positive class (\(Y_i = 1\)). Because the true labels of individual instances are hidden, the model must combine its per-instance predictions into this single collective score so the loss can be calculated against the only available ground truth: the bag label.

Four aggregation rules appear frequently:

\begin{equation} \begin{aligned} P_i^{\text {max}} &= \max _{j} p_{i,j}, \\[2pt] P_i^{\text {mean}} &= \tfrac {1}{n_i}\sum _{j=1}^{n_i} p_{i,j}, \\[2pt] P_i^{\text {noisy-OR}} &= 1 - \prod _{j=1}^{n_i} (1 - p_{i,j}), \\[2pt] P_i^{\text {att}} &= \sum _{j=1}^{n_i} \alpha _{i,j}\, p_{i,j}, \qquad \alpha _{i,j} = \frac {\exp (\bw ^\top \phi (\bx _{i,j}))}{\sum _{j'} \exp (\bw ^\top \phi (\bx _{i,j'}))}. \end {aligned} \end{equation}

Max-pooling is the direct probabilistic counterpart of Eq. (15.36). Mean-pooling is the instance-axis analog of soft vote combining (Sec. 15.3.3). Noisy-OR is the analog of LLR fusion (Sec. 15.3.5) under the assumption that instance predictions are conditionally independent. Attention pooling learns the mixing weights \(\alpha _{i,j}\) from the data and plays the role of a learned combiner (Sec. 15.3.7).

Because individual instance labels \(y_{i,j}\) are completely absent during training, a standard per-instance loss cannot be computed directly. Instead, training an instance-level classifier relies on one of three main paradigms:

End-to-End Backpropagation

The instance-level classifier (parameterized by \(\bw \)) and the aggregation function \(g\) are optimized simultaneously. Because instance-level labels are unknown, the loss is evaluated strictly at the bag level. The aggregated bag prediction \(P_i\) is compared to the true bag label \(Y_i\) using standard Binary Cross-Entropy (Sec. 8.4):

\begin{equation} \loss = - \left [ Y_i \log P_i + (1 - Y_i) \log (1 - P_i) \right ] \end{equation}

During backpropagation, the error signal must pass through the aggregation function \(g\) to reach the instance-level predictions \(p_{i,j}\). By the chain rule, \(g\) acts as a structural bottleneck that dictates exactly how the bag-level gradient is distributed among the individual instances. The choice of aggregation directly determines this gradient flow:

• • Max-pooling: The max operation acts as a hard mathematical switch. The gradient routes entirely to the single instance that yielded the maximum probability. The remaining \(n_i - 1\) instances in the bag receive a gradient of zero, meaning their parameters receive no learning signal from this forward pass.

• • Mean-pooling: The gradient is divided evenly by \(n_i\) and passed to all instances in the bag. Every instance is updated equally, which means noisy or non-informative instances receive the exact same learning signal as the truly predictive ones.

• • Attention-pooling: The gradient is scaled proportionally by the learned attention weights \(\alpha _{i,j}\). The model dynamically routes stronger learning signals to the instances it deems most relevant (high \(\alpha \)), while suppressing the updates for instances treated as background noise (low \(\alpha \)).

Label Inheritance

A heuristic approach where every instance inside a bag is simply assigned the bag’s label (\(y_{i,j} = Y_i\)). A standard binary classifier is then trained on the instances directly. This introduces significant noise into positive bags (since a positive bag may contain many negative instances), but it is computationally cheap and often used as a pre-training step to initialize weights.

Iterative Pseudo-Labeling

An alternating expectation-maximization (EM) style approach to refine instance predictions:

• 1. Train an initial classifier (often using Label Inheritance).

• 2. Use this classifier to score all instances in the training bags. Assign hard “pseudo-labels” based on these scores (e.g., in a positive bag, label the top-scoring instance as \(1\) and the rest as \(0\)).

• 3. Retrain the classifier using these new pseudo-labels, and repeat the process until convergence.