Machine Learning & Signals Learning

14 Classifier Comparison

14.1 Comparison Between Two Non-Probabilistic Classifiers

General methods concentrate on a performance of the particular classifier. In the following, dedicated inter-classifier methods are provided.

14.1.1 Summary

The following methods compare two classifiers by analyzing their hard label predictions on a shared test set. All are built on the four counts of the contingency table (Sec. 14.1.2).

14.1.2 Contingency Table

Denote the four cells entries (Fig. 14.1) as:

• • \(n_{11}\) – both classifiers correct

• • \(n_{10}\) – C₁ correct, C₂ incorrect

• • \(n_{01}\) – C₁ incorrect, C₂ correct

• • \(n_{00}\) – both classifiers incorrect

with \(n_{11}+n_{10}+n_{01}+n_{00} = M\).

The accuracy of each classifier in terms of the contingency table entries is

\begin{equation} \text {Accuracy}_1 = \frac {n_{11}+n_{10}}{M}, \qquad \text {Accuracy}_2 = \frac {n_{11}+n_{01}}{M} \end{equation}

The diagonal cells (\(n_{11}\), \(n_{00}\)) represent samples on which both classifiers agree. The off-diagonal cells (\(n_{10}\), \(n_{01}\)) are the informative ones: they count samples where exactly one classifier succeeds and the other fails. If \(n_{10} \gg n_{01}\), classifier 1 is superior; if \(n_{01} \gg n_{10}\), classifier 2 is superior; if \(n_{10} \approx n_{01}\), they perform similarly despite possibly making errors on different samples.

14.1.3 Disagreement Measure

The disagreement measure is the ratio of instances on which the two classifiers produce different outcomes to the total number of instances:

\begin{equation} \text {Dis} = \frac {n_{10} + n_{01}}{M} \end{equation}

Range: \(\text {Dis}\in [0,\,1]\). \(\text {Dis}=0\) indicates that the classifiers always make the same predictions (zero diversity). As \(\text {Dis}\to 1\), the disagreement increases.

14.1.4 McNemar’s Test

The comparison matrix shows how two classifiers differ, but does not indicate whether the difference is due to chance. McNemar’s test addresses this by testing the null hypothesis:

\begin{equation} \begin{aligned} H_0\colon n_{10} = n_{01}\\ H_1\colon n_{10} \ne n_{01} \end {aligned} \end{equation}

i.e., the two classifiers have the same error rate, and any observed difference is due to sampling variability.

Parametric and non-parametric forms McNemar’s is the parametric (asymptotic) form of the test: under \(H_0\) it is based on probabilistic assumptions. Its non-parametric (distribution-free) counterpart is a permutation test (Sec. 13.3.1).

p-value interpretation The p-value is the probability of observing a test statistic, assuming \(H_0\) is true ¹. The decision is made by comparing the p-value to a predetermined significance level \(\alpha \), which is the maximum probability of incorrectly rejecting \(H_0\) (i.e., concluding the classifiers differ when they actually do not). Common choices are \(\alpha = 0.05\) (5%) and \(\alpha = 0.01\) (1%).

• • If \(p < \alpha \), we reject \(H_0\): the observed difference is statistically significant at level \(\alpha \).

• • If \(p \ge \alpha \), we fail to reject \(H_0\): we cannot conclude that one classifier is better.

14.1.5 Cohen’s Kappa Coefficient

McNemar’s test determines whether two classifiers differ significantly, but does not measure how much they agree. Cohen’s kappa coefficient \(\kappa \) addresses this by comparing observed agreement with the agreement expected under independent (random) predictions.

Using the contingency table notation (Fig. 14.1), define the observed agreement as the fraction of samples on which both classifiers give the same result (both correct or both incorrect):

\begin{equation} p_o = \frac {n_{11} + n_{00}}{M} \end{equation}

Recall that the accuracy of each classifier is

\begin{equation} \text {Accuracy}_1 = \frac {n_{11}+n_{10}}{M}, \qquad \text {Accuracy}_2 = \frac {n_{11}+n_{01}}{M} \end{equation}

If the two classifiers were independent, the probability that both are correct on the same sample is \(\text {Accuracy}_1 \cdot \text {Accuracy}_2\), and the probability that both are incorrect is \((1-\text {Accuracy}_1)(1-\text {Accuracy}_2)\). The expected agreement by chance is the sum of these two cases:

\begin{equation} p_e = \underbrace {\text {Accuracy}_1 \cdot \text {Accuracy}_2}_{\text {both correct}} + \underbrace {(1-\text {Accuracy}_1)(1-\text {Accuracy}_2)}_{\text {both incorrect}} \end{equation}

Cohen’s kappa is then

\begin{equation} \kappa = \frac {p_o - p_e}{1 - p_e} \end{equation}

Range: \(\kappa = 1\) indicates perfect agreement (the classifiers always agree). \(\kappa = 0\) indicates agreement no better than chance. \(\kappa < 0\) indicates agreement worse than chance (systematic disagreement).

14.1.6 Q-Statistic (Yule’s Q)

While Cohen’s \(\kappa \) measures the absolute level of agreement, the Q-statistic (Yule’s Q) measures the direction and strength of association between the classifiers’ outcomes. Using the contingency table notation (Fig. 14.1):

\begin{equation} Q = \frac {n_{11}\, n_{00} - n_{10}\, n_{01}}{n_{11}\, n_{00} + n_{10}\, n_{01}} \end{equation}

Range: \(Q\in [-1,\,1]\).

• • \(Q=1\): the classifiers are perfectly positively associated (whenever one is correct, so is the other).

• • \(0<Q<1\): the classifiers are positively correlated, meaning they tend to recognize the same instances correctly and incorrectly.

• • \(Q=0\): the classifiers are independent.

• • \(-1<Q<0\): the classifiers are negatively correlated, meaning they tend to make errors on different instances.

• • \(Q=-1\): perfect negative association (one is correct if and only if the other is incorrect).

Summary of Non-Probabilistic Comparisons

For quick reference, Table 14.3 summarizes how to interpret each subrange of the four metrics.

The four methods above answer related but distinct questions. The contrasts between them clarify when each one is needed:

• • Disagreement (Sec. 14.1.3) vs. McNemar’s test (Sec. 14.1.4). Disagreement reports the raw fraction of samples on which the classifiers differ. McNemar adds a significance check: it asks whether the asymmetry between \(n_{10}\) and \(n_{01}\) is real or could plausibly arise by chance.

  • Example 14.2: Two classifier pairs (\(M=100\)) share the same disagreement rate \(d=0.30\) but differ sharply on McNemar.

    .
    		C1					C1
    		Cor	Inc				Cor	Inc
    C2	Cor	60	15		C2	Cor	60	3
    	Inc	15	10			Inc	27	10

    Symmetric (left): \(d=30/100=0.30\), \(\chi ^2=(15-15)^2/(15+15)=0\), \(p=1\). Asymmetric (right): \(d=0.30\), \(\chi ^2=(27-3)^2/(27+3)=19.2\), \(p\approx 10^{-5}\). The same raw disagreement leads to opposite conclusions about whether the accuracy gap is real.

• • McNemar’s test (Sec. 14.1.4) vs. Cohen’s \(\kappa \) (Sec. 14.1.5). McNemar tests whether the two classifiers have significantly different accuracy. Cohen’s \(\kappa \) measures how much they agree beyond chance. Two classifiers can pass McNemar (no significant accuracy gap) and still have low \(\kappa \) if their joint agreement is no better than independent guessing.

  • Example 14.3: A table with all four cells equal, \(M=100\).

    .
    		C1
    		Cor	Inc
    C2	Cor	25	25
    	Inc	25	25

    Both accuracies equal \(0.50\) and the off-diagonals match, so McNemar gives \(\chi ^2=0\), \(p=1\). But \(p_o=(25+25)/100=0.50\) and \(p_e=0.5^2+0.5^2=0.50\), hence \(\kappa =0\). Passing McNemar is compatible with chance-level agreement.

• • Cohen’s \(\kappa \) (Sec. 14.1.5) vs. Yule’s \(Q\) (Sec. 14.1.6). High \(Q\) with low \(\kappa \) indicates strongly correlated errors at different accuracy levels. \(Q\approx 0\) with moderate \(\kappa \) indicates classifiers that agree often but whose errors are uncorrelated, a desirable property for ensemble methods.

  • Example 14.4: Two highly accurate classifiers, \(M=100\).

    .
    		C1
    		Cor	Inc
    C2	Cor	95	2
    	Inc	2	1

    \(Q=(95\cdot 1-2\cdot 2)/(95\cdot 1+2\cdot 2)=91/99\approx 0.92\), a strong positive association. Yet Acc\({}_1=\)Acc\({}_2=0.97\) gives \(p_e=0.97^2+0.03^2=0.9418\), and with \(p_o=0.96\) we obtain \(\kappa \approx 0.31\), only fair agreement beyond chance. When accuracy is very high the chance baseline \(p_e\) is already close to \(1\), so \(\kappa \) shrinks even when \(Q\) reports strong association.

• • Disagreement (Sec. 14.1.3) vs. Yule’s \(Q\) (Sec. 14.1.6). Disagreement counts how many predictions differ. \(Q\) asks whether the errors line up on the same samples. Two classifiers can share an identical disagreement rate yet have very different \(Q\), depending on how their mistakes overlap.

  • Example 14.5: Two classifier pairs (\(M=100\)) with identical \(d=0.20\) but opposite \(Q\).

    .
    		C1					C1
    		Cor	Inc				Cor	Inc
    C2	Cor	80	10		C2	Cor	40	10
    	Inc	10	0			Inc	10	40

    Left: \(d=0.20\), \(Q=(80\cdot 0-10\cdot 10)/(80\cdot 0+10\cdot 10)=-1\). Both classifiers are accurate but their few errors never coincide, an ideal regime for ensembling. Right: \(d=0.20\), \(Q=(40\cdot 40-10\cdot 10)/(40\cdot 40+10\cdot 10)=1500/1700\approx 0.88\). The same off-diagonal mass now sits beside a large \(n_{00}\), so \(Q\) flips from perfect anti-correlation to strong positive association.

14.2 Comparison Between Two Probabilistic Classifiers

The non-probabilistic methods (Sec. 14.1) reduce each classifier’s output to a hard label \(\hat {y}\in \{0,1\}\), discarding the confidence information. When classifiers produce probabilistic outputs

\[{p_k = f_{\bth _k}(\bx ) = \Pr (\hat {y}=1\mid \bx )},\]

richer comparisons are possible.

14.2.1 Summary

The following methods in Table 14.4 exploit the full predicted probability \(p_i\in [0,1]\) rather than a hard label. They are grouped into three tasks: scoring accuracy (Brier score, calibration curve, paired score difference), significance testing (paired t-test, Wilcoxon, DeLong), and diversity assessment (error correlation, Spearman rank correlation).

14.2.2 Brier Score

AUC (Sec. 12.5) is scale-invariant: it is insensitive to the absolute predicted probability values \(\hat {y}_i\) and only evaluates whether positive samples receive higher probabilities than negative ones. The Brier score complements AUC by evaluating how close the predicted probabilities are to the actual outcomes. It is defined as the MSE between the predicted probability and the actual class label:

\begin{equation} \text {BS} = \frac {1}{M}\sum _{i=1}^{M}\left (p_i - y_i\right )^2 \end{equation}

where \(p_i\in [0,1]\) is the predicted probability and \(y_i\in \{0,1\}\) is the actual label.

Range and reference values: \(0 \le \text {BS} \le 1\), lower is better.

• • \(\text {BS}=0\): perfect classifier, assigns probability \(1\) to the correct class on every sample.

• • \(\text {BS}=1\): worst case, assigns probability \(1\) to the wrong class on every sample (confidently wrong).

• • \(\text {BS}=0.25\): constant prediction \(p_i=0.5\) for all samples, regardless of the label.

• • \(\text {BS}=\pi _1(1-\pi _1)\): base-rate classifier that always predicts the marginal positive rate \(p_i=\pi _1=\Pr (y=1)\) as a probability. This is the optimal constant-probability predictor (it minimizes expected Brier among all constants) and serves as the natural “no-information” probabilistic baseline: on a balanced set it equals \(0.25\); on an imbalanced set (\(\pi _1=0.1\)) it drops to \(0.09\), so a small Brier score on imbalanced data is not automatically impressive.

• • \(\text {BS}=\min (\pi _1,\,1-\pi _1)\): majority-vote classifier that always predicts the hard majority label (\(p_i=0\) if \(\pi _1<0.5\), else \(p_i=1\)). It is the accuracy-optimal constant classifier but is worse than the base-rate baseline as a probabilistic predictor: at \(\pi _1=0.1\) it gives \(0.10\) versus \(0.09\) for the base-rate classifier.

• • Skill score: \(\text {BSS}=1-\text {BS}/\text {BS}_{\text {ref}}\) with \(\text {BS}_{\text {ref}}=\pi _1(1-\pi _1)\) rescales the score so that \(\text {BSS}=1\) is perfect, \(\text {BSS}=0\) is no better than the base-rate baseline, and \(\text {BSS}<0\) is worse.

Comparison with AUC:

• • AUC measures discrimination: do positive samples receive higher predicted probabilities \(f_\bw (\bx _i)\) than negative ones? It is scale-invariant and threshold-invariant.

• • Brier score measures calibration: are the predicted probabilities \(p_i\) close to the actual labels \(y_i\)? It is sensitive to the absolute probability values.

A classifier can have high AUC (good discrimination) but poor Brier score (poorly calibrated probabilities), or vice versa.

Fig. 14.2 illustrates this distinction. Both classifiers achieve the same AUC (all positive samples are scored higher than all negative samples), but Classifier A predicts probabilities close to the actual labels (\(y_i=0\) or \(y_i=1\)), yielding a low Brier score. Classifier B compresses its predictions toward \(0.5\), degrading calibration and increasing the Brier score, while maintaining the same discrimination ability.

14.2.3 Calibration Curve

The Brier score (Sec. 14.2.2) summarizes calibration with a single number. The calibration curve (also called reliability diagram) shows where along the probability range the classifier is miscalibrated.

Construction. The construction is a probability-type histogram (Sec. 1.4) of the predicted probabilities \(\{p_i\}_{i=1}^M\), but with each bin reporting the empirical positive rate \(\bar {y}_b\) instead of the relative frequency \(n_i/N\) of eq. (1.15).

• 1. Partition the predicted probabilities \(\{p_i\}_{i=1}^M\) into \(B\) bins \(I_1,\dots ,I_B\) over \([0,1]\) (equal-width or equal-count/quantile).

• 2. For each bin \(b\), compute the mean predicted probability and the observed positive fraction,

  \(\seteqnumber{0}{}{13}\)

  \begin{equation} \bar {p}_b = \frac {1}{|I_b|}\sum _{i\in I_b} p_i,\qquad \bar {y}_b = \frac {1}{|I_b|}\sum _{i\in I_b} y_i \end{equation}

• 3. Plot \(\bar {y}_b\) versus \(\bar {p}_b\), together with the diagonal \(\bar {y}=\bar {p}\) as the reference for perfect calibration.

Interpretation.

• • Curve on the diagonal: well-calibrated. A predicted probability of \(0.7\) corresponds to a positive in \(70\%\) of such cases.

• • Curve below the diagonal (\(\bar {y}_b<\bar {p}_b\)): the classifier is over-confident, i.e. predicted probabilities exceed the realized frequencies.

• • Curve above the diagonal (\(\bar {y}_b>\bar {p}_b\)): the classifier is under-confident.

• • Deviations are common (e.g., naive Bayes, SVM with Platt scaling) and can be corrected by post-hoc methods such as Platt scaling or isotonic regression².

Expected Calibration Error. The calibration curve summarizes to a single scalar by weighting each bin’s deviation from the diagonal by its share of samples:

\begin{equation} \text {ECE}_k = \sum _{b=1}^{B} \frac {|I_b|}{M} \left |\bar {y}_b - \bar {p}_b\right | \end{equation}

A lower ECE indicates better calibration. Comparing \(\text {ECE}_1\) and \(\text {ECE}_2\) complements the Brier score by isolating the calibration component from the refinement (sharpness) component.

Relation to AUC and Brier score:

• • AUC (Sec. 12.5) is invariant to any monotonic rescaling of \(p_i\), so a classifier can have \(\mathsf {AUC}\approx 1\) while being severely miscalibrated.

• • Brier score gives a scalar summary of calibration plus refinement; the calibration curve shows the shape of the miscalibration across probability ranges, which is the actionable information for choosing a recalibration method.

14.2.4 Paired Scoring Rule Comparison

Given a proper scoring rule \(S\) , such as the Brier score \(S_{\text {BS}}(p,y) = (p - y)^2\) (Sec. 14.2.2) or the CE loss \(S_{\text {LL}}(p,y) = -[y\log p + (1{-}y)\log (1{-}p)]\) (Sec. 8.4), denote the per-sample score \(S_{k,i} = S(p_{k,i},\, y_i)\) and compute the score difference:

\begin{equation} \delta _i = S_{1,i} - S_{2,i} \end{equation}

where \(p_{k,i} = f_{\bth _k}(\bx _i)\) is the probability predicted by classifier \(k\) for sample \(i\). If \(\delta _i > 0\), classifier 2 is better on that sample; if \(\delta _i < 0\), classifier 1 is better.

The average score difference summarizes the overall comparison:

\begin{equation} \bar {\delta } = \frac {1}{M}\sum _{i=1}^{M} \delta _i = \bar {S}_1 - \bar {S}_2 \end{equation}

where

\begin{equation} \bar {S}_k = \frac {1}{M}\sum _{i=1}^{M} S_{k,i} \end{equation}

is the mean score of classifier \(k\). A negative \(\bar {\delta }\) favors classifier 1; a positive \(\bar {\delta }\) favors classifier 2.

14.2.5 Paired t-test on Score Differences

The paired scoring rule comparison provides a descriptive measure \(\bar {\delta }\), but does not indicate whether the difference could have arisen by chance. A paired statistical test addresses this by testing:

• • \(H_0\): \(\E [\delta _i] = 0\) — the classifiers are equally good.

• • \(H_1\): \(\E [\delta _i] \neq 0\) — one classifier is systematically better.

The paired t-test on \(\{\delta _1, \ldots , \delta _M\}\) yields a test statistic:

\begin{equation} t = \frac {\bar {\delta }}{s_\delta / \sqrt {M}} \end{equation}

where \(s_\delta \) is the unbiased sample standard deviation:

\begin{equation} s_\delta = \sqrt {\frac {1}{M-1}\sum _{i=1}^{M}(\delta _i - \bar {\delta })^2} \end{equation}

If \(p < \alpha \), reject \(H_0\): the scoring difference is statistically significant.

14.2.6 Wilcoxon Signed-Rank Test on Score Differences

The Wilcoxon signed-rank test uses the ranks of \(|\delta _i|\) rather than their magnitudes, making it robust to outliers and non-normal distributions. The procedure is:

• 1. Discard any samples where \(\delta _i = 0\) (ties). Let \(M'\) denote the remaining count.

• 2. Rank the absolute differences \(|\delta _1|, \ldots , |\delta _{M'}|\) from smallest to largest. Assign average ranks to tied values.

• 3. Compute the signed-rank sums:

  \(\seteqnumber{0}{}{21}\)

  \begin{equation} W^+ = \sum _{\delta _i > 0} R_i, \qquad W^- = \sum _{\delta _i < 0} R_i \end{equation}

  where \(R_i\) is the rank of \(|\delta _i|\). The test statistic is \(W = \min (W^+, W^-)\).

The hypotheses are:

• • \(H_0\): the median of \(\delta _i\) is zero — the classifiers are equally good.

• • \(H_1\): the median of \(\delta _i\) is not zero — one classifier is systematically better.

Under \(H_0\), positive and negative differences are equally likely at each rank, so \(W^+ \approx W^-\). A very small \(W\) indicates that one sign dominates the high ranks, i.e., one classifier is consistently better on the samples with the largest differences. If \(p < \alpha \), reject \(H_0\).

14.2.7 DeLong’s Test for AUC Comparison

The paired scoring rule test compares calibration (how close probabilities are to the true labels). DeLong’s test addresses a complementary question: do the classifiers differ in their ability to rank positive samples above negative ones, as measured by the AUC (Sec. 12.5)?

Given two classifiers with \(\text {AUC}_1\) and \(\text {AUC}_2\) evaluated on the same test set, the hypotheses are:

• • \(H_0\): \(\text {AUC}_1 = \text {AUC}_2\).

• • \(H_1\): \(\text {AUC}_1 \neq \text {AUC}_2\).

The DeLong test statistic is:

\begin{equation} z = \frac {\text {AUC}_1 - \text {AUC}_2}{\sqrt {\text {Var}(\text {AUC}_1 - \text {AUC}_2)}} \end{equation}

Under \(H_0\), \(z\) is approximately standard normal. The variance in the denominator accounts for the correlation between \(\text {AUC}_1\) and \(\text {AUC}_2\), since both are computed on the same test samples and their estimates are not independent³. If \(p < \alpha \), the classifiers have significantly different discrimination.

14.2.8 Error Correlation

Yule’s Q (Sec. 14.1) measures the association between two classifiers’ binary correct/incorrect outcomes. For probabilistic classifiers, we can measure the same concept on the continuous per-sample scores \(S_{k,i}\): when classifier 1 has a large error, does classifier 2 also have a large error?

The error correlation is the Pearson correlation coefficient between the per-sample scores:

\begin{equation} r = \frac {\sum _{i=1}^{M}(S_{1,i} - \bar {S}_1)(S_{2,i} - \bar {S}_2)}{\sqrt {\sum _{i=1}^{M}(S_{1,i} - \bar {S}_1)^2}\;\sqrt {\sum _{i=1}^{M}(S_{2,i} - \bar {S}_2)^2}} \end{equation}

Range: \(r \in [-1, 1]\).

• • \(r \approx 1\): errors are positively correlated — both classifiers fail on the same samples. Combining them provides little benefit.

• • \(r \approx 0\): errors are uncorrelated — ensembles can potentially average out errors effectively.

• • \(r < 0\): errors are negatively correlated — the classifiers fail on different samples, which gives high potential for ensembles.

14.2.9 Spearman Rank Correlation of Scores

The Pearson error correlation (Sec. 14.2.8) measures the linear association between \(S_{1,i}\) and \(S_{2,i}\). When the relationship is monotonic but non-linear, or when scores contain outliers, the Spearman rank correlation provides a more robust measure.

Let \(R_{k,i}\) denote the rank of \(S_{k,i}\) among \(\{S_{k,1},\ldots ,S_{k,M}\}\) (with average ranks for ties). The Spearman rank correlation (Sec. 2.5) is the Pearson correlation computed on the ranks:

\begin{equation} r_s = \frac {\sum _{i=1}^{M}(R_{1,i} - \bar {R})(R_{2,i} - \bar {R})}{\sqrt {\sum _{i=1}^{M}(R_{1,i} - \bar {R})^2}\;\sqrt {\sum _{i=1}^{M}(R_{2,i} - \bar {R})^2}} \end{equation}

where \(\bar {R} = (M+1)/2\) is the mean rank.

Range: \(r_s \in [-1, 1]\). The interpretation mirrors the Pearson error correlation:

• • \(r_s \approx 1\): classifiers find the same samples difficult — limited ensemble benefit.

• • \(r_s \approx 0\): difficulty rankings are unrelated — good ensemble diversity.

• • \(r_s < 0\): when one classifier struggles, the other excels — high ensemble potential.

Summary of Probabilistic Comparisons

For quick reference, Table 14.5 summarizes how to interpret each subrange of the probabilistic comparison metrics.

The methods above answer related but distinct questions. The contrasts between them clarify when each one is presented:

• • Brier score (Sec. 14.2.2) vs. ECE (Sec. 14.2.3). Brier mixes calibration and refinement into one scalar. ECE isolates the calibration component by bin, so it reveals where along the probability range miscalibration concentrates; two classifiers with similar Brier can have very different ECE shapes.

• • Paired score \(\bar {\delta }\) (Sec. 14.2.4) vs. paired t-test (Sec. 14.2.5). \(\bar {\delta }\) is the descriptive mean gap. The t-test asks whether that gap could plausibly arise by chance; a small \(|\bar {\delta }|\) with large per-sample variance can be non-significant, and conversely a tiny but consistent \(\bar {\delta }\) on a large \(M\) can be highly significant.

• • Paired t-test (Sec. 14.2.5) vs. Wilcoxon (Sec. 14.2.6). The t-test compares means under approximate normality of the \(\delta _i\). Wilcoxon ranks the magnitudes, so it is robust to heavy tails and outliers; the two can disagree when a few extreme \(\delta _i\) dominate the mean.

• • DeLong (Sec. 14.2.7) vs. paired t-test on Brier (Sec. 14.2.5). DeLong tests AUC, a ranking property invariant to monotone rescaling. The Brier-difference t-test tests calibrated accuracy. A classifier can keep DeLong’s verdict unchanged after a sharpening transform that destroys Brier (Fig. 14.3).

• • Pearson error correlation (Sec. 14.2.8) vs. Spearman rank correlation (Sec. 14.2.9). Pearson measures the linear association between per-sample error scores; Spearman uses only ranks. They diverge whenever the score-error relationship is monotonic but non-linear, or when a few large errors inflate the Pearson value.

• • Scoring accuracy (Brier, ECE, \(\bar {\delta }\)) vs. diversity (\(r\), \(r_s\)). The first family asks how good each classifier is individually; the second asks whether their failures overlap. A pair of strong but redundant classifiers (\(r \approx 1\)) gives less ensemble benefit than a pair of weaker but complementary ones (\(r \approx 0\)).

14.3 Comparison Between Two Multi-Class Classifiers

In the binary setting, the joint behaviour of two classifiers is captured by a \(2\times 2\) contingency table (Sec. 14.1.2). With \(K\ge 2\) classes, the natural object is a \(K\times K\) agreement matrix, whose entries count samples by the pair of predicted labels.

14.3.1 Summary

Table 14.6 maps each binary comparison method to its multi-class counterpart. Methods marked “same formula” need no modification; per-sample-score methods (Sec. 14.2.4, 14.2.5, 14.2.6, 14.2.8, 14.2.9) apply unchanged because they operate on scalar scores \(S_{k,i}\) that do not see the class count \(K\).

14.3.2 Agreement Matrix

Let \(\hat {y}_{k,i}\in \{1,\dots ,K\}\) denote the prediction of classifier \(k\) on sample \(i\). The agreement matrix \(N=[n_{jk}]\in \mathbb {N}^{K\times K}\) has entries

\begin{equation} n_{jk} = \sum _{i=1}^{M} \indFunc [\hat {y}_{1,i}=j]\,\indFunc [\hat {y}_{2,i}=k], \qquad \sum _{j,k} n_{jk} = M. \end{equation}

The diagonal entries \(n_{jj}\) count samples on which both classifiers predict class \(j\), and their sum \(\sum _{j=1}^{K} n_{jj}\) is the total number of samples on which the two classifiers issue the same label (whether correct or not). The off-diagonal entry \(n_{jk}\) with \(j\ne k\) counts samples where \(C_1\) predicts class \(j\) and \(C_2\) predicts class \(k\), revealing which class confusions drive the disagreement.

14.3.3 Disagreement Measure (Multi-Class)

The binary formula of Sec. 14.1.3 generalizes directly:

\begin{equation} \text {Dis} = \frac {1}{M}\sum _{i=1}^{M}\indFunc [\hat {y}_{1,i}\ne \hat {y}_{2,i}] = 1 - \frac {1}{M}\sum _{j=1}^{K} n_{jj}, \end{equation}

where \(n_{jj}\) are the diagonal entries of the agreement matrix (Sec. 14.3.2) that count the samples on which both classifiers predict the same class \(j\).

14.3.4 Bowker’s and Stuart-Maxwell Tests (*)

Unlike McNemar’s test (Sec. 14.1.4), which uses the correct/incorrect contingency table and answers “which classifier is more accurate,” Bowker and Stuart-Maxwell operate on the agreement matrix of Sec. 14.3.2, where ground truth \(y\) is absent. They cannot rank classifiers by accuracy; they detect whether the two classifiers behave systematically differently from each other. For multi-class accuracy comparison, apply McNemar unchanged to the correct/incorrect \(2\times 2\) collapse: each prediction is right or wrong regardless of \(K\).

The actionable outcomes are: (i) detect divergence between two models before labels are available (e.g. champion vs. challenger on live traffic); (ii) catch prior or threshold drift between two versions of a model (Stuart-Maxwell); (iii) localize which class pairs drive disagreement, by inspecting the per-pair Bowker contributions before summation (debugging); and (iv) decide whether an ensemble of the two should use symmetric (majority vote) or class-asymmetric weighting.

With \(K\) classes there are \(K(K-1)/2\) off-diagonal pairs, and the single binary equality \(n_{10}=n_{01}\) splits into two distinct, non-equivalent null hypotheses:

• • Cell-wise symmetry. Every individual pair is balanced: \(n_{jk}=n_{kj}\) for all \(j<k\). This asks whether each specific class-confusion pattern (e.g. “\(C_1\) predicts A while \(C_2\) predicts B”) occurs as often as its mirror (“\(C_1\) predicts B while \(C_2\) predicts A”). Bowker’s test addresses this null. Rejecting \(H_0\) means that at least one off-diagonal pair \((j,k)\) is directionally biased: one classifier systematically substitutes class \(j\) for class \(k\) more often than the other does the reverse, so the two classifiers do not make the same kinds of confusions in the same proportions.

• • Marginal homogeneity. For each class \(j\), the row marginal \(\sum _k n_{jk}\) (number of times \(C_1\) predicts class \(j\)) equals the column marginal \(\sum _k n_{kj}\) (number of times \(C_2\) predicts class \(j\)). This asks only whether the two classifiers issue each label at the same overall rate, without constraining which pairs of classes are confused: an excess of “\(C_1\) says A, \(C_2\) says B” can be cancelled out in the A-marginal by an excess of “\(C_1\) says C, \(C_2\) says A”, leaving the marginals balanced while individual off-diagonal pairs are not. Stuart-Maxwell’s test addresses this null. Rejecting \(H_0\) means that at least one class is predicted at a different overall rate by the two classifiers, indicating an aggregate bias in class usage (e.g. \(C_1\) predicts class A more often than \(C_2\) does, regardless of which other class \(C_2\) substitutes in its place).

The two coincide at \(K=2\) (where each marginal equation is equivalent to the single off-diagonal equality), but for \(K\ge 3\) cell-wise symmetry implies marginal homogeneity and not vice versa: asymmetries on different off-diagonal pairs can cancel in the marginals.

Bowker’s test of symmetry. \(H_0\colon n_{jk}=n_{kj}\) for every \(j<k\), i.e. the agreement matrix is symmetric about its diagonal:

\begin{equation} \chi ^2_{\text {B}} = \sum _{1\le j<k\le K} \frac {(n_{jk}-n_{kj})^2}{n_{jk}+n_{kj}}, \qquad \text {df} = \frac {K(K-1)}{2}. \end{equation}

The degrees of freedom \(K(K-1)/2\) count the number of off-diagonal pairs above the diagonal, i.e. the number of distinct equalities \(n_{jk}=n_{kj}\) being tested; it is the parameter passed to the \(\chi ^2\) table (or to scipy.stats.chi2.sf) to obtain the p-value. At \(K=2\) this collapses to McNemar’s statistic (Sec. 14.1.4)⁴.

Stuart-Maxwell test of marginal homogeneity. A weaker null only requires that the row marginals match the column marginals:

\begin{equation} H_0\colon \sum _{k=1}^{K} n_{jk} = \sum _{k=1}^{K} n_{kj} \quad \text {for all } j. \end{equation}

The test statistic \(\chi ^2_{\text {SM}}\) combines the \(K\) marginal differences into a single number using a covariance correction (one marginal is redundant because the totals must match), and is referred to a \(\chi ^2\) distribution with df\({}=K-1\). Standard implementations such as SquareTable.homogeneity compute it directly from \(N\). The p-value is interpreted as in Sec. 14.1.4: reject \(H_0\) when \(p<\alpha \).

14.3.5 Omnibus Permutation Test (*)

Just as the permutation test is the non-parametric counterpart of McNemar (Sec. 14.1.4), the omnibus permutation test is the non-parametric counterpart of Bowker and Stuart-Maxwell (Sec. 14.3.4).

The test summarizes the asymmetry of the agreement matrix (Sec. 14.3.2) with a single scalar, for example Bowker’s itself or the total off-diagonal asymmetry

\begin{equation} S = \sum _{1\le j<k\le K} \lvert n_{jk}-n_{kj}\rvert , \end{equation}

used directly as a discrepancy score rather than referred to a reference distribution. The null distribution is built under exchangeability of the two classifiers: for each disagreeing sample \(i\) (where \(\hat {y}_{1,i}\ne \hat {y}_{2,i}\)), the ordered prediction pair is swapped, \((\hat {y}_{1,i},\hat {y}_{2,i})\to (\hat {y}_{2,i},\hat {y}_{1,i})\), with probability \(1/2\). The agreement matrix is rebuilt, the statistic recomputed, and the procedure repeated \(T\) times (Sec. 13.3.1) to obtain the null distribution and the \(\frac {1}{T+1}\) p-value. Swapping only disagreeing samples leaves the diagonal \(n_{jj}\) fixed, mirroring how the permutation form of McNemar reassigns only the discordant pairs.

14.3.6 Cohen’s Kappa (Multi-Class)

The definition of Sec. 14.1.5 extends without change; only the ingredients are computed on the \(K\times K\) agreement matrix. The observed agreement is the diagonal mass:

\begin{equation} p_o = \frac {1}{M}\sum _{j=1}^{K} n_{jj}. \end{equation}

Define the empirical class-prediction frequency of each classifier from the marginals:

\begin{equation} p_{1,j} = \frac {1}{M}\sum _{k=1}^{K} n_{jk}, \qquad p_{2,j} = \frac {1}{M}\sum _{k=1}^{K} n_{kj}. \end{equation}

Under independence, the probability that both classifiers issue the same class \(j\) is \(p_{1,j}p_{2,j}\), so the expected chance agreement is

\begin{equation} p_e = \sum _{j=1}^{K} p_{1,j}\,p_{2,j}, \end{equation}

and Cohen’s kappa is \(\kappa =(p_o-p_e)/(1-p_e)\).

Range: \(\kappa = 1\) indicates perfect agreement, \(\kappa = 0\) indicates agreement no better than chance, and \(\kappa < 0\) indicates systematic disagreement. The interpretation is identical to the binary case (Sec. 14.1.5).

14.3.7 Multi-Class Brier Score

With one-hot labels \(\by _i\in \{0,1\}^K\) (where \(y_{i,k}=1\) iff sample \(i\) belongs to class \(k\)), the multi-class Brier score is the mean squared \(L_2\) distance between \(\bp _i\) and \(\by _i\):

\begin{equation} \text {BS} = \frac {1}{M}\sum _{i=1}^{M}\sum _{k=1}^{K}\left (p_{i,k}-y_{i,k}\right )^2. \end{equation}

Range: \(0 \le \text {BS} \le 2\). A perfect classifier (\(\bp _i=\by _i\)) achieves \(\text {BS}=0\). A uniform predictor (\(p_{i,k}=1/K\)) achieves \(\text {BS}=1-1/K\).

14.3.8 Multi-Class Calibration: Top-Label and Classwise ECE

The binary ECE bins predictions by their probability and compares the bin mean to the empirical positive rate. With \(K\) classes, two standard reductions to the binary machinery are used.

Top-label ECE Each sample is reduced to its winning probability and predicted class,

\begin{equation} \hat {p}_i = \max _{k} p_{i,k}, \qquad \hat {y}_i = \argmax _k p_{i,k}. \end{equation}

Bin the values \(\{\hat {p}_i\}_{i=1}^{M}\) as in Sec. 14.2.3. In each bin \(b\), compare the mean confidence \(\bar {\hat {p}}_b\) to the bin accuracy \(\bar {a}_b = (1/|I_b|)\sum _{i\in I_b}\indFunc [\hat {y}_i=y_i]\):

\begin{equation} \text {ECE}_{\text {top}} = \sum _{b=1}^{B}\frac {|I_b|}{M}\,\left |\bar {a}_b - \bar {\hat {p}}_b\right |. \end{equation}

This is the metric most commonly reported in the literature (e.g. for deep network calibration).

Classwise (marginal) ECE Treat each class as its own binary calibration problem with \((p_{i,k},\,\indFunc [y_i=k])\) and average:

\begin{equation} \text {ECE}_{\text {cw}} = \frac {1}{K}\sum _{k=1}^{K}\text {ECE}^{(k)}, \qquad \text {ECE}^{(k)} = \sum _{b=1}^{B}\frac {|I_b^{(k)}|}{M}\,\left |\bar {y}^{(k)}_b - \bar {p}^{(k)}_b\right |. \end{equation}

Both reductions produce reliability diagrams of the same shape as Fig. 14.3: one curve per class for classwise, or a single curve in confidence space for top-label.

14.3.9 Multi-Class AUC and DeLong

With \(K\) classes, the ROC curve is not unique. Two standard aggregations summarize discrimination in a single scalar.

One-vs-Rest macro-AUC. For each class \(k\) compute the binary AUC of \(\{(p_{i,k},\indFunc [y_i=k])\}_i\), then average:

\begin{equation} \text {AUC}_{\text {OvR}} = \frac {1}{K}\sum _{k=1}^{K}\text {AUC}^{(k)}. \end{equation}

One-vs-One AUC. For every ordered pair of classes \((j,k)\) with \(j\ne k\), compute the AUC restricted to samples whose true label is \(j\) or \(k\), using \(p_{i,k}\) as the score, and average over the \(\binom {K}{2}\) unordered pairs (Hand and Till’s \(M\)-measure).

DeLong for multi-class. DeLong’s test (Sec. 14.2.7) gives a paired variance estimate for the difference between two AUCs. In the multi-class setting it is applied per class on the OvR scores, yielding \(K\) p-values that can be combined (e.g. Bonferroni correction across classes), or on \(\text {AUC}_{\text {OvR}}\) directly using the per-class DeLong covariances aggregated into the macro variance. Standard implementations (sklearn.metrics.roc_auc_score with multi_class=’ovr’ or ’ovo’) compute the point estimates; dedicated libraries⁵ provide the DeLong variance.

14.3.10 Per-Sample-Score Methods (Carry Over Unchanged)

A comparison method is \(K\)-aware only if its formula directly indexes a class label or a class-conditional probability. Methods that reduce each sample to a single scalar \(S_{k,i}\) before doing anything else are \(K\)-agnostic: once you have a column of scalars per classifier, the test code is the same whether the scalars came from a binary or a multi-class problem.

The scalar score. For probabilistic classifiers the natural per-sample score is either the multi-class Brier (Sec. 14.3.7) or the cross-entropy loss (Sec. 8.4):

\begin{equation} S_{k,i} = \sum _{c=1}^{K}\left (p_{k,i,c}-y_{i,c}\right )^2 \qquad \text {or}\qquad S_{k,i} = -\sum _{c=1}^{K} y_{i,c}\,\log p_{k,i,c} \end{equation}

where \(y_{i,c}\in \{0,1\}\) is the one-hot label and \(p_{k,i,c}\) is classifier \(k\)’s probability for class \(c\) on sample \(i\). Both reduce to their binary forms when \(K=2\).

What each method answers and what it sees.

• • Paired score difference (Sec. 14.2.4). Goal: quantify which classifier produces better-calibrated probabilities, sample by sample. Input: the per-sample gap \(\delta _i=S_{1,i}-S_{2,i}\); the sign of \(\bar \delta \) picks the winner.

• • Paired t-test (Sec. 14.2.5). Goal: test whether the score difference is statistically significant, assuming approximately normal \(\delta _i\). Input: the vector \(\{\delta _i\}_{i=1}^{M}\); tests \(\E [\delta _i]=0\).

• • Wilcoxon signed-rank (Sec. 14.2.6). Goal: non-parametric significance test on the median score difference, robust to outliers and non-normality. Input: ranks of \(|\delta _i|\) with their signs.

• • Pearson error correlation (Sec. 14.2.8). Goal: measure how correlated the two classifiers’ errors are (probabilistic analog of Yule’s \(Q\)); low correlation signals ensemble potential. Input: the two columns \(\{S_{1,i}\},\{S_{2,i}\}\); returns \(r\in [-1,1]\).

• • Spearman rank correlation (Sec. 14.2.9). Goal: non-parametric error-correlation diagnostic; test whether the two classifiers rank samples by difficulty in the same order. Input: ranks of those columns; returns \(r_s\in [-1,1]\).

None of these formulas index a class. The class count \(K\) is absorbed into \(S\) at the score-construction step and is invisible from then on.

What is gained and what is lost. The same library calls (scipy.stats.ttest_rel, wilcoxon, pearsonr, spearmanr) work for any classification task; only the score function changes. The price is that these tests answer only “do the classifiers’ overall score distributions differ?” and not “on which class does one classifier do worse?” For per-class diagnosis use the classwise ECE (Sec. 14.3.8) or per-class DeLong (Sec. 14.3.9), both of which loop over \(K\) explicitly.

14.3.11 Ensemble Recommendation

The diagnostics produce four checkpoints. The recommendation is conservative: an ensemble is justified only when all the following conditions hold. Otherwise, deploy the better single classifier.

Both classifiers are individually useful Verify that each classifier outperforms the trivial baseline (constant prediction at the majority class). Then run McNemar (Sec. 14.1.4, applied to the correct/incorrect \(2\times 2\) for \(K>2\)). If the test rejects and the accuracy gap is large (rule of thumb: \(>5\) percentage points), drop the weaker classifier; ensembling rarely recovers from a wide accuracy gap because the better model’s votes are diluted by the weaker one.

The errors are not redundant This is the critical condition: ensembling helps only when the two classifiers fail on different samples.

• • Hard predictions: Yule’s \(Q\) on the correct/incorrect \(2\times 2\) (Sec. 14.1.6; for \(K>2\), apply to the collapse). \(Q\ge 0.8\) signals highly correlated errors, so ensembling will produce marginal gains at best.

• • Probabilistic outputs: Pearson error correlation \(r\) (Sec. 14.2.8) or Spearman \(r_s\) (Sec. 14.2.9). \(r\ge 0.8\) is the same red flag; \(r\le 0\) is the ideal case (negatively correlated errors).

If both diagnostics indicate near-perfect correlation, stop: the two classifiers are effectively the same model and an ensemble will not help.

Choose symmetric vs. asymmetric fusion With diversity confirmed, the choice of fusion rule depends on the shape of disagreement:

• • Bowker (Sec. 14.3.4) fails to reject: disagreements are symmetric across class pairs. Use symmetric fusion (majority vote, soft vote / score averaging, Sec. 15.3).

• • Bowker rejects: at least one class pair is asymmetrically confused. Use asymmetric fusion: weighted averaging (weight by per-class accuracy) or class-conditional routing (send disputed-pair samples to the classifier known to be stronger on that boundary).

Calibration before soft averaging (optional) Soft vote and probability averaging assume the inputs are comparable. Before averaging probabilities, check the calibration of both classifiers (Sec. 14.2.3 for binary, Sec. 14.3.8 for multi-class). If either is poorly calibrated, recalibrate first (Platt scaling, isotonic regression) or fall back to hard voting, which is unaffected by calibration.