Machine Learning & Signals Learning

12 Classifier Comparison

12.1 Comparison Between Two Non-Probabilistic Classifiers

General methods concentrate on a performance of the particular classifier. In the following, dedicated inter-classifier methods are provided.

12.1.1 Contingency Table

Denote the four cells entries (Fig. 12.1) as:

• • \(n_{11}\) – both classifiers correct

• • \(n_{10}\) – C₁ correct, C₂ incorrect

• • \(n_{01}\) – C₁ incorrect, C₂ correct

• • \(n_{00}\) – both classifiers incorrect

with \(n_{11}+n_{10}+n_{01}+n_{00} = M\).

The accuracy of each classifier in terms of the contingency table entries is

\begin{equation} \text {Accuracy}_1 = \frac {n_{11}+n_{10}}{M}, \qquad \text {Accuracy}_2 = \frac {n_{11}+n_{01}}{M} \end{equation}

The diagonal cells (\(n_{11}\), \(n_{00}\)) represent samples on which both classifiers agree. The off-diagonal cells (\(n_{10}\), \(n_{01}\)) are the informative ones: they count samples where exactly one classifier succeeds and the other fails. If \(n_{10} \gg n_{01}\), classifier 1 is superior; if \(n_{01} \gg n_{10}\), classifier 2 is superior; if \(n_{10} \approx n_{01}\), they perform similarly despite possibly making errors on different samples.

12.1.2 Disagreement Measure

The disagreement measure is the ratio of instances on which the two classifiers produce different outcomes to the total number of instances:

\begin{equation} \text {Dis} = \frac {n_{10} + n_{01}}{M} \end{equation}

Range: \(\text {Dis}\in [0,\,1]\). \(\text {Dis}=0\) indicates that the classifiers always make the same predictions (zero diversity). As \(\text {Dis}\to 1\), the disagreement increases.

12.1.3 McNemar’s Test

The comparison matrix shows how two classifiers differ, but does not indicate whether the difference is due to chance. McNemar’s test addresses this by testing the null hypothesis:

\begin{equation} \begin{aligned} H_0\colon n_{10} = n_{01}\\ H_1\colon n_{10} \ne n_{01} \end {aligned} \end{equation}

i.e., the two classifiers have the same error rate, and any observed difference is due to sampling variability.

The test statistic is

\begin{equation} \chi ^2 = \frac {(n_{10} - n_{01})^2}{n_{10} + n_{01}} \end{equation}

The numerator measures the squared disagreement between the classifiers, and the denominator normalizes by the total number of disputed samples.

p-value interpretation The p-value is the probability of observing a test statistic, assuming \(H_0\) is true ¹. The decision is made by comparing the p-value to a predetermined significance level \(\alpha \), which is the maximum probability of incorrectly rejecting \(H_0\) (i.e., concluding the classifiers differ when they actually do not). Common choices are \(\alpha = 0.05\) (5%) and \(\alpha = 0.01\) (1%).

• • If \(p < \alpha \), we reject \(H_0\): the observed difference is statistically significant at level \(\alpha \).

• • If \(p \ge \alpha \), we fail to reject \(H_0\): we cannot conclude that one classifier is better.

12.1.4 Cohen’s Kappa Coefficient

McNemar’s test determines whether two classifiers differ significantly, but does not measure how much they agree. Cohen’s kappa coefficient \(\kappa \) addresses this by comparing observed agreement with the agreement expected under independent (random) predictions.

Using the contingency table notation (Fig. 12.1), define the observed agreement as the fraction of samples on which both classifiers give the same result (both correct or both incorrect):

\begin{equation} p_o = \frac {n_{11} + n_{00}}{M} \end{equation}

Recall that the accuracy of each classifier is

\begin{equation} \text {Accuracy}_1 = \frac {n_{11}+n_{10}}{M}, \qquad \text {Accuracy}_2 = \frac {n_{11}+n_{01}}{M} \end{equation}

If the two classifiers were independent, the probability that both are correct on the same sample is \(\text {Accuracy}_1 \cdot \text {Accuracy}_2\), and the probability that both are incorrect is \((1-\text {Accuracy}_1)(1-\text {Accuracy}_2)\). The expected agreement by chance is the sum of these two cases:

\begin{equation} p_e = \underbrace {\text {Accuracy}_1 \cdot \text {Accuracy}_2}_{\text {both correct}} + \underbrace {(1-\text {Accuracy}_1)(1-\text {Accuracy}_2)}_{\text {both incorrect}} \end{equation}

Cohen’s kappa is then

\begin{equation} \kappa = \frac {p_o - p_e}{1 - p_e} \end{equation}

Range: \(\kappa = 1\) indicates perfect agreement (the classifiers always agree). \(\kappa = 0\) indicates agreement no better than chance. \(\kappa < 0\) indicates agreement worse than chance (systematic disagreement).

12.1.5 Q-Statistic (Yule’s Q)

While Cohen’s \(\kappa \) measures the absolute level of agreement, the Q-statistic (Yule’s Q) measures the direction and strength of association between the classifiers’ outcomes. Using the contingency table notation (Fig. 12.1):

\begin{equation} Q = \frac {n_{11}\, n_{00} - n_{10}\, n_{01}}{n_{11}\, n_{00} + n_{10}\, n_{01}} \end{equation}

Range: \(Q\in [-1,\,1]\).

• • \(Q=1\): the classifiers are perfectly positively associated (whenever one is correct, so is the other).

• • \(0<Q<1\): the classifiers are positively correlated, meaning they tend to recognize the same instances correctly and incorrectly.

• • \(Q=0\): the classifiers are independent.

• • \(-1<Q<0\): the classifiers are negatively correlated, meaning they tend to make errors on different instances.

• • \(Q=-1\): perfect negative association (one is correct if and only if the other is incorrect).

12.1.6 Summary

12.2 Comparison Between Two Probabilistic Classifiers

The non-probabilistic methods (Sec. 12.1) reduce each classifier’s output to a hard label \(\hat {y}\in \{0,1\}\), discarding the confidence information. When classifiers produce probabilistic outputs \(p_k = f_{\bth _k}(\bx ) = \Pr (\hat {y}=1\mid \bx )\), richer comparisons are possible.

12.2.1 Paired Scoring Rule Comparison

Given a proper scoring rule \(S\) , such as the Brier score \(S_{\text {BS}}(p,y) = (p - y)^2\) (Sec. 10.5.3) or the CE loss \(S_{\text {LL}}(p,y) = -[y\log p + (1{-}y)\log (1{-}p)]\) (Sec. 8.4), denote the per-sample score \(S_{k,i} = S(p_{k,i},\, y_i)\) and compute the score difference:

\begin{equation} \delta _i = S_{1,i} - S_{2,i} \end{equation}

where \(p_{k,i} = f_{\bth _k}(\bx _i)\) is the probability predicted by classifier \(k\) for sample \(i\). If \(\delta _i > 0\), classifier 2 is better on that sample; if \(\delta _i < 0\), classifier 1 is better.

The average score difference summarizes the overall comparison:

\begin{equation} \bar {\delta } = \frac {1}{M}\sum _{i=1}^{M} \delta _i = \bar {S}_1 - \bar {S}_2 \end{equation}

where

\begin{equation} \bar {S}_k = \frac {1}{M}\sum _{i=1}^{M} S_{k,i} \end{equation}

is the mean score of classifier \(k\). A negative \(\bar {\delta }\) favors classifier 1; a positive \(\bar {\delta }\) favors classifier 2.

12.2.2 Paired t-test on Score Differences

The paired scoring rule comparison provides a descriptive measure \(\bar {\delta }\), but does not indicate whether the difference could have arisen by chance. A paired statistical test addresses this by testing:

• • \(H_0\): \(\E [\delta _i] = 0\) — the classifiers are equally good.

• • \(H_1\): \(\E [\delta _i] \neq 0\) — one classifier is systematically better.

The paired t-test on \(\{\delta _1, \ldots , \delta _M\}\) yields a test statistic:

\begin{equation} t = \frac {\bar {\delta }}{s_\delta / \sqrt {M}} \end{equation}

where \(s_\delta \) is the unbiased sample standard deviation:

\begin{equation} s_\delta = \sqrt {\frac {1}{M-1}\sum _{i=1}^{M}(\delta _i - \bar {\delta })^2} \end{equation}

If \(p < \alpha \), reject \(H_0\): the scoring difference is statistically significant.

12.2.3 Wilcoxon Signed-Rank Test on Score Differences

The Wilcoxon signed-rank test uses the ranks of \(|\delta _i|\) rather than their magnitudes, making it robust to outliers and non-normal distributions. The procedure is:

• 1. Discard any samples where \(\delta _i = 0\) (ties). Let \(M'\) denote the remaining count.

• 2. Rank the absolute differences \(|\delta _1|, \ldots , |\delta _{M'}|\) from smallest to largest. Assign average ranks to tied values.

• 3. Compute the signed-rank sums:

  \(\seteqnumber{0}{}{19}\)

  \begin{equation} W^+ = \sum _{\delta _i > 0} R_i, \qquad W^- = \sum _{\delta _i < 0} R_i \end{equation}

  where \(R_i\) is the rank of \(|\delta _i|\). The test statistic is \(W = \min (W^+, W^-)\).

The hypotheses are:

• • \(H_0\): the median of \(\delta _i\) is zero — the classifiers are equally good.

• • \(H_1\): the median of \(\delta _i\) is not zero — one classifier is systematically better.

Under \(H_0\), positive and negative differences are equally likely at each rank, so \(W^+ \approx W^-\). A very small \(W\) indicates that one sign dominates the high ranks, i.e., one classifier is consistently better on the samples with the largest differences. If \(p < \alpha \), reject \(H_0\).

12.2.4 DeLong’s Test for AUC Comparison

The paired scoring rule test compares calibration (how close probabilities are to the true labels). DeLong’s test addresses a complementary question: do the classifiers differ in their ability to rank positive samples above negative ones, as measured by the AUC?

Given two classifiers with \(\text {AUC}_1\) and \(\text {AUC}_2\) evaluated on the same test set, the hypotheses are:

• • \(H_0\): \(\text {AUC}_1 = \text {AUC}_2\).

• • \(H_1\): \(\text {AUC}_1 \neq \text {AUC}_2\).

The DeLong test statistic is:

\begin{equation} z = \frac {\text {AUC}_1 - \text {AUC}_2}{\sqrt {\text {Var}(\text {AUC}_1 - \text {AUC}_2)}} \end{equation}

Under \(H_0\), \(z\) is approximately standard normal. The variance in the denominator accounts for the correlation between \(\text {AUC}_1\) and \(\text {AUC}_2\) — since both are computed on the same test samples, their estimates are not independent². If \(p < \alpha \), the classifiers have significantly different discrimination.

12.2.5 Calibration Curve Comparison

The Brier score provides a single scalar summary of calibration quality. A calibration curve (reliability diagram) reveals how the calibration breaks down by comparing predicted probabilities to observed frequencies across bins.

Partition the predicted probabilities into \(B\) equal-width bins \(I_1,\ldots ,I_B\). For each bin \(b\), compute:

• • Average predicted probability: \(\;\bar {p}_b = \frac {1}{|I_b|}\sum _{i\in I_b} p_{k,i}\)

• • Observed frequency: \(\;\bar {y}_b = \frac {1}{|I_b|}\sum _{i\in I_b} y_i\)

A perfectly calibrated classifier satisfies \(\bar {y}_b = \bar {p}_b\) for all bins, corresponding to the diagonal line. Deviations above the diagonal indicate underconfidence; deviations below indicate overconfidence.

The Expected Calibration Error (ECE) summarizes the calibration curve as a single scalar:

\begin{equation} \text {ECE}_k = \sum _{b=1}^{B} \frac {|I_b|}{M} \left |\bar {y}_b - \bar {p}_b\right | \end{equation}

A lower ECE indicates better calibration. Comparing \(\text {ECE}_1\) and \(\text {ECE}_2\) complements the Brier score by isolating the calibration component from the refinement (sharpness) component.

12.2.6 Error Correlation

Yule’s Q (Sec. 12.1) measures the association between two classifiers’ binary correct/incorrect outcomes. For probabilistic classifiers, we can measure the same concept on the continuous per-sample scores \(S_{k,i}\): when classifier 1 has a large error, does classifier 2 also have a large error?

The error correlation is the Pearson correlation coefficient between the per-sample scores:

\begin{equation} r = \frac {\sum _{i=1}^{M}(S_{1,i} - \bar {S}_1)(S_{2,i} - \bar {S}_2)}{\sqrt {\sum _{i=1}^{M}(S_{1,i} - \bar {S}_1)^2}\;\sqrt {\sum _{i=1}^{M}(S_{2,i} - \bar {S}_2)^2}} \end{equation}

Range: \(r \in [-1, 1]\).

• • \(r \approx 1\): errors are positively correlated — both classifiers fail on the same samples. Combining them provides little benefit.

• • \(r \approx 0\): errors are uncorrelated — ensembles can potentially average out errors effectively.

• • \(r < 0\): errors are negatively correlated — the classifiers fail on different samples, which gives high potential for ensembles.

12.2.7 Spearman Rank Correlation of Scores

The Pearson error correlation (Sec. 12.2.6) measures the linear association between \(S_{1,i}\) and \(S_{2,i}\). When the relationship is monotonic but non-linear, or when scores contain outliers, the Spearman rank correlation provides a more robust measure.

Let \(R_{k,i}\) denote the rank of \(S_{k,i}\) among \(\{S_{k,1},\ldots ,S_{k,M}\}\) (with average ranks for ties). The Spearman rank correlation (Sec. 2.5) is the Pearson correlation computed on the ranks:

\begin{equation} r_s = \frac {\sum _{i=1}^{M}(R_{1,i} - \bar {R})(R_{2,i} - \bar {R})}{\sqrt {\sum _{i=1}^{M}(R_{1,i} - \bar {R})^2}\;\sqrt {\sum _{i=1}^{M}(R_{2,i} - \bar {R})^2}} \end{equation}

where \(\bar {R} = (M+1)/2\) is the mean rank.

Range: \(r_s \in [-1, 1]\). The interpretation mirrors the Pearson error correlation:

• • \(r_s \approx 1\): classifiers find the same samples difficult — limited ensemble benefit.

• • \(r_s \approx 0\): difficulty rankings are unrelated — good ensemble diversity.

• • \(r_s < 0\): when one classifier struggles, the other excels — high ensemble potential.

12.2.8 Summary