Machine Learning & Signals Learning

9 Classification Performance Metrics

Definitions:

• • \(\by \) - target values vector of the test database, \(\by \in \real ^M\)

• • \(\hat {\by }\) - predicted value, \(\hat {\by }\in \real ^M\), output of some classifier \(\hat {\by } = f_\bth (\bX )\).

Typically, in binary classification, \(y_i\in \left \{0,1\right \}\).

9.1 Definitions

Basic terminology:

• • ‘1’ – positive group or result

• • ‘0’ – negative group or result

• • \(Y\) – actual class

• • \(\hat {Y}\) – predicted class

Positive/negative terminology is rather arbitrary. Typically, the result of interest is termed positive.

9.2 Confusion matrix

The summarization is in the for m of a 2D non-normalized histogram of \((Y,\hat {Y})\).

The (test) database has \(M\) values, among them:

• • TP + FN positive values

• • FP + TN negative values

This is the most common way to summarize the performance of a particular classifier on a particular dataset. It can be easily extended for multi-class classifiers.

9.3 Performance Metrics

9.3.1 Accuracy

\begin{equation} \begin{aligned} \text {Accuracy} &= \frac {\text {correct predictions}}{\text {total predictions}}\\[3pt] &= \frac {TP + TN}{TP + TN + FP + FN} \end {aligned} \end{equation}

In the example, FN=67 is a bad performance, and FP=0 is probably something good. However, accuracy does not reflect the discrepancy between these two. Additional metrics are used to quantify these aspects.

9.3.2 Precision

From the probability theory,

\begin{align} \Pr (Y=1|\hat {Y}=1) &= \frac {\Pr (Y=1,\hat {Y}=1)}{\Pr (\hat {Y}=1)} \\ \Pr (\hat {Y}=1) &= \Pr (\hat {Y}=1,Y=0) + \Pr (\hat {Y}=1,Y=1) \end{align}

\begin{equation} \text {Precision} = \dfrac {TP}{FP+TP} = \dfrac {\text {Correctly predicted 1's}}{\text {All predicted 1's}} \end{equation}

TP = Correctly predicted 1’s
TP + FP = All predicted 1’s

9.3.3 Recall (sensitivity)

From the probability theory,

\begin{align} \Pr (\hat {Y}=1|Y=1) &= \frac {\Pr (Y=1,\hat {Y}=1)}{\Pr (Y=1)} \\ \Pr (Y=1) &= \Pr (Y=1,\hat {Y}=0) + \Pr (Y=1,\hat {Y}=1) \end{align}

\begin{equation} \text {Recall} = \frac {TP}{TP + FN} = \frac {\text {Correctly predicted 1's}}{\text {Actual 1's}} \end{equation}

Medical meaning: portion of correctly classified ill among all the ill.

9.3.4 Specificity

\begin{equation} \text {Specificity} = \frac {TN}{FP+TN} = \frac {\text {Correctly predicted 0's}} {\text {Actual 0's}} \end{equation}

Medical meaning: portion of classified healthy among all the healthy.

9.3.5 F₁-score

The harmonic mean between precision and recall,

\begin{equation} F_1 = \frac {2}{\frac {1}{recall} + \frac {1}{precision}} = \frac {TP}{TP + \frac {1}{2}\left (FP+FN\right )} \end{equation}

9.3.6 Per-class performance

The metrics above are defined with respect to the positive class (\(Y=1\)). To assess both classes, the metrics can be computed separately for each class by treating it as the positive class.

For class \(c\), define:

• • \(TP_c\) – samples of class \(c\) correctly predicted as class \(c\)

• • \(FP_c\) – samples of other classes incorrectly predicted as class \(c\)

• • \(FN_c\) – samples of class \(c\) incorrectly predicted as another class

The per-class metrics are then

\begin{equation} \text {Precision}_c = \frac {TP_c}{TP_c + FP_c}, \quad \text {Recall}_c = \frac {TP_c}{TP_c + FN_c}, \quad F_{1,c} = \frac {2\cdot \text {Precision}_c\cdot \text {Recall}_c}{\text {Precision}_c + \text {Recall}_c} \end{equation}

Note the symmetry: for binary classification, \(FP_0 = FN_1\) and \(FP_1 = FN_0\), so precision for one class equals recall for the other only when \(FP_0 = FP_1\).

9.4 Imbalanced Dataset

Imbalanced dataset: Dataset with significant differences between the numbers of labels of each class. The following examples present a few problems related to imbalanced datasets.

Majority classifier

Majority classifier is a classifier that always predicts the most frequent class in the dataset, as in Example 9.3.

Despite achieving high accuracy on imbalanced data, a majority classifier has zero recall and zero \(F_1\) for the minority class. It is commonly used as a baseline: any useful classifier should outperform it on metrics beyond accuracy.

Small dataset problem

In imbalanced datasets, the minority class may contain very few samples. Performance metrics computed on small subsets are subject to high sampling variability, making them unreliable estimates of the true classifier performance.

This is a problem of confidence in performance evaluation. While confidence interval analysis is out of the scope of this document, the key takeaway is: metrics computed on small subsets can deviate significantly from the true performance and should be interpreted with caution.

Anomaly detection Anomaly detection is a sub-field of classification where the class of interest (the anomaly) is extremely rare compared to the normal class. Examples include fraud detection, network intrusion detection, and equipment failure prediction. The extreme class imbalance (e.g., 1:1 000) makes accuracy meaningless and amplifies the small dataset problem for the minority class. Precision and recall are the primary evaluation metrics in this setting.

9.5 Multi-class performance

9.5.1 Categorical Encoding

Label encoding¹ assigns each category an integer. For a feature with categories \(\{A, B, C\}\), label encoding maps \(A \mapsto 0\), \(B \mapsto 1\), \(C \mapsto 2\). This is appropriate for ordinal features where the order is meaningful (e.g., low, medium, high).

One-hot encoding² creates a binary indicator column for each category. A feature with \(C\) categories is replaced by \(C\) binary features, increasing the dimensionality from \(N\) to \(N + C - 1\).

Pros:

• • One-hot encoding avoids imposing artificial ordinal relationships between categories.

• • Label encoding is memory-efficient and preserves dimensionality.

Cons:

• • One-hot encoding increases dimensionality significantly for high-cardinality features.

• • Label encoding introduces a spurious ordinal relationship for nominal features.

9.5.2 Performance

For multi-class classification, confusion matrices, accuracy and per-class performance (Sec. 9.3.6) are used.

To get a single number, two averaging strategies are applied.

Macro-averaging

Averaging with equal weight to each class, regardless of the number of instances.

Micro-averaging

Equal weight to each instance, regardless of the class label and the number of instance in the class. All TP, FP, etc. are summed across all the classes.

Further reading: Accuracy, precision, and recall in multi-class classification

9.6 Decision threshold

With a probabilistic classifier, the output is the estimated probability of the positive class,

\begin{equation} \Pr (\hat {y}=1\mid \bx ) = f_\bth (\bx ) \end{equation}

The binary decision is obtained by comparing \(f_\bth (\bx )\) with a threshold \(\mathsf {thr}\):

\begin{equation} \hat {y} = \begin{cases} 1 & f_\bth (\bx ) \ge \mathsf {thr}\\ 0 & f_\bth (\bx ) < \mathsf {thr} \end {cases} \end{equation}

The default threshold is \(\mathsf {thr}=0.5\). Changing \(\mathsf {thr}\) shifts the trade-off between TP, FP, FN, and TN, and therefore changes all derived metrics (precision, recall, specificity, \(F_1\)).

9.6.1 Receiver Operating Characteristics (RoC)

The RoC curve plots the following two quantities as \(\mathsf {thr}\) varies from \(0\) to \(1\):

• • True Positive Rate (TPR): synonym for recall,

  \(\seteqnumber{0}{}{19}\)

  \begin{equation} TPR = \frac {TP}{TP+FN} = \text {recall} \end{equation}

• • False Positive Rate (FPR): complement of specificity,

  \(\seteqnumber{0}{}{20}\)

  \begin{equation} FPR = \frac {FP}{FP+TN} = 1 - \text {specificity} \end{equation}

As \(\mathsf {thr}\to 0\), everything is classified as positive (\(TPR\to 1\), \(FPR\to 1\)). As \(\mathsf {thr}\to 1\), everything is classified as negative (\(TPR\to 0\), \(FPR\to 0\)).

9.6.2 Area under curve (AUC)

AUC: AUC is the area under the RoC curve.

Range: A random (coin-toss) classifier has \(\mathsf {AUC}=0.5\) and an ideal classifier has \(\mathsf {AUC}=1\). All practical classifiers fall in the range \(0.5 \le \mathsf {AUC} \le 1\).

The relationship between classifier quality and the RoC curve is illustrated in Fig. 9.2. When the two class distributions (positive and negative) are well separated, the classifier achieves high TPR at low FPR, producing a RoC curve that hugs the top-left corner (\(\mathsf {AUC}\to 1\)). As the distributions overlap, the FN and FP regions grow, and the RoC curve shifts toward the diagonal (\(\mathsf {AUC}\to 0.5\)).

The choice of threshold \(\mathsf {thr}\) controls which part of the overlap region is assigned to each class. Lowering \(\mathsf {thr}\) classifies more samples as positive, increasing TP but also FP. Raising \(\mathsf {thr}\) increases TN but also FN. Fig. 9.3 illustrates this trade-off: moving the threshold left or right redistributes errors between FN and FP while tracing the RoC curve.

Advantages:

• • Scale-invariant: AUC measures how well predictions are ranked, rather than their absolute values.

• • Threshold-invariant: AUC summarizes performance across all thresholds, without requiring a specific threshold choice.

Limitations:

• • Scale invariance may be undesirable when well-calibrated probabilities are needed (e.g., for risk assessment), since AUC is insensitive to the predicted probability values.

• • Threshold invariance may be undesirable when the application requires a specific trade-off between false negatives and false positives. For example, in spam detection, minimizing false positives (legitimate email marked as spam) is more important than minimizing false negatives. AUC does not capture such asymmetric cost preferences.