Machine Learning & Signals Learning

6 Regression Losses and Metrics

For regression, loss and metrics are often the same quantity.

6.1 Preface

The predicted range of \(\hat {y}_i=f(\bx _i)\) results from the particular applied model.

Metric A performance metric is a real-valued function \(J:\by ,\hat {\by }\rightarrow \mathscr {R}\) that is used to evaluate and quantify the effectiveness of a model. These metrics provide insight into how well a model is performing according to various aspects of the data it predicts.

Loss function The loss function is a metric of the form \(L:\by ,\hat {\by }\rightarrow \mathscr {R}\) that is calculated over the test set, such that optimal parameters that correspond to the minimum loss

\begin{equation} \bw = \arg \min _{\bw } \loss (\hat {\by },\by ) \end{equation}

can be evaluated, e.g., by GD (Sec. 3.4).

For example cross-entropy loss in classification is not a metric and R-square is not a loss.

Summary Metrics are for communication; losses are for training. A loss is the objective minimized during training, while a metric is reported to summarize performance. They may coincide (e.g., MSE as both loss and metric), but need not.

6.2 Loss Function Properties

A loss function has a few desired properties that are presented below. While not obligatory, these properties may significantly ease the evaluation of the parameters \(\bw \). In the following, only the basic description of these properties is provided, sacrificing mathematical rigor for brevity.

Continuity: Single unbroken (without jumps) curve for all possible input values, i.e., without discontinuities.

Lipschitz continuity: A formal stricter continuity requirement that limits the pace of function changes. A real-valued function \(f(\cdot ):\mathcal {R}\rightarrow \mathcal {R}\) is called Lipschitz continuous if there exists a positive real constant \(K\) such that, for all real \(x_1\) and \(x_2\),

\begin{equation} \abs {f(x_1) - f(x_2)} \le K \abs {x_1 - x_2}. \end{equation}

This feature formally limits the maximum gradient values of a loss function.

Differentiability: A differentiable function of one real variable is a function whose derivative exists at each point in its domain.

• • If the function is differentiable, it is also continuous.

• • If the derivative is a continuous function, it is also locally Lipschitz continuous.

Convexity and strict convexity: Each segment lies above the graph between two points (Fig. 6.1). Formally, it means that the line segment between \(\left (x_1,f(x_1)\right )\) and \(\left (x_2,f(x_2)\right )\) is always above or just meeting the graph of the function \(f(x),x_1\le x \le x_2\). Mathematically, for all \(0\le t \le 1\) and \(\forall x_1,x_2\in \real \),

\begin{equation} f\left (tx_1+(1-t)x_2\right )\le tf(x_1) + (1-t)f(x_2) \end{equation}

A twice-differentiable function of a single variable is convex if and only if its second derivative is nonnegative on its entire domain. For a strict convexity, the second derivative is always positive. In the context of loss function properties, convexity guarantees that any local minimum is also a global minimum. A strictly convex function has at most one global minimum.

All these properties are particularly important in GD-related methods.

6.3 Losses

Per-sample and vector notation error notation are,

\begin{align*} e_i &= y_i - \hat {y}_i\quad i=1,\ldots ,M\\ \be &= \by - \hat {\by } \end{align*} Note, the order of subtraction is sometimes important in the context of this section.

Some of the following losses are also used as metrics.

6.3.1 Mean-squared error (MSE) or L₂-loss

• • Continuous, differentiable, convex

MSE loss (also termed L₂-loss) and metric is ((2.5))

\begin{equation} \begin{aligned} J(\by ,\hat {\by }) &= \frac {1}{M}\norm {\by - \hat {\by }}^2=\frac {1}{M}\be ^T\be \\ &= \frac {1}{M}\sum _{i=1}^M e_i^2 \end {aligned} \end{equation}

When used as loss, sometimes sum-of-squared error (SSE) ((2.4)) or the version with factor \(\frac {1}{2M}\) is applied to “compensate” for the derivative are also used (see also Sec. 2.1.1).

Important properties:

• • Popular regression loss.

• • Popular metric.

• • Analytical gradient that is error-dependent.

• • Sometimes, analytical solution is available, e.g. normal equation.

• • The main drawback is inherent outlier sensitivity.

6.3.2 RMSE

• • Continuous, differentiable, convex

Complementary convenient metric for MSE is root-MSE (RMSE).

\begin{equation} \begin{aligned} J(\by ,\hat {\by }) &= \frac {1}{\sqrt {M}}\norm {\by - \hat {\by }}\\ &= \sqrt {\frac {1}{M}\sum _{i=1}^M e_i^2} \end {aligned} \end{equation}

• • Theoretically, RMSE can also be used as loss, but it is very similar to MSE with only a difference in gradients.

• • Easier human interpretation.

6.3.3 Mean absolute error (MAE)

MAE is used both as loss and metric.

\begin{equation} \begin{aligned} \loss (\by ,\hat {\by }) &= \frac {1}{M}\sum _{i=1}^M \abs {e_i} \end {aligned} \end{equation}

Important properties:

• • Popular loss and metric.

• • All errors are similarly important and, therefore, less sensitive to outliers than MSE.

• • Error-independent gradient that may result in slower convergence under certain conditions. In particular, the gradient is high even for very small errors. To fix this, a dynamic learning rate that decreases as we move closer to the minima is required. MSE behaves nicely in this case and will converge even with a fixed learning rate. The gradient of MSE loss is high for larger loss values and decreases as loss approaches 0, making it more precise at the end of training.

• • Non-differentiable at \(\loss (\by ,\hat {\by }) =0\), also without dramatic influence on most learning algorithms.

A brief numerical example of MSE, RMSE and MAE is presented in Table 6.1 and in Fig. 6.2.

The visual comparison of a few regression losses is presented in Fig. 6.3 and detailed in Sec. 6.3.4.

6.3.4 Special Losses (*)

Huber loss

• • Lipschitz continuous, differentiable, convex

\begin{equation} \loss (e_i) = \begin{cases} \dfrac {1}{2}e_i^2 & \abs {e_i}\le \delta \\[7pt] \delta \left (\abs {e_i} - \dfrac {1}{2}\delta \right ) & \text {otherwise} \end {cases} \end{equation}

For small \(e_i\) it behaves like MSE and for larger \(e_i\) like MAE.

The problem with Huber loss is the need to tune the hyperparameter \(\delta \), which is a non-trivial process.

Log-cosh loss

\begin{equation} \loss (e_i) = \log {\cosh {e_i}} \end{equation}

Properties:

• • Twice differentiable everywhere, \(\dfrac {\partial }{\partial e_i}L(e_i)=\tanh {e_i}\).

• • Approximation:

  \(\seteqnumber{0}{}{8}\)

  \begin{equation} L(e_i)\approx \begin{cases} \dfrac {e_i^2}{2} & e_i \text { small}\\[5pt] \abs {e_i}-\log (2) & e_i \text { large} \end {cases} \end{equation}

• • No hyper-parameters.

• • Similar to Huber loss with \(\delta =1\).

• • Requires non-trivial error handling, otherwise may get stuck in either of two regions.

• • Explicit hyper-parameter optimization of Huber loss is recommended.

Cauchy

\begin{equation} \loss (e_i) = \log (1+\left (\frac {e_i}{d}\right )^2) \end{equation}

• • \(d\) is “sharpness” parameter

• • Robust against outliers more than MAE but less than atan loss.

Atan

\begin{equation} \loss (e_i) = \arctan (e_i^2) \end{equation}

• • Atan tends to \(\pi /2\) as input tends to infinity, and its derivatives tend to 0.

• • This means extreme outliers will have a negligible effect on the search direction compared to non-outliers.

• • It is important to ensure the data is well scaled and that the starting point is a reasonable guess at the true solution, if possible (similar to Log-cosh loss).

6.3.5 Mean Squared Logarithmic Error (MSLE)

MSLE is the relative difference between the log-transformed actual and predicted values.

\begin{equation} \begin{aligned} \loss (y_i,\hat {y}_i) &= \frac {1}{M}\sum _{i=1}^M \left (\log (y_i+1)-\log (\hat {y}_i+1)\right )^2\\ & = \frac {1}{M}\sum _{i=1}^M\log (\frac {y_i+1}{\hat {y}_i+1})^2 \end {aligned} \end{equation}

‘1’ is added to both \(y\) and \(\hat {y}\) for mathematical convenience since \(\log (0)\) is not defined but both \(y\) and \(\hat {y}\) can be 0.

• • Addresses \(y\) with high dynamic range, i.e. addresses relative error. Less useful for low dynamic range.

• • MSLE tries to treat small and large differences between the actual and predicted values similarly, e.g. in Table 6.2.

• • Penalizes underestimated values more than overestimated values.

• • Root MSLE (RMSLE) is also used, e.g. in scikit.

6.4 Relative Metrics

Relative squared error (RSE)

Normalized MSE loss.

\begin{equation} \begin{aligned} J(y_i,\hat {y}_i) &=\dfrac {\MSE }{s_\by ^2}\\ &= \dfrac {\sum _i \left ( y_i - \hat {y}_i\right )^2}{\sum _i\left (y_i-\overline {y}\right )^2} =\frac {\norm {\by - \hat {\by }}^2}{\norm {\by - \bar {\by }}^2} \end {aligned} \end{equation}

Recall, that \(\hat {y}_i = \bar {y}\) results \(\MSE =s_\by ^2\). The loss shows the fraction of the explained variance, with closer to 0 is better and \(>1\) is abnormal.

R²

The common metric in social sciences,

\begin{equation} R^2 = 1 - RSE \end{equation}

Opposite to RSE, i.e., \(R^2\) close to 1 is better. In highly undesirable cases, it may be negative.

For example, R² of 40% indicates that your model has reduced the mean squared error by 40% compared to the baseline, which is the mean model. This is the same as RSE of 60%.

For additional aspects, see also Sec. 2.3.1.

Normalized Root Mean Squared Error Expressed as a percentage, defined as:

\begin{equation} J(y_i,\hat {y}_i) =100\left (1-\frac {\RMSE }{s_\by }\right ) =100\left (1-\sqrt {RSE}\right ) =100\left (1-\frac {\norm {\by - \hat {\by }}}{\norm {\by - \bar {\by }}}\right ) \end{equation}

Relative absolute error (RAE)

Normalized MAE loss.

\begin{equation} J(y_i,\hat {y}_i) = \dfrac {\displaystyle \sum _i \abs {y_i - \hat {y}_i}}{\displaystyle \sum _i\abs {y_i-\overline {y}}} = \frac {MAE}{\displaystyle \frac {1}{M}\sum _i\abs {y_i-\overline {y}}} \end{equation}

Recall, that \(\hat {y}_i = \bar {y}\) results in \(MAE=\frac {1}{M}\sum _i\abs {y_i-\overline {y}}\). Closer to 0 is better and \(>1\) is abnormal.

Mean Absolute Percentage Error (MAPE)

Scaled error metric.

\begin{equation} J = \frac {1}{M}\sum _{i=1}^M\frac {\abs {y_i-\hat {y}_i}}{y_i}\times 100\% \end{equation}

• • Beware of small denominator!

• • Can exceed 100%.

• • Asymmetric, as described in Table 6.3.

Additional Reading Additional reading on regression metrics: [3]

6.5 Information Criteria

Adding parameters always improves the training MSE, which encourages overfitting. Information criteria replace the raw MSE with a penalized score that balances goodness-of-fit against the number of parameters \(N\), given \(M\) data points. All criteria below assume that the residuals are approximately Gaussian.

6.5.1 Akaike’s Final Prediction Error (FPE)

FPE estimates the expected prediction error on new data by scaling the training MSE with a bias-correction factor:

\begin{equation} FPE = \MSE \frac {1+N/M}{1-N/M} = \MSE \frac {M+N}{M-N} \end{equation}

6.5.2 Akaike’s Information Criterion (AIC)

AIC trades off goodness-of-fit against complexity. Lower AIC indicates a better model:

\begin{equation} \Delta AIC = 2N + M\ln (\MSE ) \end{equation}

AICc adds a small-sample correction, recommended when \(M/N < 40\):

\begin{equation} AICc = AIC + 2N\frac {N+1}{M-N-1} \end{equation}

6.5.3 Bayesian Information Criterion (BIC)

BIC applies a stronger penalty on the number of parameters, especially as the sample size \(M\) grows, so it tends to favor simpler models than AIC:

\begin{equation} BIC = M\ln (\MSE ) + N\ln (M) \end{equation}

These criteria are computed directly from residuals and parameter count without any additional model fitting, and they enable principled comparison between models of different complexity. Their main limitation is the Gaussian-residual assumption, which can be misleading when violated; AIC and BIC can also disagree about which model is best, so no single criterion is universally preferred. A likelihood-based version of these criteria, used for time-series models, appears in Sec. 20.3.

6.6 Summary

Common Pitfalls:

• • Scale confusion: MSE/RMSE/MAE are scale-dependent; compare across datasets only after normalization or via dimensionless metrics.

• • MAPE near zero: Avoid MAPE when \(y\) can be small or cross zero.