Machine Learning & Signals Learning
24 Regression Metrics
Given targets \(y[n]\) and predictions \(\hat y[n]\), \(n=0,\ldots ,L-1\), the prediction error (residual) is
\(\seteqnumber{0}{}{0}\)\begin{equation} \label {eq:metric-residual} e[n]=y[n]-\hat y[n]. \end{equation}
Every metric below is a summary of these same \(L\) residuals.
24.1 Scale-Dependent Metrics
Scale-dependent metrics retain the original units of the data, making them appropriate when the size of the error, or the direction it leans, is meaningful in itself.
Mean squared error (MSE) and root mean squared error (RMSE)
\(\seteqnumber{0}{}{1}\)\begin{equation} \label {eq:metric-mse} \mathrm {MSE}=\frac {1}{L}\sum _{n} e^2[n],\qquad \mathrm {RMSE}=\sqrt {\mathrm {MSE}}. \end{equation}
Heavily penalizes large errors; sensitive to outliers. MSE is reported in squared units and RMSE returns to the units of the data. MSE is also commonly used as a training loss (differentiable, convex).
Mean absolute error (MAE) and median absolute error (MedAE)
\(\seteqnumber{0}{}{2}\)\begin{equation} \label {eq:metric-mae} \mathrm {MAE}=\frac {1}{L}\sum _{n}|e[n]|,\qquad \mathrm {MedAE}=\operatorname {median}_n\,|e[n]|. \end{equation}
Robust to outliers (especially MedAE). MAE is also used as a training loss (convex, but nondifferentiable at \(0\)).
Mean error (ME)
\(\seteqnumber{0}{}{3}\)\begin{equation} \label {eq:metric-me} \mathrm {ME}=\frac {1}{L}\sum _{n} e[n]. \end{equation}
Residuals of opposite sign cancel, so ME can sit at zero while the predictions are worthless; it is read next to MSE, never alone. The pair is tied together by
\(\seteqnumber{0}{}{4}\)\begin{equation} \label {eq:metric-me-decomp} \mathrm {MSE}=\mathrm {ME}^2+\frac {1}{L}\sum _n\bigl (e[n]-\mathrm {ME}\bigr )^2, \end{equation}
an exact identity splitting the MSE into a systematic offset and the spread around it, which is the sample form of the bias-variance split of Eq. (4.13).
Where ME says nothing
Two situations in which a zero ME carries no information:
-
• On the training residuals of a least-squares fit with an intercept, \(\mathrm {ME}=0\) by construction (Sec. 3.3.3), whatever the quality of the fit. The metric is informative only on held-out data.
-
• ME is the bias of one predictor on one record, not the \(\mathrm {Bias}[\hat y]\) of Sec. 4.4, which averages over training sets at unseen points. A model can hold \(\mathrm {ME}=0\) on every run and still be badly biased in that sense.
-
Example 24.1 (A six-sample record): A predictor is evaluated on \(L=6\) samples:
\(n\) 0 1 2 3 4 5 \(y[n]\) 10 12 9 11 13 10 \(\hat y[n]\) 9 14 9 10 14 5 \(e[n]\) 1 \(-2\) 0 1 \(-1\) 5 The residuals of the last row are the only quantities that need to be computed; each metric reuses them. Squaring and summing gives \(\sum _n e^2[n]=32\), so \(\mathrm {MSE}=32/6=5.33\) and \(\mathrm {RMSE}=2.31\). Summing the magnitudes gives \(\sum _n|e[n]|=10\), so \(\mathrm {MAE}=10/6=1.67\). Sorting those magnitudes, \(0,1,1,1,2,5\), gives \(\mathrm {MedAE}=1\).
The single bad sample at \(n=5\) contributes \(25\) of the \(32\) squared units, which is why RMSE lands \(39\%\) above MAE, while MedAE reports the typical sample and does not see that failure at all.
-
Example 24.2 (Two predictors, one target): Two predictors are compared on the same target signal (Fig. 24.1). Predictor A has intermediate, uniformly distributed errors. Predictor B is more accurate on most samples but contains three large outliers. The bar chart shows how different metrics rank the two: RMSE penalizes the outliers heavily (B is much worse), MAE is similar for both, and MedAE favors B (ignoring the outliers entirely). This illustrates why reporting more than one metric provides a more complete picture of prediction quality.
What these numbers say
All three metrics answer “how large is the error, in the units of the data”:
-
• RMSE \(\ge \) MAE always holds, and the size of that gap measures how unevenly the error is spread over the record.
-
• RMSE together with MedAE brackets the question of outliers.
24.2 Scale-Free Metrics
When comparing forecasts across series with different units or scales, scale-free metrics normalize the error so that results are comparable.
Mean absolute percentage error (MAPE) and symmetric MAPE (sMAPE)
\(\seteqnumber{0}{}{5}\)\begin{equation} \label {eq:metric-mape} \mathrm {MAPE}=\frac {100}{L}\sum _n \frac {|e[n]|}{|y[n]|+\varepsilon },\qquad \mathrm {sMAPE}=\frac {100}{L}\sum _n \frac {2|e[n]|}{|y[n]|+|\hat y[n]|+\varepsilon }. \end{equation}
A small \(\varepsilon \) is added to avoid division by zero. Because MAPE divides by the target alone, the same absolute error is charged differently at different targets: an error of \(40\) is \(40\%\) against \(y=100\) but \(300\%\) against \(y=20\), as tabulated in Table 6.3. The charge is also one-sided, since a prediction cannot fall below zero: under-prediction saturates at \(100\%\), while over-prediction is unbounded. sMAPE divides instead by the mean of target and prediction, which bounds it in \([0,200]\%\), but the bound is reached rather than avoided: as \(y[n]\to 0\) the denominator collapses onto \(|\hat y[n]|\) and the term saturates at \(200\%\) however small the absolute error, as the left panel of Fig. 24.2 shows.
Mean absolute scaled error (MASE)
MASE normalizes the MAE by the error of an in-sample naive forecast (\(\hat {y}[n]=y[n-1]\)):
\(\seteqnumber{0}{}{6}\)\begin{equation} \label {eq:metric-mase} \mathrm {MASE}= \frac {\frac {1}{L}\sum _{n}|e[n]|} {\frac {1}{L-1}\sum _{n=1}^{L-1}|y[n]-y[n-1]|}. \end{equation}
MASE \(<1\) means the model outperforms the naive baseline. The metric is robust across scales and can be extended to a seasonal naive denominator when seasonality is present. How demanding that baseline is depends on the signal: on a strongly correlated record the naive forecast is already close to optimal (Sec. 21.3).
Normalized RMSE (NRMSE)
NRMSE normalizes the RMSE by a measure of the target’s spread:
\(\seteqnumber{0}{}{7}\)\begin{equation} \label {eq:metric-nrmse} \mathrm {NRMSE}_{\sigma }=\frac {\mathrm {RMSE}}{\sigma _y},\qquad \mathrm {NRMSE}_{\mathrm {range}}=\frac {\mathrm {RMSE}}{\max y-\min y}. \end{equation}
Which denominator to trust
A scale-free metric is only as defensible as the quantity it divides by:
-
• MAPE divides by the target itself, which fails whenever the target passes near zero.
-
• sMAPE divides by the mean of target and prediction, which bounds the score without rescuing it: near a zero target the denominator follows the prediction down, and the term pins to \(200\%\) however small the absolute error.
-
• MASE divides by the naive one-step forecast, so it answers “better than repeating the last sample?”.
-
• NRMSE\(_{\sigma }\) divides by the spread of the record, so it answers “better than predicting the mean?”: the constant \(\bar y\) scores exactly 1.
-
• NRMSE\(_{\mathrm {range}}\) divides by \(\max y-\min y\), a figure two samples decide between them. It places the error on a convenient scale but carries no baseline, so no value of it passes or fails.
When a metric is quoted without its baseline, the number is not comparable across studies.
-
Example 24.3 (The same record, scale-free): The residuals of Example 24.1 are reused, so only the denominators change:
Metric Divided by Value MAPE \(|y[n]|\), per sample \(15.6\%\) sMAPE mean of \(|y[n]|\) and \(|\hat y[n]|\), per sample \(18.3\%\) MASE naive step, \(12/5=2.4\) \(0.69\) NRMSE\(_{\sigma }\) \(\sigma _y=1.34\) \(1.72\) NRMSE\(_{\mathrm {range}}\) \(\max y-\min y=4\) \(0.58\) The first two normalize each sample before averaging; the last three divide a single aggregate, MAE or RMSE, by one number.
Percentage errors. The sample at \(n=5\) alone accounts for more than half of MAPE, its ratio \(|e|/|y|=0.50\) against \(0.17\) or less everywhere else. sMAPE is the larger of the two here because that prediction lies below the target, so the symmetric denominator shrinks with it.
Naive baseline. MASE lands below 1, so the model beats the one-step naive forecast.
Spread. NRMSE\(_{\sigma }\) exceeds 1, meaning the errors are larger than the spread of the record itself, so predicting the constant \(\bar y\) would have scored better, while NRMSE\(_{\mathrm {range}}\) reports a comfortable \(0.58\). Two baselines, two verdicts: which one is quoted decides whether this predictor looks good.
Rescaling. Multiplying both \(y[n]\) and \(\hat y[n]\) by 100, as if the record were reported in centimeters instead of meters, multiplies RMSE, MAE and MedAE by \(100\) and MSE by \(100^2\), and leaves MAPE, sMAPE, MASE and NRMSE untouched, as the right panel of Fig. 24.2 shows.
24.3 Information Criteria
The metrics above rank predictions; information criteria rank models, charging each one for the parameters it spends. They were introduced in Sec. 6.5, together with Akaike’s final prediction error (FPE); this section applies them to the choice of a model order for a time series.
Adding parameters always improves the training MSE, which encourages overfitting. Information criteria replace the raw MSE with a penalized score that balances goodness-of-fit against the number of parameters \(k\), given \(L\) samples:
\(\seteqnumber{0}{}{8}\)\begin{equation} \label {eq:ic-aic-bic} \mathrm {AIC}=L\ln (\mathrm {MSE})+2k,\qquad \mathrm {AICc}=\mathrm {AIC}+\frac {2k(k+1)}{L-k-1},\qquad \mathrm {BIC}=L\ln (\mathrm {MSE})+k\ln L. \end{equation}
Lower values indicate a better model. The \(L\ln (\mathrm {MSE})\) term rewards fit, and the second term charges for complexity: Akaike’s information criterion (AIC) charges \(2\) per parameter; its corrected version (AICc) should be preferred when \(L/k\) is small; the Bayesian information criterion (BIC) charges \(\ln L\) per parameter and therefore favors simpler models as the record grows. All three assume the residuals are approximately Gaussian.
-
Example 24.4 (Choosing an AR order): An auto-regressive (AR) process of order 2 (Sec. 21.3),
\(\seteqnumber{0}{}{9}\)\begin{equation*} y[n]=1.2\,y[n-1]-0.6\,y[n-2]+\epsilon [n],\qquad \sigma _\epsilon =1, \end{equation*}
is simulated for \(300\) samples (Fig. 24.3, left). Candidate AR(\(p\)) models with \(p=1,\ldots ,10\) are fitted by least squares on the same \(N=290\) rows, so their residuals are comparable, and each model is charged \(k=p+1\) parameters: \(p\) coefficients plus the noise variance. Evaluating Eq. (24.9) with \(L=N=290\) gives the working forms \(N\ln (\mathrm {MSE})+2k\) and \(N\ln (\mathrm {MSE})+k\ln N\).
The training MSE (dashed, right panel) falls from \(1.55\) at \(p=1\) to \(0.93\) at \(p=10\) and never marks a stopping point. Both criteria drop steeply from \(p=1\), where they reach \(132\) and \(139\) and leave the plotted range, to a minimum at \(p=2\), the true order, and rise afterwards: every additional coefficient buys less fit than the \(2\) (AIC) or \(\ln N=5.7\) (BIC) it costs. The BIC curve rises about three times faster, which is why on any record long enough that \(\ln N>2\) the BIC order cannot exceed the AIC order.
Using the criteria
The criteria cost nothing once the model is fitted: they need only the achieved MSE and the parameter count, not a second fit or a held-out split. Their values are meaningless in isolation and comparable only between models fitted on the same samples, so a single number such as “AIC \(=-8.8\)” should never be reported alone. The Gaussian-residual assumption is their main limitation, and AIC and BIC can disagree, in which case the disagreement itself is the message: the data do not settle the order.
24.4 Evaluation Protocol
Every metric in this chapter is only as trustworthy as the samples it is evaluated on. For a time series, the split that produces those samples cannot be random: shuffling the record lets the model interpolate between neighboring samples it has already seen, and each metric then reports a value the model will not reproduce on future data.
A temporal split is used instead: all data before a cutoff time \(t_c\) is used for training, and data after \(t_c\) for testing. The test set must always lie in the future relative to the training set.
Time-series cross-validation Standard \(k\)-fold CV shuffles the data, violating temporal order. Instead, time-series CV (Fig. 24.4) uses an expanding (or sliding) window:
-
1. Start with a minimal training set of the earliest samples.
-
2. Train the model and evaluate on the next time step (or window).
-
3. Expand the training set to include the previous test point, and repeat.
Each fold preserves the temporal ordering: training data always precedes test data (reference; example in Python). Reporting the mean of a metric over the folds, rather than its value on a single split, also removes the dependence on where the one cutoff happened to fall.
Optionally, a gap of \(g\) samples can be inserted between the training and test sets. This prevents leakage from auto-correlated signals where adjacent windows carry similar information. The gap size \(g\) is a hyper-parameter that depends on the correlation length of the signal.
Possible data leakage
A random train–test split, standard for i.i.d. tabular data, is invalid for a time series. Adjacent samples share temporal context, so a shuffled split lets information from the test period reach the model and produces over-optimistic values of every metric in this chapter.
24.5 Summary
Table 24.1 collects the properties that decide which metric to report; every value it lists is meaningful only when computed under the temporal protocol of Sec. 24.4. The two questions it answers are whether a metric survives a change of units, and what a few bad samples do to it. A defensible report combines one metric in the units of the data with one scale-free metric, and states the baseline the second one uses.
| Metric | Eq. | Units | Scale-free | Outlier-robust | Also used as loss | Main pitfall |
| MSE | (24.2) | data\(^2\) | No | No | Yes | Dominated by the largest residual |
| RMSE | (24.2) | data | No | No | Via MSE | Same, in the units of the data |
| MAE | (24.3) | data | No | Partly | Yes | - |
| MedAE | (24.3) | data | No | Yes | No | Blind to the tail, even when the tail is what matters |
| ME | (24.4) | data | No | No | No | Cancels; a zero value does not mean small errors |
| MAPE | (24.6) | % | Yes | No | No | Diverges as \(y[n]\to 0\); asymmetric |
| sMAPE | (24.6) | % | Yes | No | No | Saturates at \(200\%\) as \(y[n]\to 0\), so the bound hides the failure |
| MASE | (24.7) | none | Yes | Partly | No | Baseline is easy to beat on strongly correlated records |
| NRMSE\(_\sigma \) | (24.8) | none | Yes | No | No | Verdict moves with the spread of the evaluated segment |
| NRMSE\(_\mathrm {range}\) | (24.8) | none | Yes | No | No | Range is set by two samples, so one outlier rescales it |
| AIC, BIC | (24.9) | none | n/a | No | No | Compare models, not predictions; same data only |