Machine Learning & Signals Learning

25 Signal Classifiers

25.1 Preface

Given a training set of labeled signals, the task is to assign a whole signal \(y[n],\,n=0,\ldots ,L-1\) to one of \(C\) discrete classes. Unlike the regression setting of earlier chapters, the input is a sequence of inter-related samples rather than a fixed-length feature vector.

Dataset Signals in a dataset may be

• • univariate (single channel \(y[n]\)) or multivariate (\(K\) channels \(y_k[n]\), e.g. ECG leads or accelerometer axes);

• • equal-length or variable-length (length \(L\) may differ between examples);

• • scarce: labeled time-series datasets are typically small relative to \(L\).

Overfitting challenge Using each sample \(y[n]\) as a feature yields an \(L\)-dimensional input from only a handful of training examples, so a raw-sample classifier overfits easily. The methods below differ mainly in how they summarize, transform, or compare signals before classification in order to control this.

Main categories of methods

• • Distance-based: define an elastic distance between raw signals and apply \(k\)-NN (Dynamic Time Warping).

• • Feature-based: extract interpretable summaries and classify on them (time-series forest / RISE, shapelets, SAX and SAX-VSM). The general feature-extraction pipeline these methods specialize is covered in Sec. 24.6.

• • Specially adapted transforms: apply a large bank of random or fixed transforms followed by a linear classifier (ROCKET, MINIROCKET), or combine several domain-specific classifiers in a meta-ensemble (HIVE-COTE).

• • Deep learning (out of scope here, pointer only): 1D CNNs, RNNs, and pretrained time-series foundation models. These are contrasted with classical methods in the closing table.

25.2 Distance Metrics

The distance below are particularly fit to signals of the certain kinds.

25.2.1 Dynamic Time Warping (DTW)

Definition

Given two signals [9]

\begin{equation} \begin{aligned} X &= \left (x_1,\ldots ,x_m,\ldots ,x_M\right ),\\ Y &= \left (y_1,\ldots ,y_n,\ldots ,y_N\right ),\\ \end {aligned} \end{equation}

the DTW algorithm takes as input a pointwise dissimilarity function

\begin{equation} d_{mn} = f(x_m,y_n)\ge 0. \end{equation}

A common choice is the 1D Euclidean distance, \(d_{mn} = \abs {x_m-y_n}\).

A warping path is a sequence of index pairs \(\pi = \{(m_k,n_k)\}_{k=1}^{K}\) that satisfies three constraints:

• • Boundary: \((m_1,n_1)=(1,1)\) and \((m_K,n_K)=(M,N)\).

• • Monotonicity: \(m_{k+1}\ge m_k\) and \(n_{k+1}\ge n_k\) (no turns back).

• • Step size: each step is vertical \((m{+}1,n)\), horizontal \((m,n{+}1)\), or diagonal \((m{+}1,n{+}1)\) (no skips).

The DTW distance is the minimum cumulative cost along any admissible path,

\begin{equation} \mathrm {DTW}(X,Y) = \min _{\pi }\sum _{k=1}^{K} d_{m_k n_k}, \end{equation}

and is obtained by dynamic programming over the cost matrix \([d_{mn}]\) in \(O(MN)\) time. The resulting path serves as an elastic distance between the two signals, allowing non-linear stretching of the time axis.

Unlike Euclidean distance, which pairs samples index-by-index and therefore penalizes any time shift or stretch, DTW is free to re-align the time axes and then measures the residual mismatch under the best alignment. The distance is therefore small whenever \(X\) and \(Y\) share the same shape, even if one is faster, slower, or shifted in time.

Multivariate DTW

The multivariate case is identical to the univariate one, except that the pointwise dissimilarity is defined between vectors rather than scalars. For \(K\)-dimensional (possibly complex-valued) samples \(\mathbf {x}_m,\mathbf {y}_n\in \mathbb {C}^K\), a natural choice is the Euclidean norm of the difference,

\begin{equation} d_{mn} = \norm {\mathbf {x}_m-\mathbf {y}_n}_2 = \sqrt {\sum _{k=1}^K \left (x_{k,m}-y_{k,n}\right )\left (x_{k,m}-y_{k,n}\right )^{*}}. \end{equation}

k-NN with DTW

Because DTW provides a pairwise distance between signals of possibly different lengths, it plugs directly into any distance-based classifier. The canonical choice is \(k\)-NN (Section 10): to classify a new signal, compute its DTW distance to every training signal and take a majority vote among the \(k\) nearest. With \(k=1\), 1-NN + DTW is a strong and frequently cited baseline for time-series classification.

25.2.2 Shapelets

A shapelet is a sub-sequence \(S = (s_1,\ldots ,s_l)\) of length \(l\) that is “maximally representative of a class” [18].

Shapelet distance

Given a shapelet \(S\) of length \(l\) and a signal \(X = (x_1,\ldots ,x_M)\), the shapelet distance is the minimum distance over all possible alignments:

\begin{equation} d(S, X) = \min _{i=1,\ldots ,M-l+1} \sqrt {\frac {1}{l}\sum _{j=1}^{l}(s_j - x_{i+j-1})^2} \end{equation}

This sliding-window RMSE measures how well the shapelet matches anywhere in the signal (Fig. 25.4).

Classification

• 1. Extract shapelet candidates from the training set (sub-sequences of varying lengths).

• 2. For each candidate, compute its distance to every training signal.

• 3. Select the shapelet(s) whose distances best separate the classes (e.g. by information gain).

• 4. Use the shapelet distances as features for a classifier (e.g. decision tree, SVM).

Properties

• • A signal may contain more than one discriminative shapelet; multiple shapelets can be combined as a feature vector.

• • Shapelets are interpretable — they correspond to physically meaningful signal patterns.

• • Optimized for time-domain analysis; less effective for signals whose discriminative information lies in repeated patterns or the frequency domain.

• • Shapelet discovery can be computationally expensive (\(O(M^2 \cdot l)\) per candidate); approximate and learned methods exist to reduce cost.

In Python: tslearn (shapelet learning), sktime (shapelet transform classifier).

25.3 Pseudo-Random-Features-Based Classifiers

25.3.1 Time-series forest

Time-series forest (TSF) is an interval-based method. Instead of operating on raw samples or shapelets, it extracts simple statistics from randomly chosen sub-intervals of the signal.

Algorithm

• 1. For each tree in the forest, randomly select \(\sqrt {M}\) intervals \([a_i, b_i] \subset \{1,\ldots ,M\}\).

• 2. From each interval, extract three summary features: mean, standard deviation, and slope.

• 3. Train a decision tree on the resulting feature vector (of length \(3\sqrt {M}\)).

• 4. Repeat for all trees and aggregate predictions by majority vote.

Properties

• • Somewhat interpretable: each split corresponds to a statistic computed over a specific time interval.

• • Computationally efficient compared to shapelet-based methods.

• • Captures interval-level (phase) information rather than point-level or subsequence-level patterns.

25.3.2 Random Interval Spectral Ensemble (RISE)

RISE replaces the time-domain summary statistics with spectral features (e.g. periodogram coefficients), making it effective for signals whose discriminative information lies in the frequency domain.

In Python: sktime (interval-based classifiers including TSF and RISE).

25.4 Symbolic aggregate approximation (SAX)

Algorithm

• 1. Piecewise Aggregate Approximation (PAA): divide the signal of length \(M\) into \(w\) equal-length segments and replace each segment by its mean, producing a reduced representation of length \(w\).

• 2. Symbolization: map each mean value to a symbol from an alphabet of size \(\alpha \) using breakpoints derived from the standard normal distribution (assuming z-normalized data). The result is a string of length \(w\) over the alphabet \(\{a, b, \ldots \}\).

SAX-VSM

SAX combined with the Vector Space Model (SAX-VSM) [16] applies text-classification ideas to time-series classification:

• 1. Convert each training signal to a SAX string using a sliding window, producing a “bag of words.”

• 2. Build a tf-idf weighted term-frequency vector for each class.

• 3. Classify a new signal by computing the cosine similarity of its bag-of-words vector to each class vector.

Properties

• • Interpretable: discriminative patterns correspond to readable symbolic words.

• • Dimensionality reduction is controlled by two parameters: number of segments \(w\) and alphabet size \(\alpha \).

• • Sensitive to the choice of \(w\) and \(\alpha \); typically selected by cross-validation.

In Python: tslearn (PAA, SAX), sktime (SAX-based classifiers).

25.5 Random Convolutions

25.5.1 ROCKET

RandOm Convolutional KErnel Transform (ROCKET) [5].

Algorithm

• 1. Generate a large number (e.g. \(10{,}000\)) of random convolutional kernels with random length, weights, bias, dilation, and padding.

• 2. Convolve each kernel with the input signal.

• 3. From each convolution output, extract two features: the maximum value and the proportion of positive values (ppv).

• 4. Train a linear classifier (e.g. ridge regression) on the resulting feature vector.

The key insight is that a sufficiently large collection of random kernels captures diverse patterns (trends, spikes, oscillations at various scales) without any learning or search over the kernel space.

Properties

• • May be slow: training scales linearly with the number of signals and kernels.

• • Accuracy comparable to deep learning and ensemble methods (e.g. HIVE-COTE).

• • Not interpretable: random kernels have no direct physical meaning.

25.5.2 MINIROCKET

MINIROCKET [6] is a faster, near-deterministic variant. The main differences are summarized below.

In Python: sktime (RocketClassifier, MiniRocketClassifier).

25.6 Ensemble Classifiers

25.6.1 HIVE-COTE

The Hierarchical Vote Collective of Transformation-based Ensembles (HIVE-COTE) [14] is a meta-ensemble that fuses the outputs of several diverse classifiers, each operating on a different signal domain:

• • Time domain (e.g. shapelet transform, TSF).

• • Frequency domain (e.g. RISE).

• • Dictionary-based representation (e.g. SAX-based methods).

• • Interval-based and whole-series distance-based classifiers.

Each component classifier produces a probability distribution over the classes. HIVE-COTE combines them using a weighted vote, where the weights reflect each component’s estimated accuracy.

Properties

• • Among the most accurate non-DL classifiers on standard benchmarks.

• • Computationally expensive: trains multiple full classifiers.

• • Modular: individual components can be replaced or upgraded independently.

25.7 Summary

Method comparison

The methods presented in this chapter are compared below. The “domain” column indicates the signal representation used by each method. Accuracy and speed are qualitative rankings on standard benchmarks.

Classical vs. DL approaches