Machine Learning & Signals Learning
Part III Learning in Signals
16 Feature Extraction from Signals
16.1 Windowing
Non-overlapping windows (Fig. 16.1a) are the standard choice. Overlapping windows (Fig. 16.1b) are less recommended since they may reduce performance — the resulting feature vectors become too similar.
For overlapping windows, the overlap is controlled by the step size \(S\) (number of samples between consecutive window starts). The overlap ratio is
\(\seteqnumber{0}{}{0}\)\begin{equation} r = \frac {L - S}{L}, \end{equation}
where \(r=0\) corresponds to non-overlapping windows and \(r=0.5\) to 50% overlap.
Overlapping windows may introduce data leakage: adjacent windows share samples, so train and test sets may contain nearly identical data if split carelessly.
Each window contains an equal number of samples, \(L\). The value of \(L\) is:
-
• Field-related, e.g. 20–40 msec in acoustic and speech processing.
-
• Hand-picked by visual analysis.
-
• Hyper-parameter (the worst case).
16.2 Workflow
Feature extraction: A mapping \(f:\real ^L\to \real ^N\) that converts a signal window of \(L\) samples into a feature vector of \(N\) numerical values.
FE from signals is presented in Fig. 16.1a.
Typical dimensions:
-
• \(L\) may vary between 100s to 10,000s samples.
-
• \(N\) may vary between few to 1,000s features.
Feature extraction and feature selection are distinct steps. FE creates new features from raw data; feature selection (FS) then removes redundant or irrelevant features from the extracted set.
FE from images FE from images uses the similar principles. The main difference is in the applied image feature functions.
16.3 Signal transformation
Notes:
-
• The transformations have to be invertible.
-
• Some of the transformations are restricted only for positive signals, \(y[n]>0\,\forall n\).
Some of the common transformations:
-
• Logarithmic transformation
\(\seteqnumber{0}{}{1}\)\begin{equation} \tilde {y}[n]= \begin{cases} \log (y[n]+1) & y[n]\geq 0\\ \log (y[n]) & y[n]> 0\\ \end {cases} \end{equation}
We use \((+1)\) to avoid value of 0.
-
• Square root transformation,
\(\seteqnumber{0}{}{2}\)\begin{equation} \tilde {y}[n]=\sqrt {y[n]} \end{equation}
-
• Exponential transformation,
\(\seteqnumber{0}{}{3}\)\begin{equation} \tilde {y}[n]=y^m[n] \end{equation}
-
• Modified Box-Cox transformations [?],
\(\seteqnumber{0}{}{4}\)\begin{equation} w_t = \begin{cases} \log (y_t) & \text {if $\lambda =0$}; \\ \text {sign}(y_t)(|y_t|^\lambda -1)/\lambda & \text {otherwise}. \end {cases} \end{equation}
with inverse
\(\seteqnumber{0}{}{5}\)\begin{equation} y_{t} = \begin{cases} \exp (w_{t}) & \text {if $\lambda =0$};\\ \text {sign}(\lambda w_t + 1)|\lambda w_t+1|^{1/\lambda } & \text {otherwise}. \end {cases} \end{equation}
-
Example 16.1: A sensor signal \(y[n]\) with values in \([0,\,10{,}000]\) is highly right-skewed (Fig. 16.2). Applying \(\tilde {y}[n]=\log (y[n]+1)\) compresses the dynamic range to \([0,\,\approx 9.2]\), making the distribution more symmetric and improving downstream classifier performance.
16.4 Signal Features
Statistical (time-domain) features
Descriptive statistics computed directly on the signal window \(y[1],\ldots ,y[L]\):
-
• Mean:
\(\seteqnumber{0}{}{6}\)\begin{equation} \bar {y} = \frac {1}{L}\sum _{n=1}^{L} y[n] \end{equation}
-
• Variance:
\(\seteqnumber{0}{}{7}\)\begin{equation} \sigma ^2 = \frac {1}{L}\sum _{n=1}^{L}\bigl (y[n]-\bar {y}\bigr )^2 \end{equation}
-
• Root mean square (RMS):
\(\seteqnumber{0}{}{8}\)\begin{equation} y_{\text {rms}} = \sqrt {\frac {1}{L}\sum _{n=1}^{L} y^2[n]} \end{equation}
-
• Zero-crossing rate (ZCR):
\(\seteqnumber{0}{}{9}\)\begin{equation} \text {ZCR} = \frac {1}{L-1}\sum _{n=2}^{L}\mathbf {1}\bigl [y[n]\cdot y[n-1]<0\bigr ] \end{equation}
where \(\mathbf {1}[\cdot ]\) is the indicator function.
-
• Additional: maximum, minimum, median, skewness, kurtosis, energy (\(\sum y^2[n]\)), number of peaks.
Spectral (frequency-domain) features
Features derived from the Fourier transform \(Y[k]\) of the signal window:
-
• Spectral centroid — the “center of mass” of the spectrum:
\(\seteqnumber{0}{}{10}\)\begin{equation} \text {SC} = \frac {\sum _{k=1}^{K} f_k |Y[k]|^2}{\sum _{k=1}^{K} |Y[k]|^2} \end{equation}
where \(f_k\) is the frequency of the \(k\)-th bin.
-
• Spectral bandwidth, spectral rolloff, spectral flatness.
Auto-correlation based features
Time-domain features derived from the auto-correlation function, capturing periodicity and temporal structure.
Mixed-domain features
Features based on STFT and wavelet transforms, capturing both time and frequency information simultaneously.
Examples: tsfel feature list
Field-tailored features
Historically-developed features. For example, cepstrum in speech processing, that is logarithm of Fourier transform.
Features from different domains (statistical, spectral, etc.) may have very different scales. Feature normalization is recommended before combining them into a single feature vector.
16.4.1 Output
At the end of the FE process, each signal window of \(L\) samples is mapped to a single \(N\)-dimensional feature vector. Stacking all \(M\) windows yields an \(M\times N\) feature matrix (dataset) ready for ML (Fig. 16.1a).
Complexity Some features (e.g. spectral or wavelet-based) have non-negligible computational cost, especially for large \(L\). Speed-up can be achieved by:
-
• Parallel computation: since windows are independent, they can be distributed across multiple CPU/GPU cores.
-
• Selecting a compact feature subset (see feature selection) to avoid computing unnecessary features.
16.5 Dedicated Libraries
Several open-source libraries provide large collections of pre-implemented signal features, eliminating the need to code them from scratch.
-
• Python-based, \(150+\) features covering statistical, spectral, and non-linear domains.
-
• Built-in statistical feature selection (relevance testing via hypothesis tests).
-
• Seamless scikit-learn integration (transformers, pipelines).
tsfel (Time Series Feature Extraction Library)
-
• Python-based, \(60+\) features organized by domain (statistical, temporal, spectral).
hctsa (highly comparative time-series analysis)
-
• Matlab-based, \(7{,}700+\) features — the largest available feature library.
-
• Designed for exploratory comparison across many time-series datasets.
catch22 (CAnonical Time-series CHaracteristics)
-
• A curated subset of 22 features selected from \(4{,}791\) hctsa features on 93 publicly available datasets.
-
• Provides strong baseline performance on a broad variety of signals.
-
• Applied on internally normalized signals; optional mean and std features.
-
• Fast C implementation with Python, R, and Matlab wrappers.
-
• Does not include signal-level feature extraction, but provides classifiers, regressors, and feature selection methods.
-
• Feature selection can be embedded in a Pipeline for end-to-end workflows.
mlxtend Supplementary ML tools including sequential feature selection (forward/backward) and model evaluation utilities.
16.6 Train-Test Split in Signals
Train and test cannot be adjacent windows.
Random train-test splitting, which is standard for i.i.d. tabular data, is invalid for time-series signals. Adjacent windows share temporal context (and samples, if overlapping), so random splitting causes data leakage and over-optimistic performance estimates.
16.6.1 Classification
When the goal is to classify signals (e.g. fault detection, speaker identification), the split must be done by source rather than by individual windows. Each source (e.g. speaker, device, or recording session) appears entirely in either the train or the test set, never both.
This is known as group splitting1 (or leave-one-group-out CV). It ensures the model is evaluated on truly unseen sources, not on different windows from a source it has already learned.
1 In Python: sklearn.model_selection.GroupKFold.
-
Example 16.2: In a speaker identification task with 20 speakers, group splitting assigns 16 speakers to training and 4 to testing. All windows from a given speaker belong to the same set. A random window-level split would leak speaker-specific patterns into training, inflating accuracy.
16.6.2 Prediction
A temporal split is used: all data before a cutoff time \(t_c\) is used for training, and data after \(t_c\) for testing. The test set must always lie in the future relative to the training set.
Time-series cross-validation Standard \(k\)-fold CV shuffles the data, violating temporal order. Instead, time-series CV (Fig. 16.3) uses an expanding (or sliding) window:
-
1. Start with a minimal training set of the earliest samples.
-
2. Train the model and evaluate on the next time step (or window).
-
3. Expand the training set to include the previous test point, and repeat.
Each fold preserves the temporal ordering: training data always precedes test data (reference).
Optionally, a gap of \(g\) samples can be inserted between the training and test sets. This prevents leakage from auto-correlated signals where adjacent windows carry similar information. The gap size \(g\) is a hyper-parameter that depends on the correlation length of the signal.
16.7 Summary
-
• Feature extraction maps signal windows of \(L\) samples to \(N\)-dimensional feature vectors via \(f:\real ^L\to \real ^N\).
-
• Non-overlapping windows are preferred to avoid redundancy and data leakage.
-
• Signal transformations (log, Box-Cox, etc.) can improve downstream model performance by reshaping distributions.
-
• Features span statistical, spectral, auto-correlation, and mixed domains; dedicated libraries (tsfresh, tsfel, catch22) provide ready-to-use implementations.
-
• Train-test splitting must respect temporal or source structure — random splits are invalid for signal data.