Machine Learning & Signals Learning
11 Feature Engineering and Selection
11.1 Feature Engineering
Feature engineering creates or transforms features to improve model performance, while feature selection chooses a subset from existing ones. In a typical pipeline, feature engineering is applied first, followed by feature selection.
11.1.1 Feature Transformations
Polynomial features1 generate interaction and higher-order terms from the original features. For \(N\) features and degree \(d\), the transformation produces all monomials up to degree \(d\). This is closely related to the polynomial kernel approach (see Kernels chapter).
1 In Python: sklearn.preprocessing.PolynomialFeatures
-
Example 11.1: For two features \(x_1, x_2\) with degree \(d=2\), the expanded feature set is:
\(\seteqnumber{0}{}{0}\)\begin{equation} [x_1, x_2] \rightarrow [1,\; x_1,\; x_2,\; x_1^2,\; x_1 x_2,\; x_2^2]. \end{equation}
The dimensionality increases from \(N=2\) to \(N'=6\) (including the bias term).
Log transform applies \(x' = \log (x)\) (for \(x > 0\)) to reduce right-skewness and compress the range of features spanning several orders of magnitude. It converts multiplicative relationships into additive ones.
Binning (discretization) converts continuous features into categorical ones by partitioning the value range into intervals. For example, an age feature can be binned into groups: \([0,18)\), \([18,65)\), \([65,\infty )\). This can reduce the effect of outliers and capture non-linear step-like relationships.
Pros:
-
• Polynomial features enable linear models to fit non-linear patterns.
-
• Log transform stabilizes variance and reduces the influence of outliers.
-
• Binning is robust to outliers and can simplify the model.
Cons:
-
• Polynomial features cause rapid dimensionality growth: \(\binom {N+d}{d}\) terms for degree \(d\).
-
• Log transform is only applicable to strictly positive values.
-
• Binning discards information within each bin.
11.1.2 Date and Time Features
A single datetime variable (e.g., a timestamp) carries no direct numerical meaning for most models. Extracting its components creates features that expose cyclic and calendar-based patterns:
-
• Day of week (1–7): captures weekly seasonality (e.g., weekday vs. weekend behavior).
-
• Day of month (1–31): captures monthly patterns (e.g., salary-related spending).
-
• Month (1–12): captures annual seasonality (e.g., weather, holidays).
-
• Hour of day (0–23): captures intra-day patterns (e.g., traffic peaks).
-
• Quarter (1–4): captures business-cycle effects.
-
• Is weekend (binary): a derived indicator for weekend days.
Because many of these features are cyclic (e.g., hour 23 is close to hour 0), encoding them as a single integer can mislead distance-based models. A common solution is cyclical encoding using sine and cosine:
\(\seteqnumber{0}{}{1}\)\begin{equation} x_{\sin } = \sin \!\left (\frac {2\pi \, x}{P}\right ), \quad x_{\cos } = \cos \!\left (\frac {2\pi \, x}{P}\right ), \end{equation}
where \(P\) is the period (e.g., \(P=7\) for day of week, \(P=24\) for hour of day). This maps each cyclic value onto the unit circle, so that the first and last values in the cycle are adjacent.
Pros:
-
• Exposes temporal patterns that are invisible in raw timestamps.
-
• Cyclical encoding preserves the natural proximity of cyclic values.
Cons:
-
• Increases dimensionality, especially with cyclical encoding (two features per component).
-
• Requires domain knowledge to choose which components are relevant.
11.2 Feature Selection
Motivation:
-
• Some features are redundant and are very similar to each other.
-
• Some features are noise, i.e does not contribute to the performance of a model.
-
• Each feature has its (significant) computation complexity.
-
• Enhanced generalization by reducing overfitting
-
• Shorter training times and lesser computational cost
Exhaustive full search over all possible combination of features is typically computationally prohibitive. For example, checking all possible subset combinations of \(N=15\) features includes \(\sum _{k=1}^{N}\binom {N}{k} = 32767\) possible combinations.
All of the practical FS methods use sub-optimal search that significantly reduce the number of the examined combinations. An example of feature-number-aware metric is adjusted \(R^2\) in Sec. 3.2.2.
11.3 Classifier-Based
Greedy search
Greedy search methods build a feature subset incrementally by making locally optimal choices at each step. While they do not guarantee a globally optimal solution, they significantly reduce computational complexity from exponential to polynomial.
Forward feature selection starts with an empty feature set and iteratively adds the feature that most improves model performance. At each step, every remaining feature is evaluated, and the one yielding the best performance gain is added. This process continues until adding more features no longer improves performance or a predetermined number of features is reached.
Backward feature selection begins by constructing a model with all available features. It then eliminates the one that produces the highest performing model based on specific evaluation metrics. In subsequent steps, it continues to remove the next least significant feature, improving performance each time. This process is repeated, eliminating one feature at a time, until a predetermined criterion is satisfied.
Forward selection is computationally cheaper when the target subset is small (\(N' \ll N\)), while backward selection is preferred when only few features need to be removed. Both methods require \(\mathcal {O}(N \cdot N')\) model evaluations, compared to \(\mathcal {O}(2^N)\) for exhaustive search.
Pros:
-
• Classifier-tailored: feature relevance is evaluated in the context of the actual model.
-
• Captures feature interactions that affect model performance.
Cons:
-
• May converge to a local optimum due to greedy nature.
-
• Computationally expensive: requires retraining the classifier for each candidate subset.
-
• Hyperparameter tuning may be needed for each set or subset size.
Constant number of the selected features For example, for the pre-defined feature subset size of \(N'=3\) and \(N=15\), it results only in \(\binom {N}{N'} = 455\) feature combinations.
11.4 Statistical-Based
There are also statistical FS methods that are not related to a particular classifier.
11.4.1 Variance Threshold
Variance threshold2 removes features with variance below a specified threshold. For a given feature, let \(M\) be the number of samples, \(x_i\) the \(i\)-th sample, and \(\bar {x}\) the mean. The variance is:
\(\seteqnumber{0}{}{2}\)\begin{equation} \Var [x] = \frac {1}{M} \sum _{i=1}^{M} (x_i - \bar {x})^2. \end{equation}
Features with \(\Var [x] < \tau \) are removed, where \(\tau \) is the threshold parameter.
This method is useful for removing constant or near-constant features that provide no discriminative information. A common use case is removing features with zero variance (all identical values).
Note that variance threshold is unsupervised and does not consider the target variable.
2 In Python: sklearn.feature_selection.VarianceThreshold
11.4.2 ANOVA
Analysis of Variance (ANOVA)3 evaluates feature relevance by comparing the variance between class means to the variance within classes. For a given feature, let:
-
• \(K\) be the number of classes,
-
• \(M_k\) the number of samples in class \(k\),
-
• \(x_{k,i}\) the \(i\)-th sample in class \(k\),
-
• \(\bar {x}_k\) the mean of class \(k\), and
-
• \(\bar {x}\) the global mean.
The between-class sum of squares measures how much class means deviate from the global mean:
\(\seteqnumber{0}{}{3}\)\begin{equation} SS_B = \sum _{k=1}^{K} M_k (\bar {x}_k - \bar {x})^2. \end{equation}
The within-class sum of squares measures variability within each class:
\(\seteqnumber{0}{}{4}\)\begin{equation} SS_W = \sum _{k=1}^{K} \sum _{i=1}^{M_k} (x_{k,i} - \bar {x}_k)^2. \end{equation}
The F-statistic is the ratio of mean squares:
\(\seteqnumber{0}{}{5}\)\begin{equation} F = \frac {SS_B / (K - 1)}{SS_W / (M - K)} = \frac {MS_B}{MS_W}, \end{equation}
where \(M = \sum _{k=1}^{K} M_k\) is the total number of samples.
A higher F-value indicates that the feature better separates the classes. Features are ranked by their F-scores, and those with the highest scores are selected. ANOVA assumes features are independent and normally distributed within each class.
3 In Python: sklearn.feature_selection.f_classif
-
Example 11.3: Consider \(K=2\) classes with \(M_1 = M_2 = 3\) samples each, and two candidate features:
Feature 1 (good separation): Class 1: \(\{2, 4, 6\}\), Class 2: \(\{8, 10, 12\}\).
-
• Class means: \(\bar {x}_1 = 4\), \(\bar {x}_2 = 10\), global mean \(\bar {x} = 7\).
-
• \(SS_B = 3(4-7)^2 + 3(10-7)^2 = 27 + 27 = 54\).
-
• \(SS_W = (2-4)^2 + (4-4)^2 + (6-4)^2 + (8-10)^2 + (10-10)^2 + (12-10)^2 = 16\).
-
• \(F = \frac {54/1}{16/4} = \frac {54}{4} = 13.5\).
Feature 2 (poor separation): Class 1: \(\{5, 6, 7\}\), Class 2: \(\{4, 6, 8\}\).
-
• Class means: \(\bar {x}_1 = 6\), \(\bar {x}_2 = 6\), global mean \(\bar {x} = 6\).
-
• \(SS_B = 3(6-6)^2 + 3(6-6)^2 = 0\).
-
• \(SS_W = 1 + 0 + 1 + 4 + 0 + 4 = 10\).
-
• \(F = 0\).
Feature 1 (\(F = 13.5\)) is selected over Feature 2 (\(F = 0\)) since it better separates the classes.
-
Pros:
-
• Fast computation: closed-form solution with no model training required.
-
• Classifier-independent: can be used as a preprocessing step for any model.
Cons:
-
• Assumes linear separability: only measures differences in means, not complex relationships.
-
• Evaluates features independently: does not capture feature interactions.
-
• Sensitive to assumptions: assumes normally distributed data within classes.
11.5 Tree-Based Feature Importance
Decision tree models provide built-in feature importance scores based on how much each feature contributes to reducing impurity (e.g., Gini impurity or entropy) across all splits in the tree.
Pros:
-
• Fast computation: importance is a byproduct of model training.
-
• Captures non-linear relationships and feature interactions.
Cons:
-
• Biased toward high-cardinality features (many unique values).
-
• Importance is split among correlated features, underestimating individual relevance.