Machine Learning & Signals Learning

Part I Machine Learning

1 Descriptive Statistics Basics

1.1 Sample and Population Variance

Preliminaries

We assume a uni-variate random experiment that is described by a real-valued random variable \(X\) with

\begin{align*} \E [X] &=\mu \\ \Var [X] &= \sigma ^2. \end{align*}

Let

\begin{equation} \bx =\{x_1,\dots ,x_n\} \end{equation}

be \(n\) observations of \(X\), where \(n\) may be fixed in advance or chosen arbitrarily.

1.1.1 Definition

Sample mean Given observations \(x_1,\dots ,x_n\), the sample mean \(\bar {\bx }\) is given by

\begin{equation} \bar \bx = \frac 1n\sum _{i=1}^n x_i \end{equation}

Sample variance Sample variance is given by

\begin{align} s_{\text {unbiased}}^2 =\frac 1{n-1}\sum _{i=1}^n\bigl (x_i-\bar {x}\bigr )^2 \end{align} This formulas is unbiased.

Population variance The more intuitive (biased) population variance formula is

\begin{equation} s_{\text {biased}}^2 =\frac 1n\sum _{i=1}^n\bigl (x_i-\bar {x}\bigr )^2\\ \end{equation}

Standard Deviation The sample standard deviation (std) is simply the square-root of the (unbiased or biased) sample variance, \(s\).

1.1.2 Repeated Experiments Characterization

The repeated experiments approach is based on

• • \(k\) experiments

  • – In each experiment \(n\) samples \(\bx _1,\ldots ,\bx _k\) are drawn from the known random variable \(X\).

Mean

The average of repeated experiments values is itself approximately \(\mu \), so on average the sample mean

\begin{equation} \bar {\mu }_k = \frac {1}{k}\sum _{j=1}^{k}\bar {\bx }_j \end{equation}

recovers the true mean

\begin{equation} \lim _{k\rightarrow \infty }\bar {\mu }_k \rightarrow \mu \end{equation}

This principle is illustrated in Fig. 1.1.

Fluctuations of \(\bar {\mu }_k\):

• • As you gather more observations (higher \(n\)), \(\bar \bx \) tends to stabilize: it fluctuates less, and its empirical distribution concentrates around the true center \(\mu \).

• • Moreover, for large \(n\), the variability of \(\bar \bx \) scales like \(\sigma /\sqrt {n}\) (or \(\Var [\bar x]\approx \dfrac {\sigma ^2}{n}\)), meaning the more data we collect, the tighter our estimate of the process’s center becomes.

Variance

In the case of biased variance, the mean of the \(s_{n_1},\ldots ,s_{n_k}\) values

\begin{equation} \bar {s}_k^2 = \frac {1}{k}\sum _{j=1}^{k}s_{n_j} \end{equation}

systematically underestimates the true population variance \(\sigma ^2\),

\begin{equation} \lim _{k\rightarrow \infty }\bar {s}_{k,\text {biased}}^2 \rightarrow \dfrac {n-1}{n}\sigma ^2 \end{equation}

This inherent difference is termed bias. However, for the unbiased variance,

\begin{equation} \lim _{k\rightarrow \infty }\bar {s}_{k,\text {unbiased}}^2 \rightarrow \sigma ^2 \end{equation}

This principle is illustrated in Fig. 1.2.

MSE

The reason that biased estimator is still useful, is the when the empirical mean-squared error (MSE) is of interest. MSE is defined by

\begin{align} \mathrm {\overline {MSE}}_{\text {unbiased}} &= \frac {1}{k}\sum _{j=1}^k \left (s_{{\text {unbiased}}_j} - \sigma ^2\right )^2 \rightarrow \frac {2}{n-1}\,\sigma ^4\\ \mathrm {\overline {MSE}}_{\text {biased}} &= \frac {1}{k}\sum _{j=1}^k \left (s_{{\text {biased}}_j} - \sigma ^2\right )^2 \rightarrow \frac {2n-1}{n^2}\,\sigma ^4 \end{align} and

\begin{equation} \mathrm {\overline {MSE}}_{n-1} \geq \mathrm {\overline {MSE}}_{n} \end{equation}

1.1.3 Bias

The bias of a sample statistic is the systematic difference between its long-run average over repeated experiments and the corresponding true value of the random variable. A statistic is called unbiased when this difference is zero, and biased otherwise.

For example, for the sample variance,

\begin{equation} \mathrm {bias}(s^2) = \lim _{k\to \infty }\bar {s}_k^2 - \sigma ^2. \end{equation}

Therefore, \(s_{\text {unbiased}}^2\) is unbiased while \(s_{\text {biased}}^2\) has bias \(-\sigma ^2/n\).

The application of biased and unbiased estimators:

• • Biased: in ML tasks (e.g., loss function), optimal (maximum-likelihood estimation) for Gaussian distribution

• • Unbiased: statistics.

Code implementation default varies:

• • Python uses biased expression, e.g. numpy.std.

• • Matlab uses unbiased expression, e.g. std.

1.2 Mode, Skewness and Median

1.2.1 Median

Median is a value or quantity lying at the midpoint of a frequency distribution of observed values or quantities, such that there is an equal probability of falling above or below it.

Equivalently, for a sample \(x_1,\dots ,x_n\) with order statistics \(x_{(1)}\le \cdots \le x_{(n)}\), the sample median is

\[ \mathrm {median}( x) = \begin {cases} x_{(\frac {n+1}2)}, & n\text { odd},\\[6pt] \dfrac {x_{(\frac n2)} + x_{(\frac n2 + 1)}}2, & n\text { even}. \end {cases} \]

1.2.2 Mode

For a finite sample \(x_1,\dots ,x_n\), the sample mode is any value(s) that occur(s) most often among the observations.

Multimodality: A distribution is called multimodal if it has more than one local maximum (mode) in its probability density function or histogram. Common cases include:

• • Unimodal: a single peak (e.g., normal distribution).

• • Bimodal: two distinct peaks, often indicating that the data is a mixture of two subpopulations.

• • Multimodal: three or more peaks.

Multimodality suggests that the data may come from a mixture of different underlying processes or groups.

1.2.3 Skewness

Skewness: Skewness measures the asymmetry of a distribution about its mean. The sample skewness is defined as

\[ g_1 = \frac {1}{n}\sum _{i=1}^{n}\!\left (\frac {x_i - \bar {x}}{s_n}\right )^{\!3}. \]

• • \(g_1 = 0\): the distribution is symmetric (e.g., normal distribution).

• • \(g_1 > 0\): the distribution is right-skewed (positively skewed) — the right tail is longer, and the mean is typically greater than the median.

• • \(g_1 < 0\): the distribution is left-skewed (negatively skewed) — the left tail is longer, and the mean is typically less than the median.

An illustration of the relationship between mean, median, and mode under different skewness is shown in Fig. 1.3.

1.3 Boxplot

1.3.1 Prelinaries

Quartiles: Given the order statistics \(x_{(1)}\le \cdots \le x_{(n)}\), the three quartiles divide the sorted data into four equal parts:

• • \(Q_1\) (first quartile, 25th percentile) — the median of the lower half of the data.

• • \(Q_2\) (second quartile, 50th percentile) — the median of the entire data set.

• • \(Q_3\) (third quartile, 75th percentile) — the median of the upper half of the data.

Equivalently, approximately 25% of the observations fall below \(Q_1\), and another 25% up to \(Q_3\).

Interquartile range: The interquartile range (IQR) measures the spread of the central 50% of the data,

\begin{equation} \mathrm {IQR} = Q_3 - Q_1. \end{equation}

Outlier: An outlier is any observation that falls outside the interval

\[ \bigl [Q_1 - 1.5\,\mathrm {IQR},\;\; Q_3 + 1.5\,\mathrm {IQR}\bigr ]. \]

Such points are considered unusually far from the bulk of the data and may indicate measurement errors, rare events, or heavy-tailed behavior.

The coefficient 1.5 was proposed empirically by statistician John Tukey in 70th.

1.3.2 Plot

A boxplot (box-and-whisker plot) summarizes a data set using five statistics:

• 1. Minimum non-outlier value.

• 2. First quartile (\(Q_1\)).

• 3. Median (\(Q_2\)).

• 4. Third quartile (\(Q_3\)).

• 5. Maximum non-outlier value.

The boxplot is drawn as follows:

• • A box spans from \(Q_1\) to \(Q_3\), with a line at the median \(Q_2\).

• • Whiskers extend from the box to the most extreme data points that lie within

  \[ \bigl [Q_1 - 1.5\,\mathrm {IQR},\;\; Q_3 + 1.5\,\mathrm {IQR}\bigr ]. \]

• • Any observation outside the whisker range is marked individually as a potential outlier.

An illustration of a boxplot with its components is shown in Fig. 1.4.

Interpretation

• • The box width (IQR) reflects variability — a wide box indicates high spread.

• • The median line position within the box reveals skewness: if the median is closer to \(Q_1\), the distribution is right-skewed; if closer to \(Q_3\), left-skewed.

• • Whisker lengths indicate the range of typical observations.

• • Individual points beyond the whiskers flag potential outliers that may warrant further investigation.

1.4 Histogram

1.4.1 Discrete Values

count type

Consider an experiment with:

• • \(k\) possible distinct outcomes \(x_{1},x_{2},\ldots ,x_{k}\), where \(k\) is relatively small.

• • A total of \(N\) trials.

• • Recorded results:

  • – \(n_{1}\) occurrences of \(x_{1}\),

  • – \(n_{2}\) occurrences of \(x_{2}\),

  • – \(\ldots \) and so on,

  with \(\sum _{i} n_{i}=N\).

The highest bar in a count histogram identifies the mode of the sample. A graphical representation of the outcomes is shown in Fig. 1.5(a).

probability type

Approximation to the PDF: The probability of a particular outcome is approximated by the ratio of its count to the total number of trials,

\begin{equation} \label {eq:rand1:numeric_PDF_discr} p_X[x_i]\approx \frac {n_i}{N}, \qquad i = 1,\ldots ,k. \end{equation}

Naturally, the approximation improves as \(N \to \infty \).

A graphical example of this histogram type is shown in Fig. 1.5(b).

1.4.2 Continuous Values

When the number of possible outcomes, \(k\), is large (on the order of hundreds or more, or continuous values) two main difficulties arise:

• • Presenting the results in a compact, readable form.

• • Some outcome categories contain very few observations because their probabilities are small.

A practical way to display the data is:

• 1. Record the extreme values, \(x_{\max }\) and \(x_{\min }\).

• 2. Partition the interval \(\bigl [x_{\min },x_{\max }\bigr ]\) into \(k\) equal-width bins of size \(\Delta x\).

• 3. Mark each bin by its midpoint

  \(\seteqnumber{0}{}{15}\)

  \begin{equation} \label {eq:rand1:mid_point} \tilde {x}_{i}=x_{\min }+\Bigl (i-\frac 12\Bigr )\,\Delta x, \qquad i=1,\dots ,k, \end{equation}

• 4. Let \(n_{1}\) be the count in the first bin, \(n_{2}\) the count in the second, and so on.

• 5. Use the pairs the pairs \(\left (\tilde {x}_{i},n_{i}\right )\) for count or probability type histograms.

An example of a count histogram for a large data set is shown in Fig. 1.6.

In this representation, the median is the value on the x-axis that divides the total area of the histogram into two equal parts. The variance \(s^2\) and standard deviation \(s\) are reflected in the width or spread of the histogram: a broad histogram indicates high variability, while a narrow, sharp peak indicates that the data is tightly clustered around the mean.

1.4.3 PDF Approximation of Continuous Random Variable (*)

In addition to the binning method described above, there is a third histogram type that directly approximates the PDF of a continuous random variable.

Approximation to the PDF of a continuous random variable via histogram:

\begin{equation} \label {eq:stat:hist_pdf} f_X(x_i) \approx \frac {n_i}{N}\cdot \frac {1}{\Delta x}, \quad i = 1,\ldots ,k \end{equation}

Note the normalization factor \(1/\Delta x\), which distinguishes this from the discrete case in (1.15).

A graphical example of this histogram type is shown in Fig. 1.7.

Derivation

Based on the principle

\begin{equation} \Pr (a<X\le b) = \int _{a}^{b}f_X(x)\,dx, \end{equation}

the probability of falling within a bin of width \(\Delta x\) centered at \(x_0\) can be approximated as

\begin{align} \Pr \!\Bigl (x_0-\tfrac {\Delta x}{2} < X \le x_0 +\tfrac {\Delta x}{2}\Bigr ) &= \int _{\Delta x}f_X(x)\,dx \approx f_X(x_0)\,\Delta x. \end{align} An illustration of this principle is shown in Fig. 1.8.

Rearranging,

\begin{equation} \label {eq:stat:histogram_deriv} f_X(x_0) \approx \Pr \!\Bigl (x_0-\tfrac {\Delta x}{2} < X \le x_0 +\tfrac {\Delta x}{2}\Bigr )\cdot \frac {1}{\Delta x}. \end{equation}

The probability of falling in bin \(i\) (centered at \(\tilde {x}_i\)) is approximated by the relative frequency,

\begin{equation} \label {eq:stat:hist_prob} \Pr \!\Bigl (\tilde {x}_i-\tfrac {\Delta x}{2} < X \le \tilde {x}_i +\tfrac {\Delta x}{2}\Bigr ) \approx \frac {n_i}{N}, \quad i = 1,\ldots ,k. \end{equation}

Substituting (1.21) into (1.20) yields the PDF approximation formula in (1.17).

Summary

Three ways to display experimental results as a histogram:

• • count: plot \(n_i\) — the raw bin counts.

• • probability: plot \(n_i/N\) — the relative frequency per bin.

• • pdf (or density): plot \(\dfrac {n_i}{N\,\Delta x}\) — the estimated probability density.

1.5 Violin Plot

A violin plot extends the boxplot by adding a (smoothed¹) histogram of the data on each side. A median line is typically drawn inside.

Fig. 1.9 compares the two representations for the same bimodal data set: the boxplot gives no indication of two groups, whereas the violin plot clearly shows two bulges.

1.6 Monte-Carlo Simulation

The repeated experiments of Sec. 1.1 are an instance of a general numerical method, known as Monte-Carlo simulation. Instead of deriving the behavior of a statistic analytically, the random experiment is reproduced \(k\) times on a computer, and the resulting \(k\) values are summarized by the very same descriptive tools of this chapter.

Formally, let \(g\) be any statistic computed from a sample of \(n\) observations, e.g. \(\bar {\bx }\), \(s^2_{\text {biased}}\) or the MSE. Its expected value is approximated by the average over \(k\) independent repetitions,

\begin{equation} \label {eq-monte-carlo} \E [g]\approx \bar {g}_k = \frac 1k\sum _{j=1}^{k}g_j, \end{equation}

where the law of large numbers guarantees \(\bar {g}_k \rightarrow \E [g]\) as \(k\rightarrow \infty \). The spread of \(g\) is estimated from the very same \(k\) values,

\begin{equation} \label {eq-mc-var} \Var [g]\approx s_g^2 = \frac 1k\sum _{j=1}^{k}\left (g_j-\bar {g}_k\right )^2,\qquad s_g=\sqrt {s_g^2}, \end{equation}

and \(\bar {g}_k-\theta \) estimates the bias with respect to a known true value \(\theta \) (Sec. 1.1.3).

The value of \(s_g\) is set by the sample size \(n\) and by the distribution of \(X\): it reports how much the statistic varies from one experiment to the next, and no number of repetitions reduces it. Making the statistic itself less variable requires a larger \(n\). The number of repetitions \(k\) controls something different, namely how accurately \(\bar {g}_k\) and \(s_g\) estimate the quantities they stand for.

The strength of the method is that (1.22) requires no derivation: it holds for any statistic, any distribution of \(X\), and any sample size \(n\), including all the cases where an analytical expression is unavailable or valid only for large \(n\).

The whole recipe is summarized in Fig. 1.10: the model feeds \(k\) independent experiments of \(n\) observations each, every experiment is reduced to a single value \(g_j\), and only the collection of the \(k\) values is interpreted.

Table 1.2 is precisely the output of this recipe, applied to the sample mean, the two variance estimators and their MSEs, with \(n\in \{5,20,100\}\) and \(k=5000\). Each of its entries condenses \(k\) stored values into a single number; Fig. 1.11 shows the values themselves behind one entry.

1.6.1 Number of Experiments

The simulated result is itself a random quantity, so it carries an error of its own, and this error is controlled by \(k\) alone. Applying \(\Var [\bar x]\approx \sigma ^2/n\) to the average (1.22), the std of a simulated mean is

\begin{equation} \label {eq-mc-error-mean} \sqrt {\Var [\bar {g}_k]} = \frac {\sqrt {\Var [g]}}{\sqrt {k}}, \end{equation}

where \(\Var [g]\) is the spread (1.23) of the individual \(g_j\): the accuracy of a simulated mean is the spread of the statistic divided by \(\sqrt {k}\). Similarly, the relative error of a simulated std \(s_g\) is

\begin{equation} \label {eq-mc-error-std} \frac {\sqrt {\Var [s_g]}}{s_g}\approx \frac {1}{\sqrt {2k}}. \end{equation}

Both decay as \(1/\sqrt {k}\), so halving the simulation error costs four times more experiments. Representative values are listed in Table 1.3 and illustrated in Fig. 1.12 for the simulated mean and in Fig. 1.13 for the simulated std.

The consequence is that a simulated value is never exact, and the number of correct digits it carries is set by \(k\). In Table 1.2, for instance, \(k=5000\) yields a std error of about \(1\%\), which is why the \(n=100\) row reads \(\bar s^2_{\mathrm {unbiased}}=1.01\) rather than the exact \(1.00\).

1.6.2 Reproducibility

A computer draws pseudo-random numbers, produced by a deterministic algorithm from an initial value called the seed. Fixing the seed makes the whole simulation exactly repeatable, which is a requirement for a reported result, not an option.

• • Python: numpy.random.default_rng(seed).

• • Matlab: rng(seed).