Machine Learning & Signals Learning

$\newcommand{\footnotename}{footnote}$ $\def \LWRfootnote {1}$ $\newcommand {\footnote }[2][\LWRfootnote ]{{}^{\mathrm {#1}}}$ $\newcommand {\footnotemark }[1][\LWRfootnote ]{{}^{\mathrm {#1}}}$ $\let \LWRorighspace \hspace $ $\renewcommand {\hspace }{\ifstar \LWRorighspace \LWRorighspace }$ $\newcommand {\TextOrMath }[2]{#2}$ $\newcommand {\mathnormal }[1]{{#1}}$ $\newcommand \ensuremath [1]{#1}$ $\newcommand {\LWRframebox }[2][]{\fbox {#2}} \newcommand {\framebox }[1][]{\LWRframebox } $ $\newcommand {\setlength }[2]{}$ $\newcommand {\addtolength }[2]{}$ $\newcommand {\setcounter }[2]{}$ $\newcommand {\addtocounter }[2]{}$ $\newcommand {\arabic }[1]{}$ $\newcommand {\number }[1]{}$ $\newcommand {\noalign }[1]{\text {#1}\notag \\}$ $\newcommand {\cline }[1]{}$ $\newcommand {\directlua }[1]{\text {(directlua)}}$ $\newcommand {\luatexdirectlua }[1]{\text {(directlua)}}$ $\newcommand {\protect }{}$ $\def \LWRabsorbnumber #1 {}$ $\def \LWRabsorbquotenumber "#1 {}$ $\newcommand {\LWRabsorboption }[1][]{}$ $\newcommand {\LWRabsorbtwooptions }[1][]{\LWRabsorboption }$ $\def \mathchar {\ifnextchar "\LWRabsorbquotenumber \LWRabsorbnumber }$ $\def \mathcode #1={\mathchar }$ $\let \delcode \mathcode $ $\let \delimiter \mathchar $ $\def \oe {\unicode {x0153}}$ $\def \OE {\unicode {x0152}}$ $\def \ae {\unicode {x00E6}}$ $\def \AE {\unicode {x00C6}}$ $\def \aa {\unicode {x00E5}}$ $\def \AA {\unicode {x00C5}}$ $\def \o {\unicode {x00F8}}$ $\def \O {\unicode {x00D8}}$ $\def \l {\unicode {x0142}}$ $\def \L {\unicode {x0141}}$ $\def \ss {\unicode {x00DF}}$ $\def \SS {\unicode {x1E9E}}$ $\def \dag {\unicode {x2020}}$ $\def \ddag {\unicode {x2021}}$ $\def \P {\unicode {x00B6}}$ $\def \copyright {\unicode {x00A9}}$ $\def \pounds {\unicode {x00A3}}$ $\let \LWRref \ref $ $\renewcommand {\ref }{\ifstar \LWRref \LWRref }$ $ \newcommand {\multicolumn }[3]{#3}$ $\require {textcomp}$ $ \newcommand {\abs }[1]{\lvert #1\rvert } $ $ \DeclareMathOperator {\sign }{sign} $ $\newcommand {\intertext }[1]{\text {#1}\notag \\}$ $\let \Hat \hat $ $\let \Check \check $ $\let \Tilde \tilde $ $\let \Acute \acute $ $\let \Grave \grave $ $\let \Dot \dot $ $\let \Ddot \ddot $ $\let \Breve \breve $ $\let \Bar \bar $ $\let \Vec \vec $ $\newcommand {\bm }[1]{\boldsymbol {#1}}$ $\require {physics}$ $\newcommand {\LWRphystrig }[2]{\ifblank {#1}{\textrm {#2}}{\textrm {#2}^{#1}}}$ $\renewcommand {\sin }[1][]{\LWRphystrig {#1}{sin}}$ $\renewcommand {\sinh }[1][]{\LWRphystrig {#1}{sinh}}$ $\renewcommand {\arcsin }[1][]{\LWRphystrig {#1}{arcsin}}$ $\renewcommand {\asin }[1][]{\LWRphystrig {#1}{asin}}$ $\renewcommand {\cos }[1][]{\LWRphystrig {#1}{cos}}$ $\renewcommand {\cosh }[1][]{\LWRphystrig {#1}{cosh}}$ $\renewcommand {\arccos }[1][]{\LWRphystrig {#1}{arcos}}$ $\renewcommand {\acos }[1][]{\LWRphystrig {#1}{acos}}$ $\renewcommand {\tan }[1][]{\LWRphystrig {#1}{tan}}$ $\renewcommand {\tanh }[1][]{\LWRphystrig {#1}{tanh}}$ $\renewcommand {\arctan }[1][]{\LWRphystrig {#1}{arctan}}$ $\renewcommand {\atan }[1][]{\LWRphystrig {#1}{atan}}$ $\renewcommand {\csc }[1][]{\LWRphystrig {#1}{csc}}$ $\renewcommand {\csch }[1][]{\LWRphystrig {#1}{csch}}$ $\renewcommand {\arccsc }[1][]{\LWRphystrig {#1}{arccsc}}$ $\renewcommand {\acsc }[1][]{\LWRphystrig {#1}{acsc}}$ $\renewcommand {\sec }[1][]{\LWRphystrig {#1}{sec}}$ $\renewcommand {\sech }[1][]{\LWRphystrig {#1}{sech}}$ $\renewcommand {\arcsec }[1][]{\LWRphystrig {#1}{arcsec}}$ $\renewcommand {\asec }[1][]{\LWRphystrig {#1}{asec}}$ $\renewcommand {\cot }[1][]{\LWRphystrig {#1}{cot}}$ $\renewcommand {\coth }[1][]{\LWRphystrig {#1}{coth}}$ $\renewcommand {\arccot }[1][]{\LWRphystrig {#1}{arccot}}$ $\renewcommand {\acot }[1][]{\LWRphystrig {#1}{acot}}$ $\require {cancel}$ $\newcommand *{\underuparrow }[1]{{\underset {\uparrow }{#1}}}$ $\DeclareMathOperator *{\argmax }{argmax}$ $\DeclareMathOperator *{\argmin }{arg\,min}$ $\def \E [#1]{\mathbb {E}\!\left [ #1 \right ]}$ $\def \Var [#1]{\operatorname {Var}\!\left [ #1 \right ]}$ $\def \Cov [#1]{\operatorname {Cov}\!\left [ #1 \right ]}$ $\newcommand {\floor }[1]{\lfloor #1 \rfloor }$ $\newcommand {\DTFTH }{ H \brk 1{e^{j\omega }}}$ $\newcommand {\DTFTX }{ X\brk 1{e^{j\omega }}}$ $\newcommand {\DFTtr }[1]{\mathrm {DFT}\left \{#1\right \}}$ $\newcommand {\DTFTtr }[1]{\mathrm {DTFT}\left \{#1\right \}}$ $\newcommand {\DTFTtrI }[1]{\mathrm {DTFT^{-1}}\left \{#1\right \}}$ $\newcommand {\Ftr }[1]{ \mathcal {F}\left \{#1\right \}}$ $\newcommand {\FtrI }[1]{ \mathcal {F}^{-1}\left \{#1\right \}}$ $\newcommand {\Zover }{\overset {\mathscr Z}{\Longleftrightarrow }}$ $\renewcommand {\real }{\mathbb {R}}$ $\newcommand {\ba }{\mathbf {a}}$ $\newcommand {\bb }{\mathbf {b}}$ $\newcommand {\bd }{\mathbf {d}}$ $\newcommand {\be }{\mathbf {e}}$ $\newcommand {\bh }{\mathbf {h}}$ $\newcommand {\bn }{\mathbf {n}}$ $\newcommand {\bq }{\mathbf {q}}$ $\newcommand {\br }{\mathbf {r}}$ $\newcommand {\bt }{\mathbf {t}}$ $\newcommand {\bv }{\mathbf {v}}$ $\newcommand {\bw }{\mathbf {w}}$ $\newcommand {\bx }{\mathbf {x}}$ $\newcommand {\bxx }{\mathbf {xx}}$ $\newcommand {\bxy }{\mathbf {xy}}$ $\newcommand {\by }{\mathbf {y}}$ $\newcommand {\byy }{\mathbf {yy}}$ $\newcommand {\bz }{\mathbf {z}}$ $\newcommand {\bA }{\mathbf {A}}$ $\newcommand {\bB }{\mathbf {B}}$ $\newcommand {\bI }{\mathbf {I}}$ $\newcommand {\bK }{\mathbf {K}}$ $\newcommand {\bP }{\mathbf {P}}$ $\newcommand {\bQ }{\mathbf {Q}}$ $\newcommand {\bR }{\mathbf {R}}$ $\newcommand {\bU }{\mathbf {U}}$ $\newcommand {\bW }{\mathbf {W}}$ $\newcommand {\bX }{\mathbf {X}}$ $\newcommand {\bY }{\mathbf {Y}}$ $\newcommand {\bZ }{\mathbf {Z}}$ $\newcommand {\balpha }{\bm {\alpha }}$ $\newcommand {\bth }{{\bm {\theta }}}$ $\newcommand {\bepsilon }{{\bm {\epsilon }}}$ $\newcommand {\bmu }{{\bm {\mu }}}$ $\newcommand {\bOne }{\mathbf {1}}$ $\newcommand {\bZero }{\mathbf {0}}$ $\newcommand {\loss }{\mathcal {L}}$ $\newcommand {\appropto }{\mathrel {\vcenter { \offinterlineskip \halign {\hfil $##$\cr \propto \cr \noalign {\kern 2pt}\sim \cr \noalign {\kern -2pt}}}}}$ $\newcommand {\SSE }{\mathrm {SSE}}$ $\newcommand {\MSE }{\mathrm {MSE}}$ $\newcommand {\RMSE }{\mathrm {RMSE}}$ $\newcommand {\toprule }[1][]{\hline }$ $\let \midrule \toprule $ $\let \bottomrule \toprule $ $\def \LWRbooktabscmidruleparen (#1)#2{}$ $\newcommand {\LWRbooktabscmidrulenoparen }[1]{}$ $\newcommand {\cmidrule }[1][]{\ifnextchar (\LWRbooktabscmidruleparen \LWRbooktabscmidrulenoparen }$ $\newcommand {\morecmidrules }{}$ $\newcommand {\specialrule }[3]{\hline }$ $\newcommand {\addlinespace }[1][]{}$ $\newcommand {\LWRsubmultirow }[2][]{#2}$ $\newcommand {\LWRmultirow }[2][]{\LWRsubmultirow }$ $\newcommand {\multirow }[2][]{\LWRmultirow }$ $\newcommand {\mrowcell }{}$ $\newcommand {\mcolrowcell }{}$ $\newcommand {\STneed }[1]{}$ $\newcommand {\tcbset }[1]{}$ $\newcommand {\tcbsetforeverylayer }[1]{}$ $\newcommand {\tcbox }[2][]{\boxed {\text {#2}}}$ $\newcommand {\tcboxfit }[2][]{\boxed {#2}}$ $\newcommand {\tcblower }{}$ $\newcommand {\tcbline }{}$ $\newcommand {\tcbtitle }{}$ $\newcommand {\tcbsubtitle [2][]{\mathrm {#2}}}$ $\newcommand {\tcboxmath }[2][]{\boxed {#2}}$ $\newcommand {\tcbhighmath }[2][]{\boxed {#2}}$

18 Exponential Smoothing

18.1 Preface

Basic models:

• Naïve, Eq. (14.50).
• Moving average, Eq. (14.49).
• Signal average,
$\seteqnumber{0}{}{0}$
\begin{equation} \hat {y}[n] = \frac {1}{n}\sum _{i=0}^{n-1}y[n] \end{equation}

18.2 Exponential Smoothing

Goal: Use decaying sequence of weights for historic values.

The sequence is of the form

\begin{equation} \sum _{k=0}^\infty (1-\alpha )^k = \frac {1}{\alpha },\quad 0 < \alpha < 1 \end{equation}

For example, for $\alpha = \frac {1}{2}$,

\begin{equation} 1+\frac {1}{2} + \frac {1}{4} + \frac {1}{8} + \cdots = 2 \end{equation}

Multiply both sides of the sum by $\alpha $ and we can write that

\begin{equation} \sum _{k=0}^\infty \alpha (1-\alpha )^k = \alpha + \alpha (1-\alpha ) + \alpha (1-\alpha )^2 + \cdots + \alpha (1-\alpha )^k + \cdots = 1 \end{equation}

The resulting prediction (forecasting) model is

\begin{equation} \begin{aligned} \hat {y}[n+1] &= y[n]\alpha \\ &+ y[n-1]\alpha (1-\alpha )\\ &+ y[n-2]\alpha (1-\alpha )^2\\ & + \cdots \\ &+ y[n-k]\alpha (1-\alpha )^k + \cdots \end {aligned} \end{equation}

The difference equation form of this model is

\begin{equation} \hat {y}[n+1] = \alpha y[n] + (1-\alpha )y[n-1] \end{equation}

that starts with some $y[0]$ and $y[1]$.

The value of $\alpha \in [0,1]$ may be considered as memory decay rate. For higher $\alpha $ the method “forgets” faster. For $\alpha =1$, just one last sample is used, $\hat {y}[n+1] = y[n]$. For $\alpha \lesssim 1$ the sequence decay is very fast. For $\alpha \gtrsim 0$ the significant number of previous samples is involved.

This method is called simple exponential smoothing (SES).

Learning parameter

The learning of $\alpha $ is numerical minimization of some loss function,

\begin{equation} \arg \min \limits _{\alpha }\sum _{i=2}^L\loss (y[i],\hat {y}[i]) \end{equation}

The most common loss is MSE,

\begin{equation} \loss (y[i],\hat {y}[i]) = (y[i]-\hat {y}[i])^2 \end{equation}

18.3 Double Exponential Smoothing

Trend

Some signals (time-series) have trend (some local slope). The basic model with trend/slope $b$ is given by

\begin{equation} y[n] = a + bn + \epsilon [n] \end{equation}

Let’s define

\begin{equation} y[n] - y[n-1] = a + bn - (a - b(n-1)) = b \end{equation}

This difference is called de-trending.

18.3.1 Method

The use of trend in smoothing results. The prediction is a combination of level and trend,

\begin{equation} \hat {y}[n+1] = \ell _n + b_n \end{equation}

Level The level is given by

\begin{equation} \begin{aligned} \ell _n &= \alpha \cdot \text {new information} + (1-\alpha )\cdot (\text {old level} + \text {trend})\\ &=\alpha y[n] + (1-\alpha )(\ell _{n-1} + b_{n-1}) \end {aligned} \end{equation}

The relation is similar to SES with trend “correction”.

Trend The trend can be used in smoothing by formula above. The “new” trend is given by difference $\ell _n-\ell _{n-1}$. The smoothing trend formula is given by

\begin{equation} \begin{aligned} b_n &=\beta \cdot \text {new trend} + (1-\beta )\cdot \text {old trend}\\ &= \beta (\ell _n-\ell _{n-1}) + (1-\beta )b_{n-1} \end {aligned} \end{equation}

Both of these functions are updated consequently with initial conditions of [?, Sec. 6.4.3.3]

\begin{equation} \begin{aligned} \ell _0 &= y[0]\\ b_0 &= \begin{cases} y[1]-y[0]\\[5pt] \frac {1}{3}\left [(y[1]-y[0])+ (y[2]-y[1])+(y[3]-y[2])\right ]=\frac {y[3]-y[0]}{3}\\[5pt] \dfrac {y[n-1]-y[0]}{n} \end {cases} \end {aligned} \end{equation}

This model is similar to ARIMA(0,1,1) model.

18.4 Triple Exponential Smoothing

18.4.1 Seasonality

Seasonal component is a repetitive component (cyclical repeating pattern) with some lower frequency, i.e. with period $M$ of tens of samples or more. To identify the value of $L$, the closest-to-zero peak of ACF can be used.

18.4.2 Method

The exponential smoothing is applied on seasonality component as well. The trend component is without change. New equations set is

\begin{align} \ell _n &=\alpha y[n] + (1-\alpha )(\ell _{n-1} + b_{n-1})\\ b_n &= \beta (\ell _n-\ell _{n-1}) + (1-\beta )b_{n-1} \end{align}

Moreover, it helps to make prediction for more steps in the future.