Machine Learning & Signals Learning

3 Two Random Variables

3.1 Discrete Random Variables

3.1.1 Joint Characterization

Joint PDF: The probability that the experiment yields the pair of values \((X=x_k,\,Y=y_j)\) is

\begin{equation} \label {eq:rand2:joint_pdf} p_{XY}[x_k,y_j] = \Pr [X=x_k,\,Y=y_j], \end{equation}

defined for every possible index pair \((j,k)\). The pair of values is the intersection of two events, \(\left \lbrace X=x_k\right \rbrace \cap \left \lbrace Y=y_j\right \rbrace \).

Properties of the joint PDF: Range of values and “sum” of values,

\begin{align} 0\le p_{XY}[x_k,y_j] &\le 1 \quad \forall j,k\\ \sum _{j,k} p_{XY}[x_k,y_j] &= 1 \end{align} where the summation runs over every possible index pair \((j,k)\).

Joint CDF: The joint cumulative distribution function of two discrete random variables is

\begin{equation} \label {eq:rand2:joint_cdf} F_{XY}(x,y) = \Pr (X\le x,\,Y\le y) \end{equation}

Link between the joint PDF and the joint CDF: The joint CDF accumulates the joint PDF, exactly as in the single-variable case (Feature 1.2.1), except that the accumulation now runs over two indices,

\begin{equation} F_{XY}(x,y) = \sum _{\substack {j,k\\ x_k\le x\,\cap \, y_j\le y}} p_{XY}[x_k,y_j] \end{equation}

Expectation of a function of two random variables: For any real-valued function \(g(x,y)\), the two-variable form of Eq. (1.11) is

\begin{equation} \label {eq:rand2:Exy_disc} \E [{g(X,Y)}] = \sum _j\sum _k g(x_k,y_j)\,p_{XY}[x_k,y_j] \end{equation}

3.1.2 Marginal Distributions

Marginal distribution: The marginal distribution of one variable is obtained by summing the joint distribution over all values of the other one,

\begin{align} p_X[x_k] &=\sum _j p_{XY}[x_k,y_j]\\ p_Y[y_j] &=\sum _k p_{XY}[x_k,y_j]\\ F_X(x) &= F_{XY}(x,\infty )\\ F_Y(y) &= F_{XY}(\infty ,y) \end{align}

3.1.3 Statistical Independence

Independent variables: \(X\) and \(Y\) are statistically independent if and only if the joint distribution factors into the product of the marginals,

\begin{align} p_{XY}[x_k,y_j] &= p_X[x_k]\,p_Y[y_j]\\ F_{XY}(x,y) &= F_X(x)\,F_Y(y) \end{align}

The marginals were derived from the joint distribution. The natural question is whether the derivation can be run backwards, that is whether

\[ p_{XY}[x_k,y_j]\; \overset {\mathbf {?}}{\longleftarrow }\; p_{X}[x_k],\;p_{Y}[y_j]. \]

3.2 Continuous Random Variables

3.2.1 Joint Characterization

Joint CDF: The definition of the joint CDF, Eq. (3.4), is the same for continuous and for discrete variables,

\begin{equation} F_{XY}(x,y) = \Pr (X\le x,\,Y\le y) \end{equation}

Joint PDF: The joint probability density function \(f_{XY}(x,y)\) is the mixed derivative of the joint CDF,

\begin{equation} f_{XY}(x,y)=\frac {\partial ^{2}F_{XY}(x,y)}{\partial x\,\partial y} \end{equation}

Link between the joint PDF and the joint CDF: Integration replaces the summation of the discrete case,

\begin{equation} F_{XY}(x,y)=\int _{-\infty }^{x}\int _{-\infty }^{y}f_{XY}(s,t)\,dt\,ds \end{equation}

Properties of the joint PDF: Range of values and “sum” of values,

\begin{align} f_{XY}(x,y) &\ge 0 \\ \int _{-\infty }^\infty \int _{-\infty }^\infty f_{XY}(x,y)\,dx\,dy &= 1 =F_{XY}(\infty ,\infty ) \end{align}

Marginal distribution: The marginal distributions are obtained by integrating the other variable out,

\begin{align} f_X(x) &= \int _{-\infty }^{\infty }f_{XY}(x,y)\,dy, \\[4pt] f_Y(y) &= \int _{-\infty }^{\infty }f_{XY}(x,y)\,dx, \\[4pt] F_X(x) &= F_{XY}(x,\infty ), \\[4pt] F_Y(y) &= F_{XY}(\infty ,y) \end{align}

Statistical independence: Two continuous random variables \(X\) and \(Y\) are statistically independent if and only if

\begin{align} f_{XY}(x,y) &= f_X(x)\,f_Y(y), \label {eq:rand2:indep}\\[4pt] F_{XY}(x,y) &= F_X(x)\,F_Y(y) \end{align} Only in this case can the joint distribution be reconstructed from its marginals.

3.2.2 Expectation

Expectation of a function of two random variables: The continuous counterpart of Eq. (3.6) is

\begin{equation} \label {eq:rand2:Exy_cont} \E [{g(X,Y)}] = \iint _{-\infty }^\infty g(x,y)\,f_{XY}(x,y)\,dx\,dy \end{equation}

Linearity of the expectation: The expectation of a sum is the sum of the expectations, for any pair of random variables,

\begin{equation} \label {eq:rand2:linearity} \E [{aX+bY}] = a\E [X] + b\E [Y] \end{equation}

Expectation of a product of independent variables: For two independent random variables and a separable function \(g(x,y)=g_1(x)\,g_2(y)\), the expectation factors,

\begin{equation} \label {eq:rand2:Eg1g2} \E [{g_1(X)g_2(Y)}] = \E [{g_1(X)}]\,\E [{g_2(Y)}] \end{equation}

The special case used throughout this chapter, valid for independent variables only, is

\begin{equation} \label {eq:rand2:ExyExEy} \E [XY] = \E [X]\,\E [Y] \end{equation}

3.2.3 Bi-variate Uniform Distribution

Bi-variate uniform distribution: The joint PDF of an axis-aligned two-dimensional uniform law over the rectangle \([a,b]\times [c,d]\) is

\begin{equation} f_{XY}(x,y)= \begin{cases} \dfrac {1}{b-a}\,\dfrac {1}{d-c}, & a \le x \le b,\; c \le y \le d,\\[6pt] 0, & \text {otherwise} \end {cases} \end{equation}

3.3 Covariance

3.3.1 Definition

Covariance: The covariance of two random variables \(X\) and \(Y\) is

\begin{equation} \label {eq:rand2:cov_def} \Cov [X,Y] = \E [{\bigl (X-\E [X]\bigr )\bigl (Y-\E [Y]\bigr )}] \end{equation}

Covariance, alternative definition: The form used in practice, because it needs only the three moments \(\E [XY]\), \(\E [X]\) and \(\E [Y]\), is

\begin{equation} \Cov [X,Y] = \E [XY]-\E [X]\,\E [Y] \end{equation}

Properties of the covariance: The covariance is symmetric, it is blind to a shift of either variable, and it scales with both,

\begin{align} \Cov [X,X] & = \Var [X] \label {eq:rand2:covxx} \\[4pt] \Cov [X,Y] & = \Cov [Y,X] \label {eq:rand2:covxy}\\[4pt] \Cov [X,a] & = 0 \\[4pt] \Cov [aX,bY] & = ab\,\Cov [X,Y] \\[4pt] \Cov [{X+a,\,Y+b}] & = \Cov [X,Y] \\[4pt] \Var [{X\pm Y}] & = \Var [X]+\Var [Y] \pm 2\,\Cov [X,Y] \label {eq:rand2:sum_cov} \end{align}

Covariance of independent variables: Two statistically independent random variables always satisfy

\begin{equation} \Cov [X,Y]=0 \end{equation}

Variance of a sum of independent variables: For independent variables only, the cross term of Eq. (3.32f) vanishes and the variance becomes additive,

\begin{equation} \label {eq:rand2:independence_var} \Var [{aX+bY}] = a^{2}\Var [X] + b^{2}\Var [Y] \end{equation}

Cauchy-Schwarz inequality: The mixed moment and the covariance are both bounded by the corresponding second moments,

\begin{align} \bigl (\E [XY]\bigr )^2 &\le \E [X^{2}]\,\E [Y^{2}],\\[6pt] \bigl (\Cov [X,Y]\bigr )^2 &\le \Var [X]\,\Var [Y]\label {eq:rand2:cauchy} \end{align}

3.3.2 Correlation Coefficient

Correlation coefficient: The correlation coefficient, also called the Pearson product-moment correlation coefficient, is the covariance normalized by the two standard deviations,

\begin{equation} \label {eq:rand2:rho} \rho _{XY}=\frac {\Cov [X,Y]}{\sqrt {\Var [X]\,\Var [Y]}} \end{equation}

Range of the correlation coefficient: The correlation coefficient is bounded,

\begin{equation} \abs {\rho _{XY}}\le 1 \end{equation}

which follows directly from dividing the Cauchy-Schwarz inequality (3.35b) by \(\Var [X]\Var [Y]\).

Perfect linear dependence: The bound is attained, \(\rho _{XY}=\pm 1\), if and only if \(Y\) is an exact linear function of \(X\). The sign of \(\rho _{XY}\) is the sign of the slope.

Uncorrelated variables: Two random variables are uncorrelated, meaning that they exhibit no linear relation, when

\begin{equation} \label {eq:rand2:uncorrelated} \Cov [X,Y]=\rho _{XY}=0 \end{equation}

Orthogonal variables: Two random variables are orthogonal when their mixed moment vanishes,

\begin{equation} \E [XY] = 0 \end{equation}

For variables with zero mean, orthogonality and uncorrelatedness coincide.

3.4 Bivariate Gaussian Distribution

Sum of independent Gaussian variables: A weighted sum of independent Gaussian variables is again Gaussian. For \(X \sim N(\mu _x,\sigma _x^2)\) and \(Y\sim N(\mu _y,\sigma _y^2)\) independent,

\begin{equation} \label {eq:rand2:gauss_sum} aX+bY \sim N\bigl (a\mu _x + b\mu _y,\;a^2\sigma _x^2 + b^2\sigma _y^2\bigr ) \end{equation}

The mean follows from Eq. (3.22) and the variance from Eq. (3.34). What is new is that the shape of the distribution is preserved.

Jointly Gaussian variables: Random variables \(X_1,X_2,\ldots ,X_N\), dependent or not, are jointly Gaussian if and only if every linear combination

\[ \sum _{m=1}^N a_m X_m, \qquad a_m\in \mathbb {R}, \]

is a Gaussian random variable.

Bivariate Gaussian PDF: A jointly Gaussian pair is fully described by five numbers: the two means, the two variances, and the correlation coefficient. Its joint PDF is

\begin{multline} \label {eq:rand2:gauss_pdf} f_{X_1X_2}(x_1,x_2) = \frac {1}{2 \pi \sigma _1 \sigma _2 \sqrt {1-\rho ^2}} \exp \left ( -\frac {1}{2(1-\rho ^2)}\times \right . \\ \left . \times \left [ \frac {(x_1-\mu _1)^2}{\sigma _1^2} + \frac {(x_2-\mu _2)^2}{\sigma _2^2} - \frac {2\rho (x_1-\mu _1)(x_2-\mu _2)}{\sigma _1 \sigma _2} \right ] \right ) \end{multline} where \(\mu _i=\E [X_i]\), \(\sigma _i^2=\Var [X_i]\) and \(\rho =\rho _{X_1X_2}\). The construction that produces this density is given in the appendix to this chapter.

Marginal distributions: The marginal distributions of a jointly Gaussian pair are Gaussian, \(X_i\sim N(\mu _i,\sigma _i^2)\).

Jointly Gaussian uncorrelated variables: For a jointly Gaussian pair, and only for such a pair, uncorrelatedness and independence are the same thing,

\begin{equation} \rho _{XY}=0 \iff X,Y \text { independent} \end{equation}

Setting \(\rho =0\) in Eq. (3.41) indeed factors the joint density into the product of the two marginals.

3.5 Exercises

3.1 (Covariance) The following relations hold for the random variables \(X,Y\),

\begin{align*} \Var [{X+2Y}] &= 40\\ \Var [{X-2Y}] &= 20\\ \Var [X] &= \Var [Y] \end{align*} Compute \(\Var [X]\) and \(\Cov [X,Y]\) numerically.

3.2 (Correlation coefficient of discrete variables) The joint distribution of the discrete random variables \(X,Y\) is

\[ p_{XY}[x,y] = \begin {cases} \Pr [X=0,Y=-1] = \tfrac 13,\\[3pt] \Pr [X=-1,Y=0] = \tfrac 13,\\[3pt] \Pr [X=1,Y=1] = \tfrac 16,\\[3pt] \Pr [X=-1,Y=-1] = \tfrac 16 \end {cases} \]

Compute the correlation coefficient \(\rho _{XY}\).

3.3 (Product and sum of uniform variables) Let \(n\) independent random variables \(X_i\sim \mathrm {U}[0,1]\), \(i = 1,\ldots ,n\), be given, and define

\begin{align*} Y &= X_1\cdot X_2\cdot \ldots \cdot X_n,\\ Z &= X_1 + X_2 + \ldots + X_n \end{align*} Compute \(\E [Y],\;\Var [Y],\;\E [Z],\;\Var [Z]\).

3.4 (Correlation coefficient) Given the joint PDF

\[ f_{XY}(x,y) = \exp \!\bigl (-\tfrac {4x+y}{2}\bigr )\,u(x)\,u(y), \]

where \(u(\cdot )\) is the unit step function, find the correlation coefficient \(\rho _{XY}\).

3.5 (Linearly related variables) Given the relation

\[ Y = -6X + 22, \]

with \(\E [X] = 3\) and \(\Var [X] = 2\):

• (a) Compute \(\E [Y]\) and \(\Var [Y]\).

• (b) Compute \(\Cov [X,Y]\) and \(\rho _{XY}\), and explain the result.

3.6 (Noisy radius measurements) Radii of circles are measured. The true radii are distributed as \(r \sim \mathrm {U}[a=1,b=3]\) and the measurement error is \(n \sim N(\mu =0,\sigma ^2=0.01)\). The true radius and the error are independent, and the observed measurement is \(X = r + n\). The circumference and the area of the observed circle are

\[ C = 2\pi X, \qquad A = \pi X^2 \]

Compute:

• (a) The expected circumference \(\E [C]\).

• (b) The variance of the circumference \(\Var [C]\).

• (c) The expected area \(\E [A]\).

• (d) The covariance \(\Cov [{X,r}]\).

3.7 (Uncorrelated but dependent) Define \(\Theta \sim \mathrm {U}[-\pi ,\pi ]\) and

\[ X = \cos (\Theta ),\qquad Y = \sin (\Theta ) \]

• (a) Prove that \(X\) and \(Y\) are uncorrelated.

• (b) Compute \(\E [X^2]\) and \(\E [Y^2]\).

• (c) Show by a counterexample that \(X\) and \(Y\) are not independent. Hint: for independent variables Eq. (3.23) must hold for every pair of functions \(g_1,g_2\).

3.8 (Variance of a difference) Let \(X,Y\) be arbitrary random variables.

• (a) Derive an expression for \(\Var [{X - Y}]\).

• (b) Now assume that \(X\sim N(\mu _1,\sigma _1^2)\) and \(Y\sim N(\mu _2,\sigma _2^2)\) are jointly Gaussian with correlation coefficient \(\rho \). Express \(\E [{X - Y}]\) and \(\Var [{X - Y}]\) in terms of the distribution parameters.

3.9 (CDF of a Gaussian pair) Two jointly Gaussian random variables are given, \(X\sim N\bigl (\tfrac 12,1\bigr )\) and \(Y\sim N\bigl (\tfrac 12,2^2\bigr )\), with correlation coefficient \(\rho =\tfrac 12\). Denote their CDFs by \(F_X(x),F_Y(y),F_{XY}(x,y)\) and express the probabilities

\[ \Pr (-1<X<1,\;Y<1), \quad \Pr (X<1,\;-1<Y) \]

in terms of them. Hint: use Example 3.7.

3.10 (Rotating a Gaussian pair) Let

\[ X_1\sim N(m,\sigma ^2),\quad X_2\sim N(m,\sigma ^2) \]

be jointly Gaussian. Do not assume that they are independent. Define the rotated pair

\begin{align*} V &= X_1\cos (\theta ) + X_2\sin (\theta ),\\ W &= -\,X_1\sin (\theta ) + X_2\cos (\theta ) \end{align*}

• (a) Find every \(\theta \in [-\pi ,\pi ]\) for which \(V\) and \(W\) are independent.

• (b) Give the joint distribution of \((V,W)\) and its parameters. Here you may write \(\E [{X_1X_2}]=b\), and you may not assume the special value of \(\theta \) found in part (a).

3.11 (Sum of Bernoulli variables) Three independent Bernoulli variables are given, \(X_1\sim \mathrm {Ber}(p_1)\), \(X_2\sim \mathrm {Ber}(p_2)\), \(X_3\sim \mathrm {Ber}(p_3)\).

• (a) Compute the probabilities \(\Pr (X_1+X_2+X_3 = 1)\) and \(\Pr (X_1+X_2+X_3 = 2)\).

• (b) Verify the result by simulation, using values \(p_i\ne 0.5\).

3.12 (Simulation) Two independent random variables \(X,Y\sim \mathrm {U}[0,1]\) are given. Show by simulation that

\begin{equation*} \Pr \left (X^{\frac {1}{3}}>\frac {Y}{2}\right )=\frac {31}{32} = 0.96875 \end{equation*}

3.13 (Simulation of a discrete pair) Simulate the sum of two fair dice and plot the empirical distribution of the sum. Compare the result to the exact distribution obtained by counting the \(36\) equally likely pairs.

Appendices

3.A Constructing a Bivariate Gaussian Pair

Eq. (3.41) was stated without derivation. The construction below shows where its five parameters come from: every jointly Gaussian pair is a linear mixture of two independent standard Gaussian variables.

Let \(W_1,W_2\sim N(0,1)\) be independent, and build

\begin{align} X_1 &= \mu _1 + \sigma _{11}W_1 + \sigma _{12}W_2\\ X_2 &= \mu _2 + \sigma _{21}W_1 + \sigma _{22}W_2 \end{align} Each of \(X_1,X_2\) is a weighted sum of independent Gaussian variables, hence Gaussian by Eq. (3.40). The same holds for any linear combination \(a_1X_1+a_2X_2\), since it too reduces to a weighted sum of \(W_1\) and \(W_2\). By Definition 3.4 the pair is therefore jointly Gaussian.

The means follow from the linearity of the expectation and \(\E [W_i]=0\),

\begin{align} \E [X_1] &= \mu _1 + \sigma _{11}\E [W_1] + \sigma _{12}\E [W_2] = \mu _1\\ \E [X_2] &= \mu _2 \end{align} The variances follow from Eq. (3.34) and \(\Var [W_i]=1\),

\begin{align} \Var [X_1] &= \sigma _{11}^2\Var [W_1]+\sigma _{12}^2\Var [W_2] = \sigma _{11}^2+\sigma _{12}^2 \equiv \sigma _1^2\\ \Var [X_2] &= \sigma _{21}^2+\sigma _{22}^2 \equiv \sigma _2^2 \end{align} For the covariance, expand the product. The cross terms in \(W_1W_2\) drop out because the two are independent with zero mean, and the terms \(\E [W_i^2]=1\) survive,

\begin{align} \E [{X_1X_2}] &=\mu _1\mu _2 + \sigma _{11}\sigma _{21}\E [W_1^2] + \sigma _{12}\sigma _{22}\E [W_2^2]\nonumber \\ &=\mu _1\mu _2 + \sigma _{11}\sigma _{21} + \sigma _{12}\sigma _{22}\\ \Cov [X_1,X_2] &= \E [{X_1X_2}] - \E [X_1]\E [X_2]\nonumber \\ &= \sigma _{11}\sigma _{21} + \sigma _{12}\sigma _{22} \end{align} and the correlation coefficient is

\begin{equation} \rho = \frac {\Cov [X_1,X_2]}{\sqrt {\Var [X_1]\Var [X_2]}} = \frac {\sigma _{11}\sigma _{21} + \sigma _{12}\sigma _{22}}{\sigma _1\sigma _2} \end{equation}

Four mixing weights \(\sigma _{11},\sigma _{12},\sigma _{21},\sigma _{22}\) produce only three distinct quantities, \(\sigma _1^2\), \(\sigma _2^2\) and \(\rho \). Together with the two means \(\mu _1,\mu _2\), these are exactly the five parameters appearing in Eq. (3.41). Changing the weights while keeping the three derived quantities fixed amounts to rotating \((W_1,W_2)\), which leaves the joint density unchanged.