Random Vectors
A random vector is a multi-dimensional random variable.
Example (1.6.1)
we roll a fair die twice. Let X be the sum of the rolls and let Y be the absolute difference between the two rolls. Then (X,Y) is a 2-dimensional random vector. The joint pdf of (X,Y) is given by:
| 0 | 1 | 2 | 3 | 4 | 5 | |
|---|---|---|---|---|---|---|
| 2 | 1 | 0 | 0 | 0 | 0 | 0 |
| 3 | 0 | 2 | 0 | 0 | 0 | 0 |
| 4 | 1 | 0 | 2 | 0 | 0 | 0 |
| 5 | 0 | 2 | 0 | 2 | 0 | 0 |
| 6 | 1 | 0 | 2 | 0 | 2 | 0 |
| 7 | 0 | 2 | 0 | 2 | 0 | 2 |
| 8 | 1 | 0 | 2 | 0 | 2 | 0 |
| 9 | 0 | 2 | 0 | 2 | 0 | 0 |
| 10 | 1 | 0 | 2 | 0 | 0 | 0 |
| 11 | 0 | 2 | 0 | 0 | 0 | 0 |
| 12 | 1 | 0 | 0 | 0 | 0 | 0 |
where every number is divided by 36.
all definitions are straightforward extensions of the one-dimensional case.
Example (1.6.2)
for a discrete random vector we have the pdf \(f(x,y) = P(X=x,Y=y)\).
Say f(4,0) = P(X=4, Y=0) = P({(2,2)}) = 1/36 or f(7,1) = P(X=7,Y=1) = P({(3,4),(4,3)}) = 1/18
Example (1.6.3)
Say \(f(x,y)=cxy\) is a pdf with \(x \in \{1,2,3\}\) and \(y \in \{0,2\}\). Find c.
\(1=\sum_{x,y} f(x,y) =\)
\(f(1,0)+f(1,2)+f(2,0)+f(2,2)+f(3,0)+f(3,2) =\)
\(c(1 \times 0+1 \times 2+2 \times 0+2 \times 2+3 \times 0+3 \times 2) = 12c\)
so \(c=1/12\)
Example (1.6.4)
Say \(f(x,y)=cxy, 0 \le x,y \le 1\) is a pdf. Find c.
\[ \begin{aligned} &1 =\iint_{\mathbb{R}^2} f(x,y)d(x,y) =\\ &\int_{-\infty}^\infty \int_{-\infty}^\infty f(x,y)dxdy = \\ &\int_0^1\int_0^1 cxy dxdy = \\ &c\int_0^1 y \left(\int_0^1 x dx\right)dy = \\ &c\int_0^1 y \left( x^2/2|_0^1) \right)dy = \\ &c\int_0^1 y/2 dy = cy^2/4|_0^1=c/4\\ \end{aligned} \] so c=4.
Example (1.6.5)
Say \(f(x,y)=cxy, 0 \le x<y \le 1\) is a pdf. Find c.
\[ \begin{aligned} &1 =\iint_{\mathbb{R}^2} f(x,y)d(x,y) =\\ &\int_{-\infty}^\infty \int_{-\infty}^\infty f(x,y)dxdy = \\ &\int_0^1\int_0^y cxy dxdy = \\ &c\int_0^1 y \left(\int_0^y x dx\right)dy = \\ &c\int_0^1 y \left( x^2/2|_0^y) \right)dy = \\ &c\int_0^1 y^3/2 dy = cy^4/8|_0^1=c/8\\ \end{aligned} \]
so c=8.
Example (1.6.6)
Say (X,Y) is a discrete rv with joint pdf \(f(x,y)=cp^x, x,y \in \{0,1,..\}, y \le x, 0<p<1\). Find c
\[ \begin{aligned} &1 =\sum_{x,y} f(x,y)= \\ &\sum_{y=0}^\infty \sum_{x=y}^\infty cp^x = \\ &c\sum_{y=0}^\infty \left[\sum_{x=0}^\infty p^x-\sum_{x=0}^{y-1} p^x \right] = \\ &c\sum_{y=0}^\infty \left[ \frac{1}{1-p}-\frac{1-p^y}{1-p} \right] = \\ & \frac{c}{1-p} \sum_{y=0}^\infty p^y = \frac{c}{(1-p)^2} \end{aligned} \]
and so \(c=(1-p)^2\)
Example (1.6.7)
Say \((X,Y,Z)\) is a continuous rv with \(f(x,y,z) = c(x+y)z\) if \(0<x,y,z<1\) and 0 otherwise. Find c
\[ \begin{aligned} &1 =\int_0^1\int_0^1\int_0^1 c(x+y)z dxdydz = \\ &c\left[ \int_0^1\int_0^1 xz dxdz+\int_0^1\int_0^1 yz dydz\right] = \\ &2c\int_0^1\int_0^1 yz dydz = \\ &2c\int_0^1z/2 dz = c/2 \end{aligned} \]
so \(c=2\).
Example (1.6.8)
Let \((X,Y)\) be a discrete random vector with density f proportional to \(g(x, y)=x/y\), \(1\le x,y \le 10\). Find \(F(5, 7)\).
“proportional to” means that the density \(f(x,y)=cg(x,y)\) for some constant c. Let’s find this constant first. The easiest way to do this is using R:
x=1:10
y=1:10
z=outer(x, y, function(x,y) x/y)
sum(z)## [1] 161.0933so c=1/161.0933. Now for for \(F(5, 7)\) we find
sum(z[1:5, 1:7])/sum(z)## [1] 0.2414307Example (1.6.9)
Let \((X,Y)\) be a continuous random vector with density f proportional to \(g(x, y)=x/y\), \(1\le x,y \le 10\). Find \(F(5, 7)\).
\[ \begin{aligned} &F(x,y) = \int_{-\infty}^x \int_{-\infty}^y cg(u,v) dudv = \\ &c \left( \int_{1}^x u du \right)\left( \int_{1}^y 1/v dv\right) = \\ &c/2(x^2-1)\log y\\ &1=F(10,10)=c/2(10^2-1)\log 10=113.978c\\ &F(5, 7) = 1/(2*113.978)(5^2-1)\log 7=0.2048 \end{aligned} \]
Example (1.6.10)
Let \((X,Y)\) be a random vector with density f proportional to \(g(x, y)=x/y\), \(1\le x \le 10\) and \(y=1,2, .., 10\). Find \(F(5, 7)\).
So now we have a mixture of a continuous and a discrete random variable. So
\[ \begin{aligned} &F(x,y) = \sum_{i=1}^y \int_{1}^x cu/i du = \\ &c(x^2-1)/2 \sum_{i=1}^y1/i \\ &1=F(10,10) = c99/2*2.929=144.9839 \\ &F(5,7) = 1/144.9839 * (5^2-1)/2 \sum_{i=1}^71/i = 0.2146 \end{aligned} \] where we again find the sums using R:
sum(1/1:10)## [1] 2.928968sum(1/1:7)## [1] 2.592857Example (1.6.11)
Let’s extend the idea of a uniform random variable to two dimensions. To begin, let’s start with the unit square \([0,1]\times [0,1]\).
Again, the idea of uniform is taken to mean that the probability of a point \((x,y)\) being in some area is proportional to the size of the area. Therefore if A is some area in \([0,1]\times [0,1]\) (which has total area 1), we have
\(P((X,Y) \in A) = \text{area}(A)\)
say \(0<x,y<1\), then
\(F(x,y) = P(X<x,Y<y) = \text{area}([0,x]*[0,y]) = xy\)
\(f(x,y) = d^2/dxdy F(x,y) =d/dx[d/dy(xy)] = d/dx[x] = 1\)
Now say \((X,Y)\) is uniform on \(\{(x,y\}: 0<x<y^\alpha <1\}\) for some \(\alpha>0\). Find the joint pdf of \((X,Y)\).
First we need the total area:
\[\int_0^1\int_0^{y^\alpha} dxdy = \int_0^1 y^\alpha dy= \frac{y^{\alpha+1}}{\alpha+1}|_0^1=\frac1{\alpha+1}\]
so \(f(x,y)=\alpha+1\) if \(0<x<y^\alpha<1\) and 0 otherwise.
Marginal Distributions
Definition (1.6.12)
Say \((X,Y)\) is a discrete (continuous) r.v. with joint pdf f. Then the marginal pdf \(f_X(x)\) of \(X\) is given by
- X discrete:
\[f_X(x)=\sum_y f(x,y)\]
- X continuous
\[f_X(x)=\int_{-\infty}^\infty f(x,y) dy\]
Example (1.6.13)
Say X is the sum and Y is the absolute difference of two dice. If we add the row and column totals to the table above we get
| 0 | 1 | 2 | 3 | 4 | 5 | X | |
|---|---|---|---|---|---|---|---|
| 2 | 1 | 0 | 0 | 0 | 0 | 0 | 1 |
| 3 | 0 | 2 | 0 | 0 | 0 | 0 | 2 |
| 4 | 1 | 0 | 2 | 0 | 0 | 0 | 3 |
| 5 | 0 | 2 | 0 | 2 | 0 | 0 | 4 |
| 6 | 1 | 0 | 2 | 0 | 2 | 0 | 5 |
| 7 | 0 | 2 | 0 | 2 | 0 | 2 | 6 |
| 8 | 1 | 0 | 2 | 0 | 2 | 0 | 5 |
| 9 | 0 | 2 | 0 | 2 | 0 | 0 | 4 |
| 10 | 1 | 0 | 2 | 0 | 0 | 0 | 3 |
| 11 | 0 | 2 | 0 | 0 | 0 | 0 | 2 |
| 12 | 1 | 0 | 0 | 0 | 0 | 0 | 1 |
| Y | 6 | 10 | 8 | 6 | 4 | 2 | 36 |
and these are the marginals. For example we find \(f_X(2) = 1/36\) or \(f_Y(3) = 6/36\).
Example (1.6.14)
Say \((X,Y)\) is a rv with joint pdf \(f(x,y)=xy/12\) with \(x \in \{1,2,3\}\) and \(y \in \{0,2\}\) Now
\[f_X(3) = f(3,0) + f(3,2) = 3\times 0\times1/12 + 3\times2\times 1/12 = 6/12 = 1/2\]
and
\[f_Y(0) = f(1,0) + f(2,0) + f(3,0) = 0\]
Example (1.6.15)
Say \((X,Y)\) is a rv with joint pdf \(f(x,y)=8xy\), \(0 \le x<y \le 1\). Find \(f_Y(y)\)
\[f_Y(y)=\int_{-\infty}^\infty f(x,y) dx = \int_0^y 8xydx = 4y\times x^2|_0^y=4y^3\]
if \(0<y<1\).
Note that \(f_Y(y)\) is s proper pdf: \(f_Y (y) \ge 0\) for all y and
\[\int_0^1 4y^3 dy = y^4|_0^1=1\]
Example (1.6.16)
Say \((X,Y,Z)\) is a continuous rv with \(f(x,y,z) = 2(x+y)z\) if \(0<x,y,z<1\) and 0 otherwise. Let’s find all the marginals.
First we find the two dimensional marginals: \[ \begin{aligned} &f_{X,Y}(x,y) =\int_0^1 f(x,y,z)dz =\int_0^1 2(x+y)z dz =x+y \\ &f_{X,Z}(x,z) =\int_0^1 2(x+y)z dy =(2x+1)z \\ &f_{Y,Z}(y,z) =\int_0^1 2(x+y)z dx =(2y+1)z \\ \end{aligned} \]
and the one dimensional ones:
\[ \begin{aligned} &f_X(x) =\int_0^1 f(x,y) dy=\int_0^1 x+ydy = x+1/2 \\ &f_Y(y) =\int_0^1 x+ydx = y+1/2 \\ &f_Z(z) =\int_0^1 f(x,z) dx=\int_0^1 (2x+1)zdx = (x^2+x)z|_0^1=2z \\ \end{aligned} \]
Conditional Random variables
Definition (1.6.17)
let \((X,Y)\) be a r.v. with joint pdf \(f(x,y)\) and marginal \(f_Y\). For any y such that \(f_Y(y)>0\) the conditional pdf of \(X|Y=y\) is defined by
\[f_{X|Y=y}(x|y) = \frac{f(x,y)}{f_Y(y)}\]
Note a conditional pdf requires a specification for a value of the random variable on which we condition, something like \(f_{X|Y=y}\). An expression like \(f_{X|Y}\) is not defined!
Note this is exactly the same as the definition for conditional probabilities of events. For example if \((X,Y)\) is a discrete rv, then
\[P(X=x|Y=y)= \frac{P(X=x,Y=y)}{P(Y=j)}= \frac{f(x,y)}{f_Y(y)}\]
Example (1.6.18)
Say X is the sum and Y is the absolute difference of two dice. Find \(f_{X|Y=5}(7|5)\) and \(f_{Y|X=7}(3|7)\)
| 0 | 1 | 2 | 3 | 4 | 5 | X | |
|---|---|---|---|---|---|---|---|
| 2 | 1 | 0 | 0 | 0 | 0 | 0 | 1 |
| 3 | 0 | 2 | 0 | 0 | 0 | 0 | 2 |
| 4 | 1 | 0 | 2 | 0 | 0 | 0 | 3 |
| 5 | 0 | 2 | 0 | 2 | 0 | 0 | 4 |
| 6 | 1 | 0 | 2 | 0 | 2 | 0 | 5 |
| 7 | 0 | 2 | 0 | 2 | 0 | 2 | 6 |
| 8 | 1 | 0 | 2 | 0 | 2 | 0 | 5 |
| 9 | 0 | 2 | 0 | 2 | 0 | 0 | 4 |
| 10 | 1 | 0 | 2 | 0 | 0 | 0 | 3 |
| 11 | 0 | 2 | 0 | 0 | 0 | 0 | 2 |
| 12 | 1 | 0 | 0 | 0 | 0 | 0 | 1 |
| Y | 6 | 10 | 8 | 6 | 4 | 2 | 36 |
\[ \begin{aligned} &f_{X|Y=5}(7|5) = \frac{f(7,5)}{f_Y(5)}= \frac{2/36}{2/36}=1\\ &f_{Y|X=7}(3|7)= \frac{f(7,3)}{f_X(7)}= \frac{2/36}{6/36}=1/3 \end{aligned} \]
Example (1.6.19)
\(f(x,y)=8xy\), \(0 \le x<y \le 1\). Find \(f_{X|Y=y}(x|y)\).
\[f_{X|Y=y}(x|y) = f(x,y)/f_Y(y) = 8xy/4y^3 = 2x/y^2\]
for x, y with \(\mathbf{0 \le x \le y}\).
Here y is a fixed number!
Again, note that a conditional pdf is a proper pdf:
\[\int_{-\infty}^\infty f_{X|Y=y}(x|y) dx = \int_0^y 2x/y^2dx = x^2/y^2|_0^y=1\]
Example (1.6.20)
Say \((X,Y,Z)\) is a continuous rv with \(f(x,y,z) = 2(x+y)z\) if \(0<x,y,z<1\) and 0 otherwise. Then some of the many conditional pdf’s are:
\[ \begin{aligned} &f_{(X,Y)|Z=z}(x, y|z) = \frac{f(x,y,z)}{f_Z(z)}= \frac{2(x+y)z}{2z}=x+y\text{, }0<x,y<1 \\ &f_{(X,Z)|Y=y}(x, z|y) = \frac{f(x,y,z)}{f_Y(y)}= \frac{2(x+y)z}{y+1/2}\text{, }0<x,z<1 \\ &f_{(Y,Z)|X=x}(y,z|x) = \frac{f(x,y,z)}{f_X(x)}= \frac{2(x+y)z}{x+1/2}\text{, }0<y,z<1 \\ &f_{X|Y=y,Z=z}(x|y,z) = \frac{f(x,y,z)}{f_{(Y,Z)}(y,z)}= \frac{2(x+y)z}{2(y+1)z}\text{, }0<x<1 \\ &f_{X|Y=y,Z=z}(x|y,z) = \frac{f(x,y,z)}{f_{(Y,Z)}(y,z)}= \frac{2(x+y)z}{2(y+1)z}\text{, }0<x<1 \\ &f_{X|Y=y}(x|y) = \frac{f(x,y)}{f_{Y}(y)}= \frac{x+y}{y+1/2}\text{, }0<x<1 \\ \end{aligned} \]
Example (1.6.21)
say \(f(x,y)=3\) if \(0<x<y^2<1\). Find the marginals and the conditional pdf’s. Verify that they are proper pdf’s.
First the marginals:
\[ \begin{aligned} &f_X(x) =\int_\sqrt{x}^{1} 3dy=3(1-\sqrt{x})\text{, }0<x<1 \\ &\int_0^1 3(1-\sqrt{x}) dx = 3(x-\frac23 x^{3/2})|_0^1=1\\ &\\ &f_Y(y) =\int_0^{y^2} 3dx=3y^2\text{, }0<y<1 \\ &\int_0^1 3y^2 dy = y^3|_0^1 =1\\ &f_X(x) =\int_\sqrt{x}^{1} 3dy=3(1-\sqrt{x})\text{, }0<x<1 \end{aligned} \] and then the conditional pdf’s:
\[ \begin{aligned} &f_{X|Y=y}(x|y) = \frac{f(x,y)}{f_Y(y)} = \frac{3}{3y^2} = \frac{1}{y^2}\text{, }0<x<y^2\\ &X|Y=y\sim U[0,y^2] \\ &\\ &f_{Y|X=x}(y|x) = \frac{f(x,y)}{f_X(x)} = \frac{3}{3(1-\sqrt x)} = \frac{1}{1-\sqrt x}\text{, }\sqrt x<y<1\\ &Y|X=x\sim U[\sqrt x,1] \\ \end{aligned} \]
Example (1.6.22)
Say \((X,Y)\) is a discrete random vector with
| 1 | 2 | |
|---|---|---|
| 1 | 1/10 | 1/10 |
| 2 | 1/10 | 1/2 |
| 3 | 1/10 | 1/10 |
Find the conditional pdf of X|Y=y
\[f_{X|Y=y}(x|y)=f(x,y)/f_Y(y)\]
\[f_Y(y) = \sum_x f(x,y)\] \[ \begin{aligned} &f_Y(1) = f(1,1)+f(2,1)+f(3,1)=3/10 \\ &f_Y(2) = f(1,2)+f(2,2)+f(3,2)=7/10 \end{aligned} \]
so
\[ \begin{aligned} &f_{X|Y=1}(1|1)=f(1,1)/f_Y(1) = (1/10)/(3/10) = 1/3 \\ &f_{X|Y=1}(2|1)=f(2,1)/f_Y(1) = (1/10)/(3/10) = 1/3 \\ &f_{X|Y=1}(3|1)=f(3,1)/f_Y(1) = (1/10)/(3/10) = 1/3 \end{aligned} \] and
\[ \begin{aligned} &f_{X|Y=2}(1|2)=f(1,2)/f_Y(2) = (1/10)/(7/10) = 1/7 \\ &f_{X|Y=2}(2|2)=f(2,2)/f_Y(2) = (1/2)/(7/10) = 5/7 \\ &f_{X|Y=2}(3|2)=f(3,2)/f_Y(2) = (1/10)/(7/10) = 1/7 \end{aligned} \]
Example (1.6.23)
Let the continuous random vector \((X,Y)\) have joint density \(f(x,y)=e^{-y}, 0<x<y<\infty\)
Show that f is indeed a proper density
\[ \begin{aligned} &\int_{-\infty}^\infty \int_{-\infty}^{\infty} f(x,y) d(x,y) = \\ &\int_{0}^{\infty} \int_0^y e^{-y} dx dy = \\ &\int_{0}^{\infty} ye^{-y}dy = \\ & -ye^{-y}|_0^\infty - \int_{0}^{\infty} -e^{-y}dy = \\ &\int_{0}^{\infty} e^{-y}dy = \\ &-e^{-y}|_0^\infty=1 \end{aligned} \] Find \(f_{Y|X=x}(y|x)\)
\[f_{Y|X=x}(y|x) = f(x,y)/f_X(x)\] \[f_X(x)= \int_{-\infty}^{\infty} f(x,y) dy = \int_x^\infty e^{-y} dy=-e^{-y}|_x^\infty=e^{-x}\]
so
\[f_{Y|X=x}(y|x)= \frac{e^{-y}}{e^{-x}}=e^{x-y}\text{, }y>x\]
Example (1.6.24)
We have a “device” which generates a random number Y according to an exponential distribution with rate \(\lambda\). We don’t know exactly what \(\lambda\) is, but we do know that \(\lambda=x\) with probability \(0.5^x\) where x=1,2,3,… Find the pdf of Y. Verify that your answer is a proper pdf.
We have a discrete r.v X with pdf
\[f_X(x)=0.5^x\] x=1,2,..
and a conditional rv Y with pdf
\[f_{Y|X=x}(y|x)=x\exp(-xy)\] y>0
We want \(f_Y(y)\). It turns out that if we are dealing with a continuous rv. it is often better to first find the cdf \(F_Y (y)=P(Y \le y)\). Now first we have
\[f_{Y|X=x}(y|x) = f(x,y)/f_X(x)\] and so
\[f(x,y)=f_X(x)f_{Y|X=x}(y|x)=0.5^xx\exp(-xy)\] and so
\[f_Y(y) = \sum_{x=1}^\infty f(x,y) = \sum_{x=1}^\infty 0.5^xx\exp(-xy)\]
\[ \begin{aligned} &F_Y(y) =P(Y\le y) = \int_{-\infty}^{y} f_Y(t)dt = \\ & \int_{-\infty}^{y} \left[ \sum_{x=1}^\infty f(x,t)\right] dt= \\ &\sum_{x=1}^\infty \left[ \int_{-\infty}^{y} f(x,t)\right] dt= \\ &\sum_{x=1}^\infty \left[ \int_{0}^{y} 0.5^xx\exp(-xt)\right] dt= \\ &\sum_{x=1}^\infty 0.5^x\left[ \int_{0}^{y} x\exp(-xt)\right] dt= \\ &\sum_{x=1}^\infty 0.5^x\left[ -\exp(-xt)|_{0}^{y}\right] = \\ &\sum_{x=1}^\infty 0.5^x\left[ 1-\exp(-xy)\right] = \\ &\sum_{x=1}^\infty (\frac12)^x\left[ 1-(e^{-y})^x\right] = \\ &\sum_{x=1}^\infty (\frac12)^x - \sum_{x=1}^\infty(e^{-y}/2)^x = \\ &\left(\sum_{x=0}^\infty (\frac12)^x-1\right) - \left(\sum_{x=0}^\infty(e^{-y}/2)^x -1\right)= \\ & \frac{1}{1-1/2}- \frac{1}{1-e^{-y}/2} =\\ &2- \frac{2e^y}{2e^y-1} \end{aligned} \]
a little bit of care: the geometric series \(\sum q^k\) only converges if \(|q|<1\). Here \(y>0\), so \(e^y>1\) so \(1/2e^y < 0.5 < 1\), we are save.
Now
\[f_Y(y) = \frac{dF_Y(y)}{dy} = -\frac{2e^y(2e^y-1)-2e^y2e^y}{(2e^y-1)^2}=\frac{2e^y}{(2e^y-1)^2}\]
This type of model is called a hierarchical model, with one rv defined conditional on another. This way of describing a model is very useful in real live.
Law(s) of Total Probability
We have previously seen the law of total probability for events. There are corresponding versions for random variables:
Theorem (1.6.25)
Say X,Y are discrete random variables. Then
\[f_X(x)=\sum_y f_{X|Y=y} (x|y)f_Y (y)\]
proof
Let \(B=\{\omega:X(\omega)=x\}\) and \(A_y =\{\omega:Y(\omega)=y\}\). Then \(\{A_y , y \in S\}\) is a partition of \(S\) and we have using (1.3.6)
\[ \begin{aligned} &f_X (x) = P(X=x) = \\ &P(B)=\sum_y P(B|A_y )P(A_y ) = \\ &\sum_y f_{X|Y=y} (x|y)f_Y (y) \end{aligned} \]
Example (1.6.26)
Say we have a random variable \(X\) with \(P(X=x)=1/3, x=1,2,3\) and a conditional rv \(Y|X=x\) with density \(f_{Y|X=x}(y|x)=cy^x, y\in\{1,2\}\). We want to find the unconditional density of \(Y\). So
\[ \begin{aligned} &f_Y(y) = \sum_x f_{Y|X=x}(y|x) f_X(x) =\\ &\sum_{x=1}^3 cy^x \frac{1}{3} = \frac{c}{3} \left[y+y^2+y^3\right] \\ &\sum_y f_y(y) = \\ &\frac{c}{3} \left[1+1^2+1^3\right]+\frac{c}{3} \left[2+2^2+2^3\right] = 17c/3 \end{aligned} \]
Theorem (1.6.27)
Say \(X\) is a discrete rv and Y is a continuous rv. Then
\[f_Y(y)=\sum_x f_{Y|X=x} (y|x)f_X(x)\]
proof
Notice that the formula is the same as in theorem 1.6.25, but this case requires a different proof. We need to be careful: for a discrete rv \(f_X(x)=P(X=x)\) makes sense, but for a continuous random variable we have
\[ \begin{aligned} &P(Y=y) = \lim_{h \rightarrow 0} P(y \le Y \le y+h) = \\ &\lim_{h \rightarrow 0} \int_y^{y+h} f_Y (t)dt = \\ &\lim_{h \rightarrow 0} (F_Y (y+h)-F_Y (y)) = 0 \end{aligned} \] for all y!
Let’s instead consider the event \(B=\{Y \le y\}\). Also let \(A_x =\{\omega:X(\omega)=x\}\), then
\[
\begin{aligned}
&F_Y(y) = P(Y\le y) = \\
&P(B) =\sum_x P(B|A_x)P(A_x) = \\
&\sum_x P(Y\le y|X=x)P(X=x) = \\
&\sum_x \int_{-\infty}^{y} f_{Y|X=x}(y|x) f_X(x) \\
\\
&f_Y(y) = \frac{dF_Y(y)}{dy} = \\
&\frac{d}{dy}\sum_x \int_{-\infty}^{y} f_{Y|X=x}(y|x) f_X(x)dx =\\
&\sum_x \frac{d}{dy}\int_{-\infty}^{y} f_{Y|X=x}(y|x) f_X(x) dx=\\
&\sum_x f_{Y|X=x}(y|x) f_X(x)
\end{aligned}
\]
The interchanges of integral and sum and of derivative and sum can be justified because densities are non-negative and the sums/integrals are finite.
Example (1.6.27a)
Say we have a discrete random variable \(X\) with \(f_X(x) = P(X=x)=1/3, x=1,2,3\) and a conditional rv \(Y|X=x\) with density \(f_{Y|X=x}(y|x)=(x+1)y^x,0<y<1\). We want to find the unconditional density of \(Y\). So
\[ \begin{aligned} &f_Y(y) = \sum_x f_{Y|X=x}(y|x) f_X(x)dx =\\ &\sum_{x=1}^3 (x+1)y^x \frac{1}{3} = \\ &\frac{1}{3} \left[2(y/2)+3(y/2)^2+4(y/2)^3\right]= \\ &\left[4y+3y^2+2y^3\right]/12 \end{aligned} \]
Theorem (1.6.28)
Let \(X\) be a rv (discrete or continuous) and \(Y\) a continuous random variable, then
\[f_X(x)=\int_{-\infty}^\infty f_{X|Y=y}(x|y)f_{Y}(y)dy\]
proof
For conditioning on the continuous rv we need to define a new discrete rv \(Y'\) as follows
\[Y'= ih \text{ if }ih \le Y<(i+1)h\]
Then
\[ \begin{aligned} &f_{Y'}(ih) =P(Y'=ih) = P(ih\le Y<(i+1)h) = \\ &\int_{ih}^{(i+1)h} f_Y(t)dt\approx f_Y(ih)h \\ &f_X(x) =\sum_{i=-\infty}^\infty f_{X|Y'=ih}(x|ih)f_{Y'}(ih) \approx \\ &\sum_{i=-\infty}^\infty f_{X|Y=ih}(x|ih)f_{Y}(ih)h \rightarrow \\ &\int_{-\infty}^\infty f_{X|Y=y}(x|y)f_{Y}(y)dy \end{aligned} \]
because this is a Riemann sum, so it converges to the corresponding integral.
Example (1.6.29)
Let \(Y\sim U[0.4, 0.6]\) and \(X\) a random variable with \(P(X=k|Y=y)= c(k+1)y^k, k=0,1, 2,..\). Find \(F_X(2)\).
by the law of total probability we have
\[ \begin{aligned} &f_X(k) = \int_{-\infty}^\infty f_{X|Y=y}(k|y)f_Y(y) dy = \\ &\int_{0.4}^{0.6} c(k+1)y^k \frac1{0.6-0.4} dy = \\ &5c y^{k+1}|_{0.4}^{0.6} = \\ &5c\left(0.6^{k+1}-0.4^{k+1}\right)\\ &F_X(k)=\sum_{i=0}^k5c\left(0.6^{i+1}-0.4^{i+1}\right) = \\ &5c\left(\sum_{j=0}^{k+1} 0.6^{j} - \sum_{j=0}^{k+1} 0.4^{j}\right) =\\ &5c\left( \frac{1-0.6^{k+2}}{1-0.6}- \frac{1-0.4^{k+2}}{1-0.4} \right) \end{aligned} \]
because these are geometric sums. Now
\[ \begin{aligned} &1 = \lim_{k\rightarrow\infty} F_X(k) = \\ &5c(1/0.4-1/0.6) = 25c/6\\ &F_X(2) = 6/5\left( \frac{1-0.6^{4}}{0.4} -\frac{1-0.4^{4}}{0.6} \right)=0.6624\\ \end{aligned} \]
Example (1.6.30)
Let \(X\) have density \(f_X(x)=(p+1)x^p\), \(0<x<1,p>-1\), and \[P(Y=0|X=x)=1-P(Y=1|X=x)=x\]
We want to find \(p\) such that \(P(Y=0)=\frac12\).
\[ \begin{aligned} &P(Y=0) = f_Y(0) = \int_{-\infty}^{\infty} f_{Y|X=x}(0|x)f_X(x) = \\ & \int_0^1 x \times (p+1)x^p dx = \\ &(p+1)\int_0^1 x^{p+1} dx = \\ & \frac{p+1}{p+2}x^{p+2}|_0^1=\frac{p+1}{p+2}=\frac12 \\ &2(p+1)=p+2\\ &p=0 \end{aligned} \]
Example (1.6.30a)
Say \(X\sim Exp(1)\) and \(Y|X=x\sim U[x,x+1]\). We want to find \(P(Y>2)\).
We have \(f_X(x)=e^{-x}I_{(0,\infty)}(x)\) and \(f_{Y|X=x}(y|x)=I_{(x,x+1)}(y)\). Note that \(I_{(x,x+1)}(y)=I_{(y-1,y)}(x)\)
\[ \begin{aligned} &P(Y>2) = 1-P(Y<2)=1-\int_0^2 f_Y(y) dy\\ &f_Y(y) = \int_{-\infty}^{\infty} f_{Y|X=x}(y|x)f_X(x)dx = \\ &\int_{-\infty}^{\infty} I_{[x,x+1]}(y) e^{-x}I_{(0,\infty)}(x) dx = \\ &\int_{-\infty}^{\infty} I_{[\max\{0,y-1]\},y]}(x) e^{-x} dx =\\ &\\ &\left\{\begin{array} .\int_0^y e^{-x} dx\ \text{ for }0<y<1\\ \int_{y-1}^y e^{-x} dx\ \text{ for }y>1 \end{array}\right. =\\ &\\ &\left\{\begin{array} .1-e^{-y} \ \text{ for }0<y<1\\ e^{1-y} - e^{-y} \ \text{ for }y>1 \end{array}\right. \end{aligned} \] Now
\[ \begin{aligned} &\int_0^2 f_Y(y)dy = \\ &\int_0^1 1-e^{-y}dy+\int_1^2 e^{1-y}-e^{-y}dy = \\ &(y+e^{-y}|_0^1+(-e^{1-y}+e^{-y}|_1^2 = \\ &1+e^{-1}-1+(-e^{-1}+e^{-2}+1-e^{-1}) = \\ &1-e^{-1}+e^{-2}=0.7675\\ &P(Y>2)=1-0.7675=0.2325 \end{aligned} \]
Let’s do a little simulation to check on these calculations:
n=1e4
x=rexp(n, 1)
y=runif(n, x, x+1)
g=function(x) {
z=0*x
for(i in seq_along(x))
z[i]=ifelse(x[i]<1, 1, exp(1-x[i]))-exp(-x[i])
z
}
hist(y, 100, freq=FALSE, main="")
curve(g, 0, 8, add=TRUE, lwd=2,col="blue")
length(y[y>2])/n## [1] 0.236Independence
Definition (1.6.31)
Two r.v. X and Y are said to be independent iff
\[f_{X,Y} (x,y)=f_X (x)f_Y (y)\]
for all x,y.
Note if X and Y are discrete random variables this is again just the definition of independence of events.
\[ \begin{aligned} &f_{X,Y} (x,y) = \\ &P(X=x,Y=y) = \\ &P\left(\{\omega:X(\omega)=x\}\cap\{\omega:Y(\omega)=y\}\right) = \\ &P\left(\{\omega:X(\omega)=x\}\right)P(\left\{\omega:Y(\omega)=y\}\right) = \\ &P(X=x)P(Y=y) = \\ &f_X (x)f_Y (y) \end{aligned} \]
Notation: we will use the notation \(X \perp Y\) if X and Y are independent.
Example (1.6.32)
Say X is the sum and Y is the absolute difference of two dice. Previously we found
\[f_{X,Y} (7,1) = 1/18\]
but
\[f_X(7)f_Y(1) = 1/6 \times 10/36 = 5/108\]
so X and Y are not independent
Theorem (1.6.33)
say \(f(x,y)\) is the joint pdf of a random vector \((X,Y)\). Then \(X\) and \(Y\) are independent if there exist functions \(g\) and \(h\) such that
\[f(x,y)=g(x)h(y)\]
proof
the only difference between the definition and the theorem is that \(g\) and \(h\) need not be proper densities. But first of all we can assume that \(g\) and \(h\) are non-negative, otherwise just take \(|g|\) and \(|h|\).
Moreover
\[ \begin{aligned} &f_X(x) = \int_{-\infty}^{\infty} f(x,y)dy =\\ &\int_{-\infty}^{\infty} g(x)h(y)dy = \\ &g(x)\int_{-\infty}^{\infty} h(y)dy = cg(x)\\ &1= \int_{-\infty}^{\infty}f_X(x)dx = \\ &\int_{-\infty}^{\infty} cg(x) dx = c\int_{-\infty}^{\infty} g(x) dx \end{aligned} \]
where \(0<c<\infty\), so \(g/c = f_X\), and similarly \(h/d=f_Y\).
Example (1.6.34)
say \(f(x,y)=\exp(-x-y)\), \(x,y>0\). Then
\[f(x,y) = \exp(-x-y) = \exp(-x)\exp(-y) = g(x)h(y)\]
so X and Y are independent.
Mostly the concept of independence is used in reverse: we assume \(X\) and \(Y\) are independent (based on good reason!) and then make use of the formula:
Example (1.6.35)
Say we use the computer to generate 10 independent exponential r.v’s with rate \(\lambda\). What is the probability density function of this random vector?
We have \(f_{X_i}(x_i )=\lambda \exp(-\lambda x_i )\) for i=1,2,..,10 so
\[ \begin{aligned} &f_{(X_1 ,..,X_{10} )} (x_1 , .., x_{10} ) = \\ &\lambda \exp(- \lambda x_1 ) \times..\times \lambda \exp(- \lambda x_{10} ) = \\ &\lambda^{10}\exp(-\lambda(x_1 +..+x_{10} )) \end{aligned} \]