\(X\) is said to have a uniform distribution on the interval \([A,B]\) (\(X\sim U[A,B]\))if
\[f(x)=\frac1{B-A}I_{[A,B]}(x)\] we already found
\[ \begin{aligned} &E[X] = \frac{A+B}2\\ &var(X) = \frac{(B-A)^2}{12} \end{aligned} \] finally
\[ \begin{aligned} &\psi_X(t) = \int_A^B e^{tx}\frac1{B-A}dx=\\ &\frac{e^{tx}}{t(B-A)}|_A^B = \\ &\frac{e^{tA}-e^{tB}}{t(B-A)} \end{aligned} \]
\(X\) is said to have an exponential distribution rate \(\lambda\) (\(X \sim Exp( \lambda )\)) if
\[f(x) = \lambda e^{-\lambda x}, x>0, \lambda >0\] Note
\[F_X(x)=\int_0^x \lambda e^{-\lambda t}dt = \int_0^{x/\lambda} e^{-y}dy = F_Y(x/\lambda)\]
where \(Y\sim Exp(1)\).
Note that we also just found the cdf of an exponential:
\[F(x)=1-e^{-\lambda x}\]
We can use this to simplify some calculations:
\[ \begin{aligned} &E[Y^k] =\int_0^\infty y^ke^{-y}dy = \\ &-y^ke^{-y}|_0^\infty - \int_0^\infty -ky^{k-1}e^{-y}dy = \\ &k\int_0^\infty y^{k-1}e^{-y}dy = kE[Y^{k-1}]\\ \end{aligned} \]
\[ \begin{aligned} &E[Y] = 1E[Y^0]=1\\ &E[Y^2] = 2E[Y^1]=1 = \\ &var(Y) = 2-1^2=1\\ &\\ &E[X] =E[Y/\lambda]=1/\lambda\\ &var(X)=var(Y/\lambda)=1/\lambda^2 \end{aligned} \]
The “trick” we used here, namely showing a result for a special case (\(\lambda =1\)) and then doing the general case, is often a good idea! Let’s use it again to find the moment generating function:
\[ \begin{aligned} &\psi_Y(t) =\int_0^\infty e^{ty} e^{-y} dy =\\ &\int_0^\infty e^{-(1-t)y} dy = \frac1{1-t} \end{aligned} \] if \(1-t>\) or \(t<1\).
Now
\[\psi_X(t) = E[e^{tX}]=E[e^{tY/\lambda}]=E[e^{(\frac{t}{\lambda})Y}]=\frac1{1-t/\lambda}=\frac{\lambda}{\lambda-t}\] if \(t<\lambda\).
We have previously talked about the memoryless property, and the fact that among discrete distributions on \(\mathbb{Z}^+\) it is unique to the geometric rv. Now we have
\(X\) has an exponential distribution iff \(X\) is a positive continuous r.v. and it has the memoryless property
\[P(X>s+t | X>s) = P(X>t)\] for all \(s,t > 0\).
proof:
Assume \(X \sim Exp(\lambda)\). Then
\[P(X>s+t|X>s) = \frac{P(X>s+t)}{P(X>s)}= \frac{1-(1-e^{s+t})}{1-(1-e^{s})}=e^{-t}=P(X>t)\]
on the other hand assume \(X\) is continuous with density \(f\) and
\[P(X>s+t | X>s) = P(X>t)\]
Above we saw that this implies
\[P(X>s+t)=P(X>s)P(X>t)\]
Let \(h(x) = P(X>x)\) and let \(\epsilon>0\). Note \(h(0) = P(X>0) = 1\) because \(X\) is positive. So
\[ \begin{aligned} &h'(0) =\lim_{\epsilon\rightarrow 0} \left(\frac{h(\epsilon)-h(0)}{\epsilon} \right)\\ &\lim_{\epsilon\rightarrow 0} \left(\frac{P(X>\epsilon)-1}{\epsilon} \right) = \\ &\lim_{\epsilon\rightarrow 0} \left(\frac{1-P(X<\epsilon)-1}{\epsilon} \right) = \\ &-\lim_{\epsilon\rightarrow 0} \left(\frac{P(X<\epsilon)}{\epsilon} \right) = \\ &-\lim_{\epsilon\rightarrow 0} \frac1\epsilon\int_0^\epsilon f(x)dx = -f(0) \end{aligned} \]
Next we define \(\beta=:f(0)\) and note that
\[ \begin{aligned} &h(x+\epsilon)-h(x) = \\ &P(X>x+\epsilon)-P(X>x) = \\ &P(X>x)P(X>\epsilon)-P(X>x)=\\ &P(X>x)\left(P(X>\epsilon)-1\right)= \\ &h(x)\left(h(\epsilon)-1\right) \end{aligned} \]
and so
\[ \begin{aligned} &h'(x) =\lim_{\epsilon\rightarrow 0} \left(\frac{h(x+\epsilon)-h(x)}{\epsilon} \right) = \\ &\lim_{\epsilon\rightarrow 0} \left(\frac{h(x)(h(\epsilon)-1)}{\epsilon} \right) = \\ &h(x)\lim_{\epsilon\rightarrow 0} \left(\frac{h(\epsilon)-1}{\epsilon} \right) = \\ &h(x)h'(0) =-\beta h(x)\\ &\frac{h'(x)}{h(x)}=-\beta\\ &\int \frac{h'(x)}{h(x)}dx=\int-\beta dx = -\beta x+c\\ &\log h(x)=-\beta x+c\\ &\log h(0)=0=-\beta 0+c\\ &h(x)=e^{-\beta x}\\ &\\ &P(X>x)=h(x)=e^{-\beta x}\\ &F_X(x)=1-P(X>x)=1-e^{-\beta x} \end{aligned} \]
and so we see \(X \sim Exp( \beta )\)
\[\Gamma(x)=\int_0^\infty t^{x-1}e^{-t}dt;x>0\]
Recall the Gamma function:
The Gamma function is famous for many things, among them the relationship
\[\Gamma (x+1) = x\Gamma (x)\]
which follows from:
\[\Gamma(x+1)=\int_0^\infty t^{x}e^{-t}dt=-t^{x}e^{-t}|_0^\infty-\int_0^\infty -xt^{x-1}e^{-t}dt=\\x\int_0^\infty t^{x-1}e^{-t}dt=x\Gamma(x)\]
This implies
\[\Gamma(n)=(n-1)!\]
so the Gamma function is a continuous version of the factorial. It has many other interesting properties, for example
\[\Gamma (1/2) = \sqrt{\pi}\]
\(X\) is said have a Gamma distribution (\(X \sim \Gamma ( \alpha , \beta )\)) with parameters \((\alpha , \beta)\) if it has density
\[f(x)=\frac{\beta^\alpha}{\Gamma(\alpha)}x^{\alpha-1}e^{-\beta x}\]
\(x,\alpha,\beta>0\).
Note sometimes one uses the following equivalent parametrization:
\[f(x)=\frac{1}{\Gamma(\alpha)\beta^\alpha}x^{\alpha-1}e^{-x/\beta}\]
Note
\[ \begin{aligned} &F(x) =P(X\le x) =\\ &\int_0^x \frac{\beta^\alpha}{\Gamma(\alpha)}t^{\alpha-1}e^{-t/\beta} dt \\ &\int_0^x \frac{1}{\Gamma(\alpha)}(\beta t)^{\alpha-1}e^{-(\beta t)} (\beta dt) = \\ &\int_0^{\beta x} \frac{1}{\Gamma(\alpha)}y^{\alpha-1}e^{-y} dy = \\ &P(Y\le \beta x) = P(Y/\beta \le x) = F_{Y/\beta}(x) \end{aligned} \] where \(Y\sim Gamma(\alpha, 1)\). So if \(Y\sim Gamma(\alpha, 1)\), then \(X=Y/\beta\sim Gamma(\alpha,\beta)\).
For the mean and variance we find
\[ \begin{aligned} &E[Y^k] = \int_0^{\infty} y^k\frac{1}{\Gamma(\alpha)}y^{\alpha-1}e^{-y} dy=\\ &\frac{\Gamma(\alpha+k)}{\Gamma(\alpha)}\int_0^{\infty} \frac{1}{\Gamma(\alpha+k)}y^{\alpha+y-1}e^{-y} dy=\\ &\frac{\Gamma(\alpha+k)}{\Gamma(\alpha)} = \frac{(\alpha+k-1)...\alpha\Gamma(\alpha)}{\Gamma(\alpha)} =(\alpha+k-1)...\alpha\\ &\\ &E[Y]=\alpha\\ &var(Y)=E[Y^2]-E[Y]^2=(\alpha+1)\alpha-\alpha^2=\alpha\\ &\\ &E[X]=E[Y/\beta]=\alpha/\beta\\ &var(X)=var(Y/\beta )=\alpha/\beta^2 \end{aligned} \] for the moment generating function we find
\[ \begin{aligned} &\psi_Y(t) = \int_0^{\infty} e^{ty}\frac{1}{\Gamma(\alpha)}y^{\alpha-1}e^{-y} dy = \\ &\int_0^{\infty} \frac{1}{\Gamma(\alpha)}y^{\alpha-1}e^{-(1-t)y} dy = \\ &\frac1{(1-t)^\alpha}\int_0^{\infty} \frac{(1-t)^\alpha}{\Gamma(\alpha)}y^{\alpha-1}e^{-(1-t)y} dy = \\ &\frac1{(1-t)^\alpha} \end{aligned} \] if \(t<1\). Finally
\[\psi_X(t) = E[e^{tX}]=E[e^{(t/\beta) Y}]=\psi_Y(t/\beta t) = \frac1{(1-t/\beta )^\alpha}\]
if \(t<\beta\).
By definition we have a random variable with \(P(X>0)=1\), and so the Gamma is the basic example of a r.v. on \((0,\infty)\), or a little more general (using a change of variables) on any open half interval.
Note if \(X \sim \Gamma (1, \beta )\) then \(X \sim E(\beta )\).
Another important special case is if \(X \sim \Gamma (n/2,2)\), then X is called a Chi-square r.v. with n degrees of freedom, denoted by \(X \sim \chi^2(n)\).
We want to find the kurtosis of \(X \sim Exp(\lambda)\)
\[ \begin{aligned} &E[X^k] = \int_0^{\infty} x^k\lambda e^{-\lambda x} dx=\\ &\lambda\frac{\Gamma(k+1)}{\lambda^{k+1}}\int_0^{\infty} \frac{\lambda^{k+1}}{\Gamma(k+1)} x^{(k+1)-1} e^{-\lambda x} dx = \\ &\frac{\Gamma(k+1)}{\lambda^{k}} = \frac{k!}{\lambda^{k}}\\ &\\ &\mu=E[X]= \frac{1}{\lambda}\\ &E[X^2]=\frac{2}{\lambda^{2}}\\ &E[X^3]=\frac{6}{\lambda^{3}}\\ &E[X^4]=\frac{24}{\lambda^{4}}\\ &\\ &\kappa_2=E[X^2]-\mu^2=\\ &\frac{2}{\lambda^{2}}-\left(\frac{1}{\lambda}\right)^2=\frac{1}{\lambda^{2}}\\ &\\ &\kappa_4=E[(X-\mu)^4]=\\ &E[X^4]-4\mu E[X^3]+6\mu^2 E[X^2]-4\mu^3 E[X]+\mu^4=\\ &\left(24-4\times 6+6\times 2-4+1\right)/\lambda^4=9/\lambda^4\\ &\\ &\gamma_2=\frac{\kappa_4}{\kappa_2^2}-3= \frac{9/\lambda^4}{(1/\lambda^2)^2}-3=6 \end{aligned} \]
so the kurtosis is greater than 0, therefore an exponential is leptocurtic.
There is an important connection between the Gamma and the Poisson distributions:
if \(X_n \sim \Gamma (n, \beta )\) and \(Y \sim Pois( \beta x)\) then
\[P(X_n \le x) = P(Y \ge n)\]
proof (by induction)
case n=1: \(X_1\sim Gamma(1,\beta)=Exp(\beta)\), and so
\[ \begin{aligned} &P(X\le x) = 1-e^{-\beta x} = \\ &1-\frac{(\beta x)^0}{0!}e^{-\beta x} = \\ &1-P(Y=0) =P(Y\ge 1) \\ \end{aligned} \] Now assume statement is true for n, then
\[ \begin{aligned} &P(X_{n+1}\le x) = \int_0^x \frac{\beta^{n+1}}{\Gamma(n+1)}t^{(n+1)-1}e^{-\beta t}dt=\\ &\frac{\beta^{n+1}}{n!}\int_0^x t^{n}e^{-\beta t}dt=\\ &\frac{\beta^{n+1}}{n!}\left[-\frac1\beta t^{n}e^{-\beta t}|_0^x -\int_0^x -\frac{n}\beta t^{n-1}e^{-\beta t}dt\right]=\\ &\frac{\beta^{n+1}}{n!}\left[-\frac1\beta x^{n}e^{-\beta x}+\frac{n}\beta \int_0^x -t^{n-1}e^{-\beta t}dt\right]=\\ &-\frac{(\beta x)^{n}}{n!} e^{-\beta x}+ \int_0^x \frac{\beta^n}{\Gamma(n)}t^{n-1}e^{-\beta t}dt=\\ &-P(Y=n)+P(X_n\le x) = \\ &-P(Y=n)+P(Y\ge n) = P(Y\ge n+1) \end{aligned} \]
\(X\) is said to have a Beta distribution with parameters \(\alpha\) and \(\beta\) (\(X \sim Beta( \alpha , \beta )\)) if
\[f(x)= \frac{\Gamma(\alpha+\beta)}{\Gamma(\alpha)\Gamma(\beta)}x^{\alpha-1}(1-x)^{\beta-1}\]
for \(0<x<1\).
The above is a proper density
proof
\[ \begin{aligned} &\Gamma(x)\Gamma(y) = \\ &\left(\int_0^\infty u^{x-1}e^{-u}du \right)\left(\int_0^\infty v^{y-1}e^{-v}dv \right) = \\ &\int_0^\infty\int_0^\infty u^{x-1}v^{y-1}e^{-(u+v)}dudv \end{aligned} \] Let’s do the transformation \(z=u+v\), \(t=\frac{u}{u+v}\). The inverse transform is
\[u=zt,v=z(1-t)\] which has Jacobian
\[J=\begin{vmatrix}t & z\\1-t& -z\end{vmatrix}=-zt-z(1-t)=-z\]
and so
\[ \begin{aligned} &\Gamma(x)\Gamma(y) = \\ &\int_0^\infty\int_0^1 (zt)^{x-1}[z(1-t)]^{y-1}e^{-z}z dtdz = \\ &\int_0^\infty\left(\int_0^1 t^{x-1}(1-t)^{y-1} dt\right)z^{x+y-1}e^{-z}dz = \\ &\left(\int_0^1 t^{x-1}(1-t)^{y-1}dt\right) \left(\int_0^\infty z^{x+y-1}e^{-z}dz\right) = \\ &\left(\int_0^1 t^{x-1}(1-t)^{y-1}dt\right)\Gamma(x+y) \end{aligned} \]
It is easy to calculate the moments of a Beta distribution:
\[ \begin{aligned} &E[X^k] = \int_0^1 x^k\frac{\Gamma(\alpha+\beta)}{\Gamma(\alpha)\Gamma(\beta)}x^{\alpha-1}(1-x)^{\beta-1}dx\\ &\frac{\Gamma(\alpha+\beta)}{\Gamma(\alpha)\Gamma(\beta)}\frac{\Gamma(\alpha+k)\Gamma(\beta)}{\Gamma(\alpha+k+\beta)}\int_0^1 \frac{\Gamma(\alpha+k+\beta)}{\Gamma(\alpha+k)\Gamma(\beta)}x^{\alpha+k-1}(1-x)^{\beta-1}dx\\ &\frac{\Gamma(\alpha+\beta)}{\Gamma(\alpha)}\frac{\Gamma(\alpha+k)}{\Gamma(\alpha+k+\beta)} = \\ &\frac{\Gamma(\alpha+\beta)}{\Gamma(\alpha)}\frac{(\alpha+k-1)...\alpha\Gamma(\alpha)}{(\alpha+\beta+k-1)...(\alpha+\beta)\Gamma(\alpha+\beta)}=\\ &\frac{(\alpha+k-1)...\alpha}{(\alpha+\beta+k-1)...(\alpha+\beta)}=\\ &\prod_{r=0}^{k-1}\frac{\alpha+r}{\alpha+\beta+r} &\\ &\\ &E[X]=\frac{\alpha}{\alpha+\beta}\\ &var(X)=\frac{(\alpha+2-1)\alpha}{(\alpha+\beta+2-1)(\alpha+\beta)}-\left(\frac{\alpha}{\alpha+\beta}\right)^2 = \\ &\frac{\alpha\beta}{(\alpha+\beta)^2(\alpha+\beta+1)} \end{aligned} \]
Next we find the moment generating function:
\[ \begin{aligned} &\psi(t) =\int_0^1 e^{tx}f(x)dx\\ &\int_0^1 \left(\sum_{k=0}^\infty \frac{(tx)^k}{k!}\right) f(x)dx = \\ &\sum_{k=0}^\infty \frac{t^k}{k!} \int_0^1 x^k f(x)dx=\\ &\sum_{k=0}^\infty \frac{t^k}{k!}E[X^k] = \\ &1+\sum_{k=1}^\infty \left(\prod_{r=0}^{k-1}\frac{\alpha+r}{\alpha+\beta+r}\right)\frac{t^k}{k!} \end{aligned} \]
Note that the Taylor expansion of the moment generating function is completely general.
By definition we have \(0<X<1\), and so the Beta is the basic example of a r.v. on [0,1], or a little more general (using a change of variables) on any open finite interval.
Special cases:
\[f(x)=cx^{p-1}(1-p)^{1-1}=cx^{p-1}=px^{p-1}, 0<x<1, p>0\]
and for this pdf we have \(E[X]=p/(p+1), var(X)=p/[(p+1)^2(p+2)]\)
Say \(X\) and \(Y\) are independent \(\Gamma ( \alpha , \beta )\). Then
\[X+Y\sim Gamma(2\alpha,\beta)\]
proof
We will show the theorem fo the case \(\beta=1\). The general case follows easily. Let \(Z=X+Y\), then from the convolution formula we have
\[ \begin{aligned} &f_Z(z) = \int_{-\infty}^{\infty} f_X(t)f_Y(z-t)dt=\\ &\int_0^z \frac1{\Gamma(\alpha)} t^{\alpha-1}e^{-t}\frac1{\Gamma(\alpha)} (z-t)^{\alpha-1}e^{-(z-t)}dt = \\ &(\frac1{\Gamma(\alpha)})^2e^{-z}\int_0^z t^{\alpha-1}(z-t)^{\alpha-1} dt = \\ &(\frac1{\Gamma(\alpha)})^2e^{-z}\int_0^1 (zy)^{\alpha-1}(z-zy)^{\alpha-1} zdy = \\ &(\frac1{\Gamma(\alpha)})^2z^{2\alpha-1}e^{-z}\int_0^1 y^{\alpha-1}(1-y)^{\alpha-1} dy = \\ &(\frac1{\Gamma(\alpha)})^2z^{2\alpha-1}e^{-z}\frac{\Gamma(\alpha)^2}{\Gamma(2\alpha)}=\\ &\frac{1}{\Gamma(2\alpha)}z^{2\alpha-1}e^{-z} \end{aligned} \] so we see that \(Z \sim \Gamma (2 \alpha , 1)\). In other words, the sum of independent Gamma r.v.’s is again Gamma.
Some special cases:
\(X,Y\) iid \(Exp(\lambda)\) then \(X+Y \sim \Gamma (2, \lambda )\) (and not exponential)
\(X\sim \chi^2(n)\), \(Y\sim \chi^2(m)\) and \(X\perp Y\), then \(X+Y \sim \chi^2(n+m)\).
Previously we found a curious relationship between the Poisson and the Gamma distributions. There is a similar one between the Beta and the Binomial:
if \(X_m \sim Beta(n,m)\) and \(Y \sim Bin(n+m-1,1-x)\) then
\[P(X_m \le x) = P(Y<m)\]
proof (by induction on m)
\[ \begin{aligned} &P(X_1\le x) = \int_0^x \frac{\Gamma(n+1)}{\Gamma(n)\Gamma(1)}t^{n-1}(1-t)^{1-1}dt=\\ &\int_0^x nt^{n-1}dt = x^n = {{n}\choose{0}}(1-x)^0(1-(1-x))^{n-0}= \\ &P(Y=0) =P(Y<1) \\ \end{aligned} \]
Assume statement is true for m, then
\[ \begin{aligned} &P(X_{m+1}\le x) = \int_0^x \frac{\Gamma(n+m+1)}{\Gamma(n)\Gamma(m+1)}t^{n-1}(1-t)^{m+1-1}dt=\\ & \frac{(n+m)!}{(n-1)!m!}\left[\frac1nt^n(1-t)^{m}|_0^x-\int_0^x \frac1nt^nm(1-t)^{m-1}(-1)dt \right] =\\ & \frac{(n+m)!}{n!m!}\left[x^n(1-x)^{m}+m\int_0^x t^n(1-t)^{m-1}dt \right]=\\ & \frac{(n+m)!}{n!m!}x^n(1-x)^{m}+\int_0^x \frac{(n+m)!}{n!(m-1)!}t^n(1-t)^{m-1}dt =\\ & {{n+m}\choose{m}}x^n(1-x)^{m}+P(X_m\le x) = \\ &P(Y=m)+(Y<m)=P(Y\le m)=P(Y<m+1) \end{aligned} \]
A rv. X has a Cauchy distribution if
\[f(x)=\frac1\pi\frac1{1+(x-\theta)^2}\]
As we saw before the Cauchy has one interesting property:
\[E[|X|]= \infty\]
so the Cauchy has no mean (and therefore no moments at all). The reason is that it has thick “tails”, that is the probability of observing a large value (+ or -) is large.
A very nice tool describing these and many other distributions as well as their relationships was created by Lawrance M. Leemis and Raghu Pasupathy and is discribed in their Chance August 2019 article “The ties that bind” can be found at http://www.math.wm.edu/~leemis/chart/UDR/UDR.html.