conditional.knit

Conditional Probability

Example (1.3.1)

A bag contains slips of paper. Each paper has a number and a letter written on them. They are: A5, A7, B1, B2, C2, C4, D2, D4, E1, E3. A slip is chosen at random, what is the probability it has a 2 on it?

Easy: 3/10

Now say somebody picks a slip and tells you it has the letter B on it. Now what is the probability it also has a 2 on it?

Again easy: 1/2

This is an example of a conditional probability. We write

P(#2 | Letter B)

(” probability of #2 given letter B)

Above we found the conditional probability by changing the sample space. First it was S={A5, A7, B1, B2, C2, C4, D2, D4, E1, E3} but once we knew the slip of paper had the letter B it changed to S={B1, B2}.

In general this changing of the sample space is too difficult, but we can find conditional probabilities using the formula

$P(A|B) = \frac{P(A \cap B)}{P(B)}$

Note: this only works if P(B)>0.

Note: this formula can also be derived using the idea of coherence and the concept of the sure looser discussed earlier.

Example (1.3.2)

using the numbers from the previous exmple:

P(#2 | Letter B) =P(#2 $\cap$ Letter B) /P(Letter B) = (1/10) / (2/10) = 1/2.

It is important to notice that conditional probabilities are just like regular ones, for example they obey the axioms of Kolmogorov:

Axiom 1: $P(A|B)=P(A \cap B)/P(B)$ , but $P(A \cap B)$ and P(B) are both regular probabilities, so $P(A \cap B) \ge 0$ , $P(B)>0$ , so $P(A|B)=P(A \cap B)/P(B) \ge 0$ .

Axiom 2: $P(S|B)=P(S \cap B)/P(B)=P(B)/P(B)=1$ .

Axiom 3: say $A_1 ,..,A_n$ are mutually exclusive, then

$\begin{aligned} &P \left(\bigcup_{i=1}^n A_i\vert B \right) = \\ & \frac{P \left(\left\{\bigcup_{i=1}^n A_i\right\}\cap B \right)}{P(B)} = \\ & \frac{P \left(\bigcup_{i=1}^n\left\{ A_i\cap B\right\} \right)}{P(B)} = \\ & \frac{\sum_{i=1}^nP \left( A_i\cap B \right)}{P(B)} = \\ & \sum_{i=1}^n\frac{P \left( A_i\cap B \right)}{P(B)} = \\ &\sum_{i=1}^n P(A_i\vert B) \end{aligned}$

Multiplication Rule

A simple manipulation of the equation above yields

$P(A \cap B)=P(A|B)P(B)$ Notice that this formula can also be applied to conditional probabilities:

$P(A\vert B\cap C) = \frac{P(A\cap B \cap C)}{P(B \cap C)}$

$P(A\cap B \cap C) = P(A\vert B\cap C)P(B \cap C)$

Example (1.3.3)

You draw two cards from a standard 52-card deck. What is the probability to draw 2 Aces?

Solution: let

A = “First card drawn is an ace”
B = “Second card drawn is an ace”

Then

P(both cards are aces) =
P(first card is an ace and second card is an ace) =
$P(A\cap B)=P(A)P(B|A) = \frac{4}{52} \frac{3}{51}$

It’s easy to extend this to more than two events: What is the probability of drawing 3 aces when drawing 3 cards?

Let $A_i$ = “i^th card drawn is an ace”

Then

$\begin{aligned} &P(\text{all three cards are aces}) = \\ &P(A_1\cap A_2\cap A_3) = \\ &P(A_1) \frac{P(A_1\cap A_2)}{P(A_1)} \frac{P(A_1\cap A_2\cap A_3)}{P(A_1\cap A_2)} = \\ &P(A_1)P(A_2\vert A_1)P(A_3\vert A_1\cap A_2) \end{aligned}$

even a little more complicated: In most Poker games you get in the first round 5 cards (Later you can exchange some you don’t like but we leave that out). What is the probability that you get 4 aces?

$\begin{aligned} &\text{all five cards are aces} = \\ &\left(A_1\cap A_2\cap A_3\cap A_4\cap A_5^c\right) \cup \\ &\left(A_1\cap A_2\cap A_3\cap A_4^c\cap A_5\right) \cup \\ &\left(A_1\cap A_2\cap A_3^c\cap A_4\cap A_5\right) \cup \\ &\left(A_1\cap A_2^c\cap A_3\cap A_4\cap A_5\right) \cup \\ &\left(A_1^c\cap A_2\cap A_3\cap A_4\cap A_5\right) \end{aligned}$

These events are mutually exclusive, and so

$\begin{aligned} &P \left(\text{all five cards are aces} \right) = \\ &P\left(A_1\cap A_2\cap A_3\cap A_4\cap A_5^c\right) + \\ &P\left(A_1\cap A_2\cap A_3\cap A_4^c\cap A_5\right) + \\ &P\left(A_1\cap A_2\cap A_3^c\cap A_4\cap A_5\right) + \\ &P\left(A_1\cap A_2^c\cap A_3\cap A_4\cap A_5\right) + \\ &P\left(A_1^c\cap A_2\cap A_3\cap A_4\cap A_5\right) =\\ &\\ & \frac{4}{52}\frac{3}{51}\frac{2}{50}\frac{1}{49}\frac{48}{48} + \\ & \frac{4}{52}\frac{3}{51}\frac{2}{50}\frac{48}{48}\frac{1}{49} + \\ & \frac{4}{52}\frac{3}{51}\frac{48}{48}\frac{2}{50}\frac{1}{49} + \\ & \frac{4}{52}\frac{48}{48}\frac{3}{51}\frac{2}{50}\frac{1}{49} + \\ & \frac{48}{48}\frac{4}{52}\frac{3}{51}\frac{2}{50}\frac{1}{49} = \\ &\\ &5\frac{4}{52}\frac{3}{51}\frac{2}{50}\frac{1}{49}=1.847\times 10^{-5} \end{aligned}$

Law of Total Probability and Bayes Rule

Definition (1.3.4)

A set of events $\{A_i\}$ is called a partition of the sample space S if

$\begin{aligned} &A_i \cap A_j = \emptyset \text{ for all } i \ne j \\ &\bigcup_{i=1}^{\infty} A_i = S\\ \end{aligned}$

Example (1.3.5)

a student is selected at random from all the undergraduate students at the Colegio

A₁ = “Student is female”, A₂ = “Student is male”

or maybe

A₁ = “Student is freshman”, .., A₄ = “Student is senior”

Theorem (1.3.6)

Law of Total Probability

Let the set of events $\{A_i\}$ be a partition, and let B be any event, then

$P(B) = \sum_{i=1}^{\infty} P(B|A_i)P(A_i)$

proof

$\begin{aligned} &P(B) = P(B \cap S) = \\ &P(B \cap ( \cup^\infty_{i=1} A_i ) = \\ &P( \cup^\infty_{i=1} (B \cap A_i ) = \\ &\sum^\infty_{i=1} P(B \cap A_i ) =\\ &\sum^\infty_{i=1} P(B|A_i )P(A_i ) \end{aligned}$

Example (1.3.7)

A company has 452 employees, 210 men and 242 women. 15% of the men and 10% of the women have a managerial position. What is the probability that a randomly selected person in this company has a managerial position?

Let A₁ = “person is female”, A₂ = “person is male”.

Let B = “person has a managerial position”

Then

$\begin{aligned} &P(A_1) = \frac{242}{452} \\ &P(A_2) = \frac{210}{452} \\ &P(B) = P(B|A_1)P(A_1)+P(B|A_2)P(A_2)=\\ &0.1\frac{242}{452}+0.15\frac{210}{452}=0.123 \end{aligned}$

Example (1.3.8)

Say you roll a fair die. If you roll an even number you roll the die again, otherwise you keep the result of the first roll. What are the probabilities of rolling a 1, or a 2 or , ..,6?

Let the event A_i=“First roll was i”. This forms a partition, and so

$\begin{aligned} &P(1) =\sum_{i=1}^6 P(1\vert A_i)P(A_i) = \\ &1 \frac{1}{6} + \frac{1}{6}\frac{1}{6}+0\frac{1}{6}+\frac{1}{6}\frac{1}{6}+0\frac{1}{6}+\frac{1}{6}\frac{1}{6} = \\ &\frac{1}{6}(1+\frac{1}{2}) = \frac{1}{4}\\ &\\ &P(2) =\sum_{i=1}^6 P(2\vert A_i)P(A_i) = \\ &0 \frac{1}{6} + \frac{1}{6}\frac{1}{6}+0\frac{1}{6}+\frac{1}{6}\frac{1}{6}+0\frac{1}{6}+\frac{1}{6}\frac{1}{6} = \\ &\frac{1}{6}(\frac{1}{2}) = \frac{1}{12}\\ \end{aligned}$

and the same for 3-6. Note that

$3 \frac{1}{4}+3 \frac{1}{12}=3 \frac{3+1}{12}=1$

Theorem (1.3.9)

Bayes’ Formula

Let the set of events $\{A_i\}$ be a partition, and let B be any event, then

$P(A_k|B)=\frac{P(B|A_k)P(A_k)}{\sum_{i=1}^{\infty} P(B|A_i)P(A_i)}$

Notice that the denominator is just the law of total probability, so we could have written the formula also in this way

$P(A_k|B)=\frac{P(B|A_k)P(A_k)}{P(B)}$

only usually the first form is the one that is needed because of the available information.

proof

$\begin{aligned} &P(A_k |B) = \\ &P(A_k \cap B)/P(B) = \\ &P(B \cap A_k )/P(B) = \\ &P(B|A_k )P(A_k )/P(B) \end{aligned}$

Example (1.3.10)

In the company above a person is randomly selected, and that person is in a managerial position. What is the probability the person is female?

$P(A_1|B) = \frac{P(B|A_1)P(A_1)}{P(B}=$
$\frac{0.1\times242/452}{0.123}=0.434$

Bayes formula sometimes results in strange answers:

Example (1.3.11)

Say in some population 1 in 500 people have a certain virus. A test for the virus has a “false-positive” (aka a person who does not have the virus gets a positive test) of 1 in 25, and a “false-negative” (aka a person who does have the virus gets a negative test) of 1 in 50. Say a randomly chose person tests positive. What is the probability that the person actually has the virus?

Let’s define the events A₁=“person has virus”, A₂=“person does not have virus” and B=“person tests positive”, then

$\begin{aligned} &P(\text{person has virus }\vert\text{ person tests positive}) = \\ &P(A_1\vert B) = \\ &\frac{P(B\vert A_1)P(A_1)}{P(B\vert A_1)P(A_1)+P(B\vert A_2)P(A_2)} = \\ & \frac{49/50\times 1/500}{49/50\times1/500+1/25\times499/500} = \\ & \frac{49/50}{49/50+1/25\times499} = 0.0468 \end{aligned}$ So also the test is very good, the probability is still very small!

Bayes’ Rule plays a very important role in Statistics and in Science in general. It provides a natural method for updating your knowledge based on data.

Independence

Sometimes knowing that one event occurred does not effect the probability of another event. For example if you through a red and a blue die, knowing that the red die shows a “6” will not change the probability that the blue die shows a “2”.

Formally we have

$P(A|B)=P(A)$

or using the multiplication rule we get the previous formula for two independent events

$P(A\cap B)=P(A|B)P(B)=P(A)P(B)$

Example (1.3.12)

Say you flip a fair coin 5 times. What is the probability of 5 “heads”?

Let A_i = “i^th flip is heads”

Now it is reasonable to assume that the A_i’s are independent and so

$\begin{aligned} &P\left(A_1\cap A_2\cap A_3\cap A_4\cap A_5\right) = \\ &P(A_1)P(A_2)P(A_3)P(A_4)P(A_5) = \\ & (\frac{1}{2})^5 = \frac{1}{32}=0.03125\\ \end{aligned}$