A bag contains slips of paper. Each paper has a number and a letter written on them. They are: A5, A7, B1, B2, C2, C4, D2, D4, E1, E3. A slip is chosen at random, what is the probability it has a 2 on it?
Easy: 3/10
Now say somebody picks a slip and tells you it has the letter B on it. Now what is the probability it also has a 2 on it?
Again easy: 1/2
This is an example of a conditional probability. We write
P(#2 | Letter B)
(” probability of #2 given letter B)
Above we found the conditional probability by changing the sample space. First it was S={A5, A7, B1, B2, C2, C4, D2, D4, E1, E3} but once we knew the slip of paper had the letter B it changed to S={B1, B2}.
In general this changing of the sample space is too difficult, but we can find conditional probabilities using the formula
\[P(A|B) = \frac{P(A \cap B)}{P(B)}\]
Note: this only works if P(B)>0.
Note: this formula can also be derived using the idea of coherence and the concept of the sure looser discussed earlier.
using the numbers from the previous exmple:
P(#2 | Letter B) =P(#2 \(\cap\) Letter B) /P(Letter B) = (1/10) / (2/10) = 1/2.
It is important to notice that conditional probabilities are just like regular ones, for example they obey the axioms of Kolmogorov:
Axiom 1: \(P(A|B)=P(A \cap B)/P(B)\), but \(P(A \cap B)\) and P(B) are both regular probabilities, so \(P(A \cap B) \ge 0\), \(P(B)>0\), so \(P(A|B)=P(A \cap B)/P(B) \ge 0\).
Axiom 2: \(P(S|B)=P(S \cap B)/P(B)=P(B)/P(B)=1\).
Axiom 3: say \(A_1 ,..,A_n\) are mutually exclusive, then
\[ \begin{aligned} &P \left(\bigcup_{i=1}^n A_i\vert B \right) = \\ & \frac{P \left(\left\{\bigcup_{i=1}^n A_i\right\}\cap B \right)}{P(B)} = \\ & \frac{P \left(\bigcup_{i=1}^n\left\{ A_i\cap B\right\} \right)}{P(B)} = \\ & \frac{\sum_{i=1}^nP \left( A_i\cap B \right)}{P(B)} = \\ & \sum_{i=1}^n\frac{P \left( A_i\cap B \right)}{P(B)} = \\ &\sum_{i=1}^n P(A_i\vert B) \end{aligned} \]
A simple manipulation of the equation above yields
\[P(A \cap B)=P(A|B)P(B)\] Notice that this formula can also be applied to conditional probabilities:
\[P(A\vert B\cap C) = \frac{P(A\cap B \cap C)}{P(B \cap C)}\]
\[P(A\cap B \cap C) = P(A\vert B\cap C)P(B \cap C)\]
You draw two cards from a standard 52-card deck. What is the probability to draw 2 Aces?
Solution: let
A = “First card drawn is an ace”
B = “Second card drawn is an ace”
Then
P(both cards are aces) =
P(first card is an ace and second card is an ace) =
\(P(A\cap B)=P(A)P(B|A) = \frac{4}{52} \frac{3}{51}\)
It’s easy to extend this to more than two events: What is the probability of drawing 3 aces when drawing 3 cards?
Let \(A_i\) = “ith card drawn is an ace”
Then
\[ \begin{aligned} &P(\text{all three cards are aces}) = \\ &P(A_1\cap A_2\cap A_3) = \\ &P(A_1) \frac{P(A_1\cap A_2)}{P(A_1)} \frac{P(A_1\cap A_2\cap A_3)}{P(A_1\cap A_2)} = \\ &P(A_1)P(A_2\vert A_1)P(A_3\vert A_1\cap A_2) \end{aligned} \]
even a little more complicated: In most Poker games you get in the first round 5 cards (Later you can exchange some you don’t like but we leave that out). What is the probability that you get 4 aces?
\[ \begin{aligned} &\text{all five cards are aces} = \\ &\left(A_1\cap A_2\cap A_3\cap A_4\cap A_5^c\right) \cup \\ &\left(A_1\cap A_2\cap A_3\cap A_4^c\cap A_5\right) \cup \\ &\left(A_1\cap A_2\cap A_3^c\cap A_4\cap A_5\right) \cup \\ &\left(A_1\cap A_2^c\cap A_3\cap A_4\cap A_5\right) \cup \\ &\left(A_1^c\cap A_2\cap A_3\cap A_4\cap A_5\right) \end{aligned} \]
These events are mutually exclusive, and so
\[ \begin{aligned} &P \left(\text{all five cards are aces} \right) = \\ &P\left(A_1\cap A_2\cap A_3\cap A_4\cap A_5^c\right) + \\ &P\left(A_1\cap A_2\cap A_3\cap A_4^c\cap A_5\right) + \\ &P\left(A_1\cap A_2\cap A_3^c\cap A_4\cap A_5\right) + \\ &P\left(A_1\cap A_2^c\cap A_3\cap A_4\cap A_5\right) + \\ &P\left(A_1^c\cap A_2\cap A_3\cap A_4\cap A_5\right) =\\ &\\ & \frac{4}{52}\frac{3}{51}\frac{2}{50}\frac{1}{49}\frac{48}{48} + \\ & \frac{4}{52}\frac{3}{51}\frac{2}{50}\frac{48}{48}\frac{1}{49} + \\ & \frac{4}{52}\frac{3}{51}\frac{48}{48}\frac{2}{50}\frac{1}{49} + \\ & \frac{4}{52}\frac{48}{48}\frac{3}{51}\frac{2}{50}\frac{1}{49} + \\ & \frac{48}{48}\frac{4}{52}\frac{3}{51}\frac{2}{50}\frac{1}{49} = \\ &\\ &5\frac{4}{52}\frac{3}{51}\frac{2}{50}\frac{1}{49}=1.847\times 10^{-5} \end{aligned} \]
A set of events \(\{A_i\}\) is called a partition of the sample space S if
\[ \begin{aligned} &A_i \cap A_j = \emptyset \text{ for all } i \ne j \\ &\bigcup_{i=1}^{\infty} A_i = S\\ \end{aligned} \]
a student is selected at random from all the undergraduate students at the Colegio
A1 = “Student is female”, A2 = “Student is male”
or maybe
A1 = “Student is freshman”, .., A4 = “Student is senior”
Law of Total Probability
Let the set of events \(\{A_i\}\) be a partition, and let B be any event, then
\[P(B) = \sum_{i=1}^{\infty} P(B|A_i)P(A_i)\]
proof
\[ \begin{aligned} &P(B) = P(B \cap S) = \\ &P(B \cap ( \cup^\infty_{i=1} A_i ) = \\ &P( \cup^\infty_{i=1} (B \cap A_i ) = \\ &\sum^\infty_{i=1} P(B \cap A_i ) =\\ &\sum^\infty_{i=1} P(B|A_i )P(A_i ) \end{aligned} \]
A company has 452 employees, 210 men and 242 women. 15% of the men and 10% of the women have a managerial position. What is the probability that a randomly selected person in this company has a managerial position?
Let A1 = “person is female”, A2 = “person is male”.
Let B = “person has a managerial position”
Then
\[ \begin{aligned} &P(A_1) = \frac{242}{452} \\ &P(A_2) = \frac{210}{452} \\ &P(B) = P(B|A_1)P(A_1)+P(B|A_2)P(A_2)=\\ &0.1\frac{242}{452}+0.15\frac{210}{452}=0.123 \end{aligned} \]
Say you roll a fair die. If you roll an even number you roll the die again, otherwise you keep the result of the first roll. What are the probabilities of rolling a 1, or a 2 or , ..,6?
Let the event Ai=“First roll was i”. This forms a partition, and so
\[ \begin{aligned} &P(1) =\sum_{i=1}^6 P(1\vert A_i)P(A_i) = \\ &1 \frac{1}{6} + \frac{1}{6}\frac{1}{6}+0\frac{1}{6}+\frac{1}{6}\frac{1}{6}+0\frac{1}{6}+\frac{1}{6}\frac{1}{6} = \\ &\frac{1}{6}(1+\frac{1}{2}) = \frac{1}{4}\\ &\\ &P(2) =\sum_{i=1}^6 P(2\vert A_i)P(A_i) = \\ &0 \frac{1}{6} + \frac{1}{6}\frac{1}{6}+0\frac{1}{6}+\frac{1}{6}\frac{1}{6}+0\frac{1}{6}+\frac{1}{6}\frac{1}{6} = \\ &\frac{1}{6}(\frac{1}{2}) = \frac{1}{12}\\ \end{aligned} \]
and the same for 3-6. Note that
\[3 \frac{1}{4}+3 \frac{1}{12}=3 \frac{3+1}{12}=1\]
Bayes’ Formula
Let the set of events \(\{A_i\}\) be a partition, and let B be any event, then
\[P(A_k|B)=\frac{P(B|A_k)P(A_k)}{\sum_{i=1}^{\infty} P(B|A_i)P(A_i)}\]
Notice that the denominator is just the law of total probability, so we could have written the formula also in this way
\[P(A_k|B)=\frac{P(B|A_k)P(A_k)}{P(B)}\]
only usually the first form is the one that is needed because of the available information.
proof
\[ \begin{aligned} &P(A_k |B) = \\ &P(A_k \cap B)/P(B) = \\ &P(B \cap A_k )/P(B) = \\ &P(B|A_k )P(A_k )/P(B) \end{aligned} \]
In the company above a person is randomly selected, and that person is in a managerial position. What is the probability the person is female?
\[P(A_1|B) = \frac{P(B|A_1)P(A_1)}{P(B}=\]
\[\frac{0.1\times242/452}{0.123}=0.434\]
Bayes formula sometimes results in strange answers:
Say in some population 1 in 500 people have a certain virus. A test for the virus has a “false-positive” (aka a person who does not have the virus gets a positive test) of 1 in 25, and a “false-negative” (aka a person who does have the virus gets a negative test) of 1 in 50. Say a randomly chose person tests positive. What is the probability that the person actually has the virus?
Let’s define the events A1=“person has virus”, A2=“person does not have virus” and B=“person tests positive”, then
\[ \begin{aligned} &P(\text{person has virus }\vert\text{ person tests positive}) = \\ &P(A_1\vert B) = \\ &\frac{P(B\vert A_1)P(A_1)}{P(B\vert A_1)P(A_1)+P(B\vert A_2)P(A_2)} = \\ & \frac{49/50\times 1/500}{49/50\times1/500+1/25\times499/500} = \\ & \frac{49/50}{49/50+1/25\times499} = 0.0468 \end{aligned} \] So also the test is very good, the probability is still very small!
Bayes’ Rule plays a very important role in Statistics and in Science in general. It provides a natural method for updating your knowledge based on data.
Sometimes knowing that one event occurred does not effect the probability of another event. For example if you through a red and a blue die, knowing that the red die shows a “6” will not change the probability that the blue die shows a “2”.
Formally we have
\[P(A|B)=P(A)\]
or using the multiplication rule we get the previous formula for two independent events
\[P(A\cap B)=P(A|B)P(B)=P(A)P(B)\]
Say you flip a fair coin 5 times. What is the probability of 5 “heads”?
Let Ai = “ith flip is heads”
Now it is reasonable to assume that the Ai’s are independent and so
\[ \begin{aligned} &P\left(A_1\cap A_2\cap A_3\cap A_4\cap A_5\right) = \\ &P(A_1)P(A_2)P(A_3)P(A_4)P(A_5) = \\ & (\frac{1}{2})^5 = \frac{1}{32}=0.03125\\ \end{aligned} \]