Bayes theorem is used to calculate the probability of an event conditional on another. Quite often a conditional probability \(P(A|B) \) is easy to obtain but we want to know \(P(B|A) \). Bayes theorem allows to calculate the latter knowing the former. Lets go ahead and derive Bayes theorem which is surprising trivial.
Bayes Theorem
$$ P(B|A) = P(A|B) \frac{P(B)}{P(A)} $$
Proof.
We know \( P(A|B) = \frac{P(A \cap B)}{P(B)} \) and similarly \( P(B|A) = \frac{P(B \cap A)}{P(A)} \) from the previous section on independence and conditional probability.
Therefore $$ P(B|A)P(A) = P(A|B)P(B) $$ and $$ P(B|A) = P(A|B) \frac{P(B)}{P(A)} $$
Also useful to calculate \( P(A) \) in the denominator of Bayes theorem is to use the disjoint events \( B \) and \( \bar{B} \) (not B) are such that \( P(B) + P(\bar{B}) = 1 \)
\begin{align*}
P(A \cap B ) + P(A \cap \bar{B} ) & = P(A) \\
\implies P(A|B)P(B) + P(A|\bar{B})P(\bar{B}) & = P(A) \\
\end{align*}
Very Useful!
Example 1.
There are 10 coins in a hat. Nine are fair coins and one is double headed.
You pick a coin from the hat at random and toss it 5 times. You get five heads. Given this information what is the probability you picked the biased coin from the hat?
Let \(A = \text{event you roll 5 heads}\)
Let \(B = \text{event you picked the biased coin from the hat}\)
We want to calculate (P(B|A))
We will use Bayes theorem.
$$P(B|A) = P(A|B)\frac{P(B)}{P(A)}$$
\begin{align*}
P(A|B) &= 1^5 \nonumber \\
P(A|\tilde{B}) &= (\frac{1}{2})^5 \\
P(A) &= P(A \cap B) +
P(A \cap \tilde{B}) \\
&= P(A|B)P(B) + P(A|\tilde{B})P(B) \\
& = 1. \frac{1}{10} + \frac{1}{32}.\frac{9}{10} \\
\end{align*}
therefore
$$ P(B|A) = \frac{1.\frac{1}{10}}{\frac{1}{10}+\frac{1}{32}.\frac{9}{10} }= 0.78 $$
Notice the probability of picking the biased coin tends to 1 as you increase the number of tosses and it always returns a
head.
Example 2.
The Monty Hall Problem
This is a classic probability puzzle. A reader posted this problem in Marilyn Vos Savant’s ‘Ask Marilyn’ column in Parade Magazine in 1990. She was famous for having an extremely high IQ and her answer courted some controversy at the time. Let’s have a look at the problem.
“Suppose you’re on a game show and you are given the choice of 3 doors. Behind one door
is a car, behind the other two doors are one goat each. You pick a door, say no 1,
and the host who knows what’s behind the doors opens another door, say No 3, which has a goat. ”
He then says ‘Do you want to pick No. 2″
Should you switch doors? The crucial assumption is whether the host pick from the remaining doors at random or not. Let’s assume he always picks the goat. Clearly if you do not switch doors the probability of picking the car is \(\frac{1}{3}\)
Let’s assume you follow the strategy of switching doors.
Let (A) be the event of choosing the car on the first pick.
Let (B) be the event of choosing the car on the second pick, after changing door.
\begin{align}
P(B) &= P(B \cap A) + P(B \cap \tilde{A}) \\
& = 0 + P(B \cap \tilde{A}) \\
P(B \cap \tilde{A}) & = P(\tilde{A}) = \frac{2}{3}
\end{align}
If we answer a slightly different question you will see that there is no benefit in switching doors.
This time we the host opens one of the 2 remaining doors at random. Again if you do not
switch the probability of winning the car is (\frac{1}{3}).
If you follow the strategy of switching then you can only do this if the host doesn’t pick the car.
Again Let (A) be the event of choosing the car on the first pick.
Let (B) be the event of choosing the car on the second pick, after changing door.
\begin{align}
&P(\text{winning the car}) \;= P(\tilde{A} \cap B) \\
& = P(B|\tilde{A})P(\tilde{A}) \\
& = \frac{1}{2} . \frac{2}{3} = \frac{1}{3} \\
\end{align}
Hence there is no benefit in following the switching strategy.