10 Conditional Probability, Independence, and Bayes' Theorem

Lecture from: 20.03.2025 | Video: Homelab

Conditional Probability (Bedingte Wahrscheinlichkeit)

Often, we have partial information about the outcome of a random experiment, and we want to know how this information affects the probabilities of other events. This leads to the concept of conditional probability.

Intuition: What is the probability of event $A$ happening, given that we know event $B$ has already occurred?

Motivating Example: Card Game

Recall the scenario: Shuffle a 52-card deck, deal 5 cards to Player 1 ( $X$ ) and 5 cards to Player 2 ( $Y$ ). The sample space $Ω$ consists of all pairs of disjoint 5-card hands $(X, Y)$ , with $∣Ω∣ = (5 52) (5 47)$ . This is a Laplace space.

Let $A$ be the event “Player 1 has all four Aces”. We previously calculated $∣ A ∣$ . To form Player 1’s hand, we must choose all 4 Aces ( $(4 4) = 1$ way) and 1 non-Ace ( $(1 48) = 48$ ways). So Player 1’s hand $X$ can be chosen in $1 \times 48 = 48$ ways. Once $X$ is chosen, Player 2’s hand $Y$ can be any 5 cards from the remaining 47 cards, so there are $(5 47)$ ways to choose $Y$ . $∣ A ∣ = 48 \times (5 47)$ . $Pr [A] = \frac{∣ A ∣}{∣Ω∣} = \frac{48 ( 5 47 )}{( 5 52 ) ( 5 47 )} = \frac{48}{( 5 52 )} = \frac{48}{2 , 598 , 960} \approx 0.000018$ .

Now, let’s view this from Player 2’s perspective. Suppose Player 2 looks at their hand $Y$ and sees a specific set of cards, say $B = {♡9, ♠5, ♣6, ♢6, ♠ K}$ . This hand $B$ contains no Aces. Does knowing Player 2’s hand change the probability that Player 1 has four Aces?

Player 2 is interested in: $Pr [Player 1 has 4 Aces ∣ Player 2 has hand B]$

This “probability of A given B” is the conditional probability.

Definition 2.7: Conditional Probability

Let $A$ and $B$ be events in a probability space $Ω$ . If $Pr [B] > 0$ , the conditional probability of $A$ given $B$ , denoted $Pr [A ∣ B]$ , is defined as:

$Pr [A ∣ B] := \frac{Pr [ A \cap B ]}{Pr [ B ]}$

Interpretation: Knowing that event $B$ has occurred effectively restricts the possible outcomes to those in $B$ . We are interested in the probability that an outcome also belongs to $A$ , relative to the “new” sample space $B$ . The probability $Pr [A \cap B]$ represents the portion of the original sample space where both $A$ and $B$ occur. We renormalize this probability by dividing by $Pr [B]$ so that the probabilities within the restricted space $B$ sum to 1.

As the image indicates:

$Pr [A ∣ B]$ is the probability that event A occurs, when we already know that event B has occurred.
The formula $\frac{Pr [ A \cap B ]}{Pr [ B ]}$ is how we calculate this conditional probability.

Applying to the Card Example (Intuitive Calculation): If Player 2 has hand $B$ (5 specific non-Ace cards), then there are 47 cards remaining for Player 1. For Player 1 to have 4 Aces, their hand must consist of the 4 Aces and 1 other card from the $47 - 4 = 43$ cards that are not Player 2’s cards and not Aces. So, given Player 2’s hand $B$ , there are 43 possible hands for Player 1 that satisfy event $A$ (“Player 1 has 4 Aces”).

How many possible hands could Player 1 have, given Player 2’s hand $B$ ? Player 1 must have 5 cards chosen from the 47 cards not in $B$ . The total number of possible hands for Player 1 is $(5 47)$ .

If we treat the set of possible hands for Player 1 given Player 2’s hand $B$ as a new Laplace space, then: $Pr [A ∣ B] = \frac{Number of A’s hands with 4 Aces}{Total number of possible hands for A} = \frac{43}{( 5 47 )} = \frac{43}{1 , 533 , 939} \approx 0.000028$

Applying the Definition (Formal Calculation): Let $B$ be the event “Player 2 has the specific hand ${♡9, ♠5, ♣6, ♢6, ♠ K}$ ”. $Pr [B] = \frac{( 5 47 )}{( 5 52 ) ( 5 47 )} = \frac{1}{( 5 52 )}$ . (Only 1 way for Player 2 to have this specific hand $B$ , given Player 1’s hand is chosen from the remaining 47). A slight subtlety: $∣Ω∣ = (5 52) (5 47)$ . The number of outcomes where Player 2 has hand B is $(5 47)$ (Player 1 can have any hand from the remaining 47). So $Pr [B] = \frac{( 5 47 )}{∣Ω∣}$ . Let $A$ be the event “Player 1 has 4 Aces”. $A \cap B$ is the event “Player 1 has 4 Aces AND Player 2 has hand B”. The number of outcomes in $A \cap B$ : Player 1’s hand must be {4 Aces, 1 non-Ace other than those in B} (43 choices for the non-Ace). Player 2’s hand is fixed as B. So $∣ A \cap B ∣ = 43$ . $Pr [A \cap B] = \frac{∣ A \cap B ∣}{∣Ω∣} = \frac{43}{( 5 52 ) ( 5 47 )}$ .

$Pr [A ∣ B] = \frac{Pr [ A \cap B ]}{Pr [ B ]} = \frac{43/∣Ω∣}{( 5 47 ) /∣Ω∣} = \frac{43}{( 5 47 )}$ . This matches the intuitive calculation. The probability increased slightly because Player 2 holding non-Aces makes it slightly more likely that the Aces are concentrated elsewhere.

Example: The Two-Child Problem

This classic problem highlights the importance of carefully defining the sample space and the conditioning event.

Scenario: A family has two children. We assume the gender of each child (M=Male, W=Female) is equally likely (like a coin flip) and independent.

Sample Space: $Ω = {MM, M W, W M, WW}$ . Here, the order matters (e.g., MW means older is Male, younger is Female). This is a Laplace space with $∣Ω∣ = 4$ . $Pr [MM] = Pr [M W] = Pr [W M] = Pr [WW] = 1/4$ .

Events: Female

$A$ : “Both children are female”. $A = {WW}$ . $Pr [A] = 1/4$ .
$B$ : “At least one child is female”. $B = {M W, W M, WW}$ . $Pr [B] = 3/4$ .

Question 1: What is the probability that both children are female, given that at least one child is female? We want $Pr [A ∣ B]$ . $A \cap B = {WW} \cap {M W, W M, WW} = {WW}$ . $Pr [A \cap B] = Pr [{WW}] = 1/4$ . $Pr [A ∣ B] = \frac{Pr [ A \cap B ]}{Pr [ B ]} = \frac{1/4}{3/4} = \frac{1}{3}$ .

Question 2 (Variation): What is the probability that both children are female, given that the older child is female? Let $C$ be the event “Older child is female”. $C = {W M, WW}$ . $Pr [C] = 2/4 = 1/2$ . We want $Pr [A ∣ C]$ . $A \cap C = {WW} \cap {W M, WW} = {WW}$ . $Pr [A \cap C] = Pr [{WW}] = 1/4$ . $Pr [A ∣ C] = \frac{Pr [ A \cap C ]}{Pr [ C ]} = \frac{1/4}{1/2} = \frac{1}{2}$ .

Comparison: Knowing at least one child is female yields a 1/3 probability for both being female. Knowing the older child is female yields a 1/2 probability. The specific information matters!

Multiplication Rule (Multiplikationssatz)

Rearranging the definition of conditional probability gives a way to calculate the probability of an intersection:

If $Pr [B] > 0$ , then $Pr [A \cap B] = Pr [A ∣ B] \cdot Pr [B]$ . Similarly, if $Pr [A] > 0$ , then $Pr [A \cap B] = Pr [B ∣ A] \cdot Pr [A]$ .

This can be generalized to the intersection of multiple events, often called the Chain Rule of Probability.

Satz 2.10 (Multiplikationssatz): Let $A_{1}, ..., A_{n}$ be events. If $Pr [A_{1} \cap \dots \cap A_{n}] > 0$ (which implies all intermediate intersections also have positive probability), then:

$Pr [A_{1} \cap \dots \cap A_{n}] = Pr [A_{1}] \cdot Pr [A_{2} ∣ A_{1}] \cdot Pr [A_{3} ∣ A_{1} \cap A_{2}] \cdot \dots \cdot Pr [A_{n} ∣ A_{1} \cap \dots \cap A_{n - 1}]$

Proof: The right-hand side is a telescoping product: $Pr [A_{1}] \cdot \frac{Pr [ A _{1} \cap A _{2} ]}{Pr [ A _{1} ]} \cdot \frac{Pr [ A _{1} \cap A _{2} \cap A _{3} ]}{Pr [ A _{1} \cap A _{2} ]} \cdot \dots \cdot \frac{Pr [ A _{1} \cap \dots \cap A _{n} ]}{Pr [ A _{1} \cap \dots \cap A _{n - 1} ]}$ All intermediate terms cancel out, leaving only $Pr [A_{1} \cap \dots \cap A_{n}]$ . (The division by probabilities is valid because the condition $Pr [A_{1} \cap \dots \cap A_{n}] > 0$ ensures that $Pr [A_{1} \cap \dots \cap A_{k}] > 0$ for all $k < n$ ).

Application: The Birthday Problem (Geburtstagsproblem)

Question: In a room with $m$ people, what is the probability that at least two people share the same birthday? (Assuming $n = 365$ possible birthdays, equally likely, ignoring leap years).

Reformulation (Balls and Bins): Randomly assign each of $m$ people (balls) to one of $n = 365$ birthdays (bins). What is the probability that at least one bin contains more than one ball?

Strategy: It’s easier to calculate the probability of the complementary event, $E$ : “All $m$ people have different birthdays”. The probability we want is $1 - Pr [E]$ .

Sample Space: We can model the assignment of birthdays as an ordered sequence of length $m$ , where each element is chosen from ${1, ..., n}$ with replacement. $Ω = {(b_{1}, b_{2}, ..., b_{m}) ∣ b_{i} \in {1, ..., n}}$ . $∣Ω∣ = n^{m}$ . This is a Laplace space.

Calculating $Pr [E]$ using the Multiplication Rule: Let $A_{i}$ be the event “The $i$ -th person’s birthday is different from the birthdays of the first $i - 1$ people”. The event $E$ (all birthdays are different) is the intersection $E = A_{1} \cap A_{2} \cap \dots \cap A_{m}$ . Using the multiplication rule: $Pr [E] = Pr [A_{1}] \cdot Pr [A_{2} ∣ A_{1}] \cdot Pr [A_{3} ∣ A_{1} \cap A_{2}] \cdot \dots \cdot Pr [A_{m} ∣ A_{1} \cap \dots \cap A_{m - 1}]$

$Pr [A_{1}]$ : The first person can have any birthday. This event is certain. $Pr [A_{1}] = 1$ .
$Pr [A_{2} ∣ A_{1}]$ : Given the first person has a birthday, there are $n - 1$ remaining birthdays out of $n$ that the second person can have to be different. $Pr [A_{2} ∣ A_{1}] = \frac{n - 1}{n}$ .
$Pr [A_{3} ∣ A_{1} \cap A_{2}]$ : Given the first two people have different birthdays, there are $n - 2$ remaining birthdays out of $n$ that the third person can have to be different from the first two. $Pr [A_{3} ∣ A_{1} \cap A_{2}] = \frac{n - 2}{n}$ .
…
$Pr [A_{i} ∣ A_{1} \cap \dots \cap A_{i - 1}]$ : Given the first $i - 1$ people have different birthdays, there are $n - (i - 1)$ remaining birthdays out of $n$ that the $i$ -th person can have. $Pr [A_{i} ∣ A_{1} \cap \dots \cap A_{i - 1}] = \frac{n - ( i - 1 )}{n} = 1 - \frac{i - 1}{n}$ .

Substituting these into the multiplication rule: $Pr [E] = Pr [A_{1} \cap \dots \cap A_{m}] = 1 \cdot (1 - \frac{1}{n}) \cdot (1 - \frac{2}{n}) \cdot \dots \cdot (1 - \frac{m - 1}{n})$ $Pr [E] = \prod_{i = 1}^{m - 1} (1 - \frac{i}{n})$ (with the first term for $i = 0$ being 1).

Probability of a Match: The probability that at least two people share a birthday is: $Pr [Match] = 1 - Pr [E] = 1 - \prod_{i = 1}^{m - 1} (1 - \frac{i}{n})$

Numerical Example (n=365, m=25): Calculating this product for $m = 25$ gives $Pr [E] \approx 0.431$ . So, $Pr [Match] \approx 1 - 0.431 = 0.569$ . There is about a 57% chance that at least two people share a birthday in a room of 25 people. The probability crosses 50% at $m = 23$ .

Excursion: Approximating Products (Birthday Problem)

For large $n$ and moderate $m$ , we can approximate the product $Pr [E]$ using the fact that $1 - x \approx e^{- x}$ for small $x$ . More accurately, $1 - x = e^{- x - O (x^{2})}$ .

$Pr [E] = \prod_{i = 0}^{m - 1} (1 - \frac{i}{n}) \approx \prod_{i = 0}^{m - 1} e^{- i / n} = e^{- \sum_{i = 0}^{m - 1} i / n} = e^{- \frac{1}{n} \sum_{i = 0}^{m - 1} i}$

Using the formula for the sum of the first $m - 1$ integers, $\sum_{i = 0}^{m - 1} i = \frac{( m - 1 ) m}{2}$ :

$Pr [E] \approx e^{- \frac{m ( m - 1 )}{2 n}}$

This approximation is quite accurate, as shown in the graph on slide 21.

Threshold Behavior: When does $Pr [Match] = 1 - Pr [E] \approx 0.5$ ? This happens when $Pr [E] \approx 0.5$ . $e^{- \frac{m ( m - 1 )}{2 n}} \approx 0.5$ $- \frac{m ( m - 1 )}{2 n} \approx ln (0.5) = - ln (2) \approx - 0.693$ $m (m - 1) \approx 2 n ln (2) \approx 1.386 n$

For large $m$ , $m (m - 1) \approx m^{2}$ . So, $m^{2} \approx 1.386 n$ , which means $m \approx 1.386 n$ . If $n = 365$ , $m \approx 1.386 \times 365 \approx 506 \approx 22.5$ . This confirms that the threshold is around $m = 23$ .

The approximation also shows:

If $m ≪ 2 n$ , then $\frac{m ( m - 1 )}{2 n}$ is small, $Pr [E] \approx e^{0} \approx 1$ . (Low probability of a match).
If $m ≫ 2 n$ , then $\frac{m ( m - 1 )}{2 n}$ is large, $Pr [E] \approx e^{- large} \approx 0$ . (High probability of a match). The transition happens around $m \approx 2 n$ .

Law of Total Probability (Satz von der totalen Wahrscheinlichkeit)

This law provides a way to calculate the probability of an event by conditioning on a set of mutually exclusive and exhaustive events (a partition).

Satz 2.12 (Satz von der totalen Wahrscheinlichkeit): Let $A_{1}, ..., A_{n}$ be pairwise disjoint events such that their union contains event $B$ (i.e., $B \subseteq A_{1} \cup \dots \cup A_{n}$ ). Often, $A_{1}, ..., A_{n}$ form a partition of the entire sample space $Ω$ . Then:

$Pr [B] = \sum_{i = 1}^{n} Pr [B ∣ A_{i}] \cdot Pr [A_{i}]$

Derivation: The sets $B \cap A_{i}$ are pairwise disjoint because the $A_{i}$ are disjoint. Also, $B = B \cap (A_{1} \cup \dots \cup A_{n}) = (B \cap A_{1}) \cup \dots \cup (B \cap A_{n})$ . Using the addition rule for disjoint events (Lemma 2.2, part 5): $Pr [B] = Pr [⋃_{i = 1}^{n} (B \cap A_{i})] = \sum_{i = 1}^{n} Pr [B \cap A_{i}]$ Using the definition of conditional probability, $Pr [B \cap A_{i}] = Pr [B ∣ A_{i}] Pr [A_{i}]$ (assuming $Pr [A_{i}] > 0$ . If $Pr [A_{i}] = 0$ , the term is 0 anyway). Substituting this gives: $Pr [B] = \sum_{i = 1}^{n} Pr [B ∣ A_{i}] Pr [A_{i}]$ .

Intuition: To find the total probability of $B$ , we consider all possible disjoint scenarios $A_{i}$ under which $B$ could occur. For each scenario $A_{i}$ , we find the probability that $B$ occurs given $A_{i}$ ( $Pr [B ∣ A_{i}]$ ) and weight it by the probability of that scenario $A_{i}$ occurring ( $Pr [A_{i}]$ ). Summing these weighted probabilities gives the total probability of $B$ .

Example: ETH vs. UZH Football Let $B$ be the event “ETH wins”. Let $A_{1}$ be the event “Star player Messaldo plays”. Let $A_{2}$ be the event “Messaldo does not play” ( $A_{2} = \overset{ˉ}{A_{1}}$ ). $A_{1}$ and $A_{2}$ form a partition of the sample space.

Given information:

$Pr [B ∣ A_{1}] = Pr [ETH wins ∣ Messaldo plays] = 0.80$ (80%)
$Pr [B ∣ A_{2}] = Pr [ETH wins ∣ Messaldo not plays] = 0.30$ (30%)
$Pr [A_{1}] = Pr [Messaldo plays] = 0.60$ (60%)
$Pr [A_{2}] = Pr [Messaldo not plays] = 1 - Pr [A_{1}] = 0.40$ (40%)

Using the Law of Total Probability to find the overall probability that ETH wins: $Pr [B] = Pr [B ∣ A_{1}] Pr [A_{1}] + Pr [B ∣ A_{2}] Pr [A_{2}]$ $Pr [B] = (0.80) (0.60) + (0.30) (0.40)$ $Pr [B] = 0.48 + 0.12 = 0.60$ The overall probability that ETH wins is 60%.

Bayes’ Theorem

Bayes’ Theorem relates the conditional probability of an event $A_{i}$ given $B$ to the conditional probability of $B$ given $A_{i}$ . It’s incredibly useful for updating beliefs based on new evidence.

Satz (Satz von Bayes): Let $A_{1}, ..., A_{n}$ be pairwise disjoint events such that $B \subseteq A_{1} \cup \dots \cup A_{n}$ . Assume $Pr [B] > 0$ and $Pr [A_{i}] > 0$ for all $i$ . Then for any specific $A_{i}$ :

$Pr [A_{i} ∣ B] = \frac{Pr [ B ∣ A _{i} ] \cdot Pr [ A _{i} ]}{Pr [ B ]}$

Using the Law of Total Probability for the denominator, we get the expanded form:

$Pr [A_{i} ∣ B] = \frac{Pr [ B ∣ A _{i} ] \cdot Pr [ A _{i} ]}{\sum _{j = 1}^{n} Pr [ B ∣ A _{j} ] \cdot Pr [ A _{j} ]}$

Derivation: Start with the definition of conditional probability:

$Pr [A_{i} ∣ B] = \frac{Pr [ A _{i} \cap B ]}{Pr [ B ]}$
$Pr [B ∣ A_{i}] = \frac{Pr [ B \cap A _{i} ]}{Pr [ A _{i} ]} ⟹ Pr [A_{i} \cap B] = Pr [B ∣ A_{i}] Pr [A_{i}]$ Substitute (2) into the numerator of (1): $Pr [A_{i} ∣ B] = \frac{Pr [ B ∣ A _{i} ] Pr [ A _{i} ]}{Pr [ B ]}$ Now substitute the Law of Total Probability for $Pr [B]$ in the denominator to get the expanded form.

Interpretation of Terms:

$Pr [A_{i}]$ : Prior probability of $A_{i}$ . Our belief about $A_{i}$ before observing $B$ .
$Pr [B ∣ A_{i}]$ : Likelihood of observing evidence $B$ given that scenario $A_{i}$ is true.
$Pr [B]$ : Evidence probability (marginal likelihood). The total probability of observing $B$ , averaged over all possible scenarios $A_{j}$ . Acts as a normalization constant.
$Pr [A_{i} ∣ B]$ : Posterior probability of $A_{i}$ . Our updated belief about $A_{i}$ after observing $B$ .

Application: Medical Testing (Breast Cancer Screening)

This is a classic application showing how Bayes’ Theorem helps interpret diagnostic test results.

Scenario: A screening test (digital mammography) for breast cancer. Events:

$K$ : Patient has breast cancer (Krebs).
$\overset{ˉ}{K}$ : Patient does not have breast cancer.
$P os$ : Test result is positive.
$N e g$ : Test result is negative.

Known Statistics (Example 1: Age 50-69):

Sensitivity: $Pr [P os ∣ K] = 0.72$ (72%). Probability test is positive, given patient has cancer.
Specificity: $Pr [N e g ∣ \overset{ˉ}{K}] = 0.98$ (98%). Probability test is negative, given patient does not have cancer.
Prevalence: $Pr [K] = 0.01$ (1%). Probability a woman in this age group has breast cancer (prior).

Derived Probabilities:

False Negative Rate: $Pr [N e g ∣ K] = 1 - Pr [P os ∣ K] = 1 - 0.72 = 0.28$ .
False Positive Rate: $Pr [P os ∣ \overset{ˉ}{K}] = 1 - Pr [N e g ∣ \overset{ˉ}{K}] = 1 - 0.98 = 0.02$ .
Prior (No Cancer): $Pr [\overset{ˉ}{K}] = 1 - Pr [K] = 1 - 0.01 = 0.99$ .

Question: If a woman in this age group tests positive, what is the probability she actually has breast cancer? We want $Pr [K ∣ P os]$ .

Applying Bayes’ Theorem: The partition events are $A_{1} = K$ and $A_{2} = \overset{ˉ}{K}$ . The evidence is $B = P os$ . $Pr [K ∣ P os] = \frac{Pr [ P os ∣ K ] \cdot Pr [ K ]}{Pr [ P os ]}$

First, find the denominator $Pr [P os]$ using the Law of Total Probability: $Pr [P os] = Pr [P os ∣ K] Pr [K] + Pr [P os ∣ \overset{ˉ}{K}] Pr [\overset{ˉ}{K}]$ $Pr [P os] = (0.72) (0.01) + (0.02) (0.99)$ $Pr [P os] = 0.0072 + 0.0198 = 0.0270$

Now calculate the posterior probability: $Pr [K ∣ P os] = \frac{( 0.72 ) ( 0.01 )}{0.0270} = \frac{0.0072}{0.0270} \approx 0.2667$

Result: Even with a positive test, the probability of actually having breast cancer is only about 26.7%! This is because the prevalence is low, and the false positive rate contributes significantly to the positive results.

Example 2 (Age 20-29): Prevalence is much lower: $Pr [K] = 0.0002$ (0.02%). $Pr [\overset{ˉ}{K}] = 0.9998$ . Sensitivity and Specificity are assumed the same. $Pr [P os] = Pr [P os ∣ K] Pr [K] + Pr [P os ∣ \overset{ˉ}{K}] Pr [\overset{ˉ}{K}]$ $Pr [P os] = (0.72) (0.0002) + (0.02) (0.9998)$ $Pr [P os] = 0.000144 + 0.019996 = 0.02014$

$Pr [K ∣ P os] = \frac{( 0.72 ) ( 0.0002 )}{0.02014} = \frac{0.000144}{0.02014} \approx 0.00715$

Result: For a younger woman with much lower prior risk, a positive test yields only about a 0.7% chance of actually having breast cancer.

CS Notes

Explorer