05 Multiple Random Variables

In many probabilistic scenarios, we are not interested in just a single random variable but in the interplay and relationships between several random variables defined on the same sample space. This section delves into the realm of multiple random variables, exploring concepts such as independence, joint distributions, and operations on sets of random variables. Understanding multiple random variables is crucial for modeling complex systems and analyzing dependencies between different aspects of a random experiment.

Independence of Random Variables (2.6.1)

Just as events can be independent, so too can random variables. Intuitively, two random variables are independent if knowing the value of one variable provides no information about the value of the other. Formally, independence of random variables is defined in terms of their joint probability distribution.

Definition 2.52: Random variables $X_{1}, X_{2}, \dots, X_{n}$ are said to be independent if and only if for all possible values $(x_{1}, x_{2}, \dots, x_{n})$ in their respective ranges $W_{X_{1}}, W_{X_{2}}, \dots, W_{X_{n}}$ , the joint probability mass function factorizes into the product of marginal probability mass functions:

P r [X_{1} = x_{1}, X_{2} = x_{2}, \dots, X_{n} = x_{n}] = P r [X_{1} = x_{1}] \cdot P r [X_{2} = x_{2}] \cdot \dots \cdot P r [X_{n} = x_{n}]

This definition extends the concept of independence from events to random variables. It requires that the joint probability of any combination of values is simply the product of the probabilities of each variable taking its respective value.

Alternative Definition using Densities: Equivalently, independence can be defined in terms of density functions: Random variables $X_{1}, \dots, X_{n}$ are independent if and only if for all $(x_{1}, \dots, x_{n}) \in W_{X_{1}} \times \dots \times W_{X_{n}}$ :

f_{X_{1}, \dots, X_{n}} (x_{1}, \dots, x_{n}) = f_{X_{1}} (x_{1}) \cdot f_{X_{2}} (x_{2}) \cdot \dots \cdot f_{X_{n}} (x_{n})

This factorization property greatly simplifies calculations and analysis when dealing with independent random variables.

Lemma 2.53: If $X_{1}, \dots, X_{n}$ are independent random variables, and $S_{1}, \dots, S_{n} \subseteq R$ are arbitrary sets of real numbers, then the events ${X_{1} \in S_{1}}, \dots, {X_{n} \in S_{n}}$ are independent. That is:

P r [X_{1} \in S_{1}, \dots, X_{n} \in S_{n}] = P r [X_{1} \in S_{1}] \cdot P r [X_{2} \in S_{2}] \cdot \dots \cdot P r [X_{n} \in S_{n}]

This lemma is a crucial extension, showing that independence of random variables implies independence of events defined in terms of these variables. It allows us to work with probabilities of random variables falling within arbitrary sets, not just taking specific values, while preserving the independence property.

Corollary 2.54: Subsets of independent random variables are also independent. If $X_{1}, \dots, X_{n}$ are independent and $I = {i_{1}, \dots, i_{k}} \subseteq [n]$ , then $X_{i_{1}}, \dots, X_{i_{k}}$ are also independent.

Satz 2.55: If $f_{1}, \dots, f_{n} : R \to R$ are real-valued functions, and $X_{1}, \dots, X_{n}$ are independent random variables, then $f_{1} (X_{1}), \dots, f_{n} (X_{n})$ are also independent.

This theorem is vital as it allows us to apply functions to independent random variables without destroying their independence. This is frequently used when constructing new random variables from existing independent ones, while preserving the independence property.

Composed Random Variables (2.6.2)

We can create new random variables by combining existing ones through functions. If we have random variables $X_{1}, \dots, X_{n}$ defined on a sample space $Ω$ , and a function $g : R^{n} \to R$ , then $Y = g (X_{1}, \dots, X_{n})$ is also a random variable. The probability distribution of $Y$ is determined by the joint distribution of $X_{1}, \dots, X_{n}$ and the function $g$ .

For example, if $Z = X + Y$ , where $X$ and $Y$ are random variables, then $Z$ is a composed random variable. The probability of $Z$ taking a specific value $z$ can be calculated by summing over all pairs $(x, y)$ such that $x + y = z$ :

P r [Z = z] = x \in W_{X} \sum P r [X = x, Y = z - x]

If $X$ and $Y$ are independent, this simplifies to:

P r [Z = z] = x \in W_{X} \sum P r [X = x] \cdot P r [Y = z - x]

This formula is known as the convolution of the density functions of $X$ and $Y$ .

Satz 2.58: If $X$ and $Y$ are independent random variables and $Z = X + Y$ , then the density function of $Z$ is given by the convolution of the density functions of $X$ and $Y$ :

f_{Z} (z) = x \in W_{X} \sum f_{X} (x) \cdot f_{Y} (z - x)

Moments of Composed Random Variables (2.6.3)

Linearity of Expectation (Satz 2.60)

Linearity of expectation extends to multiple random variables and remains one of the most powerful and versatile tools in probability.

Satz 2.60 (Linearity of Expectation): For random variables $X_{1}, \dots, X_{n}$ and constants $a_{1}, \dots, a_{n} \in R$ , the expectation of a linear combination is the linear combination of expectations:

E [a_{1} X_{1} + a_{2} X_{2} + \dots + a_{n} X_{n}] = a_{1} E [X_{1}] + a_{2} E [X_{2}] + \dots + a_{n} E [X_{n}]

Crucially, linearity of expectation holds regardless of whether the random variables $X_{1}, \dots, X_{n}$ are independent or dependent. This makes it a widely applicable tool.

Multiplicativity of Expectation for Independent Variables (Satz 2.61)

For products of random variables, a similar simplification holds, but only under the assumption of independence.

Satz 2.61 (Multiplicativity of Expectation): If $X_{1}, \dots, X_{n}$ are independent random variables, then the expectation of their product is the product of their expectations:

E [X_{1} \cdot X_{2} \cdot \dots \cdot X_{n}] = E [X_{1}] \cdot E [X_{2}] \cdot \dots \cdot E [X_{n}]

This multiplicativity property is a direct consequence of independence and significantly simplifies calculations involving products of independent random variables.

Variance of Sums of Independent Variables (Satz 2.62)

For the variance of sums of random variables, a similar additive property holds, again under the condition of independence.

Satz 2.62: If $X_{1}, \dots, X_{n}$ are independent random variables and $X = X_{1} + \dots + X_{n}$ , then the variance of the sum is the sum of the variances:

Va r [X] = Va r [X_{1} + \dots + X_{n}] = Va r [X_{1}] + Va r [X_{2}] + \dots + Va r [X_{n}]

This additivity of variance for independent random variables is a powerful tool for analyzing the variability of sums of independent components.

Wald’s Identity (2.6.4)

Wald’s Identity (Satz 2.65) provides a formula for the expectation of a sum of a random number of random variables. It is particularly useful in analyzing processes that proceed in stages, where the number of stages is itself a random variable.

Satz 2.65 (Wald’s Identity): Let $N$ and $X$ be independent random variables, where $N$ takes values in natural numbers ( $N$ ). Let $X_{1}, X_{2}, \dots$ be independent copies of $X$ . Define $Z = \sum_{i = 1}^{N} X_{i}$ . Then, the expectation of $Z$ is given by:

E [Z] = E [N] \cdot E [X]

Wald’s Identity is a powerful and elegant result that connects the expectation of a sum with the expectations of the number of terms and the individual terms, under the crucial assumption of independence between $N$ and the $X_{i}$ ‘s. It has applications in various areas, including queuing theory, branching processes, and the analysis of randomized algorithms.

In summary, this section has expanded our toolkit for working with random variables to encompass multiple random variables and their interactions. We have explored the crucial concept of independence, its implications for joint distributions and moments, and powerful tools like linearity of expectation and Wald’s Identity for analyzing composed random variables. These concepts will be essential for understanding and designing randomized algorithms, where randomness often arises from multiple sources and interacts in complex ways.

Prev: 04 Important Discrete Distributions | Next: 06 Estimating Probabilities

CS Notes

Explorer