Conditional Probability and Independence

Conditional probability describes how the probability of an event changes after information about another event is available.

Conditional probability

DefinitionConditional probability

For events $A$ and $B$ with $P(B)>0$, the conditional probability of $A$ given $B$ is

$$P(A\mid B)=\frac{P(A\cap B)}{P(B)}.$$

The definition immediately gives the multiplication rule

$$P(A\cap B)=P(A\mid B)P(B)=P(B\mid A)P(A),$$

whenever both conditional probabilities are defined. Conditioning restricts attention to $B$ and renormalizes the probabilities inside that event.

TheoremLaw of total probability

If $B_1,\ldots,B_n$ are pairwise disjoint events whose union is $\Omega$, then

$$P(A)=\sum_{i=1}^{n}P(A\mid B_i)P(B_i).$$

The events $B_i$ form a partition of the sample space. The theorem is just the disjoint decomposition

$$A=\bigcup_{i=1}^{n}(A\cap B_i)$$

followed by the multiplication rule.

TheoremBayes' theorem

For events $A$ and $B$ with positive denominators,

$$P(B\mid A)=\frac{P(A\mid B)P(B)}{P(A)}.$$

If $B_1,\ldots,B_n$ is a partition, then

$$P(B_j\mid A)= \frac{P(A\mid B_j)P(B_j)} {\sum_{i=1}^{n}P(A\mid B_i)P(B_i)}.$$

The numerator combines a prior probability $P(B_j)$ with the likelihood $P(A\mid B_j)$. The denominator normalizes over all possible explanations of the evidence $A$.

ExampleA diagnostic test

Suppose a condition has prior probability $P(D)=0.01$, a test has sensitivity $P(+\mid D)=0.95$, and its false-positive rate is $P(+\mid D^c)=0.05$. Then

$$P(+)=0.95(0.01)+0.05(0.99)=0.059,$$

and Bayes’ theorem gives

$$P(D\mid +)=\frac{0.95(0.01)}{0.059}\approx0.161.$$

Even a fairly accurate test can have a modest positive predictive value when the condition is rare.

Independence

DefinitionIndependent events

Events $A$ and $B$ are independent if

$$P(A\cap B)=P(A)P(B).$$

If $P(B)>0$, this is equivalent to $P(A\mid B)=P(A)$.

Independence means that learning whether one event occurred does not change the probability of the other. It is stronger than being disjoint. If $A$ and $B$ are disjoint and both have positive probability, they cannot be independent because $P(A\cap B)=0$ while $P(A)P(B)>0$.

For independent events, complements preserve independence. For example, if $A$ and $B$ are independent, then $A^c$ and $B$ are independent because

$$P(A^c\cap B)=P(B)-P(A\cap B)=P(B)(1-P(A))=P(A^c)P(B).$$

Sequential conditioning

For events $A_1,\ldots,A_n$ with positive conditioning probabilities, repeated use of the multiplication rule gives the chain rule

$$P\left(\bigcap_{i=1}^{n}A_i\right) =P(A_1)P(A_2\mid A_1)P(A_3\mid A_1\cap A_2)\cdots P\left(A_n\mid\bigcap_{i=1}^{n-1}A_i\right).$$

This identity is useful for sequential experiments and for probabilistic models that factor a joint distribution into local conditional distributions.

Remark

Pairwise independence does not necessarily imply mutual independence. For three or more events, mutual independence requires the product identity for every nonempty subcollection, not only for each pair.