Weak Law of Large Numbers

Overview

The Weak Law of Large Numbers (WLLN) is a fundamental theorem in probability theory that describes the behavior of the average of a large number of independent and identically distributed (i.i.d.) random variables.

TheoremWeak Law of Large Numbers

Let X1,X2,...,XnX_1, X_2, ..., X_n be a sequence of i.i.d. random variables with finite expected value E[Xi]=μ\mathbb{E}[X_i] = \mu and finite variance V(Xi)=σ2\mathbb{V}(X_i) = \sigma^2.

Define the sample mean as: Xˉn=1n∑i=1nXi\bar{X}_n = \frac{1}{n}\sum_{i=1}^{n} X_i

The Weak Law of Large Numbers states that for any ϵ>0\epsilon > 0: lim⁡n→∞P(∣Xˉn−μ∣≥ϵ)=0\lim_{n \to \infty} P(|\bar{X}_n - \mu| \geq \epsilon) = 0

Or equivalently: Xˉn→Pμ as n→∞\bar{X}_n \xrightarrow{P} \mu \text{ as } n \to \infty

This is called convergence in probability.

ProofUsing Chebyshev's Inequality

Step 1: Compute the expected value of Xˉn\bar{X}_n

E[Xˉn]=E[1n∑i=1nXi]=1n∑i=1nE[Xi]=1n⋅nμ=μ\mathbb{E}[\bar{X}_n] = \mathbb{E}\left[\frac{1}{n}\sum_{i=1}^{n} X_i\right] = \frac{1}{n}\sum_{i=1}^{n} \mathbb{E}[X_i] = \frac{1}{n} \cdot n\mu = \mu

Step 2: Compute the variance of Xˉn\bar{X}_n

Since the XiX_i are independent: V(Xˉn)=V(1n∑i=1nXi)=1n2∑i=1nV(Xi)=1n2⋅nσ2=σ2n\mathbb{V}(\bar{X}_n) = \mathbb{V}\left(\frac{1}{n}\sum_{i=1}^{n} X_i\right) = \frac{1}{n^2}\sum_{i=1}^{n} \mathbb{V}(X_i) = \frac{1}{n^2} \cdot n\sigma^2 = \frac{\sigma^2}{n}

Step 3: Apply Chebyshev’s Inequality

For any ϵ>0\epsilon > 0: P(∣Xˉn−μ∣≥ϵ)≤V(Xˉn)ϵ2=σ2nϵ2P(|\bar{X}_n - \mu| \geq \epsilon) \leq \frac{\mathbb{V}(\bar{X}_n)}{\epsilon^2} = \frac{\sigma^2}{n\epsilon^2}

Step 4: Take the limit

lim⁡n→∞P(∣Xˉn−μ∣≥ϵ)≤lim⁡n→∞σ2nϵ2=0\lim_{n \to \infty} P(|\bar{X}_n - \mu| \geq \epsilon) \leq \lim_{n \to \infty} \frac{\sigma^2}{n\epsilon^2} = 0

Since probabilities are non-negative, the limit must be exactly 0.

Interpretation

The WLLN tells us that as the sample size increases, the sample mean Xˉn\bar{X}_n converges in probability to the true mean μ\mu. This means that for large nn, the sample mean will be close to the population mean with high probability.

Applications

  1. Statistics: Justifies using sample averages to estimate population parameters
  2. Gambling: Explains why casinos have consistent profits
  3. Insurance: Forms the basis for risk pooling and premium calculation
  4. Quality Control: Validates using sample means to monitor processes

Relationship to Strong Law

The Strong Law of Large Numbers (SLLN) states almost sure convergence: P(lim⁡n→∞Xˉn=μ)=1P\left(\lim_{n \to \infty} \bar{X}_n = \mu\right) = 1

The SLLN implies the WLLN, but not conversely. The WLLN is sufficient for most practical applications.

ExampleCoin Flipping

For a fair coin with P(Heads)=0.5P(\text{Heads}) = 0.5, let Xi=1X_i = 1 if the ii-th flip is heads, 00 otherwise.

  • μ=E[Xi]=0.5\mu = \mathbb{E}[X_i] = 0.5
  • σ2=V(Xi)=0.25\sigma^2 = \mathbb{V}(X_i) = 0.25

The proportion of heads in nn flips is Xˉn\bar{X}_n. By WLLN: lim⁡n→∞P(∣Xˉn−0.5∣≥ϵ)=0\lim_{n \to \infty} P(|\bar{X}_n - 0.5| \geq \epsilon) = 0

This means that as we flip the coin more times, the proportion of heads will approach 0.5.

For more details on expectation and variance, see Expectation and Variance.