Foundations of Statistical Inference: Three Convergence Proofs for the Sample Variance

Statistical inference is a cornerstone of modern data science and machine learning theory. In this tutorial, we explore a classic problem: given a sequence of independent and identically distributed (iid) random variables, how do we prove that the sample variance estimator $S_n$ converges to the population variance $\sigma^2$ in probability and almost surely, and further characterize its asymptotic normality? These three modes of convergence — and their relationships — are high-frequency topics in statistical inference exams and research.

Problem Setup and Preparation

Let $X_1, X_2, X_3, \dots$ be iid random variables with finite variance $\tau$. Define the sample variance estimator:

$$S_n = \frac{1}{n}\sum_{i=1}^{n}(X_i - \bar{X})^2$$

We need to prove three propositions:

• (a) $S_n$ converges in probability to $\sigma^2$;
• (b) $S_n$ converges almost surely to $\sigma^2$;
• (c) $\sqrt{n}(S_n - \sigma^2)$ converges in distribution to a normal distribution.

Before diving into the proofs, a key algebraic manipulation is needed. The trick is to subtract and add the population mean $\mu$ inside the parentheses, writing $(X_i - \bar{X}) = (X_i - \mu) + (\mu - \bar{X})$. This leaves the expression unchanged but reveals useful structure.

Expanding yields:

$$S_n = \frac{1}{n}\sum_{i=1}^{n}(X_i - \mu)^2 - (\bar{X} - \mu)^2$$

The cross terms cancel neatly and fold into the $(\bar{X}-\mu)^2$ term. We assume $E[X_i] = \mu$, $\mathrm{Var}(X_i) = \sigma^2$, both finite — standard assumptions even if not explicitly stated in the problem.

Introducing an Auxiliary Variable to Simplify the Structure

To apply the Law of Large Numbers and the Central Limit Theorem, define $Y_i = (X_i - \mu)^2$. This substitution is the pivotal move in the entire proof: it transforms a quadratic expression into a new sequence of iid random variables.

First, compute the expectation:

$$E[Y_i] = E[(X_i - \mu)^2] = \mathrm{Var}(X_i) = \sigma^2$$

This is precisely the target quantity we wish to estimate. The variance of $Y_i$ equals the given $\tau$, which is finite. Since $X_1, X_2, \dots$ are iid, the $Y_i$, as functions of them, are also iid. At this point, $Y_i$ satisfies all conditions required to apply limit theorems: iid, finite mean ($\sigma^2$), and finite variance ($\tau$).

Propositions (a) and (b): Starting from Almost Sure Convergence

There is a clear hierarchy among the three modes of convergence: almost sure convergence is the strongest, implying convergence in probability, which in turn implies convergence in distribution. Therefore, proposition (b) is stronger than (a) — proving (b) immediately gives (a) for free. This is an efficient proof strategy: tackle the strongest conclusion first.

The proof of (b) splits into two parts. Part one: Since the $Y_i$ are iid with mean $\sigma^2$, the Strong Law of Large Numbers (SLLN) gives $\bar{Y}_n \xrightarrow{a.s.} \sigma^2$.

Part two: The $X_i$ are iid with mean $\mu$, so by the SLLN, $\bar{X} \xrightarrow{a.s.} \mu$. By the Continuous Mapping Theorem (CMT), it follows that $(\bar{X}-\mu)^2 \xrightarrow{a.s.} 0$.

Finally, apply Slutsky's theorem to combine both parts:

$$S_n = \bar{Y}_n - (\bar{X}-\mu)^2 \xrightarrow{a.s.} \sigma^2 - 0 = \sigma^2$$

Proposition (b) is established. Since almost sure convergence implies convergence in probability, proposition (a) follows immediately.

---

The Strong Law of Large Numbers (SLLN) states: if $Z_1, Z_2, \dots$ are iid with $E[|Z_1|] < \infty$, then the sample mean $\bar{Z}n = \frac{1}{n}\sum{i=1}^n Z_i$ converges to $E[Z_1]$ with probability 1, i.e., $P(\lim_{n\to\infty}\bar{Z}_n = E[Z_1]) = 1$. This is stronger than the Weak Law of Large Numbers (WLLN), which only guarantees $P(|\bar{Z}_n - E[Z_1]| > \varepsilon) \to 0$ for all $\varepsilon > 0$ — convergence in probability — whereas the SLLN ensures convergence along almost every sample path. The Continuous Mapping Theorem (CMT) provides a complementary tool: if $W_n \xrightarrow{a.s.} c$ and $g$ is continuous at $c$, then $g(W_n) \xrightarrow{a.s.} g(c)$. Here we take $g(x) = (x - \mu)^2$, which is obviously continuous, so $\bar{X} \xrightarrow{a.s.} \mu$ directly implies $(\bar{X}-\mu)^2 \xrightarrow{a.s.} 0$ without further argument.

---

Proposition (c): Proving Asymptotic Normality

Proposition (c) requires showing that $\sqrt{n}(S_n - \sigma^2)$ converges in distribution to a normal distribution. We again split the expression into two parts.

Substituting $S_n$ and rearranging:

$$\sqrt{n}(S_n - \sigma^2) = \sqrt{n}(\bar{Y}_n - \sigma^2) - \sqrt{n}(\bar{X}-\mu)^2$$

Part one: The $Y_i$ are iid with mean $\sigma^2$ and finite variance $\tau$. Applying the Central Limit Theorem (CLT) directly:

$$\sqrt{n}(\bar{Y}_n - \sigma^2) \xrightarrow{d} N(0, \tau)$$

Part two: Cleverly rewrite $\sqrt{n}(\bar{X}-\mu)^2$ as $\sqrt{n}(\bar{X}-\mu) \cdot (\bar{X}-\mu)$.

Here, $\sqrt{n}(\bar{X}-\mu)$ converges in distribution to $N(0, \sigma^2)$ by the CLT, while the other factor $(\bar{X}-\mu)$ converges in probability to $0$ by the Law of Large Numbers. By Slutsky's theorem, their product converges in probability to $0$.

Combining both parts and applying Slutsky's theorem once more:

$$\sqrt{n}(S_n - \sigma^2) \xrightarrow{d} N(0, \tau)$$

Proposition (c) is proved.

---

Slutsky's theorem is the essential tool for combining multiple limit results. Its key content: if $A_n \xrightarrow{d} A$ and $B_n \xrightarrow{p} c$ (where $c$ is a constant), then $A_n + B_n \xrightarrow{d} A + c$ and $A_n \cdot B_n \xrightarrow{d} c \cdot A$. Crucially, the theorem requires one of the sequences to converge to a constant — when both sequences converge only in distribution, they cannot generally be added or multiplied directly. In the proof of (c), the second term $\sqrt{n}(\bar{X}-\mu)^2$ is factored as the product of two terms: $\sqrt{n}(\bar{X}-\mu)$, which converges in distribution to $N(0,\sigma^2)$, and $(\bar{X}-\mu)$, which converges in probability to the constant $0$. By Slutsky's theorem, the product converges in distribution to $0 \cdot N(0,\sigma^2) = 0$, i.e., converges in probability to $0$. Subtracting this from the first part — which converges in distribution to $N(0,\tau)$ — and applying Slutsky's theorem again yields the final result $N(0,\tau)$.

---

Methodological Summary

This problem weaves together the most important tools in statistical inference, illuminating both their conditions of use and how they work together:

• Convergence hierarchy: Almost sure convergence ⇒ convergence in probability ⇒ convergence in distribution. Understanding this chain can dramatically simplify proof strategies.
• iid and finite moment conditions: The Law of Large Numbers requires iid and finite mean; the Central Limit Theorem additionally requires finite variance. These prerequisites are non-negotiable.
• Auxiliary variable substitution: Setting $Y_i=(X_i-\mu)^2$ is the key to simplification — it transforms a nonlinear problem into a standard iid sequence problem.
• Slutsky's theorem and the Continuous Mapping Theorem: Indispensable when combining multiple convergence results.

Mastering this proof framework not only prepares you for exams, but also lays a solid foundation for deeply understanding the consistency and asymptotic distributions of estimators.