Flip a coin 100 times to learn how biased it is. Do you read your friend all 100 results — H, T, T, H, …? Of course not. You just say "57 heads." That single count tells them everything about the coin's bias; the order the flips came in never mattered. That number is a sufficient statistic — a summary that throws away nothing important about the thing you're trying to learn.
Call the measurements $X_1,X_2,\dots,X_n$ and the quantity you want to learn $\theta$ (the coin's true heads-probability, or a population average). A statistic $T$ is just some number computed from the data. We say $T$ is sufficient for $\theta$ when, once you know $T$, the leftover details of the data tell you nothing more about $\theta$. For coin flips — and a surprising number of other cases — that magic summary is simply the total, $T=\sum_i X_i$. With $n=100$ flips and $T=57$ heads, the best estimate is $\hat p = T/n = 0.57$. You never needed the individual flips at all.
The Factorization Theorem makes "sufficient" exact: $T$ is sufficient iff the likelihood splits as $f(\mathbf{x};\theta)=g\!\big(T(\mathbf{x}),\theta\big)\,h(\mathbf{x})$, where $\theta$ enters only through $T$. Every exponential-family law is built this way, $f(x;\theta)=h(x)\exp[\eta(\theta)T(x)-A(\theta)]$, so the sufficient statistic falls straight out of the formula. Conditioning any unbiased estimator on $T$ can only shrink its variance (Rao–Blackwell) — which is exactly why estimators built from $T$ are so efficient. In the sim, the True theta slider sets $\theta$, n sets the sample size, and the cards report $T$, the MLE $=f(T)$, and the Cramér–Rao bound $1/(nI)$.
(1) On the Rao-Blackwell tab, compare the spread of the MLE (which uses all the data through $T$) against using only the first observation — the sufficient-statistic estimator is dramatically tighter. (2) On Estimator Comparison, slide n upward and watch the MLE variance ride right along the Cramér–Rao floor. (3) Switch the Distribution to Poisson or Exponential and confirm the same $T=\sum_i X_i$ keeps working — one summary, many distributions.
Sufficient Statistics — Fisher 1922 / Neyman 1935
$T(X)$ sufficient for $\theta$ iff: $f(\mathbf{x};\theta)=g(T(\mathbf{x}),\theta)\cdot h(\mathbf{x})$ (Factorization Theorem)
$$\boxed{\text{Exponential family: }f(x;\theta)=h(x)\exp\!\left[\eta(\theta)T(x)-A(\theta)\right]}$$
| Distribution | Sufficient Statistic T | MLE from T |
|---|---|---|
| Normal(mu,1) | $\sum x_i$ (= n·xbar) | $\hat\mu=T/n=\bar x$ |
| Bernoulli(p) | $\sum x_i$ (number of 1s) | $\hat p=T/n$ |
| Poisson(lambda) | $\sum x_i$ | $\hat\lambda=T/n=\bar x$ |
| Exponential(lam) | $\sum x_i$ | $\hat\lambda=n/T=1/\bar x$ |
If $\hat\theta$ is unbiased and $T$ is sufficient, then $\tilde\theta=E[\hat\theta\mid T]$ satisfies: (1) $E[\tilde\theta]=\theta$ (still unbiased), (2) $\text{Var}(\tilde\theta)\le\text{Var}(\hat\theta)$. Conditioning on sufficient statistic never increases variance.
Sufficient statistic = no information loss.
T(X) is sufficient if knowing T(X) tells you everything the full data X tells you about theta. Once you have T=sum(Xi), the ordering and individual values of X1,...,Xn are irrelevant for estimating theta. The MLE is always a function of the sufficient statistic in the exponential family.
📖 Casella & Berger — Ch. 6.2
Five waiting times are observed: 2.0, 0.5, 1.5, 3.0 and 1.0. The joint density is $\lambda^5 e^{-\lambda\sum x_i}$, and it depends on the data only through $T=\sum x_i=8$. By the factorization theorem $T$ is sufficient. Any other sample with the same total, such as 1.6 repeated five times, gives exactly the same likelihood function and the same estimate $\hat\lambda=n/T=5/8=0.625$. The individual values add nothing once you know the sum.
Ten coin tosses give 1, 0, 0, 1, 1, 1, 0, 1, 0, 1, so $T=\sum X_i=6$. The estimator $X_1$ is unbiased for $p$ but crude: its variance is $p(1-p)$, which is 0.25 at $p=0.5$. Conditioning on the sufficient statistic gives $E[X_1\mid T]=T/n=0.6$, the sample mean. For $p=0.5$ its variance is $0.25/10=0.025$, ten times smaller, and it is still unbiased. That is the Rao-Blackwell theorem in one line: conditioning on a sufficient statistic never increases variance.
Open the Sufficiency Demo tab and keep $n=30$. Press New Sample several times and watch how the estimate based on $T$ behaves compared with one that throws data away. Then open the Rao-Blackwell tab and compare the spread of the crude estimator with the conditioned one. Finally use the Estimator Comparison tab and raise $n$ from 5 to 200: the spread of the estimators shrinks, and you can compare how quickly each one settles.
1. For Poisson data 3, 5, 2, 4, what is the sufficient statistic and the MLE of $\lambda$? 2. For $N(\theta,1)$ data, is the sample median a sufficient statistic? 3. In which of these is the sufficient statistic not a sum: Normal mean, Bernoulli, Uniform(0, $\theta$)?
Answers. 1. $T=\sum x_i=14$ and $\hat\lambda=14/4=3.5$. 2. No. The median throws away information that the sum keeps, so it is not sufficient. 3. Uniform(0, $\theta$): the sufficient statistic is the maximum $X_{(n)}$, because the density is $\theta^{-n}$ only when every observation is at most $\theta$.
Sufficiency justifies data compression in statistics: you can store a few numbers (a sum, a sum of squares) instead of the whole sample. It is the starting point for the Rao-Blackwell and Lehmann-Scheffe theorems, which build minimum-variance unbiased estimators. In Bayesian work, conjugate priors rely on exactly the same structure, and in machine learning the exponential-family form is behind generalized linear models.