From "where are my keys?" to the Cramer-Rao bound
You lose your keys somewhere in the house. A friend asks where you think they are, and you point to the one spot that best fits everything you remember — the most believable place. Maximum likelihood estimation (MLE) does exactly this with numbers: given the data you actually collected, it picks the parameter value that makes that data the least surprising.
Flip a bent coin 10 times and get 7 heads. What is its true heads-probability? Try a value and ask: "how probable is 7-out-of-10 heads if the coin were like that?" For the coin this chance is proportional to $p^7(1-p)^3$. A fair coin ($p=0.5$) makes that smaller than $p=0.7$ does, and maximizing lands exactly on $\hat p = 7/10 = 0.7$ — the observed fraction. In general the likelihood $L(\theta)=\prod_{i=1}^n f(x_i;\theta)$ is the chance of seeing your whole dataset if the parameter were $\theta$, and the MLE is the $\theta$ that maximizes it.
Multiplying many small probabilities underflows to zero on a computer, so we maximize the log-likelihood $\ell(\theta)=\sum_{i=1}^n\log f(x_i;\theta)$ instead — same maximizer, because $\log$ is increasing. Setting its derivative (the "score") to zero, $\ell'(\theta)=0$, gives the clean Normal answers $\hat\mu=\bar x$ and $\hat\sigma^2=\frac1n\sum(x_i-\bar x)^2$. How sharply $\ell$ peaks is the Fisher information $I(\theta)=E[-\ell''(\theta)]$: a sharp peak means the data pin $\theta$ down tightly. As $n$ grows, $\sqrt{n}(\hat\theta-\theta_0)\overset{d}{\to}N(0,I(\theta_0)^{-1})$, and the variance settles onto the Cramer-Rao floor $1/(nI)$ — no fair estimator beats it. The sliders set the true parameters and $n$; each "New Data" press refits.
On the Log-Likelihood Surface tab, slide $n$ upward and watch the peak grow narrower — that is Fisher information increasing. On the Cramer-Rao Bound tab, raise $n$ and see the yellow empirical-variance dots drop onto the red $1/(nI)$ line. Finally, set sigma small versus large and notice how a tighter true spread sharpens the likelihood peak and shrinks the sampling distribution of the estimate.
Maximum Likelihood Estimation — R.A. Fisher, 1912
$$\boxed{\hat{\theta}_{MLE}=\arg\max_\theta\ell(\theta)=\arg\max_\theta\sum_{i=1}^n\log f(x_i;\theta)}$$
Asymptotic normality: $\sqrt{n}(\hat\theta-\theta_0)\overset{d}{\to}N(0,I(\theta_0)^{-1})$
| Distribution | MLE | Fisher Info I(theta) |
|---|---|---|
| Normal(mu, sigma²) | $\hat\mu=\bar x$, $\hat\sigma^2=(1/n)\sum(x_i-\bar x)^2$ | $1/\sigma^2$ (for mu) |
| Exponential(lambda) | $\hat\lambda=1/\bar x$ | $1/\lambda^2$ |
| Poisson(lambda) | $\hat\lambda=\bar x$ | $1/\lambda$ |
$I(\theta)=E\!\left[-\frac{\partial^2}{\partial\theta^2}\log f(X;\theta)\right]$. For any unbiased $\hat\theta$: $\text{Var}(\hat\theta)\ge1/(nI(\theta))$. MLE achieves this bound asymptotically — it is efficient.
$\hat\sigma^2_{MLE}=(1/n)\sum(x_i-\bar x)^2$ is biased downward. The unbiased estimator divides by $n-1$. MLE trades small bias for minimum variance — often acceptable for large $n$.
"MLE is always unbiased."
MLE is asymptotically unbiased but can be biased in finite samples. Classic: sigma²_MLE = (1/n)sum(xi-xbar)² divides by n, not n-1, giving a downward-biased estimate of population variance. The unbiased S² divides by n-1.
📖 Casella & Berger — Ch. 10
Maximizing likelihood instead of log-likelihood numerically
✅ Always maximize log-likelihood. Product of 100 small probabilities numerically underflows to exactly 0. Log converts products to sums, avoiding underflow. argmax L(theta) = argmax log L(theta) since log is monotone increasing.
Treating likelihood as a probability distribution over theta
✅ L(theta|x) = f(x|theta) is a function of theta, not a probability distribution over theta. It does NOT integrate to 1 over theta. The posterior P(theta|x) = L(theta|x)*P(theta)/P(x) requires a prior P(theta) — that's Bayesian inference.
Four measurements: 4, 6, 8, 10. The MLE of the mean is $\hat\mu=\bar x=7$. The sum of squared deviations is $\sum(x_i-\bar x)^2=20$, so the MLE of the variance is $\hat\sigma^2=20/4=5$. The unbiased estimator divides by $n-1$ instead and gives $20/3\approx6.67$. The MLE is biased: $E[\hat\sigma^2]=\frac{n-1}{n}\sigma^2$. With $n=5$ the factor is 0.8, and a simulation of 200,000 samples confirms an average of 0.800. The bias disappears as $n$ grows, so the MLE is still consistent.
For exponential data the MLE is $\hat\lambda=1/\bar x$. For the five values 2.0, 0.5, 1.5, 3.0 and 1.0 the mean is 1.6 and $\hat\lambda=0.625$. The Fisher information per observation is $1/\lambda^2$, so the asymptotic standard error is $\lambda/\sqrt n$. At $\lambda=2$ it is $2/\sqrt{50}\approx0.283$ for $n=50$ and $2/\sqrt{200}\approx0.141$ for $n=200$. Four times the data halves the error. For Poisson data with $\lambda=4$ and $n=100$ the Cramer-Rao bound is $\lambda/n=0.04$, a standard deviation of 0.2, and the sample mean reaches it exactly.
On the MLE Fit tab choose Normal and press New Data a few times: the fitted curve moves because every sample is different. On the Log-Likelihood Surface tab check that the peak sits at the estimate. On the Sampling Distribution tab increase $n$ from 10 to 300 and watch the cloud of estimates narrow around the true value. On the Cramer-Rao Bound tab compare the observed variance of the MLE with the bound.
1. For Poisson data 3, 5, 2, 4, find the MLE of $\lambda$. 2. By what factor does the standard error change if $n$ goes from 25 to 100? 3. Why do we maximize the log-likelihood instead of the likelihood?
Answers. 1. $\hat\lambda=\bar x=14/4=3.5$. 2. It halves, because the error scales as $1/\sqrt n$. 3. The logarithm turns a product of many densities into a sum, which is easier to differentiate and numerically stable, and it has its maximum at the same parameter value.
Maximum likelihood is the default fitting method in logistic regression, Poisson regression, survival models and most of machine learning (cross-entropy loss is a negative log-likelihood). Its appeal is that, under regularity conditions, it is consistent, asymptotically normal and asymptotically efficient, meaning it reaches the Cramer-Rao bound as $n$ grows.