Every morning a bakery's loaves come out at slightly different weights, and suppose you weigh just a handful of them. From those few numbers, what is your single best guess for the typical weight the oven is really aiming for? That is the whole game of estimation: using a small sample of data to pin down a hidden number — a parameter — that you can never observe directly.
The oldest trick is to match averages. Your data has an average you can compute, the sample mean $\bar{x}=\frac{1}{n}\sum_i x_i$, and the model has a "true" average. The method of moments (MOM) just sets them equal and solves for the unknown. Weigh five loaves at $498, 502, 495, 505, 500$ grams: the sample mean is $500$ g, so the MOM estimate of the oven's target is simply $\hat{\mu}=500$ g. One average, one equation, done.
Maximum likelihood asks a sharper question: of all possible parameter values $\theta$, which one makes the data you actually saw the most probable? Each point has a density $f(x_i;\theta)$, and if the points are independent the chance of the whole sample is the product $\prod_i f(x_i;\theta)$. Maximizing that product — in practice by maximizing its logarithm, the log-likelihood $\ell(\theta)=\sum_i\log f(x_i;\theta)$, which turns an awkward product into a friendly sum — gives the estimate
$$\hat{\theta}_{MLE}=\arg\max_\theta\,\ell(\theta)=\arg\max_\theta\sum_i\log f(x_i;\theta).$$
For the normal distribution both recipes land on the same answer, $\bar{x}$; for skewed models like the gamma they part ways and MLE usually wins. On the panel above the sliders set the model's parameters, the dashed curve is the truth your samples are drawn from, and the solid curve is the fitted estimate.
Generate just a few samples and watch the fitted curve wobble far from the dashed true curve. Then add $+1000$ more and see it snap onto the truth — that is consistency, the promise that the estimate homes in on the right value as data piles up. Finally, watch how tightly the fitted peak tracks the true peak: MLE squeezes the most information out of every observation, which is why it is the gold-standard estimator for large samples.
Point Estimation — MLE & MOM — Maximum Likelihood and Method of Moments
$$\hat\theta_{MLE} = \arg\max_\theta \ell(\theta;\mathbf{x}) = \arg\max_\theta \sum_i\log f(x_i;\theta)$$
$\text{MOM: set } E[X^k]=\frac{1}{n}\sum x_i^k, \text{ solve for } \theta$
This distribution has important theoretical and applied properties. See the simulation tabs to explore how shape changes with parameters.
Parameter estimation for any statistical model. MLE: logistic regression, linear regression (= OLS under normality), mixture models, survival models. MOM: quick estimates, method of moments for generalized method of moments (GMM) in econometrics. MLE has optimal large-sample properties (efficiency); MOM is computationally simpler but less efficient.
Use the interactive simulation above to explore how the PDF shape responds to parameter changes. The Live Sampling tab demonstrates convergence of empirical frequencies to theoretical probabilities.
Always verify parameter definitions before computing probabilities.
Different textbooks and software packages use different parameterizations. Always check which convention is being used (e.g., rate vs scale for exponential/gamma; shape vs rate for beta). Plug in extreme values and check that the PDF shape matches your expectation.
See the simulation to visualize how parameters affect the distribution shape.
Five observations from Uniform(0, $\theta$): 1.2, 3.1, 2.4, 0.8 and 2.9. The method of moments sets $E[X]=\theta/2$ equal to the sample mean 2.08, so $\hat\theta_{MoM}=2\bar x=4.16$. Maximum likelihood needs $\theta\ge$ every observation and the likelihood $\theta^{-n}$ falls as $\theta$ grows, so $\hat\theta_{MLE}=\max x_i=3.1$. The moments estimate is unbiased but can even fall below the largest data point; the MLE can never do that.
Which is better? For $\theta=1$ and $n=5$ the MoM estimator has variance $\theta^2/(3n)=0.067$. The MLE is biased, since $E[\max]=\frac{n}{n+1}\theta=0.833$, but its variance is only $0.020$. A simulation of 200,000 samples confirms 0.833, 0.067 and 0.020.
For exponential data with mean 1.6 (the values 2.0, 0.5, 1.5, 3.0, 1.0), matching $E[X]=1/\lambda$ to $\bar x$ gives $\hat\lambda=1/1.6=0.625$. Maximum likelihood gives the same answer. The same happens for the Poisson mean and the Normal mean and variance: MoM and MLE coincide, so the comparison only becomes interesting for distributions such as the Gamma, Beta and Uniform.
Open the Live Sampling tab, press Play, then add 100 and 1000 samples. Watch how the estimated parameters move toward the true values as the sample grows. The Parameter Sweep tab shows how the fit changes across the range of parameter values.
1. Uniform(0, $\theta$) data: 0.5, 1.9, 1.1. Find the MoM and MLE estimates. 2. Poisson data: 3, 5, 2, 4. What do MoM and MLE give? 3. What is the main reason to prefer MoM over MLE in practice?
Answers. 1. The mean is 1.167, so MoM is 2.333; the MLE is the maximum, 1.9. 2. Both give the sample mean 3.5. 3. MoM often needs only algebra and no optimization, so it is useful as a quick estimate or as a starting value for a numerical MLE search.
MoM is the quick first estimate, for example for Gamma and Beta parameters, and is often used to initialize numerical MLE routines. MLE is preferred when you can afford the optimization because of its efficiency. The generalized method of moments extends MoM and is standard in econometrics.