20 · Point estimation, confidence intervals, and MLE
Maximum Likelihood Estimation gives you the best-guess parameter; a confidence interval tells you how uncertain that guess is — both rooted in the CLT.
One idea, one try. At your own pace.
LEARN BY DOINGPut your intuition to work3 steps
20 · Point estimation, confidence intervals, and MLE
Use the lesson’s examples and explanation to check your idea.
Personal reflection. Nothing is sent to an AI service.A point estimate is your best guess; a confidence interval quantifies the uncertainty; MLE is the principle behind both.
Without this:
Without estimation, every model output would be a number with no uncertainty attached — useless for decisions.
Point estimation is the process of using sample data to produce a single value — a point estimate — that serves as a best guess of an unknown population parameter.
Key vocabulary:
- Parameter (θ) — a fixed but unknown property of the population (e.g. the true mean μ, or the true proportion p).
- Estimator (θ̂) — a function of the sample data that we use to estimate θ. The sample mean X̄ is an estimator of μ.
- Estimate — the specific numerical value of θ̂ for a given sample.
Properties of good estimators:
- Unbiasedness: E[θ̂] = θ. The estimator is correct on average. Sample mean is unbiased for μ; sample variance with n−1 (not n) is unbiased for σ².
- Consistency: As n → ∞, θ̂ → θ in probability. More data brings you closer to the truth.
- Efficiency: Among all unbiased estimators, the one with the smallest variance is most efficient.
Maximum Likelihood Estimation (MLE): The MLE finds the parameter value θ̂ that makes the observed data most probable. If the data x₁, …, xₙ are iid from f(x | θ), the log-likelihood is:
ℓ(θ) = Σᵢ log f(xᵢ | θ)
and θ̂_MLE = arg max_θ ℓ(θ).
For the Normal distribution, setting ∂ℓ/∂μ = 0 gives θ̂_MLE = x̄ — the sample mean.
Confidence intervals: A 95% confidence interval for the mean (when σ is known) is: x̄ ± 1.96 · σ/√n
Interpretation: if you repeated the experiment many times and computed this interval each time, 95% of those intervals would contain the true μ. It is NOT "95% probability that μ is in this interval" — μ is fixed, not random.
When σ is unknown, replace σ with the sample standard deviation s and replace z with the t-quantile (t-distribution, covered next lesson).
Bootstrap CI (model-free): When no closed-form formula exists, resample from the data with replacement many times (10,000+), compute the statistic each time, and take the 2.5th and 97.5th percentiles as the 95% CI.
Python (in browser)
MLE for the Normal mean is simply the sample mean — obtained by solving ∂ℓ/∂μ = 0 analytically. scipy.stats.norm.fit returns the same MLE values. Note the small negative bias in the MLE σ̂ (ddof=0); using ddof=1 corrects it.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
The 95% CI is x̄ ± 1.96·σ/√n. Running 1000 independent experiments and checking whether each CI captures the true μ = 50 confirms the coverage is indeed ≈ 95%. This is the frequentist meaning of a confidence interval.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Bootstrap CI: resample the data with replacement 10,000 times, compute the mean each time, take the 2.5th and 97.5th percentiles. No distributional assumptions needed — it works for any statistic (mean, median, AUC, F1, …).
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Maximising the Normal log-likelihood with respect to μ gives μ̂ = x̄ — the sample mean. For σ², the MLE is the biased (1/n) formula; the unbiased version uses n−1. This derivation is the algebraic link between least squares and the Gaussian assumption.
What does 'unbiased estimator' mean?
The MLE derivation and its connection to MAP estimation (MLE + prior = MAP) are worked out in detail in MML lesson 43. Understanding the log-likelihood calculus there will let you derive the MLE for any distribution — not just the Normal.
- **MLE:** find θ that maximises ℓ(θ) = Σ log f(xᵢ | θ). For Normal data, MLE gives μ̂ = x̄.
- **95% CI:** x̄ ± 1.96·σ/√n. Coverage interpretation: 95% of such intervals (across repeated experiments) contain the true μ.
- **Bootstrap CI:** resample with replacement → compute statistic → take 2.5th & 97.5th percentiles. Works for any statistic.
Every parameter in every supervised model is an MLE (under Gaussian residuals = least squares; under Bernoulli labels = logistic regression). Bootstrap CIs are how you put error bars on F1, AUC, RMSE.
If you remove it: You can't justify why minimizing MSE is the 'right' loss for linear regression — it's the MLE under Gaussian residuals.