Bento/Slides
ESTIMATION · INFORMATION · LIMITS
Cramér–Rao Bound
How precise could an estimate possibly be?
Precision has
a floor.
Connect what measurements reveal to what estimators can achieve.
Fisher information · estimator bias
localization geometry
Open the interactive guide →
Repeated sample means at n = 4, 16, and 64. Their spread decreases around the same true mean.
Cramér–Rao Bound · Precision has a floor
S1 →
S2 →
01 / 23
01 · THE EXPERIMENT
An estimate is one outcome. Precision is a distribution.
Keep the unknown parameter fixed. Repeat the entire measurement process.
01
A fixed world
The unknown θ stays fixed across trials.
02
New measurements
Each trial produces a fresh random dataset X.
03
A new estimate
Apply the same rule T(X). Its spread is the variance.
T=T(X),\qquad \mathbb E_\theta[T]=\theta
How narrow can the cloud be?
Unbiased means correct on average over repeated datasets.
Precision describes repeated estimates. One close estimate does not establish precision.
Cramér–Rao Bound · Precision has a floor
S1 →
02 / 23
01 · THE INTUITION
Nearby worlds are hard to tell apart.
Information measures how sensitively the data distribution changes when the parameter moves.
Same separation · different noise
Densities under θ and θ + δ overlap more for noisy observations.
Data densities; not posterior probabilities.
A change must leave a trace.
Almost indistinguishable data cannot support arbitrarily concentrated, locally unbiased estimates.
I_n(\theta)=\mathbb E_\theta\!\left[\left(\partial_\theta\log p(X;\theta)\right)^2\right]
More Fisher information permits a lower variance bound.
Cramér–Rao Bound · Precision has a floor
S1 →
S2 →
03 / 23
02 · THE MECHANISM
Likelihood → score → Fisher information.
The likelihood is a function of the candidate parameter, with the observed dataset held fixed.
1 · Likelihood
\ell(\theta;X)=\log p(X;\theta)Two likelihood curves: the narrower curve has greater curvature.
A likelihood is not a probability density over θ.
2 · Score
U_\theta(X)=\partial_\theta\ell(\theta;X)
Under regularity, the score has mean zero.
3 · Fisher information
I_n(\theta)=\mathbb E_\theta[U_\theta^2]=-\mathbb E_\theta[\ell^{\prime\prime}(\theta;X)]
Expected squared score = expected curvature, under the required interchange conditions.
Cramér–Rao Bound · Precision has a floor
S1 →
S2 →
04 / 23
02 · THE THEOREM
The scalar Cramér–Rao Bound.
A floor for the variance of locally unbiased estimators in a regular model.
The variance floor
\operatorname{Var}_\theta(T)\;\ge\;\frac{1}{I_n(\theta)}\operatorname{SD}_\theta(T)\;\ge\;\frac{1}{\sqrt{I_n(\theta)}}
Variance: squared units · SD: parameter units
The theorem has a scope.
Unbiasedness
Eθ[T] = θ in a neighborhood.
Regularity
Differentiate under the expectation; Eθ[Uθ] = 0.
Finite information
0 < Iₙ(θ) < ∞ and finite estimator variance.
Known model
The likelihood describes the experiment.
Cramér–Rao Bound · Precision has a floor
S1 →
05 / 23
02 · WHY IT IS TRUE
One score identity. One Cauchy–Schwarz inequality.
An unbiased estimator must move, on average, at the same rate as the true parameter.
The score identity
1 Differentiate unbiasedness
\partial_\theta\mathbb E_\theta[T]=1
2 Move the derivative inside
\mathbb E_\theta[TU_\theta]=1
3 Use the zero-mean score
\operatorname{Cov}_\theta(T,U_\theta)=1
Cauchy–Schwarz
\operatorname{Cov}(T,U_\theta)^2\le\operatorname{Var}(T)\operatorname{Var}(U_\theta)1\le\operatorname{Var}(T)\,I_n(\theta)\boxed{\operatorname{Var}(T)\ge 1/I_n(\theta)}
An unbiased estimator must respond to the score.
Cramér–Rao Bound · Precision has a floor
S1 →
06 / 23
03 · A BOUND YOU CAN ATTAIN
For a Gaussian mean, averaging is already optimal.
Assume independent measurements Xi ∼ N(μ, σ²), with σ known.
Compute the information
\ell(\mu)=C-\frac{1}{2\sigma^2}\sum_{i=1}^n(X_i-\mu)^2U_\mu=\frac{1}{\sigma^2}\sum_{i=1}^n(X_i-\mu)I_n(\mu)=\frac{n}{\sigma^2}
Compare with the sample mean
\overline X=\frac1n\sum_{i=1}^nX_i,\qquad\mathbb E[\overline X]=\mu\operatorname{Var}(\overline X)=\frac{\sigma^2}{n}=\mathrm{CRB}
Exact equality for every n ≥ 1.
Cramér–Rao Bound · Precision has a floor
S1 →
07 / 23
03 · LAB 1 · PREDICT FIRST
What will happen when you collect more measurements?
In the next slide, every histogram bar comes from genuinely simulated datasets.
Predict → change → explain
16 → 64 samples
What happens to variance and SD?
Double the noise σ
Which quantity becomes four times larger?
Use only the first observation
Does collecting unused data help?
Reference experiment
16
samples / trial
2
noise σ
1
true μ
I_n=4,\quad \mathrm{CRB}=0.25,\quad \mathrm{SD}_{\min}=0.5
Truth stays fixed. Each trial uses fresh observations.
Cramér–Rao Bound · Precision has a floor
S1 →
08 / 23
03 · LAB 1 · LIVE EXPERIMENT
Can you reach the precision floor?
Change the experiment, repeat it, and compare empirical spread with the exact bound.
Seeded histogram of 2,000 sample means, with exact sampling-density reference curves.
0.2570
empirical variance
0.2500
exact variance
0.2500
unbiased CRB
Change the experiment
Independent Gaussian datasets.
n = 16 · σ = 2 · μ = 1
2,000 trials · seed 42
Increase n, change noise, or discard all but one observation. Compare the estimator with the same full-data bound.
Interactive controls in the HTML slides.
A finite simulated variance can fall below the CRB through Monte Carlo fluctuation.
Cramér–Rao Bound · Precision has a floor
S1 →
09 / 23
03 · SCALING
Four times the data. Half the standard deviation.
Information adds for independent measurements; variance and standard deviation scale differently.
Log-scale curves: variance bound 4/n and standard-deviation bound 2/sqrt(n).I_n=nI_1,\quad \mathrm{CRB}\propto n^{-1},\quad \mathrm{SD}_{\min}\propto n^{-1/2}
n
Variance bound
SD bound
4
1.0000
1.00
16
0.2500
0.50
64
0.0625
0.25
Correlated or non-identical data require the information of the joint likelihood.
Cramér–Rao Bound · Precision has a floor
S1 →
S2 →
10 / 23
04 · THE IMPORTANT EXCEPTION
A narrow cloud can be centered in the wrong place.
The unbiased CRB is not a universal lower bound on mean squared error.
Bias–variance decomposition
b(\theta)=\mathbb E_\theta[T]-\theta\operatorname{MSE}_\theta(T)=\operatorname{Var}_\theta(T)+b(\theta)^2An unbiased density centered at truth and a narrower biased density displaced from truth.
The correct biased extension
\operatorname{Var}_\theta(T)\ge\frac{[1+b^\prime(\theta)]^2}{I_n(\theta)}\operatorname{MSE}_\theta(T)\ge\frac{[1+b^\prime(\theta)]^2}{I_n(\theta)}+b(\theta)^2
The derivative of the bias matters.
Cramér–Rao Bound · Precision has a floor
S1 →
11 / 23
04 · LAB 2 · EXACT DISTRIBUTIONS
Can you “beat” the bound by shrinking toward zero?
Watch the variance fall, then move the true mean away from the shrinkage target.
Exact distributions of the unbiased sample mean and shrinkage estimator, at alpha = 0.55 and true mean = 0.35.Exact variance, squared bias, total MSE, and the unbiased CRB for the default shrinkage example.
Shrink toward zero
Tα = α X̄
α = 0.55 · μ = 0.35
Variance = 0.0756
Squared bias = 0.0248
MSE = 0.1004
Move μ away from zero.
Does the MSE advantage remain?
Interactive controls in the HTML slides.
The unbiased CRB is 0.25. A biased estimator can have a smaller pointwise MSE.
Cramér–Rao Bound · Precision has a floor
S1 →
S2 →
12 / 23
04 · ATTAINABILITY
“Lower bound” does not mean “always achievable.”
Equality requires a very specific relationship between the estimator and the score.
Equality in Cauchy–Schwarz
T(X)-\theta=\frac{U_\theta(X)}{I_n(\theta)}\quad\text{almost surely}\text{Gaussian mean:}\quad\frac{U_\mu}{I_n}=\overline X-\mu
The same statistic must work throughout the parameter range.
Keep three claims distinct.
Unbiased
Expectation equals the target.
Efficient
Variance attains the CRB.
Maximum likelihood
Maximizes likelihood; finite-sample bias is possible.
Regular MLEs may attain the bound asymptotically.
Cramér–Rao Bound · Precision has a floor
S1 →
S4 →
13 / 23
05 · BEYOND ONE PARAMETER
Information has directions.
A vector parameter has a covariance matrix—not a single uncertainty number.
The matrix CRB
\mathbf J(\boldsymbol\theta)=\mathbb E[\mathbf u\mathbf u^{\mathsf T}]\operatorname{Cov}(\widehat{\boldsymbol\theta})\succeq\mathbf J(\boldsymbol\theta)^{-1}\mathbf q^{\mathsf T}\operatorname{Cov}(\widehat{\boldsymbol\theta})\mathbf q\ge\mathbf q^{\mathsf T}\mathbf J^{-1}\mathbf q
Nonsingular J; regular model; locally unbiased estimator.
Covariance-bound ellipse with a long axis in the weak-information direction.
Larger information eigenvalue → smaller directional variance bound.
Cramér–Rao Bound · Precision has a floor
S1 →
S2 →
14 / 23
05 · LOCALIZATION
One range measurement constrains one local direction.
A simplified 2D ranging model: known anchor positions, independent Gaussian range noise.
Independent Gaussian ranges
z_i=\lVert\mathbf p-\mathbf a_i\rVert+\varepsilon_i,\quad\varepsilon_i\sim\mathcal N(0,\sigma_i^2)\mathbf u_i=\frac{\mathbf p-\mathbf a_i}{\lVert\mathbf p-\mathbf a_i\rVert}\mathbf J(\mathbf p)=\sum_i\frac{1}{\sigma_i^2}\mathbf u_i\mathbf u_i^{\mathsf T}One anchor constrains motion along the radial line of sight; tangential motion has zero first-order range sensitivity.
Each term is rank one. Anchors are known; noise variance is known and independent of position.
Cramér–Rao Bound · Precision has a floor
S3 →
15 / 23
05 · LAB 3 · PREDICT FIRST
Same sensors. Different geometry. Different limits.
Four accurate ranges can still leave one position direction poorly constrained.
Predict the weak direction
Surround
Anchors point toward the target from different directions.
Cluster
Anchors share almost the same line of sight.
Collinear
Anchors and target lie on the same straight line.
Position error bound
\mathrm{PEB}=\sqrt{\operatorname{tr}(\mathbf J^{-1})}
For an unbiased position estimator:
Euclidean RMSE ≥ PEB.
Finite local information does not rule out distant mirror solutions.
Cramér–Rao Bound · Precision has a floor
S3 →
16 / 23
05 · LAB 3 · LIVE GEOMETRY
Move the sensors. Reshape the bound.
Drag any anchor or the target. The Fisher matrix and its inverse update immediately.
Four surrounding anchors and a true target, with the unit-Mahalanobis covariance-bound ellipse.
Default: σ = 0.60 m · PEB = 0.625 m
Reshape the information
Move an anchor or the target.
Compare Surround, Cluster, and Collinear.
Then estimate an unknown common range offset. Watch the equivalent position information decrease.
Interactive controls in the HTML slides.
The ellipse is not a confidence region.
Cramér–Rao Bound · Precision has a floor
S3 →
17 / 23
06 · WHAT ELSE IS UNKNOWN?
An unknown clock offset consumes position information.
The same measurements must now explain both the target position and a common range offset β.
Partition the information
\boldsymbol\theta=\begin{bmatrix}\mathbf p\\ \beta\end{bmatrix},\qquad\mathbf J=\begin{bmatrix}\mathbf A&\mathbf b\\\mathbf b^{\mathsf T}&c\end{bmatrix}\mathbf J_{\mathrm e}=\mathbf A-\mathbf b\,c^{-1}\mathbf b^{\mathsf T}
The position block of J⁻¹ is Jₑ⁻¹.
Some effects imitate each other.
An unknown offset can explain changes in range that would otherwise constrain position.
\mathbf J_{\mathrm e}\preceq\mathbf A
No loss when b = 0. Clustered lines of sight can lose much more.
Explore the geometry lab →
Cramér–Rao Bound · Precision has a floor
S1 →
S3 →
18 / 23
06 · WHEN THE USUAL FORMULA FAILS
A moving support breaks the familiar proof.
Not every model allows differentiation to pass through the expectation.
Xi ∼ Uniform(0, θ)
p(X;\theta)=\theta^{-n}\prod_{i=1}^n\mathbf1\{0<X_i<\theta\}U_\theta=-\frac n\theta\quad\text{inside the support}\mathbb E_\theta[U_\theta]=-\frac n\theta\ne0
The moving boundary invalidates the regular score identity.
Check the estimator directly.
T=\frac{n+1}{n}X_{(n)},\qquad\mathbb E_\theta[T]=\theta\operatorname{Var}_\theta(T)=\frac{\theta^2}{n(n+2)}
The variance follows from the distribution of the sample maximum.
Cramér–Rao Bound · Precision has a floor
S1 →
S2 →
19 / 23
07 · USE THE BOUND
Design the experiment, not only the estimator.
A new independent, correctly modeled measurement adds a positive-semidefinite information term.
Linear Gaussian measurements
\mathbf z=\mathbf H\boldsymbol\theta+\boldsymbol\varepsilon,\qquad\boldsymbol\varepsilon\sim\mathcal N(0,\mathbf R)\mathbf J=\mathbf H^{\mathsf T}\mathbf R^{-1}\mathbf H
H sets sensitivity and geometry.
R sets noise and correlation.
Choose a design objective
Total variance
Minimize tr(J⁻¹)
Ellipsoid volume
Maximize log det J
Weakest direction
Maximize λmin(J)
Local criteria depend on the parameterization and units.
Cramér–Rao Bound · Precision has a floor
S2 →
S3 →
20 / 23
07 · INTERPRETATION
Read a CRB result like a careful reviewer.
Before comparing an algorithm with a bound, make sure they solve the same statistical problem.
01 · Estimator class
Is the comparison valid?
Analyze bias before comparing a learned, constrained, or regularized estimator with an unbiased bound.
02 · Parameter vector
What was treated as known?
Include unknown offsets, calibration, orientation, and other nuisance parameters before inverting.
03 · Metric and model
Are the assumptions aligned?
Match variance, MSE, and RMSE with the correct bound. Model measurement correlations.
04 · Local versus global
What can the bound miss?
Inspect rank, conditioning, competing solutions, and low-information regimes.
Cramér–Rao Bound · Precision has a floor
S1 →
S2 →
S3 →
21 / 23
08 · TAKEAWAYS
Three ideas to keep.
More information lowers the bound. Bias changes the comparison. Geometry decides the weak directions.
01 · Sensitivity
Information belongs to the experiment.
I_n=\mathbb E[U_\theta^2]
Noise and sensing design change the limits before any estimator runs.
02 · Precision
Unbiased variance has a floor.
\operatorname{Var}(T)\ge I_n^{-1}
An estimator within the theorem’s class cannot beat the floor.
03 · Direction
The matrix tells you where.
\operatorname{Cov}(\widehat{\boldsymbol\theta})\succeq\mathbf J^{-1}
Complementary measurement directions improve the weakest constraints.
Cramér–Rao Bound · Precision has a floor
S1 →
S2 →
S3 →
22 / 23
08 · SOURCES & EXTENSIONS
Continue from the bound to the bigger picture.
The equations and examples are teaching material; the sources below give the wider statistical context.
Primary references
S1 · Stanford STATS 200, Lecture 15 →
Fisher information, scalar bound, and efficiency.
S2 · Ly et al. (2017) →
A Tutorial on Fisher Information.
S3 · Shen & Win (2010) →
Fundamental Limits of Wideband Localization, Part I.
S4 · Stanford STATS 200, Lecture 14 →
Regular asymptotic normality of the MLE.
Explore and take it with you
Interactive guide & experiments →
Download PDF →
Download PowerPoint →
Download source package →
Frame registration →
Radar SLAM →
Cramér–Rao Bound · Precision has a floor
S1 →
S2 →
S3 →
S4 →
23 / 23