samples / trial
Estimation · Information · Limits
Cramér–Rao Bound · Interactive slides
AN INTERACTIVE FIELD GUIDEPrecision has
Precision has
a floor.
The Cramér–Rao Bound connects
what measurements reveal
to what estimators can achieve.
From repeated noisy measurements to Fisher information, bias, and the geometry of localization.
01 · The experiment
An estimate is one outcome. Precision is a distribution.
Keep the unknown parameter fixed. Repeat the entire measurement process.
01
A fixed world
The unknown value θ does not change between trials.
02
New measurements
Each trial produces a fresh random dataset X.
03
A new estimate
Apply the same rule T(X). Its spread is the variance.
The estimator
Unbiased means correct on average, across repeated datasets.
The question
How narrow can that cloud be?
Not “How close was this single estimate?”
Not “How small is the observation noise?”
01 · The intuition
Nearby worlds are hard to tell apart.
Information measures how sensitively the data distribution changes when the parameter moves.
Same separation · different noise
Could these observations tell you which world is true?
These are data densities, not posterior probabilities.
From ambiguity to precision
Small changes must leave a detectable trace.
If θ and θ + δ generate almost indistinguishable data, a locally unbiased estimator cannot be arbitrarily concentrated.
Fisher information turns local sensitivity into a number. Larger information permits a lower variance bound.
02 · The mechanism
Likelihood → score → Fisher information.
The likelihood is a function of the candidate parameter, with the observed dataset held fixed.
1 · Likelihood
How compatible is a candidate?
A likelihood is not automatically a probability distribution over θ.
2 · Score
Which way does the evidence push?
Under regularity, Eθ[Uθ] = 0.
3 · Fisher information
Expected squared score = expected curvature
The curvature identity requires additional differentiability and interchange conditions.
02 · The theorem
The scalar Cramér–Rao Bound.
A floor for the variance of locally unbiased estimators in a regular model.
The bound
Variance has squared parameter units.
Standard deviation has parameter units.
Read the fine print
The theorem has a scope.
Unbiasedness: Eθ[T] = θ in a neighborhood.
Regularity: differentiate under the expectation; the score has mean zero.
Finite information: 0 < Iₙ(θ) < ∞; T has finite variance.
Known model: the likelihood describes the actual experiment.
02 · Why it is true
One score identity. One Cauchy–Schwarz inequality.
An unbiased estimator must move, on average, at the same rate as the true parameter.
The score identity
1
Differentiate unbiasedness:
2
Move the derivative inside:
3
Use Eθ[Uθ] = 0:
Cauchy–Schwarz
Correlation cannot exceed one.
The entire lower bound follows from how strongly an unbiased estimator must respond to the score.
03 · A bound you can attain
For a Gaussian mean, averaging is already optimal.
Assume independent measurements Xi ∼ N(μ, σ²), with σ known.
Compute the information
Compare with an estimator
The sample mean meets the bound exactly.
No asymptotic approximation is needed.
For this model, equality holds for every n ≥ 1.
03 · Lab 1 · Predict first
What will happen when you collect more measurements?
In the next slide, every histogram bar comes from genuinely simulated datasets.
Try these three moves
Predict → change → explain
16 → 64 samples: what happens to variance and standard deviation?
Double σ: which quantity gets four times larger?
Use only X₁: does collecting unused data improve that estimator?
Reference experiment
Independent Gaussian measurements
16
2
noise σ
1
true μ
The truth is fixed across trials. Randomness comes only from new observations.
03 · Lab 1 · Live experiment
Can you reach the precision floor?
Change the experiment, repeat it, and compare empirical spread with the exact bound.
Sampling distribution · one estimate per trial
Simulated estimatesEfficient mean referenceChosen estimator, exact
—across repeated trials
—theoretical, not fitted
—σ² / n, using all data
A simulated variance slightly below the CRB is Monte Carlo fluctuation, not a violation.
Independent N(μ, σ²) data
Truth μ = 1 stays fixed.
Parameter changes replay 2,000 trials from the chosen seed.
03 · Scaling
Four times the data. Half the standard deviation.
Information adds for independent measurements; variance and standard deviation scale differently.
Gaussian scaling · σ = 2
The useful arithmetic
| n | Variance bound | SD bound |
|---|---|---|
| 4 | 1.0000 | 1.00 |
| 16 | 0.2500 | 0.50 |
| 64 | 0.0625 | 0.25 |
The iid shortcut fails for correlated or non-identical data. Use the information of the joint likelihood.
04 · The important exception
A narrow cloud can be centered in the wrong place.
The unbiased CRB is not a universal lower bound on mean squared error.
Bias–variance decomposition
The correct biased extension
The derivative of the bias matters.
This follows from the same proof, replacing 1 by ∂θEθ[T].
04 · Lab 2 · Exact distributions
Can you “beat” the bound by shrinking toward zero?
Watch the variance fall, then move the true mean away from the shrinkage target.
Exact sampling distributions · no Monte Carlo
Unbiased sample meanShrinkage estimatorTrue μ
—α²σ² / n
—(1 − α)²μ²
—variance + squared bias
Tα = α X̄ · target = 0
n = 16 and σ = 2, so the unbiased variance bound is 0.25.
Fixed α; no prior is assumed. A lower MSE here is a pointwise bias–variance tradeoff, not a violation of the unbiased CRB.
04 · Attainability
“Lower bound” does not mean “always achievable.”
Equality requires a very specific relationship between the estimator and the score.
Equality in Cauchy–Schwarz
All estimation error aligns with the score.
For uniform attainment, this must define the same statistic T(X) throughout the parameter range.
Three distinct claims
Do not collapse them into one.
Unbiased: expectation equals the target.
Efficient: variance attains the CRB.
MLE: maximizes the likelihood; it may be biased at finite n.
Regular MLEs often approach the bound asymptotically. Finite-sample equality needs its own check.
05 · Beyond one parameter
Information has directions.
A vector parameter has a covariance matrix—not a single uncertainty number.
Matrix CRB
The vector score is u = ∇θ log p(X; θ).
The inequality holds in every direction q.
Assume J is nonsingular and the vector estimator is locally unbiased.
Eigenvectors tell the story
Large information eigenvalue → small variance bound along that eigenvector.
05 · Localization
One range measurement constrains one local direction.
A simplified 2D ranging model: known anchor positions, independent Gaussian range noise.
The measurement model
Each rank-one term adds information along a line of sight. Here uᵢ is a unit direction, not the score vector.
Local sensitivity
A small radial move changes the range to first order. A tangential move does not.
The target must not coincide with an anchor. Noise variance is assumed known and independent of position.
05 · Lab 3 · Predict first
Same sensors. Different geometry. Different limits.
Four accurate ranges can still leave one position direction poorly constrained.
Try these configurations
What should the uncertainty ellipse do?
Surround: place anchors in different directions.
Cluster: move them to almost the same line of sight.
Collinear: place anchors and target on one straight line.
Then turn on an unknown common range offset. How much position information is lost?
The scalar summary
Position error bound
For an unbiased position estimator:
root mean squared Euclidean error ≥ PEB.
A finite ellipse describes local information. It does not rule out distant mirror solutions.
05 · Lab 3 · Live geometry
Move the sensors. Reshape the bound.
Drag any anchor or the target. The Fisher matrix and its inverse update immediately.
Known anchors · local information at the true target
Anchors A1–A4True target PCRB ellipse
Unit-Mahalanobis CRB ellipse; not a confidence region.
06 · What else is unknown?
An unknown clock offset consumes position information.
The same measurements must now explain both the target position and a common range offset β.
Partition the information
The position block of J⁻¹ is Jₑ⁻¹, not A⁻¹. Assume the required blocks are invertible.
Interpret the subtraction
Some parameter effects can imitate each other.
If moving the target changes all ranges similarly, an unknown offset can explain part of that same change.
The Schur complement removes information that cannot distinguish position from the nuisance parameter.
No loss when the coupling b is zero. A large loss is possible for clustered lines of sight.
06 · When the usual formula fails
A moving support breaks the familiar proof.
Not every model allows differentiation to pass through the expectation.
A useful warning example
Xi ∼ Uniform(0, θ)
The boundary moves with θ. The omitted boundary contribution is exactly why the score argument fails.
Do not force the shortcut
Check the estimator directly.
This variance is obtained from the sample maximum’s distribution—not from the invalid regular CRB calculation.
Other warning signs: boundaries, nondifferentiability, singular information, and misspecified noise.
07 · Use the bound
Design the experiment, not only the estimator.
A new independent, correctly modeled measurement adds a positive-semidefinite information term.
Linear Gaussian measurements
H controls sensitivity and geometry.
R controls noise and correlation.
Both matter. Counting measurements alone is not enough.
Choose a design objective
Different questions favor different designs.
| Design goal | One common criterion |
|---|---|
| Small total variance | Minimize tr(J⁻¹) |
| Small ellipsoid volume | Maximize log det J |
| Protect the weakest direction | Maximize λmin(J) |
Use a well-defined parameterization and units. These are local design criteria at the assumed true state.
07 · Interpretation
Read a CRB result like a careful reviewer.
Before comparing an algorithm with a bound, make sure they solve the same statistical problem.
01 · Estimator class
Is the comparison valid?
Measure or analyze bias. A learned, constrained, or regularized estimator is not automatically unbiased.
02 · Parameter vector
What was treated as known?
Include unknown offsets, calibration, orientation, or other nuisance parameters before inverting.
03 · Metric and model
Are the units and assumptions aligned?
Match variance with variance, MSE with MSE, and RMSE with a square-root bound. Model correlations correctly.
04 · Local versus global
What can the bound miss?
Inspect rank, conditioning, competing solutions, and low-information regimes. A small local CRB is not a guarantee of successful localization.
08 · Takeaways
Three ideas to keep.
More information lowers the bound. Bias changes the comparison. Geometry decides the weak directions.
01 · Sensitivity
Information belongs to the experiment.
The same estimator can face very different limits under different noise or measurement designs.
02 · Precision
Unbiased variance has a floor.
A better algorithm cannot beat this floor while remaining in the theorem’s estimator class.
03 · Direction
The matrix tells you where.
Independent, complementary measurements are often more valuable than repeated views of the same direction.
08 · Sources & extensions
Continue from the bound to the bigger picture.
The equations and examples are teaching material; the sources below give the wider statistical context.
Primary references
S1 Stanford STATS 200 · Lecture 15
Fisher information, Cramér–Rao proof, efficiency, and nuisance-parameter intuition. Autumn 2016.
S2 Ly, Marsman, Verhagen, Grasman & Wagenmakers (2017)
A Tutorial on Fisher Information. arXiv:1705.01064.
S3 Shen & Win (2010)
Fundamental Limits of Wideband Localization—Part I: A General Framework. IEEE TIT 56(10), 4956–4980.
S4 Stanford STATS 200 · Lecture 14
Consistency and asymptotic normality of the MLE.
Extensions on bailiping.com
Connect the ideas to inference and sensing.
Radar SLAM →
Sensing, motion, registration, and pose-graph correction.
Frame registration →
Geometry, alignment objectives, and observability.
Variational inference →
Approximation, optimization, and uncertainty.
All demos run locally. No tracking, network requests, external fonts, or external JavaScript are required.