BLP Visual notes

Interactive systems note · 2026

Make the renderer
part of the graph.

Gaussian Splatting gives SLAM a photorealistic map. GraphSLAM gives it global consistency. The key move is surprisingly small: turn the render error into a GTSAM factor.

Build it step by step Read the GTSAM post
GTSAM GraphSLAM poses · odometry · IMU · loops
SplatFactor
PyTorch Gaussian map means · scales · colors · opacity

render → compare pixels → residual + Jacobian

00 · THE ONE-SENTENCE IDEA

A camera image becomes a measurement on a pose node.

Render the current Gaussian map from a candidate pose, compare that render with the observed keyframe, and tell GTSAM how the pixel error changes when the pose moves.

Now photometric alignment, odometry, IMU, priors, and loop closure all constrain the same pose variables in the same incremental solver.

01 · INTERACTIVE WALKTHROUGH

Build Splatting SLAM, one factor at a time.

Use the numbered steps, or press play. The diagrams are a concept model—not a prerecorded video.

CONCEPT MODEL / RUNNING
01 / 07

INPUT · RGB-D KEYFRAME

A new observation anchors the story.

The first camera pose receives a strong prior. Its color and depth seed a small set of 3-D Gaussians—the beginning of the map.

A World + camera trajectory seed map
B Factor graph one pose + prior
THE MEASUREMENT Image evidence The observed keyframe initializes color.
Observed Ik
Rendered R(𝒢,Tk)
| residual |
MAP SIZE 128 splats

What changed: Depth lets us back-project pixels into 3-D; color initializes each Gaussian.

Z₀ = { I₀, D₀ } → 𝒢₀

02 · INSIDE THE BRIDGE

What exactly crosses into GTSAM?

The map stays in PyTorch. During pose optimization it acts as a fixed, differentiable measurement model.

1

Render

Project the 3-D Gaussians from the candidate camera pose.

Î = R(𝒢, T)
2

Compare

Sample image pixels and stack their RGB differences.

r = vec(Î[P] − I[P])
3

Linearize

Measure how every residual changes under a small SE(3) pose update.

J = ∂r / ∂ξ
4

Insert

Return the residual and Jacobian as a standard nonlinear factor.

φsplat(xk; 𝒢)
GTSAM / iSAM2Owns pose belief
  • Pose variables and priors
  • Odometry / IMU / loop factors
  • Photometric SplatFactors
  • Bayes tree + pose covariance
residual r Jacobian J The narrow interface
PyTorch / gsplatOwns scene belief
  • Gaussian means + rotations
  • Scales + opacities
  • Appearance / color
  • Adam map refinement

03 · COMPOSITION, NOT REPLACEMENT

GraphSLAM stays GraphSLAM.

The new factor does not replace odometry, IMU, or loop closure. It joins them. Toggle the available evidence and watch what the graph knows.

LIVE FACTOR GRAPH11 factors
LOCAL TRACKINGstrong
GLOBAL DRIFT17.4 cm
POSE σ8.1 cm

04 · FROM IDEA TO RUNNING SYSTEM

The concept has measurable consequences.

These are outputs from the linked implementation, not from the simplified browser model above.

KITTI 00 · ATE RMSE29.14 m 7.76 m73% lower
TUM fr1/desk · ATE RMSE0.189 m 0.113 m40% lower
POSE JACOBIAN CHECK< 0.1%relative error
Animation of loop closure correcting a drifting TUM fr1/xyz trajectory
TUM fr1/xyzA revisit becomes a global trajectory correction.

DINOv2 proposes a loop, geometry verifies it, and iSAM2 updates the affected Bayes-tree cliques.

Ground-truth TUM desk frame beside a render from the trained Gaussian map
THE MEASUREMENT MODELThe map can render what the camera should see.

The difference between these two views is exactly the evidence packaged by the SplatFactor.

See the KITTI 00 trajectory resultbird’s-eye trajectory + frame error
KITTI sequence 00 trajectory and frame error comparison

THE TAKEAWAY

The Gaussian map is not a separate correction system.
It becomes evidence inside the SLAM posterior.

That is the whole architectural shift: modern neural rendering on one side, decades of factor-graph machinery on the other, connected by a residual and its pose Jacobian.

Read the technical post Explore the implementation