L1 measures pixel error.
It measures the mean absolute RGB difference. It is direct and stable, but does not fully capture perceived local structure.
SIGGRAPH 2023 · Kerbl et al.
3D Gaussian
Splatting,
unpacked.
Calibrated photographs seed an explicit radiance field. Each iteration projects the Gaussians, evaluates image error, propagates gradients, and adapts the representation; later sections release the fixed camera pose for SLAM.
00
Core reconstruction pipeline
SfM→render→loss→backprop→update↺
Rendering uses splat rasterization, not ray marching.
01
Interactive training loop
One iteration,
frame by frame.
The sequence follows one iteration in the paper and reference training code, from fixed SfM calibration through rendering, loss evaluation, backpropagation, and adaptive density control.
Camera icons select the training view
02
Training objective
Pixels and structure,
in one loss.
ℒ
=
(0.8) ℒ1
+
(0.2) (1 − SSIM)
example training view
camera 03
combined loss
0.159
mean absolute pixel error
0.142
1 − local structural similarity
0.226
In the reference code, the paper’s D-SSIM contribution is evaluated as
1 - ssim(render, target). Then loss.backward()
differentiates through the rasterizer.
SSIM measures local structure.
It compares local luminance, contrast, and structure. The paper uses λ = 0.2, so L1 still carries 80% of the objective.
Optional depth regularization.
The 2024-updated repository can optionally add a monocular inverse-depth L1 term. That is an extension, not the original paper’s base loss.
03
Many cameras.
The role of SfM cameras
Many cameras.
One view at a time.
COLMAP provides one calibrated pose for every registered image, placing multiple cameras and the sparse point cloud in a common world frame.
-
01
SfM establishes shared 3D coordinates.
The sparse points and every camera pose live in the same world frame, so a Gaussian has one persistent 3D position across all views.
-
02
A camera projects the current model.
Its extrinsics
[Rᵢ | tᵢ]move world Gaussians into camera space; its intrinsicsKᵢmap them onto that image plane. -
03
The matching photo supplies supervision.
The rendered image is compared only with photo
Iᵢfrom that same pose. Across iterations, different views constrain the same scene.
view selector
Selectable cameras
POSES FIXED
selected
CAM 03
world → camera
R₃, t₃
supervision
image_003.jpg
Fixed-pose assumption
The original method treats the SfM calibration as known input. The Gaussians move and change while the camera poses remain fixed. Releasing the poses produces the transition to SLAM ↓
04
The renderer
Forward splatting,
without ray tracing.
There are no rays sampling an MLP through empty space. Known Gaussian primitives are projected toward the screen, sorted, and composited.
01
Project
Project covariance
Σ′ = J W Σ Wᵀ Jᵀ
W is the view transform and J is the local Jacobian
of perspective projection. The upper-left 2 × 2 portion becomes the ellipse
footprint on screen.
Composite a pixel
C = Σ cᵢ αᵢ ∏j<i(1 − αⱼ)
Each splat’s opacity is its learned opacity times its 2D Gaussian value at the pixel. Transmittance falls as nearer splats accumulate.
NeRF-style ray marching
sample space along every ray
3D Gaussian Splatting
send known primitives to tiles
05
Trainable scene parameters
Each point becomes
a small model.
Position
Where the Gaussian center sits in the shared world frame.
Scale
Three positive axis lengths, activated exponentially.
Rotation
A normalized quaternion that orients the ellipsoid.
Opacity
A sigmoid-constrained weight controlling contribution and visibility.
SH color
Spherical-harmonic coefficients encode view-dependent appearance.
Rather than optimize a covariance matrix directly, the method constructs Σ = R S Sᵀ Rᵀ. This keeps the covariance physically valid while allowing anisotropic Gaussians to stretch along surfaces.
Complete reconstruction loop
Calibrate once.
Render and correct, repeatedly.
01Photos + SfMfixed cameras, sparse points
→
023D Gaussiansexplicit trainable scene
→
03Splat a viewproject, sort, composite
→
04Compare + learnL1, SSIM, gradients
↺
Forward render → loss evaluation → backward pass
Repeated across views.
06
From reconstruction to SLAM
Releasing the camera pose
extends 3DGS to SLAM.
The differentiable renderer and image loss remain available, while the estimated state expands: an online system must recover each incoming camera pose and maintain the Gaussian map.
Method boundary
The original SIGGRAPH 2023 method is offline reconstruction—not SLAM. It receives SfM camera poses as fixed input. Gaussian rendering can nonetheless become a differentiable measurement model for tracking, mapping, and GraphSLAM.
Known poses,
unknown map.
𝒢̂ = argmin𝒢 Σᵢ ℒ(R(𝒢; Tᵢ), Iᵢ)
SfM supplies every Tᵢ. Gradient descent changes only the Gaussians.
Unknown pose,
evolving map.
alternate: Tt ↔ 𝒢
Track the current camera against the map, then update the map from accepted poses.
ORIGINAL PAPER · OFFLINE
SfM supplies fixed camera calibration.
All calibrated views supervise one shared Gaussian scene. A training iteration chooses a camera, but that camera is data—not a variable.
freeze {Tᵢ} · optimize 𝒢
Camera posesfixed
Gaussian maplearning
Image evidenceall views
Global graphnot needed
One GraphSLAM architecture
Render error becomes a pose factor.
With the Gaussian map temporarily frozen, a candidate pose produces an image residual and a pose Jacobian. A nonlinear solver can combine that factor with odometry, IMU, priors, and loop closures on the same pose nodes.
This is one integration pattern—not a claim that every Gaussian SLAM system uses GTSAM, iSAM2, or loop closure.
Î = R(𝒢,T)r = vec(Î − I)J = ∂r / ∂ξφsplat(xₖ;𝒢)
Solver-level formulation
A dedicated walkthrough develops the rendered image residual as a GraphSLAM factor.
Open the factor-graph lab
07
Next chapter · radio Splat-SLAM
Optimization with an
unknown UE pose.
Visual splatting compares rendered pixels with a camera image. A radio extension needs a different forward model: trace a candidate multipath, predict its delay and angles, then optimize the UE pose and reflecting geometry from the residual.
Visual factor
pose + Gaussian map
→
render pixels → image residual
Radio factor
pose + reflecting geometry
trace multipath → delay / angle residual
Model boundary
§3.1–§3.5 assume synchronized clocks, but no compass. The measurements are path length L = cτ, BS-side departure ψ, and UE body-frame arrival φbody. The global arrival direction is therefore φbody + θ, with heading θ kept inside the unknown state. §3.6 deliberately adds a stronger, globally referenced displacement sensor for its rank test.
01Hypothesizeu, θ, walls, incidence points
→
02Traceordered specular path
→
03Predictτ̂, φ̂body, ψ̂
→
04Compareradio residual
↺
The endpoint E is the mirror image of the UE for the selected path. Re-pinning
the remaining path length at P and rotating φbody by a candidate θ
constructs a valid UE and an implied wall. One bounce therefore cannot
identify position, wall, or heading by itself.
Shared optimization structure
Visual and radio estimation share the pattern state → forward model → residual → update, while radio measurements introduce different null directions. This structure supports persistent factors connecting UE poses, reflecting splats, and data-association variables.
Primary sources