Inspect the loss. Change which parameters can learn. See what an extra camera makes observable.
0.80 × L1 + 0.20 × (1 − SSIM)
Step once to see the optimizer respond.A whole line of depths matches camera 1.
Computed 2D teaching model: anisotropic footprints, fixed depth order, constant RGB colors. This is not the full 3DGS rasterizer.
Click an image above, or choose a pixel. Layers appear in the renderer’s front-to-back order.
| Layer | Opacity o | Footprint g | α = o · g | Transmittance T | Weight Tα | RGB contribution |
|---|
Rendered RGB—
Target RGB—
∂L / ∂RGB at this pixel—
The background uses the transmittance left after every splat. The RGB gradient includes the L1 term and all overlapping SSIM windows. It is the signal passed back through the compositing calculation.
Each splat has a 2D center, two positive scales, a rotation, an opacity and an RGB color. At a pixel, its unnormalized Gaussian footprint multiplies its opacity. The renderer then blends the splats in a fixed front-to-back order, including the remaining background. Overlap and occlusion affect which parameters receive useful gradients.
L1 is the mean absolute error over every pixel and RGB channel. SSIM compares local means, variances and covariance. The combined objective is (1 − λ)L1 + λ(1 − SSIM). The paper uses λ = 0.2. The colored residual plot shows the per-pixel mean absolute RGB error, not the SSIM contribution. Its scale stays fixed, so a darker image means less error.
The model differentiates the image loss through alpha compositing and the Gaussian footprint. Adam updates the allowed parameter groups. Shape includes both scales and rotation. Positive scales use an exponential parameterization. Opacity and RGB use sigmoid parameters in this teaching model. The target image stays fixed, and no reference splat parameters enter the optimizer.
Different overlapping splats can produce similar pixels. Color and opacity can compensate for one another, and a frozen parameter group can force other groups to absorb the error. This lab uses a fixed number of 2D splats. Original 3DGS additionally projects 3D covariances, handles view-dependent colors and changes model capacity through cloning, splitting and pruning.
Run 50 steps and inspect the residual. Freeze Position, reset the splats, and repeat. Then enable Position and continue. Change λ to see the exact loss arithmetic update. The history restarts when the objective changes, so values from different objectives are not joined as one curve.
Demo-specific numerical settings. Adam uses smaller rates for position and shape, with all rates multiplied by 1 / (1 + t / 180) at iteration t. Stability bounds keep centers in [−0.15, 1.15], scales in [0.02, 0.5], and opacity/color logits in [−8, 8]. These are teaching-model settings, separate from the original 3DGS schedule.
This lab renders one isotropic Gaussian with fixed size, color and opacity. Only its 3D center (x, y, z) learns. Calibrated orthographic camera 1 sees (x, y), camera 2 sees (z, y), and a held-out camera sees a 45° mixture of x and z. The images have the same pixel loss as the fitting lab, with λ = 0.2.
Moving this Gaussian along z leaves the first rendered image unchanged. The image-loss derivative with respect to z is therefore zero. Training can match the first target while preserving an incorrect initial depth. The gray vertical line marks centers with the same camera-1 horizontal projection as the reference.
Camera 2 sees the z displacement as a horizontal image error. When both cameras train the same center, their constraints intersect. The optimizer receives image gradients in the previously invisible direction. The held-out camera provides an additional image check and never contributes to training.
The cameras and appearance are known, the scene has one splat, and the targets are noise-free. Real 3DGS has more unknowns, occlusions, calibration error and potentially unobserved surfaces. Multiple views provide stronger constraints, but low training loss alone does not prove that the inferred scene is the unique physical geometry.
Keep Camera 1 only and run 50 steps. Compare the small training loss with the held-out error and the wrong z coordinate. Select Cameras 1 + 2 and run another 50 steps without resetting. Watch the shared center move toward the intersection of the two image constraints.
Demo-specific numerical settings. Adam uses rate 0.035 / (1 + t / 100), with center coordinates bounded to [−0.1, 1.1]. In the one-camera case, z and its optimizer momentum remain fixed because z has no effect on that image. Switching cameras preserves the estimated center.