One image loss. Two kinds of unknown.

The photographs stay fixed. Camera pose and Gaussian centers learn from the same L1 + SSIM residual.

Fit both observed images

Iteration 0

Mean absolute RGB error—L1

Mean local similarity—SSIM

Weighted image loss—Initial state

(1 − λ)L1 + λ(1 − SSIM)

—

Measured over both images. No true pose enters the loss.

Image loss actual updates

Computed 2D orthographic teaching model: four anisotropic splats, shared centers, one anchored camera. Shape, color, opacity and layer order stay fixed. It is not a full 3D SLAM system.

The shared map and the estimated cameraWorld coordinates · 1σ Gaussian ellipses · no reference geometry

One residual, two gradient paths

Image derivatives flow through the renderer to each splat center and to the camera pose. Rotating the camera changes both projected centers and anisotropic footprints.

‖∇_centersL‖—‖∇_poseL‖—

These are gradients of the current image objective, including frozen blocks. A frozen block receives no update. Their norms use different parameter units; they are not an importance ranking.

Pose convention: m = ½ + R(θ)(μ − ½) + t. The figure shows the corresponding inverse camera frame in world coordinates. Fixing camera 0 defines the coordinate frame for this reduced example.

How can L1 be known without the true pose?

Choose a pixel in a photo. The comparison is between its observed RGB and the current prediction at that same pixel.

View x y Click any image, or enter zero-based pixel coordinates.

Observed RGB · fixed—

Rendered RGB · current estimates—

Absolute channel errors—

Mean error at this pixel—

The optimizer does not need to know which scene point produced this pixel, or the true camera pose. It renders using its current pose estimate, compares against the fixed observed image, and differentiates that discrepancy. The full L1 averages these errors over pixels, RGB channels and both views. SSIM additionally compares local image statistics; the residual heatmap shows only absolute RGB error.

The distinction to keep

Joint pose–splat optimization can also use L1 + SSIM. Removing SSIM is a MonoGS online-loss choice, not a requirement for pose gradients.

Unknown pose does not mean unknown photo

A real camera supplies an RGB image even if its position and orientation are unknown. Here the two observed images are synthetic and are generated once. Their generating parameters are discarded; the optimizer receives pixels, the current estimates and the calibrated projection model. Changing an estimate changes the render, never the photo.

SSIM remains in the joint objective

With λ = 0.2, this lab minimizes 0.8 × L1 + 0.2 × (1 − SSIM). Local SSIM uses an 11 × 11 Gaussian window, standard deviation 1.5, with zero padding and constants C₁ = 0.01² and C₂ = 0.03² for RGB values in [0, 1]. There is no extra factor of ½. Both terms contribute to pose and splat gradients.

Simultaneous and alternating updates

Simultaneous updates use both gradient blocks at the same estimates. Alternating updates change the pose, then recompute the residual before changing the splats. Both solve the same coupled objective. The model uses gradient steps with backtracking to keep this small experiment stable; it does not reproduce MonoGS’s full optimizer.

A small residual is not a proof of unique geometry

Anchoring camera 0 removes a coordinate freedom in this 2D model. Its known scales and appearance also constrain the problem. General 3D monocular reconstruction has additional ambiguities, including metric scale. Good initialization, overlapping views and useful scene structure matter; SSIM does not remove those ambiguities.

Try this sequence

Run 50 joint steps at λ = 0.2. Reset, freeze the pose, and repeat: the shared map must absorb what it can of the pose error. Then enable both blocks and continue. Try λ = 0 (L1 only), or λ = 1 (SSIM only): either objective can supply pose gradients. History starts a fresh segment whenever the objective or update choices change.