The Fatal Flaw of Gaussian Splatting: Oxford's Survey-Grade Dataset Reveals the Truth About Off-Trajectory Collapse

The "Pseudo-Photorealistic" Illusion of Gaussian Splatting

If you've ever played with 3D Gaussian Splatting, you've likely been amazed by its photorealistic rendering quality. But this impressive result often comes with a hidden prerequisite: your camera must move along the training trajectory. The moment you deviate from the training path, geometric structures begin to collapse—previously fine surfaces develop stretching artifacts, floating blobs, and obvious deformations.

How 3DGS Works and Its Inherent Limitations

3D Gaussian Splatting (3DGS) is a novel 3D scene representation and rendering method proposed by France's INRIA team in 2023. Unlike NeRF, which uses implicit neural networks, 3DGS explicitly represents scenes using millions of 3D Gaussian ellipsoids, each with position, covariance matrix, color (spherical harmonics coefficients), and opacity attributes. During rendering, these Gaussians are projected ("splatted") onto the 2D image plane and composited into final pixel colors via alpha blending. Thanks to its explicit representation and rasterization-based rendering pipeline, 3DGS achieves real-time (30+ fps) high-quality rendering with training speeds far exceeding NeRF. However, explicit representation also brings an inherent flaw: the distribution of Gaussians is heavily dependent on training viewpoint coverage, and areas lacking observational constraints tend to develop degenerate geometric structures—this is the root cause of the "off-trajectory collapse" problem.

This is precisely the core issue that the Oxford Robotics Institute aims to address with their latest dataset release. The crux of the matter is: this "off-trajectory collapse" has almost never been truly quantified. The reason is simple—to evaluate reconstruction accuracy, your ground truth reference must be more precise than the object being evaluated. Obtaining such high-precision ground truth means hauling survey-grade scanners to the site and spending days on data collection.

What True "Survey-Grade Ground Truth" Means

Many existing Novel View Synthesis (NVS) benchmarks suffer from a methodological flaw: the so-called "test images" are actually just held-out frames from the training path. This means test viewpoints are highly similar to training viewpoints, unable to truly expose a model's generalization capability from unfamiliar angles.

The Fundamental Flaw in Existing Evaluation Methodology

The standard evaluation pipeline for novel view synthesis typically involves randomly splitting input images into training and test sets (e.g., holding out every 8th frame for testing), then computing image quality metrics like PSNR, SSIM, and LPIPS on test frames. The fundamental problem with this approach is that test frames come from the same camera trajectory as training frames, with very small viewpoint differences (usually just a few centimeters of translation and a few degrees of rotation). Models only need to perform minimal interpolation to achieve high scores, completely failing to test performance under large baseline shifts or entirely new observation directions. It's like an exam where all questions are variations of homework problems—high scores don't mean true mastery.

The Oxford team went all-in this time. They collected data for six Oxford landmarks with the following specifications:

• 24 sequences covering 125,000 square meters of space
• A handheld acquisition device integrating three synchronized fisheye cameras, a 64-beam Hesai LiDAR, and an IMU
• Each site was reference-scanned with a Leica RTC360 scanner, achieving accuracy of 1.9 mm at 10 meters
• Trajectory registration accuracy controlled to 1-2 centimeters

Leica RTC360: Guaranteeing Survey-Grade Accuracy

The Leica RTC360 is a professional-grade 3D laser scanner widely used in architectural surveying, heritage preservation, and industrial measurement. It acquires precise 3D coordinates of object surfaces by emitting laser pulses and recording reflection times, capturing approximately 2 million points per second. Its 1.9 mm ranging accuracy at 10 meters means that for typical architectural scenes (10-30 meter scale), the absolute error of scan results is controlled at the millimeter level. In comparison, the best current 3D reconstruction algorithms (such as 3DGS or NeRF) typically have reconstruction errors in the centimeter to decimeter range for outdoor scenes. This order-of-magnitude precision difference ensures that scan results serve as unquestionable ground truth. Scanning a complete site typically requires setting up dozens of stations, with each station taking about 2 minutes to scan, plus target placement and registration—a single site often requires days of work.

Multi-Sensor Acquisition System Design

The Hesai 64-beam LiDAR is a mechanically rotating LiDAR sensor that achieves 3D scanning of the surrounding environment through 64 vertically distributed laser emission/reception channels, producing approximately 1.2 million points per second. Compared to solid-state LiDAR, mechanical LiDAR offers the advantage of 360-degree full field-of-view coverage, making it particularly suitable for mobile mapping applications. In multi-sensor systems, time synchronization is a critical challenge—data from cameras, LiDAR, and IMU must be precisely aligned to the same time reference (typically achieving microsecond-level synchronization through hardware trigger signals or PTP protocols), otherwise temporal offsets during motion cause severe spatial registration errors. The choice of fisheye cameras maximizes field-of-view coverage, with three synchronized fisheye cameras achieving near-panoramic image capture.

The significance of this setup is that the ground truth precision far exceeds the output of any 3D reconstruction algorithm, enabling millimeter-level objective assessment of reconstruction quality.

Key Design: Truly "Off-Trajectory" Testing

The most valuable design aspect of this dataset lies in how the test set is constructed. Novel view test images are not frames extracted from the training path, but from a separate walk through the same site, facing completely different directions.

This is precisely the scenario that breaks Gaussian Splatting. When a model faces observation directions never seen during training, can it still maintain geometric consistency? This is the true standard for measuring 3D reconstruction robustness, rather than chasing high PSNR scores on familiar trajectories.

SLAM Pose Estimation and Trajectory Registration

SLAM (Simultaneous Localization and Mapping) is a core technology in robotics and autonomous driving that allows mobile devices to simultaneously estimate their own position and build a map of an unknown environment. In this dataset, the SLAM system fuses LiDAR point clouds, IMU inertial measurements, and visual information to estimate the acquisition device's six-degree-of-freedom pose (3D position + 3D orientation) at each moment. However, SLAM trajectories suffer from drift errors and need to be registered to the absolute coordinate system established by the Leica scanner. The registration process typically uses ICP (Iterative Closest Point) algorithms or feature-matching methods to align SLAM trajectories into the survey coordinate system. The 1-2 centimeter registration accuracy means that the camera position for every frame in the dataset is precisely localized, which is crucial for evaluating reconstruction depth accuracy.

Multi-Modal Data Integration and Visualization

After collecting massive amounts of multi-sensor data, making it convenient for researchers to use is equally challenging. The team packaged six acquisition episodes into MCAP format, a container format designed for multi-modal time-series data.

MCAP Format and the Robotics Data Ecosystem

MCAP is an open-source data container format developed by Foxglove, designed specifically for multi-modal time-series data. Its predecessor traces back to the rosbag format in the ROS (Robot Operating System) ecosystem, but MCAP features a more modern design: supporting efficient random access, multiple serialization protocols (Protobuf, FlatBuffers, ROS messages, etc.), built-in indexing, and chunked compression. Unlike traditional video files or image sequences, MCAP can encapsulate heterogeneous data streams—images, point clouds, IMU readings, pose estimates, etc.—in a single file while maintaining precise timestamp alignment. This allows researchers to replay and analyze the entire acquisition process in a unified manner, without manually synchronizing separate files from different sensors.

Using the FiftyOne visualization tool, you can synchronously view on the same timeline:

• Feeds from three fisheye cameras
• LiDAR point clouds
• IMU data
• SLAM poses

Furthermore, LiDAR depth information is "painted" onto each image frame, directly mapping 2D images to 3D geometry, greatly facilitating alignment verification and debugging.

Ready-to-Use Online Experience

For researchers wanting to get started quickly, the project offers two access methods:

• Dataset location: Voxel51/oxford-spires-multimodal on Hugging Face
• Online Space: Explore harpreetsahota/oxford-spires-multimodal-explorer directly in your browser without installing anything

This "zero-installation" online exploration approach significantly lowers the barrier to evaluating your own reconstruction results.

Why This Matters for the 3D Reconstruction Field

As NeRF and 3D Gaussian Splatting move from academic demos toward real-world applications like robot navigation, AR/VR, and digital twins, geometric accuracy and viewpoint generalization become far more critical than simply "looking good." A model that's perfect on training trajectories but collapses off-trajectory is unreliable in real-world scenarios involving autonomous robot exploration or free user roaming.

Hard Requirements for Geometric Accuracy in Real Applications

In robot navigation scenarios, the geometric accuracy of 3D reconstruction directly impacts the reliability of path planning and obstacle avoidance—if a reconstructed wall position is off by 5 centimeters, a robot might collide or fail to pass through narrow corridors. In AR/VR applications, virtual objects need to be precisely "anchored" to real-world surfaces; geometric errors cause objects to float or sink into the ground, breaking immersion. Digital Twins require millimeter-level geometric fidelity to support remote inspection, deformation monitoring, and construction verification. The common characteristic of these application scenarios is that users or robots observe scenes from arbitrary directions rather than along preset fixed trajectories, making viewpoint generalization a hard deployment requirement rather than a nice-to-have bonus.

The value of the Oxford Spires dataset lies precisely in providing a sufficiently precise measuring stick. With survey-grade ground truth and a genuinely off-trajectory test set, researchers can finally quantitatively answer a long-avoided question: how much does my reconstruction degrade from unfamiliar viewpoints?

From a methodological perspective, this also reminds the entire field that benchmark design itself often drives technological progress more than chasing SOTA scores. When evaluation methods cannot expose real deficiencies, even the highest metrics are self-deception. For any team serious about 3D reconstruction, cross-validating their models with such high-quality ground truth datasets is an essential step toward practical deployment.

Key Takeaways