Where Does Autonomous Driving Training Data Come From? Cosmo 3 Uses Scene Graphs to Generate Multi-Sensor Data

When training autonomous driving models, one core challenge is always unavoidable: where does high-quality training data come from? Real-world road testing is expensive and long-tail scenarios are scarce, while pure simulation often falls short in realism, making model transfer difficult. In a recent technical sharing session, Cosmo 3 presented a noteworthy solution—controlled generation via world scene graphs that simultaneously outputs RGB and LiDAR data, enabling closed-loop simulation to continuously produce training samples.

Why Autonomous Driving Training Data Is Hard to Obtain

Autonomous driving models need to cover extreme weather, lighting conditions, sudden obstacles, and other long-tail scenarios—events that occur with extremely low probability on real roads. Collecting data with real vehicles means long road-testing cycles and expensive annotation costs; relying on traditional simulation often compromises sensor realism. Cosmo 3's starting point is precisely to fill this gap: using generative methods to produce controllable, multi-modal driving data on demand that can evolve continuously.

According to the technical sharing session, the core capability of Cosmo 3 for autonomous driving is "controlled generation via world scene graphs"—developers can constrain generated content based on scene graph descriptions rather than passively accepting random sampling results.

Three Key Capabilities of Cosmo 3

1. World Scene Graph-Driven Controllable Generation

A world scene graph is essentially a structured description of objects, road structures, and their relative relationships within a scene. Using it as a condition, the generative model produces data that aligns with predetermined driving contexts, making "targeted data supplementation" possible—generate whatever scenarios are missing rather than relying on chance.

World Scene Graph is a structured representation that abstracts a 3D scene into nodes and edges: nodes correspond to entities in the scene (such as vehicles, pedestrians, traffic signs, and road surfaces), while edges represent spatial or semantic relationships between entities (such as "located on the left" or "facing the ego vehicle"). Compared to directly inputting a reference image or a text description, a scene graph explicitly decomposes "what is in the scene and how they are arranged" into an editable symbolic structure, allowing precise addition or removal of objects and layout adjustments without affecting other parts. In generative models, scene graphs are typically used as conditional inputs, injected into diffusion/autoregressive generation processes through graph neural networks or cross-attention, thereby constraining outputs to align with predetermined contexts. This is also the foundation for enabling "generate what's missing" targeted data generation.

2. Synchronized and Consistent RGB and LiDAR Generation

Multi-sensor fusion is the mainstream approach in autonomous driving perception, but spatiotemporal alignment between different sensor data has always been a simulation challenge. Cosmo 3 emphasizes "simultaneously and consistently generating RGB sensor and LiDAR sensor outputs," meaning images and point clouds correspond to each other in content rather than being independently generated and then hard-aligned.

LiDAR point clouds are obtained by emitting pulses from a laser radar and measuring echo times, outputting a 3D point set with reflection intensity that is sparse, unordered, and viewpoint-dependent; RGB images, on the other hand, are dense pixels on a regular grid. These two modalities differ enormously in data structure, and the traditional approach of rendering separately and then aligning through calibration parameters is prone to geometric misalignment or semantic contradictions (e.g., a car appearing in the image but missing from the point cloud). Synchronized consistent generation requires the model to jointly decode both modalities under the same scene latent representation, ensuring that object contours in images strictly correspond to point clusters in point clouds in 3D space. A common approach is to share an intermediate 3D feature field or voxel representation, then project separately onto the camera plane and LiDAR scanning pattern to guarantee spatiotemporal alignment.

3. Continuous Generation for Closed-Loop Simulation

Single-frame data can only support perception training, whereas closed-loop simulation requires scenes to evolve continuously over time and respond to ego-vehicle behavior. Cosmo 3's continuous generation capability is designed precisely for closed-loop simulation scenarios—the model acts within the simulation, the environment continuously generates new observations, forming an iterable training loop.

The key difference between closed-loop simulation and open-loop replay is that the simulation environment generates new observations in real time based on the ego vehicle's decisions, and the ego vehicle updates its decisions based on these observations, forming a feedback loop. This imposes temporal consistency requirements on generative models—scene layouts, object motion trajectories, and lighting changes between adjacent frames must be smooth and coherent, rather than independently randomly sampled each frame. This typically requires incorporating historical frames as conditions, maintaining latent variables of scene state, and ensuring generation speed meets real-time or near-real-time simulation requirements. Closed-loop simulation can expose error accumulation issues that open-loop data cannot reveal, and is therefore regarded as an important step in validating end-to-end autonomous driving models.

What This Means for Autonomous Driving Data Pipelines

Combined, these three capabilities point to a "generative data pipeline": using scene graphs to define requirements, multi-sensor consistent generation to ensure data quality, and continuous generation to support closed-loop validation. Compared to pure real-world data collection, its advantage lies in controllability and scalability; compared to traditional simulation, its advantage lies in content realism and modality completeness.

Of course, whether generated data can fully replace real-world road testing still depends on the physical consistency and distribution coverage of generated content, which needs to be validated in actual training.

Open Access and Further Observations

The presenter stated that all related capabilities have been opened up and invited developers to try them out and provide feedback ("We open up everything and I hope you give it a try"). For teams building autonomous driving data pipelines, such tools are worth adding to the evaluation list—with focus on the transfer performance of generated data on downstream perception tasks and stability in closed-loop simulation.

This article is based on a public technical sharing session. Please refer to official documentation for specific capabilities and access scope.

One-line Summary

Cosmo 3 uses world scene graphs to controllably generate synchronized RGB and LiDAR data, supporting closed-loop simulation training.

Paragraph Summary

The article introduces Cosmo 3's solution for autonomous driving data generation: addressing the high cost of real-world road testing, scarcity of long-tail scenarios, and insufficient realism of traditional simulation, it proposes controllable generation conditioned on world scene graphs. Its three key capabilities are: scene graph-driven targeted scenario generation, synchronized and consistent multi-modal RGB and LiDAR output, and continuous temporal generation for closed-loop simulation. Together they form a "generative data pipeline" whose advantages lie in controllability, scalability, and modality completeness. The article also notes that whether generated data can replace real-world road testing still depends on physical consistency and distribution coverage, requiring validation in downstream tasks, and recommends teams focus on transfer performance and simulation stability.