Synthesizing 190° Fisheye Driving Videos: The Geometric Consistency Challenge for ADAS Training Data

Introduction: The Data Scarcity Dilemma in ADAS Training

The capabilities of Advanced Driver Assistance Systems (ADAS) are highly dependent on the scale and quality of training data. However, collecting large-scale driving data that covers various extreme scenarios in the real world is not only expensive but also faces a series of challenges including privacy concerns, safety risks, and the difficulty of reproducing long-tail scenarios. For these reasons, Synthetic Data has become a key area of exploration in the industry.

Synthetic data refers to data artificially created through computer programs, simulation engines, or generative models, rather than directly collected from the real world. In the autonomous driving domain, synthetic data is typically produced by game engines (such as Unreal Engine, Unity), professional simulation platforms (such as CARLA, NVIDIA DRIVE Sim), or neural network generative models (such as diffusion models, NeRF). Its core advantages lie in the ability to precisely control scene parameters, automatically generate pixel-level annotations (semantic segmentation, depth maps, optical flow, etc.), and reproduce or vary scenes without limitation. Industry estimates suggest that over 95% of testing miles in autonomous driving development rely on simulation environments, making synthetic data an indispensable component of training and validation pipelines.

Recently, a developer shared a representative practice in a technical community: they generated a synthetic fisheye driving video with a field of view of approximately 190° based on camera calibration parameters, with a primary focus on evaluating whether the video's geometric structure could maintain consistency under such extreme wide-angle distortion. This work touches on a core pain point when using synthetic data for ADAS training—geometric fidelity.

Why Choose a 190° Fisheye Perspective

The Real-World Need for Surround Perception

Modern vehicle surround view systems (Surround View Monitoring, SVM) typically rely on multiple fisheye cameras distributed around the vehicle body, using ultra-wide angles to achieve close-range panoramic coverage. A typical configuration includes one camera each at the front, rear, left, and right, with fields of view between 180°-220°, used for parking assistance, blind spot monitoring, and low-speed environment perception. The system fuses multiple fisheye images into a Bird's Eye View (BEV) or bowl-shaped panoramic image through image stitching algorithms. In recent years, deep learning-based BEV perception frameworks (such as BEVFormer, BEVDet) can directly extract 3D spatial features from multiple fisheye images without explicit stitching steps—these algorithms have extremely high requirements for geometric accuracy in training data.

A 190° ultra-wide field of view can cover the maximum area around the vehicle with the fewest cameras, but at the cost of introducing extremely significant optical distortion—objects at image edges are heavily stretched and curved. Fisheye lenses sacrifice rectilinear projection relationships in exchange for an ultra-large field of view, with optical projection models fundamentally different from traditional pinhole cameras. Common fisheye projection models include equidistant projection (r=fθ), equisolid angle projection (r=2f·sin(θ/2)), orthographic projection (r=f·sin(θ)), and stereographic projection (r=2f·tan(θ/2)). When the field of view exceeds 180°, objects behind the camera's optical axis are also imaged, and straight lines appear as curved arcs in the image—this radial distortion is particularly severe at image edges.

For perception algorithms, accurately identifying objects and estimating distances under such distortion is a unique challenge. Therefore, the ability to generate high-quality synthetic fisheye data has direct value for training and validating surround perception models.

The Technical Approach Starting from Camera Calibration

The key to this practice lies in generating synthetic video from camera calibration parameters. Camera Calibration is the process of determining a camera's intrinsic and extrinsic parameters. Intrinsic parameters describe the camera's own optical and geometric properties, including focal length (fx, fy), principal point coordinates (cx, cy), and various orders of distortion coefficients (radial distortion k1-k6, tangential distortion p1-p2, etc.). Extrinsic parameters describe the camera's pose in the world coordinate system, namely the rotation matrix R and translation vector t.

For fisheye cameras, the traditional Zhang's calibration method needs to be extended to support fisheye models, and vehicle surround system calibration also involves the relative poses between multiple cameras as well as rigid transformations between cameras and the vehicle body coordinate system. By embedding the physical optical model of a real camera into the synthesis pipeline, the generated video can potentially approximate the imaging characteristics of real fisheye cameras geometrically, rather than simply applying post-processing distortion to ordinary images.

This calibration-based generation approach theoretically allows synthetic data to remain consistent with the target deployment hardware camera, thereby reducing the Domain Gap between training data and actual inference scenarios. Domain gap is a core concept in transfer learning and domain adaptation, describing the statistical distribution differences between the source domain (e.g., synthetic data) and target domain (e.g., real sensor data). In autonomous driving scenarios, the domain gap manifests primarily at the pixel level (texture realism, noise patterns, color response curves), geometric level (projection model accuracy, object scale relationships), semantic level (scene layout plausibility), and temporal level (motion pattern naturalness). Research shows that geometric-level domain gaps often have a greater impact on perception task performance than appearance-level domain gaps—this is precisely why this work focuses on geometric consistency.

Geometric Consistency: The Core Test for Synthetic Data

What Is Geometric Consistency

Geometric consistency refers to whether the shape, size, relative position, and temporal motion of objects in synthetic video conform to the rules determined by real-world physics and the camera's optical model. For strongly distorted imaging systems like fisheye, geometric consistency is particularly fragile:

• Spatial distortion consistency: Whether the degree of distortion of the same object at different positions in the image matches the predictions of the calibration model. Fisheye lens distortion typically requires 8-12 or more distortion parameters for precise description, including high-order radial distortion, tangential distortion, thin prism distortion, etc. Any deviation in a single coefficient will lead to distortion of local geometric relationships.
• Temporal consistency: Whether the displacement and scaling of objects between consecutive frames are coherent and natural as the vehicle moves. For fisheye video, temporal consistency is even more challenging—when an object moves from the image center toward the edge, its distortion degree changes dramatically, and the generative model must precisely simulate this gradual transition. Under a 190° field of view, the same object may undergo the complete process from minimal distortion to extreme deformation in a short time, placing extremely high demands on the generative model's optical flow prediction and geometric transformation capabilities. Current mainstream video generation methods (such as Video Diffusion Models) typically maintain temporal consistency through temporal attention mechanisms and optical flow guidance, but their performance in strongly distorted scenarios remains to be verified.
• Depth and projection relationships: Whether depth information is correctly encoded when a 3D scene is projected onto a 2D fisheye plane. Since BEV perception frameworks fundamentally rely on accurate camera projection relationships to construct 3D spatial representations, deviations in projection relationships directly propagate as errors in 3D perception results.

If geometric relationships deviate during the synthesis process, the trained perception model may learn incorrect priors, leading to systematic errors during real deployment—which is unacceptable for safety-critical ADAS systems.

The Significance of Evaluation Over Generation Itself

Notably, this work did not stop at "generating a video that looks like fisheye" but explicitly placed its focus on verifying whether geometry remains consistent. This reflects mature engineering thinking.

In the synthetic data domain, there is a vast chasm between visually "looking right" and geometrically "being actually correct." Many generative models can output visually realistic images but cannot withstand scrutiny at the pixel-level geometric accuracy. For ADAS tasks requiring precise distance measurement and localization, the accumulation of geometric errors could lead to catastrophic consequences. Therefore, making geometric consistency evaluation the core objective is a necessary threshold for determining whether synthetic data can truly be deployed for training.

Specifically, quantitative evaluation of geometric consistency may involve reprojection error analysis, projection verification based on known 3D structures, comparison between optical flow fields and theoretical optical flow, and checking whether straight lines are restored after distortion correction. These evaluation methods form the technical foundation of a synthetic data quality assurance system.

Opportunities and Limitations of Synthetic Data for ADAS

Opportunities: Long-Tail Scenarios and Low-Cost Scaling

The greatest appeal of synthetic fisheye data lies in its controllability and scalability. Developers can generate various rare but dangerous scenarios on demand—such as fisheye perspectives under extreme weather, abnormal lighting, unusual obstacle layouts, etc.—without risking collection on real roads. Meanwhile, annotation information can be automatically obtained during the generation process, eliminating the expensive manual annotation step. Manual annotation in fisheye images is far more difficult than in regular images because distortion causes irregular object boundaries, making it hard for annotators to accurately judge the actual extent of objects.

The calibration parameter-based generation approach also enables customized datasets for specific vehicle models and specific camera configurations, achieving tight "train as you deploy" matching. This approach also supports Domain Randomization strategies—by randomly varying lighting, textures, weather, and other parameters during synthesis, the model is forced to learn feature representations that are robust to these variables, thereby alleviating the domain gap problem to some extent.

Limitations: Sim-to-Real Domain Gap and Validation Costs

However, synthetic data always faces the fundamental challenge of the Sim-to-Real domain gap. Even with highly consistent geometry, factors such as material rendering, lighting physics, and sensor noise may still cause distribution differences between synthetic data and real images. Real fisheye cameras also exhibit chromatic aberration, vignetting, lens flare, and other optical effects, as well as CMOS sensor-specific rolling shutter effects, which are difficult to perfectly replicate in synthesis. This is precisely why geometric consistency verification is so critical—it is the first line of defense for controlling the domain gap, providing a foundation of geometric correctness upon which other dimensions of distribution differences can be progressively reduced.

Additionally, validation itself carries costs. How to quantitatively measure geometric errors under a 190° field of view, and how to establish credible evaluation benchmarks, are directions that such work needs to continuously advance. The industry currently lacks standardized geometric evaluation protocols for ultra-wide-angle synthetic video, which is also one of the bottlenecks constraining large-scale application of synthetic data.

Conclusion: A Critical Step from Generation to Trustworthiness

This practice targeting 190° synthetic fisheye driving video represents a commendable trend in the synthetic data field: shifting from pursuing visual realism to pursuing geometric trustworthiness. For safety-critical systems like ADAS, the physical correctness of data is far more important than visual appearance.

As autonomous driving and assisted driving technologies continue to evolve toward surround view and multi-camera fusion, the ability to synthesize and validate ultra-wide-angle fisheye data will become a core infrastructure capability. Whoever can ensure geometric consistency under extreme distortion can provide more reliable training data for perception models. Although this exploration is still in its early stages, the question it raises—whether the geometry of synthetic data can truly be trusted—is a key proposition that the entire industry must answer on the path toward reliable autonomous driving.

From a longer-term technological evolution perspective, once geometric consistency issues are reliably resolved, the next challenge will be building a complete closed-loop validation system for synthetic data—including automated assessment of synthetic data quality, optimal mixing ratio strategies for synthetic-real data hybrid training, and safety certification standards for models trained on synthetic data in real-world scenarios. The resolution of these issues will ultimately determine whether synthetic data can be elevated from a supporting role to the primary data source for autonomous driving development.