Roadside Camera + 3D Radar: A Lightweight Two-Stage Approach for Real-Time 3D Object Detection

The Roadside Perception Dilemma: Why End-to-End Approaches Don't Work

In autonomous driving and advanced driver assistance systems (ADAS), end-to-end deep learning 3D object detection based on camera-radar fusion has become quite mature. But when the same technology is moved to roadside infrastructure, it faces a fundamental problem: sensor installation conditions are highly variable.

In vehicle-mounted scenarios, sensor height, pitch angle, and relative geometry are largely uniform, allowing end-to-end models to generalize well. Roadside installation is completely different—pole heights, tilt angles, and fields of view vary enormously across locations, causing end-to-end models to fail easily, and each new intersection requires large amounts of location-specific annotation data for adaptation.

To address this pain point, a recent study proposes a low-cost, resource-efficient two-stage approach that uses roadside RGB cameras combined with 3D radar to achieve real-time 3D shape and orientation estimation.

End-to-End Learning refers to using a single neural network to directly regress final task outputs (such as 3D bounding boxes) from raw sensor inputs (such as camera images and radar point clouds), without any explicit step-by-step processing in between. Its advantage lies in avoiding hand-designed features and pipelines, letting the model automatically learn optimal representations. But the cost is that the model depends heavily on the distribution of training data—once the test environment differs systematically from training data in geometry, lighting, installation angle, and other aspects, performance drops sharply. In vehicle-mounted scenarios, sensor installation poses are basically consistent, so end-to-end models generalize easily; in roadside scenarios, however, every intersection has different pole heights, tilt angles, and fields of view, which is equivalent to requiring the model to simultaneously adapt to hundreds or thousands of different "camera extrinsics," causing annotation and training costs to explode. This is precisely the root cause of end-to-end approaches failing at the roadside.

Two-Stage Decomposition: Detect 2D First, Then Build 3D

The core idea of this approach is: rather than designing a complex end-to-end pipeline to directly regress 3D bounding boxes, it's better to split the task into two steps.

Stage One: Reuse Mature 2D Detection

Stage one directly calls any off-the-shelf, state-of-the-art 2D detection method to extract 2D bounding boxes for each road user from camera images, while also obtaining corresponding 3D radar point cloud data. This step requires no retraining or fine-tuning—any method that can produce 2D boxes and radar point associations can be plugged in directly.

The benefit of this design is decoupling: 2D visual detection is one of the most mature perception capabilities today, with a rich open-source ecosystem, so developers don't need to reinvent the wheel for roadside scenarios.

2D Object Detection is one of the most mature tasks in computer vision, aiming to find the bounding rectangle and category of each object of interest in an image. Mainstream methods can be divided into two-stage detectors (such as Faster R-CNN, which first generates candidate regions and then classifies and regresses) and one-stage detectors (such as YOLO, SSD, RetinaNet, which directly predict boxes and categories in one step). In recent years, Transformer-based DETR series and various real-time variants have further improved accuracy and speed. Since 2D detection already has numerous pretrained models on large-scale datasets like COCO, with abundant open-source implementations, developers can directly use them without training from scratch. This approach precisely leverages this ecosystem, delegating "looking at images to find objects" to mature models, thereby concentrating effort on estimating 3D geometric quantities.

Stage Two: Predict Orientation with a Minimalist Network

Stage two converts 2D detection results into 3D oriented bounding boxes, with the following division of labor:

• 3D position: Determined directly by radar data, without relying on visual depth estimation
• Object dimensions: Using predefined templates matched to each category of road user (pedestrian, bicycle, car), avoiding network-learned dimension regression
• Yaw angle: Predicted by a very small neural network, optimized for specific sites

The most clever design here is the separation of dimension templating and orientation prediction. The physical dimensions of road users vary within a limited range, so using templates instead of network regression reduces both learning difficulty and data requirements. Yaw angle, however, is a position-dependent parameter that must be predicted in conjunction with specific installation conditions, so a small neural network is retained for site adaptation.

Yaw angle is an orientation parameter describing an object's rotation about the vertical axis, typically expressed as the angle relative to a reference direction (such as the camera optical axis or road direction). For autonomous driving and roadside perception, knowing only an object's position and dimensions is insufficient—you must also know which way it's "facing." For example, determining whether a pedestrian is about to cross or whether a vehicle is turning both depend on accurate yaw angle. The difficulty in estimating yaw angle is that the same object looks very different from different viewpoints, and the roadside camera's installation angle significantly changes the mapping between appearance and orientation. Therefore, this approach retains a small neural network specifically for yaw angle prediction, optimized for each site—essentially letting the network learn the "viewpoint-orientation" mapping for that site, rather than trying to cover all installation conditions with a single general model.

Validation Results: Curved Intersection, 25 FPS Real-Time Operation

The research team validated the approach on a curved road segment with an intersection, covering three categories of road users: pedestrians, bicycles, and cars.

Key performance metrics:

• Achieves 25 frames per second on NVIDIA Jetson ORIN, meeting real-time application requirements
• The entire second stage requires only a "tiny neural network," with extremely low computational overhead
• No need for location-specific large-scale datasets to train end-to-end models

Jetson ORIN is NVIDIA's embedded platform for edge computing. Achieving a full 25 FPS on this class of hardware indicates high feasibility for actual roadside deployment—roadside devices are typically constrained by power supply, heat dissipation, and cost, and cannot accommodate server-grade GPUs.

NVIDIA Jetson ORIN is an embedded computing module introduced by NVIDIA for edge AI, integrating an ARM CPU with an Ampere architecture GPU, and offering different power tiers (approximately 10W to 60W). It is commonly used in scenarios with strict constraints on size, power consumption, and heat dissipation, such as robots, drones, and intelligent transportation roadside devices, with computing power far below server-grade GPUs (such as A100, H100). Achieving 25 FPS 3D perception on such hardware means single-frame processing time of approximately 40 milliseconds, while simultaneously completing 2D detection, radar point association, and orientation regression. This result demonstrates that the two-stage approach allocates its computational budget reasonably and can operate stably under the power supply and heat dissipation constraints of real roadside devices, which is key evidence of its engineering deployability.

Insights from This Route

The most valuable aspect of this work is not innovation in any particular module, but rather trade-offs at the system design level.

When end-to-end approaches collapse in roadside scenarios due to data distribution differences, decomposing the problem into "mature 2D perception + radar localization + lightweight orientation regression," and replacing data-driven regression with prior knowledge (dimension templates), actually achieves better deployability and generalization capability.

For teams working on smart transportation and roadside perception system development, this "good enough" engineering mindset may be more worth learning from than pursuing end-to-end SOTA. Of course, whether templated dimensions remain effective for objects with large shape variations (such as trucks and special vehicles), and what the calibration cost for site adaptation entails, still require validation across more scenarios.

One-line Summary

Two-stage roadside 3D perception: reusing 2D detection + radar localization + tiny network for orientation prediction, achieving 25 FPS on Jetson.

Paragraph Summary

Addressing the problem of highly variable sensor installation conditions in roadside scenarios and the difficulty of generalizing end-to-end 3D detection, this study proposes a low-cost two-stage approach: stage one directly reuses off-the-shelf 2D detectors to extract 2D bounding boxes for road users and associate 3D radar point clouds; stage two uses a minimalist network to convert 2D results into 3D oriented bounding boxes, where 3D position is determined directly by radar, object dimensions are matched using predefined templates, and yaw angle is predicted by a tiny neural network optimized for each site. In validation on a curved intersection with three target categories—pedestrians, bicycles, and cars—the approach achieves 25 FPS real-time performance on the NVIDIA Jetson ORIN embedded platform. Its core value lies in replacing data-driven dimension regression with prior knowledge, trading off system design considerations for deployability and generalization capability.