PanoPed: A New Sim-to-Real Benchmark for Panoramic Pedestrian Tracking with the Sextant Angular Localization Head

Panoramic cameras enable full-field-of-view person tracking for fixed surveillance systems and mobile robots alike — but conventional planar bounding boxes cannot accurately describe a person's true position on a spherical surface. A new arXiv paper introduces PanoPed, a project that tackles this problem at its root by constructing a sim-to-real benchmark specifically designed for full-sphere pedestrian tracking, alongside an extremely lightweight angular localization module called Sextant.

Why Bounding Boxes Fail in Panoramic Scenes

Full-sphere cameras cover the entire field of view in both horizontal and vertical directions, making them highly valuable for fixed surveillance, quadruped robots, and drones. However, when an image is projected onto an Equirectangular Projection (ERP) plane, a person's actual spherical position becomes severely distorted.

The paper identifies a core problem: a rectangular box on the ERP plane cannot uniquely determine the spherical center position and angular extent of a visible person. In other words, the same rectangular box may correspond to multiple distinct real-world positions on the sphere. This means that tracking systems relying solely on detector-output boxes are actually losing critical spatial information.

Interestingly, the researchers also identified a promising breakthrough: the visual queries produced by the detector still carry information about spherical centers and angular extents — it's just that this information hasn't been fully exploited. This insight laid the groundwork for the lightweight localization approach that follows.

Equirectangular Projection (ERP) is the most common storage format for panoramic images: it maps the longitude of a spherical coordinate system to the horizontal axis and latitude to the vertical axis, effectively "unrolling" the entire sphere into a rectangular image. This projection introduces minimal distortion near the equator, but objects near the poles (top and bottom of the image) are severely stretched horizontally. A pedestrian standing on the ground near the edge of the frame will have a bounding box that diverges significantly from their true spherical footprint due to polar distortion. Traditional 2D detectors (such as YOLO and DETR variants) are trained on planar images and output bounding boxes in pixel coordinates with no spherical geometry information. As a result, two bounding boxes with identical pixel coordinates but at different latitudes can correspond to completely different solid angles and real-world positions on the sphere — this is the mathematical root of the "non-uniqueness" problem.

The PanoPed Dataset: A Dual-Track Sim-to-Real Design

One of PanoPed's core contributions is a substantial dual-track dataset that directly addresses the sim-to-real transfer challenge.

Synthetic Split: PanoPed-S

PanoPed-S contains 108,000 frames of synthetic data captured from three perspectives: fixed cameras, quadruped robot-mounted cameras, and drone-mounted cameras. Crucially, every frame comes with synchronized segmentation masks, depth information, camera poses, and 3D pedestrian states. This kind of dense, multimodal annotation is nearly impossible to obtain at scale in the real world — which is precisely where synthetic data shines.

Real-World Split: PanoPed-R

PanoPed-R supplements the dataset with 28,002 frames of real footage from fixed cameras, of which 16,247 frames are densely annotated. The pairing of real and synthetic data allows researchers to systematically evaluate whether models trained on synthetic data can transfer directly to real-world scenarios without fine-tuning.

Sextant: An Angular Localization Head Inspired by the Nautical Sextant

The most elegant design in the paper is the Sextant angular localization module. Its inspiration comes from the nautical sextant, which locates objects through angular measurement — and since spherical localization is fundamentally an angular problem, the approach measures angles directly.

Sextant has standout engineering properties:

• Extremely small parameter count — only ~0.035M (35,000) parameters, almost negligible;
• Plug-and-play — reuses a frozen detector without altering track identities;
• No additional image encoder required — avoids the overhead of a heavy visual backbone.

This design philosophy is highly practical for real-world deployment: you can attach Sextant directly to an existing detection-tracking pipeline, add virtually no computational burden, and immediately gain spherical angular localization capability.

The concept of visual queries originates from Transformer-based object detectors (such as DETR and its variants). In these detectors, the decoder maintains a learnable query vector for each candidate object; after repeated cross-attention with image features, this vector progressively aggregates appearance and position information from the corresponding region and is ultimately decoded into bounding box coordinates and class probabilities. Sextant's key insight is that while this query vector was never explicitly supervised to encode spherical angles, it has already "seen" the image features of that region, meaning its internal representation latently retains sufficient spatial cues. Rather than introducing a new visual encoder from scratch, Sextant takes the frozen detector's output query vectors as input and trains a tiny regression head to predict the spherical center azimuth and elevation, along with the corresponding angular extent. This "parasitic" design compresses the parameter count to roughly 35,000 while ensuring seamless compatibility with the existing detection-tracking pipeline.

Experimental Results: Consistent and Broad Improvements

On the PanoPed-S test set, Sextant achieves state-of-the-art performance. It improves the HOTA metric of the strongest baseline, MOTIP, from 47.30 to 49.49, with gains across all eight test sequences — this across-the-board rather than selective improvement speaks to the method's robustness.

Even more noteworthy is the sim-to-real performance. Without fine-tuning on any real data, the localization head trained purely on synthetic data improves MOTIP and HAT on real videos by 0.96 to 1.14 HOTA, with both random seeds delivering improvements on every real-world sequence. This consistency indicates that the synthetic-to-real transfer is stable, not coincidental.

Among systems that do not add a localization image encoder, HAT+Sextant achieves the best score, further validating the competitiveness of the lightweight approach.

HOTA (Higher Order Tracking Accuracy) is a comprehensive evaluation metric widely adopted in multi-object tracking in recent years. It is the geometric mean of DetA (detection accuracy) and AssA (association accuracy), simultaneously reflecting detection quality and identity consistency — addressing the shortcoming of the earlier MOTA metric, which overemphasized missed and false detections while neglecting identity switches. In panoramic tracking, HOTA is especially important: spherical localization errors not only affect the matching threshold for detection boxes, but also directly cause the same pedestrian to be judged as different identities across frames, dragging down the AssA component. The HOTA improvements brought by Sextant therefore reflect a synergistic gain in both spatial matching and identity maintenance, driven by improved spherical localization accuracy.

Significance and Outlook

PanoPed's work delivers dual value to the panoramic perception community. On one hand, it provides a sim-to-real benchmark with complete 3D annotations and multi-viewpoint sources, filling the gap left by the absence of standardized evaluation for full-sphere pedestrian tracking. On the other hand, Sextant demonstrates a minimalist technical philosophy — rather than stacking larger models, it mines underutilized information already present in existing detectors, achieving spherical localization capability at minimal cost.

For applications that rely on omnidirectional perception — robotics, drones, and intelligent surveillance — this approach, accurate yet lightweight, has clear deployment potential. As panoramic cameras become more prevalent on mobile platforms, moving beyond bounding boxes toward genuine spherical spatial understanding may well become the next frontier in pedestrian tracking.