Boreas Dataset: A Multimodal Autonomous Driving Dataset with 44 Drives Through Wind and Snow

When Autonomous Driving Meets Real Extreme Weather

Most autonomous driving datasets have an unspoken "cheat": they're almost all captured on sunny days in California or Phoenix. Current mainstream autonomous driving datasets like the Waymo Open Dataset (primarily collected in Phoenix, San Francisco, etc.), nuScenes (Boston and Singapore), and KITTI (Karlsruhe, Germany) are mostly collected under mild weather conditions. This data bias has led to a systemic problem known in the industry as "sunshine bias": models perform excellently within the training distribution but suffer dramatic performance degradation in out-of-distribution adverse weather scenarios. Statistics show that approximately 38% of vehicle accidents in the United States occur in adverse weather conditions, yet adverse weather samples typically account for less than 5% of training data. These datasets produce models that perform brilliantly during training, but once the weather turns bad—heavy rain pouring down, snow blowing everywhere—they tell us almost nothing about what will happen.

The Boreas dataset, developed by the University of Toronto Institute for Aerospace Studies (UTIAS), is a direct response to this gap. Rather than choosing sunny desert highways, the team mounted a sensor suite on a vehicle and drove the same Toronto route across an entire year through all four seasons, completing 44 traversals—capturing complete scene variations under sunshine, rainfall, and snowfall.

Why Weather Is So Deadly for Autonomous Driving Sensors

This dataset tackles the core pain point of autonomous driving perception: the drastically different performance of various sensors under adverse weather conditions.

• Lidar: Lidar perceives the environment by emitting near-infrared laser pulses and measuring reflected echoes. During snowfall, every snowflake scatters the laser signal, producing what's known as "snow clutter" in the point cloud. Falling snowflakes appear as thousands of false objects in the point cloud. Research shows that moderate snowfall can introduce over 10,000 false echo points in a single point cloud frame, distributed within tens of meters around the sensor, forming a three-dimensional noise layer resembling a "snow curtain." For algorithms, every snowflake could be misidentified as an obstacle, directly interfering with object detection and path planning. More seriously, snow covering road surfaces and vehicle exteriors changes reflective properties, forcing ground segmentation and object detection algorithms to simultaneously face the dual challenge of increased noise and signal degradation.
• Radar: In most snowy scenarios, radar is nearly "unaware" that it's snowing, possessing natural resistance to weather interference, but with limited resolution and semantic information. This is because millimeter-wave radar operates in the 76-81GHz band, with electromagnetic wavelengths (approximately 4mm) far larger than raindrops and snowflakes, giving it exceptional penetration capability against precipitation. However, the trade-off is that its angular resolution (approximately 1.8°) is far lower than lidar (approximately 0.1°), and it cannot provide rich semantic texture information.
• Camera: Significantly affected by visibility, reflections, and snow coverage. Snowfall and heavy rain reduce image contrast, water droplets and snowflakes on the lens cause local blurring, and snow coverage changes the visual features of road markings and curbs, causing significant performance degradation in vision-based lane detection and semantic segmentation algorithms.

It is precisely this complementarity and conflict between sensors that makes multimodal fusion the key to deploying autonomous driving in real weather conditions. The core idea of multimodal fusion is: when one sensor fails due to environmental factors, the system can dynamically adjust the confidence weights of each sensor, relying on other unaffected sensors to maintain overall perception capability. What Boreas provides is exactly the synchronized multi-sensor data covering extreme weather that is urgently needed to study this problem.

Boreas Dataset Hardware Configuration and Data Scale

Boreas's sensor suite represents a high-specification setup by industry standards, providing ample raw signals for multimodal research.

Sensor Configuration

• 128-beam Lidar: High-density point clouds capable of finely capturing three-dimensional environmental structure. 128 beams means 128 laser emitter/receiver channels in the vertical direction, providing several times the point cloud density compared to 64-beam or 32-beam lidars, with significantly enhanced detection capability for distant small objects (such as pedestrians).
• 360-degree Omnidirectional Radar: Boreas employs an omnidirectional scanning radar based on Frequency-Modulated Continuous Wave (FMCW) technology, achieving full-coverage through mechanically rotating antennas, providing omnidirectional, weather-resistant detection capability. Compared to traditional automotive millimeter-wave radars that typically have only a ±60° forward field of view limitation, omnidirectional radar can perceive the dynamic environment 360 degrees around the vehicle.
• 5MP Camera: High-resolution visual information providing rich pixel-level data for semantic understanding and texture feature extraction.
• Centimeter-level accuracy ground truth: Typically based on a combination of high-precision RTK-GNSS (Real-Time Kinematic differential positioning) and Inertial Navigation Systems (INS), providing high-accuracy reference benchmarks for localization and mapping tasks. This level of positioning accuracy is crucial for evaluating SLAM algorithms and mapping quality.

Annotation Data Volume

The dataset's annotation scale is equally impressive:

• 326,180 3D bounding boxes, covering three key traffic participant categories: cars, pedestrians, and cyclists. 3D bounding boxes include not only the spatial position (x, y, z) and dimensions (length, width, height) of objects but also heading angle, providing a complete geometric description for spatial understanding of targets.
• 7,111 annotated data frames.

These annotations are not only sufficient in quantity—more importantly, they are distributed across different weather and seasonal conditions, enabling models to learn the critical domain shift problem of "how the same road differs in snowy versus sunny conditions." Domain shift is one of the core challenges in machine learning, referring to differences between training data distribution and testing/deployment data distribution. The main technical approaches for addressing domain shift include domain adaptation, domain randomization, and self-supervised methods based on contrastive learning. Boreas's "same route, different weather" design provides perfect controlled experimental conditions for these research directions. For teams studying sensor robustness and cross-weather generalization capabilities, this represents scarce real-world samples.

Synchronized Multimodal Visualization with FiftyOne

Having massive multi-sensor data is only the first step—how to efficiently explore and verify this data is equally important. FiftyOne is an open-source machine learning data management and visualization tool developed by Voxel51, focusing on exploration, analysis, and quality management of computer vision and multimodal data. It supports unified management of multiple data types including images, video, and point clouds, providing interactive data browsing interfaces, model evaluation dashboards, and data quality analysis features.

Boreas loads natively in mcap format into FiftyOne, delivering a remarkably intuitive interactive experience. mcap is an open-source data container format developed by Foxglove, designed specifically for multi-sensor temporal data in robotics and autonomous driving. It supports efficient time indexing, multi-channel parallel reading, and streaming access, capable of synchronously storing heterogeneous data streams from different sensors (point clouds, images, radar scans, IMU, etc.) in a single file while maintaining precise timestamp alignment. Compared to the traditional ROS bag format, mcap offers better cross-platform compatibility and higher read performance, and has become one of the emerging standards for autonomous driving data storage.

Synchronized Browsing Across Three Modalities

In FiftyOne, users can drag along a single synchronized timeline to browse camera, lidar, and radar data. This means you can precisely align what three sensors "see" at the same moment—when snowflakes create overwhelming noise in the lidar point cloud, the "calm" radar image at the same instant forms a striking contrast.

Furthermore, the system supports:

• Real-time rendering of 3D bounding boxes on point clouds, providing intuitive observation of object detection results. Researchers can rotate and zoom the point cloud view to check whether annotations precisely enclose target objects, or to discover difficult annotation cases caused by snowflake occlusion.
• Bounding box projection onto camera and radar images, enabling cross-modal annotation consistency verification. This feature achieves 3D-to-2D coordinate transformation through inter-sensor extrinsic calibration matrices, helping to discover calibration drift or annotation errors.

This visualization capability is extremely valuable for debugging perception algorithms, discovering annotation errors, and understanding sensor failure modes.

Ready-to-Use Resources

Boreas is fully open, and researchers can get started immediately:

• Dataset address: Voxel51/boreas-multimodal on Hugging Face
• Interactive experience: Hugging Face Space harpreetsahota/boreas-multimodal, explore data directly without local deployment.

What Boreas Brings to Autonomous Driving Research

The true significance of Boreas lies not in how advanced its sensors are, but in the fact that it fills the data gap for adverse weather long-tail scenarios.

Autonomous driving systems performing well in sunshine is no longer news—the real challenge has always been "edge cases": blizzards, heavy rain, low visibility. In machine learning, these low-frequency but high-risk scenarios are known as the "long-tail distribution" problem—they occur at extremely low frequency in the data, yet their impact on system safety is critical. Having the same route repeatedly collected 44 times over a year essentially constructs a controlled-variable weather comparison experiment—the route, object categories, and road structure remain unchanged, with the only variation being weather and season. This provides ideal ground for studying the following problems:

1. Sensor fusion strategies: How to rely more on radar when lidar fails? This involves designing adaptive fusion architectures, such as attention mechanism-based dynamic weight allocation or uncertainty estimation-based sensor reliability assessment. The system needs to determine each sensor's current "health status" at runtime and adjust fusion strategies accordingly.
2. Point cloud denoising: How to distinguish real obstacles from snowflake noise? Current mainstream methods include statistical filtering (such as DSOR denoising), deep learning-based point-wise classification (such as WeatherNet), and temporal consistency-based motion filtering. The abundant snowy point cloud data provided by Boreas is a valuable resource for training and validating these denoising algorithms.
3. Cross-domain generalization: How can models trained on sunny days adapt to snowy conditions? Unsupervised Domain Adaptation (UDA) and test-time adaptation are currently active research directions aimed at improving model performance in new environments without requiring target domain annotated data.

For researchers committed to moving autonomous driving out of the "California sunshine" comfort zone, Boreas provides a valuable data asset that closely reflects the real world. As autonomous driving expands to more climatically diverse regions—from Nordic ice and snow to Southeast Asian monsoon rains—the value of open-source datasets like this one, covering all seasons with multiple modalities, will only continue to grow. It not only drives algorithmic progress but also provides benchmarks under adverse weather conditions for autonomous driving safety validation standards, helping the industry build more comprehensive safety assessment frameworks.

Key Takeaways