New Breakthrough in Consumer-Grade Sonar: The HaCKRS Dual-Modality Fish School Monitoring Dataset

Can Consumer-Grade Sonar Take Up the Banner of Ecological Monitoring?

Underwater ecological monitoring has long relied on optical imaging, but in dark or murky waters, conventional cameras are almost useless. Although imaging sonar can penetrate these obstacles, it is difficult to popularize due to its high price. Now, a new study turns its attention to consumer-grade forward-looking sonar—Garmin LiveScope—and releases the first fish school monitoring dataset pairing RGB and sonar, HaCKRS, attempting to answer a key question: can the noisy images of inexpensive sonar support reliable automated fish school monitoring?

HaCKRS Dataset: Paired Observations and Calibration Information

According to the paper, the HaCKRS (Huntington and Caltech Koi RGB and Sonar) dataset pairs imaging sonar recordings from Garmin LiveScope with top-down RGB videos of koi, containing more than 105,000 time-aligned frame pairs, with a total duration of about 3.3 hours, covering four distance and orientation configurations. The dataset provides per-frame metadata, sonar geometry and coordinate information, synchronized data for the four configurations, and planar registration for three horizontal configurations. You may not have noticed that the dataset does not provide task-specific fish body annotations, but instead focuses on paired observations and calibration, aiming to provide a foundation for the subsequent development of detection, tracking, and counting methods.

This design approach differs from common annotated datasets. The researchers clearly hope to first solve the underlying problem of "data alignment," allowing the community to build annotations and algorithms on this basis. For consumer-grade sonar, images are noisy and limited in resolution, making direct annotation extremely costly; providing precise synchronization and geometric information instead lowers the barrier for subsequent development.

The Meaning of Cross-Modal Registration and Planar Registration

Paired observations refer to data collected at the same moment and in the same scene by two different sensors, here RGB video frames and sonar frames. Because the imaging principles, viewpoints, resolutions, and coordinate systems of the two sensors are completely different, spatial correspondence must be established through calibration. Sonar geometry and coordinate information describe the sonar's installation position, orientation, beam angle, and range, while planar registration aligns the sonar image and the RGB image to the same coordinate system on the horizontal plane. Specifically, the researchers use known calibration objects or scene features to estimate the transformation matrix between the two sensors, so that the position of the same target in the two images can be mapped to each other. This registration is the foundation of cross-modal learning: only after alignment can high-quality RGB images be used to interpret and annotate low-quality sonar images, or models be trained to transfer between the two modalities. The dataset provides registration for three horizontal configurations, meaning researchers can verify algorithm robustness under different installation conditions.

Opportunities and Challenges of Consumer-Grade Sonar

The advantage of consumer-grade forward-looking sonar lies in its much lower cost than research-grade imaging sonar, enabling more researchers and enthusiasts to conduct underwater observations. However, its image noise also brings difficulties in annotation and automated processing. The release of HaCKRS is precisely intended to systematically study these challenges. Through paired high-quality RGB videos, researchers can use optical images to understand targets in sonar images, and then develop robust detection and tracking algorithms.

The paper explicitly states that the goal of this dataset is to support the development of fish school detection, tracking, and counting methods, and to explore the potential of consumer-grade sonar in ecological monitoring. This means that in the future we may be able to use low-cost devices to estimate fish school numbers in murky waters, providing new tools for fisheries management and ecological research.

The Difference Between Forward-Looking Sonar and Imaging Sonar

Forward-looking sonar is an active acoustic sensor that emits sound pulses in the detection direction and receives echoes, generating two-dimensional images through the time delay and intensity of the echoes. Unlike side-scan sonar, the beam of forward-looking sonar points directly ahead, making it suitable for installation on small boats or underwater robots for navigation and obstacle avoidance. Garmin LiveScope is a representative of this type of consumer-grade product; it was originally designed for fishing enthusiasts to observe fish schools and terrain ahead in real time. Imaging sonar broadly refers to systems capable of generating high-resolution acoustic images. Research-grade equipment usually has wider frequency bands, higher dynamic range, and more precise beamforming, but can cost tens of thousands to hundreds of thousands of dollars. Consumer-grade forward-looking sonar provides usable imaging capability at a price of several thousand dollars, at the cost of noisier images, lower resolution, and limited update rates. Understanding this hardware positioning helps grasp the practical significance of the HaCKRS dataset: it tests how far automated algorithms can go under low-cost, low-signal-to-noise conditions.

Open Data Advances Underwater Vision Research

The release of HaCKRS continues the tradition of open datasets in the computer vision field. Similar to large datasets such as ImageNet, it provides a standardized benchmark that allows different methods to be compared fairly. But unlike general vision datasets, HaCKRS focuses on cross-modal pairing of underwater sonar and optics, which provides new test scenarios for multimodal learning, domain adaptation, and weakly supervised learning.

At present, the dataset has been made public, including synchronized data and registration information, and researchers can use it to carry out tasks such as fish school detection, tracking, and counting. As more teams use this dataset, the actual effectiveness of consumer-grade sonar in ecological monitoring will be evaluated more clearly.

The Role of Multimodal Learning, Domain Adaptation, and Weakly Supervised Learning in Underwater Scenarios

Multimodal learning studies how to fuse or align information from different sensors, for example using both RGB and sonar images to improve detection accuracy. Domain adaptation focuses on maintaining performance when a model is transferred from one data distribution to another, which here can refer to adapting from the optical image domain to the sonar image domain. Weakly supervised learning uses incomplete, imprecise, or inexpensive annotations for training, which fits precisely with HaCKRS's design of not providing fish body annotations: researchers can use detection results in RGB images as weak labels for sonar images, or use the pairing relationship for self-supervised pretraining. Together, these methods reduce dependence on manual annotation and make automated monitoring with consumer-grade sonar more feasible. The paired structure of HaCKRS provides a natural experimental platform for these learning paradigms, because the same scene contains both high-quality and low-quality observations, making it convenient to design comparative experiments and transfer strategies.

Conclusion

The HaCKRS dataset fills the data gap for consumer-grade sonar in the field of ecological monitoring, and its paired design and calibration information lay a solid foundation for subsequent research. Although sonar image noise remains a major challenge, open data and community collaboration are expected to accelerate algorithm innovation and make low-cost underwater monitoring a reality.

One-line Summary

The first consumer-grade sonar and RGB paired fish school dataset, HaCKRS, is released, exploring low-cost underwater ecological monitoring.

Paragraph Summary

The article introduces the HaCKRS dataset, the first dataset to pair consumer-grade forward-looking sonar Garmin LiveScope with top-down RGB videos of koi. It contains more than 105,000 time-aligned frame pairs, with a total duration of about 3.3 hours, covering four distance and orientation configurations. The dataset does not provide task-specific fish body annotations, but instead focuses on paired observations, sonar geometry and coordinate information, synchronized data, and planar registration, aiming to provide a foundation for the subsequent development of detection, tracking, and counting methods. The article analyzes the characteristics of consumer-grade sonar—low cost but high noise—points out that HaCKRS provides a benchmark for systematically studying these challenges, and discusses its potential value in multimodal learning, domain adaptation, and weakly supervised learning.