Acoustic Detection of Multi-Rotor UAVs: Fusing Handcrafted Features with Pre-Trained Embeddings

Introduction: The Acoustic Path to UAV Detection and Evaluation Pitfalls

Small multi-rotor drones produce distinctive acoustic signatures during flight, enabling passive detection at very low cost using microphone arrays. Public benchmarks often report extremely high recognition accuracy, but these results are typically built on randomly split datasets—segments from the same long recording may appear in both training and test sets, leading to data leakage and inflated performance. A new study takes DroneAudioDataset as its subject, systematically exposes the severity of this evaluation bias, and proposes a stacked ensemble that fuses handcrafted acoustic descriptors with pre-trained YAMNet embeddings, achieving robust detection performance under a strict source-disjoint protocol.

Dataset and Evaluation Protocol: Leakage-Aware Source-Disjoint Design

DroneAudioDataset contains 11,704 one-second audio clips, of which 1,332 drone clips come from 257 five-second chunks cut from 7 recordings, and 96.4% of the non-drone clips originate from the ESC-50 sound dataset. The study compares two splitting schemes: random splitting and a grouped source-disjoint protocol (five-fold, three repetitions). The latter ensures that clips from the same original recording never appear across training/test sets, thereby eliminating data leakage. Experiments show that random splitting inflates the accuracy of four models by 1.89 to 3.22 percentage points—a gap sufficient to mislead model selection in benchmark comparisons.

Data Leakage and Source-Disjoint Protocols

In machine learning evaluation, data leakage refers to information overlap between training and test sets, causing the model to have "seen" data it should not have during testing, thereby overestimating generalization performance. For audio datasets, a common form of leakage is when the same long recording is split into multiple short clips and randomly assigned to training and test sets, allowing the model to easily memorize recording-specific background noise or channel characteristics. A source-disjoint protocol requires splitting data at the level of original recordings (sources) as the minimum unit, ensuring that all clips from the same source appear in only one set. Grouped five-fold cross-validation is a common implementation: sources are randomly divided into five groups, each time one group is used as the test set and the other four for training, repeated multiple times to assess stability. This design more realistically simulates model performance when facing entirely new recording environments.

Feature Engineering: Complementarity of Handcrafted Descriptors and Pre-Trained Embeddings

Each clip is described in two ways: first, 57 handcrafted acoustic descriptors, reduced via a hybrid minimum Redundancy Maximum Relevance (mRMR) and ReliefF selection or NSGA-II multi-objective search; second, 1024-dimensional embeddings output by the pre-trained YAMNet network. Random Forest and XGBoost are trained on the two views respectively, then fused via a stacked ensemble. Hybrid selection matches the accuracy of all 44 baseline descriptors with just 16 to 18 features, while NSGA-II retains 4 to 6 features but with slightly lower precision. This indicates that a carefully selected small set of handcrafted features already captures most discriminative information, while YAMNet embeddings provide additional semantic supplementation.

mRMR and ReliefF Feature Selection

Minimum Redundancy Maximum Relevance (mRMR) is a filter-based feature selection method that simultaneously optimizes two objectives: maximizing the mutual information between features and class labels (relevance) and minimizing the mutual information among features (redundancy). ReliefF is an instance-based weighting algorithm that evaluates a feature's ability to distinguish different classes by randomly selecting samples and comparing their nearest neighbors. Combining the two (hybrid selection) balances relevance and redundancy, yielding a more compact and informative feature subset. NSGA-II (Non-dominated Sorting Genetic Algorithm II) is a multi-objective optimization algorithm that can simultaneously optimize feature count and classification error rate, producing a set of Pareto-optimal solutions for trade-off selection. These methods help reduce feature dimensionality while maintaining discriminative power, improving model efficiency and interpretability.

Performance Results: Stable Gains and Computational Cost

Under the grouped protocol, the dual-view stacked model achieves an average accuracy of 95.29% (standard deviation 1.62 percentage points across 15 test folds), specificity of 98.29%, an area under the ROC curve of 0.990, and correctly classifies 98.78% of unseen ESC-50 source clips as non-drone. Adding YAMNet embeddings improves the accuracy of the handcrafted view by 1.11 percentage points and performs better on 12 of the 15 folds—a small but consistent gain. In terms of computational cost, inference takes 51.0 milliseconds per clip on a single CPU thread, meeting real-time detection requirements.

Stacked Ensembles and YAMNet Embeddings

A stacking ensemble is a model fusion strategy that trains a meta-learner to integrate predictions from multiple base models. In this study, the handcrafted feature view (after feature selection) and the YAMNet embedding view are each trained with Random Forest and XGBoost, and the meta-learner (typically logistic regression or a simple model) learns how to weight or combine the outputs of these base models to improve final performance. YAMNet is an audio event classification network pre-trained by Google on AudioSet; its 1024-dimensional embedding layer output captures rich acoustic semantic information and provides complementary representations even without fine-tuning on drone data. This dual-view stacking allows the model to leverage both the conciseness of handcrafted features and the semantic richness of deep embeddings, achieving stable gains at a moderate increase in computational cost.

Conclusions and Implications

The study conveys two key messages: first, on datasets like DroneAudioDataset derived from a small number of long recordings, source-disjoint evaluation is essential for fair benchmarks; otherwise, random splitting systematically overestimates performance. Second, the fusion of handcrafted acoustic features and pre-trained embeddings yields stable, small gains at moderate computational cost—a complementary strategy worth promoting in embedded acoustic monitoring systems. For practical deployment, the study recommends prioritizing the rigor of the evaluation protocol before pursuing increased model complexity.

One-line Summary

Study reveals random splitting overestimates performance in drone acoustic detection; proposes source-disjoint protocol with stacked ensemble of handcrafted features and YAMNet achieving 95.29% accuracy.

Paragraph Summary

Using DroneAudioDataset, this paper systematically analyzes data leakage in drone acoustic detection evaluation. Random splitting inflates accuracy by 1.89 to 3.22 percentage points, while a grouped source-disjoint protocol ensures fair benchmarks. The study proposes a stacked ensemble fusing 57 handcrafted acoustic descriptors (selected via mRMR-ReliefF or NSGA-II) with 1024-dimensional pre-trained YAMNet embeddings, achieving an average accuracy of 95.29%, AUC 0.990, and specificity of 98.29% for unseen ESC-50 source clips under a strict protocol. Adding YAMNet embeddings improves the handcrafted view's accuracy by 1.11 percentage points, consistently better on 12 of 15 folds. Single CPU thread inference takes 51.0 milliseconds, meeting real-time detection needs.