Multi-Sensor Kalman Fusion in Practice: A Complete Breakdown of a Drone Tracking System

From Detection to Tracking: The Evolution of an Open-Source Project

A developer recently shared on Reddit a major upgrade to his Shahed-136 drone detection system. The Shahed-136 is an Iranian-developed one-way attack drone (commonly known as a "kamikaze drone") with a wingspan of about 2.5 meters and a delta-wing configuration. Its low-altitude, low-speed, low-radar-cross-section flight characteristics pose unique challenges to traditional air defense systems. The Shahed-136's low observability makes it a classic problem for modern air defense: its engine noise is around 65 decibels (comparable to a motorcycle), and its radar cross section (RCS) is estimated to be on the order of 0.01 to 0.1 square meters—comparable to a large bird. Traditional phased-array radars are highly prone to missing it against complex ground clutter. For this reason, computer-vision-centric soft-kill detection approaches have received widespread attention, and the Shahed-136 has become one of the de facto standard scenarios for academia to evaluate small-target, low-speed tracking algorithms.

Previously, he built a real-time drone detector based on YOLOv8—a real-time object detection framework released by Ultralytics in 2023 that continues the YOLO series' design philosophy of "completing detection in a single forward pass." Since Joseph Redmon introduced the YOLO (You Only Look Once) series in 2016, it has consistently led the real-time detection field. Compared to its predecessors, YOLOv8 introduces a Decoupled Head and an Anchor-Free mechanism, achieving an excellent balance between accuracy and speed. Its model family, ranging from nano to extra-large, covers the full spectrum of needs from edge devices to server-side deployment. But what really taught him something was this reconstruction of the "tracking" component.

The project's core pain point is very typical: the original tracker used a Constant-Velocity Kalman Filter based on camera detection. The Kalman filter is a recursive Bayesian estimation algorithm proposed by Rudolf Kalman in 1960. Its core consists of two alternating steps: the Predict step uses the state transition matrix F and the process noise covariance Q to propagate the current estimate forward; the Update step fuses the new observation with the prediction by computing the Kalman gain K. The gain formula K = PH^T(HPH^T + R)^{-1} essentially represents a dynamic trade-off—when measurement noise R is small, it trusts the observation more; when the prediction uncertainty P is small, it trusts the model more—allowing the system to achieve theoretically optimal estimation under linear Gaussian conditions. This mechanism performs well when the target flies in a straight line, but the moment it encounters occlusion, glare, or a sharp turn—precisely the most critical moments for tracking—the target is lost. To address this, the author rebuilt a multi-sensor fusion system from scratch (sensor_fusion.py) and used it as a hands-on exercise for learning Kalman filtering in depth.

The value of this project lies not in showing off, but in revealing the true difficulty of Kalman filtering in engineering practice: how to handle asynchronous, noisy, multi-rate real-world data streams, rather than the clean single data source found in textbooks.

From Constant Velocity to Constant Acceleration: Teaching the Filter to "Turn"

The first key change the author made was upgrading the motion model from constant velocity (CV) to a Constant-Acceleration (CA) model, expanding the state vector from [x, y, vx, vy] to [x, y, vx, vy, ax, ay].

Why the CV Model Fails

The underlying assumption of the constant-velocity model is that the target won't change speed or direction. This assumption collapses the instant the target maneuvers—the Kalman filter either reacts too slowly or "overshoots" during sharp turns, predicting a path that deviates from the true trajectory.

In the field of target tracking, motion models form a spectrum from simple to complex: constant position (CP) → constant velocity (CV) → constant acceleration (CA) → interacting multiple model (IMM). Introducing the acceleration term gives the filter the ability to respond to turns. In actual engineering, the CA model absorbs higher-order maneuvering uncertainty by increasing the process noise covariance matrix Q. For more highly maneuverable targets, a more advanced approach is the Interacting Multiple Model (IMM) algorithm: it runs multiple motion models simultaneously (such as CV and CA), dynamically computing model probabilities based on each model's residuals and outputting a probability-weighted mixture. This handles the target's transitions between constant-velocity and maneuvering states more elegantly. The IMM algorithm was formally introduced by Blom and Bar-Shalom in 1988. Its core idea is to maintain a model probability vector μ and describe the target's switching probabilities between different maneuvering states via a Markov transition matrix. At each time step, IMM first performs a probability-weighted mixing of each model's state estimate (the interaction step); after each sub-filter independently completes its prediction and update, it then updates the model probabilities based on the likelihood function and fuses the outputs. IMM's computational complexity grows linearly with the number of models, and in engineering implementations, 2 to 4 models is usually appropriate, with typical combinations being CV+CT (constant turn) or CV+CA. For the Shahed-136 in this project, the CA model is already the watershed between "barely usable" and "truly reliable."

Multi-Sensor Fusion: Noise Characteristics Differences Are Key

The second change was upgrading the single-sensor architecture to a pluggable multi-sensor interface. The author added an add_external_measurement() interface, allowing RF (radio frequency), radar, or a second camera to all feed into the same filter.

A One-Size-Fits-All Covariance Won't Work

Architecturally, multi-sensor fusion is divided into three levels: data-level, feature-level, and decision-level. The Kalman filter implements a state-level fusion, whose advantage lies in its ability to precisely characterize each sensor's uncertainty using the measurement covariance matrix R. The most valuable insight in multi-sensor fusion is this: cameras and RF-type sensors have entirely different noise characteristics and sampling rates.

Camera: 30Hz sampling, low noise, good at precise angle/pixel position but lacks depth information
RF sensor: 5Hz sampling, high noise, good at all-weather range measurement but limited resolution

Here, the "noise" is described by the measurement covariance matrix R—the diagonal elements of the R matrix correspond to the variance of each measurement dimension and directly determine the magnitude of the Kalman gain: the larger R is, the less the filter trusts that sensor's readings and the more it relies on its own prediction; the smaller R is, the faster it responds to changes in observation. If the same measurement covariance is used for all sensors, the fusion effect is greatly diminished. The correct approach is to configure a separate covariance matrix for each sensor, allowing the filter to dynamically adjust weights based on the reliability of different sensors. The natural complementarity of these two types of sensors—cameras providing high-precision localization, RF providing continuity during occlusion—is precisely the core value that distinguishes multi-sensor fusion from single-source filtering.

The Hardest Nut to Crack: Out-of-Sequence Measurement Processing (OOSM)

The author admitted that the most difficult part of the entire project to implement correctly was Out-of-Sequence Measurement (OOSM) processing.

The Essence of the Problem

The OOSM problem is ubiquitous in distributed sensor networks, wireless transmission, and multi-processor systems. Its root causes are processing delays, transmission delays, and clock offsets among different sensors. When a reading from a slower sensor (such as a 5Hz RF sensor) arrives, the filter may have already advanced its time step based on faster camera data. At this point, you cannot simply overlay this "late" measurement directly—it corresponds to a state at some past moment, and directly fusing it would contaminate the current estimate.

The OOSM problem has a large body of academic research; systematic study began with Bar-Shalom et al.'s seminal 2002 paper. The main approaches include: Algorithm A (approximating the processing of late measurements using the current time step's Kalman gain, with a computational cost only twice that of a normal update step, suitable for scenarios where the delay is within 1 to 2 sampling periods, but introducing approximation error), Algorithm B (storing historical cross-covariance matrices for exact posterior correction, offering optimal accuracy but with storage requirements growing linearly with the maximum delay window—in multi-sensor, high-sampling-rate scenarios, the memory overhead may become a bottleneck), and the rewind-and-replay approach adopted by this project. In addition, approaches based on the Information Filter have a natural architectural advantage in asynchronous multi-sensor fusion due to their additive nature. In real-time embedded systems, it's usually also necessary to set a maximum delay tolerance window, discarding measurements that exceed the window to control resource overhead.

The Rewind-and-Replay Approach

The author ultimately implemented a "rewind-and-replay" mechanism:

During operation, the filter creates checkpoints for its state
When a late measurement appears, the filter rewinds to the nearest checkpoint
It then replays all measurements in strict chronological order, including this late piece of data

This approach guarantees that regardless of the order in which sensor data arrives, the final state estimate reflects the correct time sequence. Its cost is the need to maintain a historical state buffer, which places certain demands on memory and computing resources, but due to its intuitiveness and universality, it is a classic solution in engineering practice for handling asynchronous multi-source data.

Trajectory Prediction with Uncertainty

The author also implemented a predict_trajectory(horizon_s) function to project the track forward and draw the 1-σ uncertainty ellipse as it grows over time.

The 1-σ uncertainty ellipse is obtained from the position sub-block (a 2×2 matrix) of the Kalman filter's error covariance matrix P via eigenvalue decomposition: the square roots of the two eigenvalues correspond to the semi-axis lengths of the ellipse, and the corresponding eigenvectors determine the ellipse's orientation. Under a two-dimensional Gaussian distribution, the 1-σ ellipse contains the true state with a probability of about 39.4%, the 2-σ about 86.5%, and the 3-σ about 98.9%. As the number of prediction steps increases, the continuous accumulation of process noise causes the P matrix to grow and the ellipse to gradually expand, clearly illustrating the decay of the filter's confidence. Notably, the shape of the ellipse also reveals the "directional uncertainty" of tracking—the uncertainty along the direction of flight is often far greater than the lateral uncertainty, which has direct value for threat assessment and interception window calculations. In air defense command-and-control systems, actual interception calculations require convolving the kill radius of the friendly interceptor with the target's predicted position distribution to obtain the kill probability distribution (the Pk plot); the shape and size of the covariance ellipse directly determine the choice of optimal interception launch timing and guidance law parameters. In an incoming threat scenario, the 3-σ ellipse is also commonly used as a "worst-case" boundary for delineating safety zones.

Measured Data: How Much Value Does Sensor Fusion Really Add?

The author validated the fusion effect with controlled experiments. In a target-loss scenario (the camera loses the target for 1.8 seconds during a turn, while the RF sensor maintains tracking at a low rate with high noise), the root mean square error (RMSE) comparison of the three approaches is as follows:

Approach	RMSE
Camera + RF Fusion	3.36 px
Camera only	5.47 px
RF only	14.06 px

The error of the fusion approach is significantly lower than that of any single sensor, fully demonstrating the value of multi-sensor complementarity—the camera provides high-precision localization, while RF provides continuity during occlusion.

Validation on Real Benchmark Data

To avoid "self-entertainment," the author also used real thermal imaging footage from the Anti-UAV410 benchmark for cross-validation. Anti-UAV410 is a public benchmark dataset released in 2023 for counter-drone research, containing 410 video sequences and over 438,000 annotated frames. It covers two modalities—visible light and infrared thermal imaging—with scenes spanning urban, mountainous, sea-surface, and various other backgrounds, and it annotates occlusion states and target size changes. The core challenge of this dataset is that drone targets in thermal imaging often occupy only a few dozen pixels, with a median pixel size of only about 20×20—a classic small-target scenario that places special demands on the receptive field design of the feature extraction backbone network. Its evaluation protocol, in addition to the classic single-object tracking metrics, specifically introduces a "Re-Detection" subset evaluation—that is, forcibly resetting the tracker's state after occlusion to separately assess its ability to capture the target when it reappears, which is precisely the core capability dimension that this project's rewind-and-replay mechanism targets. The validation results show that RMSE is below 3px during normal flight, and the target is cleanly re-captured after real occlusion ends, rather than drifting off and being lost as before.

On the detection side, the fine-tuned YOLOv8s achieved mAP@50 of 99.5% on the shahed class. mAP@50 represents the mean AP across classes at an IoU (Intersection over Union) threshold of 0.5, and a 99.5% score means the detector has almost no missed detections or false positives at this threshold—however, given that the dataset has a single class and has undergone dedicated fine-tuning, the actual generalization significance of the high mAP still needs to be evaluated in conjunction with real-world validation. As the author says, the tracker is the more educationally valuable part of the entire project from an engineering standpoint.

Practical Lessons for Similar Projects

The most profound summary of this project is this: "Kalman filtering 'clicks' much faster once you're forced to deal with asynchronous, noisy, multi-rate data rather than a clean single data stream."

For developers working on target tracking and sensor fusion, this open-source project offers several directly applicable practical takeaways:

The choice of motion model should match the target's maneuvering characteristics (CV→CA→IMM, with complexity matched to target maneuverability)
The success or failure of multi-sensor fusion largely depends on covariance tuning, and each sensor needs to be modeled independently
OOSM handling is not optional but an essential capability for real multi-source systems
Validate with real benchmark data, not just simulation

The author provides a standalone, reproducible demo that runs without video or a model (simulate_fusion_demo.py), and interested developers can experience the fusion logic directly. The project is open-sourced on GitHub, and the author has expressed a willingness to discuss the details of the OOSM replay logic and covariance tuning in depth.

Project URL: github.com/alexandre196/Drone-Shahed-AI-Multi-Sensor-Tracker