PG-FM: Keeping RGB-T Detectors Robust Under Signal Corruption

RGB-thermal (RGB-T) object detection achieves more reliable perception in complex scenes by fusing visible-light and thermal signals. However, when the signal from either modality is corrupted, detector accuracy drops significantly. Traditional restoration-based methods can improve input quality, but they introduce extra latency and the downstream accuracy gains are inconsistent. The arXiv paper "Shifting Focus by Reliability: Geometric Fusion Modulation for Robust RGB-T Detection" proposes PG-FM (Prompt-Guided Feature Modulation), a lightweight adaptation scheme that skips input restoration and instead directly adapts the detector to changes in signal quality.

Problem: Detection Degradation Under Corrupted Signals

RGB-T detectors rely on the complementarity of visible light and thermal imaging, but in real-world scenes sensor noise, harsh weather, or hardware failure can cause the signal quality of one modality to plummet. Most existing methods use image restoration or enhancement to recover the input before feeding it to the detector. Such methods not only introduce significant inference latency, but the restored images are not necessarily beneficial for downstream detection—the restoration objective and the detection objective are misaligned, leading to unstable accuracy gains.

Through controlled corruption sweep experiments, the authors found that under near-clean data conditions, different types of corruption affect the detector similarly; but as corruption severity increases, the effects of different corruption types begin to diverge. This means the detector needs an adaptation mechanism that can perceive corruption type and severity and dynamically adjust for each input.

mAP50 (mean Average Precision at IoU threshold 0.5) is a core evaluation metric in object detection. It computes the mean of average precision (AP) across all classes at an intersection-over-union (IoU) threshold of 0.5. IoU measures the overlap between the predicted box and the ground-truth box; a threshold of 0.5 means a prediction counts as correct when the overlap area exceeds half. mAP50 places relatively lenient demands on localization accuracy and is commonly used to measure whether a detector can "find the object," whereas mAP50-95 further requires higher localization precision. In RGB-T corruption scenarios, degraded signal quality mainly affects feature extraction and fusion, leading to more missed and false detections, so a drop in mAP50 directly reflects degraded detector robustness.

PG-FM: Prompt-Guided Feature Modulation

The core idea of PG-FM is: instead of restoring the corrupted input, let the detector modulate features according to the estimated corruption type and severity. Specifically, PG-FM estimates the corruption type and severity for each modality, generates corresponding prompt vectors, and uses these prompts to modulate the fused features. This modulation is input-dependent and can track the diverse corruptions that appear at test time.

Compared with conventional parameter-efficient fine-tuning (PEFT), PG-FM's advantage is that its correction is not input-agnostic. PEFT typically learns a fixed set of adaptation parameters and cannot cope with corruptions that change dynamically at test time. Through prompt guidance, PG-FM achieves customized adaptation for each input, thereby maintaining detection accuracy under corruption conditions.

Parameter-efficient fine-tuning (PEFT) is a class of techniques that adapt a pretrained large model to downstream tasks by updating only a small number of parameters; common methods include Adapter, LoRA, Prefix Tuning, and others. The core idea is to freeze most of the backbone's parameters and inject task-specific information via lightweight modules or low-rank decomposition, greatly reducing training cost while preserving pretrained knowledge. However, the adaptation parameters learned by conventional PEFT are static and input-agnostic, unable to adjust dynamically based on the quality of the input signal at test time. PG-FM borrows PEFT's lightweight adaptation philosophy but uses a prompt-vector generation mechanism so that the modulation coefficients vary with the input corruption type and degree, thereby overcoming the limitations of static adaptation.

Experimental Results: Significant mAP50 Gains

Under controlled RGB-T corruption simulation, PG-FM preserves performance on clean datasets while improving mAP50 under corruption by up to 20.8 points on the FLIR dataset and up to 19.9 points on the KAIST dataset. More importantly, PG-FM transfers to unmodified, naturally captured external RGB-T datasets without retraining, demonstrating strong generalization ability.

This result shows that directly adapting the detector to changes in signal quality is more efficient and more effective than the restore-then-detect route. PG-FM's lightweight nature also makes it suitable for real deployment, avoiding the latency issues of restoration methods.

FLIR and KAIST are two widely used public datasets in the RGB-T object detection field. The FLIR dataset is collected with FLIR thermal cameras and contains aligned visible-light and thermal images, mainly annotated for classes such as pedestrians and vehicles, with scenes dominated by daytime and nighttime urban roads. The KAIST dataset is larger, containing synchronized visible-thermal pairs from multiple sensors, covering diverse scenes such as daytime, nighttime, campus, and streets, with annotated classes including pedestrians, cyclists, vehicles, and more. The complementarity of these two datasets lies in: visible-light images provide texture detail, while thermal imaging is unaffected by lighting, making them suitable for evaluating multimodal fusion detectors under complex environments. The magnitude of PG-FM's gains on these two datasets verifies its cross-dataset effectiveness.

Insight: From Restoring Inputs to Adapting Models

PG-FM's work offers a new direction for robust RGB-T detection: rather than pursuing perfect input restoration, let the model learn to focus on reliable information within signals of uneven quality. This strategy of "shifting focus by reliability" echoes the paper's title, "Shifting Focus by Reliability." For real-world RGB-T systems, PG-FM promises to relax demanding requirements on sensor hardware and improve usability in harsh environments.

In the future, how to estimate corruption type and severity more accurately, and how to extend PG-FM to more modalities and tasks, are directions worth exploring.

One-line Summary

PG-FM replaces input restoration with prompt-guided feature modulation, letting RGB-T detectors dynamically adapt to signal corruption, with mAP50 gains of up to 20.8 points.

Paragraph Summary

PG-FM is a lightweight adaptation method for RGB-T object detection, aimed at solving the sharp drop in detection accuracy when the signal from either modality is corrupted. Traditional restoration-based methods introduce extra latency and unstable downstream accuracy gains, whereas PG-FM abandons input restoration and instead generates prompt vectors based on the estimated corruption type and severity for each modality, applying input-dependent modulation to the fused features. Under controlled corruption simulation, PG-FM improves mAP50 by up to 20.8 and 19.9 points on the FLIR and KAIST datasets respectively, while preserving clean-data performance and transferring to naturally captured external datasets without retraining. This work shows that letting the model dynamically focus by reliability is more efficient and effective than restoring inputs, offering a new direction for robust multimodal perception.