Deformable DETR's Convergence Speedup Not Working? A Parameter Scaling Experiment Reveals the Truth

A Practitioner's Puzzling Discovery

In the field of object detection, DETR (Detection Transformer) has revolutionized the traditional Anchor- and NMS-based detection pipeline with its end-to-end detection paradigm since its introduction in 2020. Traditional object detection pipelines rely on the Anchor mechanism to preset a large number of candidate boxes, then filter them through classification and regression branches, and finally use NMS (Non-Maximum Suppression) to remove overlapping predictions. This pipeline involves a large number of hand-designed hyperparameters (such as anchor size ratios, NMS thresholds, etc.), resulting in high engineering complexity. DETR redefines detection as a set prediction problem, leveraging the Transformer's global attention mechanism and the Hungarian Matching algorithm to directly output a fixed number of predictions, completely eliminating anchor generation and NMS post-processing steps, achieving truly end-to-end training and inference.

However, DETR's most criticized issue is its extremely slow convergence — the original paper required training for up to 500 epochs to achieve ideal performance. This is precisely why Deformable DETR was developed, claiming to improve training convergence speed by approximately 10x.

But recently, a Reddit user raised an intriguing question: in their own experiments, Deformable DETR didn't seem to deliver the expected convergence speedup. Their experimental setup was: using a training set of 5,600 images, scaling both DETR and Deformable DETR down to approximately 2 million (2M) parameters, and training each for 70 epochs. The result was that both models achieved only about 0.2 mAP50 on the test set.

This result is indeed surprising, and it raises a topic worth exploring in depth: Where does Deformable DETR's convergence advantage actually come from? And why does this advantage disappear under certain experimental conditions?

The Core Mechanisms Behind Deformable DETR's Faster Convergence

To understand this problem, we first need to clarify the core mechanisms that enable Deformable DETR's faster convergence.

Sparse Attention Replaces Global Attention

The original DETR uses standard Global Attention, where each query must compute attention weights with all spatial positions on the feature map. The computational complexity of the global self-attention mechanism in a standard Transformer is O(N²), where N is the sequence length. On image feature maps, N equals the spatial resolution H×W. For a typical feature map (e.g., 32×32 = 1,024 positions), the attention matrix is already quite large.

The more critical issue lies at the optimization level: during the initial stages of training, the Softmax-normalized attention weights approach a uniform distribution of 1/N, meaning each query assigns nearly equal attention to all positions and cannot effectively focus on target regions. The model must undergo a lengthy gradient update process to gradually shift attention from "looking everywhere" to "looking at the right places" — this is the fundamental reason why DETR requires 500 epochs.

Deformable DETR introduces the Deformable Attention Module, drawing inspiration from Deformable Convolution. Specifically, for each query, the model first determines a Reference Point, then predicts K sampling offsets through a linear projection layer. These offsets define the specific positions around the reference point that need attention. The sampling process uses bilinear interpolation to extract feature values from corresponding positions on the feature map, which are then combined via a weighted sum with learned attention weights. Since K is typically set to 4, each query interacts with only 4 positions rather than the entire feature map, reducing computational complexity from O(N²) to O(NK), i.e., O(N) linear complexity. More importantly, the offsets are learnable, giving the model a spatial prior from the very beginning — focusing only on local regions near the reference point. This drastically reduces the search space that attention needs to "explore," allowing the model to learn meaningful attention distributions much faster.

The Boost from Multi-Scale Feature Fusion

Deformable DETR natively supports multi-scale feature fusion, enabling better handling of objects of different sizes, which is particularly important for small object detection. Multi-scale feature fusion is a classic strategy in object detection, with FPN (Feature Pyramid Network) being the most representative example. Objects of different scales are more easily detected on feature maps of different resolutions: large objects are better suited for low-resolution (high-semantic) features, while small objects benefit from high-resolution (high-detail) features. The original DETR uses only a single-scale feature map (typically the 1/32 resolution from the backbone's last layer), limiting its small object detection capability. Deformable DETR uses Multi-Scale Deformable Attention, allowing each query to simultaneously sample from feature maps at multiple resolutions without explicitly constructing a feature pyramid, achieving adaptive cross-scale feature fusion. This not only improves small object detection performance but also provides richer gradient signals that contribute to faster convergence.

It's important to note that the convergence advantage reported in the paper was validated on large-scale datasets like COCO with standard parameter configurations. Once you deviate from these premises, the conclusions may not hold.

Three Reasons Why the Convergence Advantage Disappeared

Returning to this user's experiment, where both models achieved only 0.2 mAP50 with no visible convergence difference, the likely reasons are as follows.

Overly Aggressive Parameter Compression

Compressing the model to 2M parameters is an extremely aggressive approach. The DETR family's performance is highly dependent on the expressive power of the backbone network and Transformer. Model Capacity refers to the complexity of the function family a model can represent, which directly determines the complexity of data distributions the model can fit. DETR-R50 (using a ResNet-50 backbone) has approximately 41 million parameters, while Deformable DETR-R50 has approximately 40 million. Compressing to 2 million parameters means retaining only about 5% of the original parameter count, severely weakening both the feature extraction layers and the Transformer encoder-decoder.

In this severe underfitting state, the model cannot even learn basic feature representations adequately, let alone demonstrate architecture-level convergence optimizations. This is analogous to two people trying to build houses with extremely crude tools — the subtle differences between tools become irrelevant when the overarching problem is that the tools are simply inadequate. Both models are constrained by the same bottleneck of insufficient model capacity, so architectural convergence advantages naturally cannot manifest.

Insufficient Training Epochs

70 epochs is still too few for the DETR family. Even for Deformable DETR, the official recommendation is 50 epochs, and that's with full parameter count and well-tuned hyperparameters. A model with drastically compressed parameters has inherently limited learning capacity, extracting useful information from data less efficiently per epoch. 70 epochs may be far from convergence.

Small Dataset and Lack of Pretraining

5,600 images constitutes a small-to-medium scale dataset. The DETR family typically requires an ImageNet-pretrained backbone as a foundation. If the backbone is also scaled down and pretrained weights are not used, the learning difficulty increases dramatically. Pretrained weights provide the model with a solid initial feature representation capability. Without this foundation, the model must simultaneously learn low-level visual features and high-level detection logic from random initialization — a nearly impossible task with limited data.

Practical Advice for Using DETR on Custom Datasets

Maintain Reasonable Model Capacity

When validating architectural advantages, use configurations as close to the original paper's standard settings as possible. If lightweight models are needed, consider knowledge distillation or purpose-built small model architectures rather than simply scaling down parameters proportionally. Knowledge Distillation is a model compression technique proposed by Hinton et al. in 2015. The core idea is to use a trained large Teacher model to guide the training of a smaller Student model. The student model learns not only from ground-truth labels (hard labels) but also from the teacher model's output probability distributions (soft labels), which contain rich "dark knowledge" such as inter-class relationships. In object detection scenarios, distillation can be applied at multiple levels including classification logits, bounding box regression, intermediate feature maps, and even attention maps. Compared to simply reducing parameter count, knowledge distillation can maintain a smaller model size while preserving as much of the large model's performance as possible, making it a more scientifically sound path to lightweight models.

Make Full Use of Pretrained Weights

Always use backbone networks pretrained on large-scale datasets. This is especially critical in data-limited scenarios, as it both accelerates convergence and improves final accuracy.

Judge Convergence Through Loss Curves

Don't judge convergence solely by final mAP — also observe the trends of loss curves and validation metrics throughout training. If the curves are still in a rapid improvement phase, horizontal comparisons are not fair.

Consider Newer DETR Variants

If convergence speed is a core pain point, consider subsequent improved versions such as DINO, DAB-DETR, and DN-DETR, which further optimize query design and denoising training strategies. Specifically, DAB-DETR (Dynamic Anchor Boxes DETR) directly parameterizes Object Queries as dynamic anchor box coordinates (x, y, w, h), giving queries explicit spatial meaning and avoiding the learning difficulties caused by the ambiguous query semantics in the original DETR. DN-DETR (Denoising DETR) adds noise to ground-truth box coordinates during training, then has the model learn to recover from noisy coordinates to ground-truth boxes. This Denoising Training provides more stable gradient signals for Hungarian Matching, significantly accelerating convergence. DINO combines multiple techniques mentioned above and introduces contrastive denoising training and mixed query selection strategies. On COCO, it achieves performance comparable to DETR's 500-epoch results in just 12 epochs, making it one of the current benchmarks in the DETR family.

Conclusion

This experiment reveals an important lesson: Performance advantages reported in papers often depend on specific experimental conditions, and blindly reproducing them outside those conditions can easily lead to misleading conclusions. Deformable DETR can indeed accelerate convergence, but this advantage only manifests when model capacity is sufficient, pretrained weights are used, and training is adequate.

When parameters are compressed to 2M without pretrained weight support, the bottleneck shifts from "convergence speed" to "representational capacity," rendering any architecture-level convergence optimizations ineffective. This also reminds us that when conducting model comparison experiments, we must ensure proper variable control to avoid irrelevant factors masking the phenomena we actually want to observe.