SL-Ground: How Sentence-Level Grounding Breaks Through the Bottleneck of Fine-Grained Open-Vocabulary Detection

Open-vocabulary object detection (OVOD) faces a core challenge: models must recognize categories never seen during training. Mainstream methods are built on the phrase grounding framework — text descriptions are decomposed into tokens, object queries are aligned with these tokens, and class scores are recovered through token-to-class aggregation. This design performs robustly in general scenarios, but when target categories are defined by complete compositional descriptions — such as "2018 Porsche 911 Carrera" — isolated token evidence becomes a bottleneck.

From Tokens to Sentences: An Overlooked Scoring Unit Problem

The researchers' observation hits the mark: grounding-based detectors build class scores on fragmented token matching, where each object query aligns only with individual tokens, and fine-grained semantics are lost before aggregation even occurs. For hierarchical labels like "vehicle type—brand—model," token-level scoring cannot distinguish the semantic gap between "Porsche" and "Porsche 911," because the two share nearly identical token sets.

SL-Ground's alternative is straightforward: encode each category description as a complete sentence-level representation and let object queries score directly against this holistic representation, rather than detouring through tokens. This change may seem simple, but it alters how the model "sees" text — shifting from puzzle-like fragment matching to whole-sentence understanding of compositional descriptions.

Open-vocabulary object detection (OVOD) refers to the setting where models see only a subset of categories during training but must detect arbitrary text-described target categories at inference. Its core challenge is mapping text semantics to visual regions. Mainstream methods adopt the phrase grounding framework: category names or descriptions are split into tokens, cross-modal attention aligns object queries with tokens, and token scores are aggregated to produce class confidence. This design relies on the flexibility of token-level matching, but under compositional descriptions (e.g., "brand + model + year"), isolated tokens cannot capture holistic semantics, causing fine-grained class scores to be distorted.

Prompt chunking is a common strategy for token-level methods to handle long text descriptions: since detectors typically limit the number of tokens each object query can align with, long descriptions are split into multiple short segments that are scored separately and then fused. While this covers all tokens, it introduces extra inference overhead, and semantic fragmentation between segments may weaken compositional consistency. SL-Ground avoids the need for chunking through whole-sentence encoding, enabling the model to process complete descriptions in one pass, simplifying deployment and reducing engineering complexity on long-tail categories.

Multi-Granularity Benchmark: The Three-Tier Test of Stanford Cars

To systematically evaluate fine-grained recognition capability, the authors constructed a multi-granularity Stanford Cars benchmark covering three levels: vehicle type, brand, and specific model. This design allows performance degradation points to be precisely located.

The results are compelling. At the finest model granularity, SL-Ground raises AP from 14.5 (token-level variant, with matched training candidate sampling) to 70.6; with full-vocabulary training, AP further reaches 86.0. Meanwhile, the model maintains open-vocabulary capability and requires no repeated prompt chunking — a common compromise of token-level methods when handling long descriptions.

You may not have noticed that the jump from 14.5 to 86.0 is not a marginal improvement from hyperparameter tuning, but a direct manifestation of fine-grained semantics being preserved after switching the scoring unit. On FG-OVD, SL-Ground likewise achieves the best average AP among compared methods, indicating its gains are not limited to a single dataset.

Stanford Cars is a dataset containing 196 vehicle categories, each defined by a "brand—model—year" combination, naturally possessing a hierarchical label structure. The multi-granularity benchmark constructed in the paper divides it into three levels — vehicle type (e.g., sedan, SUV), brand (e.g., Porsche), and specific model (e.g., 2018 Porsche 911 Carrera) — to diagnose the model's recognition capability at different semantic granularities. This layered evaluation reveals the performance collapse of token-level methods at fine-grained levels, because finer categories share more tokens, making differentiation difficult through token aggregation alone.

Why Sentence-Level Grounding Suits Fine-Grained OVOD Better

The fundamental limitation of token-level scoring is an information bottleneck: aggregation occurs after scoring, and fine-grained differences are already diluted before aggregation. Sentence-level representations front-load compositional semantics into the encoding stage, so alignment between queries and descriptions happens within the full semantic space. This explains why model granularity benefits the most — the deeper the hierarchy, the more pronounced the gap between compositional descriptions and isolated tokens.

Another practical advantage is engineering simplicity. Avoiding prompt chunking means less inference overhead and a more direct deployment path — a point that cannot be overlooked for real-world systems handling large numbers of long-tail categories.

FG-OVD is a benchmark dataset for fine-grained open-vocabulary detection, focused on distinguishing fine-grained subcategories within the same broad class (e.g., different car models, different bird species). Its evaluation metric is average precision (AP), requiring models to accurately localize and classify unseen fine-grained categories. SL-Ground achieves the best average AP among compared methods on this benchmark, demonstrating that sentence-level scoring units are not only applicable to the automotive domain but also generalize across broader fine-grained scenarios, offering new ideas for text-side design in open-vocabulary detection.

Limitations and Outlook

Current evaluations focus on the automotive domain and the FG-OVD benchmark. Whether sentence-level encoding remains equally robust under more open and noisier description distributions remains to be verified. The computational cost of sentence-level representations compared to token-level methods is also not fully elaborated in the abstract. Nevertheless, this work points to a clear direction: the bottleneck of fine-grained open-vocabulary detection may not lie in the detector itself, but in the choice of text-side scoring unit.

For practitioners building fine-grained recognition systems, SL-Ground offers a signal worth noting — when category definitions trend toward compositional descriptions, re-examining text encoding granularity may be more effective than stacking detection heads.