[KongchangAI]
· 1 min read· 841 words

279M Text Ranker Beats 2B/7B Vision-Language Models: A New Breakthrough in Urdu GUI Grounding

279M Text Ranker Beats 2B/7B Vision-Language Models: A New Breakthrough in Urdu GUI Grounding

A 279M text ranker ranks DOM candidate elements to far outperform 2B/7B VLMs on Urdu GUI grounding with 73.4% accuracy.

A UI grounding study on Urdu mobile web pages proposes "rank, don't regress": instead of regressing coordinates, it extracts candidate elements from the page DOM and uses a 279M-parameter cross-encoder (based on mDeBERTa-v3-base) to select the correct element. On a frozen test set of 304 instructions, the ranker achieves 73.4% point-in-box accuracy, while zero-shot UI-TARS-2B and Qwen2.5-VL-7B achieve only 29.6% and 23.7%. On the referential instruction subset, the ranker scores 65.8%, while the string-matching baseline is only 30.2%. Under the same data, a coordinate-regression model achieves only 1.6% accuracy, and the best variant reaches 75.7% after adding element geometry information. Errors are mainly concentrated on elements with identical text and ordinal instructions. The study demonstrates that small models can significantly outperform large VLMs on language-specific GUI grounding tasks, but the method depends on DOM structure and needs improvements in element discrimination and ordinal reasoning.

UI grounding tasks require models to locate the screen element that a user's instruction refers to. The mainstream approach uses vision-language models (VLMs) to read screenshots and output pixel coordinates. However, a study focused on Urdu mobile web pages proposes a different idea: instead of regressing coordinates, extract candidate elements from the page DOM and train a 279M-parameter cross-encoder (based on mDeBERTa-v3-base) to select the correct element.

Rank, Don't Regress paper title screenshot

Method: Rank, Don't Regress

The research team built a frozen test set of 304 instructions from six Urdu news websites. Of these, 149 (49%) are referential instructions that never quote the full target text. Test domains were banned from the training data before data collection began, ensuring fair evaluation.

Unlike VLMs that directly output coordinates, this method uses the page DOM to obtain a list of candidate elements and lets the model choose among them. The click point is the center of the selected element's box, so as long as the correct element is chosen, the click location is exact.

Cross-Encoder is an architecture that concatenates two input sequences (such as a query and a candidate document) and feeds them into the same Transformer encoder for joint modeling, outputting a relevance score. Unlike dual encoders (which encode the query and document separately and then compute similarity), cross-encoders can capture fine-grained interactions between the query and candidates, but computational cost grows linearly with the number of candidates. In this paper, the model concatenates the Urdu instruction and the DOM element text, encodes them with mDeBERTa-v3-base to output a matching score, and then ranks all candidates.

mDeBERTa-v3-base is the multilingual DeBERTa V3 base model proposed by Microsoft, with approximately 279 million parameters, supporting more than 100 languages. DeBERTa V3 uses disentangled attention and ELECTRA-style replaced token detection pretraining, and performs excellently on multilingual natural language understanding tasks. This paper chooses this model as the ranker backbone precisely because of its multilingual capability and moderate size, making it easy to fine-tune on low-resource languages such as Urdu.

Experimental Results: Small Model Leads by a Large Margin

On point-in-box accuracy, the 279M ranker reaches 73.4%, while zero-shot UI-TARS-2B achieves only 29.6% and Qwen2.5-VL-7B only 23.7%. On the referential instruction subset, the ranker scores 65.8%, while the training-free string-matching baseline is 30.2%.

A small controlled comparison shows that a coordinate-regression model of similar size achieves only 1.6% accuracy. After adding element geometry information as input tokens, the best variant reaches 75.7%.

Point-in-box accuracy is a commonly used evaluation metric in UI grounding tasks: a predicted click coordinate is considered correct if it falls within the bounding box of the target element. This metric directly reflects whether the user can successfully click the target, and is closer to actual experience than absolute coordinate error. All accuracies in this paper are calculated based on this metric.

Error Analysis: Duplicate Text and Ordinal Instructions

The ranker's errors are concentrated in two cases: elements with identical text, and ordinal instructions (such as "the fourth link"). This indicates that the model still has difficulty distinguishing elements with the same text and handling relative-position instructions.

Ordinal instructions refer to users indicating interface elements by relative position (such as "the fourth link" or "the second button") rather than relying on element text content. Such instructions require the model to understand the ordering of page elements and count them, posing a challenge for models that rank solely based on text, because the candidate element text itself contains no positional information. The error analysis in this paper shows that the ranker performs weakly on ordinal instructions, suggesting that layout or order features need to be introduced in the future.

Significance and Limitations

This study establishes the first UI grounding benchmark on Urdu mobile web pages, demonstrating that the ranking approach is significantly better with far fewer parameters than VLMs. However, the method depends on DOM structure, and the error patterns suggest that further improvements are needed in element discrimination and ordinal reasoning.

One-line Summary

A 279M text ranker uses DOM candidate element ranking to far outperform 2B/7B vision-language models on Urdu GUI grounding with 73.4% accuracy.

Paragraph Summary

A UI grounding study on Urdu mobile web pages proposes a new "rank, don't regress" idea: instead of directly regressing coordinates, extract candidate elements from the page DOM and use a 279M-parameter cross-encoder (based on mDeBERTa-v3-base) to select the correct element. On a frozen test set of 304 instructions, the ranker achieves 73.4% point-in-box accuracy, while zero-shot UI-TARS-2B and Qwen2.5-VL-7B achieve only 29.6% and 23.7%, respectively. On the referential instruction subset, the ranker scores 65.8%, while the string-matching baseline is only 30.2%. Under the same data, a coordinate-regression model achieves only 1.6% accuracy, and the best variant reaches 75.7% after adding element geometry information. Errors are mainly concentrated on elements with identical text and ordinal instructions. The study demonstrates that small models can significantly outperform large VLMs on language-specific GUI grounding tasks, but the method depends on DOM structure and needs improvements in element discrimination and ordinal reasoning.

#UI grounding#Urdu GUI grounding#text ranker#vision-language models#DOM candidate elements#cross-encoder#mDeBERTa-v3-base#coordinate regression#point-in-box accuracy
Share:

Related articles