Encoders Understand Bengali Compound Verbs, But LLMs Only Translate Literally

Title

Encoders Understand Bengali Compound Verbs, But LLMs Only Translate Literally

Meta Description

BengaliCV-Probe benchmark tests 26 models and finds: encoders can internally distinguish the two meanings of Bengali compound verbs with accuracy up to 93%, but large language models rarely invoke this knowledge. Translation systems literalize 15%-55% of the time, and a small model outperforms Qwen2.5-7B on correct collocations.

Article Content

In Bengali, a main verb is often followed by a "vector verb" that loses its original meaning and only adds grammatical nuances such as completion. For example, likhe phello means "finished writing," not "wrote and threw away." The same verb combination can also describe two independent events, distinguished entirely by context. Such compound verbs are a typical challenge in multiword expressions and serve as a touchstone for testing whether language models truly understand language.

BanglaCV-Probe: A Benchmark Specifically Designed to "Interrogate" Models

The research team constructed BanglaCV-Probe, containing 17 vector verbs, 1,044 minimal pairs, 204 entailment questions, and 197 sets of correct/incorrect collocations, all verified by native speakers. They used this benchmark to test 26 models—encoders, large language models, and translation systems—covering four skills: internally distinguishing the two meanings, understanding the additional meaning of vector verbs, preferring correct collocations, and preserving vector meaning in translation.

Encoders "Secretly Grasp It," LLMs "Don't Say It Out Loud"

Probes on five encoders show that they can internally distinguish the two meanings, with accuracy still reaching 93% even when surface cues are removed from the training data. This indicates that encoders do capture the semantic distinction of compound verbs.

But prompted models rarely invoke this knowledge. Accuracy on entailment questions remains between 59% and 81%. Multiple large models answer "yes" or "no" to almost all questions, and nine out of ten models fail to recognize that "ate up all the rice" implies no rice is left. The knowledge exists but cannot be reliably used during generation.

Probe is a technique for analyzing the internal representations of neural networks: a simple classifier is trained on the model's hidden-layer representations to test whether that representation linearly encodes a certain linguistic property. High probe accuracy indicates that the model indeed captures the relevant semantic distinction internally; however, probe results do not guarantee that the model will use this information in generation tasks. In this paper, probes help reveal that encoders can distinguish the two meanings of compound verbs, while prompted models cannot invoke this knowledge, forming a "knowledge exists but cannot be used" gap.

Translation Systems: Literalization Is a Common Problem

Most instruction-tuned large models translate 15%–20% of vector verb sentences literally, and for the vector verb mora (die), the literal translation rate reaches as high as 55%. This means the models treat grammaticalized vector verbs as independent content verbs, and the translations lose additional meanings such as completion and thoroughness.

Literalization refers to machine translation systems mistranslating grammaticalized components (such as vector verbs) as independent content verbs, thereby losing their grammatical function. For example, the phello (throw) in the Bengali likhe phello (finished writing) is literally translated as "throw," resulting in the incorrect translation "wrote and threw away." Literalization is a typical failure mode of grammaticalization phenomena in machine translation, stemming from the model's failure to recognize the integrity of multiword units and the lack of sufficient parallel structures in training data to learn this non-compositional mapping.

Small Models Can Also Topple Large Models

On the correct collocation preference task, a small model trained solely on Bengali reaches 88%, on par with BLOOM-7B1's 87%, and both surpass Qwen2.5-7B's 74%. Parameter scale is not the decisive factor; language-specific training is more effective.

Consensus and Divergence from Multiple Perspectives

According to the study and community discussions of the benchmark, the core consensus is the capability gap between encoders and generative models: semantics that the representation layer can encode may not be deliverable by the decoding layer. The divergence lies in attribution—one side believes that prompting strategies and instruction-tuning objectives cause models to "cut corners" and take the literal path, while the other points out that compound verbs are highly context-dependent and that the minimal pairs in the benchmark may amplify the model's conservative tendencies. Observations from a single source also mention that the literalization rate of translation systems for high-frequency vector verbs like mora is abnormally high, possibly related to their distribution in training data, but this requires further verification.

Minimal Pair is a type of stimulus pair commonly used in linguistic experimental design, referring to two sentences that differ in only one key component, thereby highlighting the semantic or grammatical role of that component. In the benchmark of this paper, minimal pairs are used to contrast the two interpretations of compound verbs (such as completive meaning versus coordinated action meaning) to test whether the model can distinguish them. However, this design may force the model to make judgments out of context, thereby amplifying its conservative tendencies and leading to biases against compound verbs that rely on context for disambiguation in natural language.

Implications for Multilingual NLP

Compound verbs are not unique to Bengali; they are widespread in South Asian languages. The value of BanglaCV-Probe lies in breaking down "whether the model understands grammaticalization" into measurable subtasks, exposing the gap between encoder capability and generation capability. For developers, scaling up parameters may not solve such fine-grained semantic problems; targeted language data and probe-driven evaluation may be more critical. The data and code are open-sourced on GitHub for reproduction and extension.

Background Supplement

Vector Verb is a core concept in the study of grammaticalization in South Asian languages, referring to a class of verbs that immediately follow the main verb in South Asian languages (such as Bengali and Hindi). They were originally content verbs (such as "throw," "eat," "die"), but in compound verb constructions they undergo grammaticalization, with semantic bleaching, retaining only aspectual information such as completion, thoroughness, suddenness, deliberateness, etc. This is functionally similar to English aspect markers (such as have done), but presented in lexical form. Vector verbs and main verbs form a semantic whole, and their additional meaning is difficult to derive literally, thus becoming a difficulty for semantic understanding in language models.

One-line Summary

Bengali compound verb benchmark reveals that encoders can distinguish semantics but LLMs cannot invoke them.

Paragraph Summary

The research team constructed the BanglaCV-Probe benchmark, containing 17 vector verbs, 1,044 minimal pairs, etc., to test 26 models' understanding of Bengali compound verbs. Results show that encoders can internally distinguish the two meanings of vector verbs (93% accuracy after removing surface cues), but prompted large models rarely invoke this knowledge, with entailment question accuracy only 59%-81% and widespread response bias; most instruction-tuned models translate literally on 15%-20% of sentences, with a literalization rate of up to 55% for mora. A Bengali-specific small model ties BLOOM-7B1 at 88% and surpasses Qwen2.5-7B on the collocation task, indicating that parameter scale is not the decisive factor. The study reveals a capability gap between the representation layer and the decoding layer, with implications for multilingual NLP evaluation and development.