Bengali Coreference Resolution Study: High-Quality Small Data Beats Mixed Large Data

Multilingual Transformer models are often assumed by default to possess cross-lingual generalization capabilities, but their actual performance on low-resource languages frequently goes unverified. A study on Bengali coreference resolution systematically evaluated multiple settings including zero-shot, fine-tuning, data augmentation, and cross-lingual transfer, and the results directly challenge two common assumptions: "more data is always better" and "multilingual models are universally capable."

The Coreference Resolution Dilemma for Low-Resource Languages

Bengali has over 250 million speakers, but well-annotated coreference resolution resources are very limited. The coreference resolution task requires models to identify which expressions in a text refer to the same entity, which is critical for discourse understanding and downstream applications. The research team chose Bengali as their entry point precisely because its gap between "large speaker population" and "scarce annotated resources" is representative.

Previous low-resource NLP work often relied on two strategies: first, directly applying multilingual pre-trained encoders in the hope that zero-shot or few-shot fine-tuning would suffice; second, compensating for annotation gaps through data augmentation or transfer from high-resource languages. This study tested each of these strategies, and the conclusions are not encouraging.

Coreference Resolution is a foundational task in natural language processing, aiming to determine which noun phrases in a text (such as pronouns, proper nouns, and descriptive phrases) refer to the same real-world entity. For example, in the sentence "Zhang San said he would come tomorrow," the model needs to determine that "he" co-refers with "Zhang San." This task is typically modeled as a clustering problem: first extract all mentions, then determine which mentions belong to the same entity cluster. Common evaluation metrics include MUC, B³, CEAF, and their average CoNLL F1. Coreference resolution is critical for discourse understanding and is a prerequisite capability for downstream tasks such as question answering, summarization, and information extraction. Unlike sentence-level tasks, it relies on entity consistency information across sentences and even across paragraphs, making it particularly sensitive to structural perturbations in data augmentation and transfer learning.

Key Finding: Annotation Quality Matters More Than Data Scale

The study's clearest conclusion is that fine-tuning with a small amount of high-quality annotated data significantly outperforms larger but mixed-quality datasets, with statistical significance reaching p < 0.01. This result implies that blindly expanding data volume in low-resource scenarios may yield little benefit and may even introduce noise. For practitioners, investing budget in refined annotation is more worthwhile than piling up low-quality data.

This finding also echoes a gradually accepted view in low-resource NLP: the core of data efficiency lies in quality rather than quantity. But this study turned this intuition into a reproducible empirical conclusion through rigorous controlled experiments.

Unexpected Failures of Data Augmentation and Cross-Lingual Transfer

Data augmentation is regarded as a lifeline for low-resource scenarios in many tasks, but this study provides a counterexample. Augmentation methods such as back-translation not only failed to improve coreference resolution performance but even led to degraded results in certain settings. The researchers point out that coreference resolution depends on discourse-level entity consistency, and the augmentation process may destroy this structural information.

Cross-lingual transfer performance was equally unstable. Transfer from distant languages like English degraded performance, while transfer from the related language Hindi did not bring consistent gains across different models. This breaks the simple assumption that "similar languages enable transfer" and shows that transfer effectiveness is highly dependent on the specific match between model and task.

Back-translation is a commonly used data augmentation technique: translating an original sentence into another language and then translating it back to the original language, thereby generating new samples with similar semantics but different expressions. It is highly effective in tasks such as machine translation and is often borrowed for low-resource scenarios to expand training data. However, back-translated text undergoes shifts in vocabulary and syntax, which may alter pronoun references, entity description methods, or discourse cohesion. For tasks like coreference resolution that require precise maintenance of entity reference consistency, back-translation may introduce referential ambiguity or break cross-sentence entity chains—this is precisely the potential mechanism behind the failure or harmfulness of augmentation methods in this study.

Hidden Risks in Multilingual Model Embedding Spaces

The study also revealed a more fundamental problem: several widely used multilingual models exhibit collapsed embedding spaces, where representations of different inputs are highly convergent, making them unsuitable for similarity-based clustering tasks. Entity clustering in coreference resolution relies on exactly this capability, so embedding collapse directly harms model performance.

This finding reminds developers that average scores on benchmarks may mask representational deficiencies in multilingual models. When selecting encoders, it is necessary to examine the distributional characteristics of their embedding spaces for specific tasks rather than relying solely on general evaluation results.

Embedding Space Collapse refers to the phenomenon where a model produces highly convergent representation vectors for different inputs, resulting in decreased discriminability between vectors. Ideally, inputs with different semantics should occupy different regions in the embedding space; when collapse occurs, representations of many inputs are squeezed into a narrow region, cosine similarities are generally high, and similarity-based clustering, retrieval, and matching tasks struggle to distinguish positive from negative samples. This phenomenon in multilingual models may stem from training objectives that over-optimize for language invariance or insufficient utilization of representation dimensions. Detection methods include observing the variance of vector distributions, computing the average similarity of random sample pairs, or visualizing the embedding distribution after dimensionality reduction. For this study, collapse directly weakened the model's ability to distinguish different entity mentions.

Implications for Low-Resource NLP Practice

Taken together, this study provides several empirical guidelines for developing coreference resolution systems for low-resource languages: prioritize investment in high-quality annotation over data scale; remain cautious about data augmentation, especially for tasks that depend on discourse structure; cross-lingual transfer requires model-by-model validation and cannot be assumed to work by default; and the selection of multilingual encoders should incorporate representational diagnostics tailored to task characteristics.

Although these conclusions come from Bengali, their methodology has reference value for other low-resource languages. The greater significance of the study lies in bringing the implicit assumptions of "multilingual models are naturally universal" and "more data is always better" to the table, using experimental data to demand that the community re-examine them.

Frequently Asked Questions

Does this study mean that data augmentation is completely ineffective in low-resource NLP? Not at all. The study focused on back-translation augmentation in the coreference resolution task, and the conclusions cannot be directly generalized to all tasks. Tasks that depend on discourse consistency may be more sensitive to augmentation, while sentence-level tasks may still benefit.

Why do multilingual models' embedding spaces collapse? The original text does not elaborate on specific causes but points out that this phenomenon makes models unsuitable for similarity clustering. The practical response is to perform representational checks for target tasks rather than relying on the general capabilities advertised for models.

What scale specifically constitutes "high-quality small data"? The original text emphasizes that "a small amount of high-quality annotated data" significantly outperforms "larger-scale mixed-quality data" but does not provide specific quantity comparisons. The core message is the directional conclusion that quality takes precedence over quantity.

One-line Summary

A Bengali coreference resolution study challenges two major assumptions: "more data is always better" and "multilingual models are universally capable."

Paragraph Summary

A systematic study on Bengali coreference resolution evaluated multiple settings including zero-shot, fine-tuning, data augmentation, and cross-lingual transfer, challenging common assumptions in low-resource NLP. The study found that fine-tuning with a small amount of high-quality annotated data significantly outperforms larger but mixed-quality datasets (p < 0.01), indicating that annotation quality matters more than data scale. Data augmentation methods such as back-translation not only fail to improve performance but may even harm it by disrupting discourse-level entity consistency. Cross-lingual transfer is equally unstable: transfer from English degrades performance, and transfer from Hindi yields no consistent gains across models. Furthermore, several widely used multilingual models exhibit embedding space collapse, making them unsuitable for similarity-based clustering tasks. The study calls on the community to re-examine assumptions about data scale and multilingual universality.