[KongchangAI]
· 2 min read· 1,038 words

BLiMP-RO: A New Benchmark for Romanian Grammar Evaluation

BLiMP-RO: A New Benchmark for Romanian Grammar Evaluation

BLiMP-RO is the first Romanian grammatical minimal-pair benchmark, revealing how data scale, composition, and architecture affect grammatical generalization.

BLiMP-RO is the first systematic Romanian grammatical minimal-pair benchmark, covering 10 domains and 50 grammatical phenomena, including 493 linguist-constructed and native-speaker-validated sentence pairs, and 5,942 semi-automatically generated and human-validated sentence pairs. The research team trained GPT-2 and GPT-BERT from scratch, finding that for every tenfold increase in subtitle data, grammatical judgment accuracy improves by about 9 percentage points; at matched scales, BabyBabelLM data generally outperforms video subtitles; and GPT-BERT consistently outperforms GPT-2. The benchmark fills the gap in Romanian grammar evaluation and provides a reusable benchmark construction methodology for low-resource languages.

Whether language models truly grasp grammar requires benchmarks that cover diverse linguistic phenomena and are validated by human judgment. The introduction of BLiMP-RO provides the first systematic minimal-pair evaluation tool for Romanian, a low-resource language, and thereby reveals how training data scale, composition, and model architecture affect grammatical generalization.

BLiMP-RO Benchmark Overview

What Is BLiMP-RO?

BLiMP-RO is a linguistic minimal-pair benchmark for Romanian, covering 10 domains and 50 grammatical phenomena. It consists of two parts: 493 sentence pairs constructed by linguists, evaluated by native speakers through five experiments; and 5,942 semi-automatically generated and human-validated sentence pairs. This design ensures broad coverage of linguistic phenomena while scaling up the data through semi-automatic expansion.

Similar to the English BLiMP, BLiMP-RO adopts the minimal-pair paradigm: each sentence pair contrasts only on a single grammatical feature (e.g., subject-verb agreement, case marking), and the model must judge which sentence is more grammatical. This design precisely probes a model's sensitivity to specific grammatical rules rather than relying on surface statistical cues.

The minimal-pair paradigm originates from linguistic experimental design, referring to two sentences that differ minimally on a single grammatical feature (e.g., "The boy laughs" vs. "The boy laugh"), thereby excluding confounds such as vocabulary and semantics. In language model evaluation, the model must judge which sentence is more grammatical, and its accuracy directly reflects sensitivity to specific grammatical rules rather than reliance on surface statistical cues. This paradigm was systematized by the English BLiMP benchmark proposed by Warstadt et al. in 2020 and has become a standard tool for probing models' grammatical knowledge.

Training Small Models from Scratch: Insights from Infant Language Data

The research team further explored grammatical generalization under limited linguistic input. They trained GPT-2 and GPT-BERT models from scratch, using training data that included Romanian BabyBabelLM data and a newly collected 10-million-word corpus of child-targeted video subtitles.

Key findings include:

  • Significant data scale effect: For every tenfold increase in subtitle training data, accuracy improves by about 9 percentage points. This shows that even with a single data source, accumulating more data can still yield substantial gains in grammatical ability.
  • Data composition affects generalization: At matched data scales, BabyBabelLM generally outperforms video subtitles. This may be because BabyBabelLM more closely resembles the natural distribution of child language acquisition, whereas video subtitles suffer from domain bias.
  • Clear architectural differences: GPT-BERT consistently outperforms GPT-2. The bidirectional encoder architecture may be more advantageous for grammatical judgment tasks because it can utilize both left and right context simultaneously.

These results highlight that training data quantity, composition, and model architecture jointly shape grammatical generalization. For low-resource languages, how to efficiently leverage limited data (such as child-directed corpora) becomes a direction worth exploring.

BabyBabelLM is a dataset that simulates the environment of child language acquisition, built on the "infant language data" hypothesis: the linguistic input children receive is highly limited in scale and distribution, yet sufficient for acquiring grammar. This dataset typically contains child-directed speech (CDS) and utterances from the target child's daily environment, ranging from one million to ten million words—far smaller than general pretraining corpora. In low-resource language research, BabyBabelLM serves as a benchmark data source for testing models' grammatical generalization under limited input.

GPT-BERT is not a standard term; here it refers to a model using a BERT-style bidirectional encoder architecture, contrasting with the autoregressive GPT-2. Bidirectional encoders predict masked words using both left and right context during pretraining, and thus may be more advantageous for grammatical judgment tasks that require integrating full-sentence information; GPT-2's autoregressive objective relies only on left context. This architectural difference is a key factor explaining the performance gap between the two.

Why Does a Romanian Benchmark Matter?

Existing grammar evaluation benchmarks are heavily concentrated in resource-rich languages such as English. Romanian, as a member of the Romance language family, has rich morphological inflection and a case system, and its grammatical phenomena differ significantly from English. BLiMP-RO fills this gap and provides a reusable methodology for multilingual grammar evaluation: linguists construct a core set, native speakers validate, semi-automatic expansion, and human secondary validation.

The release of this benchmark also echoes the AI community's growing attention to fair evaluation for low-resource languages. As language models expand to multilingual settings, the lack of targeted benchmarks can lead to misjudgments of model capabilities. The emergence of BLiMP-RO gives the measurement of Romanian grammatical ability a comparable yardstick.

Implications for Model Developers

From the experimental results of BLiMP-RO, developers can extract several practical takeaways:

  1. Data scale remains the most direct lever: On grammatical generalization tasks, a tenfold increase in data brings about a 9-percentage-point improvement, indicating that continuously expanding high-quality corpora still pays off.
  2. Data sources must be chosen carefully: Child-directed corpora and general corpora each have advantages; at matched scales BabyBabelLM is better, but video subtitles can also serve as a supplement in specific scenarios.
  3. Architecture choice cannot be ignored: The consistent advantage of GPT-BERT suggests that for grammar-sensitive tasks, bidirectional context modeling may be more effective than autoregressive modeling.

BLiMP-RO is not only a benchmark but also an experimental framework for studying grammatical generalization in low-resource languages. Its open data and evaluation methods are expected to promote the construction of minimal pairs for more languages.

Future Directions

The research team notes that future work can extend to more linguistic phenomena and larger-scale native-speaker validation, and explore the impact of different pretraining objectives on grammatical generalization. For the Romanian NLP community, BLiMP-RO provides the first systematic grammar diagnostic tool, on which targeted improvement methods can be developed.

One-line Summary

BLiMP-RO is the first Romanian grammatical minimal-pair benchmark, revealing how data scale, composition, and architecture affect grammatical generalization.

Paragraph Summary

BLiMP-RO is the first systematic Romanian grammatical minimal-pair benchmark, covering 10 domains and 50 grammatical phenomena, including 493 linguist-constructed and native-speaker-validated sentence pairs, and 5,942 semi-automatically generated and human-validated sentence pairs. The research team trained GPT-2 and GPT-BERT from scratch, finding that for every tenfold increase in subtitle data, grammatical judgment accuracy improves by about 9 percentage points; at matched scales, BabyBabelLM data generally outperforms video subtitles; and GPT-BERT consistently outperforms GPT-2. The benchmark fills the gap in Romanian grammar evaluation and provides a reusable benchmark construction methodology for low-resource languages.

#BLiMP-RO#Romanian grammar benchmark#minimal pairs#low-resource languages#grammatical generalization#GPT-BERT
Share:

Related articles