Multilingual Relation Extraction in Practice: Label Leakage and Evidence Sparsity Pitfalls Explained

When building multilingual information extraction systems, Named Entity Recognition (NER) is often relatively straightforward to tackle, while Relation Extraction (RE) frequently becomes the bottleneck of the entire pipeline. Recently, a developer shared on Reddit their hard-won lessons from training a GLiNER-style joint model (targeting EU multilingual scenarios): NER reached approximately 80% F1, while RE struggled around 30%. This massive gap reveals two easily overlooked pitfalls in zero-shot label model training.

reddit source

GLiNER-Style Joint Models: Extending from NER to RE

GLiNER (Generalist and Lightweight model for Named Entity Recognition) is a zero-shot NER architecture proposed by Urchade Zaratiana et al. in 2024. Unlike traditional NER models that train on fixed label sets, GLiNER transforms entity recognition into a matching problem between candidate labels and text spans: the model simultaneously encodes input text and candidate label descriptions, determining whether corresponding entities exist in the text by computing semantic similarity between them. This design allows the model to accept arbitrary new labels at inference time without retraining. Extending this approach to relation extraction (i.e., a GLiNER-style joint model) requires the model to not only match entity labels but also match relation labels given entity pairs—which far exceeds pure NER tasks in both computational complexity and depth of semantic understanding.

Label Order Leakage: The Model Learned to "Always Pick the First One"

The first problem this developer discovered serves as a powerful cautionary tale: the gold-standard relation label was always positioned first in the candidate list. This meant the model never actually learned genuine semantic relationships—it simply took a shortcut by learning a trivial rule: "pick the first candidate."

The verification method was straightforward: randomly shuffle the candidate labels during evaluation. The result was devastating—F1 plummeted from 37% to 14%, nearly halved. This number brutally revealed that the model's previous "high score" was largely a false signal caused by data construction flaws.

Why Label Leakage Is a Common Pitfall for Zero-Shot Models

For any label-based zero-shot model, the ordering of candidate labels can become a hidden shortcut feature. When training data construction fails to sufficiently randomize the candidate set, models will keenly detect this positional bias and use it as a prediction basis.

In Transformer-based label matching models, positional bias has deep technical roots. While Transformer self-attention is theoretically permutation equivariant, in practice, positional encoding (whether absolute or relative like RoPE) assigns different representations to tokens at different positions in the sequence. When candidate labels are concatenated as part of the input sequence fed to the model, the first-position label consistently receives the same positional encoding signal. If the correct label always appears at a fixed position in training data, the model's attention weights gradually learn to associate that position with positive predictions—essentially a form of confounding where positional information becomes spuriously correlated with label correctness.

The danger of this type of leakage lies in its subtlety: scores look normal on standard evaluation sets, but once deployed in real scenarios (where candidate order is random), performance collapses. Therefore, the author recommends: any team training label-based zero-shot models should proactively shuffle candidate order during evaluation as a routine robustness check.

Evidence Sparsity Is More Deadly Than Label Sparsity

The second finding carries even greater methodological value: evidence sparsity trumps label sparsity.

The author's analysis revealed that approximately 90% of relations in the validation set corresponded to "(evidence pattern, label)" combinations that never appeared in the training set. In other words, even if a label appears frequently during training, the specific textual evidence patterns that trigger that relation may be entirely novel.

Understanding the Technical Meaning of "Evidence Patterns"

In the context of relation extraction, "evidence patterns" refer to the specific linguistic structures in text that express semantic relationships between two entities. For example, the relation "headquartered in" might be expressed through multiple evidence patterns: "Company X has its headquarters in Y," "X, headquartered in Y," "Y is the home base of X," etc. Each expression constitutes an independent evidence pattern. The core difficulty of relation extraction lies in the fact that the evidence pattern space for any given relation is enormous and open-ended—different languages, genres, and domains produce vastly different expressions. This contrasts sharply with NER: entity boundary recognition primarily relies on local features (capitalization, word morphology, context windows), while relation identification often requires understanding complex semantic structures spanning multiple clauses or even multiple sentences.

Label Frequency Is Nearly Useless for Predicting RE Performance

A counterintuitive conclusion emerges: label frequency is almost entirely non-predictive of model performance on that label. Conventional wisdom suggests that high-frequency labels are easier to learn well, while low-frequency labels represent long-tail difficulties. But in relation extraction, what truly determines generalization capability is whether the model has seen similar evidence expressions, not how many times the label itself appeared.

This insight has direct implications for data construction: rather than simply pursuing balanced sampling at the label level, focus on diversity coverage of evidence patterns. If evidence patterns in training data are too homogeneous, the model will still fail when encountering new expressions, even with perfectly balanced label distributions.

Long-Tail Label Space: To Canonicalize or Not?

Another practical challenge the author faces is an extreme long-tail label distribution: 14,000 labels, of which 9,000 are singletons (appearing only once).

Long-tail label distributions are extremely common in information extraction, especially in open-domain knowledge graph construction scenarios. Take Wikidata as an example: it contains over 11,000 relation types, but high-frequency relations (like "instance of" or "country") appear thousands of times more often than low-frequency ones. The theoretical premise of zero-shot learning is that models can perform generalized reasoning through semantic descriptions of labels, but this requires the label descriptions themselves to carry sufficient information and discriminability. When the label count inflates to 14,000 with many semantically similar labels (e.g., "founded by" vs. "co-founded by"), the model needs extremely fine-grained semantic discrimination capability, placing very high demands on the encoder's representation space.

The author currently chooses not to perform canonicalization, reasoning that it preserves zero-shot generalization capability. This represents a classic trade-off:

Benefits of not canonicalizing: Preserves original semantic granularity of labels; the model has the opportunity to perform zero-shot reasoning on unseen new labels, aligning with the design philosophy of GLiNER-style models.
Costs of not canonicalizing: 9,000 singletons mean vast numbers of labels have virtually no supervisory signal, making training highly unstable, and evaluation metrics easily dominated by noise.

Compromise Approaches for Long-Tail Labels

For such situations, a common community practice is hierarchical processing: apply lightweight clustering or mapping to semantically highly overlapping labels (alleviating the singleton problem), while maintaining an open zero-shot entry point for handling genuinely new labels. Completely abandoning canonicalization maximizes generalization space in theory, but with inconsistent label quality and excessive singletons, it often drags down overall training convergence. Whether to canonicalize should be determined based on the downstream task's actual requirements for label granularity.

Special Challenges in Multilingual Scenarios

Multilingual scenarios add multiple layers of extra complexity to relation extraction. First, there are cross-lingual differences in evidence patterns: different languages may use entirely different syntactic structures to express the same relation (e.g., SOV vs. SVO word order), causing the evidence pattern space to explode further. Second, multilingual pretrained models (such as mBERT, XLM-R) have uneven representation quality across languages—high-resource languages (English, French) typically have superior representation quality compared to low-resource languages (such as Latvian, Maltese), which is particularly pronounced in EU multilingual scenarios. Additionally, annotation resources vary dramatically across languages, causing high-resource languages to dominate gradient updates during joint training, further suppressing RE performance in low-resource languages.

This also explains why the developer's NER achieved 80% F1 while RE only reached 30%: the local features NER relies on (word morphology, context windows) are already well captured by multilingual pretrained models, while the cross-lingual relational reasoning capability required by RE remains far from adequately learned.

Core Takeaways for Multilingual Relation Extraction Practitioners

Synthesizing this sharing, several broadly applicable lessons emerge for similar projects:

Watch out for construction-induced leakage: Data construction details like candidate ordering and label arrangement can become shortcuts for model cheating. Always perform randomization verification during evaluation.
Focus on evidence, not labels: The generalization bottleneck in relation extraction lies in evidence pattern coverage. Data augmentation should center on expression diversity.
NER and RE difficulty are asymmetric: In joint models, RE is typically far harder than NER. Design separate training strategies and evaluation benchmarks for RE rather than masking problems with aggregate metrics.
Long-tail labels require trade-offs: Find a balance between zero-shot generalization and training stability. Extreme singleton distributions often require some degree of aggregation.

Multilingual relation extraction remains an open challenge. This developer's candid "failure log" provides invaluable guidance for those who follow—sometimes an honest number dropping from 37% to 14% is worth more than a polished experiment report.