IntLawNER: The First Named Entity Recognition Dataset and Benchmark for International Law

International law provides the normative framework through which nations coordinate action, regulate armed conflict, and protect human rights — yet legal texts in this domain have long lacked token-level named entity recognition (NER) resources. A new paper published on arXiv introduces IntLawNER, an NER dataset and benchmark specifically targeting codified international law sources, aiming to fill this gap.

Why International Law Needs Its Own NER Resources

Named entity recognition is a foundational NLP task focused on identifying entities such as person names, organizations, and locations within text. But general-purpose NER models tend to struggle with international legal texts, because entities in this domain depend not just on surface form but heavily on institutional-functional context.

The research team drew corpora from three authoritative legal sources: International Court of Justice (ICJ) judgments, United Nations Security Council resolutions, and European Court of Human Rights (ECtHR) rulings. IntLawNER ultimately comprises 2,987 manually gold-annotated sentences and 8,094 entity spans, divided into seven entity types defined by institutional characteristics. This institution-centric annotation scheme is precisely what distinguishes it from general-purpose NER datasets.

International legal texts exhibit several distinctive linguistic properties that make generic NER tools difficult to apply. First, they contain a large number of "institutional-function-dependent entities" — terms such as "contracting party," "applicant," and "High Commissioner" whose entity type depends on the role they play in a specific legal proceeding rather than on the literal word itself. The same term may belong to completely different entity categories in different contexts. Second, international legal texts are highly citational, saturated with cross-references to treaty articles, resolution numbers, and case names — all of which constitute entities that must be recognized. Third, ICJ judgments, Security Council resolutions, and ECtHR rulings differ substantially in writing style, legal tradition, and institutional conventions, making it difficult to generalize training data from one setting to another. These properties collectively explain why international law NER requires a dedicated, institution-function-based annotation scheme rather than borrowing existing frameworks from domains like news or biomedicine.

A Hybrid Annotation Pipeline: Algorithm and Agent in Tandem

Building a high-quality legal annotation dataset from scratch is prohibitively expensive. The research team designed a hybrid algorithm-agent pipeline that balances cost and quality, compressing a raw source corpus of 468,000 sentences into a compact annotated set.

The pipeline consists of three stages: candidate retrieval, LLM-based vetting, and human review. Notably, 89.6% of entity spans in the final gold annotations were accepted directly from the "silver layer" (i.e., the initial machine-generated annotations), indicating a high baseline quality from the automated stage. This design substantially reduces the burden of pure manual annotation and offers a reusable paradigm for constructing domain-specific datasets.

The Pitfall of Agreement Metrics: High Kappa ≠ High Quality

The most methodologically significant finding in the paper comes from the analysis of the silver-to-gold conversion process. The research exposes an easily overlooked problem: in domain-specific NER tasks, human-machine aggregate agreement metrics can be misleading.

Specifically, Cohen's kappa reached 0.964 on matched spans — seemingly near-perfect agreement. But once missed entities, boundary errors, and label corrections are factored in, macro-F1 drops sharply to 0.753. This means that a high kappa value can mask numerous deficiencies in actual annotation quality. For researchers who rely on expert annotation, this serves as a cautionary reminder: metric selection must be commensurate with task complexity, and a single high score cannot represent overall quality.

Cohen's kappa is a classic statistic for measuring annotation agreement between two annotators (or between a human and a model), with a range of -1 to 1 where 1 indicates perfect agreement and 0 is equivalent to random labeling. In NER tasks, kappa is typically computed over "identified spans" — that is, only positions where both parties agree that an entity exists — which naturally skips cases where one party annotated an entity that the other missed, causing the metric to be artificially inflated. Macro-F1, by contrast, computes precision and recall separately for each entity class and then averages them, weighting rare and common classes equally, which fully exposes a model's failures on low-frequency entities. The dramatic gap between the two metrics (0.964 vs. 0.753) fundamentally reflects the difference between "span boundary hit rate" and "complete entity recall" — high kappa only indicates that the labels on recognized spans are accurate, while concealing the large number of entities that were never identified at all.

Model Benchmarks: Zero-Shot Collapse to Few-Shot Recovery

In the benchmarking phase, the research team evaluated a range of mainstream approaches, with instructive results.

The zero-shot span-based GLiNER model nearly completely failed on entity types that depend on institutional function rather than surface form, achieving a micro-F1 of only 0.243. This confirms the core difficulty of international law entity recognition — models cannot determine the legal meaning of an entity from lexical patterns alone. Fine-tuned Transformer models performed better overall but struggled with rare labels.

The turning point came from carefully selected few-shot examples. When prompts included contrastive few-shot samples that demonstrated label distinctions, all large language models outperformed their zero-shot counterparts. Claude Opus 4.6 achieved the best results, with a micro-F1 of 0.873. This outcome demonstrates that in specialized domain tasks, the quality and contrastive design of examples often matters more than simply scaling up model size.

GLiNER (Generalist and Lightweight model for NER) is a class of open NER models built on dual-encoder or span-based architectures that supports zero-shot recognition by describing entity types in natural language, without retraining for each new type. The core idea is to map entity type descriptions and text spans into a shared vector space and classify them via similarity matching. This design excels in general settings because common entity types (person names, place names, organization names) have linguistic descriptions that closely correspond to surface-level text features. In international law, however, identifying entities like "applicant state" or "respondent" requires understanding procedural roles and legal context rather than relying on surface lexical patterns — precisely the blind spot of GLiNER's zero-shot mode. Few-shot prompting, by including several annotated examples in the input, lets the model learn task boundaries and label contrast logic from demonstrations. This is essentially in-context learning compensating for the model's lack of domain knowledge, which is why it often delivers significant performance gains in specialized domain tasks.

Significance and Outlook

IntLawNER has been publicly released as the first token-level NER benchmark and reusable resource for international legal texts, primarily intended for extracting citations and entities from international legal documents.

Its value lies not only in the dataset itself, but also in the two deeper issues it surfaces: first, the reliability of evaluation metrics in domain-specific NER deserves reexamination; second, institutional-function-dependent entities pose a substantive challenge to existing models. For applications in legal technology, compliance retrieval, and cross-jurisdictional judicial text analysis, this resource provides a solid foundation. As large language models continue to penetrate specialized domains, construction methodologies like IntLawNER — balancing cost and quality — may well become the standard approach for building high-value vertical datasets.