Weak Supervision + LLM Annotation Refinement: Extracting Dataset Citations from Humanitarian Documents
Weak supervision + LLM refinement enables low-cost automated dataset citation extraction from humanitarian documents.
This arXiv paper tackles the challenge of tracking dataset citations in Forced Displacement and FCV humanitarian documents by proposing a weak supervision framework that requires no large-scale manual annotation. The core approach divides labor between a lightweight model that generates candidate mentions and a frontier LLM that refines them in context, supplemented by synthetic and contrastive examples. The fine-tuned lightweight model achieves 88.2% passage-level accuracy and 89.5% precision on passages containing citations across a 1,706-passage gold benchmark, demonstrating practical utility for coarse-grained classification and offering a replicable, low-cost path for building domain-specific supervision signals in annotation-scarce settings.
Development organizations and humanitarian agencies produce vast volumes of surveys, administrative records, and other data resources each year to support research, policymaking, and field operations. Yet a persistent challenge remains: which publications actually cite these datasets? References are scattered across research papers, project documents, and humanitarian reports — making it difficult to track how data is actually used or to identify gaps in data availability and dissemination.
A newly published arXiv paper (arXiv:2609.12107) proposes a weak supervision framework targeting documents in the Forced Displacement and Fragile, Conflict, and Violence (FCV) domains. The goal is to automate dataset citation extraction without requiring large-scale manually annotated corpora.
The Core Problem: Domain Adaptation Under Annotation Scarcity
Dataset extraction is fundamentally an information extraction task — identifying mentions of specific datasets within text. Reasonably mature methods exist for general research literature, but when applied to specialized domains like humanitarian affairs and conflict studies, the scarcity of labeled data becomes the primary bottleneck. Building a sufficiently large domain-specific training set by hand is expensive and requires subject matter experts.
The research team's approach sidesteps this requirement entirely: rather than constructing large-scale manual annotations upfront, they use a "weak supervision + LLM refinement" strategy to automatically generate usable training signal. This has strong practical relevance for real-world scenarios where annotation resources are constrained.
Method Breakdown: Lightweight Model for Candidates, Frontier LLM as Judge
The overall pipeline can be understood as a series of coordinated stages:
Lightweight Model Generates Candidates
A lightweight model trained on general research literature first processes unannotated domain documents to generate candidate dataset mentions. This step casts a wide net — quickly surfacing potential citation locations across large volumes of text — though with limited precision.
Frontier LLM Reviews in Context
Candidates are then passed to a frontier large language model (LLM), which verifies or rejects each one in context and corrects extraction boundaries. Compared to the lightweight model, the LLM's semantic understanding allows it to determine whether a passage is genuinely citing a dataset and to accurately delineate the mention's span. This step is critical for improving annotation quality.
Synthetic and Contrastive Examples Added
Building on the LLM-refined annotations, the team further supplements the training data with targeted synthetic examples and contrastive examples. Contrastive examples help the model learn to distinguish "looks like a dataset citation but isn't" cases, reducing false positives.
Fine-Tune the Lightweight Model for Large-Scale Extraction
These refined annotations are ultimately used to fine-tune the lightweight model. The key advantage: at deployment time, only the low-cost lightweight model is used, enabling large-scale extraction, while the expensive LLM is invoked only once during training data construction. This division of labor — "use a large model to create data, use a small model for inference" — is a pragmatic cost-control strategy increasingly common in weak supervision practice.
Evaluation Results: Performance on the Gold Benchmark
The researchers evaluate their model on an independent gold-standard benchmark comprising 1,706 text passages spanning research, humanitarian, and operational document types. Key results include:
- Mention level: overall precision of 74.1%, recall of 70.5%;
- Among passages that actually contain dataset citations: precision rises to 89.5%;
- Passage level: accuracy of 88.2% for distinguishing passages with vs. without citations, with specificity reaching 88.6%.
These figures indicate that the model performs quite robustly on the coarser-grained task of determining "does this passage involve a dataset citation," while there remains room for improvement on fine-grained mention extraction. The gap between mention-level and passage-level metrics also reflects that precisely locating mention boundaries is more challenging than classifying passage topic.
Significance and Limitations
The value of this work lies not only in the specific model, but in the replicable pathway it demonstrates: when domain-labeled data is limited, how to construct domain-specific supervision signal at low cost. For humanitarian data governance, the ability to systematically track dataset citations means better understanding of how data is used and where dissemination gaps exist — laying a technical foundation for downstream data gap analysis.
On the limitations side, mention-level precision and recall hovering around 70% means that real-world large-scale deployment will still produce a meaningful rate of missed detections and false positives, making it difficult to fully eliminate human review in the near term. Additionally, the framework is heavily dependent on the refinement quality of the frontier LLM, and any biases in that LLM may propagate into the final training data.
For readers interested in information extraction, weakly supervised learning, and AI applications in the social good space, this paper offers a reference case study with both methodological value and practical grounding.
Related articles
CCPS Sampling: Preserving Reasoning Diversity to Boost LLM Performance Without Fine-Tuning
A new arXiv paper proposes CCPS, which boosts LLM reasoning accuracy without any training by preserving reasoning diversity via Chopthin resampling and semantic majority selection — achieving up to 10.6pp absolute gains.
MIT Spinout Transforms Plastic Waste into High-Resilience Building Materials
MIT spinout Atlas Building Composites converts plastic waste into resilient structural components for buildings and infrastructure, pioneering a high-value recycling pathway.
Repair Before Reinforce: Context-Augmented Knowledge Graph Reasoning Tackles Multi-Hop QA
New arXiv paper proposes a context-augmented KG reasoning framework using a "Repair Before Reinforce" strategy to improve LLM multi-hop QA. Validated on Gastroparesis and Diabetes KGs with Qwen3-14B, achieving 100% single-hop accuracy after repair.