Multi-Document Contradiction Discovery: The Retrieval and Reasoning Challenge Beyond NLI

An NLP Task With Almost No Literature

Most NLP tasks come with a well-worn trail of existing research to follow. But a third-year NLP PhD student recently shared an unusual predicament on Reddit: his advisor handed him a research problem he'd never seen documented anywhere — given a private organization's HR policy document collection (100 to 500 PDFs), find all pairs of mutually contradictory clauses within it.

At first glance, this might not seem particularly novel. There's a vast body of research on NLI (Natural Language Inference)-style contradiction classification. But the student quickly realized that the truly tricky part was how the problem was framed — it was fundamentally different from anything he'd seen. Traditional NLI tasks assume someone has already handed you a sentence pair, and your job is simply to classify it as entailment, neutral, or contradiction. In multi-document contradiction discovery, identifying which sentence pairs are even candidates in the first place is the core challenge.

Combinatorial Explosion: Why Brute Force Doesn't Work

The scale of the problem defines the boundaries of any approach. In a corpus of roughly 1,000 to 2,000 clauses, pairwise comparison would produce millions of candidate pairs. That means:

• Per-pair LLM calls are infeasible: Millions of large model calls are prohibitive in both cost and time;
• Stuffing full documents into prompts also fails: Cramming all documents into a context window triggers the classic "lost-in-the-middle" problem, where critical information buried in long contexts tends to be overlooked by the model.

This is precisely what distinguishes this task from conventional contradiction detection — it's not a classification problem, but a discovery problem in an enormous search space.

What the Existing Literature Covers

The researcher conducted thorough literature review, only to find his problem sitting in the cracks between several mature fields:

• Legal NLP (e.g., ContractNLI): Stanford's dataset for NLI tasks contains real contradictions, but its task format is "hypothesis against clause" — not a symmetric clause-to-clause comparison;
• Conflict detection in requirements engineering: Focuses on conflicts between software requirements — conceptually similar, but applied in a very different context;
• RAG conflict research: Deals with information conflicts in retrieval-augmented generation — again, an adjacent field but not a direct match.

The closest work involves injecting synthetic contradictions into synthetic corpora to test detectors. The researcher borrowed this evaluation strategy, constructing benchmarks by injecting contradictions into his corpus. He used three data sources: a university's HR handbook, a synthetic corpus with injected contradictions, and ContractNLI with external annotations.

The Subtle Difference in Task Formulation

You might not have noticed that these three data sources don't share exactly the same task format. The university HR handbook and synthetic corpus involve clause-to-clause symmetric comparisons, while ContractNLI is fundamentally an asymmetric hypothesis-to-clause comparison. This inconsistency in form reflects the fact that "multi-document contradiction discovery" as a standalone task has yet to be fully defined.

A Two-Stage Pipeline: Balancing Recall and Precision

Faced with combinatorial explosion, the researcher designed a two-stage pipeline built around a simple idea: first dramatically reduce the candidate space, then carefully verify what remains.

Stage 1: HyDE-Based Contradiction Retrieval

He adopted a HyDE (Hypothetical Document Embeddings)-style retrieval strategy, with one clever twist: the query is not the original clause, but a hypothetical, contradictory version of it.

The reasoning is elegant — if you want to find content that contradicts a given clause, querying with "the opposite of that clause" will naturally surface more genuinely conflicting material. This feeds into a recall-oriented candidate retrieval phase driven by LLM-generated candidate pairs.

Stage 2: Precision-Oriented Contradiction Verification

Once candidate pairs are generated, the pipeline moves into a precision-oriented verification stage. Each candidate pair is returned to its source document for re-reading, and an LLM decides whether it constitutes a genuine contradiction. Two key findings emerged:

1. Contextual information significantly improves performance: Drawing on Anthropic's contextual retrieval approach, adding surrounding context sentences improved both retrieval and verification precision. The surrounding document context helps the model more accurately understand a clause's scope and preconditions;
2. Agentic approaches actually perform worse: This is a counterintuitive result. The researcher tried multi-step, tool-calling agentic verification methods, and found they underperformed single-pass prompting.

Why Agentic Verification Doesn't Win

This finding deserves attention. In the current industry climate where "everything should be an agent," multi-step reasoning doesn't always pay off. For tasks like contradiction verification that require holistic judgment rather than stepwise decomposition, a single well-crafted prompt may actually preserve a more coherent reasoning chain — avoiding error accumulation and attention drift across multiple steps.

Empirical Results and Open Questions

As a case study, the researcher ran the full pipeline on a publicly available government policy corpus and successfully identified several genuine contradictory clauses. Three baselines have been established so far: an NLI baseline, a direct prompting baseline, and an agentic baseline.

But several questions remain open:

• Is there a research community being missed? It's hard to believe no one has studied this systematically — there may be a relevant body of literature he simply hasn't been able to name;
• Can this reach a top-tier venue? His advisor leans toward lower-tier conferences or journals, but he wonders whether this work could be competitive at premier NLP venues;
• How to strengthen the experiments short-term? He wants to know what additional ablations or comparisons should be added beyond the existing baselines.

The Real Value of Multi-Document Contradiction Discovery

From a research methodology standpoint, this student's predicament is itself quite instructive. Task formulation often determines a paper's originality more than the methods themselves do. Multi-document contradiction discovery appears niche largely because it straddles the boundaries of legal NLP, information retrieval, and requirements engineering — no single community has claimed it fully.

Several directions seem worth exploring further:

• Establishing a more rigorous evaluation protocol: Contradictions are subjective and context-dependent, requiring inter-annotator agreement checks to ensure benchmark reliability;
• Building a taxonomy of contradiction types: Distinguishing direct contradictions, conditional contradictions, temporal contradictions, and so on would significantly strengthen the paper's theoretical contribution;
• Quantifying the upper bound of retrieval recall: The ceiling of a two-stage pipeline is determined by first-stage recall — it's essential to demonstrate that genuine contradiction pairs aren't being filtered out during retrieval.

Regardless of which venue this work ultimately targets, it addresses a real and commercially valuable pain point: corporate compliance review, policy auditing, and legal document management all urgently need tools that can automatically surface internal contradictions across large document collections. As LLM reasoning capabilities continue to grow, figuring out how to deploy them at the level of discovery rather than mere judgment may be exactly the next frontier worth pursuing.