Dual-Axis Policy Entailment Benchmark DAPE: A New Paradigm for Policy Change Reasoning

Why Policy Change Reasoning Needs a New Benchmark

A core question in public policy research is: does a new policy continue, broaden, narrow, or reverse the previous policy? Traditional methods often rely on manual coding, but when faced with massive volumes of policy text, annotators tend to classify difficult policy pairs as "unrelated," thereby blurring the subtle direction of policy change. To address this, researchers propose the Dual-Axis Policy Entailment (DAPE) benchmark, designed to simultaneously evaluate the relationship between policy pairs along two dimensions: "domain" and "direction of change."

How DAPE Works: Dual-Axis Design and Five Label Categories

The core innovation of DAPE lies in decomposing the policy pair judgment into two ordered decisions:

• Binary judgment: Do the two policy instructions govern the same policy domain?
• Five-way judgment: Does the latter policy continue, broaden, narrow, or reverse the former policy, or does it shift to a different domain?

Together, the two judgments form a five-category label (continuation, broadening, narrowing, reversal, different domain), which serves as the primary evaluation metric. This design avoids conflating errors in one dimension with a vague "unrelated" label, while also making annotation more rigorous—coders cannot arbitrarily mark difficult policy pairs as domain-irrelevant but must make explicit, checkable decisions.

Entailment here is borrowed from the natural language inference (NLI) task: given a premise sentence and a hypothesis sentence, determine whether the latter can be logically inferred from the former. DAPE adapts this into "policy entailment"—treating the former policy as the premise and the latter policy as the hypothesis, and using entailment relations to characterize the semantic position of the new policy relative to the old one: "continuation" corresponds to pure entailment, "broadening" corresponds to the hypothesis covering a broader scope of application, "narrowing" is the opposite, "reversal" means the semantic direction is opposite, and "different domain" indicates that the premise and hypothesis do not form a comparable relationship. The value of this analogy lies in transforming the vague qualitative judgment of policy change into an annotatable, computable ordered classification problem.

Expert-Annotated Benchmark and Calibration Protocol

Researchers constructed an expert-annotated benchmark containing 1,000 ordered policy instruction pairs, all sampled from the vast policy corpus of the Chinese government. They also contributed an annotation calibration protocol (including an adjudication package and pre/post consistency measurement) as a methodological tool to ensure annotation quality is quantifiable and reproducible.

Model Performance: Even Frontier Flagships Fail Zero-Shot

In the zero-shot setting, 14 open-weight models and 4 frontier API flagship models all failed the task. Even the best-performing frontier flagship model achieved a random-corrected agreement with expert labels far below the human upper bound (Cohen's κ ≈ 0.29 vs. 0.53). This means that existing large models still have a significant gap with human experts in policy change reasoning.

Cohen's κ (Cohen's kappa) is a statistic measuring agreement between two annotators (or between a model and an expert), typically ranging from 0 to 1, calculated after subtracting the agreement that could be achieved by random chance alone: κ = (Po − Pe) / (1 − Pe), where Po is the observed actual agreement rate and Pe is the expected random agreement rate. Generally, κ > 0.6 is considered substantial agreement, and 0.4–0.6 moderate agreement. Therefore, the gap between κ ≈ 0.29 and 0.53 in this paper is not just a difference in numerical magnitude—it means the model's agreement with experts is only slightly above chance level, representing a qualitative gap from the moderate agreement among human experts. The use of a random-corrected metric rather than raw accuracy is to avoid inflated performance caused by imbalanced class distributions (e.g., a high proportion of "continuation").

Fine-Tuned Open-Source Models Approach Human Level

Research further shows that fine-tuning open-source models can substantially narrow the gap. After fine-tuning Qwen3-14B with LoRA SFT using five-fold cross-validation, the model reached the human inter-coder upper bound (κ = 0.524 vs. 0.527). This result demonstrates that through targeted fine-tuning, open-source models can approach human annotator agreement levels on specialized policy reasoning tasks.

LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning method: it freezes all original weights of the pre-trained model and injects two low-rank matrices A and B alongside the weight matrices of attention and other layers, training only these two small matrices, thereby approximating full fine-tuning with very few trainable parameters (typically less than 1% of the original model). SFT (Supervised Fine-Tuning) refers to continuing to train the model with labeled input–output examples using standard cross-entropy loss. The combination is "LoRA SFT," which offers the advantages of low memory usage, fast training, and the ability to save independent lightweight adapters for each task. Five-fold cross-validation divides the 1,000 policy pairs into five parts, training on four and evaluating on one in rotation, reporting the average κ across five runs to reduce evaluation variance caused by a single split.

Application Example: China AI Policy Change Map (1980–2024)

As an application of the method, researchers used DAPE to analyze China's artificial intelligence policies (1980–2024) and mapped the direction of policy change across different eras. This demonstrates the benchmark's practical value in real policy analysis and provides policy researchers with a scalable quantitative tool.

Significance and Outlook

The introduction of DAPE not only fills the gap in policy change reasoning benchmarks but also enhances annotation rigor through its dual-axis design. Its calibration protocol and fine-tuning approach provide reusable methodology for future research. Although current large models perform poorly in the zero-shot setting, fine-tuned open-source models have approached human level, bringing hope for automated policy analysis. In the future, the benchmark is expected to expand to more languages and domains, advancing the development of intelligent policy analysis.

One-line Summary

DAPE dual-axis benchmark reveals that large models struggle with zero-shot policy change judgment, but fine-tuning can approach human level.

Paragraph Summary

The article introduces the Dual-Axis Policy Entailment benchmark DAPE, designed to determine whether a new policy continues, broadens, narrows, reverses, or shifts to a different domain relative to an old policy. Its core is to decompose the judgment into a binary decision of "whether the same policy domain" and a five-way decision of "direction of change," combined into a five-category label, preventing annotators from vaguely classifying difficult policy pairs as unrelated. Researchers constructed 1,000 expert-annotated Chinese government policy pairs and an annotation calibration protocol. In zero-shot tests, 14 open-source models and 4 frontier API flagships all failed, with the best Cohen's κ only about 0.29, far below the human upper bound of 0.53; however, after fine-tuning Qwen3-14B with LoRA SFT, κ reached 0.524, approaching human level. The method was also used to map China's AI policy changes from 1980 to 2024.