DAPE
A benchmark for evaluating reasoning about policy change. It decomposes judgments about policy into two ordered decisions, a domain judgment and a change-direction judgment, forming five label categories.
Timeline (last 90 days)
Researchers proposed the Dual-Axis Policy Entailment (DAPE) benchmark, aimed at simultaneously evaluating policy relationships along two dimensions: 'domain' and 'direction of change.'
DAPE decomposes the policy-pair judgment into two ordered decisions: a binary judgment (whether they are in the same policy domain) and a five-way judgment (continuation, broadening, narrowing, reversal, different domain).
The DAPE benchmark contains 1000 ordered policy instruction pairs, all drawn from Chinese government policy corpora.
The researchers contributed an annotation calibration protocol (including adjudication packages and before-and-after consistency measurement) as a methodological tool to ensure annotation quality is quantifiable and reproducible.
In the zero-shot setting, 14 open-weight models and 4 frontier API flagship models all failed the DAPE policy change reasoning task.
The best-performing frontier flagship model achieved a chance-corrected Cohen's κ of about 0.29 against expert labels, far below the human ceiling of about 0.53.
After fine-tuning Qwen3-14B with LoRA SFT under five-fold cross-validation, the model reached κ = 0.524, close to the inter-coder human ceiling of κ = 0.527.
The researchers applied DAPE to analyze Chinese AI policy (1980-2024) and charted the directions of policy change across different eras.
The concept of 'entailment' in DAPE is borrowed from the natural language inference (NLI) task, treating the earlier policy as the premise and the later policy as the hypothesis, using the entailment relation to characterize the semantic position of the new policy relative to the old one.
All Facts (9)
The concept of 'entailment' in DAPE is borrowed from the natural language inference (NLI) task, treating the earlier policy as the premise and the later policy as the hypothesis, using the entailment relation to characterize the semantic position of the new policy relative to the old one.
50%UnverifiedDAPE decomposes the policy-pair judgment into two ordered decisions: a binary judgment (whether they are in the same policy domain) and a five-way judgment (continuation, broadening, narrowing, reversal, different domain).
50%UnverifiedThe DAPE benchmark contains 1000 ordered policy instruction pairs, all drawn from Chinese government policy corpora.
50%UnverifiedIn the zero-shot setting, 14 open-weight models and 4 frontier API flagship models all failed the DAPE policy change reasoning task.
50%UnverifiedThe best-performing frontier flagship model achieved a chance-corrected Cohen's κ of about 0.29 against expert labels, far below the human ceiling of about 0.53.
50%UnverifiedAfter fine-tuning Qwen3-14B with LoRA SFT under five-fold cross-validation, the model reached κ = 0.524, close to the inter-coder human ceiling of κ = 0.527.
50%UnverifiedThe researchers contributed an annotation calibration protocol (including adjudication packages and before-and-after consistency measurement) as a methodological tool to ensure annotation quality is quantifiable and reproducible.
50%UnverifiedResearchers proposed the Dual-Axis Policy Entailment (DAPE) benchmark, aimed at simultaneously evaluating policy relationships along two dimensions: 'domain' and 'direction of change.'
50%UnverifiedThe researchers applied DAPE to analyze Chinese AI policy (1980-2024) and charted the directions of policy change across different eras.
50%