[KongchangAI]
· 1 min read· 961 words

What Do Small Models Learn Under an Exact Verifier?

What Do Small Models Learn Under an Exact Verifier?

An exact verifier reveals small models can learn sufficiency-checking procedures but struggle to transfer to new policies.

Using minimal sufficient reasons—exactly computable over an enumerable context space—as a zero-error verifier, the study examines what small language models learn. A 4B model fine-tuned on 2000 deletion-algorithm trajectories reaches 92% accuracy on the training policy, and algorithm teaching beats frontier-model teaching across all seeds. Yet the student learns a rigid sufficiency-checking schedule that does not transfer: on unseen policies, answers stay sufficient (94%) but are no longer minimal (48%). RL with exact rewards reverses the probing policy but does not improve held-out accuracy.

When an authorization policy blocks an agent from calling a tool, the audit log should list only the facts that forced the decision—nothing more, nothing less. For executable policies over an enumerable context space, this minimal sufficient reason can be computed exactly, so explanations can be scored against a ground truth rather than relying on proxy metrics. Researchers leverage this exactness to explore what a model can learn from a zero-error verifier.

rss source: Procedure without Judgment: What Small Language Models Learn under an Exact Verifier

Experimental Setup: Interpreter and Thirty Rounds of Interaction

The interpreter sees only a single request and a single decision, never the policy itself, but has thirty rounds of interaction: it can call the policy on modified requests and ask whether a set of candidate facts is sufficient. Frontier models using tools alone find the exact reason in about half of cases, and a single "narrow your answer" prompt boosts Haiku 4.5's accuracy from 58% to 75%.

This setup strips away policy visibility and focuses the learning target on the meta-skill of "sufficiency checking." The model must approximate the truth through active probing rather than memorizing policy rules.

Minimal sufficient reason is a core concept in the field of formal explanation: given a policy P and a denial decision d, a set of reasons R is sufficient if and only if, over the enumerable context space of the policy, any request satisfying R leads to the same decision d; R is minimal if and only if removing any fact from R makes it no longer sufficient. Unlike proxy metrics common in machine learning such as attention weights or saliency maps, a minimal sufficient reason has an exact set-theoretic definition and can be computed via exhaustive search over a finite context space to yield a unique answer, thus serving as a zero-error verifier that directly scores model outputs.

Teacher Type Determines Learning Outcomes

A 4B model fine-tuned on 2000 deletion-algorithm trajectories reaches 92% accuracy on the training policy. What really drives performance is the teacher type: at matched data volume, students taught by the algorithm outperform students taught by a frontier model across all three training seeds, and the two groups' results never overlap.

What the student learns is a fixed sufficiency-checking schedule, and this schedule does not transfer. On three unseen public policies, its answers remain sufficient (94%) but are no longer minimal (48%), whereas Haiku remains exact on all 47 segments under the same prompt.

Deletion-based algorithm is the classic method for computing minimal sufficient reasons: starting from the full set of request facts, try deleting each fact one by one and re-invoke the policy; if the decision is unchanged after deletion, the fact is redundant and can be permanently removed; iterate until no fact in the remaining set can be deleted, yielding the minimal sufficient reason. In the worst case with n facts, this algorithm requires O(n²) policy calls but guarantees an exact minimal solution. The teacher trajectories in this study are automatically generated by this algorithm on the training policy, so "algorithm teaching" is essentially using an exact computational process rather than model intuition to provide the supervision signal.

Reinforcement Learning and the Transfer Dilemma

Reinforcement learning based on exact rewards keeps held-out accuracy at the same level but reverses the policy: from a probing strategy to spending the entire budget on sufficiency checks. The exact verifier makes training outcomes readable, but does not make training itself better.

This reveals a deeper tension: exact rewards can clearly measure behavioral change, but the procedural knowledge acquired by small models is highly dependent on the training distribution and hard to generalize to new policies. Although algorithm teaching is more efficient, it is likewise constrained by the rigidity of the schedule.

Policy reversal here refers to a qualitative change in the model's behavioral pattern before and after reinforcement learning, not a rise or fall in accuracy numbers. Before training, the model tends to make a few probing calls first and then give a guessed reason; after training, the model learns to spend the entire thirty-round budget on sufficiency checks of candidate facts, and probing behavior nearly disappears. This shows that the reward signal does shape behavior, but because the reward only measures whether the final answer is correct and does not penalize process redundancy, the model converges to a local optimum of "checking to the end," with held-out policy accuracy on par with pre-training.

Implications for AI Programming and Agent Development

For developers building authorization-policy explanation systems, this study suggests: small models can efficiently learn a fixed procedure for sufficiency checking, but cross-policy deployment still requires frontier models or richer training distributions. The exact verifier is a diagnostic tool, not a panacea.

In AI programming scenarios, if a small model is used to generate audit logs, ensure the policy space matches the training distribution; otherwise it may produce sufficient but non-minimal explanations, increasing the audit burden.

One-line Summary

An exact verifier reveals that small models can learn the sufficiency-checking procedure but struggle to transfer to new policies.

Paragraph Summary

This study uses minimal sufficient reasons—exactly computable over an enumerable context space—as a zero-error verifier to examine what small language models can learn. The experiment has an interpreter, blind to the policy, approximate the exact reason through thirty rounds of interactive probing. Results show that a 4B model fine-tuned on 2000 deletion-algorithm trajectories reaches 92% accuracy on the training policy, and algorithm teaching leads frontier-model teaching across all seeds. However, what the student learns is a fixed sufficiency-checking schedule that does not transfer: on unseen policies, answers remain sufficient (94%) but are no longer minimal (48%). Reinforcement learning with exact rewards reverses the probing policy but does not improve held-out accuracy.

#exact verifier#minimal sufficient reason#small language models#algorithm teaching#reinforcement learning#procedural knowledge transfer
Share:

Related articles