[KongchangAI]
· 2 min read· 1,061 words

GT-FENG: A New Framework for Rule Attribution in Language Model Factual Evaluation

GT-FENG: A New Framework for Rule Attribution in Language Model Factual Evaluation

GT-FENG evaluates contextual correctness in rule-governed domains through documented rule attribution, reducing error confusion.

GT-FENG is a general framework unifying contextual/temporal correctness with rule-based error attribution for model evaluation in rule-governed domains like law and policy. Traditional evaluation only compares answer values, ignoring potential rule misuse. GT-FENG uses a two-step mechanism: checking answer correctness against documented rules, then retrieving matching rules to attribute errors (rule confusion, outdated rules, or hallucination). In Medicaid income threshold experiments, introducing the queried state's documented records reduced cross-state rule confusion errors by 68-78%, with over half of stable errors matching two or more documented sources. Open-sourced as Python package gtfeng, it offers a shift from "answer accuracy" to "rule faithfulness" in rule-intensive domains.

Whether a language model's answer is correct often depends on the specific context—the same value may be correct under one jurisdiction, date, or legal definition, but wrong under another. This context-dependence makes model evaluation tricky: looking only at the answer itself, it's hard to tell whether the model truly understands the rules or just happened to guess the right number. To address this pain point, researchers have proposed GT-FENG (General Theory of Factual Evaluation and Normative Grounding), a general framework that unifies contextual/temporal correctness with rule-based error attribution.

The Evaluation Dilemma: Rule Dependence Behind Answer Correctness

In rule-governed domains (such as law, policy, and social welfare), factual correctness is rarely absolute. Take Medicaid income thresholds as an example: each state has its own standard, so the same applicant may qualify in one state and not in another. Traditional evaluation only checks whether the answer matches the reference answer's value, ignoring the possibility that the model may have borrowed rules from another jurisdiction. The core insight of GT-FENG is that when multiple rules yield the same value, the answer alone cannot distinguish which rule the model actually used.

GT-FENG framework diagram

This perspective advances evaluation from "does the answer match" to "is the answer supported by the applicable rule," providing a more granular measure of model reliability in rule-intensive scenarios.

Rule-governed domains refer to domains where correctness and incorrectness are defined by explicit, written rules (such as legal statutes, administrative regulations, and policy guidelines). In such domains, the correct answer is not a universal constant but changes with rule version, applicable jurisdiction, and time. For example, Medicaid income thresholds are set by each state individually and may be adjusted year by year. Therefore, when evaluating a model, one cannot simply check whether the final value matches; one must also confirm whether the model relied on the correct rule. This contrasts with open-domain question answering, where facts are relatively stable, and requires evaluation methods that can track rule provenance and applicability conditions.

Framework Design: Rule Matching and Error Attribution

GT-FENG's mechanism works in two steps. First, it checks the correctness of the answer against applicable rules—rules that are documented and traceable, rather than vague knowledge implicit in model parameters. Second, for incorrect answers, the framework searches for whether any documented rule exactly matches the erroneous value, thereby determining whether the error stems from rule confusion, outdated rules, or pure hallucination.

This attribution capability makes evaluation results more actionable. Developers can know not only that the model was wrong, but also which rule it went wrong on, enabling targeted improvements to retrieval augmentation or fine-tuning strategies.

Retrieval-Augmented Generation (RAG) is a technical paradigm that combines external knowledge retrieval with language model generation. In rule-intensive scenarios, a RAG system first retrieves relevant rule provisions from a document repository, then generates answers based on the retrieval results. GT-FENG's attribution approach is complementary: it not only evaluates whether the answer is right or wrong, but also compares the answer with documented rules to determine whether the model is "faithful" to the retrieved rules, thereby distinguishing whether the model correctly used the rules or just happened to match the value. This combination helps developers optimize retrieval strategies or adjust model prompts to reduce rule confusion.

Empirical Findings: Documented Records Significantly Reduce Rule Confusion

The researchers conducted Medicaid income threshold question-answering experiments on two models. When documented records of the queried state itself were introduced, the number of incorrect answers that matched only other states' values dropped by 68%–78%. This means a considerable portion of errors stem from the model confusing rules across states, rather than complete fabrication. More notably, more than half of the stable incorrect answers, after documented sources were introduced, matched two or more documented sources—suggesting that the model may be wavering between different rules, or that there are conflicting signals in the training data.

This result indirectly demonstrates that providing authoritative, structured rule documents has substantive value for improving model performance in policy-related question answering.

Stable incorrect answers refer to incorrect answers that persist across different evaluation settings or multiple runs, rather than occasional errors arising from random fluctuations. Such errors often reflect systematic biases in the model's internal knowledge or reasoning process. In the GT-FENG experiments, more than half of stable incorrect answers, after documented sources were introduced, matched two or more documented rules, suggesting that the model may be wavering between different rules or that the training data contains conflicting signals. Identifying stable errors helps locate the model's knowledge deficiencies and guides targeted data cleaning or rule alignment.

Open-Source Tooling and Future Directions

GT-FENG has been released as an open-source Python package gtfeng, allowing researchers and developers to reproduce experiments or integrate it into their own evaluation pipelines. The authors position it as infrastructure for model evaluation in rule-governed domains and look forward to future benchmarks and evaluation toolkits being built on this foundation.

For teams focused on deploying AI in high-stakes scenarios such as law and policy, GT-FENG offers an evaluation approach that shifts from "answer accuracy" to "rule faithfulness." It reminds us that in rule-intensive domains, a correct answer without the correct rule supporting it may be merely a statistical coincidence rather than genuine understanding.

As language models increasingly intervene in professional decision support, fine-grained evaluation frameworks like this will become an indispensable part of trustworthy AI.

One-line Summary

GT-FENG evaluates contextual correctness in rule-governed domains through documented rule attribution, reducing error confusion.

Paragraph Summary

GT-FENG is a general framework that unifies contextual/temporal correctness with rule-based error attribution, aiming to solve the challenge of model evaluation in rule-governed domains (such as law and policy). Traditional evaluation only compares answer values, ignoring the possibility that the model used the wrong rule. GT-FENG uses a two-step mechanism: first, it checks answer correctness against documented rules; then, for incorrect answers, it retrieves matching rules to achieve error attribution (rule confusion, outdated rules, or hallucination). In Medicaid income threshold question-answering experiments, after introducing documented records of the queried state itself, incorrect answers that matched only other states' values decreased by 68-78%, and more than half of stable incorrect answers matched two or more documented sources, suggesting rule confusion or training data conflicts. GT-FENG has been open-sourced as the Python package gtfeng, offering a new approach for model evaluation in rule-intensive domains that shifts from "answer accuracy" to "rule faithfulness."

#GT-FENG#language model evaluation#rule attribution#factual accuracy#contextual correctness#rule-governed domains#Medicaid income thresholds#retrieval-augmented generation#RAG#rule confusion#open-source Python package#trustworthy AI
Share:

Related articles