Faithfulness Alignment for Theory-Scale Autoformalization: Interpreting the FaithAlign Benchmark

Autoformalization Is Not Just Compilation: Faithfulness Is the Real Bottleneck

The goal of autoformalization (AF) is to translate natural language mathematical propositions into formal statements. But a recent study points out that passing compilation does not equal correct translation—the formal statement must preserve the semantics of its informal source, a property known as "faithfulness."

Most existing evaluation methods stop at the statement level, outputting only a binary judgment of whether the whole is faithful. They cannot reveal which specific expression corresponds to which, nor can they explain which contextual interpretations support that judgment. This flaw is amplified in theory-scale settings: the meaning of a target statement often derives from surrounding mathematical material and unstated conventions, and evaluation divorced from context is almost doomed to distortion.

Autoformalization (AF) is a core task at the intersection of formal mathematics and AI. Its roots trace back to 20th-century formal mathematics practice, where projects such as Mizar and Isabelle gradually translated textbook-level mathematics into machine-verifiable proofs. Traditionally, such work relied on human labor and was extremely costly. In recent years, with large language models improving in code and mathematical reasoning, AF has been seen as a path toward scalability: a model reads a natural language proposition or proof and outputs a statement acceptable to proof assistants such as Lean, Coq, or Isabelle.

It is important to distinguish two layers in AF output: first, syntactic acceptability—the output can be parsed and type-checked by a proof assistant; second, semantic correctness—the formal statement expresses what the informal original actually means. The former is like "the code compiles"; only the latter is truly "translating correctly" in the mathematical sense. For a long time, most work reported only compilation success rates or type-checking pass rates, because these metrics are objective and automatically measurable—but this sidesteps the hardest and most critical part.

From "Holistic Judgment" to "Align Then Annotate"

The research team redefines faithfulness evaluation as a context-aware "alignment-then-annotation" problem: first identify phrase-level correspondences between informal and formal statements, then assign a faithfulness label to each correspondence pair under explicitly given contextual interpretations and assumptions.

This approach changes the granularity of evaluation. In the past it was "is the whole sentence right or wrong"; now it is "this phrase corresponds to that formal expression under what assumptions." It is theoretically more rigorous and practically more diagnosable—when a model errs, you can pinpoint the specific expression rather than receiving a vague failure signal.

Phrase-level alignment draws on ideas from word alignment and structured alignment in natural language processing, such as IBM models from the statistical machine translation era, and later alignment annotations used in semantic parsing to connect natural language fragments with logical forms. Its core assumption is that the translation correctness of a complex statement can be decomposed into several local correspondences; as long as each "informal fragment—formal expression" pair is judged faithful under the given context, the whole becomes more interpretable and diagnosable.

The "re-annotation" in "align-then-annotate" emphasizes that faithfulness is conditional: the same formal expression may be faithful or unfaithful under different assumptions. Therefore, the evaluation output is not a single True/False, but a triple structure of the form "under assumption H, informal phrase p corresponds to formal expression e and is faithful." This structure makes errors attributable and traceable, and provides finer signals for subsequent automated checking.

FaithAlign: The First Theory-Scale Faithfulness Benchmark

The paper proposes FaithAlign, claimed to be the first theory-scale faithfulness benchmark, containing 460 entries and 4,982 phrase-level annotation links. Each entry includes a target statement pair, relevant context, formal dependencies, and (when needed) assumptions specifying a given interpretation.

Also released is FaithAlign-Gap, with 146 entries, whose distinguishing feature is that it hides the informal context and assumptions. This design directly tests whether a model can recognize "undecidable" cases when information is incomplete, rather than forcing a judgment.

Theory-scale means the evaluation target is not limited to isolated lemmas or exercises, but sits within a systematic mathematical theory: definitions, theorems, conventions, and notational practices depend on one another, and the meaning of a single proposition is often determined by the overall framework of the theory it belongs to. This has two consequences: first, a statement divorced from context may admit multiple reasonable formalizations; second, whether it is "faithful" must be judged relative to a set of explicit or implicit background assumptions.

The "Gap" in FaithAlign-Gap refers to the information gap: by hiding informal context and assumptions, it constructs entries that the model should be unable to decide. This draws on the design philosophy of "answerability" and "abstention" tasks in natural language processing—a good system should not only be able to judge, but also clearly indicate uncertainty when information is insufficient, rather than producing plausible hallucinated conclusions.

Eight Models Tested: Highest Defect F1 Only 47.8%

In evaluations across 8 models, GPT-6 Astra achieved the highest defect F1 at 47.8%. Even more revealing is the fine-grained performance: among entries containing two or more defects, Astra identified all defects in only 4.5% of cases.

On FaithAlign-Gap, Astra correctly identified only 28.2% of undecidable entries as "undecidable." This shows that when context and assumptions are hidden, models tend to force a judgment rather than acknowledge insufficient information.

Defect F1 is a metric for evaluating a model's ability to identify "unfaithful correspondences," combining precision and recall: it must find as many real defects as possible (recall) without misjudging faithful correspondences as defects (precision). In faithfulness evaluation, this is a conservative metric—it penalizes both "misses" and "false alarms."

Undecidable (indeterminate) here is not undecidability in the Gödelian sense, but rather cases where faithfulness cannot be uniquely determined given the current information. Correctly outputting "undecidable" is often more valuable in evaluation than giving a guess that happens to be correct, because it reflects the model's awareness of its own knowledge boundaries. Astra identifying only 28.2% of undecidable entries means that many samples that should have been abstained from were forcibly judged faithful or unfaithful.

What These Numbers Show

A defect F1 of 47.8% means that even the strongest model still misses or misjudges more than half of the defects in phrase-level faithfulness judgments. The 4.5% all-defect identification rate is even more direct: multi-defect entries are almost completely missed.

These results point to a core difficulty—recovering context-dependent correspondences is extremely hard. The semantics of formal statements are often anchored in broader theoretical conventions, and models lack mechanisms to explicitly track these anchors. This is precisely where FaithAlign's value lies: it turns "context-aware fine-grained faithfulness evaluation" into a quantifiable, reproducible testbed.

Implications for Autoformalization Research

If faithfulness cannot be reliably evaluated, the outputs of autoformalization cannot be trusted, and downstream theorem proving and mathematical reasoning will inherit these hidden errors. The gap revealed by FaithAlign suggests two directions: first, models need to explicitly model context and assumptions rather than absorbing them implicitly; second, evaluation itself must upgrade from "scoring" to "alignment," making every judgment traceable.

For teams building mathematical AI toolchains, this means introducing phrase-level alignment checks into the pipeline, rather than just running compilation once.

One-line Summary

A new study argues that the bottleneck of autoformalization is faithfulness, and proposes the theory-scale benchmark FaithAlign.

Paragraph Summary

Autoformalization aims to translate natural language mathematical propositions into formal statements, but passing compilation does not equal correct translation. A recent study points out that existing statement-level binary evaluation cannot reveal phrase-level correspondences and is especially distorted in theory-scale settings. The research reframes faithfulness evaluation as a context-aware "align-then-annotate" problem and proposes FaithAlign, the first theory-scale faithfulness benchmark, containing 460 entries and 4,982 phrase-level annotation links. Its companion FaithAlign-Gap hides context and assumptions to test a model's ability to recognize undecidable cases. Among 8 models, GPT-6 Astra's defect F1 tops out at only 47.8%, its all-defect identification rate on multi-defect entries is only 4.5%, and it correctly identifies only 28.2% of undecidable entries. These numbers show that recovering context-dependent correspondences is extremely difficult, and evaluation must upgrade from "scoring" to "alignment."