Gemini Decision Closure Benchmark Results: Interpreting 99.3% Pass Rate Across 285 Runs

An Overlooked Evaluation Dimension: Decision Closure Errors

Most LLM evaluations focus on whether a model can produce the correct answer, but a researcher published a novel benchmark on Zenodo that measures something different — not "what did it get right," but whether the model can correctly "close" a decision.

"Decision Closure" refers to whether an LLM retains all conditions necessary to legitimately close a decision when evidence, applicability, identity, conflict resolution, transfer rules, context, or record state undergo changes. In other words, this benchmark examines the model's logical consistency and state management capabilities under complex condition transitions, rather than simple knowledge Q&A.

This benchmark, named CFC (Conditional Closure) by its author, has a historical sequence spanning V1 through V100. The final reproducible test set includes 95 variants with a total of 285 repeated runs.

Core Data: 283 Pass, 2 Partial Pass, 0 Semantic Failures

Across the 285 test runs, Gemini's results were as follows:

• 283 semantic PASS
• 2 semantic PARTIAL
• 0 recoverable semantic FAIL

This yields a strict semantic pass rate of 99.3%. The strongest-performing fully retained block, V78–V100, achieved a perfect 69/69 semantic pass score.

This number may seem stunning at first glance, but the author himself provides important qualifications with considerable restraint.

Data Transparency: The Author's Acknowledged Imperfections

To the researcher's credit, he did not attempt to embellish the data archive. He explicitly notes:

• V1–V4 are unrecoverable and have been excluded
• V6 is an early unstable baseline that cannot be precisely aggregated

In other words, the 99.3% figure is derived from the "reconstructable and scoreable" subset, not from all 100 versions. This kind of honesty about data gaps is actually rare in an AI evaluation landscape saturated with marketing-driven benchmarks.

The Most Critical Methodological Insight: Semantic Correctness ≠ Format Compliance

The author states that the most valuable finding from the entire experiment was separating semantic correctness from output format compliance in scoring.

This is a distinction that evaluators frequently conflate, yet it is critically important. He found:

Gemini was often semantically correct even when it violated the required serialization format.

This means that if an evaluation marks "didn't output in JSON format" as a direct failure, it would severely underestimate the model's actual reasoning capability. The model's "errors" are often not about thinking incorrectly, but about not packaging the answer in the way you requested.

Why Separating Semantic and Format Scoring Matters So Much

In practical engineering applications, format issues can often be mitigated through post-processing, constrained decoding, or retry mechanisms, whereas semantic errors represent fundamental defects that are difficult to repair. Conflating the two leads to systematic bias in our assessment of model capabilities — either overestimating (treating format workarounds as correct reasoning) or underestimating (treating format violations as reasoning failures).

By explicitly splitting these two dimensions, this benchmark provides a cleaner scoring framework for subsequent cross-model comparisons.

Frozen Benchmark Design: Paving the Way for Fair Cross-Model Comparison

The author states that this first benchmark will be "frozen" and kept unchanged. The next step is to run Claude and Grok on the same frozen test set, completing this before introducing any CFC rule changes to ensure fairness in cross-model comparison.

This is a commendable experimental design principle: fix the test standard first, then compare horizontally, avoiding data contamination from "changing rules while testing." Only when the test set is completely identical do the numbers between Gemini, Claude, and Grok become truly comparable.

Maintaining a Cautious Interpretive Stance

A point the author repeatedly emphasizes — and the one readers most need to remember:

99.3% is not a universal "CFC accuracy" value, nor is it a general reliability score for Gemini. It is only the score on this specific, reproducible decision closure benchmark.

This reminds us that high scores on any single benchmark should not be over-interpreted. A specifically designed boundary test set can only represent model performance on that particular class of failure modes and cannot be extrapolated to general capabilities.

Furthermore, this remains an exploratory benchmark published by an individual researcher. The sample size (285 runs, 95 variants) is statistically moderate and has not yet undergone large-scale peer review. The author himself is publicly soliciting criticism of his methodology, scoring strategy, and boundary test category design.

Conclusion: Small but Rigorous Evaluations Deserve More Attention

In an era where large model evaluations are increasingly becoming an "arms race," work like this — focused on specific failure modes, transparent about data gaps, and committed to frozen benchmarks for fair comparison — actually offers greater reference value.

It doesn't chase a headline-grabbing "comprehensive capability leaderboard." Instead, it answers a specific and profound question: when the conditions underlying a decision change, can the model maintain the boundaries of logical closure?

For any developer building LLM applications that require rigorous state management and decision consistency (such as workflow engines, compliance review, multi-turn reasoning systems), this methodology — especially the separation of semantic and format scoring — is worth adopting. The complete data has been published on Zenodo (record number 22045494), and more researchers are welcome to participate in verification and critique.