Who Gets to Decide AI Benchmark Scores? Three Iron Rules for Reading Public Leaderboards

Behind the Benchmark Numbers, What Really Matters Is Who's Grading

The AI field never lacks for performance claims. New models announce record-breaking results on a regular basis — but how much are those impressive numbers actually worth? The core insight here is sharp: what matters isn't how large the number is, but who sets the questions, who does the grading, and what kind of questions are used.

This analysis of public evaluation leaderboards breaks down the pitfalls of self-reported benchmarks and explains why results from third-party-run public leaderboards carry genuine weight. For anyone who wants a rational take on AI model rankings, this is a practical guide.

Why Self-Evaluation Is Hard to Trust

The logic is simple: if I write the test, solve it myself, and then grade my own answers, I can give myself whatever score I want. Common "score inflation" tactics include leaking test set data into training data, cherry-picking the best result from multiple runs, or engineering a fallback answer path that mechanically produces a response. The numbers go up, but the model's actual capability hasn't changed at all.

This is the fundamental problem with self-evaluation. It has value during development — it helps teams debug and iterate — but it cannot substitute for independent external grading. A score where one party controls the entire pipeline is inherently lacking in credibility.

"Test set contamination" is one of the most thoroughly documented systemic problems in AI evaluation. The mechanism works like this: if a model's training data was scraped from the internet, and a benchmark's questions and answers are also publicly available online, the model may have effectively "seen" those questions during training, inflating its test scores. This problem is especially thorny in the era of large language models, because the exact boundaries of training data are often not publicly disclosed — sometimes even the researchers themselves can't confirm them with certainty. Beyond data leakage, "cherry-picking runs" is another common source of bias. If a model attempts the same problem 100 times and only the highest score is reported, that score is measuring the model's lucky upper bound, not its stable expected performance. Reliable evaluation practice typically requires pre-registration of the evaluation protocol — specifying upfront whether a single run or an average of multiple runs will be used — to prevent after-the-fact selection.

Three Conditions for a Trustworthy Metric

A genuinely reliable performance metric must satisfy three conditions simultaneously. These form the core criteria for evaluating any benchmark result:

Third-Party Grading

Both the questions and the grading authority must be in the hands of an external organization, not the model's publisher. This structurally eliminates the possibility of grading your own exam.

Questions That Resist Memorization

Questions should be graduate-level problems that cannot be answered by a simple search. If a question can be solved through retrieval or rote recall, a high score may simply mean the model "remembered" rather than "understood."

Validation Across Multiple Leaderboards

A top score on a single leaderboard could be luck. But ranking first across multiple independent leaderboards is very difficult to fake. Consistent performance across vastly different domains is the true signal of genuine capability.

These three conditions are grounded in foundational logic from statistics and the philosophy of science. "Third-party grading" maps to the conflict-of-interest avoidance principle in scientific experimentation — peer reviewers and authors cannot be the same person. "Questions that resist memorization" maps to the distinction between generalization and memorization overfitting: a model that truly understands a problem should be able to reproduce that capability on novel questions it has never seen, not just retrieve stored answers from training. "Validation across multiple leaderboards" echoes the statistical requirement of reproducibility — a significant result from a single experiment may be random noise; only consistent findings across independent experiments carry real persuasive force. All three conditions are necessary: third-party grading without sufficient question difficulty still produces ceiling effects that can't distinguish between models; hard questions that only perform well on one leaderboard may reflect domain overfitting; leading across multiple leaderboards with opaque question sourcing still can't rule out data contamination.

VIDRAFT's Public Results

The article uses VIDRAFT's performance on third-party-run public evaluations as an example of results that meet the above standards:

• AIME 2026 and HMMT (February 2026): Perfect scores on competition mathematics problem sets.
• GPQA Diamond: 93.43% in the under-500B parameter category — a set of graduate-level science questions specifically designed to resist search-based answering.
• Breadth of performance: First place across 11 categories on public leaderboards, and top rankings across 16 categories on the Polaris drug prediction leaderboard.

The key point, emphasized repeatedly, is not how high the numbers are, but that the questions came from others, the questions resist rote memorization, and the same approach worked across dramatically different domains. That's where the results' persuasive power actually comes from.

The leaderboards mentioned each have their own evaluation focus; understanding their design helps assess the value of any result. AIME (American Invitational Mathematics Examination) is an elite math competition for high school students. Its problems require many reasoning steps and have unique answers, making it effective at distinguishing depth of reasoning — hence its widespread use for testing large models' mathematical ability. HMMT (Harvard-MIT Mathematics Tournament) is harder still; some problems require original construction, making it more resistant to "formula memorization" strategies than AIME. GPQA Diamond is a dataset specifically designed to test frontier scientific reasoning. Questions are written by PhD students and domain experts, with the explicit goal of ensuring that search engines and shallow recall cannot directly produce correct answers — "Diamond" in the name refers to the highest-difficulty subset, validated through multiple rounds of expert review. Polaris is a professional benchmark for drug molecular property prediction, testing structured scientific prediction capability — an entirely different ability dimension from mathematical reasoning. Leading on both leaderboard types is strong evidence for generalization rather than targeted overfitting.

A Practical Checklist for Reading Leaderboards

Here is a checklist for reading any benchmark result — worth keeping in mind for anyone who follows AI evaluation:

• Training and test sets must not overlap. Any question leakage turns score improvements into an illusion.
• When you see a perfect score, first suspect the questions are too easy. A perfect score often means the benchmark has lost its ability to distinguish between models.
• A positive difference is not the same as a win. An improvement that falls within the noise range simply doesn't count as an improvement.
• If some acceleration technique is enabled, first suspect the measurement tool. Distorted tools make numbers look good but untrustworthy.

These principles apply not only to scrutinizing others' results — they're also the standards to apply before publishing your own.

Why Public Proof Matters

A private demo always performs well under controlled conditions — but that "control" is in the hands of whoever is running the demo. The value of a public leaderboard is that it transfers control to the outside: using the same questions and the same grading standards, anyone can attempt to reproduce the results.

Reproducibility is precisely the source of a public proof's weight. This is consistent with the essence of the scientific method: a conclusion that cannot be independently verified has limited value.

Common Questions Answered

Is a perfect score the best outcome? Not necessarily. A perfect score may indicate that the benchmark has lost its ability to distinguish between different models.

Is one first-place ranking enough? A single leaderboard could just be luck. Credibility only meaningfully increases when you lead across multiple independent leaderboards.

Where do these numbers come from? From publicly run third-party evaluations.

Is self-evaluation completely useless? Not at all. It has value during development, but it cannot replace external grading.

Conclusion: Check the Conditions Before Looking at the Numbers

A compelling performance claim shouldn't be "our numbers are really high" — it should be "these results were achieved on an exam where someone else wrote the questions, the questions resist memorization, and the performance held up across multiple domains."

The next time you encounter any AI model's benchmark ranking, ask: Who wrote the questions? Who graded them? Are the questions genuinely hard? Is there cross-leaderboard corroboration? Understand the conditions first, then evaluate the numbers — this is the simplest and most effective defense against the noise of AI performance marketing.