How LLM Benchmarks Produce Comparable Scores: Dissecting a Reverse Engineering Benchmark

LLM leaderboard numbers are everywhere, but few people stop to ask: where do these scores actually come from? Why can model A's score of 78 be meaningfully compared to model B's score of 43 on the same table? The answer lies not in the models themselves, but in the design of the evaluation pipeline. Using LFORLA's Reverse Engineering benchmark as a case study, this article breaks down how an evaluation framework transforms raw model outputs into comparable numbers.

Where Comparable Numbers Come From

Comparability doesn't happen by accident — it's the output of a fixed pipeline. Using the reverse engineering benchmark as a concrete example, we can identify four key components. Understanding these four points gives you a solid foundation for judging the credibility of any leaderboard.

Same Prompt, Same Task

The benchmark in question is called Reverse Engineering (Binary to Source). The task is to reconstruct C source code from a stripped binary — one with all symbols removed. Models can choose between a tool-assisted mode or a no-tool mode. Crucially, every model receives exactly the same prompt and the same binary file. All variables are locked between runs except the model itself. This is the first cornerstone of comparability — if prompts vary across models, cross-model score comparisons lose their meaning.

A Fixed and Deterministic Judge

Scoring uses deterministic token similarity comparisons. Reference answers are stored server-side and never shown to the models. The judge doesn't improvise — it compares tokens, not "vibes." This determinism is what keeps scores comparable across both models and time. In other words, the same answer will always receive the same score, regardless of when it's submitted.

Bag-of-Words Cosine Similarity measures the cosine of the angle between two texts in token-frequency vector space: the text is broken into a set of tokens, order is ignored, and only token frequency is counted, then the cosine of the angle between two frequency vectors is computed. This method is sensitive to vocabulary overlap but insensitive to word order. Longest Common Subsequence (LCS), by contrast, finds the longest shared token sequence between two texts while preserving relative order — it is stricter about structural and sequential matching. Blending the two at a 60/40 ratio is an engineering tradeoff between "breadth of coverage" and "structural fidelity": the cosine component rewards broad vocabulary overlap, while the LCS component penalizes outputs whose vocabulary looks similar but whose logical order is scrambled. There is no theoretically optimal weighting here — this is a subjective design choice by the benchmark creators, something the original post honestly acknowledges in its limitations section.

Multi-Axis Scoring

A single number hides too much information, so this benchmark reports across four dimensions:

• Completeness: The percentage of reference answer tokens covered by the submission, from 0 to 100% — higher is better.
• Code Similarity: A blended token similarity score, composed of 60% Bag-of-Words cosine similarity and 40% Longest Common Subsequence (LCS) ratio.
• Percent Reversed: An overall blended score, again 60% cosine and 40% LCS.
• Function Recovery: A binary threshold — a function only counts as recovered if token similarity meets or exceeds 0.75.

The value of this multi-axis design is that it tells you where a model genuinely excels and where it's essentially guessing.

The 0.75 binary threshold used for Function Recovery carries special significance in a reverse engineering context. Source code functions have strong structural constraints: variable names, control flow, and return types all matter. A function with a token similarity of only 0.6 may already represent completely different logic in the compiler's eyes. Setting a hard pass/fail line — rather than a continuous score — simulates engineering reality: code either works or it doesn't. There's no "kind of works." The side effect of this binary design is that it's very unforgiving to models near the threshold: 0.74 and 0.80 are worlds apart in score, yet the actual capability gap may be negligible. When reading leaderboards, watching for these thresholds often explains why certain models rank differently than intuition would suggest.

Publicly Archived Verbatim Submissions

Every submission is preserved, and anyone can go back and review what a model actually said. If a score looks suspicious, you can open that submission and check the tokens yourself. This is the fundamental difference between a leaderboard and a black box — auditability.

Real Score Comparison

The original post provides empirical data for two models:

Model	Provider	Score
Nemotron 3 Ultra (free, via opencode)	deepseekAd	43.49
GLM 5.2 (author's own submission)	lforla	78.0

It's worth noting that GLM 5.2 is the author's own submission, and the original post honestly acknowledges that this number should be scrutinized with the same critical eye as any other model's results.

What the Gap in Rankings Actually Means

The gap between 78 and 43 is not rounding noise — it directly reflects how much of the reference source code each model actually reconstructed. A score near 78 means the submission covered most of the reference answer's tokens and matched fairly tightly on the 60% cosine + 40% LCS blended metric. A score near 43 means roughly half of the reference content was missing or misaligned.

The Function Recovery axis is even more demanding: it only triggers when token similarity reaches 0.75. This means a model might look decent on completeness yet still fail to clear this binary threshold. All the dimensions together are what truly sketch out a model's capability profile.

How Practitioners Should Use This to Choose a Model

Don't select a model based on a single headline number. Instead, focus on the dimension that best matches your workload.

If you need complete source code reconstruction, prioritize Completeness and Function Recovery. If you need tight token-level fidelity, look at Code Similarity. If you just want a rough signal, check Percent Reversed. Then go read the verbatim submissions for the cases you actually care about.

A model that performs brilliantly on simple binaries but falls apart on hard samples is a completely different beast from a model with consistent, stable performance. Averages routinely smooth over this distinction.

Five Steps for Reading a Leaderboard

The original post offers a practical checklist:

1. Confirm that the prompts and judge are fixed. If either changes between runs, the numbers aren't comparable.
2. Understand what each axis measures before looking at rankings. Blended scores obscure how weights are allocated.
3. Look for binary thresholds like Function Recovery — they tell you whether a model truly solved the task.
4. Open at least one verbatim submission to verify the score matches the actual answer.
5. Compare models on the dimension that fits your use case, not just the overall ranking.

Honest Limitations

The original post proactively lists the boundaries of its methodology — which is itself a mark of credibility:

Token similarity does not equal semantic correctness. A submission may match tokens yet still be structurally wrong. The 60% cosine + 40% LCS blend is a design choice, not a universal truth. The reverse engineering benchmark covers only one family of tasks; results don't automatically transfer to other code or reasoning tasks. The leaderboard shown here is a snapshot — scores will shift as more models are submitted.

Reverse engineering as an evaluation task has a noteworthy property worth flagging: the reference answer is typically unique (the original source code), but in practice the same binary can be compiled from multiple semantically equivalent C sources. Token similarity comparisons will treat these equivalent alternatives as errors, because their token sequences differ from the reference. This is a fundamental dilemma facing all string-matching-based code evaluation — not something unique to LFORLA. A more rigorous approach would recompile model outputs and execute them, comparing binary behavior rather than source text. But that requires a fully executable test environment, with far higher engineering costs than token comparison. Understanding this limitation helps position the benchmark's scores as measuring "text-level reconstruction ability" rather than "semantic-level reverse engineering comprehension."

Conclusion

LLM evaluation is a mechanism, not a mystery. Identical prompts, a fixed and deterministic judge, multi-axis scoring criteria, and publicly archived verbatim submissions — these are the four pillars that transform raw answers into comparable numbers. For anyone who wants to seriously interpret model capabilities, understanding this mechanism and learning to decompose scores by dimension is a far more reliable approach than fixating on who sits at the top of the leaderboard.