HakemBench: A Deep Dive into the Turkish "Typed Decisions" Benchmark

What Is HakemBench

HakemBench is a "typed decisions" benchmark designed for the Turkish language. Its core task format differs somewhat from typical Q&A benchmarks: the model under evaluation reads a passage, a question, and a fixed set of options, then returns a probability value for each option rather than simply selecting a single answer.

This design allows HakemBench to measure not only whether a model answers correctly, but also whether its confidence in its own judgments is reliable — precisely the capability most lacking in real-world decision-making tasks.

The current Version 1.0, released under the CC BY 4.0 license, is fully open-source and contains 2,346 entries and 4,275 questions. Question types include multiple-choice, yes/no, and scoring, distributed across seven tracks.

"Typed Decisions" is a task format that structures decision outputs as discrete options with probability distributions, distinguishing it from open-ended generation tasks. Under this paradigm, a model must not only select the most likely answer but also assign a normalized confidence probability to each candidate option — formally similar to a softmax output in classification tasks, except that evaluation focuses on the quality of the probabilities themselves rather than just whether the argmax is correct. This design enables the benchmark to simultaneously capture a model's discriminative ability and its capacity for uncertainty quantification. The latter is critical in high-stakes automation scenarios such as medical diagnostic assistance, legal risk assessment, and content compliance review — contexts where an overconfident wrong answer is far more harmful than an honest "I'm not sure."

Seven Tracks and Evaluation Dimensions

The Seven Task Tracks

HakemBench's seven tracks are closely aligned with real-world application scenarios:

• Fact-check triage
• Education
• Guardrails
• Legal routing
• Moderation
• Spam and phishing detection
• Customer support

Most of these tracks belong to high-risk domains requiring "trustworthy decisions," where errors or overconfidence carry real costs. Simple accuracy scores are therefore insufficient.

A Three-Dimensional Scoring Framework

HakemBench uses a unified evaluation harness to assess model performance across three dimensions, then combines them into a composite score via geometric mean:

1. Decision quality: Measured by macro F1, focusing on balanced performance across all categories;
2. Calibration: Based on normalized Brier score, measuring whether the model's output probabilities match its actual accuracy;
3. Selective automation: Based on the normalized area under the generalized risk-coverage curve, measuring how well the model trades off between "auto-processable" and "requires human intervention" cases.

The composite score is also accompanied by confidence intervals derived from 2,000 bootstrap samples, along with robustness probes for perturbations such as option order shuffling, paraphrasing, English translation, and name substitution. This level of methodological rigor is rare among benchmarks for lower-resource languages.

Calibration refers to the consistency between a model's output probabilities and its actual accuracy. For example, a well-calibrated model that outputs "80% confidence" should have an actual accuracy of around 80% on similar questions. The Brier score is a classic metric for measuring probabilistic prediction error, calculated as the mean squared difference between predicted probabilities and true outcomes (0 or 1) — lower is better. HakemBench uses a normalized version that enables cross-comparison across different question types.

Selective Automation addresses a more practical question: when a model is allowed to "abstain" and escalate uncertain cases to humans, is the overall risk-coverage trade-off of the system reasonable? The risk-coverage curve plots "the proportion of cases handled automatically" on the x-axis against "the error risk at that coverage level" on the y-axis. A smaller area under the curve indicates more reliable judgments in high-confidence regions. This dimension directly maps to the core deployment decision of "when to trust the model and when to escalate to human review."

An Important Note on Label Sources

HakemBench has a transparency detail that is both worth flagging and worth applauding: most gold labels were not human-verified.

These labels were derived from blind-test results from one AI model family, then cross-referenced against votes from an LLM jury composed of models from other families. In other words, the labels are essentially "model consensus" rather than "human consensus."

This means that when interpreting HakemBench scores, one must recognize that the evaluation's reference frame carries inherent model preferences. For tracks like legal routing and moderation, where annotation quality is especially critical, this is a limitation that must be factored in. The benchmark team's proactive disclosure of this fact actually enhances the interpretability of the results.

In NLP benchmark construction, "gold labels" typically refer to reference data annotated by human experts and treated as ground truth. Using model consensus to construct gold labels is an alternative approach that has grown in popularity due to the high cost of human annotation — sometimes called the "LLM-as-annotator" paradigm. Its advantages include fast scalability and low cost, but an inherent risk is that if multiple reviewer models share similar training data or alignment approaches, they may systematically agree on the same class of errors, making label bias difficult to detect through majority voting. For tasks like legal routing or content moderation, there may be systematic gaps between normative human judgment (e.g., interpretation of legal provisions, culturally contextualized determinations of offensiveness) and model tendencies — something to approach with extra caution when using this benchmark.

The Leaderboard and the "Self-Evaluation" Problem

On the 16-row leaderboard, the top performer achieves a composite score of 0.888, while the benchmark-publishing lab's own model ranks 7th with a composite score of 0.660.

The team candidly acknowledges that their model's results were not blind-tested. More critically, early test results had influenced that model's training data, so its scores on the guardrails, moderation, and customer support tracks are explicitly flagged as "potentially contaminated."

To provide a fairer comparison, when all models are scored only on the remaining four tracks, the lab's model achieves a composite score of 0.678, ranking 6th. This kind of proactive disclosure of potential data contamination is a rare act of self-discipline in benchmark development.

Why HakemBench Is Worth Paying Attention To

HakemBench's value lies not only in filling the gap in Turkish-language evaluation, but also in demonstrating a more mature benchmark philosophy:

• It goes beyond accuracy to ask whether the model "knows it might be wrong";
• It uses robustness probes to guard against data leakage and surface-level pattern matching;
• It maintains a high degree of transparency about label sources and self-evaluation bias.

For researchers and engineering teams focused on multilingual AI, model calibration, and safety/compliance deployments, this is an open-source resource whose methodological value outweighs its absolute scores. As more applications enter high-stakes decision-making scenarios, evaluation paradigms like HakemBench's — emphasizing calibration and selective automation — may well become the mainstream direction for future benchmark design.