HakemBench
A typed-decision benchmark for the Turkish language that requires models to return probability values for each answer option. It evaluates three dimensions—decision quality, calibration, and selective automation—and is released under the CC BY 4.0 license.
Timeline (last 90 days)
HakemBench is a "typed decisions" benchmark targeting the Turkish language.
HakemBench's task format requires the evaluated model to read a text, a question, and fixed options, and return a probability value for each option rather than providing a single answer.
HakemBench Version 1.0 is fully open-sourced under the CC BY 4.0 license.
HakemBench Version 1.0 contains 2,346 entries and 4,275 questions, covering three question types: multiple-choice, yes/no, and scoring.
HakemBench includes seven tracks: fact-checking triage, education, safety guardrails, legal routing, content moderation, spam and phishing detection, and customer support.
HakemBench evaluates models across three dimensions—decision quality (macro F1), calibration (normalized Brier score), and selective automation (normalized area under the generalized risk-coverage curve)—and combines them into a composite score via geometric mean.
HakemBench's composite scores are given with confidence intervals based on 2,000 bootstrap samples, and robustness probes for perturbations such as option order, paraphrasing, English translation, and name replacement are reported.
Most of HakemBench's gold labels are not human-verified but are derived from blind-test results from one AI model family and then compared against LLM jury votes from other model families.
On HakemBench's 16-row leaderboard, the top performer achieved a composite score of 0.888.
The lab that published HakemBench ranked its own model 7th with a composite score of 0.660.
2 more timeline events
All Facts (12)
When all models are scored only on the remaining four tracks, the lab's model's composite score becomes 0.678, ranking it 6th.
50%UnverifiedHakemBench's task format requires the evaluated model to read a text, a question, and fixed options, and return a probability value for each option rather than providing a single answer.
50%UnverifiedHakemBench is a "typed decisions" benchmark targeting the Turkish language.
50%UnverifiedHakemBench Version 1.0 contains 2,346 entries and 4,275 questions, covering three question types: multiple-choice, yes/no, and scoring.
50%UnverifiedHakemBench includes seven tracks: fact-checking triage, education, safety guardrails, legal routing, content moderation, spam and phishing detection, and customer support.
50%UnverifiedHakemBench evaluates models across three dimensions—decision quality (macro F1), calibration (normalized Brier score), and selective automation (normalized area under the generalized risk-coverage curve)—and combines them into a composite score via geometric mean.
50%UnverifiedHakemBench's composite scores are given with confidence intervals based on 2,000 bootstrap samples, and robustness probes for perturbations such as option order, paraphrasing, English translation, and name replacement are reported.
50%UnverifiedMost of HakemBench's gold labels are not human-verified but are derived from blind-test results from one AI model family and then compared against LLM jury votes from other model families.
50%UnverifiedOn HakemBench's 16-row leaderboard, the top performer achieved a composite score of 0.888.
50%UnverifiedThe lab that published HakemBench ranked its own model 7th with a composite score of 0.660.
50%UnverifiedThe lab that published HakemBench's own model results were not blind-tested, and early test results had influenced the model's training data, causing its scores on three tracks—safety guardrails, content moderation, and customer support—to be flagged as potentially contaminated.
50%UnverifiedHakemBench Version 1.0 is fully open-sourced under the CC BY 4.0 license.
50%