IFEval
A benchmark for measuring the instruction-following capability of large language models. It evaluates how accurately models execute complex instructions by applying verifiable constraints that allow objective assessment of compliance.
Timeline (last 90 days)
Saluki scored 93.5 on the IFEval benchmark (versus the original model's 91.5), and scored 72.7 on IFBench, surpassing the original model's 71.0.
HumanEval and SWE-bench are commonly used to evaluate large model coding capabilities, MMLU and GPQA for comprehensive reasoning, and IFEval for instruction following.
The evaluation covered five benchmarks: MMLU-Pro, GPQA Diamond, GSM8K, German MGSM, and IFEval.
On GSM8K, Flash-Next scored 97.5% versus FP8's 90.0%; on German MGSM, 92.5% versus 90.0%; and on IFEval, both tied at 89.6%.
An early iteration scored five percentage points lower than the Huihui base model on the IFEval instruction-following benchmark.
The author has not yet re-run the IFEval benchmark on the currently released checkpoint, nor has coding performance been tested.
All Facts (5)
Saluki scored 93.5 on the IFEval benchmark (versus the original model's 91.5), and scored 72.7 on IFBench, surpassing the original model's 71.0.
50%UnverifiedHumanEval and SWE-bench are commonly used to evaluate large model coding capabilities, MMLU and GPQA for comprehensive reasoning, and IFEval for instruction following.
50%UnverifiedThe evaluation covered five benchmarks: MMLU-Pro, GPQA Diamond, GSM8K, German MGSM, and IFEval.
50%UnverifiedAn early iteration scored five percentage points lower than the Huihui base model on the IFEval instruction-following benchmark.
50%UnverifiedThe author has not yet re-run the IFEval benchmark on the currently released checkpoint, nor has coding performance been tested.
50%