[KongchangAI]
Benchmark

TDM-Gauntlet

A benchmark comprising 46 English test sets for evaluating zero-shot decision models within realistic academic evaluation frameworks.

Timeline (last 90 days)

Nov 17

The researchers replaced the predictors of 24 ACL conference papers with TypeSafe's Jev (also called System One or the System 1 model), keeping each paper's data split, label definitions and evaluation metrics unchanged, and built the TDM-Gauntlet benchmark comprising 46 English test sets.

Unverified50%
Nov 17

Under zero-shot prompting, Jev directly matched the original paper's predictor on 12 of the 46 test sets.

Unverified50%

All Facts (2)

Source Articles