TDM-Gauntlet
A benchmark comprising 46 English test sets for evaluating zero-shot decision models within realistic academic evaluation frameworks.
Timeline (last 90 days)
The researchers replaced the predictors of 24 ACL conference papers with TypeSafe's Jev (also called System One or the System 1 model), keeping each paper's data split, label definitions and evaluation metrics unchanged, and built the TDM-Gauntlet benchmark comprising 46 English test sets.
Under zero-shot prompting, Jev directly matched the original paper's predictor on 12 of the 46 test sets.
All Facts (2)
The researchers replaced the predictors of 24 ACL conference papers with TypeSafe's Jev (also called System One or the System 1 model), keeping each paper's data split, label definitions and evaluation metrics unchanged, and built the TDM-Gauntlet benchmark comprising 46 English test sets.
50%UnverifiedUnder zero-shot prompting, Jev directly matched the original paper's predictor on 12 of the 46 test sets.
50%