WorkspaceBench
A benchmark for evaluating the ability of large models to activate a global workspace (intermediate representations) and read it out into text; it includes multiple evaluation families and problems covering topics such as safety, logical reasoning, and multi-hop computation, along with a dedicated hallucination evaluation.
Timeline (last 90 days)
WorkspaceBench is a benchmark specifically designed to evaluate the reading capability of the "global workspace," used to judge whether a tool can restore intermediate variables in the forward pass into readable text.
The WorkspaceBench benchmark contains 27 evaluation families with a total of 3356 questions, covering topics such as safety, logical reasoning, and multi-hop computation.
WorkspaceBench also provides a dedicated hallucination evaluation, and uses "precision/recall relative to J-lens" as an auxiliary metric.
Problem quality control requires the model to have an accuracy above 80% at n=10, temperature=0.8, and without chain-of-thought, and the intermediate quantities must never appear in the prompt.
The authors strongly recommend using a judge of equal or greater capability for original evaluations, because simpler or weaker judges let in a large number of misjudgments.
The evaluation families include basic, computational, safety, associative, anti-bag-of-words, as well as hallucination evaluation and J-lens concept precision/recall.
All Facts (6)
WorkspaceBench is a benchmark specifically designed to evaluate the reading capability of the "global workspace," used to judge whether a tool can restore intermediate variables in the forward pass into readable text.
50%UnverifiedThe WorkspaceBench benchmark contains 27 evaluation families with a total of 3356 questions, covering topics such as safety, logical reasoning, and multi-hop computation.
50%UnverifiedWorkspaceBench also provides a dedicated hallucination evaluation, and uses "precision/recall relative to J-lens" as an auxiliary metric.
50%UnverifiedProblem quality control requires the model to have an accuracy above 80% at n=10, temperature=0.8, and without chain-of-thought, and the intermediate quantities must never appear in the prompt.
50%UnverifiedThe authors strongly recommend using a judge of equal or greater capability for original evaluations, because simpler or weaker judges let in a large number of misjudgments.
50%UnverifiedThe evaluation families include basic, computational, safety, associative, anti-bag-of-words, as well as hallucination evaluation and J-lens concept precision/recall.
50%