ConfabQA
A benchmark for evaluating whether language models refuse to answer or confabulate when they lack knowledge. It includes ambiguous pre-cutoff categories to separate knowledge cutoff from a tendency to confabulate.
Timeline (last 90 days)
The study constructed a new benchmark, ConfabQA, containing 784 entries, and set up a "blurry pre-cutoff" category to disentangle correctness from knowledge cutoff confusion.
On Llama 3.2 3B, the marginal improvement on the ConfabQA benchmark was within the range of fold-to-fold fluctuation, but refusal-specific probes achieved 18–25 percentage point improvements on the PopQA and TriviaQA datasets.
The correctness signal is model-dependent and transferable only on Llama 3.2 3B: when probe margins were transferred among ConfabQA, PopQA and TriviaQA, 81% of the margin was preserved on the test majority class.
The study open-sourced the benchmark, judge and code.
All Facts (4)
The study constructed a new benchmark, ConfabQA, containing 784 entries, and set up a "blurry pre-cutoff" category to disentangle correctness from knowledge cutoff confusion.
50%UnverifiedOn Llama 3.2 3B, the marginal improvement on the ConfabQA benchmark was within the range of fold-to-fold fluctuation, but refusal-specific probes achieved 18–25 percentage point improvements on the PopQA and TriviaQA datasets.
50%UnverifiedThe correctness signal is model-dependent and transferable only on Llama 3.2 3B: when probe margins were transferred among ConfabQA, PopQA and TriviaQA, 81% of the margin was preserved on the test majority class.
50%UnverifiedThe study open-sourced the benchmark, judge and code.
50%