Unverified50% confidenceFactExact time
The wording of the prompt was chosen on 100 BIRD questions that are part of the test set, so there is tuning contamination.
1
Sources
50%
Confidence
Medium-term (~90 days)
Relevance
10/17/2026
First Seen
Valid until: 1/15/2027
Sources
Related Entities
Related Claims
UnverifiedThe benchmark uses a unified set of 100 questions, with only one search issued per question and the same answering model used across all configurations to process the returned content.70% similarUnverifiedThe test designed 50 automatically verifiable tasks, divided into five categories: multi-step math computation, extracting JSON from order notes, classifying support tickets, conditional filtering and sorting, and strict-format writing, with 10 questions per category, each with a standard answer.66% similarUnverifiedThe test questions are divided into 40 easy-to-verify questions and 40 hard-to-verify questions, and all standard answers of the math, code, and date types are validated in Python.66% similarUnverifiedBuilding a small test set should start with 30 to 50 real questions, each accompanied by the correct answer and the document containing the answer, measuring retrieval (whether the correct text chunk appears in the top results) and generation (whether the final answer is supported by the retrieved text) separately.66% similarUnverifiedThe benchmark was designed with 40 requests across 6 categories, covering faithfulness, unit conversion, multilingual (Bengali, Hindi), injection attacks, out-of-scope, and ambiguous input.65% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1050464API
curl https://kongchang.com/api/v1/knowledge/claims/1050464MCP
get_claim(id=1050464)