Unverified50% confidenceBenchmarkExact time
After injecting two Oracle-specific rules into the prompt, first-attempt failed queries dropped from 694 to 85.
1
Sources
50%
Confidence
Medium-term (~90 days)
Relevance
10/20/2026
First Seen
Valid until: 1/18/2027
Sources
Related Entities
Related Claims
UnverifiedThe substantive threshold was validated on the 128 attempts from the filtering round: 121 passed, and the 7 rejected were 6 empty submissions and 1 bare SELECT, with no real attempt lost.69% similarUnverifiedThrough benchmark leakage detection, SerpApi excluded 40 questions that exposed benchmark answers or evaluation prompts, ultimately answering 937 of the remaining 960 questions correctly.68% similarUnverifiedIn the early test rounds, 101 of 122 attempts contained syntax errors and could not run at all.67% similarUnverifiedWhen the open-source small model gemma2:2b was used as the checker, it accepted 8 of the 28 wrong answers (6 wrong party and 2 wrong action), while rejecting 10 of the 14 correct answers.67% similarUnverifiedThe schemagate authors note its limitations: only one model was tested, and the prompt wording was chosen on 100 BIRD questions; removing these two factors leaves the score essentially unchanged (90.08).66% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1073559API
curl https://kongchang.com/api/v1/knowledge/claims/1073559MCP
get_claim(id=1073559)