Unverified50% confidenceBenchmarkExact time
On the ring_push task, four models (Claude Haiku 4.5, Gemma 4 31B, GPT-5.4 mini, Gemini 3.7 Flash) completed the required changes in both tasks, yet lost old behaviors not restated in the prompt, passing fail-to-pass but failing pass-to-pass, and this failure reproduced stably across eight runs.
1
Sources
50%
Confidence
Short-term (~14 days)
Relevance
10/12/2026
First Seen
Valid until: 10/26/2026
Sources
Related Entities
Related Claims
Unverified此前的PokeAgent挑战证明，即使是GPT-4V、Gemini等顶级多模态模型，在没有专门设计的辅助框架时，几乎无法在宝可梦RPG中取得任何进展69% similarUnverifiedIn Bench-Corp tests, the guardrail missed twice out of 520 runs, both on the same task with Gemini 3.7 Flash.67% similarUnverifiedDeepSeek V4 Flash failed 13 out of 22 multi-turn tests (fail_fast 45%), and Gemini also failed 13.67% similarUnverifiedThe eight models tested include GPT-6 Luna, Claude Haiku 5.5, Gemini 3.5 Flash-Lite, Qwen 3 Next 80B, Gemma 4 31B, GPT-6 Sol, Claude Sonnet 5, and Gemini 3.7 Flash, with each model running 107 investigations per track.64% similarUnverifiedThe revised experiments use Gemini 3.7 Flash, Claude Sonnet 5.5, and GPT-5.4, with each model independently performing three optimization attempts, and prompts using temperature 0 and seed 0.64% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1007728API
curl https://kongchang.com/api/v1/knowledge/claims/1007728MCP
get_claim(id=1007728)