Unverified50% confidenceBenchmarkExact time
For long-horizon tasks (such as migrating an entire application or building a complex application from scratch), even frontier models achieve only a 3% to 40% success rate on Android Bench v2.
1
Sources
50%
Confidence
Medium-term (~90 days)
Relevance
10/30/2026
First Seen
Valid until: 1/28/2027
Sources
Agentic Coding with Small Models: 100x Lower Cost, Double the Success Rate
youtubeAndroid Developers10/30/2026
Related Entities
Related Claims
UnverifiedOn Android Bench, a certain model had a pass rate of 22% with only a natural prompt like "migrate nav2 to nav3"; after providing in advance a test suite that would fail, the overall benchmark pass rate jumped to 90%.66% similarUnverifiedOn a real Android device, the same 4B small model jumped from a 0/20 to a 20/20 success rate simply by changing the policy of "when to consult the model".66% similarUnverifiedOn AutomationBench and VitaBench, SpecAgent recovers 44% to 75% of the quality gap between the two models, while reducing latency and memory overhead.66% similarUnverifiedIn Google's Android World benchmark, Artemis reported a success rate of 99%, covering more than 20 apps and over 100 multi-step tasks.65% similarUnverifiedAutomationBench-AA's 68% completion rate indicates that models still have about one-third of failure space in automated tasks, and there is still a gap to reliable production-grade agents.61% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1117970API
curl https://kongchang.com/api/v1/knowledge/claims/1117970MCP
get_claim(id=1117970)