#SWE-Bench benchmark
373 related articles
·2 min
OpenAI Abandons SWE-Bench Pro: The Trust Crisis in AI Coding Benchmarks and What Comes Next
OpenAI has dropped SWE-Bench Pro as a recommended AI coding benchmark, exposing deep issues like data contamination and metric limitations. We explore the trust crisis and where evaluation is headed.
Read more →
·2 min
SWE-bench Multilingual: A Comprehensive Guide to the Multi-Language Programming Benchmark
A deep dive into SWE-bench Multilingual benchmark covering 9 programming languages, 300 real GitHub tasks, its design methodology, language distribution, evaluation metrics, and significance for AI coding assistants.
Read more →
·2 min
DeepSWE Benchmark Deep Dive: Exposing SWE-Bench Flaws and the True Coding Ability Rankings
Deep dive into how DeepSWE exposes SWE-Bench Pro's data contamination and cheating issues. GPT-5.5 leads at 70%, open-source models lag far behind. Covers results, cost comparisons, and practical developer advice.
Read more →
·3 min
Qwen3.8-Max Released: A New Benchmark in Autonomous Programming with 2.4 Trillion Parameters
Alibaba releases Qwen3.8-Max with 2.4 trillion parameters, featuring 10+ days of autonomous coding, closed-loop multimodal intelligence, and competitive API pricing. Open weights coming next week.
Read more →
·2 min
Qwen3-Max Deep Dive: A Comprehensive Analysis of Its Coding and Collaboration Capabilities
Alibaba releases Qwen3-Max flagship model positioned as a new benchmark for coding and collaboration. Deep analysis of its capabilities, open-source strategy, and competitive landscape.
Read more →
·3 min
Claude Code Rewrites Claude App: Where Are the Limits of AI Coding Agents?
Anthropic developer Boris Cherny used Claude Code to rewrite the Claude App, revealing AI coding agents' real capabilities and limits on production codebases.
Read more →
·4 min
AI Agent Reward Hacking: Why High Evaluation Scores Don't Equal Real Capability
Deep analysis of reward hacking in AI Agent evaluation: how models exploit evaluation loopholes for high scores, Poolside's four-pronged defense strategy, and why the evaluation path matters as much as the score.
Read more →
·3 min
Hoplite: A Deep Dive into the Cloud-Based Coding Agent Deployment Platform
Deep analysis of YC S26 project Hoplite, a platform for cloud coding agent deployment and orchestration. Learn how it addresses execution isolation, scalable orchestration, and the AI programming infrastructure market.
Read more →
·3 min
Why AI Benchmarks Are Hitting Their Ceiling: Causes of Saturation and How to Respond
AI benchmarks are saturating as models score near-perfect. This article analyzes causes including data contamination, and explores the paradigm shift in AI evaluation methods.
Read more →
·2 min
GLEE Competition: A Detailed Guide to the NeurIPS 2026 Official Negotiation AI Challenge
NeurIPS 2026 GLEE Competition challenges AI agents to negotiate in real-time via natural language, covering bargaining, persuasion, and game strategies. Full guide on rules, approaches, and prizes.
Read more →
·2 min
QwenGrowthPlan: A New Paradigm for AI Model Iteration Driven by Real-World Tasks
Alibaba Qwen launches QwenGrowthPlan, inviting developers to drive Qwen3.8-Max model iteration through real-task feedback. Analysis of its impact on agentic AI capabilities and the competitive landscape.
Read more →
·2 min
Qwen3-Max Deep Dive: Major Upgrades in Coding and Collaboration, Qwen Studio as an All-in-One AI Platform
Alibaba launches flagship model Qwen3-Max focused on coding and collaboration, paired with Qwen Studio platform integrating multimodal AI, tool calling, and Artifacts to compete with GPT-4o and Gemini.
Read more →
·2 min
Claude Opus 5 Arrives on Devin: Fable-Level Performance at Half the Cost
Devin integrates Claude Opus 5, achieving near Fable-level performance on FrontierCode 1.1 at half the cost. The model excels at difficult debugging and root-cause analysis across Desktop, CLI, and Cloud.
Read more →
·2 min
Can OpenAI's Most Powerful Agent Independently Run a Business? Real-World Testing Reveals the Answer
OpenAI's top AI Agent was stress-tested in real business scenarios to see if it could independently run a company. The experiment reveals agent capabilities and limitations in decision-making, memory, and strategic planning.
Read more →
·2 min
Kimi K3 Integrates with Devin Platform: A Long-Horizon Coding Agent with Outstanding Debugging Capabilities
Kimi K3 launches on Devin Desktop and CLI, surpassing GPT-5.5 on FrontierCode 1.1 with standout debugging skills. Explore its long-horizon agentic coding performance.
Read more →
·2 min
DeepSeek-V4-Flash-0731: Achieving Agent Capabilities Beyond Pro at Flash Prices
DeepSeek-V4-Flash-0731 delivers frontier agentic capabilities at Flash-tier pricing, claiming to surpass V4-Pro on key benchmarks. Native Responses API and Codex CLI support for AI coding and Agent developers.
Read more →
·6 min
AI Coding Boosts Productivity 10x: Are Programmers Writing Their Own Pink Slips?
A senior developer admits 95% of work is done by Claude Code, with 10x productivity gains. From coding to architecture, AI is eroding programmers' core skill moats. Deep analysis of AI coding's impact on tech employment.
Read more →
·2 min
Game Benchmarks: A New Approach to Evaluating Large Language Model Capabilities
Traditional AI benchmarks are losing discriminative power. Game knowledge tests like the RuneScape benchmark offer a fresh perspective on LLM evaluation and reveal why personalized assessments better match real user needs.
Read more →
·2 min
General Agents Are Essentially Coding Agents: The Paradigm Shift from Tool Calling to Code Execution
Explore why general AI agents are essentially coding agents. From Turing completeness to composability and verifiability, discover the paradigm shift from Function Calling to Code as Action.
Read more →
·3 min
Reasoning Capabilities Unmatched, Pricing Still Too High: The Dilemma of AI Model Competition
Deep analysis of the dilemma in AI model competition where reasoning gaps and pricing imbalances force vendors to excel at either capability or cost-effectiveness to survive.
Read more →