Claude Haiku 4.5 Real-World Test: Fast but Outclassed in Value by GPT-5 Mini

Nearly a year later, Anthropic has finally updated its lightweight model, Haiku. Claude Haiku 4.5 claims intelligence levels close to Sonnet 4 at one-third the price and twice the speed. However, when we place it in the broader competitive landscape, the results are decidedly mixed.

Speed Test: Haiku 4.5 Is Indeed Much Faster Than Sonnet 4

To verify Anthropic's speed claims, the video author conducted a head-to-head speed comparison between Haiku 4.5 and Claude Sonnet 4 in the Cursor editor. Cursor is one of the most popular AI-native code editors available today — built on VS Code's open-source codebase with deep integration of multiple large language models, allowing developers to invoke AI directly for code generation, refactoring, debugging, and explanation while writing code. Since Cursor lets users freely switch between underlying models (including the Claude series, GPT series, etc.), it serves as an ideal testing platform for comparing different AI models' actual coding performance.

The results were clear: Haiku 4.5 won by over one minute. To rule out interference from Cursor's own features, the author also re-ran the test in Claude Code, with essentially identical results. Claude Code is Anthropic's command-line AI programming tool that lets developers operate on codebases directly through natural language instructions, executing complete development workflows from code writing to Git commits. These two tools represent the two mainstream paradigms of AI-assisted programming — IDE-integrated and command-line agent-based — and the consistent speed advantage across both paradigms demonstrates that Anthropic's speed claims are well-founded.

Interestingly, the new Haiku is the first model in the series to support reasoning capabilities, featuring a 200K token context window and a maximum output length of 64K tokens. Reasoning mode is one of the most important technical breakthroughs in large language models over the past two years: traditional LLMs generate answers directly upon receiving a question, while reasoning-capable models perform an internal "Chain-of-Thought" process before outputting their final answer — breaking complex problems into multiple steps and deriving conclusions incrementally. This mechanism was first popularized at scale by OpenAI's o1 model, with other major players quickly following suit. The tradeoff of reasoning mode is additional token consumption and increased latency, but for tasks requiring multi-step thinking — such as math, programming, and logical reasoning — the performance improvement is very significant. As the first Haiku model to support reasoning, Haiku 4.5 means users can access this deep thinking capability on a lightweight model without paying for the more expensive Sonnet or Opus models.

As for the 200K token context window, it means Haiku 4.5 can process approximately 150,000 English words or about 300,000 Chinese characters in a single pass — enough to accommodate a medium-length technical book. For developers, a larger context window means more code files, documentation, and conversation history can be fed into the model at once, reducing information loss. The 64K token maximum output length determines how much content the model can generate in a single response, which is particularly important for scenarios requiring complete code files or lengthy documents in one shot. For reference, GPT-4o has a context window of 128K tokens, giving Haiku 4.5 a clear advantage in this dimension. These specifications are quite competitive among lightweight models.

Code Quality Comparison: It's Fast, but How Good Is the Output?

In the speed test, both models generated a typing game. Looking at the actual output, the author felt Haiku's UI design looked better, though the text scrolling speed was too fast. Sonnet 4's version, on the other hand, excelled in feature completeness, "unlocking a whole new level."

However, a single case isn't enough to draw conclusions. Let's look at more systematic benchmark data.

Benchmarks: Official Numbers Shine, but Third-Party Tests Tell a Different Story

Anthropic's Official Benchmark Results

According to Anthropic's own published data, Haiku 4.5's performance is indeed impressive:

• Beat Sonnet 4 on SWEbench, even surpassing GPT-5
• Matched or beat Sonnet 4 on most benchmarks
• Fell behind Sonnet 4 on only four tests

SWEbench (Software Engineering Benchmark) is a benchmark developed by a Princeton University research team specifically designed to evaluate AI models' software engineering capabilities. Unlike traditional code generation tests, SWEbench extracts real bug-fix tasks from actual GitHub open-source projects — models need to understand the project's codebase structure, locate the problematic files and lines of code, then generate correct fix patches. This testing approach more closely mirrors real software development scenarios, making it widely regarded as one of the "gold standards" for measuring AI coding ability. Haiku 4.5 beating Sonnet 4 on SWEbench suggests Anthropic prioritized practical engineering capability during training.

This also indicates that Anthropic clearly prioritized coding ability during training, making certain tradeoffs in other areas.

Third-Party Independent Evaluation Results

However, when we turn to third-party evaluation platforms like Artificial Analysis, the picture changes. Artificial Analysis is one of the most influential independent evaluation platforms in the AI model space, conducting standardized cross-vendor comparisons across multiple dimensions including intelligence level, speed, and price. Unlike vendor-published benchmark scores, the value of third-party evaluations lies in their consistent testing conditions and neutrality — all models are tested under identical prompts and scoring criteria, avoiding the problem of vendors "gaming benchmarks." A well-known phenomenon exists in the industry where models perform excellently on official benchmarks but significantly worse in real-world use or third-party evaluations — typically because training data contained samples similar to the benchmark tests, causing the model to "overfit" to specific tests. Therefore, third-party evaluation results are generally considered more reflective of a model's true capabilities.

On the coding index (with reasoning mode enabled), Haiku 4.5 trails Sonnet 4 by only two points — which seems acceptable. But the problem is:

• GPT-5 Mini High scores 9 points higher
• GLM 4.6 also leads by 1 point
• And both competing models are priced significantly lower than Haiku 4.5

Price Comparison: Anthropic's Pricing Strategy Is Puzzling

This is the most disappointing part. Let's look at the numbers directly:

Model	Input Price (/million tokens)	Output Price (/million tokens)
Claude Haiku 4.5	$1.00	$5.00
Claude Haiku 3.5 (last year)	$0.80	$4.00
GPT-5 Mini	$0.25	$2.00
GLM 4.6	$0.50	$1.75

AI model API pricing is typically measured in "per million tokens," with input and output priced separately. Output prices are usually higher than input prices because the generation (output) process requires autoregressive inference token by token, with computational density far exceeding the encoding process that handles input text all at once. A model's operational costs are primarily determined by GPU compute, and GPU utilization efficiency, model parameter count, and inference optimization techniques (such as quantization, speculative decoding, KV cache optimization, etc.) all directly affect the marginal cost per token.

A rather ironic fact: Haiku 4.5 is more expensive than last year's Haiku 3.5 — input price increased 25%, output price increased 25%. In an era where AI model prices are generally declining, Anthropic raising prices against the trend is somewhat baffling. The current market shows a rapid downward trend in AI model pricing — thanks to advances in inference hardware, maturation of model distillation techniques, and fierce competition among vendors. Anthropic's decision to raise prices against the trend is controversial precisely because it contradicts this industry-wide trend.

More critically, GPT-5 Mini's input price is just one-quarter of Haiku 4.5's, its output price is less than half, and it performs better in third-party evaluations.

Comprehensive Analysis: Value and Speed Combined

When we plot intelligence level against cost on the same chart, the conclusion is very clear:

• GPT-5 Mini leads by a wide margin in value-for-money, making it the optimal choice among current lightweight models
• GLM 4.6 also outperforms Haiku 4.5
• Even in the intelligence vs. output speed comparison, Haiku with reasoning mode enabled merely ties with GLM 4.6

It's worth noting that GLM 4.6 comes from Chinese AI company Zhipu AI, incubated by a Tsinghua University technical team and one of China's earliest companies focused on general-purpose large model R&D. In recent years, Chinese AI vendors have demonstrated extremely strong competitiveness in model cost-effectiveness through more efficient training methods, lower operational costs, and aggressive pricing strategies, forming a unique competitive advantage in the global market. GLM 4.6 delivers performance comparable to top international models at extremely low prices — this "volume through pricing" strategy is profoundly reshaping global AI model pricing dynamics, forcing Western vendors like Anthropic and OpenAI to consider more competitive pricing.

Is Haiku 4.5 Still Worth Using? Which Scenarios Suit It?

Despite not winning the pure value-for-money battle, Haiku 4.5 still has its unique value:

Advantages:

• Genuinely fast, suitable for development scenarios requiring rapid iteration
• Deep integration with Claude Code — low switching costs for users already in the Anthropic ecosystem
• Anthropic models often perform more reliably in real codebases than benchmarks suggest
• First Haiku model to support reasoning — a qualitative leap in functionality

Disadvantages:

• Pricing isn't competitive enough — even more expensive than its predecessor
• Third-party evaluation performance trails GPT-5 Mini and GLM 4.6
• Obviously skewed toward coding, with general capabilities sacrificed

Final Thoughts: Competition Is a Developer's Best Friend

Objectively speaking, Haiku 4.5 isn't a "disappointing" model — it genuinely achieves intelligence levels close to Sonnet 4 with faster speed. The real issue is that competitors are advancing faster and pricing more aggressively.

OpenAI's GPT-5 Mini and Zhipu's GLM 4.6 both deliver comparable or better performance at lower prices. This intense competitive dynamic is an absolute win for developers — lightweight, high-value AI models are becoming better and cheaper.

For Anthropic, Haiku 4.5 feels more like a defensive product update rather than an offensive market breakthrough. In this price-sensitive market segment, Anthropic may need to reconsider its pricing strategy, or risk gradually losing ground in this value-for-money battle.

Key Takeaways

• Claude Haiku 4.5 is over one minute faster than Sonnet 4 and is the first Haiku model to support reasoning
• In official benchmarks, Haiku 4.5 beats Sonnet 4 and GPT-5 on SWEbench, but in third-party evaluations, GPT-5 Mini scores 9 points higher
• Haiku 4.5 is priced at $1/$5 (input/output), more expensive than last year's Haiku 3.5, while GPT-5 Mini is just $0.25/$2
• In comprehensive value analysis, both GPT-5 Mini and GLM 4.6 outperform Haiku 4.5
• Haiku 4.5 appears to be a defensive update from Anthropic; the fierce lightweight model market competition benefits developers