Haiku 4.5 vs GPT-5 Mini vs GLM-4.6: Real-World Benchmark of Budget Coding Models

The Battle of Small Coding Models

As AI coding tools become increasingly prevalent, developers face a practical question: not every project needs flagship models like Claude Sonnet 4.5 or GPT-5. In day-to-day development, the models actually integrated into IDEs and command lines are typically the fast, affordable "terminal coding models."

Terminal Coding Models refer to lightweight AI models specifically optimized for integration into developer workflows. Unlike flagship models that pursue maximum reasoning capabilities, these models seek a balance between parameter size, inference latency, and API call costs. They're typically embedded in IDE plugins like VS Code, Cursor, and Windsurf, or command-line tools, handling high-frequency tasks such as code completion, code generation, and refactoring. Since developers may trigger hundreds of model calls during a single coding session, even tiny per-call costs accumulate into significant expenses—making cost-effectiveness a critical consideration for this category.

KiloCode recently published an in-depth evaluation comparing three of the most representative small coding models today—Claude Haiku 4.5, GPT-5 Mini, and GLM-4.6—under the same testing framework. The tests weren't simple text generation; they covered real programming scenarios including async logic, persistence, concurrency control, and tool calling. The results are quite illuminating.

Test Design: Fair and Close to Real Development

The testing methodology deserves mention. The evaluation team used KiloCode's Q&A mode to write prompts, then switched to code mode for execution. The test task was: Build a task queue system with TypeScript and SQLite that supports delayed execution, implements persistence, and demonstrates a complete example.

This task is cleverly designed—it covers async logic, data persistence, and concurrency control, which happen to be exactly where small models are most likely to fail. All three models received identical prompts with no model-specific prompt engineering, each starting from a blank project.

KiloCode's evaluation methodology represents an important trend in AI coding model assessment: shifting from static benchmarks to dynamic real-world tests. Traditional code generation evaluations (such as HumanEval, MBPP, SWE-bench) typically test a model's ability to generate code at the isolated function level. KiloCode's test requires models to complete a full engineering task involving multiple files, modules, and technology stacks. This end-to-end evaluation approach better reflects model performance in real development scenarios, because actual programming involves not just writing correct functions, but also project structure design, dependency management, error handling, and runtime behavior. This also explains why the three models may show minimal differences on traditional benchmarks but demonstrate vastly different engineering philosophies in this real-world test.

Core Results: The Speed-Cost-Quality Triangle

Hard Metrics Comparison

Model	Time	Total Cost	Key Characteristic
GPT-5 Mini	6 minutes	$0.05	Concurrency-safe, production-grade quality
Haiku 4.5	3 minutes	$0.08	Fastest speed, most features
GLM-4.6	4 minutes	$0.14	Best structure, but lacking stability

A counterintuitive finding: The model with the lowest per-token price doesn't necessarily have the lowest total cost. Although GLM-4.6 has a cheaper list price, it generates excessively verbose content and consumes massive tokens during inference, ultimately making it the most expensive of the three.

This is a highly practical insight—when evaluating AI coding model costs, you can't just look at the pricing table; you need to consider the "actual cost per run." In enterprise applications, this difference gets amplified further: if a team executes thousands of model calls per day, a $0.09 difference in per-run cost translates to thousands of dollars in additional monthly spending.

Breaking Down the Three Models: Distinctly Different Engineering Philosophies

GPT-5 Mini: The Systems Engineer Mindset

GPT-5 Mini is the only model among the three that truly understands SQLite's concurrency limitations. To appreciate why this matters, you need to understand SQLite's core architectural characteristics: SQLite is an embedded relational database known for being lightweight and zero-configuration, widely used in mobile apps, desktop software, and small services. However, it uses file-level locking, allowing only one write operation at a time. When multiple processes or threads attempt simultaneous writes, SQLite returns a SQLITE_BUSY error. In production environments, without proper handling of concurrent writes—such as using WAL (Write-Ahead Logging) mode, transaction retries, or application-level locks—data loss or crashes will occur.

GPT-5 Mini addresses this concurrency issue through a lease-based locking system. Lease locks are a common concurrency control pattern in distributed systems: when a Worker acquires a task, instead of permanently locking it, a time-limited lease is set (e.g., 30 seconds). If the Worker completes the task before the lease expires, the lock is released normally; if the Worker crashes or times out, the lease automatically expires and other Workers can re-acquire the task. This mechanism avoids deadlock problems and is widely used in Redis distributed locks (such as the Redlock algorithm) and Kubernetes Leader Election. GPT-5 Mini implements locking through timestamps via a locked_until field, and also employs transactions and exponential backoff—retrying failed operations at exponentially growing intervals (e.g., 1s, 2s, 4s, 8s) to prevent "retry storms" that would further increase system load under high concurrency.

This is real engineering logic. If you're deploying code to production, this is exactly how it should be done. While it encountered some minor tool calling issues, it automatically recovered from failures. GPT-5 Mini's strategy is clear: correctness first—no frills, just working code that won't crash.

GLM-4.6: The Software Architect Mindset

GLM-4.6 excels in code structure—multi-file architecture, complete type system, enums, priority queues, and even hand-written UUID generation functions instead of using existing libraries. This approach is a bit "over-engineered," but it demonstrates strong capability in handling low-level coding tasks.

However, GLM's classic problem resurfaces: reasoning mode causes tool calling to fail. This reflects a common tension in current large language model architectures. Reasoning mode typically enables Chain-of-Thought mechanisms, where the model performs multi-step internal reasoning before generating a final answer. However, this reasoning process alters the model's output format and token distribution, potentially breaking or malforming structured outputs (such as the JSON format required for tool calling). In some models, reasoning capabilities and tool calling capabilities are optimized at different stages or on different datasets during training, creating a certain "capability competition" between the two. OpenAI and Anthropic are actively working to resolve this in their latest models, but for many models, balancing reasoning depth with tool calling stability remains an active research direction.

The evaluation team had to disable reasoning mode to get GLM-4.6 working properly, which also slowed it down. More critically, it tracks active tasks in memory—meaning if the application crashes, all state is lost.

In one sentence: The code looks clean in the repository, but gives you endless headaches at runtime.

Haiku 4.5: The UX Designer Mindset

Claude Haiku 4.5 is the speed king, completing the task in just 3 minutes while adding extra features like statistics, task cleanup, and index optimization. This aligns with Claude's typical style—focused on developer experience, providing thoughtful bonus features.

For tool calling, Haiku performed flawlessly with zero failures. Claude has always been top-tier in file editing precision. But the fatal flaw is: no concurrency control, no locking, no transactions, no safety guarantees whatsoever. The code runs, but should absolutely never be called via API in a production environment.

Additionally, the evaluation found that Haiku sometimes gets stuck in loops—if a task hangs midway, it may enter an infinite loop of repeated operations.

Tool Calling: The Overlooked Critical Dimension

One of the most valuable aspects of this evaluation is the dedicated testing of tool calling capabilities. In the context of AI coding assistants, tool calling (Tool Use / Function Calling) means the model doesn't just generate code as text—it can directly manipulate the development environment: creating files, editing specific lines of code, executing terminal commands, reading project structures, etc. This requires the model to output structured JSON instructions that the host application (such as an IDE plugin) parses and executes. Tool calling reliability directly determines the practicality of an AI coding assistant: if a model generates correct code but can't accurately write it to the target file, or loses context during multi-step operations causing file editing chaos, developers still need extensive manual intervention.

Tool calling is typically the first thing to break in AI coding assistants, and the key dividing line between "can chat about code" and "can write code."

The results are clear:

• Haiku 4.5: Zero tool calling failures, flawless performance. This means developers can trust every file operation from the model without repeatedly checking and correcting.
• GPT-5 Mini: Recovered after retries, acceptable
• GLM-4.6: Tool calling completely fails in reasoning mode; must disable reasoning mode

For developers, what matters isn't what the model claims it can do, but what it can actually build when you run it.

AI Coding Model Selection Guide

The evaluation article offers an elegant analogy that perfectly captures the three models' mindsets:

GPT-5 Mini is the systems engineer, GLM-4.6 is the software architect, and Haiku 4.5 is the UX designer.

Based on these real-world test results, here are selection recommendations for different scenarios:

• Production deployment and data-sensitive projects → GPT-5 Mini. It's the only model with production-grade concurrency safety guarantees, and it has the lowest total cost. For backend services requiring concurrent writes, transaction integrity, and failure recovery, GPT-5 Mini's systems engineer mindset effectively reduces production incident risk.
• Rapid prototyping or demos → Haiku 4.5. Unmatched speed, thoughtful bonus features, and stable tool calling make it ideal for quickly validating ideas. In hackathons, product demos, or internal proof-of-concepts, completing a fully functional task queue system in 3 minutes provides an overwhelming efficiency advantage.
• Emphasis on code structure and architecture design → GLM-4.6. Beautifully organized code, but remember to disable reasoning mode—not recommended for scenarios requiring high reliability. Best suited for generating project scaffolding, designing module interfaces, or serving as code review references.

Conclusion

The value of this evaluation lies in not ranking models by simple benchmark scores, but rather exposing each AI coding model's strengths and weaknesses through a real engineering task. For developers, choosing a coding model has never been about "which is strongest" but rather "which best fits my current scenario."

It's worth noting that the performance differences among these three models also reflect different training priorities and product positioning across AI companies. Anthropic has invested heavily in optimizing tool calling precision, OpenAI continues to deepen system-level reasoning and safety, while Zhipu demonstrates unique advantages in code structuring and architecture design capabilities. As these models iterate rapidly, current shortcomings may see significant improvements in the next version.

Looking forward to seeing more real-world evaluations like this, especially comparisons in more complex scenarios like API development and frontend construction—where the trade-offs between models truly matter.