Databricks Tests Coding Agents in Production: The Truth About Cost, Performance, and Tool Selection

A Real Codebase as the Testing Ground

AI coding agents are multiplying rapidly, but most benchmarks are built on small, clean example projects. Databricks took a more demanding approach — running systematic benchmark tests on their production-grade codebase spanning millions of lines of code. This environment far more closely mirrors the complexity of real enterprise software development, making the findings far more actionable.

The analysis challenges several industry assumptions: from how model costs are structured, to the true capability ceiling of open-source models, to the outsized role that the agent execution framework (the "harness") plays in determining real-world outcomes. Here are the core findings worth unpacking.

The Pareto Frontier: No Single Winner

A key concept in this analysis is the Pareto Frontier — the set of best achievable quality outcomes at any given cost level. The term comes from economist Vilfredo Pareto's theory of Pareto optimality: in multi-objective optimization, it describes the set of solutions where you cannot improve one objective without degrading another. In the context of AI model evaluation, "cost" and "quality/performance" serve as the two optimization dimensions. Every point on the Pareto frontier means: at that cost level, no other model delivers higher quality — and getting higher quality necessarily requires higher cost. This framework elevates model comparison from a single-dimension ranking to a multi-dimensional tradeoff, which better reflects how enterprise decisions are actually made.

Databricks found that this frontier curve includes models from OpenAI, Anthropic, and the open-source ecosystem alike. The implication is clear: no single tool delivers frontier performance across all scenarios. Different models excel at different task difficulty levels and cost ranges. For enterprises, the optimal strategy isn't betting on one vendor — it's building a flexible toolchain capable of routing across multiple models.

In a fast-moving field like coding agents, a diversified model portfolio may be the only way to genuinely balance cost and quality. The "pick a side" mentality deserves a second look.

The Rise of Open-Source: GLM 5.2's Breakthrough

A long-standing assumption in the industry has been that open-source models fall noticeably behind closed proprietary models on high-difficulty tasks. This test tells a different story.

Databricks specifically noted that open-source models — particularly GLM 5.2 — are now capable of handling the highest difficulty tier of coding tasks. The GLM (General Language Model) series is co-developed by Tsinghua University and Zhipu AI. Its architecture is based on an autoregressive blank-filling objective, which differs from the purely autoregressive approach of the GPT series. GLM 5.2 is a significant iteration of the series, with notable improvements in code comprehension, long-context processing, and multi-step reasoning. As an open-source model, GLM can be deployed in private environments without sending code data to third-party servers — a natural advantage in industries with strict data compliance requirements, such as finance, healthcare, and government.

This signals that open-source models are rapidly closing the gap with closed frontier models in complex code understanding and generation — and in some dimensions, matching them outright. It also validates the broader trend of China's open-source LLM ecosystem catching up with the international frontier. For enterprises prioritizing data privacy, cost control, or self-hosted deployment, this is a significant development. Open-source models are no longer just a "good enough" fallback — they are viable, reliable options for handling hard tasks in production. This further reinforces the value of a hybrid toolchain strategy: open-source models are becoming an indispensable part of the Pareto frontier.

The Cost Myth: Token Price Per Unit Is Misleading

Perhaps the most counterintuitive finding concerns cost. The common practice is to estimate usage costs using a model's price per token — but Databricks' tests show this is an extremely poor cost metric.

Tokens are the basic unit by which large language models process text — roughly 1–2 tokens per English word, and similarly for Chinese characters. Current mainstream models are typically priced per million tokens. However, completing a real coding task often involves multiple rounds of dialogue, code generation, error correction, and test validation — meaning total token consumption can be tens of times higher than a single call.

The actual end-to-end cost of a task depends on the total tokens consumed to complete it, not the unit price. The tests found:

• Larger models tend to be more token-efficient — they complete tasks with fewer interaction rounds and more precise reasoning. This property is called "Task Token Efficiency";
• As a result, a more expensive model per token may actually result in lower total cost for a complete task.

This finding matters enormously for cost-conscious teams. Choosing a model based solely on per-token price risks selecting one that appears cheap but burns more money under real workloads. Evaluating the cost of an AI coding agent must be done on a full end-to-end, per-task basis.

The Decisive Role of Harness Design

The final core finding points to a frequently overlooked variable: the execution framework, or harness.

A harness refers to the outer framework that invokes the model, manages context, organizes tool calls, and runs the execution loop. Common agent frameworks include LangChain, AutoGen, SWE-agent, and OpenHands — responsible for task decomposition, tool invocation, context management, and execution cycles. The same underlying model can deliver wildly different cost and quality outcomes depending on which harness it runs under.

Complex frameworks often introduce multi-layer planning, reflection mechanisms, and multi-agent collaboration. These can meaningfully improve performance on certain tasks, but they also introduce more intermediate steps, higher token consumption, and more potential failure points. One particularly thought-provoking result from Databricks' tests: simpler harnesses (such as Pi) consistently performed best on their workloads. This shows that more complex, "smarter" agent frameworks don't always produce better outcomes — over-engineered orchestration logic can introduce noise, increase costs, and reduce reliability. This aligns neatly with the software engineering principle of YAGNI (You Aren't Gonna Need It).

This finding is a reminder that when building AI coding systems, the framework carrying the model deserves just as much scrutiny as the model itself. A clean, efficient architecture is sometimes the true optimal solution.

Summary and Selection Guidance

Databricks' real-world testing on a million-line codebase yields several counterintuitive lessons for enterprises evaluating coding agents:

1. Don't bet on a single vendor — frontier performance is distributed across OpenAI, Anthropic, and open-source models; a hybrid toolchain is the current optimal strategy.
2. Reconsider open-source models — models like GLM 5.2 can now take on the hardest tasks.
3. Abandon the token price myth — measured by end-to-end task cost, larger models may actually be cheaper.
4. Take harness design seriously — simpler frameworks sometimes outperform complex orchestration.

As AI coding agents move into enterprise production environments, benchmarks grounded in real, large-scale codebases will become increasingly critical. Benchmarks that are disconnected from real workloads can lead to serious misjudgments. When selecting and deploying tools, testing against your own codebase and your own task scenarios is the only way to reach conclusions you can actually trust.