Claude Haiku 5.5 vs. GPT-6 Luna: Same Sticker Price, Wildly Different Bills — An Agentic Coding Benchmark

Two small models with identical list prices — $0.10 per million input tokens and $0.50 per million output tokens — ran through the same agentic coding benchmarks and produced bills that differed by 3.6×. That's the core finding from LeanZero's head-to-head test of Claude Haiku 5.5 and GPT-6 Luna across two proprietary agentic benchmarks: same sticker price, dramatically different actual spend, and Haiku winning both rounds on quality.

This article unpacks a fact buried beneath vendor marketing slides — the price on the label is not the price for agentic workloads.

Two Benchmarks, Haiku Wins Both

Testing was conducted on two automated benchmarks, both running goose (a fork of Block's open-source agent) through OpenRouter with a budget of 150 calls and zero human intervention.

On Forge 1.0 — building an Atlassian Forge application in an offline environment — Haiku scored 0.9661 versus Luna's 0.395. Haiku ranked fourth out of 31 runs, trailing the much larger Claude Opus 5.5 by just 0.0106. On Gauntlet 7.2 (a payments application), Haiku scored 0.661 versus Luna's 0.4797.

Each benchmark consists of hundreds of automated checks organized into tiers covering storage, UI, 3D interaction, and more. Tier scores account for 88% of the total (referred to as the core score), with the remaining 12% being an "excellence" bonus that primarily measures economy — how many API calls the application consumed. A "critical" failure multiplies the entire run's score by 0.6, while a "band" is a set of must-pass checks whose failure caps the run at 0.699 or 0.799.

The authors are careful to flag a caveat that runs throughout the entire post: each model ran each benchmark exactly once. A single run cannot separate model capability from luck, and a single early decision in a Forge build can shift an entire run across a band boundary. All conclusions should be read as "what happened in these runs," not as "laws about these models."

goose is Block's (formerly Square's) open-source agent framework, designed to let LLMs autonomously complete multi-step software development tasks. It interacts with real environments through tool calls (shell commands, file I/O, API requests, etc.) and keeps the full conversation history in the prompt of every request — a design decision that is the technical root cause of the cost gap in this article. OpenRouter is a unified API gateway that lets developers call models from Anthropic, OpenAI, and other providers through a single interface, with request-level token telemetry that makes cross-model cost comparisons possible. Atlassian Forge is Atlassian's official serverless app development platform, allowing developers to build custom functionality on top of Jira, Confluence, and other products. Its strict sandbox constraints — no external network access, dependencies must be declared in the manifest file upfront — make "offline builds" and "manifest completeness" genuine tests of model capability rather than artificial obstacles.

Same List Price, 3.6× the Bill

Both models carry identical pricing: $0.10/M input tokens, $0.50/M output tokens, and $0.01/M for cache reads. Yet Haiku's two runs totaled $1.62, while Luna's came to just $0.46.

The key difference is hidden in pricing tiers. Anthropic's pricing page states clearly that when a request's prompt exceeds 100K tokens, prices jump to $0.50/$2.50 — and the cache read cost rises to $0.05 — with the entire request billed at the higher rate, even if part of it is a cache hit. Luna's tier doesn't kick in until 272K tokens, nearly three times later. Haiku 5.5 is also currently the only Claude model Anthropic excludes from its "standard pricing up to 1M tokens" scheme.

Because goose resends the entire conversation history with every call, the prompt only ever grows. In the Gauntlet run, 24 of Haiku's 39 calls crossed the 100K threshold, carrying 88.1% of the prompt tokens. In the Forge run, 40 of 72 calls crossed the line.

The authors re-priced the calls using Anthropic's two-tier structure based on telemetry data, achieving a model error within 5.5%. The conclusion: without that tier, the same calls would have cost roughly $0.30 for Gauntlet and $0.19 for Forge — essentially on par with Luna's $0.23 and $0.22. The gap wasn't caused by Haiku being verbose. It was caused by an early toll booth.

A Bug That Created a Cliff

The Gauntlet chart shows a sharp cliff at call 16 — the prompt jumps from 57,094 tokens to 148,478 tokens in one step. The cause: Haiku chained fourteen curl calls into a single shell command to test its own service, producing 188,758 bytes of output in just 14 lines. goose's shell tool has a 50,000-byte hard cap, but its preview logic counts lines rather than bytes. With 14 lines falling below the 50-line threshold, it flagged the output as "over limit" but injected the full content anyway.

The authors acknowledge this as a bug in their own goose fork and have submitted a fix. Using the re-pricing model, they estimate the run would have cost approximately $0.48 rather than ~$1.15 if the output had been correctly truncated. This kind of transparency — openly quantifying how your own tooling inflated the results — is rare in benchmark writeups.

The "pricing tier" is a common but rarely discussed mechanism in LLM API pricing: when a single request's prompt exceeds a threshold, the entire request is billed at the higher rate — not just the portion above the threshold. Anthropic's tier for Claude Haiku 5.5 sits at 100K tokens, after which input pricing jumps from $0.10 to $0.50 and output from $0.50 to $2.50 (a 5× increase). The commercial rationale is usually that very long contexts put nonlinearly increasing pressure on inference hardware (especially KV cache VRAM), and the higher price reflects that computational cost. For agentic workloads, frameworks like goose that use a "full conversation history replay" design cause the prompt to grow monotonically with each call. Once it crosses the tier threshold, there's no going back — every subsequent call will continue to be billed at the higher rate, like paying highway tolls for the entire remaining journey the moment you pass the first booth.

Forge: 0.9661 vs. 0.395 — The Gap Runs Deep

Forge 1.0 required building a real Atlassian Forge application — specifically a "scope ledger" that records every Jira issue entering and leaving an active sprint — in an offline environment. Models had to rely entirely on their own knowledge of Forge.

Of Haiku's 66 checks, only two scored below 1.0, and both were on economy grounds. All core functional checks were perfect — identical to Opus 5.5's core score. The 0.0106 gap between them comes almost entirely from Haiku making two extra Jira calls during backfill.

Luna's performance exposed Forge's core challenge. Its real-time ledger was nearly flawless: all 22 real-time event rows were correct, and all 28 reads used the correct sprint index. But it never imported history — an app installed mid-sprint must backfill all prior changes on its first scheduled run, and Luna's ledger recorded 0 entries after the first round, leaving all 62 historical changes untracked.

This backfill check was also flagged as a critical failure labeled "data loss — changes silently missing," multiplying the entire run by 0.6. Luna scored 0.6583 before the multiplier, dropping to 0.395 after.

The authors examined all eight screenshots of Luna's sprint modal. Every row in the Source column read "event." Every row in Haiku's read "reconcile" — one model only saw real-time events; the other actually replayed history.

Two Missing Items in the Manifest

Luna's manifest had two issues that would only bite at runtime: it was missing the Jira Software scope required to list sprint issues (resulting in a 401 from the server), and its code called @forge/llm without declaring the llm module, causing Forge to reject the call outright and scoring its interpretation feature a zero.

Interestingly, Luna's UI was actually more polished — cards, live indicators, a clear legend, earning a perfect 1.0 on the visual tier, matching Haiku. But Luna scored only 0.48 on the functional tier versus Haiku's 1.0. A prettier shell couldn't compensate for missing history.

The "scope ledger" is the core challenge of this Forge task. It requires the application to track the complete history of every Jira issue entering and leaving an active sprint — not just responding to real-time webhook events, but also proactively backfilling all changes that occurred before the app was installed. This demands understanding the "event sourcing" design pattern: the real-time path and the historical replay path are two distinct code flows. An app that omits the latter will appear to "work normally" but expose a data void the first time a historical query is made. The screenshot evidence (every Source entry reading "event" rather than "reconcile") confirms Luna only implemented the former. This "looks like it runs, but silently loses data" failure mode isn't an overly harsh test criterion — in real payment or project management scenarios, silent data loss is often far more dangerous than a visible error.

What Other Benchmarks Show

Most independent leaderboards point in the same direction: Haiku scores higher and costs more per task.

On Vals AI's Terminal-Bench 4.0, Haiku scores 35.35% at $3.05 per test; Luna scores 13.64% at $0.35 — 8.7× the cost for 2.6× the score. Artificial Analysis's Intelligence Index places Haiku at 43 and Luna at 38, noting that Haiku uses roughly 162K output tokens per task versus Luna's ~50K. On WebDev Arena, they're virtually tied at 1587 vs. 1581, within margin of overlap.

Luna does lead in some areas: Artificial Analysis puts Luna's knowledge accuracy at 44% versus Haiku's 36%, and Luna also leads on AutomationBench (where AA suggests Haiku is penalized by excessive refusals). Each vendor's marketing materials use different harnesses, effort settings, and tasks, making apples-to-apples comparisons difficult.

Limitations Worth Naming Explicitly

The authors provide a lengthy list of ways their results could be wrong — and that honesty deserves its own section:

• The two runs weren't in the same environment: Luna built a week earlier using an older version of goose, a different machine, and a different scorer version. Haiku's wins on several timing-related checks were run on a different Mac.
• Luna's build files are on a laptop the authors can't access, so its draft behavior, Forge token count, and stopping reason are all unknown.
• The billing figures are a lower bound from OpenRouter; the re-pricing is a model, not an actual invoice.
• Forge 1.0 has confirmed scorer defects that inflate Haiku's lead over Sonnet and may make the top-five rankings unreliable — including Haiku's fourth-place finish. These defects do not, however, affect the backfill failure that determined Luna's score.
• Forge 1.0 is now frozen; the harder Forge 2.0 has launched, and neither Haiku nor Luna has results there yet.

An Honest Conclusion: Price It Per Delivered App

In these runs, Haiku 5.5 was the better agentic builder of the two — particularly on Forge, where the gap was decisive. It passed every core check that Opus 5.5 passed, for $0.52, while Opus spent $4.26. Luna built something that looked better, but never imported sprint history.

For picking a cheap model to build autonomously, the authors would choose Haiku — but with two caveats: actually enforce tool output limits, because one oversized result will push every subsequent call past the tier threshold; and budget at 3× the list price, unless you've already verified that your agent consistently stays under 100K tokens. The number to compare is "$0.52 to deliver a working Forge app," not "$0.10 per million tokens."

If your workload is thousands of small, focused calls — classifiers, routers, single-diff reviews — Luna is worth trying first. It's the same price below 100K tokens but produces roughly one-tenth the output per call that Haiku does. These tests didn't cover that scenario; what they showed was that when faced with an entire application to build, Luna exhausted its call budget on one and failed to import history on the other.

Two models at the same list price, with agentic real-world costs that differ by 2.3× to 4.8× — that's the line the marketing slides won't show you.