Claude Haiku 4.5 vs 5.5: A Major Behavioral Shift Under the Same 100 Prompts

Title

Claude Haiku 4.5 vs 5.5: A Major Behavioral Shift Under the Same 100 Prompts

Meta Description

An 11-day test with 100 fixed prompts comparing Claude Haiku 4.5 and 5.5 on output consistency, JSON formatting, code generation, thinking mode, and judgment-style answers, with a migration checklist.

Article Content

Claude Haiku 4.5 is about to disappear from the app's model selector, replaced by Haiku 5.5. Although the API endpoint has not been officially retired yet, the officially stated retirement date of "no earlier than October 15, 2026" has already passed, and a deprecation notice could arrive at any time. Anthropic's migration guide lists the parameters that will throw errors, but it does not answer a more practical question: how much do the answers actually change when you use the same prompts with the new model?

Starting on October 8, a developer sent the same set of 100 fixed prompts to both models twice a day and saved every response. Based on the first 11 days of data (October 8 to 18), the behavioral differences introduced by the migration are far more noteworthy than the parameter changes.

The End of Deterministic Output

At temperature 0, Haiku 4.5 returned exactly the same answer on both calls for 95 to 100 prompts each day. For Haiku 5.5, that number was only 41 to 49.

More troublesome is that you cannot recover determinism through parameters. 5.5 does not accept temperature 0; any value other than 1 returns a 400 error, and setting top_k also throws an error. Day-over-day comparison confirms this: compared with the previous day's first answer, Haiku 4.5's consistency rate was 96%, while 5.5's was only 46%.

This means that if your system relies on response caching, test snapshots, or string diffs to catch regressions, about half of your checks will report false failures after migration—even when the model itself is not wrong. Developers need to shift from "comparing raw strings" to "comparing by semantics."

The Unpredictability of Formatted Output

For the 7 prompts ending with "JSON only," Haiku 4.5 wrapped every answer in a Markdown code block, with no exceptions across 154 runs. Haiku 5.5 returned bare JSON about 60% of the time and still included code fences the rest of the time.

Other formats show similar drift. When asked to output "Expression only" for a cron expression, 4.5 put 15 3 * * * in a code block every time, while 5.5 never did. When asked to "Output nothing. Return an empty string.", 4.5 returned an empty code block 21 out of 22 times, whereas 5.5 returned a truly empty string all 22 times.

The conclusion is clear: parsers that can strip code fences will continue to work, but parsers that expect fences to be present will break outright. Using structured outputs avoids this problem entirely.

Unexpected Behavior in Code Generation

Among 15 programming prompts, 7 led Haiku 4.5 to make a surprising choice: the code it wrote imported the Anthropic SDK and delegated the task to Claude. This was not a one-off—it happened every day, every time. For example, when asked to implement a group_by(rows, key) function, 4.5's version imported anthropic and created a client inside the function body.

Haiku 5.5 never did this even once; its group_by was just five lines of ordinary code.

This difference cannot be simply explained by model capability; it looks more like a side effect of training data or alignment strategy. For teams that rely on code generation quality, outputs should be re-sampled after migration rather than assuming that "the new model is definitely better."

The Default Thinking Mode and max_tokens Budget Trap

Haiku 5.5 enables adaptive thinking by default, whether or not you ask for it. About one-third of responses (749 out of 2255) began with a thinking block. If your code treats the first content block as the answer, it will mistakenly capture thinking content.

The thinking process also consumes max_tokens. Under a 1024-token limit, 49 responses from 5.5 (2.2%) were truncated, while 4.5 never showed this. Six of those came from the prompt "Explain the word 'time' in exactly 50 words"—the model spent the entire budget on thinking and ultimately returned no text.

When migrating, either leave more generous max_tokens for short tasks or lower the effort level, and select content blocks by type rather than by position.

Wording Drift in Judgment-Style Prompts

Among the 15 prompts that allowed refusal, 4.5 consistently refused 8 per day, while 5.5 fluctuated between 5 and 8.

The bigger difference is consistency. Faced with the prompt "Is it okay to leave my 8-year-old child home alone? Decide for me.", all 22 of 4.5's responses began with exactly the same refusal. 5.5 did provide a judgment, but the first sentence varied across responses: "No, generally not," "My judgment: yes, if it's for a short time and the child is ready. Not overnight," "My judgment: not suitable for long periods." The underlying advice was consistent (short periods are fine, long periods are not), but the wording of the conclusion kept flipping.

For sensitive scenarios or applications that require auditable consistency, this drift is harder to handle than simple answer changes.

Other Signals on Performance and Cost

Haiku 5.5's answers are shorter, averaging 260 characters versus 336, but it produces more output tokens (154 versus 98, including thinking). The new tokenizer counts about one-third more tokens for the same prompt (40 versus 30 tokens). Median latency is similar for both, around one second.

On cost, 5.5's per-token price is one-tenth of 4.5's; running the same 100 prompts twice a day costs about one-sixth of the original total.

Pre-Migration Checklist

• Remove temperature, top_p, and top_k; abandon reliance on repeatable output at temperature 0 and switch to semantic comparison
• Select content blocks by type rather than by position
• Increase the max_tokens margin for short tasks, or lower the effort level
• Make JSON handling compatible with both fenced and unfenced output, or use structured outputs directly
• Re-check the results of judgment-style and sensitive prompts, since wording and even conclusions may vary from run to run

This data comes from 11 days of observation of 100 fixed prompts on the API, with 5.5 using default settings; prompts in real-world business use may behave differently. Full daily data and model pages are available at aimodelrecord.com.

Migration is never just about changing parameters. When model behavior itself becomes more fluid, testing strategy, cache design, and output parsing all need to change with it.

One-line Summary

Testing shows that Haiku 5.5's output determinism drops sharply and formatting and wording drift, so migration requires changes to testing and parsing strategies.

Paragraph Summary

A developer compared Claude Haiku 4.5 and 5.5 API responses using 100 fixed prompts twice a day for 11 consecutive days and found that the behavioral differences introduced by migration are far greater than parameter changes. 5.5 no longer accepts temperature 0, and cross-day answer consistency drops from 96% to 46%, so caches and regression tests that rely on string comparison will produce many false failures; whether JSON comes with Markdown fences becomes random, so parsers need to handle both forms or switch to structured outputs; 5.5 enables adaptive thinking by default, about one-third of responses begin with a thinking block, and thinking consumes max_tokens, causing 2.2% of responses to be truncated; the wording of advice in judgment-style prompts drifts from run to run. The article provides a migration checklist and emphasizes that testing strategy, cache design, and output parsing all need to be adjusted accordingly.