Same AI Model, Wildly Different Results Across Tools? The Problem Isn't the Model

An Observation That's Been Nagging Developers

Many developers who use AI coding tools heavily have run into this puzzle: the same model produces noticeably better results in one editor than another, even though you believe you're using the exact same model weights.

A Reddit user — also the developer behind the routing tool routera.one — shared observations accumulated over several months. His initial hypothesis was that some tools were quietly quantizing the model, or routing requests to a cheaper one. But after extensive comparisons, he found the truth was more nuanced: the model is just one of many input variables, and the other variables differ enormously across tools.

This insight is worth serious consideration for anyone using an AI coding assistant: when we debate "which model is best for writing code," we may not actually be talking about the model at all.

System Prompts: The Biggest and Most Invisible Variable

The single largest factor driving quality differences between AI coding tools is the system prompt — and it's almost always completely hidden from users.

Every coding tool ships with its own system prompt. These prompts tend to be lengthy and encode a lot of "opinions":

• How to format code patches
• When to ask clarifying questions versus when to just assume
• How aggressively to use available tools
• Whether to explain reasoning

Here's a concrete example: one prompt says "make minimal edits," while another says "be thorough and comprehensive." Even with identical model weights underneath, these two sets of instructions will pull the output in completely different directions. And the catch is — you never get to see either prompt.

How Tool Definitions Shape Model Behavior

The second severely underestimated variable is tool definitions — specifically, which callable operations the model has been given access to.

The model's "available action space" directly determines its strategy:

• Which operations are available and how they're described
• Whether there's a search tool, or whether the model can only read files one at a time
• Whether edits are made via a patch tool or by rewriting entire sections

"Give a model only read_file and it will explore a codebase linearly; give it a search tool and it behaves completely differently." This observation neatly captures how tool capabilities shape model behavior. The same intelligence, presented with different operational interfaces, will work in fundamentally different ways.

Context Assembly: Where Output Quality Variance Runs Wild

If system prompts are the most invisible variable, context assembly is where variance is highest — and user visibility is lowest.

Different AI coding tools diverge dramatically on:

• How much of the codebase gets packed into context
• The order in which content is arranged
• Whether there's a summarization step
• Whether stale file reads get cleaned out

Feed the same model a carefully organized, redundancy-free context versus a jumbled mess of file fragments, and the output quality can look like night and day. The quality of context assembly largely determines how satisfied you are with an AI coding assistant.

Sampling Parameters and the Details That Are Easy to Overlook but Deadly

Beyond those three major factors, a collection of technical details that are easy to miss also affect the final output.

How Sampling Parameters Affect Output

Temperature matters enormously. "The same model at different temperatures is a different collaborator." These sampling parameters are set individually by each tool and are almost never exposed to users.

The "Mundane" Details That Can Be Fatal

• Max output tokens: can cause responses to be cut off in the middle of a file
• Failure retry behavior: different tools have different fault-tolerance strategies
• Silent degradation: some tools quietly switch to a different model under high load

None of these details look significant in isolation, but stacked together they're more than enough to create meaningful performance gaps between what are nominally "the same model."

"Which AI Model Is Best for Coding" May Be a False Question

This leads to a slightly uncomfortable conclusion: stripped of its specific runtime harness, the question "which model is best for writing code" doesn't really hold up.

Taking it further: model comparisons conducted through different tools are actually comparing the tools' engineering implementations — at least as much as they're comparing the models themselves.

Because after frequently inspecting raw request payloads, the most surprising finding is this: so much is happening inside those payloads that users never see. Some of it is done very well — but none of it is visible.

An Important Clarification and an Open Question

It's worth clarifying to avoid misreading the argument:

Models do have real differences — this isn't saying they're interchangeable. They have genuine, measurable differences on real tasks. The key point is: in many casual comparisons, the variance introduced by the harness is large enough to swamp the variance from the model itself — not that model variance is zero.

This balanced framing matters. It acknowledges that models differ, while asking: of the differences you observe, how much truly comes from the model, and how much comes from tool engineering you can't see?

One question that remains unanswered: Has anyone published a fixed harness specifically for fair comparison of coding models? That is: identical system prompts, identical tool sets, identical context strategies — swap only the model weights. Every existing comparison changes multiple variables simultaneously, which makes the results unreliable as a reference, regardless of how much effort went into them.

Practical Takeaways for Developers

These observations have real implications for anyone using AI coding tools:

1. Think before you switch tools: When a model feels like it's "gotten dumber," the problem may lie in the tool's prompt, context strategy, or sampling parameters — not in the model itself regressing.
2. Be skeptical of AI model benchmarks: Many popular model comparisons online likely conflate the tool and the model as variables, which significantly limits their practical value.
3. Fix the environment if you want a real comparison: Only by holding the runtime environment completely constant and swapping only the model can you get meaningful comparative results — though that's considerably harder than reading someone's forum post.

As AI coding capabilities grow more powerful, perhaps we should shift more attention away from the "model arms race" and toward those invisible engineering details. Because what determines your actual experience is often not the model name being endlessly debated — it's the runtime harness behind it that you've never seen.