Self-Hosting Open-Source Models for Coding Agents: The Model Is the Easy Part

More and more teams want to move their coding workloads off frontier APIs and onto self-hosted open-source models. But once you actually start, you discover that model selection is the easiest step in the whole engineering effort. We spent several months testing a batch of open-weight models on real software engineering tasks, pairing each model with hardware that could actually serve it, and swapping different agent harnesses on top. The measurements point to three counterintuitive conclusions.

Capability Pricing Is a Step Function, Not Linear

The common intuition is that stronger models cost proportionally more to run. That's not what happens. Open-source model quality improves gradually across the board, but the underlying hardware cost jumps discontinuously — the key question is whether the weights fit into the memory of the machine you already have.

Once you cross that line, you're not paying the price of a few more B parameters — you're paying the price of a whole new machine. So some models are terrible value and some are excellent, and this has little to do with where they sit on the leaderboard.

In our test set, a mid-sized model cost 4x as much to deploy as a smaller one, but improved application performance by only about 2%. Another, much larger model cost only slightly more than the tier below it, yet solved significantly more tasks.

The genuinely useful question isn't "which model is best" but "which model lands just inside a hardware tier" — and you need to measure it by cost per solved task, not cost per token.

What "weights fit into memory" means: When you serve an open-source model on a GPU, parameters aren't the only thing in VRAM. Weights, KV cache (which grows with concurrency and context), activations, and framework overhead all share the same memory. Crossing the threshold of "this model fits on the machine I already have" means either moving to a card with more VRAM or going multi-GPU, and the whole-machine cost jumps. This also explains why adding a few B parameters within the same tier costs almost nothing extra, while the next tier of memory requires a whole new machine.

Concurrency Is the Variable Nobody Budgets For

With an API, concurrency is someone else's problem. You get relatively predictable response times and never have to think about it.

Once you run your own stack, concurrency becomes a hard constraint. Every user brings a context, every context has to fit in memory, and latency becomes unacceptable long before the hardware is truly exhausted.

Across our configurations, how many developers a single machine can comfortably support varied enormously — and in the opposite direction from what the spec sheet implies. The largest, most powerful configuration served the fewest people at once.

This is where most self-hosting business cases quietly collapse. The weaker configurations we tested needed breakeven utilization far above what enterprises actually achieve. Real deployments already report low utilization, and well-managed clusters aren't much better. If your plan requires running near full capacity, it won't work.

The Harness Changes the Bill, Barely the Application Performance

We compared three widely used coding harnesses on identical hardware and identical inference settings, expecting the orchestration layer to make a huge difference. In solve rate, it barely moved the needle — the gap between harnesses was a few tasks either way.

But on cost, it mattered a lot. Same model, same hardware, solving the same tasks, total wall-clock time differed by 2x depending on the pairing. When you rent GPUs, wall-clock time is what actually defines your cost.

And once you've picked a harness, evaluation is critical, because things break easily. In our experiments, one harness was the clear winner — until the next software release, when performance fell off a cliff. The cause was a small change in how prefix caching was handled; the cache mechanism suddenly stopped working entirely, and GPU rental costs spiked 5x.

The key takeaway: pick the harness your developers want to use, then monitor it closely, because a point release can affect your costs more than swapping models.

Why prefix caching determines cost: In coding agent scenarios, the system prompt, tool definitions, and repo context often make up the bulk of the input and reappear on every turn of the conversation. Prefix caching lets the KV computation for these fixed prefixes be reused, directly eliminating a large amount of prefill compute. Once its handling is changed or breaks, every request has to prefill from scratch, GPU compute time and wall-clock time rise together, and the bill reflects a multiple-x cost difference.

What This Means for You

The question isn't "is self-hosting cheaper." That question has no answer, because it depends on your workload, your peak, and your tolerance for a slower agent.

The better questions are: what is your real utilization? And what are the things you absolutely cannot put on someone else's API? Data that isn't allowed to leave the building, a stack nobody can rate-limit, prices that won't change mid-quarter — those reasons survive a spreadsheet. Purely cost-based reasons usually don't.