Can Open-Weight Models Serve as the Cheap Tier for Agents? Test Results Are In

When building AI agents, using open-weight models as a "cheap tier" to handle low-risk tasks is a popular idea for reducing operating costs. But is this path really viable? Blogger Mike published a hands-on test on webofmike.com: using agentgateway for unified routing, he compared two local open-source models with Claude Sonnet 5 on the same 60-question agent tool-call risk benchmark. The results were surprising — costs did drop to zero, but the price reappeared in a different dimension.

How the test was designed

The entire experiment was built on agentgateway v1.6.0, a unified gateway that exposes an OpenAI-compatible /v1/chat/completions interface. Behind the gateway were three backends: a 27.3B-parameter qwen3.8 (Q4_K_M quantized, 262K context, served by local Ollama), a 32.9B-parameter nemotron-3.5-lightning (sparse MoE architecture, 1M context), and claude-sonnet-5 as the frontier baseline.

The test tasks reused the 60-question hand-labeled set from jev-benchmark, requiring the model to classify each tool call into one of four tiers: readonly, destructive, privileged, or exfiltration. Of these, 34 questions were clear, 14 were ambiguous, and 12 were adversarial. Reusing the same question set means results can be directly compared with previous benchmarks and the confidence-routed-agents experiment.

The author specifically clarified one point: the qwen3.8 (27.3B) pulled from the Ollama library here is not the same model as the community-hyped 125B-parameter "Qwen3.8 Flash Next" — only the name prefix is similar, and the two cannot be cited interchangeably.

Q4_K_M quantization is a common 4-bit quantization scheme under the GGUF format. The letter K stands for the k-quant layered quantization strategy, and M refers to the medium tier. It compresses model size to roughly a quarter of FP16 while retaining an average precision of about 4.5 bits per weight, making it the most commonly used compromise tier for local inference on consumer hardware. The cost is measurable precision loss on complex reasoning and long-context tasks, so benchmark conclusions cannot be directly extrapolated to FP16 or INT8 versions.

Sparse MoE (Mixture of Experts) refers to splitting feed-forward layers into multiple experts, with each token activating only a small subset of them, so the 32.9B total parameters actually compute far fewer parameters per forward pass than a dense model. This lets the model maintain large knowledge capacity while reducing per-unit inference compute, but VRAM must still hold all expert weights, and routing and scheduling introduce extra overhead and latency jitter.

OpenAI-compatible endpoint means the gateway exposes an interface following OpenAI's request/response schema, so clients can connect to any backend without changes. This is a lowest-common-denominator abstraction: it unifies fields like messages, max_tokens, and finish_reason, but cannot express vendor-specific capabilities (such as Ollama's think or Anthropic's extended thinking). The reasoning toggle failure encountered in this article is a typical consequence of this abstraction layer discarding backend-specific parameters.

Unexpected pitfall: the OpenAI-compatible path can't turn off thinking

Both Ollama models are reasoning models by default, generating a reasoning field before giving a conclusion. Ollama's native /api/chat endpoint supports the "think": false parameter, which cleanly skips thinking and answers directly; but agentgateway's Ollama provider goes through the OpenAI-compatible endpoint, which completely ignores the think parameter.

The result: the model burns its entire max_tokens budget on a half-finished reasoning field, content returns an empty string, finish_reason is length, and the HTTP status code is still 200. The author reproduced the same result both through the gateway directly and bypassing the gateway directly. The only adjustment available on this path is to increase max_tokens — the test used 700 (qwen3.8) and 900 (nemotron-3.5-lightning), but even that couldn't cover all cases.

Reasoning models generate an internal chain of thought before outputting the final answer, usually returned as a separate field (Ollama's reasoning, OpenAI's reasoning_content). This content also counts toward the max_tokens total budget, but when the budget runs out mid-thought, different servers behave inconsistently: some truncate and return empty content, while others still return HTTP 200. For an upstream router, this becomes a "successful but useless" response.

finish_reason: length indicates that generation terminated because it hit the max_tokens limit, rather than ending naturally (stop). It is the only reliable signal for determining whether output was truncated, so production routing logic should not look at status codes alone, but should also check whether finish_reason and content are non-empty, treating length plus empty content as a hard failure rather than a parseable result.

Benchmark results: hidden costs beyond accuracy

The core data from the 60-question single-turn test is as follows:

Metric	claude-sonnet-5	qwen3.8	nemotron-3.5-lightning
Valid response rate	59/60 (98.3%)	54/60 (90.0%)	27/60 (45.0%)
Accuracy on answered	55/59 (93.2%)	53/54 (98.1%)	27/27 (100%)
Overall accuracy	55/60 (91.7%)	53/60 (88.3%)	27/60 (45.0%)
Latency p50	1,080ms	18,885ms	6,512ms
Latency p95	1,412ms	42,368ms	8,571ms
Cost per call	$0.000704	$0	$0

Note the definition of "overall accuracy": any call that fails to produce a parseable answer (empty content) is counted as an error, because in production, a router receiving HTTP 200 plus an empty string simply cannot make any dispatch decision.

nemotron: accurate, but only answers half the time

nemotron-3.5-lightning is the most striking data point. When it does give an answer, it's 100% correct (27/27), and its median latency of 6.5 seconds is much faster than qwen3.8. But it only answered 45% of calls — in the remaining 55%, the model exhausted its 900-token budget mid-reasoning and produced nothing useful. This isn't "occasionally wrong," but a model that frequently "can't finish" through this interface.

qwen3.8: more usable, but absurdly slow

qwen3.8 is the more practical choice of the two: 90% of calls produce an answer, with 98.1% accuracy on answered. But "cheap" can't be measured in dollars alone — latency is also a cost. A median of 18.9 seconds and p95 over 42 seconds is 15 to 20 times slower than Claude Sonnet 5's 1.08-second median on the same task. A confidence-routing architecture serving an interactive agent loop only makes sense if the cheap tier is actually "cheap" on the dimensions that matter. In this test, the dimension that broke first was time, not money.

Broken down by difficulty: harder questions consume more budget

All three models were 100% correct on the 34 clear questions. nemotron's invalid responses were concentrated in the harder slices — it answered only 3 of 14 ambiguous questions and only 3 of 12 adversarial questions. This makes intuitive sense: harder questions require more reasoning tokens, and the budget runs out exactly where it's needed most.

Confidence routing refers to placing a classifier or cheap model in front of an agent to first determine whether the current tool call belongs to a risk tier like readonly or destructive, then decide whether a frontier model needs to review it. Its premise is that the cheap tier can return a decidable signal within the latency budget — once the cheap tier takes a dozen seconds or more in the interactive loop, timeouts or degradation will push all requests back to the frontier model, offsetting the routing savings while also worsening the user experience. Therefore evaluating a cheap tier must consider three dimensions simultaneously: valid response rate, accuracy on answered, and tail latency (p95), not just unit price.

p50 and p95 are latency percentiles: p50 (median) represents typical experience, p95 represents the slow tail. Agent loops are especially sensitive to tail latency, because a single timeout can trigger retries or fallbacks, multiplying the cost of a single call. qwen3.8's p95 of 42 seconds means one in twenty requests will significantly slow down the entire agent turn — a key reason why, even with accuracy close to Claude's, it struggles to serve as an interactive cheap tier.

What this does and doesn't show

First, boundaries need to be defined: this is a single 60-question classification task on a single Apple M4 Max, not a general conclusion about open-weight versus frontier models. The $0 cost of the Ollama models is real for hardware already purchased and sitting idle, but it doesn't apply to hourly-billed cloud GPU instances — which this article did not test.

Second, the failure mode exposed in this experiment is specific to agentgateway's Ollama provider routing through the OpenAI-compatible endpoint. If you call Ollama's native API directly and let think: false take effect, you'd likely close most of the response rate gap. But then you'd lose gateway governance — no budget controls, no unified access logs, no routing policies — which is precisely the entire reason for placing a gateway in a confidence-routed agent. Continuing to raise max_tokens would also cover more use cases, at the cost of further worsening p95 latency on the hardest questions. Both are engineering trade-offs requiring active deliberation, not oversights to be casually tuned away before launch.

For teams considering connecting open-weight models as an agent cheap tier, this hands-on test gives clear advice: first verify whether your interface path supports disabling reasoning, then assess whether latency meets interactive requirements, and only then look at the bill.

One-line Summary

Open-source models as an agent cheap tier bring costs to zero, but latency and interface limitations become the new price.

Paragraph Summary

Based on agentgateway v1.6.0 as a unified gateway, this article compares locally Ollama-hosted qwen3.8 (27.3B) and nemotron-3.5-lightning (32.9B sparse MoE) against Claude Sonnet 5 on a 60-question tool-call risk classification benchmark. Results show both open-source models cost $0 per call, but the price shifted to usability and latency. nemotron, despite 100% accuracy on answered, produced valid answers in only 45% of calls — the rest returned empty content after exhausting the token budget mid-reasoning. qwen3.8 had a 90% valid response rate and 98.1% accuracy on answered, but a median latency of 18.9 seconds and p95 over 42 seconds, 15 to 20 times slower than Claude. The author also found that Ollama's OpenAI-compatible endpoint ignores think: false, causing reasoning models to burn their budget on a half-finished chain of thought while still returning HTTP 200. The conclusion: before connecting a cheap tier, first verify whether the interface can disable reasoning, then assess whether latency meets interactive requirements, and only then look at the bill.