Claude Code Orchestrating Local LLMs: 78 vs 76 in Real Testing

Splitting Work into Stages Lets a Local Model Pull Ahead

Running Claude Code with a local LLM is cheap and keeps your data private, but the model's capabilities usually lag behind. A developer rented an RTX PRO 4000 Blackwell (24GB, $0.57/hour) on RunPod, ran a Qwen3.8-27B Q4_K_M model on llama.cpp, and set up a controlled experiment: let Claude act as the "foreman," handing five small modules off to the local model one by one, then reviewing each, and sending them back for rework. The control group, meanwhile, fed the entire spec to the local model in one shot.

Claude Code using a local LLM

The result was clean-cut: in orchestration mode, the local model scored 78/78 and 77/78; in single-shot mode it only got 76, 74, and 71 (out of 78). All five test runs shared the same GPU, the same rubric, and the same hidden test set.

The more telling number isn't the total score but the "max prompt length." In orchestration mode, no single task prompt exceeded 8K tokens; in single-shot mode the max prompt hit 32,001 tokens, just 800 shy of the 32K context limit. Once the context fills up, the model starts losing track of key information — the longest-running attempt actually scored the lowest.

How the Experiment Was Set Up

The architecture had two roles. Claude ran in a Claude Code session on the developer's Mac, responsible for planning, writing task prompts, reviewing code, writing probe tests, and deciding what to send back for rework. The local model was the "worker," invoked via a Python script worker.py. Each task opened a stateless conversation, equipped with four tools:

list_files / read_file: reads restricted to the workspace directory
write_file: also path-restricted, rejecting absolute paths, .., and symlink escapes
run_command: only allows python3 scripts, python3 -c, python3 -m unittest, ls, cat, pwd, and only relative paths

The server startup command was one line: llama-server -m Qwen3.8-27B-Uncensored-Q4_K_M.gguf --host 127.0.0.1 --port 8080 -c 32768 -ngl 99 --jinja -fa on --alias local. It listens on a local port, and the Mac connects via an SSH tunnel, bypassing the public proxy on RunPod.

The local model was Qwen3.8-27B's Q4_K_M quantized version, 16.8GB, which just fits into 24GB of VRAM. Temperature 0.2, chain-of-thought disabled, max output 4096 tokens.

Q4_K_M and llama.cpp's quantization mechanism. Qwen3.8-27B's original weights in bfloat16 require about 54GB of VRAM, far more than a single 24GB card. Q4_K_M is the codename for llama.cpp's GGUF quantization scheme: Q stands for quantization, 4 means weights are compressed to roughly 4 bits on average, K indicates k-quant block quantization, and M means medium mixed precision. It doesn't just chop every tensor down to 4 bits uniformly; it preserves higher bit widths for precision-sensitive layers like attention projections and feed-forward, and compresses more robust layers harder. The final model file is 16.8GB, so a 17B-class model can fit into 24GB along with the KV cache. Quantization inevitably raises perplexity, but for code completion tasks, the degradation at 4 bits is usually acceptable — which is why it's the most common choice for local deployment.

Temperature 0.2 and disabled chain-of-thought. Temperature controls sampling randomness; 0.2 approaches greedy decoding, making output highly consistent across multiple runs with the same prompt — crucial for experiments that need to reproduce defects and score stably. Disabling chain-of-thought means the model doesn't output intermediate reasoning before answering, directly filling in tool calls and code. This saves tokens and reduces latency, and avoids long reasoning chains eating up precious context budget; the cost is weaker performance on complex logic tasks, which also explains why an explicit review stage matters so much.

Why Pick a Project Full of Sharp Edges

The developer wrote a toy project called minikv: a key-value store implemented in pure Python, with five modules depending on each other in layers, each hiding a corner that trips people up.

Module	Trap
`store.py`	TTL rounding, expired keys still lingering in the dict
`txn.py`	Keys deleted in a transaction must return as "nonexistent" even if they're still in the store
`protocol.py`	Quoting rules, error ordering, KEYS accepting no arguments
`aof.py`	Truncated last line must be ignored, empty lines preserved
`session.py`	Replaying the log must not write the replayed lines again; transactions must buffer their own log lines

The test set has 78 unittest cases: 20 visible to the worker, 58 hidden. The scoring script takes the denominator from the test file, not from the submission; the sandbox only copies the five spec modules; Python is run with -I isolation mode — faking unittest.py or tampering with the summary results won't help.

How Orchestration Mode Runs

The five modules were split into tasks in dependency order. The first stage sent three independent modules in parallel; the slowest finished in 87 seconds. Claude then reviewed each against the spec, probed edge cases with one-liner python3 -c commands, and sent back issues it found. At most two rounds of rework; anything past that was scored as-is.

Each task prompt contained: the module's spec text, the interfaces of the modules it depends on, the test command, and the most important line — "the final score uses additional tests you can't see, so implement strictly according to the requirements."

Rework prompts contained only three things: what the defect was, how to reproduce it, how the spec text described it. No hints about the cause.

Single-shot mode fed the entire spec in one go, all five modules stuffed into one prompt, same tools, same visible tests, budget 90 turns.

Stateless conversations and prompt accumulation. The repeatedly mentioned "max prompt length" is the key to understanding the whole experiment. The local model uses stateless calls: worker.py opens a fresh conversation for each task, the model doesn't remember what it did last turn, and all context is fed in through the prompt in one shot. In orchestration mode, each worker only gets its own module's spec and dependency interfaces, so prompts are naturally short. But single-shot mode stuffs all five modules into one prompt, and then every turn's tool results get appended back into the same conversation — the context grows monotonically, approaching the 32K limit by turn 90. This is a direct consequence of the "stateless + accumulating prompt" structure, not the model itself getting dumber.

--jinja and tool calls. For the local model to call tools like list_files and write_file like Claude does, the server must render tool definitions into the prompt using the model's own chat template, and correctly parse the model's structured output. --jinja makes llama-server use the Jinja template embedded in the GGUF to do this. If the template doesn't match, the model outputs malformed tool calls, and the whole orchestration workflow breaks. This is why tool-calling ability is a hard requirement for "a local model as worker," not just a matter of model scores.

What the Review Stage Caught

Every defect the review caught had passed the visible tests — which is exactly the point of hidden tests.

Module	Defect	Round 1	Round 2
`protocol.py`	KEYS command not implemented at all	Fixed in one round	Fixed in one round
`txn.py`	Deleted/expired keys still treated as existing	Fixed in one round	Fixed in one round
`txn.py`	Keys deleted by an outer transaction still visible in a nested inner transaction	Didn't appear	Still unfixed after rework rounds exhausted
`session.py`	Replaying the log at startup writes each line back, file doubles on every restart	Fixed in round 2	Fixed in one round
`session.py`	Inner ROLLBACK retains that transaction's log lines	Fixed in round 2	Didn't appear

The most telling one was the log-replay bug. All three single-shot runs delivered with this bug, and two orchestration-mode workers also wrote the same code. The difference was simply: Claude read __init__, saw the replay loop called execute(), and knew execute() writes logs. It's a bug that's easy to write and easy to spot — provided someone is looking.

The developer's one-line summary nails it: The local model will tell you it's done, but the replay bug is still there.

Hidden tests vs. visible tests. This is the most important evaluation design in the experiment. The 20 tests the worker can see guide development; the 58 hidden cases are what's actually used for scoring. If only visible tests exist, the model writes code precisely targeting them — the fact that all those defects passed the visible tests is proof: visible tests are the model's objective function, hidden tests are the real rubric. The scoring script taking the denominator from the test file rather than the submission, the sandbox copying only the five spec modules, and running Python with -I isolation mode — all these close off the path of "faking unittest.py or tampering with summary results." A lenient visible test set is almost a net negative for a local model; it teaches the model to pass checks rather than satisfy the spec.

Why the reviewer can see bugs the worker can't. The log-replay bug escaped all three single-shot runs because the worker only focused on the function it was currently writing. Claude read __init__, saw the replay loop called execute(), and connected the dots that execute() writes logs. This is a classic cross-module visibility problem: the cause and effect of a defect are scattered across different files, and only someone who has read through the whole structure can connect the two. The value of the review stage isn't being smarter — it's standing at a vantage point where the full picture is visible.

The second orchestration run lost one point, and deservedly so. The spec required: only quote when the argument is empty, contains whitespace, or contains double quotes. The worker additionally quoted values containing backslashes, so format_command output SET k "a\\\\b" where the hidden test expected SET k a\\b. Both forms parse back the same way, so a round-trip check couldn't detect it — catching it would require character-by-character comparison of the output, which the review didn't cover.

Also, the nested-transaction bug remained after the second rework round was exhausted, but none of the 58 hidden tests covered this scenario, so no points were lost. The review found defects the tests couldn't see; the tests found defects the review missed. Neither is a complete checking mechanism.

The More Turns You Give It, the Worse Single-Shot Mode Gets

The most easily criticized part of the first round was the unfair budget: orchestration used 88 turns, single-shot only 40. So this time single-shot got 90 turns, and one run had duplicate protection disabled to allow using them all.

It used all the turns and scored 71/78, the lowest of the five. At the end, prompt length reached 32,001 tokens, just 800 from the 32,768 limit, because every turn appends tool results into the conversation. The two shorter runs failed 2 and 4 hidden tests respectively; this one failed 7, including transaction log tests that shorter runs had passed.

Same model, longer rope, worse code. The two runs with protection enabled stopped themselves at 23 and 11 turns respectively, with better results.

The real source of the gap isn't that orchestration mode got more turns — it's that each turn happened in a short, focused conversation, with a reviewer deciding what to discuss next.

Real Pitfalls Hit

The command filter is not a sandbox. In one rework, the worker spent 12 of its 20 turns repeatedly running the same check script, changing the filename each time: /tmp/aof1.txt to /tmp/aof12.txt. The absolute path was hidden inside a python3 -c string, the filter couldn't parse it, and it wrote 11 files in the developer's Mac /tmp. And because the filename differed each turn, no two turns were identical, so duplicate protection never triggered. The worker's code runs as you, on your machine — don't keep anything important in the workspace.

finish: done is just the model's opinion. Every task that ended with defects came back with an enthusiastic summary saying "all tests pass." And they did — the visible ones.

max-turns means you have to read the code yourself. Three worker calls exhausted their turns in the orchestration run; one left code that passed visible tests but hid two defects. Status only tells you it stopped, not what it left behind.

The second rework must spell out where to change. The first session rework (defect + reproduction + spec) changed nothing; the second named the method and described the fix direction, and it was done in seven turns. This is the orchestrator doing more work, which contradicts the claim that "Claude only reviews."

Command whitelisting vs. OS-level sandboxing. The point that "the command filter is not a sandbox" deserves emphasis. The protection here is implemented at the Python layer with string matching and argument validation: only python3, ls, cat, etc. are allowed, absolute paths and .. are rejected. But the model only needs to hide an absolute path inside a python3 -c string for the filter to miss it. A true sandbox should enforce restrictions at the OS level — containers, chroot, a separate user identity, read-only mounts, resource limits — so that even if a command is allowed, out-of-bounds writes are physically impossible. String filtering can never keep up with the model's imagination; it stops accidents, not any purposeful bypass. In the experiment, the worker wrote 11 files into the developer's Mac /tmp — that's the price of lacking OS-level isolation.

Boundaries of This Approach

The developer honestly listed several limitations:

Only two runs per condition, too small a sample. Models aren't deterministic; with the same prompt, single-shot mode scored 76, 74, and 71. The gap held in every paired comparison, but don't take the decimal place seriously.
The second round's review wasn't fully blind. Before the second round, when scoring single-shot mode, Claude had glimpsed a brief failure summary of hidden tests. Though the second-round review still followed the same probes and spec-reading standards, the notes had been seen.
The spec must be precise for the review to have grounds. If the spec is vague, the review can catch less. Part of the benefit comes from the precise spec — which applies to humans too.
The orchestrator's cost isn't counted. Claude's tokens are the expensive part; the comparison above only covers the worker side.
Context was never the constraint in orchestration mode. Max prompt under 8K — this project only proves that decomposition keeps the model away from the context limit, not that it can rescue a codebase that won't fit in 32K.

Takeaways Worth Carrying

Local models are already good enough for small, well-specified tasks. The developer put it plainly: "I'll use it for Python or Bash automation scripts, but I won't let it touch my database."

The entire session — two conditions, five runs — consumed 42 minutes of GPU time, about $0.40. What really makes the approach work is the review stage.

In other words, if you want to save money with a local model, what you're really buying isn't model capability — it's a "write–review–revise" workflow, plus a sufficiently precise spec. The local model types, Claude thinks, and defects die in review.