Why LLM Orchestration Fails: Real Tests on Consumer Hardware

Introduction: Does a Better Model Make Orchestration Reliable?

Anyone running multi-agent systems locally has probably been through this loop: when an Agent executes a long task, it drops steps, treats delegations that should have been executed as text output, or returns empty results after tool calls. Intuition tells us the problem is that the model isn't strong enough.

The previous post finally got the 77GB coordinator model running stably—VRAM stopped thrashing, and the worker was moved to a second machine. So the question that had been postponed could finally be addressed head-on: now that a large model really can run, can a bigger coordinator solve the orchestration problem?

The answer is no. And that negative answer unexpectedly reshaped how the whole team works.

The Ceiling Remains: Big Models Also Fail

Once memory was no longer the bottleneck, the coordinator's orchestration ability could finally be evaluated on its own. Its performance was indeed more reliable than Gemma's—delegation order was more stable. But it couldn't escape the same failure modes:

• It still drops steps in long plans, issuing some delegations, quietly skipping others, then just writing a summary
• It occasionally outputs a delegation as text instead of actually executing it (the previous post recorded one instance: a handoff task printed as bare JSON and then abandoned)
• It periodically returns empty results after tool calls, requiring external prodding to continue:

⚠ Model returned empty after tool calls — nudging to continue

The frequency of this warning is high enough to serve as diagnostic evidence. The key point is that these are not errors of a weak model, but errors of a sufficiently capable model being asked to do something it is structurally unsuited for—maintaining a long control loop in its head while also reasoning inside that loop.

Diagnosis: The Ceiling Is in the Loop, Not the Model

The decisive observation: three failure modes—dropped steps, narrative replacing execution, and empty turns—appear in large models in exactly the same way as in small ones. The common factor is not model scale, but the agentic loop (agent loop) itself.

Asking a language model to do orchestration means asking it to play four roles at once: planner deciding the steps, scheduler issuing each delegation, bookkeeper tracking which delegations are not yet complete, and evaluator judging whether returned results are acceptable. This requires carrying a large amount of state across many turns of conversation, and cross-turn state retention is precisely the weakest link for this class of models.

A larger model only raises the ceiling a little; it doesn't remove it.

agentic loop (agent loop) refers to the control structure that makes a language model iterate repeatedly between "think—act—observe": the model first decides the next action (usually calling a tool or delegating a subtask), the system executes that action and feeds the result back into the context, and the model then decides the next step accordingly, looping until the task is complete or a termination condition is reached. The advantage of this pattern is flexibility—it can handle open-ended tasks; the cost is that every round must feed the full historical state back to the model, and the context window is limited while attention decays with length. Therefore, the longer the loop, the easier it is for the model to "forget" what it has already done and what remains undone. This is also why errors such as dropped steps, repeated delegations, and empty turns tend to cluster in the latter half of long tasks—the problem is not the quality of a single inference, but the loop's continuous demand for cross-turn state retention.

The Solution: Take the Loop Out of the Model's Hands

If the problem is the loop, then the loop should not be the model's job. The solution is deterministic orchestration: use a very small script to hold the control loop—it maintains the task list, dispatches each delegation, waits for and collects results, and decides what to run next. The model is invoked only for the two things it is genuinely good at: planning (what are the steps?) and evaluation (is this result acceptable?). It never needs to remember what it has dispatched, because the script keeps the books.

The improvement is immediate and measurable: for the same task sequence, the model-driven loop got stuck halfway, completing only three of five steps before losing the thread; the same plan handed to a script ran all five steps to completion. The model didn't become stronger—it was simply relieved of the work it couldn't reliably do.

Deterministic orchestration, as opposed to a model-driven autonomous loop, has as its core idea separating control flow from probabilistic components and handing it to ordinary imperative code. The script usually maintains an explicit task queue or state machine, with the transition conditions for each step hard-coded: take the next pending item, call the model to generate delegation instructions, execute the delegation, write the result back to the task record, and determine whether the completion condition is met. The model is downgraded to a stateless "function call," receiving only the current subproblem each time and returning structured output, holding no cross-turn memory. An additional benefit of doing this is observability and replayability—the inputs and outputs of each step are recorded by the script, so failures can be pinpointed to the exact step and rerun without polluting global state, whereas errors in pure model loops are often hard to reproduce and attribute.

An Expensive Lesson

When the agent loop is unreliable, the instinctive reaction is to switch to a bigger model. That instinct is wrong here, and costly—it is exactly the reasoning logic behind the coordinator search in Part 4 and the VRAM wall in Part 5, and in the end it did not solve the problem it was chasing.

Orchestration reliability is not a model capability that can be bought with more parameters; it is a property of the loop structure itself. The durable fix is architectural: move the deterministic parts—the loop, the bookkeeping, the dispatch—into deterministic code, and leave the model for the parts that truly require judgment. The control loop is the program's job, not the language model's job—the earlier this boundary is drawn, the more reliable the system becomes.

Cross-turn state retention is a measure of a model's ability to maintain consistent internal representations across multi-turn interactions, and it is a recognized weak point of current language models. Even when the context window is long enough, the model's utilization of early information drops significantly, and content in the middle positions is especially likely to be overlooked. For orchestration tasks that require dozens of consecutive rounds of delegation and waiting, this means the model is effectively "re-guessing what it did before" every round. Scaling up parameters improves the quality of single-round reasoning, but it cannot fundamentally change this mechanism—the state is not in the model's parameters, but in the conversation history, and the efficiency of using that history is limited by the attention mechanism itself. This is precisely the theoretical basis for externalizing the bookkeeping responsibility to a script: rather than hoping the model remembers, let the code remember.

Next: Execution Is Reliable—What About Judgment?

Deterministic orchestration makes the team's execution reliable: plans no longer drop steps, nor do they fabricate progress. But when a plan hits a wall—when the correct next step is simply not in the plan and someone needs to rethink the approach rather than just execute—it is powerless.

Reliable execution and good judgment are two different capabilities. The next post will discuss how to add the latter without sacrificing the former.

Key Takeaways

• Real tests on consumer hardware: after the 77GB coordinator model runs stably, orchestration failure modes still exist—model scale is not the cure
• Three typical failures: long plans drop steps, delegations are treated as text output, and tool calls return empty
• The root cause is cramming four state-intensive responsibilities—planning, scheduling, bookkeeping, and evaluation—onto the language model, when cross-turn state retention is precisely its weak point
• The solution is deterministic orchestration: the script holds the control loop, and the model is responsible only for planning and evaluation
• The same plan went from completing three of five steps to completing all five—the capability didn't change, the responsibilities did