How Much Scaffolding Does a Capable Agent Actually Need for Autonomous ML Engineering?

The Overlooked Question: Is More Scaffolding Always Better?

Autonomous Machine Learning Engineering (MLE) is quickly becoming one of the hottest battlegrounds in AI Agent research. In recent years, various MLE Agents have made remarkable strides on public leaderboards, frequently pushing the state of the art. But behind this progress lies a trend worth scrutinizing: to keep Agents moving forward on long-horizon tasks and to push past the native limitations of large language models (LLMs), researchers have been stacking increasingly complex "harnesses" — multi-agent orchestration systems, dedicated retrieval sub-agents, and elaborate task decomposition modules.

These mechanisms do deliver meaningful performance gains, but they also raise a critical question: How much scaffolding does a truly capable Agent actually need to handle autonomous machine learning engineering? As the underlying LLMs themselves grow more powerful, are these external structures still necessary — or are they gradually becoming redundant, even a liability?

Where Complex Scaffolding Came From

MLE tasks are inherently long-horizon work: understanding the problem, designing a solution, writing code, training models, debugging, evaluating, and iterating — the full pipeline often demands dozens or even hundreds of sequential decisions. Early LLMs were prone to "progress stagnation" on tasks like these — running in circles, losing direction, or failing to recover from errors.

To compensate, the dominant approach became "equip the model from the outside":

• Multi-agent orchestrators: Breaking complex tasks into roles — planner, executor, reviewer — with different agents handling each.
• Dedicated retrieval sub-agents: Pulling relevant information from documentation, past experience, or external knowledge bases to patch the model's context and knowledge limitations.
• Complex state management and memory mechanisms: Helping Agents maintain consistency across long workflows.

The logic was straightforward: if a single LLM call has limited capacity, compensate with engineered system architecture. So the scaffolding kept growing, and the system started to resemble an elaborate production pipeline.

The challenges long-horizon tasks pose to LLMs have deeper technical roots. LLM inference is inherently stateless — each call can only process a finite context window, with no natural working memory across steps. When an MLE task requires dozens of iterative rounds (e.g., exploring data, adjusting feature engineering, training, then backtracking based on validation results), this statelessness causes the model to "forget" the premises of earlier decisions, leading to contradictory actions or losing sight of the global objective amid local optimization. Multi-agent orchestration architectures attempt to externalize memory and coordination through systems engineering — storing global state in external data structures and having the orchestration layer pass context between agent calls. This design borrows from software engineering principles of "single responsibility" and "separation of concerns." It was genuinely effective at reducing per-call complexity when early LLMs were weaker, but the tradeoff is that end-to-end behavior becomes difficult to trace.

The Road Less Traveled: Simpler but Stronger Coding Agents

In contrast to stacking complex scaffolding sits a long-overlooked alternative: more primitive but more capable coding Agents.

The core idea is to let the LLM directly interact with the execution environment through a minimal set of primitives:

• read: Read file contents
• write: Write or modify files
• bash: Execute command-line instructions

In other words, instead of wrapping the model in layers of intermediaries, treat it like an engineer with terminal access — let it read code, write code, run commands, and observe results. This minimal interaction style grants the Agent a high degree of freedom and flexibility.

As the coding and reasoning capabilities of underlying LLMs continue to advance rapidly, this "light scaffolding, strong model" approach deserves a fresh look. A sufficiently capable model, paired with direct environment access, may be able to handle tasks that previously required complex multi-agent systems.

This read/write/bash interaction pattern aligns closely with the Unix philosophy — "do one thing well, compose tools through the filesystem and pipes." Via bash, an Agent can invoke any command-line tool (conda, pip, git, nvidia-smi, etc.); via read/write, it directly manipulates code files and logs. The entire execution environment becomes the Agent's "workbench," requiring no intermediate translation layers. The key advantage of this design is an extremely tight feedback loop: the model executes a command and immediately sees stdout or error output, adjusting strategy in the very next step — almost identical to how a human engineer works in a terminal. The evolution of code agent benchmarks like SWE-bench supports this trend: top-performing systems tend to keep their tool interfaces minimal, concentrating complexity in the model's reasoning rather than in external orchestration logic.

The Core Tradeoff: Intelligence in the Model or in the System?

At its heart, this is a debate about where intelligence should live.

The heavy-scaffolding approach embeds much of the intelligence and logic into the system architecture. Its advantages are controllability, interpretability, and reduced dependence on raw model capability. Its disadvantages are system complexity, high maintenance costs, and the risk that carefully engineered workflows become obsolete as models improve — structures built to compensate for an older model's weaknesses may be pure overhead for a newer one.

The light scaffolding + strong coding Agent approach bets on the model itself. It's simpler, more generalizable, and naturally benefits from model upgrades as they come. But it demands a great deal from the underlying model — when the model falls short, long-horizon tasks can quickly spiral out of control or stall.

This is the central tension in Agent engineering today: As models grow more capable, should we keep thickening the scaffolding — or should we trust the model to do more on its own?

This tension is not new to AI system design. A similar debate played out between deep learning and feature engineering: early machine learning depended heavily on hand-crafted feature extraction pipelines ("intelligence in the system"), but the rise of deep learning showed that a large enough model, given enough data, could internalize those hand-crafted rules one by one. The MLE Agent field is approaching a similar inflection point. Notably, the relationship between "scaffolding complexity" and "system performance" is not monotonically increasing — excessive task decomposition can sever the Agent's ability to reason holistically over the full context, and interface overhead between sub-agents can cancel out the gains from specialization. This is therefore not just a tradeoff between model capability and engineering effort, but a deeper choice between "emergent holistic ability" and "decomposed controllability."

Practical Takeaways for Practitioners

For teams building MLE Agents or other autonomous agent systems, this discussion offers several concrete points to consider:

First, don't default to "more complex is better." Complex multi-agent orchestration may boost leaderboard scores, but it also makes debugging harder, raises costs, and introduces fragility. Before adopting it, ask honestly: are these mechanisms compensating for genuine model weaknesses, or duplicating capabilities the model already has?

Second, retest simple baselines with the latest generation of models. Much of today's complex scaffolding was designed around earlier, weaker models. It's worth running a controlled experiment with a minimal setup — just give the model read/write/bash access — to see whether the gap has narrowed or even reversed.

Third, think about long-term maintainability. In a landscape where models evolve rapidly, over-customized system architectures tend to become technical debt. Keeping architecture simple usually means better adaptability going forward.

Conclusion

"How much scaffolding does a capable Agent need" doesn't have a universal answer — it depends on task difficulty, model generation, and cost constraints. But the question itself is a useful reminder: in the current frenzy of stacking agent engineering mechanisms, revisiting the simpler path of "let a strong model do the work directly" may yield surprising gains in both efficiency and elegance. As LLMs' native capabilities continue to advance, the balance may be quietly shifting.