Agent Sandbox in Practice: Boxing Software Factories with Three-Layer Architecture

Developers Are the Bottleneck: The New Turning Point in Agent Engineering

Amid numerous discussions about Agentic Engineering, YouTube channel IndyDevDan makes a striking point: If you're still inside the loop, you are the bottleneck. Most engineers are accustomed to running AI programming agents in a small corner of their own machines, or relying too heavily on CI/CD pipelines and containers. But engineers running high-intensity workloads at top AI labs stopped doing that long ago.

They've realized a simple truth: the more autonomous and secure an agent is, the more it can do for you with zero risk. The core of this article is how to get engineers truly out of the loop and achieve unprecedented leverage by putting a "Software Factory" into an Agent Sandbox.

Why Aren't Containers Enough?

Many ask: with Docker containers available, why do we need dedicated agent sandboxes? IndyDevDan provides three key reasons—true Isolation, amazing Scale, and Agency.

Containers do provide isolation, but they can't give you scale and agency. To understand this, you need to know the technical nature of container isolation: Docker containers implement process-level isolation based on Linux kernel namespaces and cgroups, with multiple containers sharing the same host kernel. This means containers share a kernel attack surface, and resources like CPU, memory, and disk I/O still compete on the same physical machine. For traditional microservice deployments, this level of isolation suffices. But for AI agents, the problem is that agents may need to execute arbitrary commands, install dependencies, modify system configurations, or even run unpredictable code—operations that carry escape risks in containers sharing a kernel. Agent sandboxes typically use lightweight virtual machine (microVM) technology, such as Firecracker or Cloud Hypervisor, providing complete kernel isolation at the hardware virtualization layer while maintaining startup speeds close to containers.

When you confine an agent to "one corner" of your computer, you're interfering with its work, and it's interfering with yours. The philosophy of agent sandboxes is: Give each agent its own entire computer. Just as developers have their own development machines, agents should have their own "development machines"—complete virtual machines with independent kernels and file systems—enabling them to work independently and take full responsibility for results.

Three-Layer Architecture: Boxing Software Factories

The author's system employs a clear three-layer orchestration architecture:

• Out-Sandbox Orchestrator: Runs on the local machine, responsible for launching and managing multiple sandboxes. The author uses Claude Code as the top-level orchestrator.
• In-Sandbox Orchestrator: Runs inside sandbox virtual machines, responsible for launching specific workflows within the box.
• ADW Agents (AI Developer Workflow Agents): The agents that actually do the work, including planners, builders, testers, documentation writers, etc., forming a complete software factory.

The author emphasizes that the core of this system isn't "throwing an expensive Claude Opus agent at it," but understanding which model should run where and what code it needs to pair with. This is the essence of a software factory: combining non-deterministic agents with deterministic code, taking the best of both. The "deterministic code" here refers to the Agent Harness—the control framework wrapping LLM calls, responsible for tool call routing, error handling, retry logic, context management, output parsing and validation. Unlike general agent frameworks such as LangChain or CrewAI, harness engineering emphasizes building precise control logic for specific workflows rather than relying on general abstractions. LLMs handle creative decisions; harnesses ensure process reliability and observability.

Technology Stack Choices

For specific tools, the author uses:

• Herder: Terminal multiplexing tool for observing multiple agents executing tasks simultaneously;
• Claude Code: As the top-level orchestrator;
• Pi (Agent SDK): As a custom agent harness, implementing "harness engineering." Pi doesn't try to be a universal framework but provides a minimal scaffold that lets engineers precisely control agent behavior at each step—embodying the harness engineering philosophy;
• exe.dev: As the primary sandbox tool, providing "persistent, fast, secure, shareable" sandbox virtual machines.

The author specifically notes that exe.dev differs from many other sandbox providers—the latter often give you only hours or days, while exe.dev is a virtual machine you truly own, which you can start and stop at will.

Best-of-N: Exploring Multiple Possibilities Simultaneously

The most exciting part of the entire demonstration is the Best-of-N mode. Best-of-N is a classic inference-time compute scaling strategy in machine learning. Its core idea stems from statistics: if the success probability of a single sample is p, then the probability of at least one success in N independent samples is 1-(1-p)^N, which rapidly approaches 1 as N increases. OpenAI has already proven in its research on the o1 model that increasing inference-time compute (such as sampling multiple times and selecting the best) is more cost-effective than increasing training-time compute for many tasks.

In the context of Agentic Engineering, Best-of-N is elevated from the token level to the entire workflow level—not sampling a single response multiple times, but executing the entire software development process multiple times independently, then selecting the best solution from multiple complete application outputs.

The author used just one prompt—asking to redesign a writing app called Inkwell into a "Quiet Room" style, making it more minimal and focused—then simultaneously launched five independent software factories, each running in its own agent sandbox with different configurations:

1. Default (default configuration, Gemini Flash + GLM 5.2 + GPT Luna)
2. Frontier (frontier model configuration)
3. Deepest Seek (pure DeepSeek V4 Flash configuration)
4. Open Weights (pure open-weight models)
5. Top Speed (fast model combination pursuing speed)

Each sandbox has an independent URL, allowing the author to separately observe the software factory's execution process and the final generated application. The value of this mode is: When you have a good idea, or multiple directions you want to try, you can run them in parallel and pick the best result. This approach is feasible precisely because the continuous decline in compute costs and maturation of sandbox technology have made the marginal cost of running multiple complete development processes in parallel acceptable.

Model Layering: From "Choosing Models" to "Combining Compute"

The author proposes an important mindset shift: If you're still agonizing over which model to use, you're already behind. He divides models into three tiers—State-of-the-art, Workhorse, and Lightweight.

He's particularly bullish on the newly released DeepSeek V4 Flash (0731), calling it "absurd for its price." DeepSeek is an AI company renowned for training high-performance models at extremely low costs, with its V3 model's training cost reportedly only about $5.6 million, far lower than Western models of comparable performance. V4 Flash continues this approach, using a Mixture of Experts (MoE) architecture that activates only some parameters during inference, dramatically reducing per-token compute costs while maintaining high-quality output.

In Artificial Analysis's cost benchmarks (an independent LLM performance benchmarking platform that comprehensively evaluates models across quality, speed, and price), DeepSeek V4 Flash sits in the lowest cost range with decent speed (about 110 tokens/s), having jumped from B-tier in the previous generation to the A-tier workhorse model range. The author calls it "stock-market-crash-level economics," perfectly suited to handle 90% of daily work. This trend confirms what the author calls "intelligence cascading down the model tiers"—today's cutting-edge capabilities will be replicated by cheap models in 6-12 months.

His core thesis is: Combine compute, don't select compute. You need a model stack, not a single model.

Test Results and Failure Case Analysis

The entire workflow ran for about 45 minutes, with interesting results:

• Top Speed configuration completed fastest, finishing in 14 minutes, consuming 2 million tokens;
• Deepest Seek completed a high-quality redesign at the lowest cost and relatively fast speed;
• Open Weights configuration's Kimi K3 model completely failed, unable to output correct JSON format;
• Frontier configuration's Opus 5 ran for 13 minutes in the planning phase, consuming massive tokens.

The author frankly states that precisely because failure is possible, the Best-of-N mode is so important—"blast radius is zero, the blast radius is just this box." Even if one configuration crashes, it doesn't affect other sandboxes or the local environment.

One detail: all Pi programming agents used OpenRouter provisioned keys, capped at $50, and destroyed the keys after completion. OpenRouter is a unified LLM API gateway that aggregates APIs from dozens of model providers including OpenAI, Anthropic, Google, Meta, and DeepSeek under a single interface, allowing developers to dynamically switch between different models with just one API key, without managing authentication and billing for each provider separately. Its provisioned keys feature allows users to create temporary keys with consumption limits that are destroyed after use—crucial for controlling costs and security risks when running untrusted agent code in sandboxes. The emergence of this model routing layer reflects an important trend in AI infrastructure: models are commoditizing, and true competitive differentiation is shifting to the orchestration, combination, and workflow layers.

The Essential Difference Between Agentic Engineering and Vibe Coding

Throughout the article, the author repeatedly emphasizes one distinction: This is Agentic Engineering, not Vibe Coding.

"Vibe coding is not knowing what your system is doing and not looking. Agentic engineering is understanding your system so thoroughly that you don't need to look."

The concept of "Vibe Coding" was first coined by Andrej Karpathy in early 2025, referring to developers "feeling their way" through programming by conversing with AI without fully understanding the code. It lowers the barrier to programming but also brings risks of uncontrollable systems. Agentic Engineering is the opposite—it requires engineers to have deep understanding of the entire system's behavior, only delegating execution to agents. While they may look similar on the surface (humans not directly writing code), Agentic Engineers are "thinking with systems thinking"—they consider the 100th run, the 1000th run, they consider observability, reusability, isolation and scale. The author warns engineers: Don't outsource your thinking to AI labs or third-party tools, stay close to the engineering itself.

He also offers a somewhat provocative judgment: "If you're using agents to directly modify application-layer code, you're wasting time." The rare exception is when you're "building the system that builds the system"—in that case, you do need to get hands-on, prompt back and forth, and watch the agent work. The logic here is: you should spend time at the meta-level—building and refining the software factory itself, not at the object-level—directly using agents to write business code. Once the factory is built, it can produce high-quality code automatically and repeatedly—that's where true leverage lies.

Toward the Age of Abundant Compute

The author's ultimate point is preparing for the "Age of Abundant Compute." He observes intelligence cascading down the model tiers—capabilities that were once cutting-edge are now becoming cheap workhorse models. This trend has deep economic and technical drivers: on one hand, open-source models (like Llama, Qwen, DeepSeek) continuously narrow the gap with closed-source frontier models through knowledge distillation and architectural innovation; on the other hand, competition in inference chips (NVIDIA, AMD, Google TPU, various AI ASIC startups) is driving down the cost per FLOP. When inference costs drop low enough, "trading compute for quality" (i.e., Best-of-N, multi-agent parallelism, large-scale verification) becomes economically viable.

He likens an engineer's role to a function: receiving input, executing, synthesizing, delivering output. Software factories plus agent sandboxes are the mechanism for amplifying the leverage of this function.

Great Agentic Engineering isn't about watching the loop constantly, but about showing up at critical moments—planning, reviewing, prompting, and verifying. The longer you stay in the loop, the more you are the bottleneck.

The author candidly concludes that as low-level leverage ("everyone using Claude Code, everyone having terminal agents") gets exhausted, the next stage of leverage requires genuine upfront investment and work. But for engineers willing to focus, this is precisely the best moment to seize the advantage. When everyone can use the same AI tools, differentiation will come from how you orchestrate those tools, how you design workflows, how you build reusable software factories—this is the core competitive advantage of Agentic Engineering.

Key Takeaways