Building an Autonomous A/B Testing AI Agent with Docker Cloud Sandboxes

Imagine an AI agent that independently handles an entire A/B testing cycle — playing the role of a user to experience a product, analyzing pain points, writing fix code, and ultimately scoring each version to pick a winner, all without human intervention. That's exactly the system we're building here. It runs on Docker Cloud Sandboxes (ephemeral cloud microVMs) and orchestrates a multi-agent workflow with roughly ten SDK calls.

The full source code is open at github.com/ajeetraina/sbx-ab-agent. Let's break down the design philosophy and key implementation details.

What the System Actually Does

Picture a bare-bones online store: all products dumped into a single long list, with no search, no filtering, and no sorting. This is an intentional "problem" waiting for an AI agent to discover and fix.

The automated flow boils down to six steps: deploy the store into a secure cloud sandbox; send several "pretend shoppers" (AI agents) to experience it and score it; have an "analyst" agent read all the shopper reports and propose a few fix variants; hand each variant to a "builder" agent to write code inside its own isolated sandbox; run the same shoppers against each new version; and finally pick the highest-scoring version and copy its code back locally.

This is the essence of A/B testing — build a few versions, see which one is better. The original store is "A"; the analyst's fix proposals are "B" and "C". Each variant is "the same store + one small change," ensuring a fair comparison.

Why Sandboxes Are Non-Negotiable

The builder agent writes and runs code that nobody has reviewed. You definitely don't want that running directly on your laptop. A Docker Cloud Sandbox is essentially a disposable cloud microVM — and it provides four critical capabilities.

Security comes first — even if an agent does something stupid, only the sandbox itself is affected. Isolation ensures that fix B and fix C each get their own sandbox and never interfere with each other. Real links mean each sandbox gets a public HTTPS URL, so you can open each version in a browser and see it with your own eyes. Secret safety means your OpenAI key is stored on the Docker side as a secret and temporarily attached when a sandbox makes a request — the key never lives inside the sandbox itself.

All of this is orchestrated by a small TypeScript program using the @docker/sandboxes SDK.

The microVM technology underlying Docker Cloud Sandboxes is fundamentally different from ordinary containers. Regular Docker containers share the host kernel — if a container escape vulnerability is exploited, an attacker could gain host-level privileges. MicroVMs (like Firecracker, open-sourced by AWS) run a separate lightweight Linux kernel for each instance, providing hardware-level isolation that makes container escapes virtually impossible. This is the prerequisite for letting AI agents run unreviewed code: even if an agent writes a malicious script or triggers a security vulnerability, the damage is strictly contained within that one microVM, which is destroyed after use without a trace. Compared to spawning a local Node.js subprocess to execute AI-generated code — or using a shared-kernel ordinary container — the difference in security boundary is orders of magnitude.

Three AI Agents: Same Model, Different Instructions

The project has three roles, all calling the same underlying model (OpenAI's Codex). The only difference is the prompt fed to each one.

The Shopper: Simulating a Real User

Three shoppers each have simple goals: budget-student is a student with a ~$20 budget looking for a gift for a developer friend; hoodie-hunter is a busy engineer looking for the cheapest hoodie in size L; sticker-collector is a sticker enthusiast who wants to browse all stickers and compare prices.

Shoppers are explicitly told "you are a real person, not a developer — do not edit any files." They use curl to view pages and imagine the clicks and scrolling a real user would need. Crucially, they must respond in JSON rather than prose, so the program can easily parse the output. The returned report includes a score (1 to 10, where 10 means the task was extremely easy), friction (pain points encountered), and a user-perspective quote.

The Analyst: Distilling Actionable Fixes

After reading all shopper reports, the analyst proposes several fix variants under strict constraints: each fix must be small and focused, only touching index.html, app.js, and styles.css; no new libraries; no backend changes; and each proposal should represent a distinct approach. It outputs each variant's title, hypothesis, and specification in JSON format.

The Builder: Actually Writing the Code

Given a specific fix (e.g., "sort by price"), the builder writes the code under rules including: only edit those three files; use plain HTML/CSS/JS without frameworks; don't restart the server; and verify the page still loads with curl when done. These constraints ensure every variant is fair — the same store plus one small change.

This "same model, different prompts" multi-agent design pattern is commonly called "Role Prompting" or "agent specialization," and is a prevalent paradigm in current multi-agent systems. The core idea is that large language models have generalist capabilities, and carefully designed system prompts can constrain their behavioral boundaries, making the same model play roles as a critic, implementer, reviewer, and so on. The advantage of this approach is avoiding the cost of maintaining multiple specialized models, while using role constraints (like "you are a real person, not a developer" and "do not edit files") to deliberately introduce cognitive bias that prevents agents from overstepping or conflating responsibilities. In this project, the shopper's constraints ensure scoring stays from the user's perspective rather than a developer's — a point that's critical to the validity of the test results.

Core SDK Operations Chained Together

The project uses the SDK quite sparingly — roughly only ten calls in total. Authentication is done via oauth() (on first use it prints a link and verification code; afterwards credentials are stored in a local file) or pat() (set DOCKER_USERNAME and DOCKER_PAT to skip the browser). The OpenAI key is stored in Docker via client.secrets.create, not inside the sandbox — note that the SDK won't automatically attach secrets for you; you must pass them in manually when creating a sandbox.

Creating and launching a sandbox uses client.kits.launchAndWait('codex', ...), where 'codex' is a pre-built kit with Codex installed and launchAndWait blocks until the sandbox is ready. Then sandbox.files.mkdir and files.write copy the store files in, sandbox.processes.start runs the Node server in the background, polling (up to 30 times, once per second) confirms the server is responding, and finally sandbox.ports.create({ number: 3000 }) exposes the port to the public internet, returning a real HTTPS URL.

All Codex task execution goes through a single runAgent function, using codex exec for a one-shot run, --ephemeral mode, and -o to write the final answer to a file. The --dangerously-bypass-approvals-and-sandbox flag disables Codex's built-in safety box — which is acceptable given we're already inside a disposable sandbox. Since AI responses are often wrapped in code blocks or extra sentences, the extractJson helper simply finds content between the first { and last } — simple but effective.

The --dangerously-bypass-approvals-and-sandbox flag deserves a separate explanation. Codex CLI runs in an approval mode by default, pausing to wait for human confirmation every time the agent wants to write a file or run a command, to prevent AI mishaps. This design is reasonable for interactive use, but it blocks an entirely automated pipeline. This project's solution is to "outsource" that safety layer to Docker Sandbox — since the entire execution environment is already a disposable, isolated microVM, Codex's built-in approval mechanism becomes redundant overhead rather than necessary protection. This reflects an important security engineering principle: security mechanisms should be layered, and when an outer layer already provides sufficient protection, an inner layer of the same type can be selectively disabled in exchange for automation efficiency.

The Main Flow and Winner Selection

The main program ab.ts chains the helpers in sequence. It first launches the original store as baseline A, then uses simulate with Promise.all to let three shoppers concurrently experience it and return JSON reports. Next, the analyst generates fix proposals. Then for each proposal it calls runVariant — which opens a new sandbox, has the builder apply the fix, and sends the same three shoppers to test it.

There's a noteworthy engineering detail here: batch execution uses Promise.allSettled rather than Promise.all. The difference is that if one fix fails, the other variants still complete — a single bad version doesn't blow up the entire run.

The winner selection logic is clear: sort by average score descending; break ties by completion rate; break further ties by fewest steps. Once a winner is selected, the program reads its files from the sandbox back to a local results/winner/ directory and generates report.md and report.json recording all scores and user quotes. A single diff -u app/app.js results/winner/app.js is essentially your pull request.

Running, Observing, and Cleanup

After configuring OPENAI_API_KEY in .env, run npm start to kick things off. A typical run (2 fixes, 3 shoppers) uses 3 small sandboxes and roughly 12 Codex calls, taking about 5–10 minutes. Flags like --variants, --personas, --goal, --cleanup, and --no-dashboard offer flexible adjustments.

During the run, visit http://localhost:4000 to see the live dashboard, where the A/B/C boxes light up as progress advances and display scores in real time. The terminal also prints a public URL for each variant, letting you watch B and C change in real time as the builder works — all in three browser tabs side by side. In one sample result, the baseline A scored 4.0 ease-of-use, 67% completion rate, and 14 steps; while the winner B (search and filtering) achieved 8.3 score, 100% completion, and only 4 steps — a 108% improvement. When you're done, npm run cleanup deletes all sandboxes prefixed with ab-.

Limitations and the Bigger Picture

The author is candid about a few limitations: shoppers read pages via curl rather than a real browser, so they can tell if search or filtering is missing but can't evaluate visual aesthetics; scores come from AI judgments by a tiny number of shoppers, making them directional signals rather than hard evidence; and the Sandboxes SDK is still experimental (the project targets version 0.53.0), with some naming potentially changing in the future.

But this store is just an example. The same pattern applies to testing prompts, tuning parameters, trying different copy — any scenario where "you can play the role of a user and give a score" can be plugged into this loop. The real core insight is this: when a safe, disposable computer can be summoned with a single function call, letting agents "give it a try and see what happens" is no longer a risky proposition. That may well be a pragmatic path forward for deploying autonomous agent engineering in the real world.