WildAWM: Synthesizing Computer Agent Training Environments with Open Models

Title

WildAWM: Synthesizing Computer Agent Training Environments with Open Models

Meta Description

Microsoft Research proposes WildAWM, using open models to synthesize computer agent training environments supporting GUI, MCP, and CLI, producing 341 long-horizon tasks via execution-based quality gates, with Qwen models showing significant gains on 8 OOD benchmarks.

Article Content

Training Computer-Use Agents (CUAs) is shifting from imitating human data toward interacting with real environments, but scaling this kind of learning directly on real computer systems carries enormous safety risks. WildAWM (Agent World Model in the Wild), proposed by Microsoft Research, offers a scalable synthesis path: using open models to automatically generate controllable environments and tasks, letting agents safely "trial and error" in virtual worlds.

From Real Systems to Synthetic Worlds: Why WildAWM Is Needed

Letting agents learn in real operating systems, browsers, and terminals means they might perform irreversible operations such as deleting files, sending emails, or calling paid APIs. Opening up permissions directly is both dangerous and hard to scale—every exploration requires isolation, rollback, and monitoring. The core idea of WildAWM is to make the environment itself a synthesizable object: rather than relying on manually built fixed testbeds, it uses models to automatically generate a batch of executable, verifiable, and difficulty-controlled computer environments.

The paper emphasizes that this pipeline is entirely driven by open models and does not depend on proprietary frontier models. This sends an important signal: high-quality CUA training environments need not be the exclusive resource of a few labs. The code is open-sourced on GitHub (MSR-Orchard/WildAWM), and the community can reproduce and extend it.

Three Interfaces Combined: One Environment, Three Interaction Modes

Existing environment synthesis work usually targets only a single interaction interface—either a graphical interface, tool calling, or the command line. What sets WildAWM apart is that the executable environments it builds simultaneously support a graphical user interface (GUI), Model Context Protocol (MCP) services, and a command-line interface (CLI). Agents can combine visual interaction, tool calling, and code execution within the same task—for example, first locating a button through the GUI, then calling an MCP service to query data, and finally using a CLI script to batch-process the results.

This "three-in-one" design is closer to the complexity of real computer use. Traditional benchmarks often split agent capabilities into isolated tracks, whereas WildAWM's tasks require agents to switch between multiple modalities, placing higher demands on generalization.

Model Context Protocol (MCP) is an open protocol proposed by Anthropic that has gradually become an industry standard, used to let AI models call external tools, data sources, and services in a structured way. Compared with traditional function calling, MCP defines a more complete client-server architecture: MCP servers expose resources, prompt templates, and tools, and agents as clients discover and invoke them through a unified interface. In computer-use scenarios, MCP enables agents to access file systems, databases, or third-party APIs in an auditable and reusable manner, without having to write a separate adapter layer for each tool. WildAWM adopts MCP as one of its three interaction interfaces precisely because of its ability to standardize tool calling, allowing tasks in synthetic environments to simulate human operations through the GUI as well as move toward programmatic, structured tool use through MCP—thereby covering two fundamentally different interaction paradigms in real computer use.

Three Quality Gates for Task Synthesis

Synthetic environments are only the first step; the tasks themselves must satisfy three conditions: solvable, sufficiently difficult, and reliably verifiable. To this end, WildAWM introduces execution-based quality gates, comprising three mechanisms:

• Heterogeneous agent rollout: use different agents to actually execute tasks, filtering out unsolvable or overly simple samples;
• Cross-judge consistency check: multiple verifiers independently judge execution results, and only conclusions that agree are retained;
• Counterfactual probes: perturb task conditions or execution trajectories to test whether verifiers can be fooled by "fake successes."

These three gates target recurring pain points in task synthesis. Solvability is confirmed through real execution, difficulty is filtered through the distribution of agent performance, and verification reliability is calibrated through adversarial probing. Compared with relying solely on model self-evaluation or manual spot checks, execution-driven gating is closer to the signals that actually matter during training.

Counterfactual probes originate from the ideas of causal inference and robustness testing: while keeping the surface form of a task unchanged, apply controlled perturbations to conditions or execution trajectories and observe whether the verifier's judgment changes correctly in response. For example, compare a trajectory that truly changes the system state with a "fake success" trajectory that merely produces the same screen output but does not change the underlying state; a reliable verifier should reject the latter. If a verifier is insensitive to perturbations, it may be relying on surface features rather than real execution results. Such probes can expose shortcut learning and reward hacking risks in verifiers and are a key means of calibrating verification reliability within execution-based gating, complementing heterogeneous agent rollout (testing solvability) and cross-judge consistency (testing judgment stability).

341 Long-Horizon Tasks: Even Frontier Models Pass Only 60%

After filtering, WildAWM produced 341 environment-task pairs. These tasks are of the long-horizon type, with some requiring more than 100 agent steps to complete. The difficulty baseline is quite direct: even the frontier model GPT-6-Astra can solve only 60% of them. This means the dataset still poses a substantive challenge to the strongest current agent systems, rather than being yet another evaluation set approaching saturation.

Interestingly, the difficulty of long-horizon tasks comes not only from the complexity of individual steps but also from state dependencies and error accumulation across multiple steps. When agents switch between GUI, MCP, and CLI, any deviation in one step can invalidate subsequent steps, testing both planning and error-correction capabilities.

Long-horizon tasks refer to tasks that require many decision steps and have strong state dependencies between steps, typically requiring dozens to hundreds of consecutive actions. Unlike single-step question answering or short-horizon tool calling, the core difficulty of long-horizon tasks lies in error accumulation and credit assignment: a subtle deviation in an early step is amplified along the execution chain and ultimately leads to overall failure, while it is hard for the agent to infer from the final result which step went wrong. In addition, long-horizon tasks require agents to have cross-stage planning, memory, and self-correction capabilities. Some tasks in WildAWM exceed 100 agent steps and require switching between GUI, MCP, and CLI, precisely to approximate the long-horizon, multimodal complexity of real computer use—which is also an important reason why the frontier model's pass rate is only 60%.

Train in Synthetic Worlds, Validate on Real Benchmarks

The research team trained two Qwen models of different sizes using only these synthetic worlds, then evaluated them on 8 out-of-distribution (OOD) benchmarks covering web agents, claw-type tasks, and computer-use agents. Both models achieved significant improvements, showing that capabilities acquired in WildAWM synthetic environments can generalize to unseen computer-use tasks.

This result addresses a key concern: will synthetic environments cause agents to merely memorize patterns of specific interfaces? Judging from OOD performance, transfer across interfaces and task types holds. Since the training data comes entirely from the synthesis pipeline, the generalization gains are more likely to stem from the transferability of task structure itself rather than implicit fitting to real data.

Out-of-distribution (OOD) evaluation refers to measuring model performance on test sets outside the training data distribution, used to test generalization ability rather than memorization. In agent learning, OOD benchmarks usually include websites, applications, task types, or interaction interfaces that did not appear during training. If a model trained only in synthetic environments improves on multiple heterogeneous OOD benchmarks, it indicates that what it has learned is transferable task structure and interaction strategies rather than pattern fitting to a specific environment. It should be noted that OOD improvement does not automatically equate to robustness in real scenarios—the distribution gap between synthetic environments and real systems still needs to be quantified, which is one of the reasons the authors list it as an open problem.

The Significance and Boundaries of the Synthetic Environment Approach

The value of WildAWM lies in turning "environments" from a scarce resource into training material that can be mass-produced. The open-model-driven pipeline lowers the barrier, execution gating ensures quality, and the three-interface joint design stays close to real usage scenarios. However, the paper also leaves some open questions: how to quantify the distribution gap between synthetic environments and real systems, how to maintain verifier reliability on more complex tasks, and whether 341 tasks are sufficient to support larger-scale training. These questions will determine how far the synthetic environment approach can go.

For teams focused on deploying AI agents, WildAWM offers a reusable idea: rather than cautiously delegating authority in real systems, first synthesize a world that is realistic enough, difficult enough, and verifiable, and let agents make all the mistakes they should make inside it.

One-line Summary

Microsoft's WildAWM uses open models to synthesize three-interface executable environments, safely training computer agents and validating generalization.

Paragraph Summary

Microsoft Research proposes WildAWM, turning the training environment for computer agents itself into a synthesizable object to avoid the safety risks and scaling challenges of trial and error in real systems. The pipeline is entirely driven by open models, generating executable environments that simultaneously support GUI, MCP, and CLI interaction modes, and uses three execution-based quality gates—heterogeneous agent rollout, cross-judge consistency checks, and counterfactual probes—to ensure tasks are solvable, sufficiently difficult, and reliably verifiable. It ultimately produces 341 long-horizon environment-task pairs, some requiring more than 100 steps, with the frontier model GPT-6-Astra solving only 60%. Two Qwen models trained only in synthetic worlds achieve significant improvements on 8 out-of-distribution benchmarks, indicating that capabilities learned in synthetic environments can transfer to unseen computer-use tasks.