ProgramGym: A New Long-Horizon Training Framework for Building Programs from Scratch with AI

From "Fixing Code" to "Building Programs": A New Challenge for Coding Agents

Most coding agents today excel at fixing bugs in existing codebases or generating code from detailed requirement documents. But the ability to truly "build a complete program from scratch" — the way a software engineer would — remains a stubborn challenge.

Two core obstacles lie at the heart of this problem. First, there's a lack of scalable, verifiable training environments that force agents to discover what needs to be done and autonomously decide how the system should be designed. Second, existing agent workflows struggle to support long-horizon development — the process of thoroughly exploring a reference program before diving into implementation.

Addressing both pain points, researchers have proposed the ProgramGym framework. It consists of two core components: Program-Env, which builds training environments, and Program-Agent, which executes development tasks.

Program-Env: Automatically Building Verifiable Training Environments

The core value of Program-Env lies in its ability to automatically construct verifiable training environments from real-world code repositories. "Verifiable" means that programs generated by an agent can be objectively checked for correctness — whether they run as expected and satisfy the intended behavior.

It employs a feedback-driven loop mechanism designed to tackle two engineering challenges: fixing the construction of runtime sandboxes (to ensure programs can actually run in an isolated environment), and improving the quality of black-box tests (so the tests are rigorous enough to reliably distinguish correct from incorrect implementations).

Using this novel pipeline, the research team ultimately produced 1,648 training environments spanning four mainstream system-level programming languages: C, C++, Go, and Rust. The language coverage is itself noteworthy — compared to many programming benchmarks that focus on Python, ProgramGym's support for compiled, engineering-heavy languages is far more representative of real-world systems software development.

Black-box testing is a fundamental software testing paradigm: testers focus solely on a program's input-output behavior, with no knowledge of its internal implementation. This property is especially important for coding agents — any given problem can have countless correct implementations, and black-box tests only require that "the result is correct," not that "the code looks a certain way." This makes them naturally suited for evaluating the functional equivalence of generated code. However, constructing high-quality black-box tests is non-trivial: insufficient test coverage lets incorrect implementations slip through, while biased test design can mistakenly reject correct ones. ProgramGym's feedback-driven loop continuously improves test quality, directly addressing this systemic challenge.

Program-Agent: A Role-Separated Long-Horizon Development Workflow

If Program-Env answers the question of where to train, then Program-Agent answers how to develop effectively.

Its design centers on a role-separated workflow that breaks the development process into three distinct phases: analysis, implementation, and review. This division of labor keeps the agent focused on different objectives at each stage, preventing the quality degradation that comes from mixing concerns in a single monolithic process.

Built on top of this, Program-Agent introduces a revision management mechanism specifically designed to handle compilation checks and failure recovery. When code fails to compile or tests fail, the agent doesn't simply give up — it systematically diagnoses the problem and attempts recovery. This is precisely the most stability-critical aspect of long-horizon development.

Long-horizon development refers to tasks that require multiple interdependent decision steps to complete, in contrast to "single-step completion" or "local patching." In long-horizon settings, early decisions — such as architectural choices and data structure design — profoundly affect every subsequent step. A wrong early direction is often impossible to correct locally and requires a full redesign. This places much higher demands on an agent's planning and state management capabilities. The value of a role-separated workflow lies here: by explicitly separating "understanding requirements," "hands-on implementation," and "review and verification" into distinct stages, it reduces decision quality degradation caused by information overload in a single context window, while making each stage's output a structured, reliable input for the next.

Experimental Results: Gains from Both Fine-Tuning and Workflow

ProgramGym's effectiveness was validated on the ProgramBench benchmark, with compelling numbers.

The value of training environments: Fine-tuning on trajectories generated from Program-Env environments boosted the Qwen3.8-27B model's performance from 34.87% to 42.84% — a gain of nearly 8 percentage points that directly demonstrates the practical value of these training environments.

Additive gains from the workflow: Program-Agent further pushed both the base model and the supervised fine-tuned (SFT) model to 55.71% and 56.25%, respectively. Even more noteworthy is that both scores surpassed certain flagship-level models — showing that a well-designed workflow can enable mid-sized open-source models to punch above their weight on specific tasks.

This result echoes a growing trend in the field: parameter scale alone is not the only determining factor. The quality of training environments and the orchestration of inference-time workflows are equally critical levers for pushing capability ceilings.

Supervised Fine-Tuning (SFT) refers to further training a pretrained model on high-quality "input-output" examples — either human-annotated or automatically generated — to align its behavior toward a desired direction. In agentic settings, SFT training data typically takes the form of "trajectories" — complete action sequences an agent follows to complete a task, including tool calls, intermediate reasoning, and final outputs. Trajectory quality directly determines fine-tuning effectiveness: low-quality trajectories reinforce incorrect behavior patterns, while high-quality trajectories from real, verifiable environments teach the model more reliable development strategies. The 1,648 verifiable environments provided by Program-Env in ProgramGym serve as precisely this infrastructure for generating high-quality training trajectories at scale.

Open-Source Plans and Industry Significance

The research team has announced plans to open-source ProgramGym, supporting further research on coding agents in the "build programs from scratch" direction.

For the field as a whole, ProgramGym's significance goes beyond a specific score improvement. It offers a scalable path for generating verifiable training environments — the most scarce infrastructure in current agent training. When training data can be automatically constructed from real repositories and test quality can be continuously improved through feedback loops, the barrier to entry for long-horizon program reproduction research drops significantly.

For developers and researchers tracking the evolution of AI programming capabilities, this is a direction worth following closely — especially in the contexts of system-level languages and long-horizon development, which more closely mirror real-world software engineering.