Building a Competitive AI Coding Agent: Engineering Practices from One-Shot Generation to Iterative Feedback

Why One-Shot AI Coding Isn't Enough

Asking a large language model to generate code directly often yields answers that "look reasonable" rather than answers that actually pass tests. When building an autonomous coding agent for a competitive environment, the original author explicitly abandoned the one-shot generation approach and instead assembled a complete engineering workflow: repository analysis, planning, implementation, execution, testing, diagnosis, iterative refinement—then learning from test results and resubmitting.

Task → Analyze → Plan → Implement → Execute → Test → Diagnose → Refine → Retest → Submit

The core value of this pipeline is treating execution results as evidence, rather than treating the model's first output as the final answer. For an agent that needs to solve difficult programming tasks, reliably passing tests matters far more than producing "beautiful code snippets."

"One-shot generation" refers to handing a programming task to a large language model as a single prompt and directly accepting the first version of the code it outputs—without running it, without validating it. The reason this mode falls short in real-world scenarios is rooted in the training objective of large language models: they fundamentally "predict the probability distribution of the next token," optimizing for linguistic plausibility rather than program semantic correctness. As a result, models can easily generate syntactically correct, neatly structured, even highly professional-looking code—yet silently make mistakes in boundary conditions, state management, concurrency, or specific API semantics. More critically, models lack reliable self-awareness of "am I actually right"—their high confidence does not equate to high accuracy. Competitive programming enforces extremely strict correctness, recognizing only whether test cases pass or not, so one-shot generation almost inevitably fails in this environment. This is precisely why execution results—not model outputs—must be treated as the source of truth.

The Feedback Loop Is the Key to Performance Gains

What truly makes an agent stronger is the diagnosis and correction stage after a test failure. When a test case fails, the agent proactively analyzes the error message, infers the likely root cause, modifies the code, and then re-runs the relevant tests.

The biggest benefit of this mechanism isn't "writing more"—it's "making fewer mistakes." It catches logic errors and edge cases before submission, preventing the agent from stopping as soon as it finds the first seemingly viable solution. Many programming tasks fail not because the model can't write code, but because the model doesn't carefully review the code it wrote. Iterative feedback fills exactly this gap.

From an engineering perspective, this design upgrades the agent from a "code generator" to a "problem solver with a verification loop." The verification signal comes from the real execution environment, not the model's own confidence, giving the entire system a qualitative leap in reliability.

"Feedback loop" here refers to the classic perception–action–observation loop in autonomous agents: the agent generates code (action), executes it in a real runtime environment and obtains test results or error messages (observation), then adjusts its next action accordingly. The key difference from traditional "zero-shot" or "few-shot" prompting is that the feedback signal comes from the external environment rather than the model itself. This external signal is called a "verifiable reward"—in programming tasks, whether tests pass is an objective criterion that the model can hardly "fake." Compared to approaches relying on model self-critique, loops based on execution feedback are far less prone to "confidently making mistakes." The industry often analogizes this pattern to trial-and-error in reinforcement learning, except here no model parameters need to be trained—significant gains can be obtained through iterative invocation during inference alone. This is also the core paradigm widely adopted by current coding agents (such as SWE-agent, OpenHands, etc.).

How to Measure Whether Optimization Is Actually Effective

The original author used the same set of benchmark tasks and execution constraints, comparing scenarios with iterative feedback enabled versus disabled. Evaluation dimensions include:

• Task success rate
• Number of tests passed per task
• Average completion time
• Compute cost per successful task

The combination of these metrics answers a key question: is a given optimization actually improving correctness and reliability, or merely introducing more latency and compute overhead? Looking only at success rate easily overlooks cost; looking only at speed may encourage cutting corners. Only by observing multiple metrics together can you judge whether an optimization is worth keeping.

The comparison data in the original article is presented as placeholders (baseline → optimized), indicating the author wants to emphasize the evaluation framework itself rather than any specific set of numbers. For teams reproducing such agents, establishing a reliable measurement system is more important than blindly tuning parameters.

When evaluating coding agents, several easily confused concepts need to be distinguished. "Task success rate" (pass@1 or overall solve rate) measures the end-to-end task-level pass ratio; "number of tests passed" is more granular, reflecting whether the agent is "close to correct"—some tasks aren't fully solved but pass most test cases, indicating the direction is right but convergence is insufficient. "Average completion time" and "compute cost per successful task" correspond to the cost dimension: the former concerns wall-clock latency, while the latter is typically measured in token consumption or dollar amounts. It's especially important to divide total cost by the number of successful tasks, not total tasks—otherwise wasted spending on failed tasks gets masked. Industry benchmarks like SWE-bench also adopt similar multi-dimensional reporting, because looking only at accuracy encourages the "brute-force correctness with massive compute" route, which is unsustainable in real deployment. Establishing a unified evaluation standard is the prerequisite for judging whether an optimization lever is worth keeping.

Beyond the Feedback Loop: What Other Optimization Levers Exist

Iterative feedback isn't the only optimization point. The original author also mentions several tunable directions: model selection, prompt design, repository context, tool invocation, reasoning parameters, and failure recovery strategies.

These directions involve trade-offs. Stronger models are usually more expensive and slower; longer repository context improves comprehension but may dilute key information; more aggressive tool invocation speeds up exploration but increases error probability. Failure recovery strategies determine the agent's robustness when encountering environment anomalies, missing dependencies, or timeouts.

These optimizations collectively serve one goal: enabling the agent to systematically use execution results to improve its solutions, rather than remaining at the "one-shot code generation" level. For developers wanting to build similar systems, this approach can be understood as a layered optimization framework—the feedback loop handles error correction, while the other levers raise the quality ceiling of each iteration.

The various optimization levers mentioned in the article each involve trade-offs in engineering practice. "Reasoning parameters" typically include temperature, top-p, maximum generation length, etc. Lower temperatures favor code stability, but excessively high values introduce non-reproducible errors. "Repository context" refers to injecting relevant files, function signatures, dependency relationships, etc. from the codebase as part of the prompt. Its quality often matters more than quantity—retrieval-augmented (RAG) selection of relevant snippets typically outperforms simply stuffing the entire repository in. "Tool invocation" encompasses file read/write, shell execution, code retrieval, unit test running, and other capabilities, directly determining the environmental boundaries the agent can reach. "Failure recovery strategies" handle failures caused by non-model reasons such as timeouts, missing dependencies, and environment crashes—the guarantee of system robustness. Understanding these levers as "amplifiers beyond the feedback loop" is more apt—they don't directly correct errors, but they determine the quality ceiling each iteration can reach, and whether the agent can operate stably long-term under real engineering constraints.

Implications for AI Coding Tool Developers

This practice conveys a clear signal: the competitiveness of a competitive-grade coding agent comes from the engineering closed loop, not from relying solely on model capabilities. Repository analysis determines depth of understanding, planning and implementation determine first-round quality, testing and diagnosis determine whether the agent can converge to a correct solution, and cost control determines whether it can run sustainably under real constraints.

If you're building AI coding-related products, consider asking yourself three questions: Does the agent actually execute and verify its own output? Is there a diagnosis and retry mechanism after failure? Are optimization effects measured jointly by success rate, number of tests passed, and compute cost? Solving these three questions well will bring more practical value than switching to a stronger base model.