[KongchangAI]
· 1 min read· 898 words

ADAPT: Using AI Agents to Generate Diff-Aware Tests for PRs

ADAPT: Using AI Agents to Generate Diff-Aware Tests for PRs

ADAPT uses autonomous agents to analyze PR diffs and generate change-centric tests, achieving an F2P rate of up to 89.9%.

ADAPT is a regression test generation method for pull requests, aiming to fill the gap of lacking focused tests for PR changes. It uses an autonomous coding agent to analyze the diff between base and head versions, identify modified functions and their context, plan and generate candidate unit tests, then validate through fail-to-pass (F2P) checking to ensure tests truly capture changed behavior, and iteratively refine based on feedback about uncovered executable changed lines. On 426 real PRs, ADAPT+GPT-4o achieved a 62.0% F2P rate, far exceeding BLAST's 35.4%; with GPT-5-mini it reached 89.9%. On the 137 PRs shared with ChaCo, ADAPT achieved complete patch coverage on 83.9% of PRs with significantly lower time and cost. Human evaluation shows its tests average 4.54/5, 4.35/5, and 4.49/5 on importance, assertion strength, and readability, often covering behaviors beyond developers' handwritten tests.

Introduction: The PR Testing Gap and ADAPT's Solution

In modern software development, pull requests (PRs) have become the core unit of collaboration. A PR typically introduces only a small set of targeted code changes, yet these changes often lack focused tests to guard them after merging. Developers frequently submit PRs without accompanying tests, causing new behaviors to lose protection in future iterations. Most existing test generation techniques target entire software units, optimizing for code coverage rather than the behavior introduced by the change. Some earlier diff-aware approaches have shown limited effectiveness in capturing changed behavior. ADAPT (Agentic Diff-Aware Patch Testing) is precisely a regression test generation method proposed to fill this gap, leveraging autonomous coding agents to generate change-centric tests for PRs.

ADAPT paper source screenshot

How ADAPT Works: From Diff Analysis to F2P Validation

ADAPT's core workflow revolves around an autonomous coding agent. Given a PR, the agent first analyzes the diff between the base version and the head version, identifies modified functions and their context, then plans and generates candidate unit tests.

Diff-Aware Test Planning

Unlike traditional unit test generation, ADAPT does not start from the entire codebase but focuses on the change points revealed by the diff. The agent parses the modified functions, their call chains, and related context, and formulates a test plan accordingly. This change-centric strategy makes the generated tests more likely to directly target the new behavior introduced by the PR.

Fail-to-Pass Validation and Iterative Refinement

ADAPT is equipped with a dedicated fail-to-pass (F2P) validation tool. Each candidate test is run on both the base and head versions: a valid F2P test should fail on the base and pass on the head, thereby proving that it actually captures the behavior introduced by this change. After F2P validation succeeds, the tool also reports which executable changed lines remain uncovered. The agent iteratively refines the tests based on this feedback, progressively closing coverage gaps.

F2P (Fail-to-Pass) tests are a standard pattern in regression testing: a test case should fail on the pre-modification code (base version) and pass on the post-modification code (head version). This property directly proves that the behavior verified by the test is newly introduced or fixed by this change. In automated test generation, F2P validation serves as a strong constraint—it filters out "lazy" tests that behave identically on base and head, ensuring that generated tests are truly related to the change rather than merely covering some code.

Patch coverage is a metric relative to overall code coverage: it only counts the proportion of executable code lines modified or added by this diff that are actually executed by tests. Unlike pursuing coverage of the entire codebase, patch coverage focuses on "whether this change is guarded by tests," making it more aligned with PR scenarios. Even if a PR has high overall coverage, if the newly added code lines are completely uncovered, regression risk still exists; patch coverage is precisely used to quantify this gap.

Experimental Results: Significant Improvements in F2P Rate and Coverage

ADAPT was evaluated on the same set of 426 real-world PRs used by the previous state-of-the-art baseline, paired with two models: GPT-4o and GPT-5-mini.

F2P Rate Comparison

ADAPT+GPT-4o achieved a 62.0% F2P rate (264/426), while BLAST reported a 35.4% F2P rate (151/426). ADAPT+GPT-5-mini reached 89.9% (383/426). This result demonstrates that the diff-aware agent approach delivers an order-of-magnitude improvement over traditional methods in generating tests that are truly related to changes.

Coverage and Cost Comparison with ChaCo

On the 137 PRs shared with Change And Cover (ChaCo), ADAPT achieved complete patch coverage on 83.9% of PRs, compared to 79.6% for ChaCo. More importantly, ADAPT is significantly lower in time and cost. This shows that ADAPT is not only on par in effectiveness but also has an advantage in efficiency.

BLAST is a representative baseline method for previous diff-aware test generation. It also attempts to generate tests for code changes but has limited effectiveness in capturing changed behavior—its F2P rate is only about 35%. ChaCo (Change And Cover) is another work on test generation for code changes, with its evaluation focused on patch coverage, so ADAPT directly compares patch coverage and cost on the subset of PRs shared with ChaCo. These two baselines represent two routes: "diff-aware but weak F2P effectiveness" and "covering changed lines but higher cost," while ADAPT attempts to achieve advantages simultaneously across three dimensions: F2P rate, patch coverage, and cost.

Human Evaluation: Test Quality Recognized

Human evaluation confirmed that tests passing F2P validation indeed target the intended changes of each PR. On the three dimensions of importance, assertion strength, and readability, average scores reached 4.54/5, 4.35/5, and 4.49/5, respectively. These tests often cover behaviors beyond what developers write by hand, demonstrating potential beyond manual testing.

Significance and Outlook: A New Paradigm for Change-Aware Testing

The emergence of ADAPT marks a paradigm shift in test generation from "covering entire units" to "covering changed behavior." By combining autonomous agents, diff analysis, and F2P validation, it can generate tests that truly guard the new behavior of PRs, rather than merely pursuing coverage numbers. For development teams, this means new behaviors have more reliable protection after PR merges, and regression risk is significantly reduced. In the future, such change-aware test generation methods are expected to be integrated into CI pipelines and become a standard part of the PR process.

ADAPT architecture diagram

#ADAPT#diff-aware testing#PR regression testing#autonomous agent#F2P validation#test generation
Share:

Related articles