How to Build an Effective AI Code Review Agent: Prompt and Model Inversion in Practice

AI code review agents are becoming new infrastructure for platform teams. When everyone in the company is vibe coding, review bottlenecks quickly emerge. Based on a real-world sharing session, this article breaks down the complete pipeline from sandbox setup to prompt optimization, model inversion, code context injection, and end-to-end testing, highlighting the most common pitfalls at each stage.

From Sandbox to PR Feedback: A Minimal Viable Review Agent

Building a working review agent isn't complicated. You need a sandbox environment (Firecracker or any commercial sandbox service), put the agent inside it, choose a model-agnostic harness like OpenCode or Py, connect to GitHub or GitLab PR webhooks, let the agent review incoming PR content, and write feedback back to the PR via the GitHub review API.

Once this minimal system is running, you can see feedback on PRs in about five minutes. Many people think this is all there is to a review agent, but real problems will surface one by one during use.

Firecracker is AWS's open-source lightweight virtualization technology that implements microVMs based on KVM, enabling millisecond-level startup while maintaining strong isolation—commonly used in scenarios requiring secure execution of untrusted code. In review agents, the sandbox allows the agent to freely run tests, install dependencies, or even execute code from the PR without contaminating the host machine or accessing sensitive data. If you don't want to build Firecracker yourself, you can also choose managed sandbox services like E2B or Modal, which provide similar isolated execution environments and APIs.

A so-called model-agnostic harness refers to a runtime framework that decouples the "reasoning model" from "tool calling, file I/O, and terminal execution," such as OpenCode or Py. It doesn't bind to a specific model vendor, making it easy to perform model inversion and cost tiering later by simply swapping the underlying model.

Four Inevitable Pitfalls: Missed Bugs, Missing Context, High Cost, and Too Verbose

The first pitfall is missed bugs. Bugs in code are nearly fractal and infinite—no system can exhaust them no matter how hard it tries. What's more troublesome is that a review agent can only tell you a bug exists, not that it doesn't exist—you never know the counterfactual.

The second pitfall is missing context. The agent doesn't understand the direction of the codebase or how a particular library works. If the company has gone through microservice decomposition and has a large number of repositories, the lack of context becomes even more severe.

The third pitfall is cost. Using a coding agent to review a single PR can cost between $3 and $25, with token consumption accumulating far faster than expected.

The fourth pitfall is nitpicky comments. Technically correct but overly pedantic comments cause double harm: human readers get drowned in noise and struggle to distinguish which issues are catastrophic; and coding agents indiscriminately address every comment one by one, ultimately making the codebase overly defensive and unrecognizable.

Optimizing Prompts: Target Specific Bug Types, Explicitly Skip Nitpicks

The lowest-hanging fruit is prompt optimization. The initial prompt is often "review this PR and output comments according to the GitHub API schema," which is far from sufficient. The prompt should be restructured around two themes: first, require the agent to look for specific bug types; second, explicitly tell it what constitutes a nitpicky comment and not to look for those.

Where do specific bug types come from? Analyze internal historical failures and incidents to identify the bug types the codebase is most prone to, build a catalog, and then have the prompt target each one. Nitpicky comments also need explicit definition—comments like "why isn't there three layers of defense here" or "error handling isn't rigorous enough" should all be excluded.

There's a hidden risk in prompt fine-tuning: a small change can cause the number of comments to fluctuate dramatically. This means you can't tell whether it's missing bugs. Since catastrophic bugs are extremely rare—according to their data team's statistics, roughly one catastrophic bug per thousand lines of code—you might release a version that looks good but quietly misses critical issues, only to discover it much later.

Calibrate with Historical PR Datasets, But Don't Expect a One-Time Fix

The second improvement is building a historical PR dataset. If the company has thorough incident records, you can compile a list of PRs that previously caused incidents, run each version of the review agent against them, and observe capture frequency. This gives more confidence in prompt tuning.

But bug topology and ontology change over time. About 80% to 90% of bugs are repetitive errors, but as coding agents get stronger, they start making unprecedented new mistakes. Unless you have millions of PRs in your dataset, it's hard to maintain high coverage of bug types—this is the ceiling of the dataset approach.

Model Inversion: Review Code with a Different Model

The third improvement is model inversion. Research found that Claude models and GPT models produce significantly different bug types: Claude models have about three times the probability of one-off errors compared to GPT models, while GPT models have a noticeably higher probability of authentication bypass than Claude models. This is considered a difference introduced by post-training.

If you use the same model to review code it wrote itself, it will catch fewer bugs because it fails in the same way during review. So use Claude to write code and GPT to review it, and vice versa.

Identifying which model generated the code is easy for 80% of PRs: Codex names branches with a codex/ prefix, and Claude writes co-authored-by in PR descriptions and commit messages. The remaining 20% require heuristic judgment, such as internal registries or developer machine telemetry. Empirical testing shows that model inversion yields higher capture rates—a significant improvement when measured in terms of incidents and failures, enough to avoid catastrophic outcomes.

Post-training refers to the stages a model goes through after pre-training, such as instruction fine-tuning, RLHF, or Constitutional AI. Different vendors' post-training data and preference objectives shape different "failure modes": for example, one type of model is reinforced to handle authentication and permissions more cautiously, while another is more inclined to quickly complete features. This difference causes the same code to produce different bug type distributions under different models, and is the fundamental reason why model inversion improves capture rates.

When heuristically determining code provenance, you can combine branch naming prefixes, co-authored-by markers in commit messages, internal developer machine telemetry, or code style features. Note that these signals may be forged or missing, so they should be treated as probabilistic inputs rather than absolute evidence.

Code Context: GREP Is Not a Panacea

Code context is the third cornerstone of a review agent. There's a popular belief that GREP has solved the code context problem, but that's not true: GREP is extremely imprecise and expensive. In coding agent call traces, a large amount of tokens are spent on GREP, file reads, and cache writes of file contents.

A more refined approach is to build an internal knowledge base: scan the entire codebase, catalog key flows like authentication, and record how they work, their flow diagrams, and all files involved. Take authentication bypass as an example—determining whether a bypass has occurred requires knowing exactly whether the current code is in an authenticated context, which is impossible without structured code context.

The limitation of GREP in code retrieval is that it's based on literal matching and cannot understand semantic equivalence or cross-file call chains. When the agent needs to determine "whether the current request passes through authentication middleware," simply searching for function names may miss dynamic registration, decorator wrapping, or configuration-driven routing. Additionally, each GREP writes large amounts of file content into the context cache, causing token costs to rise linearly with repository size.

Building an internal knowledge base typically involves: statically scanning entry points and middleware, generating call graphs, annotating trust boundaries of key flows, and regularly updating as code changes. It's essentially giving the agent a structured map so that when reviewing security issues like authentication bypass, it can directly locate relevant files rather than searching blindly.

Code Without Bugs Can Still Be Bad: Intent and End-to-End Testing

Even if code has no bugs, it can be of poor quality or fail to implement the author's expressed intent. When people treat Slack as an IDE and ask an agent to change code with a sentence or two, the instructions are highly ambiguous, and the agent may misinterpret intent and write correct but completely wrong code.

The way to counter this is to give the agent more context. The two most important types of context are documentation and tickets (Jira, Linear) and codebase indexes. Much tribal knowledge subconsciously helps humans write better code, and without it, agents appear clueless.

Computer use capabilities have improved dramatically, performing far better in the new generation of GPT and Opus models than previous generations, and end-to-end QA is approaching being solved for the first time. What you need to do is document all application flows and prepare sufficiently simple build scripts so the agent can spin up the entire environment. Then the agent can test code end-to-end, using visual models to catch issues like "this button technically exists but is hidden on iPhone."

Cost Tiering: Reserve Frontier Models for Critical Tasks

After all the above optimizations, costs are still not low—a single PR review can cost $3 to $15 or $20. The solution is task tiering: frontier models are still stronger than open-source models, but the gap is narrowing. A reference ratio is about 20% frontier models for the most critical tasks, with long-tail tokens (exploration, file reads, etc.) delegated to various smaller models.

A typical division of labor: small models find context, lightweight computer-use models test user flows, and frontier models analyze bugs, execute review processes, and submit. In practice, you can break this into about 20 subtasks, each with a specific model combination. Even simple task substitution can achieve 70-80% of the effect.

But this also depends on datasets. Without a good dataset, you might swap a sub-agent for a smaller, faster model without knowing its bug-catching ability has significantly degraded—only to realize it after an incident. Another trick is to use cheap models to rerank context and decide what should be loaded into the frontier model's cache—the most expensive and scarce resource.

Context reranking refers to using cheap small models to score and rank candidate code snippets, files, or historical comments, sending only the most relevant few into the frontier model's cache. Since frontier model input tokens are far more expensive than small models, this step can significantly reduce per-review costs while preventing irrelevant content from diluting model attention. In practice, you can combine embedding retrieval with lightweight classifiers to dynamically adjust the content budget sent to frontier models based on task type.

No Silver Bullet, Only Continuous Calibration

Building an effective AI code review agent is not a one-time engineering effort but a continuously calibrated system: prompts target specific bug types, datasets cover historical and novel failures, model inversion breaks same-source blind spots, code knowledge bases provide precise context, end-to-end testing validates intent, and finally cost tiering controls expenses. Every step depends on honest observation of your own codebase and failure history, not generic templates.

One-line Summary

AI code review agents require continuous calibration of prompts, datasets, model inversion, and cost tiering.

Paragraph Summary

Based on a real-world sharing session, this article breaks down the complete pipeline of AI code review agents from minimal viable system to continuous calibration. The minimal system only needs a sandbox, a model-agnostic harness, PR webhooks, and the GitHub review API, but it exposes four major problems: missed bugs, missing context, high cost, and nitpicky comments. Improvement paths include: restructuring prompts around specific bug types and explicitly excluding nitpicky comments; calibrating with historical PR datasets while acknowledging their coverage ceiling; leveraging bug type differences between Claude and GPT for model inversion; building an internal knowledge base cataloging key flows to replace imprecise GREP; validating code intent through end-to-end testing; and finally tiering by task, using frontier models for only about 20% of critical tasks. The core conclusion is that there is no silver bullet—every step depends on honest observation of your own codebase and failure history.