[KongchangAI]
· 3 min read· 1,737 words

How the Grok Team Hits 11,000 PRs a Month: Replacing Manual Review with Deterministic Constraints

How the Grok Team Hits 11,000 PRs a Month: Replacing Manual Review with Deterministic Constraints

Grok's team uses deterministic constraints and automated verification to hit 11,000 PRs a month with a 0.5% rollback rate.

Based on a talk by xAI engineer Potato, this article explains how the Grok team achieves 11,000 production PRs a month with only a 0.5% rollback rate through deterministic constraints and automated verification. It prioritizes code-level checks—architecture, Lint rules, and tests—over prompts, keeps the codebase pure by banning useEffect, cross-process imports, and 99% of comments, and turns verification into the new review so most PRs can auto-merge.

Is AI-generated code all garbage? The Grok team has data that says otherwise. An engineer at xAI working on Grok (known online as Potato) revealed in a technical talk that he personally submitted 3,000 PRs to production last month, while his team submitted 11,000 in a single month—with a PR rollback rate holding steady at around 0.5%. What supports this number isn't a stronger model, but an engineering system built around "deterministic constraints + automated verification."

Thinking about how to turn it into something deterministic

From Low Trust to High Trust: The Four-Layer Pyramid of the Agent Environment

Potato breaks a team's trust in agents into four layers. The bottom two layers are fully deterministic: solid API design and architecture, refactoring, plus Lint rules, type systems, compilers, and tests. What these tools share is that they don't rely on the agent's "understanding"—they use the code itself to force the agent to default to correct behavior.

Above that come runtime verification and formal verification, responsible for finding bugs, checking properties that should always hold in the system, and confirming whether the agent truly understood the goal. At the very top are soft constraints that require judgment: prompts, rules, skills, AGENTS.md, and the like. Potato's advice is direct: every time you correct an agent, first ask whether you can turn it into a deterministic check written into the codebase, rather than adding another line to a rules file—code is far more reproducible.

The Codebase Is the Agent's Biggest Input

One metaphor runs through the entire talk: the codebase is a garden. In the past, an engineer might handle only a few PRs at a time; now code pours in like a continuous stream, and each engineer is effectively leading a team of hundreds or even thousands of agents.

This means any workaround, anti-pattern, or tech debt in the codebase is likely to be copied by the next agent—because it looks like an "established pattern" in the codebase. Like weeds in a garden, it spreads unnoticed. So keep the codebase as pure as possible, and what agents copy will always be the good stuff.

Encoding Architectural Constraints Directly into Lint Rules

The Grok team's client framework, Dune, encodes a great deal of experience into constraints and Lint rules. The most typical example is banning React's useEffect: it has its uses, but it's often misused—the industry jokes that using it well requires a PhD, and agents aren't good at it. Their approach is to ban it outright in product code, allowing it only in specific places inside the Dune API, so the problem vanished entirely from code review.

Another pattern is using directory structure to provide context. In the Electron app Caster, agents frequently imported main-process code into the renderer, causing jank. In the Grok and Dune apps, two top-level directories, Electron Renderer and Electron Main, were set up, and imports between them are forbidden by a dependency graph analyzer and Lint rules. The rule itself is simple, but it means one less thing to worry about during review.

But it means that during code review

useEffect and the Fragility of React State Management

React's useEffect is used to perform side effects in function components (such as data fetching, subscriptions, and manual DOM manipulation), but it couples side effects to the render cycle, and missing or incorrect dependency arrays easily lead to infinite re-renders, memory leaks, or stale closure values. Using it correctly requires a deep understanding of render timing, cleanup functions, and dependency tracking, hence the community's self-deprecating joke that "using useEffect well requires a PhD." Agents tend to reach for useEffect as a quick fix but struggle to reason systematically about its side effects, causing anti-patterns to spread through the codebase. By banning useEffect in product code via Lint rules and forcing developers toward more declarative data flow or custom Hooks, this class of problem can be eliminated at the root while removing useEffect as an "escape hatch" for agents.

Isolation Between the Electron Main Process and Renderer Process

An Electron app contains a main process (Node.js environment, responsible for system-level operations) and a renderer process (browser environment, responsible for UI). The two communicate via IPC, but if the renderer directly imports main-process modules, it pulls in Node.js dependencies, breaking the sandbox and causing performance problems (such as jank). The Grok team separates Electron Renderer and Electron Main as top-level directories and uses a dependency graph analyzer and Lint rules to forbid cross-directory imports. This directory-structure-based constraint gives agents clear contextual boundaries, so they naturally follow process isolation principles when writing code, reducing the burden of manual judgment during code review.

TLA+ and Formal Verification

TLA+ is a formal specification language for designing, modeling, and verifying concurrent systems and distributed algorithms. It describes system states and state transitions with mathematical logic and uses a model checker (such as TLC) to exhaustively explore the state space, checking whether properties such as invariants and liveness hold. The Grok team uses TLA+ to model user flows such as cross-device notifications to find logic holes at the design stage. Although TLA+ has a steep learning curve and high maintenance cost, agents can automatically generate specifications and run the checks, greatly lowering the barrier to formal verification. This lets the team catch subtle concurrency errors early without relying on expensive testing or manual review.

Banning 99% of Comments and Closing the Escape Hatches

One phenomenon surprised Potato: agents treat Markdown docs, code comments, and skill rules as places to "write down knowledge of what not to do," and even overreact to a small human correction, treating it as a hard rule that must never be broken.

In JavaScript performance optimization, imperative timers (setTimeout, requestAnimationFrame) cause garbage collection and jank when created and destroyed in large numbers. Agents, however, like to use them as quick fixes and explain in comments why the patch is justified—so timers spread everywhere. Having learned their lesson, the Grok team simply banned 99% of comments: comments go stale easily, humans are too lazy to read them, and they give agents an escape hatch for "this is too hard to do"—worst of all, comments can't be enforced. Now their Lint rules produce clear, useful error messages that guide agents to fix the root cause rather than patch the surface.

Verification Becomes the New Review

With static constraints blocking most problems, the code review process shrinks to inspecting verification artifacts: videos, screenshots, charts, and formal verification results.

The most basic example is runtime verification—giving agents the ability to run code themselves and use debugging tools. They have something called a "feature map," similar to a sitemap, that lists every available feature in the app from the user's perspective, so agents don't have to guess what "that thing in the left sidebar" refers to. Paired with a custom CLI, agents can reuse the same driver every time, saving tokens and making testing easier.

Going further is formal verification. The team uses TLA+ for model checking of key user flows (for example, reminder notifications across multiple devices and platforms) and Lean to prove that their in-house virtualization library won't flicker or jitter. Potato admits this is still experimental and not a panacea, but it has already found quite a few bugs. They also use the property-based testing library Bombadil (as stated) to test the UI, which likewise caught several UI bugs.

According to a tweet Potato cited, Cinder said six months ago that she couldn't trust agents because subtle bugs were being merged in; six months later, 70% of her daily tokens go to verification. Potato says the bulk of his token usage is likewise verification. None of these techniques are unprecedented inventions—they were just prohibitively expensive to maintain in the past, and agents have slashed that cost dramatically. Software testing and verification are entering a golden age.

It's basically like a sitemap

The Promised Land: 11,000 PRs and a 0.5% Rollback Rate

The return on this investment shows up in the numbers. Last month the Grok team submitted 11,000 PRs to production, the client ships roughly a new stable release every day, and in October about 227 people submitted at least one PR—many of them "drive-by contributors" with limited context.

More counterintuitive is the quality: the relative proportion of rolled-back PRs actually went down, holding at around 0.5%. Grok's performance has not only stayed stable but improved over the launch version across dimensions such as startup, typing, and conversation switching—and remember, it's still an Electron app.

Will this turn into a mess?

A concrete example: on every new release, Grok automatically spins up a new Cursor project, fuzz-tests the app and the build, uses the app like a real user to find bugs, then opens a fix PR for each bug and auto-merges it. That sounds crazy, but the premise is that the codebase is good enough that PRs are usually good by default, and most product code no longer needs deep manual review.

Potato's recommended order for teams is: before automating everything, try to improve the codebase every time you correct an agent—write a Lint rule to prevent an anti-pattern from recurring, clean up old patterns, and then make sure the verification approach is solid. If the inner coding loop produces poor code, automation only produces more poor code, and you'll have to throw more and more tokens at it—an extremely expensive path.

In other words, what turns poor code into gold isn't a stronger model—it's the foundation you lay for your agents.

One-line Summary

The Grok team uses deterministic constraints and automated verification to achieve 11,000 PRs a month with a 0.5% rollback rate.

Paragraph Summary

Based on a talk by xAI engineer Potato, this article reveals how the Grok team uses an engineering system of "deterministic constraints + automated verification" to achieve 11,000 production PRs a month with only a 0.5% rollback rate. The core is a four-layer trust pyramid that prioritizes deterministic tools—architecture, Lint rules, and tests—and turns every correction into a code-level check rather than a prompt. The team treats the codebase as a garden and keeps it pure by banning useEffect, restricting cross-process imports, and banning 99% of comments, so agents produce correct code by default. Verification becomes the new review, with runtime verification, formal verification, and feature maps letting most PRs auto-merge. Ultimately, quality doesn't depend on a stronger model but on laying a solid foundation for agents.

#Grok#xAI#AI coding agents#deterministic constraints#Lint rules#formal verification#code review automation
Share:

Related articles