The Real Bottleneck in AI Coding Isn't the Model — It's Closed-Loop Verification

The Model Isn't the Bottleneck Anymore — Verification Is

A blogger overseas put forward a counterintuitive argument: in today's AI coding practice, model capability is no longer the biggest bottleneck. What truly determines success or failure is closed-loop verification.

He quoted Boris Cherny, the creator of Claude Code, who said he stopped writing prompts long ago — he now "writes the loop." A real loop consists of five components: Trigger, Goal, Work, Memory, and Verification. Verification is the only feedback gate allowed to say "no, we're not done yet."

The blogger's take is blunt: nearly everyone is tweaking models and prompts, but very few are seriously building out the verification step. In his view, the quality of your loop is only as good as how strict your feedback gate is.

A Painful Lesson: Green Tests That Lied for Two Days

To drive the point home, the blogger shared a real experience. He let an AI agent run for two full days, with every iteration showing all-green tests. He opened the app with excitement — only to find the entire checkout flow was completely broken. Buttons didn't respond. No errors. Just dead.

What went wrong? The agent had been running unit tests against its own generated mocks. Everything was green on its own machine, and it confidently reported "done" — but real users couldn't use the product at all.

This raises a core question: what signal actually closes the loop? The answer is simple — the deployed thing works correctly in front of real users. Green tests on the agent's own machine prove nothing.

What is a Mock? Mock (simulated data/mock objects) is a common practice in software testing: pre-configured fake data or fake interfaces replace real external dependencies (like databases, payment gateways, or third-party APIs), allowing unit tests to run quickly in an isolated environment. Mocks are fast, repeatable, and don't rely on network or external services — but as the blogger discovered, they only verify "whether the code behaves as expected under controlled assumptions," not "whether real users can complete actions in a production environment." When an agent generates its own mocks, the problem is even worse: it's both the question-setter and the answer-writer, so tests will naturally pass while any divergence from real business logic stays completely hidden. This phenomenon is sometimes called "testing theater" in software engineering — it looks like quality assurance, but it's really just self-confirmation.

Using TestSprite CLI as a "Strict Judge"

The blogger's solution is to introduce an open-source validator: TestSprite CLI (Apache 2.0 license). The agent calls it during the build process, and it interacts with the live deployed application like a real user — not with mocks.

When the checkout flow broke again, it didn't just report "failure" — it returned a screenshot showing exactly what the user would see: that frozen, dead button. The agent read the screenshot, fixed the issue, and re-ran — all without any human intervention. That's what a closed loop actually feels like: not a bigger brain, but a stricter judge.

Installation is straightforward: requires Node 20.19, 22.13, or 24+, install via npm install -g @testsprite/testsprite-cli, grab an API key from the console, run testsprite setup to paste the key, and confirm the connection with testsprite auth status.

The blogger's advice: when designing your next project, don't start with the model — start with the validator.

The Tech Behind TestSprite CLI TestSprite CLI's core technical approach falls under end-to-end testing (E2E) or visual regression testing. Unlike unit tests, E2E tests simulate real user browser behavior — clicking buttons, filling forms, navigating pages — to verify that an entire business flow works end to end. The screenshot-return design is especially important: it gives AI agents "visual feedback," letting them directly perceive page rendering state rather than just parsing structured test reports. This closely mirrors how a human developer visually inspects a page, making the agent's "perceive–judge–fix" loop much more aligned with real debugging. The Apache 2.0 license means the tool can be used freely in commercial projects without requiring that modifications be open-sourced, lowering the barrier for enterprise adoption.

Subscription Showdown: Codex, Claude Code, and GLM

The second half of the video is a thorough subscription value comparison. The blogger admits that for many coding tasks he personally still prefers Fable 5.1 over GPT-6 Astra — but if you ask him which subscription to recommend right now, his answer is Codex. On a tight budget, GLM is also worth a look.

Codex: Generous Limits, Complete App Experience

Codex's paid tiers are ChatGPT Plus ($20/month), Pro 5x ($100), and Pro 20x ($200). The blogger tested the $200 Pro 20x plan: roughly 43 minutes of development work moved the weekly usage meter from 0 to 3%, and after auditing, that amounted to approximately $34 in API equivalent usage. From this, he made an "illustrative, unverified" extrapolation — if used to capacity each week, the Plus plan is worth roughly $240 in API equivalent, Pro 100 roughly $1,220, and Pro 200 roughly $4,900.

He repeatedly stressed the unreliability of these numbers: rounding at the decimal level and uncertain update timing both make the conversion imprecise. These figures represent what one sample implies, not what any account is guaranteed.

The Codex desktop app is a big reason he prefers it: you can generate illustrations or game assets with GPT's image model while building an app, and the built-in browser lets you click, inspect, and verify fixes. The ChatGPT chat interface (including mobile) is also useful beyond just coding. His only concern is that usage rules and promotional terms change too frequently — he'd prefer more predictable subscriptions.

Claude Code: Great Model, Hard to Recommend as a Subscription

Claude's tiers are Pro ($20), Max 5X ($100), and Max 20X ($200). When the blogger tested Pro, roughly $0.13 in API equivalent activity corresponded to a 4 percentage point increase on the 5-hour meter, extrapolating to roughly $101/month in API equivalent for Pro.

His position is clear: Fable 5.1 is his favorite model, especially for frontend work and complex debugging. But at the subscription level it's hard to recommend — in the Max plan, Fable can at most consume half the weekly quota, and going beyond that costs extra. He still prefers the Claude Code terminal (CLI) experience over Codex, so people who primarily work in the terminal have a stronger case for it — but the desktop app experience is noticeably weaker.

The Hidden Value of Caching

The blogger also raised a commonly overlooked factor: caching. Within the same project, models can reuse context from earlier requests at low cost. In his samples, roughly 94% of Astra's input and 79% of Sonnet's input came from cache. He modeled a hypothetical 98% reuse scenario, finding that Codex workloads could handle roughly 25% more tokens while Claude handled roughly 9.3% more — the difference stems from the fact that most of the API value in the Claude sample was spent on output, and output doesn't benefit from input caching discounts.

How LLM API Caching Works Large language model API caching (Prompt Caching) allows providers to reuse repeated context tokens within the same request over a short period, charging a lower rate for that portion of input (typically 10–50% of the original price). In coding agent scenarios, system prompts, codebase context, and tool descriptions barely change across multiple turns — so cache hit rates are extremely high. The fact that roughly 94% of Astra's input and 79% of Sonnet's input came from cache in the blogger's sample means actual API costs are far lower than the face-value token count would suggest. Understanding this is crucial for evaluating subscription plans: subscriptions internalize the caching benefit as "more usable quota," while developers calling the API directly need to proactively design prompt structures to maximize cache hits — otherwise the real cost for the same workload will be significantly higher.

GLM: A Surprising Option for Tight Budgets

GLM's coding plans are priced at $18, $80, and $168/month (standard), with Zcode available as a free download. The blogger was impressed by the $18 plan — the sheer generosity of its quota puts it in the same conversation as hundred-dollar plans, especially if "running out of credits" is your main pain point. It works well with Zcode, includes browser automation, and can keep pushing toward a goal across long-running tasks.

Final Recommendation: Start with the Smallest Plan

The blogger's overall advice is clear: most people should choose Codex right now — especially those who want a desktop app. Astra is capable enough, the quota feels more generous, and the image generation and browser workflow are solid bonuses. If Fable consistently outperforms on your critical tasks and you prefer the Claude Code CLI, Max is still a reasonable choice. On a tight budget, try Zcode and the $18 GLM plan.

His parting advice is practical: start with the smallest plan that can handle your workload, and only upgrade once you've genuinely maxed out the smaller quota. Big plans look great on a spreadsheet, but paying for capacity you don't use won't help you get things done.