Why AI Coding Makes Your Codebase Worse Over Time: HumanLayer CEO Breaks Down the Truth

When Coding Agents Become a Code Quality Liability

A widely overlooked fact is coming to light: AI coding agents tend to degrade code quality as they continuously iterate on the same codebase. In a conversation with Dex Horthy — founder and CEO of HumanLayer and the person credited with coining the term "context engineering" — what once felt like a vague intuition has become a quantifiable reality.

Dex pointed to a benchmark called Slop Code Bench, built by researcher Gabe Orlansky's lab at the University of Wisconsin. Its goal is straightforward: prove that models make codebases worse over time. This stands in sharp contrast to most mainstream coding benchmarks today, and exposes a structural blind spot in how we evaluate AI coding capabilities.

Why Existing Benchmarks Don't Reflect Real Development

Dex breaks down past coding benchmarks into roughly three categories. The first is the "small slice" approach, exemplified by SWE-bench: roll a real open-source repo back to before a given PR, hand the model a historical issue, ask it to fix the bug, then validate with the benchmark's built-in tests. "A model scores 95% on SWE-bench and everyone says coding is solved — but that's nothing like real software development."

The second category is "marathon" tasks — rebuilding something like Redis or Excel from a spec document, with individual tasks potentially taking 4 to 400 hours. The third comes from Cognition's approach: using larger tasks with a "judge model" that evaluates afterward whether the code violated the codebase's rules, conventions, and patterns.

But all three share a fundamental flaw: they hand the model the complete problem from the start. Real software development is the opposite — you rarely know all the requirements upfront.

SWE-bench is one of the most influential code evaluation benchmarks today, released in 2023 by a Princeton research team. It pulls resolved issues from real open-source GitHub projects (like Django and scikit-learn), rolls the codebase back to when the problem existed, and asks the model to generate a fix patch based solely on the issue description, then validates using the original repo's unit tests. The design is clever in that it mirrors "real problems with known answers" — but as Dex points out, it's fundamentally a closed, single-shot repair task: a clean starting point, a well-defined goal, and pre-existing tests. This is a far cry from what engineers face daily — production codebases that evolve constantly and carry years of technical debt. Moreover, because the benchmark is so well-known, model training data may already include the relevant issues and PRs, introducing data contamination risk that further undermines the scores' reliability.

What Makes Slop Code Bench Clever

Slop Code Bench works differently: it has models build a project from scratch (greenfield), then keeps adding features — "now add this feature to the codebase, now add this one, now add another." Each challenge includes 3 to 10 checkpoints, simulating a model working continuously without human oversight and without full knowledge of the big picture.

The results are telling: the number of failing tests trended upward for every model tested. Dex revealed that GPT-5.5 Codex scored around 15% on one run, while Astra — which he tested independently — came in at roughly 16.5%, just one percentage point higher. The benchmark hasn't yet been widely adopted by model cards, partly because it's slow and expensive to run, but it's the first to provide quantifiable evidence that "models aren't good at maintaining codebases over time."

What's especially concerning, Dex notes, is that while models have made impressive gains in areas like computer use and game generation, they've made almost no progress on writing maintainable code, managing complexity, deleting code, and changing abstractions — the capabilities that truly reflect engineering judgment.

"Slop" is slang for low-quality, bloated content, and has been adopted by the AI community to describe the redundant, superficially correct but unmaintainable code that large models tend to generate. The name Slop Code Bench makes its critique explicit: it's not testing whether a model can solve a problem in one shot, but whether the model will keep producing garbage as tasks accumulate. Greenfield development refers to building a new project from scratch with no legacy code constraints — the most ideal starting condition. Slop Code Bench deliberately starts from this optimal baseline and layers on requirements one by one to observe how code quality decays, making the degradation curve clearly visible. The benchmark's high runtime cost stems from the fact that it requires multiple full rounds of code generation and test execution rather than single-pass inference, making it far harder to run at scale compared to SWE-bench.

Don't "Mortgage" Your Codebase

There's been a popular rationalization floating around: "Let the code get messy — when it really becomes a problem, that'll be GPT-7's problem." Dex's response is blunt: "We're on GPT-6 now, and there hasn't been much progress on this front."

His core advice: don't bet your codebase on the fantasy that a model smart enough to fix everything will arrive in a year. The major labs are clearly more focused on climbing new hills like computer use, and the last 10% of software maintainability may be just as hard as the first 90% — or harder.

Collaboration Over One-Shot: The Value of Design Docs

When it comes to enterprise adoption, Dex emphasizes that HumanLayer has the biggest impact on engineering teams of 100 or more — because these teams have decade-old codebases, hundreds of repositories, complex dev environments, and all kinds of "special conventions" that cloud-based agents will constantly stumble over.

He pushes back hard against the "one-shot" fantasy: "No amount of effort will let you hold a five-hour meeting to make a perfect plan, feed it to an agent, and get a perfect result." What actually works is what engineering teams have been doing for the past 30 or 40 years — aligning on what you're building before you start building it.

HumanLayer calls this process a "design doc," deliberately distinguishing between "where we're going / the end state" and "how we get there / the specific changes." The key insight: from a 25,000-foot view, a one-page overview can eliminate 50% of the directions that could go wrong; zoom to 10,000 feet and you can eliminate 80% of bad approaches. By the time code is produced, rework rates can drop from the common 25%–75% range down to 0%–10%.

The "Instagram Phase": Most AI Engineering Flex Is Fake

Dex is unsparing in his critique of current industry culture. Borrowing a phrase from another guest, he calls this the "Instagram phase" of AI engineering — everything is filters, gloss, and performance. Say something isn't working for you and someone fires back "skill issue."

He witnessed a telling scene firsthand: a legacy company with thousands of engineers, where a department head heard another company claim "90% of our code is AI-written," while his own senior engineers were drowning in "30,000-line garbage PRs" generated by AI. The result: an executive class gripped by competitive anxiety — "Am I in the 90th percentile or the 99th? Am I being left behind?"

As for so-called token maxing — bragging about how many tokens you burned, how you blew through six Codex subscriptions in a week — Dex is direct: "What did you actually ship?" He thinks forcing the narrative that "if you're not maxing out six subscriptions you're a bad engineer" is genuinely harmful to the industry. His position is consistent: reason from first principles about what actually works and what's practical, rather than reverse-engineering from whatever you saw online.

Realistic Productivity Multipliers: Stop Chasing 100x

When asked about reasonable productivity multipliers, Dex gives a grounded answer: "If every engineer is 2 to 4x faster, that's transformative for human society." But many people will miss out on that gain because they're "too busy chasing 100x."

He advocates getting to a stable 2–3x first, then compounding another 2–3x on top — multiplicative growth. Trying to tear down all past engineering practices and rebuild the entire engineering system from scratch "doesn't work." This is ultimately the classic YAGNI principle — don't over-engineer for an imagined future.

Code Review and the Challenge of Passing Down "Taste"

Dex distinguishes between two purposes of code review: ensuring correctness and ensuring maintainability. Screenshots and videos can verify that features work correctly; deterministic tools (like cyclomatic complexity checks) can catch broad categories of bad code — but "I've never seen any prompt or deterministic tool that can meaningfully counteract a model's tendency to write unmaintainable code without human intervention."

Maintainability depends on "taste," and taste comes from being burned by bad architecture — those 3 a.m. firefights, those moments of swearing "I will never let anyone merge this pattern again," accumulated into a senior engineer's judgment. The question is: in the age of agents, how does the next generation of engineers build taste?

Historically it was passed down through PR reviews, but the worst-case scenario now is: a developer submits code they don't actually understand, takes review comments, pastes them into AI to fix, "in which case I'm doing all the important work for you." Dex admires Mitchell Hashimoto's approach — leaving comments on PRs that just say "ask your Claude why this is wrong," with no hints, forcing the developer to genuinely understand. He also predicts that as design docs become standardized, "teaching and mentoring at the design stage" will become just as important as — or more important than — code review, because a design doc is essentially a prompt.

Cyclomatic Complexity was introduced by Thomas McCabe in 1976 and is a classic metric for measuring the logical complexity of code — its value equals the number of linearly independent paths through a piece of code's control flow graph. The higher a function's cyclomatic complexity, the more nested branches and loops it contains, and the harder it is to understand, test, and maintain. It's generally recommended to keep individual functions below a complexity of 10. These deterministic static analysis tools (which also include line count, coupling metrics, duplication detection, etc.) can automatically catch obvious structural problems and are common quality gates in CI/CD pipelines. But as Dex points out, these tools can identify "formal messiness" — they cannot judge "whether this abstraction layer fits the overall design philosophy of this particular codebase." That kind of judgment is an intuition only engineers who've suffered through architectural debt can develop. YAGNI (You Aren't Gonna Need It) is a core principle of Extreme Programming (XP), advocating that you only implement what's genuinely needed right now and avoid over-engineering for hypothetical future scenarios — perfectly aligned with Dex's argument against betting on an imagined future model.

Dex's Real Agentic Workflow

Cutting through the performance, Dex breaks down HumanLayer's actual workflow:

• 50%–60% of work is basically done in one shot: Changes under a few hundred lines get no upfront planning — just do it, test it, add a few more prompts.
• About 10% is handled overnight by "janitor bots": Control loops set based on the "desired end state of the codebase," operating like a thermostat to automatically generate small PRs for cleanup, refactoring, and endpoint migrations. At most one PR of each type is open at a time, labels prevent duplicate fixes, and they're deliberately kept small — "if a PR is too big it never gets merged."
• About 30% goes through an RPI process (Research-Plan-Implement, with evolved variants called crispy/cordoy, etc.): Start with codebase research, build a query plan, then use a fresh session that doesn't know the original ticket to do the research, building up the plan in layers.

For large changes in the tens of thousands of lines, Dex's approach is to check in at intervals or break the work into stacked PRs: "Checking every thousand lines is far less painful than writing ten thousand lines, finding everything is broken, and debugging from there." He also mentions cross-vendor sub-agent orchestration techniques, such as having one model orchestrate sub-agents from another.

Stacked PRs are an engineering management strategy for large changes: decomposing a massive change into a series of interdependent small PRs, where each subsequent PR branches off the previous one, forming a "stack." The benefit is that each PR is small, easy to review, and surfaces problems quickly after merging — avoiding the design flaws that only appear when thousands of lines of changes pile up. In the age of agents, Dex recommends this approach for a specific reason: models tend to "drift" in long contexts, and staged commits provide natural checkpoints for humans to step in and evaluate at each critical juncture, rather than waiting until the agent has run the entire task only to find the direction has gone off the rails. The RPI process (Research-Plan-Implement) explicitly breaks agent tasks into three independent phases, with the core idea being to use "a fresh session that doesn't know the original requirements" for research — avoiding the bias and path dependency from the first conversation contaminating later planning. This is methodologically consistent with the classic software engineering principle of "separation of concerns."

Do Less in Parallel — Focus on the One Most Important Thing

On parallel execution, Dex holds a counterintuitive but practical view: "I don't want to wake up every morning to ten thousand lines of code." Real work should involve more human-in-the-loop engagement.

He pushes back on reflexive parallelism: instead of flying 10 things at once, forgetting two, dragging four out for days, and only delivering two — do "two at a time." End-to-end value delivery is actually faster. His principle: always know what the one most important thing is, and the moment it's unblocked, drop everything else and push it forward. Everything else is just gap-filling while you wait.

On token anxiety, Dex identifies a critical mismatch: the Twitter celebration of "maxing out six subscriptions" is completely disconnected from the real world — for security and data retention reasons, nearly all enterprises pay per token, with no subscription plans. He predicts we'll look back on this era of "subscription maxing" and recognize it as a brief illusion created by a particular model distribution mechanism.

As he keeps coming back to: "What has always been true in engineering is still true across all of this work."