Harvard Study: AI Coding Agents Increase Code Volume by 30%, But Delivery Stays Flat

Harvard economists Fiona Chen and James Stratton recently released a working paper that used data from the Jellyfish platform to track the work patterns of 718 companies and over 700,000 engineers. The conclusion is quite counterintuitive: AI coding tools increased the amount of code engineers wrote by 30%, but the actual amount of work completed barely changed.

How the Study Was Conducted

The data for this study comes from Jellyfish—a commercial analytics company that specializes in tracking engineering activity. The dataset covers 718 companies and over 700,000 workers, spanning from 2021 to 2026. It tracks not just code commits, but also issue tickets, calendar schedules, and other engineering management dimensions.

The AI coding tools involved include Claude Code, Cursor, and Devin. These tools have spread through enterprises very quickly: from nearly zero at the end of 2024 to over 95% adoption among surveyed companies by early 2026.

There's a clever aspect to the research design—because companies adopted AI tools at different times, the researchers could compare teams that had "already switched" against teams that "hadn't switched yet," thereby isolating the impact of the AI tools themselves rather than mistakenly attributing broader industry trends to AI.

A working paper is a common form of research output in economics: authors publicly release a draft before formally submitting to a journal or undergoing peer review, in order to quickly disseminate findings, gather feedback, and stake a claim to priority. These are typically published on platforms like NBER (National Bureau of Economic Research) or SSRN, without vetting by journal editors or anonymous reviewers, so the robustness of conclusions and reproducibility of data have not been independently verified. Readers should be cautious when citing conclusions from working papers—especially when they involve rapidly changing fields (such as AI tools), where the sample window, variable definitions, and identification strategies may all be adjusted in subsequent versions, potentially even leading to conclusions opposite to the initial draft. This is also why the authors themselves actively emphasize that the paper has "not yet undergone peer review."

Difference-in-differences is the core of this paper's identification strategy. The basic idea is: treat teams that have "already adopted AI tools" as the treatment group and teams that "haven't adopted yet" as the control group, then compare the difference in changes between the two groups before and after adoption. The key advantage of this approach is that it can eliminate common trends affecting both groups (such as productivity fluctuations across the industry due to economic cycles or technology stack upgrades), thereby isolating the impact of the "adopting AI" event itself. This paper leverages the staggered adoption timing across companies to construct a natural quasi-experimental condition—a classic application scenario for this method.

Writing Code Got Faster, But the Bottleneck Shifted

The output-side data is indeed impressive. Per-capita code volume grew by about 30%, commits by about 20%, and pull requests by about 23%. From the pure act of "writing code," the efficiency gains from AI coding tools are real and substantial.

The problem lies downstream. After code is written, it must go through review before it can be merged and deployed, and the review process simply hasn't kept up: review time increased by 49%, the proportion of PRs requiring rework nearly doubled, the number of review comments increased by about one-third, and the number of engineers pulled in to participate in reviews is also rising.

The paper's exact words are: "Productivity gains did not fully pass through to changes in software output or employment." In other words, writing more code doesn't equal delivering more software.

Why "Writing More" Didn't Become "Delivering More"

This result actually reveals an old problem in software engineering: software development was never a single-stage game—it's an assembly line. When AI dramatically speeds up the "writing code" stage, the bottleneck of the entire pipeline shifts to the slowest stage—review and verification.

AI-generated code requires human review, and reviewers are facing more code that may vary widely in quality. PR volume increased by 23%, but the review time and number of revisions required per PR are both rising. Reviewers are overwhelmed by large volumes of low-quality or rework-requiring code, which actually reduces overall throughput efficiency.

The implication for teams is direct: if you only invest in AI coding tools without correspondingly strengthening code review capacity (whether by adding reviewers, improving review processes, or using AI-assisted review), the returns on your investment will be eaten up by the downstream bottleneck. The paper's authors are clear—to capture the real benefits of AI coding, you must simultaneously invest in the review stage, not just make code get written faster.

Amdahl's Law is a classic conclusion in computer architecture, but it applies equally well to understanding the phenomenon in this paper: the overall speedup of a system is limited by the proportion of the system that cannot be accelerated. Suppose that in a software delivery pipeline, "writing code" accounts for 30% of total time and "review and verification" accounts for 70%. Then even if you make the code-writing stage infinitely fast, the upper limit on overall delivery speed improvement is only about 43%. AI coding tools trigger bottleneck shift precisely in this sense—they accelerate the accelerable parts, but make the unaccelerated or not-yet-optimized review stage the new constraint. This explains why code output grew by 30% while end-to-end delivery barely changed: the gains were consumed by downstream fixed costs.

Bottleneck shift is a core concept in the Theory of Constraints. The theory holds that the output of any system is determined by its weakest link; optimizing non-bottleneck stages does not increase overall output but only leads to accumulation of work-in-progress (WIP). In this paper's data, PR volume up 23%, review time up 49%, and rework ratio doubling are classic signals of "upstream acceleration, downstream congestion": more code did not flow through the review gate to become deliverable software, but instead piled up in review queues or even flowed back in the form of rework.

Limitations of the Study

One additional point: this is a working paper that has not yet undergone peer review. The data comes from the commercial analytics company Jellyfish, which may involve selection bias among platform users. The authors themselves emphasize that these are short-term effects—as teams gradually adapt to new tools and adjust workflows, long-term results may differ.

But even with these caveats, the core finding that "coding acceleration ≠ delivery acceleration" is still worth taking seriously by every team that is introducing or preparing to introduce AI coding tools. It reminds us: when evaluating the value of AI coding tools, you cannot look only at code output metrics—you must also look at end-to-end delivery efficiency.

Selection bias refers to a situation where the composition of a sample is not random, causing observed results to systematically deviate from the true population. All data in this paper comes from clients of the commercial analytics platform Jellyfish. Companies willing to pay for engineering efficiency tracking tools tend to already be larger, have more standardized engineering management, and be more sensitive to measurement metrics—such companies may differ systematically from smaller teams that don't use such platforms in terms of AI adoption speed and review process maturity. Therefore, whether the study's conclusions can be extrapolated to the entire software industry remains an open question. This is a common challenge faced by all research based on commercial platform data.

One-line Summary

AI coding tools increased code output by 30%, but delivery barely changed, with the bottleneck shifting to code review.

Paragraph Summary

Two Harvard economists tracked 718 companies and over 700,000 engineers using the Jellyfish platform and found that AI coding tools (Claude Code, Cursor, Devin) increased per-capita code volume by about 30%, with commits and PRs growing 20% and 23% respectively, but actual software delivery barely changed. The reason is that the bottleneck shifted downstream: review time increased by 49%, the proportion of PRs requiring rework nearly doubled, and both review comments and participant numbers are rising. The study leveraged staggered AI adoption across companies to isolate the true impact of the tools themselves. The paper concludes that productivity gains did not pass through to software output or employment. The authors note this is a working paper that hasn't been peer-reviewed, the data has platform selection bias, and these are short-term effects—but the core finding that "coding acceleration ≠ delivery acceleration" still deserves attention from all teams adopting AI coding tools.