METR Study Breaks Down: Developers Won't Work Without AI, Speed Measurement Fails

Title

METR Study Breaks Down: Developers Won't Work Without AI, Speed Measurement Fails

Meta Description

METR's AI coding productivity RCT failed on its second attempt: many developers refused to work without AI, and the control group vanished. Self-reported speed gains were 3x, but earlier trials showed subjective perception overestimates by 40 percentage points. This article analyzes the measurement dilemma and provides a practical weekly landed/dropped/reverted counting script.

Article Content

METR once conducted a rare randomized controlled trial to measure the real impact of AI on coding productivity, with surprising results. When they tried to replicate the study, they found the experiment itself could no longer be conducted—too many developers refused to work without AI. The issues this reveals are more worth pondering than "does AI make you faster."

First trial: felt 20% faster, actually 19% slower

In early 2025, METR recruited 16 experienced open-source developers at $150/hour to work on 246 real issues in large open-source repositories they had known for years. Each issue was randomly assigned to allow or forbid AI use.

Result: when AI was allowed, task completion time actually increased by 19%. But developers predicted before starting that AI would speed them up by 24%, and after completing they still believed AI had accelerated them by 20%. According to METR's July 2025 report, this gap of about 40 percentage points between subjective perception and objective measurement became the core of subsequent discussion.

The trial design itself was rigorous—randomization, real tasks, experienced developers. The problem lay in human perception.

Randomized controlled trial (RCT) is the gold standard for causal inference, eliminating selection bias and confounding factors by randomly assigning interventions (here, whether AI use is allowed). RCTs are rare in software engineering research because it is difficult to achieve randomization, control variables, and ensure ecological validity in real development scenarios. METR's trial randomly assigned each issue to allow or forbid AI use, making the two groups statistically comparable in developer experience, task difficulty, and other dimensions, thereby isolating the causal effect of the AI tool. But RCTs have limitations: they measure average effects under specific tools, specific tasks, and specific populations, and extrapolation requires caution.

Second trial: the control group disappeared

In August 2025, METR launched a larger follow-up experiment: 57 developers, 143 repositories, over 800 tasks. The results announced in February 2026 left even the researchers themselves in a difficult position.

METR stated outright in its report that this batch of data "cannot provide us with a reliable signal about the current productivity effects of AI tools."

The core problem lay in recruitment. METR observed that "the number of developers who chose not to participate in the study rose sharply, due to unwillingness to work without AI." Even those who participated self-selected: 30% to 50% of developers said they would actively not submit certain tasks because they did not want to complete them without AI. METR believes both effects would cause the data to underestimate AI's speedup effect.

Two other problems compounded: compensation dropped from $150 to $50 per hour, introducing additional selection bias; and for developers running multiple AI agents simultaneously, METR admitted that its task time measurement itself was unreliable—if you start an agent and go do something else, how long did the task actually take?

The data METR obtained: the returning developer group showed "-18% speedup, with a confidence interval between -38% and +9%"; newly recruited developers showed "-4% speedup, confidence interval -15% to +9%." Note that the negative sign here indicates shorter task time—the opposite direction from the 2025 results. But both confidence intervals cross zero, meaning the conclusions are not significant. METR's own interpretation is: developers may indeed be faster now than in early 2025, but the evidence this batch of data can provide is very weak.

A randomized controlled trial designed to measure AI coding speed lost its control condition.

Confidence interval is an interval used in frequentist statistics to express estimation uncertainty, usually paired with a 95% confidence level. If the interval crosses zero (e.g., -38% to +9%), it means that under the null hypothesis (no effect), the probability of observing the current data or more extreme cases is not less than 5%, so the hypothesis of "no effect" cannot be rejected. Interval width reflects precision: smaller samples and larger variance produce wider intervals. METR's confidence intervals in both trials crossed zero, indicating that even if point estimates show speedup, the evidence is insufficient for a significant conclusion.

How unreliable self-reported data is

METR also conducted a survey covering 349 technical workers between February and April 2026, including 87 software engineers. Respondents' self-reported median speedup was 3x, and median work value increase was 1.4 to 2x. Respondents also said they would be willing to give up 29% of their salary to keep AI tools for one month—METR itself flagged this item as weak evidence.

METR attached an explicit warning when publishing these numbers: "Survey results are not necessarily based on reality." The same article reviewed the 2025 trial—"people on average overestimated the impact of AI on their task time by 40 percentage points."

This is not an argument that AI makes you slower. The 2025 trial used early tools (mainly Cursor Pro with Claude 3.5 and 3.7 Sonnet), operating on large mature repositories very familiar to developers. METR itself believes the situation has changed. What is truly noteworthy is: the measurement tool everyone relies on most—"how fast does it feel"—was once calibrated against a stopwatch, with a deviation of about 40 percentage points. And now the stopwatch is hard to run, because the people being timed are unwilling to put down their tools. All that remains is feeling. And feeling is precisely the only measurement method with a known error bar, and that error bar is huge.

Selection bias refers to samples not being randomly drawn, causing estimates to systematically deviate from the true population value. In METR's follow-up trial, developers who refused to participate or actively withheld certain tasks may have been exactly those who found AI less helpful or used it in unusual ways, thereby distorting the measurements. Similarly, lower compensation may attract participants with different motivations. Selection bias cannot be eliminated by increasing sample size; it can only be mitigated by improving recruitment design or using methods such as instrumental variables.

"How many agents I started" is a new kind of feeling metric

"I run five agents at once" sounds like a number, but it is just an input quantity.

METR's stopwatch failed precisely because developers were running multiple agents simultaneously and switching to other work in between—which is exactly the workflow people often use to prove productivity gains. Starting more agents is easy to count and easy to feel good about, but it does not tell you how much code ultimately made it into the main branch.

An agent that runs for forty minutes and finally hands you a discarded diff still consumed your review time. An agent whose change is merged and then reverted two days later costs even more. Neither is visible in "number of agents started."

There are three metrics that can be counted without a stopwatch or intuition:

• Landed: the number of tasks whose changes were merged into the branch you use for releases
• Dropped: the number of PRs closed without being merged
• Reverted: the number of changes that were undone after being merged

None of these measure speed; they count output that survived review—closer to what teams actually pay for. Crude, lagging, but not dependent on "how did this week feel."

A counting script you can run in one minute each week

If your agent work ultimately ends in pull requests, GitHub CLI and git can give you these three numbers. The following script counts the last seven days in the current repository:

#!/usr/bin/env bash

**GitHub CLI (gh)** is GitHub's official command-line tool. It supports filtering PRs using GitHub search syntax (e.g., `created:>=2025-01-01`) via the `--search` parameter, and extracting structured fields via `--json` and `--jq`. In the script, `gh pr list` by default returns only open PRs; you must explicitly specify `--state all` or `--state merged`/`--state closed`. `git log --grep='^Revert'` leverages the git commit message convention: the default `git revert` generates a commit message starting with `Revert "..."`, so rollback counts can be derived from this. These commands depend on timely synchronization between the local repository and the remote API.
# weekly-landed.sh: what landed in the last 7 days
since=$(date -d '7 days ago' +%F 2>/dev/null \
  || date -v-7d +%F)

opened=$(gh pr list --state all --limit 500 \
  --search "created:>=$since" \
  --json number --jq length)

landed=$(gh pr list --state merged --limit 500 \
  --search "merged:>=$since" \
  --json number --jq length)

dropped=$(gh pr list --state closed --limit 500 \
  --search "is:unmerged closed:>=$since" \
  --json number --jq length)

reverted=$(git log main --since="$since" \
  --oneline --grep='^Revert' | wc -l | tr -d ' ')

echo "since:    $since"
echo "opened:   $opened"
echo "landed:   $landed"
echo "dropped:  $dropped"
echo "reverted: $reverted"

The date line is compatible with both GNU (-d) and macOS (-v). The revert count only captures records that retained git's default Revert "..." commit message, and assumes your main branch is called main and your local copy is up to date. Branches abandoned without opening a PR will not appear in dropped. PR counts include PRs submitted by everyone (or every agent) in the repository, which is fine: you are tracking team output.

Run it once every Friday and record the numbers. Compare with last week, not with other teams. If adding one agent doubles "opened" while "landed" stays flat, the answer is clear.

What tools can and cannot help with

The author of this article is a developer of Hermes IDE; the following section should be read with that context in mind.

Counting landed work is easier when each agent task has a clear start and end. In Hermes, ⌘N (Ctrl+Shift+N on Windows/Linux) starts each task in a fresh hermes/<name> branch and worktree, with the task itself as the agent's first prompt. Each session displays a status: working, needs approval, asking you a question, done, error, or idle, and Hermes indicates when a status is guessed.

A task remains a task until you decide where it goes. Hermes saves every round of agent interaction and can diff or restore any single round. Review Desk (⌘G) shows changes by round or by file, with risk markers, and inline comments can be sent directly back to the agent that wrote that line of code.

Land completes commit, open PR, or local merge in one step, merging into the branch where the task started, and is undoable. Your custom checks can determine when a task counts as done. When a PR opened by Land is merged, the script above will count it. Local merges do not appear in PR numbers; if that is how you land, you need to count from git log.

Hermes has no productivity dashboard and does not measure your speed. The numbers still come from git and GitHub. Each agent runs its own interface in a real terminal, using your own login credentials, and Hermes only adds branches, status, review, and the Land step around the periphery. If you only use one vendor, its own tools may be enough: Claude Code by default moves background sessions to worktrees under .claude/worktrees/, and the Codex app runs chats in parallel under $CODEX_HOME/worktrees.

Hermes is source-available under BSL 1.1 and supports macOS, Windows, and Linux.

So, are you actually faster?

For most people: unknown. METR had randomization, a stopwatch, and a budget, and still could not get a reliable number this year. You will not get an answer from how some Tuesday feels either.

Run the script on Friday, run it again next Friday, and then consider adding the next agent.

Would you accept METR's no-AI task, even at $150/hour? If you track agent output in a team, what do you count?

One-line Summary

METR's follow-up RCT lost its control group because developers refused to work without AI; subjective perception diverges hugely from objective measurement, suggesting countable metrics like landed, dropped, and reverted should replace feeling.

Paragraph Summary

METR's 2025 randomized controlled trial found that developers subjectively felt 20% faster with AI, while actual task time increased by 19%, a deviation of about 40 percentage points. The follow-up trial launched in August 2025 lost its control group because many developers refused to work without AI, and METR itself judged the data as an unreliable signal. Survey self-reports showed a median speedup of 3x, but METR warned that survey results are not necessarily based on reality. The article argues that since the stopwatch is hard to run, we should turn to countable metrics such as landed, dropped, and reverted, and provides a weekly bash script to help teams track output that survives review.