Do AI Coding Assistants Really Make Developers Faster?
The honest answer to whether AI coding assistants make developers faster is that it depends who you measure, on what task, and what you count as finished. Four credible controlled studies have produced results ranging from a 55.8% speedup to a 19% slowdown, and none of them is obviously wrong. The disagreement is the finding.
The study everyone quotes: a 55.8% speedup
The number in most vendor decks comes from a 2023 controlled experiment run by Sida Peng, Eirini Kalliamvakou, Peter Cihon and Mert Demirer, published as “The Impact of AI on Developer Productivity: Evidence from GitHub Copilot”. Ninety-five professional programmers recruited through Upwork were randomised (45 with Copilot, 50 without) and asked to implement an HTTP server in JavaScript. Completion was measured automatically as the time from repository creation to the first commit passing all twelve tests in a suite the developers could read but not modify.
The treated group averaged 71.17 minutes; the control group averaged 160.89 minutes. That is the 55.8% figure, with a p-value of 0.0017 and a 95% confidence interval spanning a 21% to 89% improvement. It is a real result. It is also a greenfield, self-contained, test-verified task performed by contractors on an unfamiliar codebase of exactly zero lines, roughly the best possible conditions for autocomplete.
At enterprise scale: about 26% more completed tasks
The largest controlled evidence comes from three randomised trials run at Microsoft, Accenture and an unnamed Fortune 100 company, covering 4,867 developers. Zheyuan Cui, Mert Demirer, Sonia Jaffe, Leon Musolff, Sida Peng and Tobias Salz reported a 26.08% increase in completed tasks, with a standard error of 10.3%. That standard error is worth sitting with: the effect is statistically distinguishable from zero, but the plausible range is wide.
The heterogeneity finding is the more actionable one. Less experienced developers had both higher adoption rates and larger productivity gains. If your engineering organisation is mostly senior, the average effect in this study is not your effect.
Inside Google: about 21%, and not significant once controlled
A team at Google ran its own randomised trial with 96 full-time software engineers on a complex, enterprise-grade task. Their paper puts the best estimate of the effect at “about 21%” faster with AI features enabled, but it reports a 95% confidence interval of [-0.51, 0.03] and a p-value of 0.086 once known influences on time-on-task are controlled for. Read plainly: directionally positive, not statistically established at n=96.
The study that found a slowdown
In July 2025, METR published a randomised controlled trial that pointed the other way. Sixteen experienced open-source maintainers, averaging repositories with more than 22,000 GitHub stars, worked through 246 real issues from their own projects, with AI permitted on a randomly assigned half. Tooling was mostly Cursor Pro with Claude 3.5 or 3.7 Sonnet, frontier models at the time.
Developers were 19% slower when AI was allowed. The striking part is the perception gap: they had forecast a 24% speedup beforehand, and after living through the slowdown still believed AI had made them roughly 20% faster. METR was careful about what the result does not show. It is not evidence that AI fails to help most developers, or that it fails on unfamiliar codebases, or that learning effects beyond about 50 hours of tool use were captured.
METR published a design update in February 2026 that is arguably more informative than the original. Its follow-up run gave a -18% speedup for returning participants (confidence interval -38% to +9%) and -4% for newly recruited developers (-15% to +9%). More importantly, METR is abandoning task-level randomisation because it has become unmeasurable: developers now decline to participate rather than work without AI even at $50 an hour, they avoid submitting tasks where AI would obviously win, and agentic tools make self-reported time unreliable because people multitask while waiting. METR’s own read is that selection effects mean the true speedup could be higher than its numbers suggest.
Four studies, four designs
| Study | Design | Sample | Measured effect |
|---|---|---|---|
| Peng et al. (2023), GitHub/Microsoft | RCT, single greenfield task (JS HTTP server) | 95 contractors | 55.8% faster; 95% CI 21% to 89% |
| Cui et al., Microsoft Research (2025) | Three enterprise field RCTs | 4,867 developers | +26.08% completed tasks (SE 10.3%) |
| Paradis et al., Google (2024) | RCT, one complex enterprise task | 96 Google engineers | ~21% faster; p=0.086 with controls |
| METR (2025, updated 2026) | RCT on maintainers’ own real issues | 16 developers, 246 issues | 19% slower; follow-up -18% and -4% |
What the large surveys add
Surveys cannot establish causation, but they capture something the RCTs miss: what happens after the code is written. Google’s DORA programme surveyed roughly 5,000 technology professionals for its 2025 State of AI-assisted Software Development report, published in September 2025. Adoption reached 90%, up from 76% a year earlier, and more than 80% of respondents reported increased productivity. Trust fell: 70% said they trusted AI output somewhat or more, down from 87.9% in 2024. DORA’s 2024 edition had found delivery throughput dropping about 1.5% for every 25% increase in AI adoption; the 2025 edition found throughput had recovered while stability had not.
DORA’s follow-up ROI analysis, published in May 2026 and reported by InfoQ, models a 500-person engineering organisation investing $8.4m and generating $11.6m in value, a 39% first-year return with roughly an eight-month payback. Buried in that model are the two numbers a founder should actually read: productivity gains of 35% to 40% on simple tasks but 10% or less on complex legacy code, and a negative $344,000 line item from change failure rates rising from 5% to 6%.
Stack Overflow’s 2025 developer survey, published in December 2025 with more than 49,000 respondents, points at the mechanism. Eighty percent use AI tools, but trust in accuracy fell to 29% from 40%; 45% named “almost right, but not quite” answers as their main frustration; and 66% said they are spending more time fixing almost-correct AI-generated code. Favourable views of AI dropped from 72% to 60% year on year.
Telemetry from engineering-analytics vendors tells a consistent story, with the usual caveat that vendors have a commercial interest in the framing. Faros AI’s 2026 engineering report, drawn from 22,000 developers across more than 4,000 teams, found average pull request size up 51.3%, PR review time up 91% on high-adoption teams, and bugs per developer up 54%.
Why the numbers disagree
Three variables explain nearly all the spread. Task novelty: AI does best where the code is new and the requirements are explicit, worst where the constraint is a decade of undocumented context in a maintainer’s head. Developer expertise: the enterprise RCT found juniors gained most, and the study that found a slowdown used maintainers with unusually deep knowledge of their own codebases. And what you count: throughput measured at the commit is not throughput measured at the incident review, which is precisely where DORA and Faros locate the losses.
METR’s own framing of the perception gap: developers forecast a 24% speedup, experienced a 19% slowdown, and afterwards still estimated they had been sped up by 20%.
How to get a real number for your own team
- Pick one metric that survives the whole pipeline: cycle time from first commit to production, not commits or lines of code.
- Track change failure rate and review time alongside it. Every dataset above that measured downstream effects found them, and they are where the ROI leaks.
- Segment by seniority and by codebase age. A single organisation-wide average will hide the effect the evidence says is real.
- Do not use self-reported speedup as your evidence. It was wrong by roughly 39 percentage points in the METR sample.
- Compare against your own baseline period, not against a published percentage from a study whose task and population look nothing like yours.
The defensible position as of 2026 is narrow but useful: AI assistants reliably speed up well-specified work on unfamiliar or new code, especially for less experienced developers, and reliably add review and rework load that shows up two steps downstream. Whether that nets out positive is an empirical question about your codebase, and it is one you can answer yourself in a quarter.
Sources
- arXiv — The Impact of AI on Developer Productivity: Evidence from GitHub Copilot (Peng et al., 2023)
- Microsoft Research — The Effects of Generative AI on High-Skilled Work: Evidence from Three Field Experiments
- METR — Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity
- METR — We are Changing our Developer Productivity Experiment Design
- arXiv — How much does AI impact development speed? An enterprise-based randomized controlled trial (Google)
- DORA — State of AI-assisted Software Development 2025
- Stack Overflow — 2025 Developer Survey results
- InfoQ — New DORA Report Claims Strong Engineering Foundations Drive AI Return on Investment
Post Comment