You still have 1 article to read before you register for a LeadDev.com free account.
Your inbox has been upgraded. Get weekly engineering insights to elevate your leadership approach.
Estimated time of reading: 2
Software engineers believe that AI unlocks more pace in their works. A new study suggests these perceptions of productivity increases might be overblown.
Researchers at the Model Evaluation & Threat Research organization (METR) randomly assigned 246 issues to contributors of large, mature GitHub project with the tags “AI allowed” and “AI disallowed”.
Participants were allowed to use the Cursor Pro IDE, and Anthropic’s Claude 3.5/3.7 model, as well as log their screen activity and time, for approximately two months between February 2025 and June 2025.
The 16 developers were asked, before being given a task to complete the task, to estimate how much they would save by using AI to complete the task. They thought they would be able to work 24% faster, but in reality it took them 19% more time than without AI assistance.
“Our main motivation for doing this study was to determine the methods we would use in order to understand if certain software developers were being sped-up by AI,” explains Nate Rush, METR and one of co-authors of paper. “We thought that we would get a very obvious outcome: a speed-up of 20%, 50%, or even two times. But, of course, this is not what we found.”
More like this
Unbelievable results
Both AI boosters and skeptics used the study to support their respective views. For those who are bullish on AI the study is so implausible it must be incorrect. For AI skeptics it’s proof that the hype does not match the reality.
Steve Newman is the co-founder and former CEO of Writely (which became Google Docs). They admitted that they thought the results “were too bad to be true”but then looked into the study’s methodology and came to the conclusion that its findings were legitimate. Rush says that the response to the paper has shown how people are hungry for information about AI’s impact in the real world.
Simon Willison is the co-creator and creator of the Django Web Framework. He says, “It feels like very credible research.” Regular user of AI coding toolHe acknowledges that the study is a small sample, but he still believes it can provide developers with useful information on how to adopt AI. “I believe that the absolute truth is the fact that people are terrible at estimating productivity performance,” says he.
The AI productivity myth has some real data to back it up, says Milan Milanovic. He is a Chief Technology Officer at 3MD and has more than 20 years of experience in different industries. Milanovic says the findings highlight a conundrum that more experienced developers might encounter – that they know their codebases better than AI could. He says that these developers “knew the codebases inside and out, working with million-line repositories accumulated over years of complexity.” “In this environment, AI is a liability and not an asset.”
What can developers learn about the paper?
Researchers are keen to point out that the results do not conclude that AI assistance hinders developers’ productivity. Rush says that “we would definitely discourage anyone from interpreting the results as: AI slows developers”. The authors suggest that it could be used as a tool to better inform people on how to use AI.
A team of researchers examined 20 possible explanations to explain the gap between perceptions and reality. The team found that an over-optimistic belief in AI, the sheer volume and peculiarities of mature repositories and the need to correct unreliable models suggestions were the most common. The agendas for LeadDev New York events are now available!
But Joel Becker, co-author of the study, also admits that their findings may only be valid for the AI models the researchers tested. Becker says, “It is possible and many people believe that AI models and tools have advanced very quickly since then. This could mean that developers in this environment today, let alone the near future, may be sped-up.” It has either highlighted the need for rigorous measurement and research. “It would be great if this inspired other well-funded organizations to do the same level of meticulous research,” he says.
