The goal of artificial intelligence coding tools is to speed up software development. However, researchers tested these tools using a randomized controlled trial and found the opposite.
Model Evaluation & Threat Research, a non-profit group of researchers, has published a study that shows AI coding tools actually slow down software developers, despite expectations.
The use of AI tools not only hampered developers, but also caused them to hallucinate. This is similar to what AIs do. The developers expected a 24 percent increase in speed, but even after concluding the study, they still believed AI had helped complete tasks 20 percent quicker when it actually delayed their work. The study found that AI tooling actually slowed down developers
“After completing the study, developers estimate that allowing AI reduced completion time by 20 percent,” . “Surprisingly, we find that allowing AI actually increases completion time by 19 percent — AI tooling slowed developers down.”
This study involved 16 experienced developers working on large open source projects. The developers provided a real-world list of issues (e.g. bug fixes, new features, etc.) They then estimated the time it would take to complete each task (246 in total). The issues were randomly assigned allowing or disallowing AI tool usage.
The developers worked on their issues using their AI tools of choice (mainly Cursor with Claude 3.5/3.7 Sonnet), when allowed. The work took place between February and June of 2025.
According to the study, the slowdown is likely due to five factors:
- “Over-optimism about AI usefulness” [developers had unrealistic expectation]
- “High developer familiarity with repositories” [the devs had enough experience that AI help was of no use to them]
- “Large and complex repositories” [AI performs worse on large repositories with 1M+ code lines]
- “Low AI reliability” [devs accepted less 44 percent of generated ideas and then spent time reviewing and cleaning up]
- “Implicit repository context”
Other factors, such as AI generation latency or failure to provide models with the optimal context (input), may have had an impact on the results. However, the researchers are unsure how these factors affected the study. Researchers show that it is possible to defeat AI scrapers using technology.
- Datacenters are feeling the heat from climate change.
- EU attempts to explain how AI can be done without breaking the laws.
- Firebase Studio’s new Agent Mode wants you to code, so they don’t have too.
According to other researchers, AI doesn’t always live up the hype. Recent research by AI coding business Qodo revealed that some of the benefits AI software assistance provided were undermined by the need for additional work to verify AI code suggestions. A Danish economic survey concluded that generative AI had no impact on wages or jobs. Intel’s study concluded that AI PCs reduce productivity. Call center workers from a Chinese electric utility say that while AI assistance may speed up some tasks, it can also slow things down because it creates more work. One of the graphics in the study clearly shows the added work that comes with AI tool usage. The study explains “When AI is allowed, developers spend less time actively coding and searching for/reading information, and instead spend time prompting AI, waiting on and reviewing AI outputs, and idle,” .
Anecdotally speaking, many coders have found that AI tools are able to help test out new scenarios in a low-risk way, and automate some routine tasks. However, this does not save time because you still need to validate the code. AI tools can make programming incrementally fun, but not more efficient.
The author’s – Joel Becker Nate Rush Beth Barnes and David Rein – caution that their work is best viewed in a limited context, as an example of a snapshot taken in time using specific experimental tools and circumstances. They say
“The slowdown we observe does not imply that current AI tools do not often improve developer’s productivity – we find evidence that the high developer familiarity with repositories and the size and maturity of the repositories both contribute to the observed slowdown, and these factors do not apply in many software development settings,” .
They continue to note that the authors’ findings do not imply that current AI systems aren’t useful or that future AI model won’t perform better. (r)
