Serving technology enthusiasts for more than 25 years. TechSpot is the place to go for tech advice and analysis you can trust.
Cutting Corners: A new study found that AI-powered coding tools may actually hinder productivity for seasoned software developers. This is contrary to the reason they use these tools, which was to increase it.
Model Evaluation & Threat Research, a non-profit organization, conducted researchto measure the impact of AI tools in software development. METR observed 16 open-source developers who were experienced in tackling 246 real programming tasks, ranging from bug fixing to new feature implementations, on large code repositories that they knew well. Each task was randomly assigned either to allow or prohibit the usage of AI coding software. Most participants chose Cursor Pro paired up with Claude 3.5, 3.7 Sonnet or 3.5 Sonnet when AI was allowed.
Before beginning, developers confidently predicted that AI would make them 24 percent faster. Even after the study concluded, they still believed their productivity had improved by 20 percent when using AI. The reality, however, was starkly different. The data showed that developers actually took 19 percent longer to finish tasks when using AI tools, a result that ran counter not only to their perceptions but also to the forecasts of experts in economics and machine learning.
Researchers investigated possible causes for this unexpected slowdown and identified several contributing factors. First, developers’ expectations of AI tools were often higher than the actual capabilities. AI was not able to provide any meaningful shortcuts because many participants were already familiar with their codebases. AI is challenged by the complexity and size of these projects, which can exceed a million lines. AI tends to be better at solving smaller, more contained issues. The AI’s suggestions were also inconsistent. Developers accepted less than 44% of the code generated by the AI, and spent significant time reviewing and fixing these outputs. AI tools also struggled to understand the implicit context of large repositories. This led to misunderstandings and inappropriate suggestions.
Methodology was rigorous. Each developer estimated the time it would take to complete a task with AI and without AI. They then worked through issues while recording their screen and self-reporting their time spent. Participants were paid $150 per hour in order to ensure their professional commitment. The results were consistent across different outcome measures and analyses. There was no evidence of bias or artifacts influencing the findings.
Researchers warn that these results shouldn’t be generalized. The study was based on highly-skilled developers who worked on familiar, complex codebases. AI tools could still be more beneficial to less experienced programmers, or those who work on smaller or unfamiliar projects. The authors acknowledge that AI technology is rapidly evolving, and future iterations may yield different results.
Many participants and researchers still use AI coding software despite the slowdown. They note that while AI may or may not speed up the development process, it can make certain parts of it easier to mentally handle, making coding a more iterative task.
