Stack Overflow data reveals hidden productivity tax for ‘almost-right’ AI code

“One of the most surprising findings was a significant shift in developer preferences for AI compared to previous years, while most developers use AI, they like it less and trust it less this year,” Erin Yepis, Senior Analyst for Market Research and Insights at Stack Overflow, told VentureBeat. This is surprising, because I would have expected that with all the investment and attention on AI in the tech news the trust would increase as technology improves.

These numbers tell the whole story. Only 33% developers will trust AI accuracy by 2025. This is down from 43% in both 2024 and 2023. AI favorability has dropped from 77% to 72% to 60% this year.

However, the survey data shows that technical decision-makers are more concerned. Developers are most frustrated by “AI solutions which are almost perfect but not quite”. 66% of developers report this problem. Debugging AI generated code takes longer than expected, according to 45% of developers. AI tools promise productivity increases but may create new categories of debt.

AI tools do not just produce obvious broken code. They generate plausible solutions, but require significant developer input to become production-ready. This creates an especially insidious problem with productivity.

Yepis explained that while AI tools promise to save time and increase productivity, developers spend time fixing the unintended workflow breakdowns caused by AI. “Most developers claim AI tools don’t address complexity. Only 29% of developers believed AI tools could solve complex problems this year – down from 35% in 2017.” Developers need to know what’s wrong, and how to fix it. Many say it would be quicker to write the code completely from scratch than debug and fix AI-generated solutions.

Workflow disruption goes beyond individual coding tasks. The survey revealed that 54% of developers used six or more tools in order to complete their tasks. This adds complexity to an already complex process by adding context-switching overhead.

Enterprise governance frameworks lag behind adoption

Rapid AI adopters have outpaced enterprise governance abilities. Security and technical debt are now potential risks that organizations haven’t fully addressed.

Ben Matthews, Senior director of engineering at Stack Overflow told VentureBeat that “Vibe coding” requires a high level of trust in AI’s output and sacrifices security and confidence in the code in order to achieve a faster turnaround.

Developers reject vibe coding in professional work. 77% of respondents said that it is not part their professional development. The survey also reveals that enterprises have gaps in their management of AI-generated code.

Matthews warns AI coding tools powered with LLMs are prone to mistakes. He said that while developers with the right knowledge can identify and test vulnerable codes, LLMs may not be able to detect any mistakes.

The security risks compound these issues. The survey data shows that 61.7% of developers still seek human help for coding. They cite “ethical and security concerns about code” in their reasoning. This suggests that AI tools present integration challenges in terms of data access, performance, and security. Organizations are still learning how to manage these.

Despite a decline in trust, developers don’t abandon AI tools. They’re developing sophisticated strategies to integrate them into workflows. In the last year, 69% of developers have spent time learning new coding languages or techniques. In 2024, 37% of developers used AI-enabled learning tools. Now, 44% use AI-enabled learning tools.

Despite the rise of AI and vibe coding, the survey data shows developers still maintain strong connections with human expertise and community resources. Stack Overflow is the most popular community platform, with 84% usage. GitHub is second at 67%, followed by YouTube at 61%. The most telling statistic is that 89% of developers use Stack Overflow more than once per month. 35% of these developers turn to Stack Overflow after experiencing issues with AI responses.

Jody Bailey, Chief Technology & Product Officer, told VentureBeat that, “although we have seen a decrease in traffic, it is not as dramatic as some might suggest.”

Bailey acknowledged that the needs of users have changed since Stack Overflow was founded 16 years ago. He said that not a single company or site is not experiencing a change in the way users interact with gen AI tools or where they come from. This shift is forcing Stack Overflow, a popular digital-age community site, to reassess its methods of measuring success.

Bailey said, “The future vitality and success of the internet will no longer be defined solely by metrics of success that were outlined in 1990s or early 2000s.” “Instead, emphasis is placed on the quality of data, reliability of information, as well as the vital role played by expert communities and individuals who meticulously create, share and curate knowledge. “

Stack Overflow data suggests a few key considerations for enterprise teams evaluating AI tools.

Investing in debugging capabilities and code reviews: With 45% developers reporting increased debugging times for AI code, organisations need stronger code reviewing processes. They need debugging software designed specifically for AI-generated solutions.

Maintain pipelines of human expertise: Continued reliance upon community platforms and human consulting shows that AI tools amplify rather than replace the needs for experienced developers. These experts can identify AI-generated code errors and correct them.

Implement a staged AI adoption: Successful AI adoption requires careful integrating with existing tools and process rather than wholesale replacing of development workflows. This allows developers the opportunity to maximize AI strengths while minimizing “almost right solution” risks.

Focusing on AI tool literacy.Developers who use AI tools daily are 88% more favorable than weekly users. This indicates that proper training and integration strategy have a significant impact on outcomes. This data shows that for enterprises looking to lead the way in AI driven development, competitive advantage will not come from AI adoption speed but from developing superior abilities in AI-human work flow integration and AI generated code quality management.

Organizations who solve the “almost-right” problem will gain significant advantages when it comes to development speed and code.

Daily insights into business use cases from VB Daily

Want to impress your boss? VB Daily can help. We provide you with the inside scoop about what companies are doing to leverage generative AI. From regulatory shifts and practical deployments, we give you the information you need to maximize your ROI.

Read our privacy policy

Thank you for subscribing. Click here to view more VB Newsletters.

An error occured.