Abstract:

This research presents a randomized controlled trial (RCT) measuring the productivity impact of early-2025 AI tools (primarily Cursor Pro with Claude 3.5/3.7 Sonnet) on experienced software developers. The study recruited 16 developers, each with years of experience on large, mature open-source repositories, to complete 246 real-world tasks (bug fixes, features, refactors). Each task was randomly assigned to either allow or disallow the use of AI tools. The core finding was that allowing AI assistance increased task completion time by 19%, making developers significantly slower. This result stands in stark contrast to the participants' own perceptions; developers forecasted a 24% speedup before the study and, even after experiencing the slowdown, estimated a 20% speedup. Experts in machine learning and economics also drastically overestimated the productivity gains. A factor analysis suggests the slowdown was driven by a combination of high developer familiarity with their codebases, the complexity of the repositories, low AI reliability, and developers' over-optimism about AI's usefulness. The study provides critical, real-world evidence that contradicts benchmark results and anecdotal reports, highlighting a major gap between the perceived and actual impact of current AI coding assistants.

AI Slows Down Experienced Developers: A Study and Community Discussion

• Core Finding: AI Makes Experienced Developers Slower A randomized controlled trial found that when experienced open-source developers were allowed to use frontier AI coding tools (early-2025 models), they took 19% longer to complete their tasks.

• Perception vs. Reality: A Striking Disconnect The most surprising result was the gap between belief and reality.

  • Developer Forecast: Before starting, developers predicted AI would make them 24% faster.
  • Developer Post-Hoc Belief: After completing the study, they still believed AI had made them 20% faster, despite the data showing the opposite.
  • Expert Forecasts: Experts in ML and economics also predicted a significant speedup of around 38%.

• Methodology: A Realistic Test The study recruited 16 highly experienced developers working on 246 real issues from their own large, high-quality open-source repositories (e.g., projects with 1M+ lines of code). Tasks were randomly assigned to "AI-allowed" (using tools like Cursor Pro with Claude 3.5/3.7 Sonnet) or "AI-disallowed" groups, with completion time as the primary outcome measure.

• Factor Analysis: Why the Slowdown? The study identified five key factors likely contributing to the negative result:

  1. Over-optimism: Developers used AI tools even when they were not helping, likely because they incorrectly believed the tools were making them more productive.
  2. High Developer Familiarity: AI provides less benefit when a developer is already an expert on their own codebase. The study showed developers were slowed down more on tasks they were highly familiar with.
  3. Large and Complex Repositories: The AI tools struggled with the context of large, mature codebases.
  4. Low AI Reliability: Developers accepted fewer than 44% of AI-generated suggestions and spent significant time (9% of their total time) reviewing and cleaning up AI code.
  5. Lack of Implicit Context: The AI could not grasp the unwritten rules, history, and tacit knowledge that experienced developers rely on.

• Hacker News Discussion: The "Steep Learning Curve" Theory A prominent theory in the community discussion is that using AI tools effectively has a high learning curve.

  • The study noted that 56% of participants had never used Cursor (the primary tool) before the study.
  • The one developer with over 50 hours of prior Cursor experience did see a productivity increase, suggesting that expertise with the tool itself is a critical factor.

• Hacker News Discussion: Is It User Error or Hype? The results sparked a debate on whether the slowdown is due to developers "not using it right" or if the technology is simply overhyped. Many pointed out that unlike other complex developer tools (like Vim or debuggers), AI assistants are marketed with extreme claims of revolutionizing productivity for everyone, setting up a clash between expectation and reality.

• Hacker News Discussion: Context is Key Many developers shared personal anecdotes aligning with the study's findings, noting that AI is most useful for:

  • Greenfield projects: Starting new projects from scratch.
  • Unfamiliar codebases/languages: Getting up to speed quickly.
  • Boilerplate and one-off scripts: Removing tedious work. However, for complex tasks in mature codebases where developers are already experts, AI often gets in the way.

• Key Takeaway: The Importance of Robust Measurement Beyond the headline result, the study's most significant contribution is demonstrating the unreliability of anecdotal reports and self-perception. The wide gap between how productive developers felt and how productive they were shows the critical need for objective, real-world experiments to accurately measure the impact of AI.