developer-tools ai-productivity open-source-development experiment-design

We are changing our developer productivity experiment design

metr.org

February 24, 2026

8 min read

🔥🔥🔥🔥🔥

48/100

Summary

METR's previous study indicated a 20% slowdown in task completion among experienced open-source developers using AI tools. A new experiment launched in August 2025 aims to assess AI's impact on developer productivity over time, but participant feedback suggests the data may be unreliable.

Key Takeaways

A previous study found that AI tools caused a 20% slowdown in task completion among experienced open-source developers from February to June 2025.
A new experiment starting in August 2025 faced significant selection effects, as many developers opted out due to a preference for working with AI, which likely biased productivity estimates downwards.
The early 2025 study showed tasks taking 19% longer with AI, while the new data suggests a potential speedup of -18% among original participants and -4% among newly-recruited developers, indicating that true productivity gains could be underestimated.
Wider adoption of AI tools has made it more challenging to measure developer productivity, as many developers are unwilling to work without AI and selectively choose tasks they believe will benefit from AI assistance.

Read original article

View story →

Coverage & discussions

Hacker News · metr.org

We are changing our developer productivity experiment design ↗

Community discussions

Hacker News discussion ↗

Historical count: 5 · 5 saved in sample · Historical sample

Analysis ready

Community Sentiment

Mixed

Positives

The new study indicates a significant shift in developer productivity, with findings showing a potential speedup of 20%, challenging previous notions of AI's limitations.
A notable percentage of developers (30-50%) express a strong preference for using AI, suggesting that AI tools are becoming integral to their workflow and productivity.
The observation that experienced developers show greater productivity gains with AI tools highlights the importance of familiarity and skill in leveraging AI effectively.

Concerns

Concerns arise from the potential for rate limits to impact profitability, which could hinder the adoption of AI tools in developer environments.
The implication that many developers are avoiding tasks without AI raises questions about the reliability of productivity metrics and the overall impact of AI dependency.