We are changing our developer productivity experiment design
metr.org
February 24, 2026
8 min read
Summary
METR's previous study indicated a 20% slowdown in task completion among experienced open-source developers using AI tools. A new experiment launched in August 2025 aims to assess AI's impact on developer productivity over time, but participant feedback suggests the data may be unreliable.
Key Takeaways
- A previous study found that AI tools caused a 20% slowdown in task completion among experienced open-source developers from February to June 2025.
- A new experiment starting in August 2025 faced significant selection effects, as many developers opted out due to a preference for working with AI, which likely biased productivity estimates downwards.
- The early 2025 study showed tasks taking 19% longer with AI, while the new data suggests a potential speedup of -18% among original participants and -4% among newly-recruited developers, indicating that true productivity gains could be underestimated.
- Wider adoption of AI tools has made it more challenging to measure developer productivity, as many developers are unwilling to work without AI and selectively choose tasks they believe will benefit from AI assistance.
Community Sentiment
MixedPositives
- The new study indicates a significant shift in developer productivity, with findings showing a potential speedup of 20%, challenging previous notions of AI's limitations.
- A notable percentage of developers (30-50%) express a strong preference for using AI, suggesting that AI tools are becoming integral to their workflow and productivity.
- The observation that experienced developers show greater productivity gains with AI tools highlights the importance of familiarity and skill in leveraging AI effectively.
Concerns
- Concerns arise from the potential for rate limits to impact profitability, which could hinder the adoption of AI tools in developer environments.
- The implication that many developers are avoiding tasks without AI raises questions about the reliability of productivity metrics and the overall impact of AI dependency.
Related Articles
How AI assistance impacts the formation of coding skills
Jan 30, 2026
Labor market impacts of AI: A new measure and early evidence
Mar 5, 2026
Measuring AI agent autonomy in practice
Feb 19, 2026
Anthropic Education the AI Fluency Index
Feb 23, 2026
Coding assistants are solving the wrong problem
Feb 3, 2026
Source
metr.org
Published
February 24, 2026
Reading Time
8 minutes
Relevance Score
48/100
Why It Matters
This page is optimized for focused reading: quick context up top, a clean summary block, and a direct path to the original source when you want the full story.