Story
Why SWE-bench Verified no longer measures frontier coding capabilities
1 report
1 website
1 discussion
Latest report
Overview
SWE-bench Verified is becoming less reliable for measuring frontier coding capabilities due to contamination. SWE-bench Pro is recommended as a more accurate alternative for assessing models on autonomous software engineering tasks.
Summary from openai.com.
Reports
One report so far. Additional coverage will appear here when it is linked to this story.
Community reactions
Each reaction summarizes sampled comments from the linked discussion.
Hacker News · openai.com
Hacker News discussion ↗
Historical count: 5 · 5 saved in sample · Historical sample
Analysis ready
Read this discussion’s reaction
Based on 5 sampled comments from this thread. Analyzed 2026-04-26.
- Praise: The introduction of novel benchmarks like Zork bench aims to address the limitations of existing coding benchmarks by focusing on unique challenges that LLMs struggle to solve.
- Praise: The anticipation for ARC-AGI-3 highlights the community's interest in reasoning-heavy benchmarks, which could better evaluate model capabilities beyond simple tasks.
- Praise: The expectation of improvement in models like Claude Opus 4.6 suggests that ongoing advancements in AI will lead to better performance in challenging benchmarks.
- Concern: The revelation that a significant portion of SWE-bench Verified's test cases were flawed raises serious concerns about the validity of benchmarks used to measure AI capabilities.
- Concern: The community's skepticism about the reliability of benchmarks reflects a broader issue in AI evaluation, where many models may not accurately represent real-world performance.
- Concern: Concerns about the potential for benchmarks to be gamed or optimized for marketing purposes undermine trust in their effectiveness for assessing true AI capabilities.