Story
DeepSWE: A contamination-free benchmark for long-horizon coding agents
1 report
1 website
1 discussion
Latest report
Overview
DeepSWE is a long-horizon software engineering benchmark designed to evaluate coding agents on original engineering tasks. It features contamination-free tasks, high diversity across 91 repositories in five programming languages, and real-world applicability.
Summary from deepswe.datacurve.ai.
Reports
One report so far. Additional coverage will appear here when it is linked to this story.
Community reactions
Each reaction summarizes sampled comments from the linked discussion.
Hacker News · deepswe.datacurve.ai
Hacker News discussion ↗
Historical count: 5 · 5 saved in sample · Historical sample
Analysis ready
Read this discussion’s reaction
Based on 5 sampled comments from this thread. Analyzed 2026-05-27.
- Praise: The normalization of the harness to mini-swe-agent is a significant improvement, allowing models to generalize better across different tools, which is crucial for practical applications.
- Praise: The initiative to create a contamination-free benchmark is commendable, as it aims to provide a clearer evaluation of long-horizon coding agents.
- Concern: The benchmark's 'contamination-free' label may only be valid for its initial release, as it still suffers from fundamental design issues common to benchmarks, such as a focus on single correct answers.
- Concern: The saturation of scores at launch suggests that the benchmark may not effectively differentiate between the capabilities of models like Claude and GPT-5.5, raising concerns about its utility.
- Concern: The lack of checks for code quality and maintainability in the verifier highlights a significant gap in evaluating the true effectiveness of coding models, which often struggle with code 'taste'.