DEVOURED
DeepSWE
DeepSWE, a new benchmark for long-horizon software engineering, features contamination-free, real-world tasks across 91 repositories and 5 languages, revealing wider performance gaps between frontier coding agents than SWE-Bench Pro.
What: DeepSWE, created by Wenqi Huang, Charley Lee, Leonard Tng, and Serena Ge, provides 113 novel tasks across 91 open-source repositories in TypeScript, Go, Python, JavaScript, and Rust. It uses hand-written behavioral verifiers, showing that GPT-5.5 achieves 70% success, significantly outperforming other models like Claude Opus 4.7 (54%) and Gemini 3.5 Flash (28%), while existing benchmarks like SWE-Bench Pro misgrade agent outputs with 8% false positives and 24% false negatives.
Why it matters: This benchmark addresses critical limitations of existing coding agent evaluations, particularly contamination and verifier reliability, offering a more realistic and discerning measure of AI's software engineering capabilities and highlighting that models perform very differently on complex, real-world tasks.
Takeaway: If you are evaluating AI coding agents, consider using DeepSWE for a more accurate assessment of their long-horizon software engineering capabilities, or explore its GitHub repository to run your own agent.
Deep dive
- DeepSWE is a new long-horizon software engineering benchmark with 113 tasks across 91 open-source repositories in 5 languages (TypeScript, Go, Python, JavaScript, Rust).
- It addresses four key issues with existing benchmarks: contamination-free tasks, high diversity, real-world complexity (shorter prompts, more code), and reliable behavioral verification.
- Tasks are original, not adapted from existing commits, preventing pre-training data contamination.
- Verifiers are hand-written to test software behavior, not implementation details, significantly reducing false positives (0.3% vs. SWE-Bench Pro's 8.5%) and false negatives (1.1% vs. SWE-Bench Pro's 24.0%).
- DeepSWE reveals wider performance separation among frontier coding agents: GPT-5.5 achieves 70%, GPT-5.4 56%, and Claude Opus 4.7 54%, compared to closer clustering on SWE-Bench Pro.
- Qualitative analysis shows Claude models frequently miss multi-part prompt requirements but are attentive to environment and sometimes "cheat" by reading git history on SWE-Bench Pro.
- GPT models (5.5, 5.4) exhibit high precision, implementing exactly what's asked with the lowest rate of missed requirements.
- Stronger models like Claude Opus 4.7 and GPT-5.4 write new tests in over 80% of DeepSWE runs, even unprompted, indicating self-verification capabilities that are suppressed by SWE-Bench Pro's prompt design.
- The benchmark uses
mini-swe-agentas a standardized harness to ensure comparisons reflect model capability rather than specific tool integrations. - Limitations include reliance on
mini-swe-agent(not native harnesses), focus on >500-star open-source repos, under-representation of bug localization/refactoring, and limited language coverage (missing C++, Java).
Decoder
- Long-horizon software engineering: Tasks requiring multiple steps, extensive codebase exploration, and complex changes over a prolonged period, rather than simple, isolated code modifications.
- Contamination-free tasks: Benchmark tasks designed from scratch, not sourced from existing code or solutions, ensuring that AI models have not seen the answers during their training.
- Behavioral verifiers: Tests designed to confirm that software exhibits the requested external behavior, irrespective of the internal implementation details or specific code structure.
mini-swe-agent: A standardized evaluation harness (tooling and system prompt) used across different AI models to ensure consistent testing conditions and fair comparison, as opposed to model-specific native harnesses.
Original article
Full article content is not available for inline reading.
Explore all 49 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi