Software Engineering & Agentic Coding
Modern code benchmarks evaluate agentic code repair within realistic multi-file GitHub repositories, terminal command execution, and deterministic unit-test pass rates.
Why Classic Benchmarks Failed
HumanEval and MBPP consist of trivial single-function Python prompts that exist in hundreds of thousands of public training files, saturating over 90% and failing to predict real-world developer utility.
The TrustTheBench & LiveBench Approach
SWE-bench Verified and LiveCodeBench evaluate real pull requests and newly released LeetCode/Codeforces problems with hidden test suites.
Verified Category Leaderboard
Ranked strictly by verified Software Engineering & Agentic Coding performance.
| Rank | Model | Research Lab | Access Type | Category Score | Speed | Pricing / 1M |
|---|---|---|---|---|---|---|
| #1 | Claude Fable 5.1 | Anthropic | proprietary | 86.4% | 54 tok/s | $5.00 in |
| #2 | Claude Fable 5 | Anthropic | proprietary | 86.0% | 45 tok/s | $10.00 in |
| #3 | GPT-6 Astra | OpenAI | proprietary | 85.9% | 62 tok/s | $4.50 in |
| #4 | DeepSeek V4.1 Flash | DeepSeek | open-weight | 85.1% | 96 tok/s | $0.18 in |
| #5 | GPT-5 | OpenAI | proprietary | 84.5% | 85 tok/s | $2.50 in |
| #6 | GPT-5.6 Sol | OpenAI | proprietary | 84.1% | 62 tok/s | $5.00 in |
| #7 | Grok 4 | xAI | proprietary | 84.0% | 75 tok/s | $5.00 in |
| #8 | o3 | OpenAI | proprietary | 84.0% | 68 tok/s | $4.00 in |
| #9 | Kimi K3 | Moonshot AI | open-weight | 83.5% | 58 tok/s | $2.80 in |
| #10 | Claude Sonnet 5 | Anthropic | proprietary | 83.2% | 95 tok/s | $3.00 in |
| #11 | Claude 3.7 Sonnet | Anthropic | proprietary | 83.2% | 72 tok/s | $3.00 in |
| #12 | Grok 3 | xAI | proprietary | 82.5% | 80 tok/s | $3.00 in |
| #13 | Qwen: Qwen3.8 Max Prime | Alibaba Cloud / Qwen | open-weight | 82.4% | 55 tok/s | $4.00 in |
| #14 | Claude Opus 5.5 | Anthropic | proprietary | 82.4% | 55 tok/s | $4.00 in |
| #15 | GPT-6 Astra Pro | OpenAI | proprietary | 82.4% | 55 tok/s | $10.00 in |
Deployment Recommendations
Claude Fable 5 (86.0%) & GPT-5.6 Sol (83.9%)
Maximum reasoning depth and lowest error rate on difficult boundary problems.
Kimi K3 (81.4%) & DeepSeek V4-Pro (81.0%)
Host locally or via cost-effective cloud providers with full data privacy.
DeepSeek V4-Flash ($0.14/1M) & Qwen 2.5 Coder 32B
Optimal price-to-performance ratio for high-volume automated agent pipelines.