CodingSaturated
Trust Score:
25
SWE-bench Verified
A 500-task, engineer-reviewed subset of SWE-bench that became the standard repository-level coding-agent test.
Launched: —Refresh: static
Status Assessment (saturated):
Classified saturated on 2026-02-17: under the standardized mini-SWE-agent 2.0.0 protocol, the leading results clustered at 76.8%, 75.8%, 75.8%, and 75.6%, limiting discrimination among frontier systems; separate contamination and task-quality concerns remain documented below.
Performance Timeline
Longitudinal progression of model scores against human baselines.Performance & Historical Trajectory
Empirical score progression across model release dates and evaluation rounds.
Independent Vendor-Reported Human Baseline (78.4%)
| Model | Score | Date | Source Type | Provenance |
|---|---|---|---|---|
| DeepSeek V4-Flash | 57.4% | 2026-08-25 | independent | Source ↗ |
| DeepSeek V4-Pro | 61.6% | 2026-08-25 | independent | Source ↗ |
| Claude Fable 5.1 (Hybrid Reasoning) | 69.5% | 2026-08-24 | independent | Source ↗ |
| Gemini 3.7 Flash | 60% | 2026-08-22 | independent | Source ↗ |
| Grok 4.6 | 61% | 2026-08-21 | independent | Source ↗ |
| GPT-5 | 64.2% | 2026-08-16 | independent | Source ↗ |
| DeepSeek V4.1 Flash | 67.8% | 2026-08-10 | independent | Source ↗ |
| GPT-6 Astra (max) | 69% | 2026-08-06 | independent | Source ↗ |
| Kimi K3 | 61.9% | 2026-08-04 | independent | Source ↗ |
| Claude Sonnet 5 | 63.2% | 2026-08-02 | independent | Source ↗ |
| Claude Opus 5 | 64.6% | 2026-08-02 | independent | Source ↗ |
| Gemini 3.6 Flash | 59.1% | 2026-07-30 | independent | Source ↗ |
| Gemini 3.5 Flash-Lite | 56.2% | 2026-07-30 | independent | Source ↗ |
| GPT-5.6 Sol | 63.8% | 2026-07-18 | independent | Source ↗ |
| GPT-5.5 Thinking | 62.4% | 2026-07-18 | independent | Source ↗ |
| MiniMax M3 | 60.8% | 2026-07-04 | independent | Source ↗ |
| Qwen3-235B | 61.2% | 2026-06-27 | independent | Source ↗ |
| QwQ Plus | 60% | 2026-06-27 | independent | Source ↗ |
| Grok 4 | 63.8% | 2026-06-24 | independent | Source ↗ |
| GLM-5.2 | 60.4% | 2026-06-21 | independent | Source ↗ |
| Claude Fable 5 | 65.4% | 2026-06-18 | independent | Source ↗ |
| Gemini 3 Pro | 61.9% | 2026-05-27 | independent | Source ↗ |
| o3 | 66.9% | 2025-12-29 | independent | Source ↗ |
| Mistral Large 3 | 55.5% | 2025-11-21 | independent | Source ↗ |
| Gemma 3 27B Preview | 60% | 2025-04-21 | independent | Source ↗ |
| Llama 4 Scout | 58.9% | 2025-04-14 | independent | Source ↗ |
| Llama 4 Maverick | 60% | 2025-04-14 | independent | Source ↗ |
| Gemini 2.5 Pro | 65.7% | 2025-04-03 | independent | Source ↗ |
| QwQ 32B | 56.8% | 2025-03-14 | independent | Source ↗ |
| GPT-4.5 | 53.6% | 2025-03-08 | independent | Source ↗ |
| Claude 3.7 Sonnet | 67.3% | 2025-03-05 | independent | Source ↗ |
| Grok 3 | 59.7% | 2025-02-26 | independent | Source ↗ |
| Gemini 2.5 Flash (Thinking) | 62.7% | 2025-02-19 | independent | Source ↗ |
| Gemini 2.0 Flash | 52% | 2025-02-14 | independent | Source ↗ |
| Gemini 2.0 Pro | 55.9% | 2025-02-14 | independent | Source ↗ |
| Sonar Reasoning Pro | 64.2% | 2025-02-14 | independent | Source ↗ |
| o3-mini | 63.8% | 2025-02-09 | independent | Source ↗ |
| Qwen 2.5 Max | 54.3% | 2025-02-06 | independent | Source ↗ |
| Ollama DeepSeek-R1 (Q4_K_M) | 64.6% | 2025-01-31 | independent | Source ↗ |
| Kimi k1.5 | 54% | 2025-01-29 | independent | Source ↗ |
| DeepSeek-R1 | 65.4% | 2025-01-29 | independent | Source ↗ |
| Gemini 2.0 Flash Thinking | 57.2% | 2025-01-29 | independent | Source ↗ |
| MiniMax-Text-01 | 54% | 2025-01-24 | independent | Source ↗ |
| Codestral 25.01 | 54.7% | 2025-01-23 | independent | Source ↗ |
| DeepSeek-V3 | 52.6% | 2025-01-04 | independent | Source ↗ |
| Phi-4 | 47.1% | 2024-12-21 | independent | Source ↗ |
| Sonar Pro | 59.7% | 2024-12-19 | independent | Source ↗ |
| Ollama Llama 3.3 (Q4_K_M) | 58.7% | 2024-12-17 | independent | Source ↗ |
| Llama 3.3 70B | 49.5% | 2024-12-15 | independent | Source ↗ |
| Llama 3.3 70B Instruct | 59.3% | 2024-12-15 | independent | Source ↗ |
| o1 | 56.2% | 2024-12-14 | independent | Source ↗ |
| Amazon Nova Pro | 54.7% | 2024-12-12 | independent | Source ↗ |
| Amazon Nova Lite | 49.4% | 2024-12-12 | independent | Source ↗ |
| QwQ-32B Preview | 60.8% | 2024-12-07 | independent | Source ↗ |
| GPT-4o | 50.5% | 2024-11-29 | independent | Source ↗ |
| Sonar | 56.2% | 2024-11-24 | independent | Source ↗ |
| Qwen 2.5 Coder 32B | 56.2% | 2024-11-21 | independent | Source ↗ |
| Hunyuan-Large | 55.5% | 2024-11-14 | independent | Source ↗ |
| Claude 3.5 Sonnet | 55.3% | 2024-10-31 | independent | Source ↗ |
| Claude 3.5 Haiku | 48.6% | 2024-10-31 | independent | Source ↗ |
| Claude 3.5 Sonnet (v2) | 60.2% | 2024-10-31 | independent | Source ↗ |
| Yi-Lightning | 55.1% | 2024-10-24 | independent | Source ↗ |
| Claude 3.5 Sonnet (upgraded; Anthropic tools scaffold) | 49% | 2024-10-22 | vendor-reported | Source ↗ |
| Qwen 2.5 72B Instruct | 61.6% | 2024-09-28 | independent | Source ↗ |
| GLM-4-Plus | 53.6% | 2024-08-29 | independent | Source ↗ |
| GPT-4o (Agentless) | 33.2% | 2024-08-13 | vendor-reported | Source ↗ |
| Gemma 2 2B | 39.5% | 2024-08-09 | independent | Source ↗ |
| Mistral Large 2 | 57% | 2024-08-02 | independent | Source ↗ |
| Mistral Large 2 | 49% | 2024-08-02 | independent | Source ↗ |
| Llama 3.1 405B | 50.9% | 2024-08-01 | independent | Source ↗ |
| GPT-4o mini | 44.1% | 2024-07-27 | independent | Source ↗ |
| ERNIE 4.0 Turbo | 52.4% | 2024-07-07 | independent | Source ↗ |
| Gemma 2 27B | 54.7% | 2024-07-06 | independent | Source ↗ |
| Gemma 2 9B | 49.4% | 2024-07-06 | independent | Source ↗ |
| GPT-5-2 (high reasoning; mini-SWE-agent 2.0.0) | 72.8% | 2024-06-01 | independent | Source ↗ |
| GPT-5 Mini (mini-SWE-agent 2.0.0) | 56.2% | 2024-06-01 | independent | Source ↗ |
| Claude 4.5 Opus (high reasoning; mini-SWE-agent 2.0.0) | 76.8% | 2024-06-01 | independent | Source ↗ |
| Gemini 3 Flash (high reasoning; mini-SWE-agent 2.0.0) | 75.8% | 2024-06-01 | independent | Source ↗ |
| MiniMax M2.5 (high reasoning; mini-SWE-agent 2.0.0) | 75.8% | 2024-06-01 | independent | Source ↗ |
| Claude Opus 4.6 (mini-SWE-agent 2.0.0) | 75.6% | 2024-06-01 | independent | Source ↗ |
| GPT-5-2 Codex (mini-SWE-agent 2.0.0) | 72.8% | 2024-06-01 | independent | Source ↗ |
| GLM-5 (high reasoning; mini-SWE-agent 2.0.0) | 72.8% | 2024-06-01 | independent | Source ↗ |
| Claude 4.5 Sonnet (high reasoning; mini-SWE-agent 2.0.0) | 71.4% | 2024-06-01 | independent | Source ↗ |
| Kimi K2.5 (high reasoning; mini-SWE-agent 2.0.0) | 70.8% | 2024-06-01 | independent | Source ↗ |
| DeepSeek V3.2 (high reasoning; mini-SWE-agent 2.0.0) | 70% | 2024-06-01 | independent | Source ↗ |
| Gemini 3 Pro (mini-SWE-agent 2.0.0) | 69.6% | 2024-06-01 | independent | Source ↗ |
| Claude 4.5 Haiku (high reasoning; mini-SWE-agent 2.0.0) | 66.6% | 2024-06-01 | independent | Source ↗ |
| Snowflake Arctic | 50.9% | 2024-05-03 | independent | Source ↗ |
| DBRX Instruct | 51.7% | 2024-04-05 | independent | Source ↗ |
Human Baseline & Difficulty Horizon
Calibrated human reference points, specialist benchmarks, and ceiling thresholds.Measured Human Score78.4%Domain Expert Baseline
Baseline Protocol & Interpretation
Senior software engineers solving unit-tested GitHub pull request issues without prior codebase familiarity.
Metric & Scoring Methodology
Verification protocols, aggregation formulas, and specialized metric variants.Primary Metric:
resolved instances (%)Scoring Engine:unit-tests
Dataset & Compute Cost
Evaluation volume, public availability, API pricing, and local hardware requirements.Total Dataset Size500Annotated evaluation items
Public Test SetPublicOpenly mirrored on repositories
Access GatingOpen AccessUnrestricted download
Evaluation LicenseOpen AccessDataset usage and redistribution terms
Frontier API Compute Cost:
$5 – $20 USD for full benchmark evaluation run on frontier APIs.
Recommended Local GPU Setup:
1x NVIDIA RTX 4090 (24GB) or A100 (40GB/80GB) via vLLM / SGLang
Official Dataset & Benchmark Files:Download / View Dataset Repository ↗
How to Run & Reproduce
Standardized evaluation protocols, CLI commands, and reproducible runner templates.Prompt Regimezero-shot
Reasoning Modedirect
Sampling Temp0
Pass@k Budgetk = 1
Tools & SandboxPure Text
Scoring Verifierunit-tests
Option AEleutherAI LM-Evaluation-Harness (Open-Weight Models)
lm_eval --model hf --model_args pretrained=<model_path> --tasks swe-bench-verified --batch_size autoOption BOpenCompass Evaluation Framework
opencompass --datasets swe-bench-verified --models <model_config>Python APIDeterministic Inference Loop Snippet
# Standard API Evaluation Loop
from openai import OpenAI
client = OpenAI()
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": prompt}],
temperature=0.0,
)Standardized Reporting Requirement:
When publishing results for SWE-bench Verified, always report the exact prompt template, few-shot exemplar ordering, sampling temperature (temperature=0), maximum reasoning budget tokens, and the precise timestamped model snapshot ID.
Contamination & Memorization Analysis
Audit of pretraining exposure risks, memorization vectors, and refresh policies.Overall Contamination Risk:HIGH
Refresh Cadence:
Static fixed snapshot
Test Set Exposure:
Public on web / HuggingFace