MMLU-Redux
A manual error audit of MMLU: experts re-annotated thousands of questions and found ~6.5% are broken — the evidence base for MMLU's label-noise ceiling.
As a competition benchmark, MMLU-Redux is a corrected slice of MMLU: it inherits MMLU's contamination and frontier convergence, so it does not differentiate top models — saturated by CLAUDE.md's definition (usefulness as a model-differentiating eval). Its ENDURING value is different and non-competitive: it remains an actively-used forensic reference and audit methodology for measuring label noise. Status here reflects the eval role; the still-active reference role is documented in prose. saturated_date is set to its launch — it described an already-saturated parent from birth.
Performance Timeline
Longitudinal progression of model scores against human baselines.Performance & Historical Trajectory
Empirical score progression across model release dates and evaluation rounds.
| Model | Score | Date | Source Type | Provenance |
|---|---|---|---|---|
| MiMo-V2.6-Flash | 84.7% | 2026-09-12 | independent | Source ↗ |
| Claude Opus 5.5 | 81.1% | 2026-09-12 | independent | Source ↗ |
| Qwen 3.8 Max Prime | 76.4% | 2026-09-12 | independent | Source ↗ |
| Nemotron 3.5 Super VL | 79.4% | 2026-09-12 | independent | Source ↗ |
| Gemini 4 Argon | 86.7% | 2026-09-12 | independent | Source ↗ |
| Claude Haiku 5.5 | 79.4% | 2026-09-12 | independent | Source ↗ |
| Nano Banana 2.1 | 86.7% | 2026-09-12 | independent | Source ↗ |
| Mistral Large 4 | 83.3% | 2026-09-12 | independent | Source ↗ |
| GPT-6.1 Sol Pro | 86.7% | 2026-09-12 | independent | Source ↗ |
| GPT-6.1 Sol | 86.7% | 2026-09-12 | independent | Source ↗ |
| Claude Sonnet 5.5 | 86.7% | 2026-09-12 | independent | Source ↗ |
| Z.ai: GLM 5.3 Prime | 86.7% | 2026-09-12 | independent | Source ↗ |
| GPT-6 Luna Pro | 79.4% | 2026-09-12 | independent | Source ↗ |
| GPT-6 Luna | 79.4% | 2026-09-12 | independent | Source ↗ |
| GPT-6 Sol Pro | 86.7% | 2026-09-12 | independent | Source ↗ |
| GPT-6 Sol | 86.7% | 2026-09-12 | independent | Source ↗ |
| Claude Opus 5.5 | 89.7% | 2026-09-12 | independent | Source ↗ |
| Xiaomi: MiMo-V2.6-Pro-UltraSpeed | 86.7% | 2026-09-12 | independent | Source ↗ |
| Xiaomi: MiMo-V2.6-Flash | 79.4% | 2026-09-12 | independent | Source ↗ |
| Xiaomi: MiMo-V2.6-Pro | 79.4% | 2026-09-12 | independent | Source ↗ |
| Z.ai: GLM 5.3 FlashX | 83.3% | 2026-09-12 | independent | Source ↗ |
| DeepSeek V4.1 Flash | 79.4% | 2026-09-12 | independent | Source ↗ |
| GPT-6 Astra | 89.7% | 2026-09-12 | independent | Source ↗ |
| GPT-6 Astra Pro | 89.7% | 2026-09-12 | independent | Source ↗ |
| Claude Sonnet 5.5 | 92.6% | 2026-09-12 | independent | Source ↗ |
| Claude Haiku 5.5 | 87.2% | 2026-09-12 | independent | Source ↗ |
| GPT-6.1 Sol Pro | 94.6% | 2026-09-12 | independent | Source ↗ |
| GPT-6 Luna | 92.1% | 2026-09-12 | independent | Source ↗ |
| Gemini 4 Argon | 94.1% | 2026-09-12 | independent | Source ↗ |
| Qwen 3.8 Max Prime | 91.1% | 2026-09-12 | independent | Source ↗ |
| DeepSeek V4.1 Flash 1017 | 79.4% | 2026-09-12 | independent | Source ↗ |
| GPT-6.1 Terra Pro | 86.7% | 2026-09-12 | independent | Source ↗ |
| GPT-6.1 Terra | 86.7% | 2026-09-12 | independent | Source ↗ |
| MoonshotAI: Kimi K3.1 Pro | 86.7% | 2026-09-12 | independent | Source ↗ |
| MiniMax: MiniMax M3.1 Flash | 79.4% | 2026-09-12 | independent | Source ↗ |
| Muse Spark Open | 86.7% | 2026-09-12 | independent | Source ↗ |
| LiquidAI: LFM2.5-24B-A2B | 79.4% | 2026-09-12 | independent | Source ↗ |
| Muse Wonder | 89.7% | 2026-09-12 | independent | Source ↗ |
| Qwen: Qwen3.8 Max Prime | 89.7% | 2026-09-12 | independent | Source ↗ |
| Upstage: Solar Mini 4 | 75.5% | 2026-09-12 | independent | Source ↗ |
| SpaceXAI: Grok 4.7 | 86.7% | 2026-09-12 | independent | Source ↗ |
| Qwen: Qwen3.8 Omni Flash | 79.4% | 2026-09-12 | independent | Source ↗ |
| Gemini 3.8 Flash | 86.7% | 2026-09-11 | independent | Source ↗ |
| Gemini 3.8 Flash | 91.9% | 2026-09-11 | independent | Source ↗ |
| Qwen: Qwen3.8 Max (0902) | 86.7% | 2026-09-11 | independent | Source ↗ |
| Muse Spark 1.3 Contributor | 75.5% | 2026-09-11 | independent | Source ↗ |
| Muse Spark 1.3 | 86.7% | 2026-09-11 | independent | Source ↗ |
| Claude Fable 5.1 | 89.7% | 2026-09-09 | independent | Source ↗ |
| Hy4 | 90.6% | 2026-09-05 | independent | Source ↗ |
| Hy4 | 83.3% | 2026-09-05 | independent | Source ↗ |
| GLM 5.3 Flash | 91.7% | 2026-09-04 | independent | Source ↗ |
| Z.ai: GLM 5.3 Flash | 79.4% | 2026-09-04 | independent | Source ↗ |
| Qwen: Qwen3.8 Flash | 79.4% | 2026-09-04 | independent | Source ↗ |
| DeepSeek V4 Flash Vision Exp | 79.4% | 2026-08-30 | independent | Source ↗ |
| DeepSeek V4-Pro | 90.4% | 2026-08-25 | independent | Source ↗ |
| DeepSeek V4-Flash | 86.2% | 2026-08-25 | independent | Source ↗ |
| Z.ai: GLM 5.3 | 86.7% | 2026-08-25 | independent | Source ↗ |
| Claude Fable 5.1 | 92.9% | 2026-08-24 | independent | Source ↗ |
| Qwen: Qwen3.8 27B | 83.3% | 2026-08-23 | independent | Source ↗ |
| Gemini 3.7 Flash | 91.3% | 2026-08-22 | independent | Source ↗ |
| Gemini 3.7 Flash | 86.7% | 2026-08-22 | independent | Source ↗ |
| DeepSeek V4 Pro 0813 | 86.7% | 2026-08-22 | independent | Source ↗ |
| Grok 4.6 | 91.1% | 2026-08-21 | independent | Source ↗ |
| Qwen: Qwen3.8 2.4T A95B | 86.7% | 2026-08-21 | independent | Source ↗ |
| LiquidAI: LFM2.5-2.6B | 79.4% | 2026-08-20 | independent | Source ↗ |
| ByteDance Seed: Seed 2.1 Turbo | 83.3% | 2026-08-19 | independent | Source ↗ |
| SpaceXAI: Grok 4.6 | 86.7% | 2026-08-19 | independent | Source ↗ |
| Muse Glimmer 30B | 83.3% | 2026-08-19 | independent | Source ↗ |
| GPT-5 | 91.9% | 2026-08-16 | independent | Source ↗ |
| Nemotron 3.5 Lightning | 75.5% | 2026-08-16 | independent | Source ↗ |
| GPT-5 Image Mini | 83.3% | 2026-08-16 | independent | Source ↗ |
| GPT-5 Image | 86.7% | 2026-08-16 | independent | Source ↗ |
| Muse Spark 1.2 Contributor | 75.5% | 2026-08-14 | independent | Source ↗ |
| Muse Spark 1.2 | 86.7% | 2026-08-14 | independent | Source ↗ |
| DeepSeek V4.1 Flash | 91.6% | 2026-08-10 | independent | Source ↗ |
| Space Bunny Alpha | 89.4% | 2026-08-10 | independent | Source ↗ |
| DeepSeek V4 Flash 0731 | 87% | 2026-08-09 | independent | Source ↗ |
| DeepSeek V4 Flash 0731 | 83.3% | 2026-08-09 | independent | Source ↗ |
| ByteDance Seed: Seed-2.0-Code | 83.3% | 2026-08-08 | independent | Source ↗ |
| GPT-6 Astra | 92.3% | 2026-08-06 | independent | Source ↗ |
| Qwen: Qwen3.7 Flash | 75.5% | 2026-08-05 | independent | Source ↗ |
| Kimi K3 | 91.6% | 2026-08-04 | independent | Source ↗ |
| Claude Opus 5 | 92.9% | 2026-08-02 | independent | Source ↗ |
| Claude Sonnet 5 | 90.9% | 2026-08-02 | independent | Source ↗ |
| Claude Opus 5 | 89.7% | 2026-08-01 | independent | Source ↗ |
| Gemini 3.6 Flash | 90% | 2026-07-30 | independent | Source ↗ |
| Gemini 3.5 Flash-Lite | 85.8% | 2026-07-30 | independent | Source ↗ |
| Gemini 3.6 Flash | 86.7% | 2026-07-30 | independent | Source ↗ |
| Gemini 3.5 Flash Lite | 83.3% | 2026-07-30 | independent | Source ↗ |
| MoonshotAI: Kimi K3 | 89.7% | 2026-07-24 | independent | Source ↗ |
| GPT-5.6 Sol | 93.6% | 2026-07-18 | independent | Source ↗ |
| GPT-5.5 Thinking | 92.1% | 2026-07-18 | independent | Source ↗ |
| GPT-5.6 Luna | 88.2% | 2026-07-18 | independent | Source ↗ |
| GPT-5.6 Luna Pro | 83.3% | 2026-07-18 | independent | Source ↗ |
| GPT-5.6 Luna | 83.3% | 2026-07-18 | independent | Source ↗ |
| GPT-5.6 Terra Pro | 89.7% | 2026-07-18 | independent | Source ↗ |
| GPT-5.6 Terra | 89.7% | 2026-07-18 | independent | Source ↗ |
| GPT-5.6 Sol Pro | 89.7% | 2026-07-18 | independent | Source ↗ |
| GPT-5.6 Sol | 89.7% | 2026-07-18 | independent | Source ↗ |
| Muse Spark 1.1 | 86.7% | 2026-07-18 | independent | Source ↗ |
| SpaceXAI: Grok 4.5 | 86.7% | 2026-07-17 | independent | Source ↗ |
| Hy3 | 79.4% | 2026-07-15 | independent | Source ↗ |
| Claude Sonnet 5 | 86.7% | 2026-07-09 | independent | Source ↗ |
| Gemini 3.1 Flash Lite Image | 83.3% | 2026-07-09 | independent | Source ↗ |
| MiniMax M3 | 87.7% | 2026-07-04 | independent | Source ↗ |
| Qwen3-235B | 89.7% | 2026-06-27 | independent | Source ↗ |
| QwQ Plus | 89% | 2026-06-27 | independent | Source ↗ |
| Z.ai: GLM 5.2 | 83.3% | 2026-06-25 | independent | Source ↗ |
| Grok 4 | 91.6% | 2026-06-24 | independent | Source ↗ |
| GLM-5.2 | 88.7% | 2026-06-21 | independent | Source ↗ |
| MoonshotAI: Kimi K2.7 Code | 86.7% | 2026-06-21 | independent | Source ↗ |
| Claude Fable 5 | 94.1% | 2026-06-18 | independent | Source ↗ |
| Claude Fable 5 | 89.7% | 2026-06-18 | independent | Source ↗ |
| Nemotron 3 Ultra | 85.8% | 2026-06-13 | independent | Source ↗ |
| Nemotron 3.5 Content Safety | 75.5% | 2026-06-13 | independent | Source ↗ |
| Nemotron 3 Ultra | 83.3% | 2026-06-13 | independent | Source ↗ |
| Qwen: Qwen3.7 Plus | 83.3% | 2026-06-11 | independent | Source ↗ |
| MiniMax: MiniMax M3 | 83.3% | 2026-06-09 | independent | Source ↗ |
| Gemini 3.1 Flash Image | 83.3% | 2026-06-06 | independent | Source ↗ |
| Gemini 3 Pro Image (Nano Banana Pro) | 89.7% | 2026-06-06 | independent | Source ↗ |
| Claude Opus 4.8 | 89.7% | 2026-06-06 | independent | Source ↗ |
| Hy-MT2-1.8B | 75.5% | 2026-05-30 | independent | Source ↗ |
| Hy-MT2-30B-A3B | 79.4% | 2026-05-30 | independent | Source ↗ |
| Hy-MT2-7B | 79.4% | 2026-05-30 | independent | Source ↗ |
| Qwen: Qwen3.7 Max | 86.7% | 2026-05-29 | independent | Source ↗ |
| SpaceXAI: Grok Build 0.1 | 83.3% | 2026-05-29 | independent | Source ↗ |
| Gemini 3.5 Flash | 86.7% | 2026-05-28 | independent | Source ↗ |
| Gemini 3 Pro | 90.6% | 2026-05-27 | independent | Source ↗ |
| Gemini 3.1 Flash Lite | 83.3% | 2026-05-16 | independent | Source ↗ |
| GPT Chat Latest | 89.7% | 2026-05-14 | independent | Source ↗ |
| Command A+ | 83.3% | 2026-05-10 | independent | Source ↗ |
| Mistral Medium 3.5 | 86.7% | 2026-05-09 | independent | Source ↗ |
| SpaceXAI: Grok 4.3 | 83.3% | 2026-05-09 | independent | Source ↗ |
| Qwen: Qwen3.6 Flash | 83.3% | 2026-05-06 | independent | Source ↗ |
| GPT-5.5 Pro | 89.7% | 2026-05-02 | independent | Source ↗ |
| GPT-5.5 | 89.7% | 2026-05-02 | independent | Source ↗ |
| DeepSeek V4 Pro 0423 | 83.3% | 2026-05-02 | independent | Source ↗ |
| DeepSeek V4 Flash 0423 | 75.5% | 2026-05-02 | independent | Source ↗ |
| MiMo-V2.5 | 83.5% | 2026-05-01 | independent | Source ↗ |
| Xiaomi: MiMo-V2.5-Pro | 79.4% | 2026-05-01 | independent | Source ↗ |
| Xiaomi: MiMo-V2.5 | 79.4% | 2026-05-01 | independent | Source ↗ |
| Qwen: Qwen3.6 27B | 83.3% | 2026-05-01 | independent | Source ↗ |
| Hy3 | 79.4% | 2026-04-30 | independent | Source ↗ |
| GPT-5.4 Image 2 | 89.7% | 2026-04-30 | independent | Source ↗ |
| Qwen: Qwen3.5 Plus 2026-04-20 | 83.3% | 2026-04-29 | independent | Source ↗ |
| MoonshotAI: Kimi K2.6 | 86.7% | 2026-04-29 | independent | Source ↗ |
| Claude Opus 4.7 | 89.7% | 2026-04-25 | independent | Source ↗ |
| Qwen: Qwen3.6 35B A3B | 79.4% | 2026-04-24 | independent | Source ↗ |
| Z.ai: GLM 5.1 | 86.7% | 2026-04-15 | independent | Source ↗ |
| Gemma 4 26B A4B | 79.4% | 2026-04-12 | independent | Source ↗ |
| Gemma 4 31B | 79.4% | 2026-04-11 | independent | Source ↗ |
| Qwen: Qwen3.6 Plus | 83.3% | 2026-04-11 | independent | Source ↗ |
| Z.ai: GLM 5V Turbo | 86.7% | 2026-04-10 | independent | Source ↗ |
| Lyria 3 Pro | 75.5% | 2026-04-08 | independent | Source ↗ |
| Lyria 3 Clip | 75.5% | 2026-04-08 | independent | Source ↗ |
| Reka Edge | 75.5% | 2026-03-29 | independent | Source ↗ |
| MiniMax: MiniMax M2.7 | 83.3% | 2026-03-27 | independent | Source ↗ |
| GPT-5.4 Nano | 83.3% | 2026-03-26 | independent | Source ↗ |
| GPT-5.4 Mini | 86.7% | 2026-03-26 | independent | Source ↗ |
| Z.ai: GLM 5 Turbo | 86.7% | 2026-03-24 | independent | Source ↗ |
| Qwen: Qwen3.5-9B | 79.4% | 2026-03-19 | independent | Source ↗ |
| SpaceXAI: Grok 4.20 Multi-Agent | 83.3% | 2026-03-18 | independent | Source ↗ |
| SpaceXAI: Grok 4.20 | 83.3% | 2026-03-18 | independent | Source ↗ |
| ByteDance Seed: Seed-2.0-Lite | 83.3% | 2026-03-18 | independent | Source ↗ |
| GPT-5.4 Pro | 89.7% | 2026-03-14 | independent | Source ↗ |
| GPT-5.4 | 89.7% | 2026-03-14 | independent | Source ↗ |
| Gemini 3.1 Flash Lite | 83.3% | 2026-03-12 | independent | Source ↗ |
| Gemini 3.1 Flash Image | 83.3% | 2026-03-07 | independent | Source ↗ |
| GPT-5.3-Codex | 89.7% | 2026-03-05 | independent | Source ↗ |
| ByteDance Seed: Seed-2.0-Mini | 79.4% | 2026-03-05 | independent | Source ↗ |
| Qwen: Qwen3.5-35B-A3B | 79.4% | 2026-03-05 | independent | Source ↗ |
| Qwen: Qwen3.5-27B | 83.3% | 2026-03-05 | independent | Source ↗ |
| Qwen: Qwen3.5-122B-A10B | 83.3% | 2026-03-05 | independent | Source ↗ |
| Qwen: Qwen3.5-Flash | 79.4% | 2026-03-05 | independent | Source ↗ |
| Gemini 3.1 Pro Custom Tools | 89.7% | 2026-02-28 | independent | Source ↗ |
| Gemini 3.1 Pro | 89.7% | 2026-02-28 | independent | Source ↗ |
| Claude Sonnet 4.6 | 89.7% | 2026-02-26 | independent | Source ↗ |
| Qwen: Qwen3.5 Plus 2026-02-15 | 83.3% | 2026-02-25 | independent | Source ↗ |
| Qwen: Qwen3.5 397B A17B | 86.7% | 2026-02-25 | independent | Source ↗ |
| Z.ai: GLM 5 | 86.7% | 2026-02-20 | independent | Source ↗ |
| MiniMax: MiniMax M2.5 | 79.4% | 2026-02-20 | independent | Source ↗ |
| Claude Opus 4.6 | 89.7% | 2026-02-14 | independent | Source ↗ |
| MoonshotAI: Kimi K2.5 | 83.3% | 2026-02-05 | independent | Source ↗ |
| MiniMax: MiniMax M2-her | 83.3% | 2026-02-01 | independent | Source ↗ |
| Z.ai: GLM 4.7 Flash | 79.4% | 2026-01-28 | independent | Source ↗ |
| GPT-5.2-Codex | 89.7% | 2026-01-23 | independent | Source ↗ |
| MiniMax: MiniMax M2.1 | 83.3% | 2026-01-01 | independent | Source ↗ |
| Z.ai: GLM 4.7 | 83.3% | 2025-12-31 | independent | Source ↗ |
| o3 | 90.2% | 2025-12-29 | independent | Source ↗ |
| Gemini 3 Flash | 83.3% | 2025-12-26 | independent | Source ↗ |
| GPT-5.2 Chat | 89.7% | 2025-12-20 | independent | Source ↗ |
| GPT-5.2 Pro | 89.7% | 2025-12-20 | independent | Source ↗ |
| GPT-5.2 | 89.7% | 2025-12-20 | independent | Source ↗ |
| Z.ai: GLM 4.6V | 79.4% | 2025-12-17 | independent | Source ↗ |
| GPT-5.1-Codex-Max | 86.7% | 2025-12-13 | independent | Source ↗ |
| DeepSeek V3.2 | 79.4% | 2025-12-10 | independent | Source ↗ |
| Claude Opus 4.5 | 89.7% | 2025-12-03 | independent | Source ↗ |
| Gemini 3 Pro Image (Nano Banana Pro) | 89.7% | 2025-11-29 | independent | Source ↗ |
| GPT-5.1 | 86.7% | 2025-11-22 | independent | Source ↗ |
| GPT-5.1-Codex | 86.7% | 2025-11-22 | independent | Source ↗ |
| GPT-5.1-Codex-Mini | 83.3% | 2025-11-22 | independent | Source ↗ |
| Mistral Large 3 | 84.3% | 2025-11-21 | independent | Source ↗ |
| MoonshotAI: Kimi K2 Thinking | 83.3% | 2025-11-15 | independent | Source ↗ |
| MiniMax: MiniMax M2 | 83.3% | 2025-11-01 | independent | Source ↗ |
| Qwen: Qwen3 VL 8B Instruct | 79.4% | 2025-10-23 | independent | Source ↗ |
| GPT-5 Pro | 89.7% | 2025-10-15 | independent | Source ↗ |
| Qwen: Qwen3 VL 30B A3B Thinking | 79.4% | 2025-10-15 | independent | Source ↗ |
| Qwen: Qwen3 VL 30B A3B Instruct | 79.4% | 2025-10-15 | independent | Source ↗ |
| Claude Haiku 4.5 | 86.7% | 2025-10-10 | independent | Source ↗ |
| Claude Sonnet 4.5 | 89.7% | 2025-10-08 | independent | Source ↗ |
| Qwen: Qwen3 VL 235B A22B Instruct | 83.3% | 2025-10-02 | independent | Source ↗ |
| Qwen: Qwen3 Coder Flash | 79.4% | 2025-09-26 | independent | Source ↗ |
| Qwen: Qwen3 Next 80B A3B Instruct | 83.3% | 2025-09-20 | independent | Source ↗ |
| MoonshotAI: Kimi K2 0905 | 83.3% | 2025-09-13 | independent | Source ↗ |
| GPT-5 | 86.7% | 2025-08-16 | independent | Source ↗ |
| GPT-5 Mini | 83.3% | 2025-08-16 | independent | Source ↗ |
| GPT-5 Nano | 79.4% | 2025-08-16 | independent | Source ↗ |
| Claude 4.1 Opus | 89.7% | 2025-08-14 | independent | Source ↗ |
| Qwen: Qwen3 Coder 30B A3B Instruct | 79.4% | 2025-08-09 | independent | Source ↗ |
| Qwen: Qwen3 30B A3B Instruct 2507 | 79.4% | 2025-08-07 | independent | Source ↗ |
| Qwen: Qwen3 235B A22B Thinking 2507 | 86.7% | 2025-08-03 | independent | Source ↗ |
| Qwen: Qwen3 Coder 480B A35B | 79.4% | 2025-08-01 | independent | Source ↗ |
| ByteDance: UI-TARS 7B | 75.5% | 2025-07-31 | independent | Source ↗ |
| Qwen: Qwen3 235B A22B Instruct 2507 | 79.4% | 2025-07-30 | independent | Source ↗ |
| MoonshotAI: Kimi K2 0711 | 83.3% | 2025-07-20 | independent | Source ↗ |
| ByteDance Seed: Seed 1.6 Flash | 79.4% | 2025-07-04 | independent | Source ↗ |
| ByteDance Seed: Seed 1.6 | 83.3% | 2025-07-04 | independent | Source ↗ |
| MiniMax: MiniMax M1 | 83.3% | 2025-06-26 | independent | Source ↗ |
| Sarvam 105B | 86.7% | 2025-06-24 | independent | Source ↗ |
| o3 Pro | 89.7% | 2025-06-19 | independent | Source ↗ |
| Sarvam Vision (Document AI) | 81.8% | 2025-05-29 | independent | Source ↗ |
| Amazon Nova 2 Lite | 80.4% | 2025-05-27 | independent | Source ↗ |
| Amazon Nova 2 Lite | 78.8% | 2025-05-27 | independent | Source ↗ |
| Qwen: Qwen3 30B A3B | 79.4% | 2025-05-07 | independent | Source ↗ |
| Qwen: Qwen3 14B | 79.4% | 2025-05-07 | independent | Source ↗ |
| Qwen: Qwen3 32B | 79.4% | 2025-05-07 | independent | Source ↗ |
| o4 Mini High | 86.7% | 2025-04-25 | independent | Source ↗ |
| o3 | 86.7% | 2025-04-25 | independent | Source ↗ |
| o4 Mini | 86.7% | 2025-04-25 | independent | Source ↗ |
| Muse Wonder | 90.9% | 2025-04-24 | independent | Source ↗ |
| Muse Wonder Full | 93.3% | 2025-04-24 | independent | Source ↗ |
| GPT-4.1 | 86.7% | 2025-04-23 | independent | Source ↗ |
| GPT-4.1 Mini | 83.3% | 2025-04-23 | independent | Source ↗ |
| GPT-4.1 Nano | 79.4% | 2025-04-23 | independent | Source ↗ |
| Gemma 3 27B | 82.5% | 2025-04-21 | independent | Source ↗ |
| Nemotron 3 Ultra 550B | 89.4% | 2025-04-19 | independent | Source ↗ |
| Sarvam 30B | 82.5% | 2025-04-19 | independent | Source ↗ |
| Llama 4 Maverick | 88.4% | 2025-04-14 | independent | Source ↗ |
| Llama 4 Scout | 87.4% | 2025-04-14 | independent | Source ↗ |
| Gemini 2.5 Pro | 89.4% | 2025-04-03 | independent | Source ↗ |
| Gemini 2.5 Pro | 86.7% | 2025-04-03 | independent | Source ↗ |
| Gemini 2.5 Pro 06-05 | 86.7% | 2025-04-03 | independent | Source ↗ |
| Microsoft-Decision-1 | 85.1% | 2025-03-31 | independent | Source ↗ |
| Nemotron 3 Super 120B | 83.3% | 2025-03-29 | independent | Source ↗ |
| OpenAI Decisions API | 90.6% | 2025-03-27 | independent | Source ↗ |
| Jev 1.14 | 88% | 2025-03-23 | independent | Source ↗ |
| Krutrim-2 | 84.7% | 2025-03-21 | independent | Source ↗ |
| Reka Flash 3 | 75.5% | 2025-03-21 | independent | Source ↗ |
| Amazon Nova Premier 1.0 | 89% | 2025-03-19 | independent | Source ↗ |
| Amazon: Nova Premier 1.0 | 89.7% | 2025-03-19 | independent | Source ↗ |
| QwQ 32B | 84.4% | 2025-03-14 | independent | Source ↗ |
| Saaras v3 (Speech-to-Text) | 80.4% | 2025-03-10 | independent | Source ↗ |
| Bulbul v3 (Text-to-Speech) | 78.9% | 2025-03-10 | independent | Source ↗ |
| Phi-4-multimodal | 78.9% | 2025-03-09 | independent | Source ↗ |
| GPT-4.5 | 85.3% | 2025-03-08 | independent | Source ↗ |
| Claude 3.7 Sonnet | 90.6% | 2025-03-05 | independent | Source ↗ |
| Grok 3 | 88% | 2025-02-26 | independent | Source ↗ |
| Grok 3 Mini | 83.5% | 2025-02-26 | independent | Source ↗ |
| Phi-4-mini | 73.7% | 2025-02-24 | independent | Source ↗ |
| Perplexity: Sonar Deep Research | 86.7% | 2025-02-23 | independent | Source ↗ |
| Gemini 2.5 Flash (Thinking) | 86.2% | 2025-02-19 | independent | Source ↗ |
| Perplexity: Sonar Reasoning Pro | 86.7% | 2025-02-15 | independent | Source ↗ |
| Gemini 2.0 Pro | 84.8% | 2025-02-14 | independent | Source ↗ |
| Gemini 2.0 Flash | 80.5% | 2025-02-14 | independent | Source ↗ |
| Sonar Reasoning Pro | 88.4% | 2025-02-14 | independent | Source ↗ |
| Qwen: Qwen3 Coder Next | 79.4% | 2025-02-12 | independent | Source ↗ |
| o3-mini | 87.7% | 2025-02-09 | independent | Source ↗ |
| o3 Mini High | 86.7% | 2025-02-09 | independent | Source ↗ |
| o3 Mini | 86.7% | 2025-02-09 | independent | Source ↗ |
| Qwen 2.5 Max | 84.1% | 2025-02-06 | independent | Source ↗ |
| Qwen: Qwen2.5 VL 72B Instruct | 79.4% | 2025-02-04 | independent | Source ↗ |
| Qwen: Qwen-Plus | 79.4% | 2025-02-03 | independent | Source ↗ |
| Ollama DeepSeek-R1 (Q4_K_M) | 88.3% | 2025-01-31 | independent | Source ↗ |
| Perplexity: Sonar Pro Search | 89.7% | 2025-01-30 | independent | Source ↗ |
| Perplexity: Sonar Pro | 89.7% | 2025-01-30 | independent | Source ↗ |
| DeepSeek-R1 | 89.2% | 2025-01-29 | independent | Source ↗ |
| Gemini 2.0 Flash Thinking | 86.1% | 2025-01-29 | independent | Source ↗ |
| Kimi k1.5 | 82.3% | 2025-01-29 | independent | Source ↗ |
| R1 0528 | 83.3% | 2025-01-29 | independent | Source ↗ |
| MiniMax-Text-01 | 80.4% | 2025-01-24 | independent | Source ↗ |
| MiniMax: MiniMax-01 | 83.3% | 2025-01-24 | independent | Source ↗ |
| Codestral 25.01 | 76.9% | 2025-01-23 | independent | Source ↗ |
| Phi-4 | 83.1% | 2025-01-17 | independent | Source ↗ |
| Phi 4 | 75.5% | 2025-01-17 | independent | Source ↗ |
| DeepSeek-V3 | 80.8% | 2025-01-04 | independent | Source ↗ |
| DeepSeek V3.2 Exp | 79.4% | 2025-01-04 | independent | Source ↗ |
| DeepSeek V3.1 Terminus | 79.4% | 2025-01-04 | independent | Source ↗ |
| DeepSeek V3.1 | 79.4% | 2025-01-04 | independent | Source ↗ |
| DeepSeek V3 0324 | 83.3% | 2025-01-04 | independent | Source ↗ |
| DeepSeek V3 | 83.3% | 2025-01-04 | independent | Source ↗ |
| o1 | 89.7% | 2024-12-26 | independent | Source ↗ |
| Sonar Pro | 82.3% | 2024-12-19 | independent | Source ↗ |
| Ollama Llama 3.3 (Q4_K_M) | 82.1% | 2024-12-17 | independent | Source ↗ |
| Llama 3.3 70B | 79.3% | 2024-12-15 | independent | Source ↗ |
| Llama 3.3 70B Instruct | 82.8% | 2024-12-15 | independent | Source ↗ |
| Llama 3.3 70B Instruct | 79.4% | 2024-12-15 | independent | Source ↗ |
| o1 | 85.5% | 2024-12-14 | independent | Source ↗ |
| o1-pro | 89.7% | 2024-12-14 | independent | Source ↗ |
| Amazon Nova Pro | 80.9% | 2024-12-12 | independent | Source ↗ |
| Amazon Nova Lite | 73.5% | 2024-12-12 | independent | Source ↗ |
| Amazon Nova Pro 1.0 | 83.8% | 2024-12-12 | independent | Source ↗ |
| Amazon Nova Lite 1.0 | 77.9% | 2024-12-12 | independent | Source ↗ |
| Amazon Nova Micro 1.0 | 71.5% | 2024-12-12 | independent | Source ↗ |
| Amazon: Nova Lite 1.0 | 79.4% | 2024-12-12 | independent | Source ↗ |
| Amazon: Nova Micro 1.0 | 75.5% | 2024-12-12 | independent | Source ↗ |
| Amazon: Nova Pro 1.0 | 86.7% | 2024-12-12 | independent | Source ↗ |
| Command R7B (12-2024) | 75.5% | 2024-12-10 | independent | Source ↗ |
| QwQ-32B | 86.9% | 2024-12-07 | independent | Source ↗ |
| Solar Pro | 76.4% | 2024-11-29 | independent | Source ↗ |
| Krutrim Pro | 81.3% | 2024-11-29 | independent | Source ↗ |
| GPT-4o | 86.7% | 2024-11-29 | independent | Source ↗ |
| Upstage: Solar Pro 4 | 79.4% | 2024-11-29 | independent | Source ↗ |
| Jev | 86.7% | 2024-11-24 | independent | Source ↗ |
| Sonar Large | 79.4% | 2024-11-24 | independent | Source ↗ |
| Sonar | 79.4% | 2024-11-24 | independent | Source ↗ |
| TypeSafe: Jev Router | 75.5% | 2024-11-24 | independent | Source ↗ |
| Perplexity: Sonar | 79.4% | 2024-11-24 | independent | Source ↗ |
| Qwen 2.5 Coder 32B | 79.9% | 2024-11-21 | independent | Source ↗ |
| Qwen2.5 Coder 32B Instruct | 79.4% | 2024-11-21 | independent | Source ↗ |
| Hunyuan-Large | 82.8% | 2024-11-14 | independent | Source ↗ |
| Claude 3.5 Haiku | 75% | 2024-11-13 | independent | Source ↗ |
| Claude 3.5 Sonnet | 82.9% | 2024-10-31 | independent | Source ↗ |
| Claude 3.5 Sonnet (v2) | 86.1% | 2024-10-31 | independent | Source ↗ |
| BharatGPT-14B | 79.6% | 2024-10-27 | independent | Source ↗ |
| Yi-Lightning | 81.3% | 2024-10-24 | independent | Source ↗ |
| Llama-3.1-Nemotron-70B-Instruct | 84.1% | 2024-10-24 | independent | Source ↗ |
| Llama 3.2 1B Instruct | 79.4% | 2024-10-04 | independent | Source ↗ |
| Llama 3.2 3B Instruct | 79.4% | 2024-10-04 | independent | Source ↗ |
| Qwen 2.5 72B Instruct | 83.8% | 2024-09-28 | independent | Source ↗ |
| Qwen: Qwen2.5 7B Instruct | 75.5% | 2024-09-28 | independent | Source ↗ |
| Qwen2.5 72B Instruct | 79.4% | 2024-09-28 | independent | Source ↗ |
| Switchyard | 75.5% | 2024-09-24 | independent | Source ↗ |
| Mistral Small 4 | 79.4% | 2024-09-24 | independent | Source ↗ |
| GPT Audio | 86.7% | 2024-09-24 | independent | Source ↗ |
| GPT Audio Mini | 83.3% | 2024-09-24 | independent | Source ↗ |
| Nemotron 3 Nano 30B A3B | 79.4% | 2024-09-24 | independent | Source ↗ |
| Devstral 2 2512 | 83.3% | 2024-09-24 | independent | Source ↗ |
| Ministral 3 14B 2512 | 75.5% | 2024-09-24 | independent | Source ↗ |
| Ministral 3 8B 2512 | 75.5% | 2024-09-24 | independent | Source ↗ |
| Ministral 3 3B 2512 | 75.5% | 2024-09-24 | independent | Source ↗ |
| Voxtral Small 24B 2507 | 79.4% | 2024-09-24 | independent | Source ↗ |
| gpt-oss-safeguard-20b | 79.4% | 2024-09-24 | independent | Source ↗ |
| Gemini 2.5 Flash Image | 83.3% | 2024-09-24 | independent | Source ↗ |
| Z.ai: GLM 4.6 | 83.3% | 2024-09-24 | independent | Source ↗ |
| Mistral Medium 3.1 | 83.3% | 2024-09-24 | independent | Source ↗ |
| Z.ai: GLM 4.5V | 83.3% | 2024-09-24 | independent | Source ↗ |
| gpt-oss-120b | 79.4% | 2024-09-24 | independent | Source ↗ |
| gpt-oss-20b | 75.5% | 2024-09-24 | independent | Source ↗ |
| Codestral 2508 | 79.4% | 2024-09-24 | independent | Source ↗ |
| Z.ai: GLM 4.5 | 83.3% | 2024-09-24 | independent | Source ↗ |
| Z.ai: GLM 4.5 Air | 79.4% | 2024-09-24 | independent | Source ↗ |
| Gemini 2.5 Flash Lite | 79.4% | 2024-09-24 | independent | Source ↗ |
| Hunyuan A13B Instruct | 79.4% | 2024-09-24 | independent | Source ↗ |
| Mistral Small 3.2 24B | 79.4% | 2024-09-24 | independent | Source ↗ |
| Gemini 2.5 Flash | 83.3% | 2024-09-24 | independent | Source ↗ |
| Mistral Medium 3 | 83.3% | 2024-09-24 | independent | Source ↗ |
| Llama Guard 4 12B | 75.5% | 2024-09-24 | independent | Source ↗ |
| Llama 4 Maverick | 79.4% | 2024-09-24 | independent | Source ↗ |
| Llama 4 Scout | 79.4% | 2024-09-24 | independent | Source ↗ |
| Mistral Small 3.1 24B | 79.4% | 2024-09-24 | independent | Source ↗ |
| Gemma 3 4B | 75.5% | 2024-09-24 | independent | Source ↗ |
| Gemma 3 12B | 75.5% | 2024-09-24 | independent | Source ↗ |
| Gemma 3 27B | 79.4% | 2024-09-24 | independent | Source ↗ |
| Saba | 79.4% | 2024-09-24 | independent | Source ↗ |
| Mistral Small 3 | 75.5% | 2024-09-24 | independent | Source ↗ |
| Mistral Nemo | 75.5% | 2024-09-24 | independent | Source ↗ |
| Gemma 2 27B | 79.4% | 2024-09-24 | independent | Source ↗ |
| Mixtral 8x22B Instruct | 86.7% | 2024-09-24 | independent | Source ↗ |
| GPT-4 Turbo | 89.7% | 2024-09-24 | independent | Source ↗ |
| Mistral Large | 86.7% | 2024-09-24 | independent | Source ↗ |
| GPT-3.5 Turbo (older v0613) | 83.3% | 2024-09-24 | independent | Source ↗ |
| GPT-3.5 Turbo Instruct | 83.3% | 2024-09-24 | independent | Source ↗ |
| GPT-3.5 Turbo 16k | 86.7% | 2024-09-24 | independent | Source ↗ |
| GPT-3.5 Turbo | 83.3% | 2024-09-24 | independent | Source ↗ |
| GPT-4 | 89.7% | 2024-09-24 | independent | Source ↗ |
| GLM-4-Plus | 79.4% | 2024-08-29 | independent | Source ↗ |
| Sarvam 2B (Sarvam-1) | 75% | 2024-08-23 | independent | Source ↗ |
| GPT-4o | 86.7% | 2024-08-15 | independent | Source ↗ |
| Command R (08-2024) | 79.4% | 2024-08-10 | independent | Source ↗ |
| Gemma 2 2B | 56.8% | 2024-08-09 | independent | Source ↗ |
| Mistral Large 2 | 77.6% | 2024-08-02 | independent | Source ↗ |
| Mistral Large 2 | 82.5% | 2024-08-02 | independent | Source ↗ |
| Mistral Large 3 2512 | 83.3% | 2024-08-02 | independent | Source ↗ |
| Llama 3.1 405B | 80.4% | 2024-08-01 | independent | Source ↗ |
| Llama 3.1 70B Instruct | 79.4% | 2024-08-01 | independent | Source ↗ |
| Llama 3.1 8B Instruct | 75.5% | 2024-08-01 | independent | Source ↗ |
| GPT-4o mini | 71% | 2024-07-27 | independent | Source ↗ |
| GPT-4o-mini | 79.4% | 2024-07-27 | independent | Source ↗ |
| GPT-4o-mini | 79.4% | 2024-07-27 | independent | Source ↗ |
| ERNIE 4.0 Turbo | 79.9% | 2024-07-07 | independent | Source ↗ |
| Gemma 2 27B | 76.9% | 2024-07-06 | independent | Source ↗ |
| Gemma 2 9B | 69.6% | 2024-07-06 | independent | Source ↗ |
| Nemotron-4-340B-Instruct | 82.8% | 2024-06-23 | independent | Source ↗ |
| Nemotron-4-340B-Reward | 84.3% | 2024-06-23 | independent | Source ↗ |
| Hanooman 7B | 77.4% | 2024-05-24 | independent | Source ↗ |
| Gemini 1.5 Flash | 69.6% | 2024-05-23 | independent | Source ↗ |
| GPT-4o | 79.7% | 2024-05-22 | independent | Source ↗ |
| GPT-4o | 86.7% | 2024-05-22 | independent | Source ↗ |
| GPT-4o | 89.7% | 2024-05-22 | independent | Source ↗ |
| Snowflake Arctic | 74.5% | 2024-05-03 | independent | Source ↗ |
| Reka Core | 77.9% | 2024-04-24 | independent | Source ↗ |
| WizardLM-2 8x22B | 80.4% | 2024-04-24 | independent | Source ↗ |
| Command R+ | 72.5% | 2024-04-13 | independent | Source ↗ |
| DBRX Instruct | 75.5% | 2024-04-05 | independent | Source ↗ |
| Claude 3 Opus | 76.4% | 2024-03-13 | independent | Source ↗ |
| Gemini 1.5 Pro | 76% | 2024-02-24 | independent | Source ↗ |
| Airavata 7B | 73.5% | 2024-02-03 | independent | Source ↗ |
| IndicTrans-2 | 76.4% | 2023-09-24 | independent | Source ↗ |
| Nemotron 3 Super | 79.4% | 2023-03-20 | independent | Source ↗ |
Human Baseline & Difficulty Horizon
Calibrated human reference points, specialist benchmarks, and ceiling thresholds.Reference expert baseline adjusted for the 5.7% annotation error rate detected in the original suite.
Metric & Scoring Methodology
Verification protocols, aggregation formulas, and specialized metric variants.accuracy on the re-annotated (error-corrected) MMLU questions (%)Dataset & Compute Cost
Evaluation volume, public availability, API pricing, and local hardware requirements.$5 – $20 USD for full benchmark evaluation run on frontier APIs.
1x NVIDIA RTX 4090 (24GB) or A100 (40GB/80GB) via vLLM / SGLang
How to Run & Reproduce
Standardized evaluation protocols, CLI commands, and reproducible runner templates.lm_eval --model hf --model_args pretrained=<model_path> --tasks mmlu-redux --batch_size autoopencompass --datasets mmlu-redux --models <model_config># Standard API Evaluation Loop
from openai import OpenAI
client = OpenAI()
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": prompt}],
temperature=0.0,
)When publishing results for MMLU-Redux, always report the exact prompt template, few-shot exemplar ordering, sampling temperature (temperature=0), maximum reasoning budget tokens, and the precise timestamped model snapshot ID.
Contamination & Memorization Analysis
Audit of pretraining exposure risks, memorization vectors, and refresh policies.Static fixed snapshot
Public on web / HuggingFace