KnowledgeSaturated
Trust Score:
25

MMLU-Redux

A manual error audit of MMLU: experts re-annotated thousands of questions and found ~6.5% are broken — the evidence base for MMLU's label-noise ceiling.

Launched: —Refresh: static
Status Assessment (saturated):

As a competition benchmark, MMLU-Redux is a corrected slice of MMLU: it inherits MMLU's contamination and frontier convergence, so it does not differentiate top models — saturated by CLAUDE.md's definition (usefulness as a model-differentiating eval). Its ENDURING value is different and non-competitive: it remains an actively-used forensic reference and audit methodology for measuring label noise. Status here reflects the eval role; the still-active reference role is documented in prose. saturated_date is set to its launch — it described an already-saturated parent from birth.

Performance Timeline

Longitudinal progression of model scores against human baselines.

Performance & Historical Trajectory

Empirical score progression across model release dates and evaluation rounds.

Independent Vendor-Reported Human Baseline (89.8%)
ModelScoreDateSource TypeProvenance
MiMo-V2.6-Flash84.7%2026-09-12independentSource ↗
Claude Opus 5.581.1%2026-09-12independentSource ↗
Qwen 3.8 Max Prime76.4%2026-09-12independentSource ↗
Nemotron 3.5 Super VL79.4%2026-09-12independentSource ↗
Gemini 4 Argon86.7%2026-09-12independentSource ↗
Claude Haiku 5.579.4%2026-09-12independentSource ↗
Nano Banana 2.186.7%2026-09-12independentSource ↗
Mistral Large 483.3%2026-09-12independentSource ↗
GPT-6.1 Sol Pro86.7%2026-09-12independentSource ↗
GPT-6.1 Sol86.7%2026-09-12independentSource ↗
Claude Sonnet 5.586.7%2026-09-12independentSource ↗
Z.ai: GLM 5.3 Prime86.7%2026-09-12independentSource ↗
GPT-6 Luna Pro79.4%2026-09-12independentSource ↗
GPT-6 Luna79.4%2026-09-12independentSource ↗
GPT-6 Sol Pro86.7%2026-09-12independentSource ↗
GPT-6 Sol86.7%2026-09-12independentSource ↗
Claude Opus 5.589.7%2026-09-12independentSource ↗
Xiaomi: MiMo-V2.6-Pro-UltraSpeed86.7%2026-09-12independentSource ↗
Xiaomi: MiMo-V2.6-Flash79.4%2026-09-12independentSource ↗
Xiaomi: MiMo-V2.6-Pro79.4%2026-09-12independentSource ↗
Z.ai: GLM 5.3 FlashX83.3%2026-09-12independentSource ↗
DeepSeek V4.1 Flash79.4%2026-09-12independentSource ↗
GPT-6 Astra89.7%2026-09-12independentSource ↗
GPT-6 Astra Pro89.7%2026-09-12independentSource ↗
Claude Sonnet 5.592.6%2026-09-12independentSource ↗
Claude Haiku 5.587.2%2026-09-12independentSource ↗
GPT-6.1 Sol Pro94.6%2026-09-12independentSource ↗
GPT-6 Luna92.1%2026-09-12independentSource ↗
Gemini 4 Argon94.1%2026-09-12independentSource ↗
Qwen 3.8 Max Prime91.1%2026-09-12independentSource ↗
DeepSeek V4.1 Flash 101779.4%2026-09-12independentSource ↗
GPT-6.1 Terra Pro86.7%2026-09-12independentSource ↗
GPT-6.1 Terra86.7%2026-09-12independentSource ↗
MoonshotAI: Kimi K3.1 Pro86.7%2026-09-12independentSource ↗
MiniMax: MiniMax M3.1 Flash79.4%2026-09-12independentSource ↗
Muse Spark Open86.7%2026-09-12independentSource ↗
LiquidAI: LFM2.5-24B-A2B79.4%2026-09-12independentSource ↗
Muse Wonder89.7%2026-09-12independentSource ↗
Qwen: Qwen3.8 Max Prime89.7%2026-09-12independentSource ↗
Upstage: Solar Mini 475.5%2026-09-12independentSource ↗
SpaceXAI: Grok 4.786.7%2026-09-12independentSource ↗
Qwen: Qwen3.8 Omni Flash79.4%2026-09-12independentSource ↗
Gemini 3.8 Flash86.7%2026-09-11independentSource ↗
Gemini 3.8 Flash91.9%2026-09-11independentSource ↗
Qwen: Qwen3.8 Max (0902)86.7%2026-09-11independentSource ↗
Muse Spark 1.3 Contributor75.5%2026-09-11independentSource ↗
Muse Spark 1.386.7%2026-09-11independentSource ↗
Claude Fable 5.189.7%2026-09-09independentSource ↗
Hy490.6%2026-09-05independentSource ↗
Hy483.3%2026-09-05independentSource ↗
GLM 5.3 Flash91.7%2026-09-04independentSource ↗
Z.ai: GLM 5.3 Flash79.4%2026-09-04independentSource ↗
Qwen: Qwen3.8 Flash79.4%2026-09-04independentSource ↗
DeepSeek V4 Flash Vision Exp79.4%2026-08-30independentSource ↗
DeepSeek V4-Pro90.4%2026-08-25independentSource ↗
DeepSeek V4-Flash86.2%2026-08-25independentSource ↗
Z.ai: GLM 5.386.7%2026-08-25independentSource ↗
Claude Fable 5.192.9%2026-08-24independentSource ↗
Qwen: Qwen3.8 27B83.3%2026-08-23independentSource ↗
Gemini 3.7 Flash91.3%2026-08-22independentSource ↗
Gemini 3.7 Flash86.7%2026-08-22independentSource ↗
DeepSeek V4 Pro 081386.7%2026-08-22independentSource ↗
Grok 4.691.1%2026-08-21independentSource ↗
Qwen: Qwen3.8 2.4T A95B86.7%2026-08-21independentSource ↗
LiquidAI: LFM2.5-2.6B79.4%2026-08-20independentSource ↗
ByteDance Seed: Seed 2.1 Turbo83.3%2026-08-19independentSource ↗
SpaceXAI: Grok 4.686.7%2026-08-19independentSource ↗
Muse Glimmer 30B83.3%2026-08-19independentSource ↗
GPT-591.9%2026-08-16independentSource ↗
Nemotron 3.5 Lightning75.5%2026-08-16independentSource ↗
GPT-5 Image Mini83.3%2026-08-16independentSource ↗
GPT-5 Image86.7%2026-08-16independentSource ↗
Muse Spark 1.2 Contributor75.5%2026-08-14independentSource ↗
Muse Spark 1.286.7%2026-08-14independentSource ↗
DeepSeek V4.1 Flash91.6%2026-08-10independentSource ↗
Space Bunny Alpha89.4%2026-08-10independentSource ↗
DeepSeek V4 Flash 073187%2026-08-09independentSource ↗
DeepSeek V4 Flash 073183.3%2026-08-09independentSource ↗
ByteDance Seed: Seed-2.0-Code83.3%2026-08-08independentSource ↗
GPT-6 Astra92.3%2026-08-06independentSource ↗
Qwen: Qwen3.7 Flash75.5%2026-08-05independentSource ↗
Kimi K391.6%2026-08-04independentSource ↗
Claude Opus 592.9%2026-08-02independentSource ↗
Claude Sonnet 590.9%2026-08-02independentSource ↗
Claude Opus 589.7%2026-08-01independentSource ↗
Gemini 3.6 Flash90%2026-07-30independentSource ↗
Gemini 3.5 Flash-Lite85.8%2026-07-30independentSource ↗
Gemini 3.6 Flash86.7%2026-07-30independentSource ↗
Gemini 3.5 Flash Lite83.3%2026-07-30independentSource ↗
MoonshotAI: Kimi K389.7%2026-07-24independentSource ↗
GPT-5.6 Sol93.6%2026-07-18independentSource ↗
GPT-5.5 Thinking92.1%2026-07-18independentSource ↗
GPT-5.6 Luna88.2%2026-07-18independentSource ↗
GPT-5.6 Luna Pro83.3%2026-07-18independentSource ↗
GPT-5.6 Luna83.3%2026-07-18independentSource ↗
GPT-5.6 Terra Pro89.7%2026-07-18independentSource ↗
GPT-5.6 Terra89.7%2026-07-18independentSource ↗
GPT-5.6 Sol Pro89.7%2026-07-18independentSource ↗
GPT-5.6 Sol89.7%2026-07-18independentSource ↗
Muse Spark 1.186.7%2026-07-18independentSource ↗
SpaceXAI: Grok 4.586.7%2026-07-17independentSource ↗
Hy379.4%2026-07-15independentSource ↗
Claude Sonnet 586.7%2026-07-09independentSource ↗
Gemini 3.1 Flash Lite Image83.3%2026-07-09independentSource ↗
MiniMax M387.7%2026-07-04independentSource ↗
Qwen3-235B89.7%2026-06-27independentSource ↗
QwQ Plus89%2026-06-27independentSource ↗
Z.ai: GLM 5.283.3%2026-06-25independentSource ↗
Grok 491.6%2026-06-24independentSource ↗
GLM-5.288.7%2026-06-21independentSource ↗
MoonshotAI: Kimi K2.7 Code86.7%2026-06-21independentSource ↗
Claude Fable 594.1%2026-06-18independentSource ↗
Claude Fable 589.7%2026-06-18independentSource ↗
Nemotron 3 Ultra85.8%2026-06-13independentSource ↗
Nemotron 3.5 Content Safety75.5%2026-06-13independentSource ↗
Nemotron 3 Ultra83.3%2026-06-13independentSource ↗
Qwen: Qwen3.7 Plus83.3%2026-06-11independentSource ↗
MiniMax: MiniMax M383.3%2026-06-09independentSource ↗
Gemini 3.1 Flash Image83.3%2026-06-06independentSource ↗
Gemini 3 Pro Image (Nano Banana Pro)89.7%2026-06-06independentSource ↗
Claude Opus 4.889.7%2026-06-06independentSource ↗
Hy-MT2-1.8B75.5%2026-05-30independentSource ↗
Hy-MT2-30B-A3B79.4%2026-05-30independentSource ↗
Hy-MT2-7B79.4%2026-05-30independentSource ↗
Qwen: Qwen3.7 Max86.7%2026-05-29independentSource ↗
SpaceXAI: Grok Build 0.183.3%2026-05-29independentSource ↗
Gemini 3.5 Flash86.7%2026-05-28independentSource ↗
Gemini 3 Pro90.6%2026-05-27independentSource ↗
Gemini 3.1 Flash Lite83.3%2026-05-16independentSource ↗
GPT Chat Latest89.7%2026-05-14independentSource ↗
Command A+83.3%2026-05-10independentSource ↗
Mistral Medium 3.586.7%2026-05-09independentSource ↗
SpaceXAI: Grok 4.383.3%2026-05-09independentSource ↗
Qwen: Qwen3.6 Flash83.3%2026-05-06independentSource ↗
GPT-5.5 Pro89.7%2026-05-02independentSource ↗
GPT-5.589.7%2026-05-02independentSource ↗
DeepSeek V4 Pro 042383.3%2026-05-02independentSource ↗
DeepSeek V4 Flash 042375.5%2026-05-02independentSource ↗
MiMo-V2.583.5%2026-05-01independentSource ↗
Xiaomi: MiMo-V2.5-Pro79.4%2026-05-01independentSource ↗
Xiaomi: MiMo-V2.579.4%2026-05-01independentSource ↗
Qwen: Qwen3.6 27B83.3%2026-05-01independentSource ↗
Hy379.4%2026-04-30independentSource ↗
GPT-5.4 Image 289.7%2026-04-30independentSource ↗
Qwen: Qwen3.5 Plus 2026-04-2083.3%2026-04-29independentSource ↗
MoonshotAI: Kimi K2.686.7%2026-04-29independentSource ↗
Claude Opus 4.789.7%2026-04-25independentSource ↗
Qwen: Qwen3.6 35B A3B79.4%2026-04-24independentSource ↗
Z.ai: GLM 5.186.7%2026-04-15independentSource ↗
Gemma 4 26B A4B79.4%2026-04-12independentSource ↗
Gemma 4 31B79.4%2026-04-11independentSource ↗
Qwen: Qwen3.6 Plus83.3%2026-04-11independentSource ↗
Z.ai: GLM 5V Turbo86.7%2026-04-10independentSource ↗
Lyria 3 Pro75.5%2026-04-08independentSource ↗
Lyria 3 Clip75.5%2026-04-08independentSource ↗
Reka Edge75.5%2026-03-29independentSource ↗
MiniMax: MiniMax M2.783.3%2026-03-27independentSource ↗
GPT-5.4 Nano83.3%2026-03-26independentSource ↗
GPT-5.4 Mini86.7%2026-03-26independentSource ↗
Z.ai: GLM 5 Turbo86.7%2026-03-24independentSource ↗
Qwen: Qwen3.5-9B79.4%2026-03-19independentSource ↗
SpaceXAI: Grok 4.20 Multi-Agent83.3%2026-03-18independentSource ↗
SpaceXAI: Grok 4.2083.3%2026-03-18independentSource ↗
ByteDance Seed: Seed-2.0-Lite83.3%2026-03-18independentSource ↗
GPT-5.4 Pro89.7%2026-03-14independentSource ↗
GPT-5.489.7%2026-03-14independentSource ↗
Gemini 3.1 Flash Lite83.3%2026-03-12independentSource ↗
Gemini 3.1 Flash Image83.3%2026-03-07independentSource ↗
GPT-5.3-Codex89.7%2026-03-05independentSource ↗
ByteDance Seed: Seed-2.0-Mini79.4%2026-03-05independentSource ↗
Qwen: Qwen3.5-35B-A3B79.4%2026-03-05independentSource ↗
Qwen: Qwen3.5-27B83.3%2026-03-05independentSource ↗
Qwen: Qwen3.5-122B-A10B83.3%2026-03-05independentSource ↗
Qwen: Qwen3.5-Flash79.4%2026-03-05independentSource ↗
Gemini 3.1 Pro Custom Tools89.7%2026-02-28independentSource ↗
Gemini 3.1 Pro89.7%2026-02-28independentSource ↗
Claude Sonnet 4.689.7%2026-02-26independentSource ↗
Qwen: Qwen3.5 Plus 2026-02-1583.3%2026-02-25independentSource ↗
Qwen: Qwen3.5 397B A17B86.7%2026-02-25independentSource ↗
Z.ai: GLM 586.7%2026-02-20independentSource ↗
MiniMax: MiniMax M2.579.4%2026-02-20independentSource ↗
Claude Opus 4.689.7%2026-02-14independentSource ↗
MoonshotAI: Kimi K2.583.3%2026-02-05independentSource ↗
MiniMax: MiniMax M2-her83.3%2026-02-01independentSource ↗
Z.ai: GLM 4.7 Flash79.4%2026-01-28independentSource ↗
GPT-5.2-Codex89.7%2026-01-23independentSource ↗
MiniMax: MiniMax M2.183.3%2026-01-01independentSource ↗
Z.ai: GLM 4.783.3%2025-12-31independentSource ↗
o390.2%2025-12-29independentSource ↗
Gemini 3 Flash83.3%2025-12-26independentSource ↗
GPT-5.2 Chat89.7%2025-12-20independentSource ↗
GPT-5.2 Pro89.7%2025-12-20independentSource ↗
GPT-5.289.7%2025-12-20independentSource ↗
Z.ai: GLM 4.6V79.4%2025-12-17independentSource ↗
GPT-5.1-Codex-Max86.7%2025-12-13independentSource ↗
DeepSeek V3.279.4%2025-12-10independentSource ↗
Claude Opus 4.589.7%2025-12-03independentSource ↗
Gemini 3 Pro Image (Nano Banana Pro)89.7%2025-11-29independentSource ↗
GPT-5.186.7%2025-11-22independentSource ↗
GPT-5.1-Codex86.7%2025-11-22independentSource ↗
GPT-5.1-Codex-Mini83.3%2025-11-22independentSource ↗
Mistral Large 384.3%2025-11-21independentSource ↗
MoonshotAI: Kimi K2 Thinking83.3%2025-11-15independentSource ↗
MiniMax: MiniMax M283.3%2025-11-01independentSource ↗
Qwen: Qwen3 VL 8B Instruct79.4%2025-10-23independentSource ↗
GPT-5 Pro89.7%2025-10-15independentSource ↗
Qwen: Qwen3 VL 30B A3B Thinking79.4%2025-10-15independentSource ↗
Qwen: Qwen3 VL 30B A3B Instruct79.4%2025-10-15independentSource ↗
Claude Haiku 4.586.7%2025-10-10independentSource ↗
Claude Sonnet 4.589.7%2025-10-08independentSource ↗
Qwen: Qwen3 VL 235B A22B Instruct83.3%2025-10-02independentSource ↗
Qwen: Qwen3 Coder Flash79.4%2025-09-26independentSource ↗
Qwen: Qwen3 Next 80B A3B Instruct83.3%2025-09-20independentSource ↗
MoonshotAI: Kimi K2 090583.3%2025-09-13independentSource ↗
GPT-586.7%2025-08-16independentSource ↗
GPT-5 Mini83.3%2025-08-16independentSource ↗
GPT-5 Nano79.4%2025-08-16independentSource ↗
Claude 4.1 Opus89.7%2025-08-14independentSource ↗
Qwen: Qwen3 Coder 30B A3B Instruct79.4%2025-08-09independentSource ↗
Qwen: Qwen3 30B A3B Instruct 250779.4%2025-08-07independentSource ↗
Qwen: Qwen3 235B A22B Thinking 250786.7%2025-08-03independentSource ↗
Qwen: Qwen3 Coder 480B A35B79.4%2025-08-01independentSource ↗
ByteDance: UI-TARS 7B75.5%2025-07-31independentSource ↗
Qwen: Qwen3 235B A22B Instruct 250779.4%2025-07-30independentSource ↗
MoonshotAI: Kimi K2 071183.3%2025-07-20independentSource ↗
ByteDance Seed: Seed 1.6 Flash79.4%2025-07-04independentSource ↗
ByteDance Seed: Seed 1.683.3%2025-07-04independentSource ↗
MiniMax: MiniMax M183.3%2025-06-26independentSource ↗
Sarvam 105B86.7%2025-06-24independentSource ↗
o3 Pro89.7%2025-06-19independentSource ↗
Sarvam Vision (Document AI)81.8%2025-05-29independentSource ↗
Amazon Nova 2 Lite80.4%2025-05-27independentSource ↗
Amazon Nova 2 Lite78.8%2025-05-27independentSource ↗
Qwen: Qwen3 30B A3B79.4%2025-05-07independentSource ↗
Qwen: Qwen3 14B79.4%2025-05-07independentSource ↗
Qwen: Qwen3 32B79.4%2025-05-07independentSource ↗
o4 Mini High86.7%2025-04-25independentSource ↗
o386.7%2025-04-25independentSource ↗
o4 Mini86.7%2025-04-25independentSource ↗
Muse Wonder90.9%2025-04-24independentSource ↗
Muse Wonder Full93.3%2025-04-24independentSource ↗
GPT-4.186.7%2025-04-23independentSource ↗
GPT-4.1 Mini83.3%2025-04-23independentSource ↗
GPT-4.1 Nano79.4%2025-04-23independentSource ↗
Gemma 3 27B82.5%2025-04-21independentSource ↗
Nemotron 3 Ultra 550B89.4%2025-04-19independentSource ↗
Sarvam 30B82.5%2025-04-19independentSource ↗
Llama 4 Maverick88.4%2025-04-14independentSource ↗
Llama 4 Scout87.4%2025-04-14independentSource ↗
Gemini 2.5 Pro89.4%2025-04-03independentSource ↗
Gemini 2.5 Pro86.7%2025-04-03independentSource ↗
Gemini 2.5 Pro 06-0586.7%2025-04-03independentSource ↗
Microsoft-Decision-185.1%2025-03-31independentSource ↗
Nemotron 3 Super 120B83.3%2025-03-29independentSource ↗
OpenAI Decisions API90.6%2025-03-27independentSource ↗
Jev 1.1488%2025-03-23independentSource ↗
Krutrim-284.7%2025-03-21independentSource ↗
Reka Flash 375.5%2025-03-21independentSource ↗
Amazon Nova Premier 1.089%2025-03-19independentSource ↗
Amazon: Nova Premier 1.089.7%2025-03-19independentSource ↗
QwQ 32B84.4%2025-03-14independentSource ↗
Saaras v3 (Speech-to-Text)80.4%2025-03-10independentSource ↗
Bulbul v3 (Text-to-Speech)78.9%2025-03-10independentSource ↗
Phi-4-multimodal78.9%2025-03-09independentSource ↗
GPT-4.585.3%2025-03-08independentSource ↗
Claude 3.7 Sonnet90.6%2025-03-05independentSource ↗
Grok 388%2025-02-26independentSource ↗
Grok 3 Mini83.5%2025-02-26independentSource ↗
Phi-4-mini73.7%2025-02-24independentSource ↗
Perplexity: Sonar Deep Research86.7%2025-02-23independentSource ↗
Gemini 2.5 Flash (Thinking)86.2%2025-02-19independentSource ↗
Perplexity: Sonar Reasoning Pro86.7%2025-02-15independentSource ↗
Gemini 2.0 Pro84.8%2025-02-14independentSource ↗
Gemini 2.0 Flash80.5%2025-02-14independentSource ↗
Sonar Reasoning Pro88.4%2025-02-14independentSource ↗
Qwen: Qwen3 Coder Next79.4%2025-02-12independentSource ↗
o3-mini87.7%2025-02-09independentSource ↗
o3 Mini High86.7%2025-02-09independentSource ↗
o3 Mini86.7%2025-02-09independentSource ↗
Qwen 2.5 Max84.1%2025-02-06independentSource ↗
Qwen: Qwen2.5 VL 72B Instruct79.4%2025-02-04independentSource ↗
Qwen: Qwen-Plus79.4%2025-02-03independentSource ↗
Ollama DeepSeek-R1 (Q4_K_M)88.3%2025-01-31independentSource ↗
Perplexity: Sonar Pro Search89.7%2025-01-30independentSource ↗
Perplexity: Sonar Pro89.7%2025-01-30independentSource ↗
DeepSeek-R189.2%2025-01-29independentSource ↗
Gemini 2.0 Flash Thinking86.1%2025-01-29independentSource ↗
Kimi k1.582.3%2025-01-29independentSource ↗
R1 052883.3%2025-01-29independentSource ↗
MiniMax-Text-0180.4%2025-01-24independentSource ↗
MiniMax: MiniMax-0183.3%2025-01-24independentSource ↗
Codestral 25.0176.9%2025-01-23independentSource ↗
Phi-483.1%2025-01-17independentSource ↗
Phi 475.5%2025-01-17independentSource ↗
DeepSeek-V380.8%2025-01-04independentSource ↗
DeepSeek V3.2 Exp79.4%2025-01-04independentSource ↗
DeepSeek V3.1 Terminus79.4%2025-01-04independentSource ↗
DeepSeek V3.179.4%2025-01-04independentSource ↗
DeepSeek V3 032483.3%2025-01-04independentSource ↗
DeepSeek V383.3%2025-01-04independentSource ↗
o189.7%2024-12-26independentSource ↗
Sonar Pro82.3%2024-12-19independentSource ↗
Ollama Llama 3.3 (Q4_K_M)82.1%2024-12-17independentSource ↗
Llama 3.3 70B79.3%2024-12-15independentSource ↗
Llama 3.3 70B Instruct82.8%2024-12-15independentSource ↗
Llama 3.3 70B Instruct79.4%2024-12-15independentSource ↗
o185.5%2024-12-14independentSource ↗
o1-pro89.7%2024-12-14independentSource ↗
Amazon Nova Pro80.9%2024-12-12independentSource ↗
Amazon Nova Lite73.5%2024-12-12independentSource ↗
Amazon Nova Pro 1.083.8%2024-12-12independentSource ↗
Amazon Nova Lite 1.077.9%2024-12-12independentSource ↗
Amazon Nova Micro 1.071.5%2024-12-12independentSource ↗
Amazon: Nova Lite 1.079.4%2024-12-12independentSource ↗
Amazon: Nova Micro 1.075.5%2024-12-12independentSource ↗
Amazon: Nova Pro 1.086.7%2024-12-12independentSource ↗
Command R7B (12-2024)75.5%2024-12-10independentSource ↗
QwQ-32B86.9%2024-12-07independentSource ↗
Solar Pro76.4%2024-11-29independentSource ↗
Krutrim Pro81.3%2024-11-29independentSource ↗
GPT-4o86.7%2024-11-29independentSource ↗
Upstage: Solar Pro 479.4%2024-11-29independentSource ↗
Jev86.7%2024-11-24independentSource ↗
Sonar Large79.4%2024-11-24independentSource ↗
Sonar79.4%2024-11-24independentSource ↗
TypeSafe: Jev Router75.5%2024-11-24independentSource ↗
Perplexity: Sonar79.4%2024-11-24independentSource ↗
Qwen 2.5 Coder 32B79.9%2024-11-21independentSource ↗
Qwen2.5 Coder 32B Instruct79.4%2024-11-21independentSource ↗
Hunyuan-Large82.8%2024-11-14independentSource ↗
Claude 3.5 Haiku75%2024-11-13independentSource ↗
Claude 3.5 Sonnet82.9%2024-10-31independentSource ↗
Claude 3.5 Sonnet (v2)86.1%2024-10-31independentSource ↗
BharatGPT-14B79.6%2024-10-27independentSource ↗
Yi-Lightning81.3%2024-10-24independentSource ↗
Llama-3.1-Nemotron-70B-Instruct84.1%2024-10-24independentSource ↗
Llama 3.2 1B Instruct79.4%2024-10-04independentSource ↗
Llama 3.2 3B Instruct79.4%2024-10-04independentSource ↗
Qwen 2.5 72B Instruct83.8%2024-09-28independentSource ↗
Qwen: Qwen2.5 7B Instruct75.5%2024-09-28independentSource ↗
Qwen2.5 72B Instruct79.4%2024-09-28independentSource ↗
Switchyard75.5%2024-09-24independentSource ↗
Mistral Small 479.4%2024-09-24independentSource ↗
GPT Audio86.7%2024-09-24independentSource ↗
GPT Audio Mini83.3%2024-09-24independentSource ↗
Nemotron 3 Nano 30B A3B79.4%2024-09-24independentSource ↗
Devstral 2 251283.3%2024-09-24independentSource ↗
Ministral 3 14B 251275.5%2024-09-24independentSource ↗
Ministral 3 8B 251275.5%2024-09-24independentSource ↗
Ministral 3 3B 251275.5%2024-09-24independentSource ↗
Voxtral Small 24B 250779.4%2024-09-24independentSource ↗
gpt-oss-safeguard-20b79.4%2024-09-24independentSource ↗
Gemini 2.5 Flash Image83.3%2024-09-24independentSource ↗
Z.ai: GLM 4.683.3%2024-09-24independentSource ↗
Mistral Medium 3.183.3%2024-09-24independentSource ↗
Z.ai: GLM 4.5V83.3%2024-09-24independentSource ↗
gpt-oss-120b79.4%2024-09-24independentSource ↗
gpt-oss-20b75.5%2024-09-24independentSource ↗
Codestral 250879.4%2024-09-24independentSource ↗
Z.ai: GLM 4.583.3%2024-09-24independentSource ↗
Z.ai: GLM 4.5 Air79.4%2024-09-24independentSource ↗
Gemini 2.5 Flash Lite79.4%2024-09-24independentSource ↗
Hunyuan A13B Instruct79.4%2024-09-24independentSource ↗
Mistral Small 3.2 24B79.4%2024-09-24independentSource ↗
Gemini 2.5 Flash83.3%2024-09-24independentSource ↗
Mistral Medium 383.3%2024-09-24independentSource ↗
Llama Guard 4 12B75.5%2024-09-24independentSource ↗
Llama 4 Maverick79.4%2024-09-24independentSource ↗
Llama 4 Scout79.4%2024-09-24independentSource ↗
Mistral Small 3.1 24B79.4%2024-09-24independentSource ↗
Gemma 3 4B75.5%2024-09-24independentSource ↗
Gemma 3 12B75.5%2024-09-24independentSource ↗
Gemma 3 27B79.4%2024-09-24independentSource ↗
Saba79.4%2024-09-24independentSource ↗
Mistral Small 375.5%2024-09-24independentSource ↗
Mistral Nemo75.5%2024-09-24independentSource ↗
Gemma 2 27B79.4%2024-09-24independentSource ↗
Mixtral 8x22B Instruct86.7%2024-09-24independentSource ↗
GPT-4 Turbo89.7%2024-09-24independentSource ↗
Mistral Large86.7%2024-09-24independentSource ↗
GPT-3.5 Turbo (older v0613)83.3%2024-09-24independentSource ↗
GPT-3.5 Turbo Instruct83.3%2024-09-24independentSource ↗
GPT-3.5 Turbo 16k86.7%2024-09-24independentSource ↗
GPT-3.5 Turbo83.3%2024-09-24independentSource ↗
GPT-489.7%2024-09-24independentSource ↗
GLM-4-Plus79.4%2024-08-29independentSource ↗
Sarvam 2B (Sarvam-1)75%2024-08-23independentSource ↗
GPT-4o86.7%2024-08-15independentSource ↗
Command R (08-2024)79.4%2024-08-10independentSource ↗
Gemma 2 2B56.8%2024-08-09independentSource ↗
Mistral Large 277.6%2024-08-02independentSource ↗
Mistral Large 282.5%2024-08-02independentSource ↗
Mistral Large 3 251283.3%2024-08-02independentSource ↗
Llama 3.1 405B80.4%2024-08-01independentSource ↗
Llama 3.1 70B Instruct79.4%2024-08-01independentSource ↗
Llama 3.1 8B Instruct75.5%2024-08-01independentSource ↗
GPT-4o mini71%2024-07-27independentSource ↗
GPT-4o-mini79.4%2024-07-27independentSource ↗
GPT-4o-mini79.4%2024-07-27independentSource ↗
ERNIE 4.0 Turbo79.9%2024-07-07independentSource ↗
Gemma 2 27B76.9%2024-07-06independentSource ↗
Gemma 2 9B69.6%2024-07-06independentSource ↗
Nemotron-4-340B-Instruct82.8%2024-06-23independentSource ↗
Nemotron-4-340B-Reward84.3%2024-06-23independentSource ↗
Hanooman 7B77.4%2024-05-24independentSource ↗
Gemini 1.5 Flash69.6%2024-05-23independentSource ↗
GPT-4o79.7%2024-05-22independentSource ↗
GPT-4o86.7%2024-05-22independentSource ↗
GPT-4o89.7%2024-05-22independentSource ↗
Snowflake Arctic74.5%2024-05-03independentSource ↗
Reka Core77.9%2024-04-24independentSource ↗
WizardLM-2 8x22B80.4%2024-04-24independentSource ↗
Command R+72.5%2024-04-13independentSource ↗
DBRX Instruct75.5%2024-04-05independentSource ↗
Claude 3 Opus76.4%2024-03-13independentSource ↗
Gemini 1.5 Pro76%2024-02-24independentSource ↗
Airavata 7B73.5%2024-02-03independentSource ↗
IndicTrans-276.4%2023-09-24independentSource ↗
Nemotron 3 Super79.4%2023-03-20independentSource ↗

Human Baseline & Difficulty Horizon

Calibrated human reference points, specialist benchmarks, and ceiling thresholds.
Measured Human Score89.8%Domain Expert Baseline
Baseline Protocol & Interpretation

Reference expert baseline adjusted for the 5.7% annotation error rate detected in the original suite.

Metric & Scoring Methodology

Verification protocols, aggregation formulas, and specialized metric variants.
Primary Metric:accuracy on the re-annotated (error-corrected) MMLU questions (%)
Scoring Engine:exact-match

Dataset & Compute Cost

Evaluation volume, public availability, API pricing, and local hardware requirements.
Total Dataset Size5,700Annotated evaluation items
Public Test SetPublicOpenly mirrored on repositories
Access GatingOpen AccessUnrestricted download
Evaluation LicenseOpen AccessDataset usage and redistribution terms
Frontier API Compute Cost:

$5 – $20 USD for full benchmark evaluation run on frontier APIs.

Recommended Local GPU Setup:

1x NVIDIA RTX 4090 (24GB) or A100 (40GB/80GB) via vLLM / SGLang

Official Dataset & Benchmark Files:Download / View Dataset Repository ↗

How to Run & Reproduce

Standardized evaluation protocols, CLI commands, and reproducible runner templates.
Prompt Regimezero-shot
Reasoning Modedirect
Sampling Temp0
Pass@k Budgetk = 1
Tools & SandboxPure Text
Scoring Verifierexact-match
Option AEleutherAI LM-Evaluation-Harness (Open-Weight Models)
lm_eval --model hf --model_args pretrained=<model_path> --tasks mmlu-redux --batch_size auto
Option BOpenCompass Evaluation Framework
opencompass --datasets mmlu-redux --models <model_config>
Python APIDeterministic Inference Loop Snippet
# Standard API Evaluation Loop
from openai import OpenAI

client = OpenAI()
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": prompt}],
    temperature=0.0,
)
Standardized Reporting Requirement:

When publishing results for MMLU-Redux, always report the exact prompt template, few-shot exemplar ordering, sampling temperature (temperature=0), maximum reasoning budget tokens, and the precise timestamped model snapshot ID.

Contamination & Memorization Analysis

Audit of pretraining exposure risks, memorization vectors, and refresh policies.
Overall Contamination Risk:HIGH
Refresh Cadence:

Static fixed snapshot

Test Set Exposure:

Public on web / HuggingFace