Multimodal VisionActive
Trust Score:
93

MMMU-Pro

Harder, filtered variant of MMMU for multimodal college-level reasoning, with more answer options and questions embedded in images.

Launched: 2024-09-04Refresh: static
Status Assessment (active):

Actively evaluated by Artificial Analysis on current frontier models; no saturation evidence recorded.

Performance Timeline

Longitudinal progression of model scores against human baselines.

Performance & Historical Trajectory

Empirical score progression across model release dates and evaluation rounds.

Independent Vendor-Reported
ModelScoreDateSource TypeProvenance
Magistral Small 1.255.5%2026-11-15independentSource ↗
Llama 4 Maverick62.1%2026-11-15independentSource ↗
Nemotron 3 Nano Omni 30B A3B Reasoning53.2%2026-11-15independentSource ↗
Magistral Medium 1.259.7%2026-11-15independentSource ↗
DeepSeek V4 Flash Vision74.8%2026-11-15independentSource ↗
Gemma 4 E2B41.8%2026-11-15independentSource ↗
Gemini 3.8 Flash85.6%2026-11-15independentSource ↗
Inkling Small74%2026-11-15independentSource ↗
Ministral 3 14B49.8%2026-11-15independentSource ↗
Ministral 3 8B46%2026-11-15independentSource ↗
Qwen3.5 Omni Flash64.7%2026-11-15independentSource ↗
NVIDIA Nemotron Nano 12B v2 VL52.9%2026-11-15independentSource ↗
Gemma 4 26B A4B69.2%2026-11-15independentSource ↗
Ministral 3 3B38.1%2026-11-15independentSource ↗
Qwen3.6 35B A3B71%2026-11-15independentSource ↗
Kimi K3.1 Pro83.9%2026-11-15independentSource ↗
Nova 2.0 Lite63.8%2026-11-15independentSource ↗
Llama 3.2 Instruct 11B (Vision)29.3%2026-11-15independentSource ↗
Qwen3 Omni 30B A3B (Reasoning)60.2%2026-11-15independentSource ↗
Agnes 2.5 Pro Beta75.4%2026-11-15independentSource ↗
Inkling 2 Small81.6%2026-11-15independentSource ↗
Mistral Medium 3.564.9%2026-11-15independentSource ↗
Agnes 2.5 Pro Theta78.4%2026-11-15independentSource ↗
MiniMax M3.1 Flash79.1%2026-11-15independentSource ↗
Gemma 4 31B73.4%2026-11-15independentSource ↗
DiffusionGemma 26B A4B66.5%2026-11-15independentSource ↗
Gemini 3.1 Pro82.4%2026-11-15independentSource ↗
GPT-5.3-Codex78.5%2026-11-15independentSource ↗
Qwen3.8 Max (0902)82.8%2026-11-15independentSource ↗
Molmo 7B-D24.5%2026-11-15independentSource ↗
Llama 4 Scout52.9%2026-11-15independentSource ↗
Step3 VL 10B64%2026-11-15independentSource ↗
Qwen3.7 Plus80.5%2026-11-15independentSource ↗
ERNIE 5.0 Thinking64.6%2026-11-15independentSource ↗
Llama 3.2 Instruct 90B (Vision)39.5%2026-11-15independentSource ↗
Qwen3.8-Flash-Next78.6%2026-11-15independentSource ↗
Qwen3 Omni 30B A3B Instruct55.5%2026-11-15independentSource ↗
GPT-6 Luna79.7%2026-11-15independentSource ↗
o370.1%2026-11-15independentSource ↗
Mistral Small 456.8%2026-11-15independentSource ↗
MiMo-V2.6-Flash73.1%2026-11-15independentSource ↗
Qwen3.5 397B A17B77.3%2026-11-15independentSource ↗
Inkling73.5%2026-11-15independentSource ↗
Nova 2.0 Pro64.5%2026-11-15independentSource ↗
GPT-6.1 Terra84.7%2026-11-15independentSource ↗
GPT-6 Astra86.9%2026-11-15independentSource ↗
Qwen3.5 2B42.7%2026-11-15independentSource ↗
Step 576.4%2026-11-15independentSource ↗
Gemini 3.5 Flash-Lite79%2026-11-15independentSource ↗
GPT-6.1 Sol86%2026-11-15independentSource ↗
Apodex 1.179.2%2026-11-15independentSource ↗
Gemma 4 12B62%2026-11-15independentSource ↗
Qwen3.5-122B-A10B70.3%2026-11-15independentSource ↗
Nova 2.0 Omni61.9%2026-11-15independentSource ↗
LFM2.5-VL-1.6B26.5%2026-11-15independentSource ↗
Muse Glimmer74.3%2026-11-15independentSource ↗
DeepSeek V4.1 Flash 101780.6%2026-11-15independentSource ↗
Gemma 4 E4B51.2%2026-11-15independentSource ↗
Doubao Seed Code68.1%2026-11-15independentSource ↗
MiniCPM-V 4.6 1.3B37.9%2026-11-15independentSource ↗
Ling-3.0-flash-VL79%2026-11-15independentSource ↗
Qwen3.5 0.8B25.7%2026-11-15independentSource ↗
Molmo2-8B37.5%2026-11-15independentSource ↗
Qwen3.5 Omni Plus70.5%2026-11-15independentSource ↗
Muse Spark Open80.9%2026-11-15independentSource ↗
Qwen3.8 27B76.3%2026-11-15independentSource ↗
Qwen3.5 4B62.1%2026-11-15independentSource ↗
Claude Opus 5.587.7%2026-11-15independentSource ↗
Command A+63.2%2026-11-15independentSource ↗
Phi-4-multimodal14.5%2026-11-15independentSource ↗
MiniMax M378.6%2026-11-15independentSource ↗
Qwen3.5-9B66.8%2026-11-15independentSource ↗

Human Baseline & Difficulty Horizon

Calibrated human reference points, specialist benchmarks, and ceiling thresholds.
Measured Human Scoren/aNot established
Baseline Protocol & Interpretation

No peer-reviewed human baseline has been published for this benchmark.

Metric & Scoring Methodology

Verification protocols, aggregation formulas, and specialized metric variants.
Primary Metric:accuracy (%)
Scoring Engine:exact-match

Dataset & Compute Cost

Evaluation volume, public availability, API pricing, and local hardware requirements.
Total Dataset Size—Annotated evaluation items
Public Test SetPublicOpenly mirrored on repositories
Access GatingOpen AccessUnrestricted download
Evaluation LicenseOpen AccessDataset usage and redistribution terms
Frontier API Compute Cost:

$5 – $20 USD for full benchmark evaluation run on frontier APIs.

Recommended Local GPU Setup:

1x NVIDIA RTX 4090 (24GB) or A100 (40GB/80GB) via vLLM / SGLang

Official Dataset & Benchmark Files:Download / View Dataset Repository ↗

How to Run & Reproduce

Standardized evaluation protocols, CLI commands, and reproducible runner templates.
Prompt Regimezero-shot
Reasoning Modedirect
Sampling Temp0
Pass@k Budgetk = 1
Tools & SandboxPure Text
Scoring Verifierexact-match
Option AEleutherAI LM-Evaluation-Harness (Open-Weight Models)
lm_eval --model hf --model_args pretrained=<model_path> --tasks mmmu-pro --batch_size auto
Option BOpenCompass Evaluation Framework
opencompass --datasets mmmu-pro --models <model_config>
Python APIDeterministic Inference Loop Snippet
# Standard API Evaluation Loop
from openai import OpenAI

client = OpenAI()
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": prompt}],
    temperature=0.0,
)
Standardized Reporting Requirement:

When publishing results for MMMU-Pro, always report the exact prompt template, few-shot exemplar ordering, sampling temperature (temperature=0), maximum reasoning budget tokens, and the precise timestamped model snapshot ID.

Contamination & Memorization Analysis

Audit of pretraining exposure risks, memorization vectors, and refresh policies.
Overall Contamination Risk:MEDIUM
Refresh Cadence:

Static fixed snapshot

Test Set Exposure:

Public on web / HuggingFace