ReasoningActive
Trust Score:
93

CritPt

Research-level physics reasoning benchmark of unpublished challenge problems written by physicists.

Launched: —Refresh: static
Status Assessment (active):

Actively evaluated by Artificial Analysis on current frontier models; no saturation evidence recorded.

Performance Timeline

Longitudinal progression of model scores against human baselines.

Performance & Historical Trajectory

Empirical score progression across model release dates and evaluation rounds.

Independent Vendor-Reported
ModelScoreDateSource TypeProvenance
JT-35B-Flash0%2026-11-15independentSource ↗
Qwen3.5 Omni Plus0.6%2026-11-15independentSource ↗
Nova 2.0 Omni0%2026-11-15independentSource ↗
Qwen3.5-122B-A10B0.9%2026-11-15independentSource ↗
Gemma 4 12B0%2026-11-15independentSource ↗
Claude Sonnet 5.531.4%2026-11-15independentSource ↗
Gemma 3 270M0%2026-11-15independentSource ↗
Apodex 1.14.6%2026-11-15independentSource ↗
GPT-6.1 Sol31.7%2026-11-15independentSource ↗
Gemini 3.5 Flash-Lite0%2026-11-15independentSource ↗
Step 520.9%2026-11-15independentSource ↗
Qwen3.5 2B0%2026-11-15independentSource ↗
Qwen3 Omni 30B A3B Instruct0%2026-11-15independentSource ↗
Granite 4.2 3B0%2026-11-15independentSource ↗
Claude Fable 5.235.4%2026-11-15independentSource ↗
GPT-6.1 Terra28%2026-11-15independentSource ↗
Nova 2.0 Pro0%2026-11-15independentSource ↗
Inkling5.4%2026-11-15independentSource ↗
Qwen3.5 397B A17B1.7%2026-11-15independentSource ↗
Granite 4.2 30B0.3%2026-11-15independentSource ↗
K2 Horizon 7B0.3%2026-11-15independentSource ↗
MiMo-V2.6-Flash12%2026-11-15independentSource ↗
Mistral Small 40.3%2026-11-15independentSource ↗
Solar Pro 46.3%2026-11-15independentSource ↗
gpt-oss-20b0%2026-11-15independentSource ↗
o31.1%2026-11-15independentSource ↗
K-EXAONE 2.00.9%2026-11-15independentSource ↗
GLM 5.3 Flash15.4%2026-11-15independentSource ↗
GPT-6 Luna19.4%2026-11-15independentSource ↗
North Mini Code0.3%2026-11-15independentSource ↗
GPT-6 Astra31.7%2026-11-15independentSource ↗
Granite 4.0 Micro0%2026-11-15independentSource ↗
Claude Haiku 5.518.9%2026-11-15independentSource ↗
Grok 4.717.7%2026-11-15independentSource ↗
Qwen3.5-9B0.6%2026-11-15independentSource ↗
Hermes 4 - Llama-3.1 70B0%2026-11-15independentSource ↗
Nemotron 3 Nano 30B A3B0%2026-11-15independentSource ↗
MiMo-V2.6-Pro26.6%2026-11-15independentSource ↗
Qwen3 Next 80B A3B Instruct0%2026-11-15independentSource ↗
MiniMax M33.7%2026-11-15independentSource ↗
Mistral Large 410%2026-11-15independentSource ↗
Nemotron 3.5 Lightning0%2026-11-15independentSource ↗
NVIDIA Nemotron Nano 9B V20%2026-11-15independentSource ↗
Command A+0.3%2026-11-15independentSource ↗
Claude Opus 5.531.7%2026-11-15independentSource ↗
EXAONE 4.0 32B0%2026-11-15independentSource ↗
G9v3-3B0%2026-11-15independentSource ↗
Qwen3.5 4B0%2026-11-15independentSource ↗
K2 Horizon 3.7B0%2026-11-15independentSource ↗
Olmo 3 7B Think0%2026-11-15independentSource ↗
HyperCLOVA X SEED Think (32B)0%2026-11-15independentSource ↗
HyperNova 60B 26050.9%2026-11-15independentSource ↗
Nova Micro0%2026-11-15independentSource ↗
Qwen3.8 27B5.4%2026-11-15independentSource ↗
Muse Spark Open14.9%2026-11-15independentSource ↗
INTELLECT-30.3%2026-11-15independentSource ↗
Muse Wonder28.3%2026-11-15independentSource ↗
Olmo 3.1 32B Think0%2026-11-15independentSource ↗
Nex-N2-Pro8.6%2026-11-15independentSource ↗
LFM2.5-VL-1.6B0%2026-11-15independentSource ↗
JT-MINI0%2026-11-15independentSource ↗
Molmo2-8B0%2026-11-15independentSource ↗
Qwen3.5 0.8B0%2026-11-15independentSource ↗
Ling-3.0-flash-VL2%2026-11-15independentSource ↗
MiniCPM-V 4.6 1.3B0%2026-11-15independentSource ↗
Sarvam 105B0%2026-11-15independentSource ↗
Llama 3.1 Instruct 405B0%2026-11-15independentSource ↗
Ring-flash-2.00.3%2026-11-15independentSource ↗
LFM2.5-1.2B-Instruct0%2026-11-15independentSource ↗
Gemini 4 Argon27.1%2026-11-15independentSource ↗
Apertus 70B Instruct0%2026-11-15independentSource ↗
Doubao Seed Code0.3%2026-11-15independentSource ↗
Gemma 4 E4B0.3%2026-11-15independentSource ↗
Ling 3.1 Flash 101926%2026-11-15independentSource ↗
K2 Horizon MoVA 36B A4B2.3%2026-11-15independentSource ↗
Granite 4.0 350M0%2026-11-15independentSource ↗
DeepSeek V4.1 Flash 101724%2026-11-15independentSource ↗
Muse Glimmer2.6%2026-11-15independentSource ↗
Solar Open 100B0%2026-11-15independentSource ↗
GPT-5.3-Codex16.9%2026-11-15independentSource ↗
LFM2.5-1.2B-Thinking0%2026-11-15independentSource ↗
K2 Think V20%2026-11-15independentSource ↗
Nemotron 3 Ultra3.1%2026-11-15independentSource ↗
LongCat 2.02.6%2026-11-15independentSource ↗
Tri-21B-Think0.3%2026-11-15independentSource ↗
Granite 4.0 H Small0%2026-11-15independentSource ↗
Llama 3.2 Instruct 11B (Vision)0%2026-11-15independentSource ↗
LongCat Flash Lite0%2026-11-15independentSource ↗
Nova 2.0 Lite0.3%2026-11-15independentSource ↗
Solar Mini 41.4%2026-11-15independentSource ↗
MiniCPM5-2B0.3%2026-11-15independentSource ↗
Kimi K3.1 Pro24.6%2026-11-15independentSource ↗
Motif-2-12.7B-Reasoning0%2026-11-15independentSource ↗
Qwen3.6 35B A3B0%2026-11-15independentSource ↗
Ministral 3 3B0%2026-11-15independentSource ↗
Beam 1.013.1%2026-11-15independentSource ↗
Ling-3.0-flash-Fin2.6%2026-11-15independentSource ↗
Gemma 4 26B A4B0%2026-11-15independentSource ↗
NVIDIA Nemotron Nano 12B v2 VL0%2026-11-15independentSource ↗
Qwen3.5 Omni Flash0%2026-11-15independentSource ↗
Magistral Small 1.20.3%2026-11-15independentSource ↗
Ministral 3 14B0%2026-11-15independentSource ↗
Inkling Small8.3%2026-11-15independentSource ↗
Gemini 3.8 Flash18.3%2026-11-15independentSource ↗
Gemma 4 E2B0%2026-11-15independentSource ↗
Sarvam 30B0.3%2026-11-15independentSource ↗
Mercury 2.50%2026-11-15independentSource ↗
DeepSeek V4 Flash Vision10.9%2026-11-15independentSource ↗
MiniCPM5-1B0%2026-11-15independentSource ↗
Jamba 1.7 Large0%2026-11-15independentSource ↗
Nemotron 3.5 Bolt 74B-A9B4.3%2026-11-15independentSource ↗
Nanbeige4.1-3B0%2026-11-15independentSource ↗
Solar Open2 250B5.7%2026-11-15independentSource ↗
Ling 3.0 Tiny0%2026-11-15independentSource ↗
Celeris-10%2026-11-15independentSource ↗
Tiny Aya Global0%2026-11-15independentSource ↗
A.X-K28.9%2026-11-15independentSource ↗
Exaone 4.0 1.2B0%2026-11-15independentSource ↗
Hermes 4 - Llama-3.1 405B0.3%2026-11-15independentSource ↗
DeepSeek V4-Pro18%2026-11-15independentSource ↗
Magistral Medium 1.20.3%2026-11-15independentSource ↗
Apertus 8B Instruct0%2026-11-15independentSource ↗
Nemotron 3 Nano Omni 30B A3B Reasoning0%2026-11-15independentSource ↗
Kimi K2.7 Code10%2026-11-15independentSource ↗
Qwen3.8 2.4T A95B20%2026-11-15independentSource ↗
Olmo 3 7B Instruct0%2026-11-15independentSource ↗
Llama 4 Maverick0%2026-11-15independentSource ↗
Olmo 3.1 32B Instruct0%2026-11-15independentSource ↗
Ministral 3 8B0%2026-11-15independentSource ↗
Llama 3.3 Instruct 70B0%2026-11-15independentSource ↗
Nemotron Cascade 2 30B A3B0.6%2026-11-15independentSource ↗
Qwen3.8-Flash-Next11.1%2026-11-15independentSource ↗
K2 Horizon 0.9B0%2026-11-15independentSource ↗
NVIDIA Nemotron 3 Nano 4B0%2026-11-15independentSource ↗
Phi-4-mini0%2026-11-15independentSource ↗
K2-V20%2026-11-15independentSource ↗
Gemini 3 Deep Think25.7%2026-11-15independentSource ↗
ERNIE 5.0 Thinking1.4%2026-11-15independentSource ↗
Ling 3.1 Flash18%2026-11-15independentSource ↗
Granite 4.0 H 350M0%2026-11-15independentSource ↗
Apriel-v1.6-15B-Thinker0.3%2026-11-15independentSource ↗
Qwen3.7 Plus9.1%2026-11-15independentSource ↗
Step3 VL 10B0%2026-11-15independentSource ↗
Llama 4 Scout0%2026-11-15independentSource ↗
Granite 4.2 8B0.3%2026-11-15independentSource ↗
ERNIE 4.5 300B A47B0%2026-11-15independentSource ↗
Motif 36.6%2026-11-15independentSource ↗
Qwen3.8 Max (0902)17.7%2026-11-15independentSource ↗
Hy34.9%2026-11-15independentSource ↗
Jamba 1.7 Mini0%2026-11-15independentSource ↗
Trinity Large Thinking0.9%2026-11-15independentSource ↗
Qwen3 Omni 30B A3B (Reasoning)0%2026-11-15independentSource ↗
K2 Horizon 375B A23B4.6%2026-11-15independentSource ↗
Gemini 3.1 Pro17.7%2026-11-15independentSource ↗
G9v3-39A5B0.3%2026-11-15independentSource ↗
DiffusionGemma 26B A4B0.3%2026-11-15independentSource ↗
GPT-5.5 Instant0%2026-11-15independentSource ↗
Falcon-H1R-7B0.3%2026-11-15independentSource ↗
Qwen3 Next 80B A3B (Reasoning)0%2026-11-15independentSource ↗
gpt-oss-120b1.1%2026-11-15independentSource ↗
GLM 5.319.1%2026-11-15independentSource ↗
Gemma 4 31B1.4%2026-11-15independentSource ↗
Llama 3.1 Nemotron Ultra 253B v10%2026-11-15independentSource ↗
MiniMax M3.1 Flash22.3%2026-11-15independentSource ↗
Agnes 2.5 Pro Theta15.1%2026-11-15independentSource ↗
Granite 4.0 H 1B0%2026-11-15independentSource ↗
Reka Flash 30%2026-11-15independentSource ↗
Mistral Medium 3.50%2026-11-15independentSource ↗
Mi:dm K 2.5 Pro0%2026-11-15independentSource ↗
LFM2.5-2.6B0%2026-11-15independentSource ↗
LFM2.5-8B-A1B0%2026-11-15independentSource ↗
Nemotron 3.5 Super VL10.6%2026-11-15independentSource ↗
GPT-5.5 Pro30.6%2026-11-15independentSource ↗
Nova Premier0%2026-11-15independentSource ↗
Muse Wonder Full32.9%2026-11-15independentSource ↗
Cogito v2.10%2026-11-15independentSource ↗
Ring-2.6-1T3.7%2026-11-15independentSource ↗
Phi-40%2026-11-15independentSource ↗
Agnes 2.5 Pro Beta15.7%2026-11-15independentSource ↗
Jamba Reasoning 3B0%2026-11-15independentSource ↗
Quasar 438B9.4%2026-11-15independentSource ↗
LFM2 24B A2B0%2026-11-15independentSource ↗
KAT Coder Pro V20%2026-11-15independentSource ↗
Qwen3 Coder Next0%2026-11-15independentSource ↗

Human Baseline & Difficulty Horizon

Calibrated human reference points, specialist benchmarks, and ceiling thresholds.
Measured Human Scoren/aNot established
Baseline Protocol & Interpretation

No peer-reviewed human baseline has been published for this benchmark.

Metric & Scoring Methodology

Verification protocols, aggregation formulas, and specialized metric variants.
Primary Metric:accuracy (%)
Scoring Engine:exact-match

Dataset & Compute Cost

Evaluation volume, public availability, API pricing, and local hardware requirements.
Total Dataset Size—Annotated evaluation items
Public Test SetPublicOpenly mirrored on repositories
Access GatingOpen AccessUnrestricted download
Evaluation LicenseOpen AccessDataset usage and redistribution terms
Frontier API Compute Cost:

$5 – $20 USD for full benchmark evaluation run on frontier APIs.

Recommended Local GPU Setup:

1x NVIDIA RTX 4090 (24GB) or A100 (40GB/80GB) via vLLM / SGLang

How to Run & Reproduce

Standardized evaluation protocols, CLI commands, and reproducible runner templates.
Prompt Regimezero-shot
Reasoning Modedirect
Sampling Temp0
Pass@k Budgetk = 1
Tools & SandboxPure Text
Scoring Verifierexact-match
Option AEleutherAI LM-Evaluation-Harness (Open-Weight Models)
lm_eval --model hf --model_args pretrained=<model_path> --tasks critpt --batch_size auto
Option BOpenCompass Evaluation Framework
opencompass --datasets critpt --models <model_config>
Python APIDeterministic Inference Loop Snippet
# Standard API Evaluation Loop
from openai import OpenAI

client = OpenAI()
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": prompt}],
    temperature=0.0,
)
Standardized Reporting Requirement:

When publishing results for CritPt, always report the exact prompt template, few-shot exemplar ordering, sampling temperature (temperature=0), maximum reasoning budget tokens, and the precise timestamped model snapshot ID.

Contamination & Memorization Analysis

Audit of pretraining exposure risks, memorization vectors, and refresh policies.
Overall Contamination Risk:MEDIUM
Refresh Cadence:

Static fixed snapshot

Test Set Exposure:

Public on web / HuggingFace