ReasoningActive
Trust Score:
93
CritPt
Research-level physics reasoning benchmark of unpublished challenge problems written by physicists.
Launched: —Refresh: static
Status Assessment (active):
Actively evaluated by Artificial Analysis on current frontier models; no saturation evidence recorded.
Performance Timeline
Longitudinal progression of model scores against human baselines.Performance & Historical Trajectory
Empirical score progression across model release dates and evaluation rounds.
Independent Vendor-Reported
| Model | Score | Date | Source Type | Provenance |
|---|---|---|---|---|
| JT-35B-Flash | 0% | 2026-11-15 | independent | Source ↗ |
| Qwen3.5 Omni Plus | 0.6% | 2026-11-15 | independent | Source ↗ |
| Nova 2.0 Omni | 0% | 2026-11-15 | independent | Source ↗ |
| Qwen3.5-122B-A10B | 0.9% | 2026-11-15 | independent | Source ↗ |
| Gemma 4 12B | 0% | 2026-11-15 | independent | Source ↗ |
| Claude Sonnet 5.5 | 31.4% | 2026-11-15 | independent | Source ↗ |
| Gemma 3 270M | 0% | 2026-11-15 | independent | Source ↗ |
| Apodex 1.1 | 4.6% | 2026-11-15 | independent | Source ↗ |
| GPT-6.1 Sol | 31.7% | 2026-11-15 | independent | Source ↗ |
| Gemini 3.5 Flash-Lite | 0% | 2026-11-15 | independent | Source ↗ |
| Step 5 | 20.9% | 2026-11-15 | independent | Source ↗ |
| Qwen3.5 2B | 0% | 2026-11-15 | independent | Source ↗ |
| Qwen3 Omni 30B A3B Instruct | 0% | 2026-11-15 | independent | Source ↗ |
| Granite 4.2 3B | 0% | 2026-11-15 | independent | Source ↗ |
| Claude Fable 5.2 | 35.4% | 2026-11-15 | independent | Source ↗ |
| GPT-6.1 Terra | 28% | 2026-11-15 | independent | Source ↗ |
| Nova 2.0 Pro | 0% | 2026-11-15 | independent | Source ↗ |
| Inkling | 5.4% | 2026-11-15 | independent | Source ↗ |
| Qwen3.5 397B A17B | 1.7% | 2026-11-15 | independent | Source ↗ |
| Granite 4.2 30B | 0.3% | 2026-11-15 | independent | Source ↗ |
| K2 Horizon 7B | 0.3% | 2026-11-15 | independent | Source ↗ |
| MiMo-V2.6-Flash | 12% | 2026-11-15 | independent | Source ↗ |
| Mistral Small 4 | 0.3% | 2026-11-15 | independent | Source ↗ |
| Solar Pro 4 | 6.3% | 2026-11-15 | independent | Source ↗ |
| gpt-oss-20b | 0% | 2026-11-15 | independent | Source ↗ |
| o3 | 1.1% | 2026-11-15 | independent | Source ↗ |
| K-EXAONE 2.0 | 0.9% | 2026-11-15 | independent | Source ↗ |
| GLM 5.3 Flash | 15.4% | 2026-11-15 | independent | Source ↗ |
| GPT-6 Luna | 19.4% | 2026-11-15 | independent | Source ↗ |
| North Mini Code | 0.3% | 2026-11-15 | independent | Source ↗ |
| GPT-6 Astra | 31.7% | 2026-11-15 | independent | Source ↗ |
| Granite 4.0 Micro | 0% | 2026-11-15 | independent | Source ↗ |
| Claude Haiku 5.5 | 18.9% | 2026-11-15 | independent | Source ↗ |
| Grok 4.7 | 17.7% | 2026-11-15 | independent | Source ↗ |
| Qwen3.5-9B | 0.6% | 2026-11-15 | independent | Source ↗ |
| Hermes 4 - Llama-3.1 70B | 0% | 2026-11-15 | independent | Source ↗ |
| Nemotron 3 Nano 30B A3B | 0% | 2026-11-15 | independent | Source ↗ |
| MiMo-V2.6-Pro | 26.6% | 2026-11-15 | independent | Source ↗ |
| Qwen3 Next 80B A3B Instruct | 0% | 2026-11-15 | independent | Source ↗ |
| MiniMax M3 | 3.7% | 2026-11-15 | independent | Source ↗ |
| Mistral Large 4 | 10% | 2026-11-15 | independent | Source ↗ |
| Nemotron 3.5 Lightning | 0% | 2026-11-15 | independent | Source ↗ |
| NVIDIA Nemotron Nano 9B V2 | 0% | 2026-11-15 | independent | Source ↗ |
| Command A+ | 0.3% | 2026-11-15 | independent | Source ↗ |
| Claude Opus 5.5 | 31.7% | 2026-11-15 | independent | Source ↗ |
| EXAONE 4.0 32B | 0% | 2026-11-15 | independent | Source ↗ |
| G9v3-3B | 0% | 2026-11-15 | independent | Source ↗ |
| Qwen3.5 4B | 0% | 2026-11-15 | independent | Source ↗ |
| K2 Horizon 3.7B | 0% | 2026-11-15 | independent | Source ↗ |
| Olmo 3 7B Think | 0% | 2026-11-15 | independent | Source ↗ |
| HyperCLOVA X SEED Think (32B) | 0% | 2026-11-15 | independent | Source ↗ |
| HyperNova 60B 2605 | 0.9% | 2026-11-15 | independent | Source ↗ |
| Nova Micro | 0% | 2026-11-15 | independent | Source ↗ |
| Qwen3.8 27B | 5.4% | 2026-11-15 | independent | Source ↗ |
| Muse Spark Open | 14.9% | 2026-11-15 | independent | Source ↗ |
| INTELLECT-3 | 0.3% | 2026-11-15 | independent | Source ↗ |
| Muse Wonder | 28.3% | 2026-11-15 | independent | Source ↗ |
| Olmo 3.1 32B Think | 0% | 2026-11-15 | independent | Source ↗ |
| Nex-N2-Pro | 8.6% | 2026-11-15 | independent | Source ↗ |
| LFM2.5-VL-1.6B | 0% | 2026-11-15 | independent | Source ↗ |
| JT-MINI | 0% | 2026-11-15 | independent | Source ↗ |
| Molmo2-8B | 0% | 2026-11-15 | independent | Source ↗ |
| Qwen3.5 0.8B | 0% | 2026-11-15 | independent | Source ↗ |
| Ling-3.0-flash-VL | 2% | 2026-11-15 | independent | Source ↗ |
| MiniCPM-V 4.6 1.3B | 0% | 2026-11-15 | independent | Source ↗ |
| Sarvam 105B | 0% | 2026-11-15 | independent | Source ↗ |
| Llama 3.1 Instruct 405B | 0% | 2026-11-15 | independent | Source ↗ |
| Ring-flash-2.0 | 0.3% | 2026-11-15 | independent | Source ↗ |
| LFM2.5-1.2B-Instruct | 0% | 2026-11-15 | independent | Source ↗ |
| Gemini 4 Argon | 27.1% | 2026-11-15 | independent | Source ↗ |
| Apertus 70B Instruct | 0% | 2026-11-15 | independent | Source ↗ |
| Doubao Seed Code | 0.3% | 2026-11-15 | independent | Source ↗ |
| Gemma 4 E4B | 0.3% | 2026-11-15 | independent | Source ↗ |
| Ling 3.1 Flash 1019 | 26% | 2026-11-15 | independent | Source ↗ |
| K2 Horizon MoVA 36B A4B | 2.3% | 2026-11-15 | independent | Source ↗ |
| Granite 4.0 350M | 0% | 2026-11-15 | independent | Source ↗ |
| DeepSeek V4.1 Flash 1017 | 24% | 2026-11-15 | independent | Source ↗ |
| Muse Glimmer | 2.6% | 2026-11-15 | independent | Source ↗ |
| Solar Open 100B | 0% | 2026-11-15 | independent | Source ↗ |
| GPT-5.3-Codex | 16.9% | 2026-11-15 | independent | Source ↗ |
| LFM2.5-1.2B-Thinking | 0% | 2026-11-15 | independent | Source ↗ |
| K2 Think V2 | 0% | 2026-11-15 | independent | Source ↗ |
| Nemotron 3 Ultra | 3.1% | 2026-11-15 | independent | Source ↗ |
| LongCat 2.0 | 2.6% | 2026-11-15 | independent | Source ↗ |
| Tri-21B-Think | 0.3% | 2026-11-15 | independent | Source ↗ |
| Granite 4.0 H Small | 0% | 2026-11-15 | independent | Source ↗ |
| Llama 3.2 Instruct 11B (Vision) | 0% | 2026-11-15 | independent | Source ↗ |
| LongCat Flash Lite | 0% | 2026-11-15 | independent | Source ↗ |
| Nova 2.0 Lite | 0.3% | 2026-11-15 | independent | Source ↗ |
| Solar Mini 4 | 1.4% | 2026-11-15 | independent | Source ↗ |
| MiniCPM5-2B | 0.3% | 2026-11-15 | independent | Source ↗ |
| Kimi K3.1 Pro | 24.6% | 2026-11-15 | independent | Source ↗ |
| Motif-2-12.7B-Reasoning | 0% | 2026-11-15 | independent | Source ↗ |
| Qwen3.6 35B A3B | 0% | 2026-11-15 | independent | Source ↗ |
| Ministral 3 3B | 0% | 2026-11-15 | independent | Source ↗ |
| Beam 1.0 | 13.1% | 2026-11-15 | independent | Source ↗ |
| Ling-3.0-flash-Fin | 2.6% | 2026-11-15 | independent | Source ↗ |
| Gemma 4 26B A4B | 0% | 2026-11-15 | independent | Source ↗ |
| NVIDIA Nemotron Nano 12B v2 VL | 0% | 2026-11-15 | independent | Source ↗ |
| Qwen3.5 Omni Flash | 0% | 2026-11-15 | independent | Source ↗ |
| Magistral Small 1.2 | 0.3% | 2026-11-15 | independent | Source ↗ |
| Ministral 3 14B | 0% | 2026-11-15 | independent | Source ↗ |
| Inkling Small | 8.3% | 2026-11-15 | independent | Source ↗ |
| Gemini 3.8 Flash | 18.3% | 2026-11-15 | independent | Source ↗ |
| Gemma 4 E2B | 0% | 2026-11-15 | independent | Source ↗ |
| Sarvam 30B | 0.3% | 2026-11-15 | independent | Source ↗ |
| Mercury 2.5 | 0% | 2026-11-15 | independent | Source ↗ |
| DeepSeek V4 Flash Vision | 10.9% | 2026-11-15 | independent | Source ↗ |
| MiniCPM5-1B | 0% | 2026-11-15 | independent | Source ↗ |
| Jamba 1.7 Large | 0% | 2026-11-15 | independent | Source ↗ |
| Nemotron 3.5 Bolt 74B-A9B | 4.3% | 2026-11-15 | independent | Source ↗ |
| Nanbeige4.1-3B | 0% | 2026-11-15 | independent | Source ↗ |
| Solar Open2 250B | 5.7% | 2026-11-15 | independent | Source ↗ |
| Ling 3.0 Tiny | 0% | 2026-11-15 | independent | Source ↗ |
| Celeris-1 | 0% | 2026-11-15 | independent | Source ↗ |
| Tiny Aya Global | 0% | 2026-11-15 | independent | Source ↗ |
| A.X-K2 | 8.9% | 2026-11-15 | independent | Source ↗ |
| Exaone 4.0 1.2B | 0% | 2026-11-15 | independent | Source ↗ |
| Hermes 4 - Llama-3.1 405B | 0.3% | 2026-11-15 | independent | Source ↗ |
| DeepSeek V4-Pro | 18% | 2026-11-15 | independent | Source ↗ |
| Magistral Medium 1.2 | 0.3% | 2026-11-15 | independent | Source ↗ |
| Apertus 8B Instruct | 0% | 2026-11-15 | independent | Source ↗ |
| Nemotron 3 Nano Omni 30B A3B Reasoning | 0% | 2026-11-15 | independent | Source ↗ |
| Kimi K2.7 Code | 10% | 2026-11-15 | independent | Source ↗ |
| Qwen3.8 2.4T A95B | 20% | 2026-11-15 | independent | Source ↗ |
| Olmo 3 7B Instruct | 0% | 2026-11-15 | independent | Source ↗ |
| Llama 4 Maverick | 0% | 2026-11-15 | independent | Source ↗ |
| Olmo 3.1 32B Instruct | 0% | 2026-11-15 | independent | Source ↗ |
| Ministral 3 8B | 0% | 2026-11-15 | independent | Source ↗ |
| Llama 3.3 Instruct 70B | 0% | 2026-11-15 | independent | Source ↗ |
| Nemotron Cascade 2 30B A3B | 0.6% | 2026-11-15 | independent | Source ↗ |
| Qwen3.8-Flash-Next | 11.1% | 2026-11-15 | independent | Source ↗ |
| K2 Horizon 0.9B | 0% | 2026-11-15 | independent | Source ↗ |
| NVIDIA Nemotron 3 Nano 4B | 0% | 2026-11-15 | independent | Source ↗ |
| Phi-4-mini | 0% | 2026-11-15 | independent | Source ↗ |
| K2-V2 | 0% | 2026-11-15 | independent | Source ↗ |
| Gemini 3 Deep Think | 25.7% | 2026-11-15 | independent | Source ↗ |
| ERNIE 5.0 Thinking | 1.4% | 2026-11-15 | independent | Source ↗ |
| Ling 3.1 Flash | 18% | 2026-11-15 | independent | Source ↗ |
| Granite 4.0 H 350M | 0% | 2026-11-15 | independent | Source ↗ |
| Apriel-v1.6-15B-Thinker | 0.3% | 2026-11-15 | independent | Source ↗ |
| Qwen3.7 Plus | 9.1% | 2026-11-15 | independent | Source ↗ |
| Step3 VL 10B | 0% | 2026-11-15 | independent | Source ↗ |
| Llama 4 Scout | 0% | 2026-11-15 | independent | Source ↗ |
| Granite 4.2 8B | 0.3% | 2026-11-15 | independent | Source ↗ |
| ERNIE 4.5 300B A47B | 0% | 2026-11-15 | independent | Source ↗ |
| Motif 3 | 6.6% | 2026-11-15 | independent | Source ↗ |
| Qwen3.8 Max (0902) | 17.7% | 2026-11-15 | independent | Source ↗ |
| Hy3 | 4.9% | 2026-11-15 | independent | Source ↗ |
| Jamba 1.7 Mini | 0% | 2026-11-15 | independent | Source ↗ |
| Trinity Large Thinking | 0.9% | 2026-11-15 | independent | Source ↗ |
| Qwen3 Omni 30B A3B (Reasoning) | 0% | 2026-11-15 | independent | Source ↗ |
| K2 Horizon 375B A23B | 4.6% | 2026-11-15 | independent | Source ↗ |
| Gemini 3.1 Pro | 17.7% | 2026-11-15 | independent | Source ↗ |
| G9v3-39A5B | 0.3% | 2026-11-15 | independent | Source ↗ |
| DiffusionGemma 26B A4B | 0.3% | 2026-11-15 | independent | Source ↗ |
| GPT-5.5 Instant | 0% | 2026-11-15 | independent | Source ↗ |
| Falcon-H1R-7B | 0.3% | 2026-11-15 | independent | Source ↗ |
| Qwen3 Next 80B A3B (Reasoning) | 0% | 2026-11-15 | independent | Source ↗ |
| gpt-oss-120b | 1.1% | 2026-11-15 | independent | Source ↗ |
| GLM 5.3 | 19.1% | 2026-11-15 | independent | Source ↗ |
| Gemma 4 31B | 1.4% | 2026-11-15 | independent | Source ↗ |
| Llama 3.1 Nemotron Ultra 253B v1 | 0% | 2026-11-15 | independent | Source ↗ |
| MiniMax M3.1 Flash | 22.3% | 2026-11-15 | independent | Source ↗ |
| Agnes 2.5 Pro Theta | 15.1% | 2026-11-15 | independent | Source ↗ |
| Granite 4.0 H 1B | 0% | 2026-11-15 | independent | Source ↗ |
| Reka Flash 3 | 0% | 2026-11-15 | independent | Source ↗ |
| Mistral Medium 3.5 | 0% | 2026-11-15 | independent | Source ↗ |
| Mi:dm K 2.5 Pro | 0% | 2026-11-15 | independent | Source ↗ |
| LFM2.5-2.6B | 0% | 2026-11-15 | independent | Source ↗ |
| LFM2.5-8B-A1B | 0% | 2026-11-15 | independent | Source ↗ |
| Nemotron 3.5 Super VL | 10.6% | 2026-11-15 | independent | Source ↗ |
| GPT-5.5 Pro | 30.6% | 2026-11-15 | independent | Source ↗ |
| Nova Premier | 0% | 2026-11-15 | independent | Source ↗ |
| Muse Wonder Full | 32.9% | 2026-11-15 | independent | Source ↗ |
| Cogito v2.1 | 0% | 2026-11-15 | independent | Source ↗ |
| Ring-2.6-1T | 3.7% | 2026-11-15 | independent | Source ↗ |
| Phi-4 | 0% | 2026-11-15 | independent | Source ↗ |
| Agnes 2.5 Pro Beta | 15.7% | 2026-11-15 | independent | Source ↗ |
| Jamba Reasoning 3B | 0% | 2026-11-15 | independent | Source ↗ |
| Quasar 438B | 9.4% | 2026-11-15 | independent | Source ↗ |
| LFM2 24B A2B | 0% | 2026-11-15 | independent | Source ↗ |
| KAT Coder Pro V2 | 0% | 2026-11-15 | independent | Source ↗ |
| Qwen3 Coder Next | 0% | 2026-11-15 | independent | Source ↗ |
Human Baseline & Difficulty Horizon
Calibrated human reference points, specialist benchmarks, and ceiling thresholds.Measured Human Scoren/aNot established
Baseline Protocol & Interpretation
No peer-reviewed human baseline has been published for this benchmark.
Metric & Scoring Methodology
Verification protocols, aggregation formulas, and specialized metric variants.Primary Metric:
accuracy (%)Scoring Engine:exact-match
Dataset & Compute Cost
Evaluation volume, public availability, API pricing, and local hardware requirements.Total Dataset Size—Annotated evaluation items
Public Test SetPublicOpenly mirrored on repositories
Access GatingOpen AccessUnrestricted download
Evaluation LicenseOpen AccessDataset usage and redistribution terms
Frontier API Compute Cost:
$5 – $20 USD for full benchmark evaluation run on frontier APIs.
Recommended Local GPU Setup:
1x NVIDIA RTX 4090 (24GB) or A100 (40GB/80GB) via vLLM / SGLang
How to Run & Reproduce
Standardized evaluation protocols, CLI commands, and reproducible runner templates.Prompt Regimezero-shot
Reasoning Modedirect
Sampling Temp0
Pass@k Budgetk = 1
Tools & SandboxPure Text
Scoring Verifierexact-match
Option AEleutherAI LM-Evaluation-Harness (Open-Weight Models)
lm_eval --model hf --model_args pretrained=<model_path> --tasks critpt --batch_size autoOption BOpenCompass Evaluation Framework
opencompass --datasets critpt --models <model_config>Python APIDeterministic Inference Loop Snippet
# Standard API Evaluation Loop
from openai import OpenAI
client = OpenAI()
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": prompt}],
temperature=0.0,
)Standardized Reporting Requirement:
When publishing results for CritPt, always report the exact prompt template, few-shot exemplar ordering, sampling temperature (temperature=0), maximum reasoning budget tokens, and the precise timestamped model snapshot ID.
Contamination & Memorization Analysis
Audit of pretraining exposure risks, memorization vectors, and refresh policies.Overall Contamination Risk:MEDIUM
Refresh Cadence:
Static fixed snapshot
Test Set Exposure:
Public on web / HuggingFace