PutnamBench
Formal theorem-proving on Putnam problems — models write a complete Lean/Isabelle/Coq proof, machine-checked by the proof assistant's kernel.
The wiki's first formal-proving benchmark and, for reasonable budgets, still far from solved. Through most of 2024–2025 the best systems solved single digits of the Lean set (Goedel-Prover 7 at pass@512; GPT-4o ~1). General reasoning models at pass@1 remain low (GPT-5 ReAct ~28–42 of ~660, single digits %). Heavy-compute systems have since climbed steeply — ByteDance's Seed-Prover reportedly ~581/672 Lean at ~10 GPU-days per problem, and in Jan 2026 Logical Intelligence's Aleph CLAIMED 668/672 (99.4%, agentic pass@1) topping the board — but these use enormous or single-source, unreproduced budgets, and the Isabelle/Coq subsets and the harder 'no-answer' variant remain far less attacked. What would move this to nearing-saturation: independent reproduction of high solve rates across languages under disclosed, comparable budgets.
Performance Timeline
Longitudinal progression of model scores against human baselines.Performance & Historical Trajectory
Empirical score progression across model release dates and evaluation rounds.
Human Baseline & Difficulty Horizon
Calibrated human reference points, specialist benchmarks, and ceiling thresholds.Median human score among elite collegiate mathematics students on formal theorem proofs in Lean 4.
Metric & Scoring Methodology
Verification protocols, aggregation formulas, and specialized metric variants.problems with a complete, kernel-verified formal proof — Lean subset (count / 672)Dataset & Compute Cost
Evaluation volume, public availability, API pricing, and local hardware requirements.$5 – $20 USD for full benchmark evaluation run on frontier APIs.
1x NVIDIA RTX 4090 (24GB) or A100 (40GB/80GB) via vLLM / SGLang
How to Run & Reproduce
Standardized evaluation protocols, CLI commands, and reproducible runner templates.lm_eval --model hf --model_args pretrained=<model_path> --tasks putnambench --batch_size autoopencompass --datasets putnambench --models <model_config># Standard API Evaluation Loop
from openai import OpenAI
client = OpenAI()
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": prompt}],
temperature=0.0,
)When publishing results for PutnamBench, always report the exact prompt template, few-shot exemplar ordering, sampling temperature (temperature=0), maximum reasoning budget tokens, and the precise timestamped model snapshot ID.
Contamination & Memorization Analysis
Audit of pretraining exposure risks, memorization vectors, and refresh policies.Static fixed snapshot
Public on web / HuggingFace