KnowledgeActive
Trust Score:
90

Humanity's Last Exam

~2,500 expert-written questions across 100+ subjects, each filtered to stump frontier models — designed to be the final closed-ended academic benchmark.

Launched: —Refresh: static
Status Assessment (active):

The clearest active benchmark on the wiki. Launch scores were single digits (GPT-4o 3.3%, o1 9.1%, DeepSeek-R1 9.4%, no tools); the current no-tools frontier is ~38% (Gemini 3 Pro, Nov 2025) — a steep climb but a long way from solved. Given the 'last exam' thesis, saturation here would mean closed-ended academic testing has run out of headroom entirely, so 'what replaces it' points toward open-ended, agentic, and interactive evaluation rather than a harder Q&A set. Note: tool-augmented and agent scores run much higher and are tracked separately (see Timeline).

Performance Timeline

Longitudinal progression of model scores against human baselines.

Performance & Historical Trajectory

Empirical score progression across model release dates and evaluation rounds.

Independent Vendor-Reported Human Baseline (84.0%)
ModelScoreDateSource TypeProvenance
GPT-4o3.3%2025-01-24independentSource ↗
Claude 3.5 Sonnet4.3%2025-01-24independentSource ↗
o19.1%2025-01-24independentSource ↗
DeepSeek-R19.4%2025-01-24independentSource ↗
Grok 424.5%2024-06-01independentSource ↗
GPT-525.3%2024-06-01independentSource ↗
Gemini 3 Pro38.3%2024-06-01independentSource ↗

Human Baseline & Difficulty Horizon

Calibrated human reference points, specialist benchmarks, and ceiling thresholds.
Measured Human Score84.0%Domain Expert Baseline
Baseline Protocol & Interpretation

Cross-disciplinary academic experts score ~84% across questions outside their exact narrow specialty (authors achieve ~95%).

Metric & Scoring Methodology

Verification protocols, aggregation formulas, and specialized metric variants.
Primary Metric:accuracy, no tools (%)
Scoring Engine:llm-judge

Dataset & Compute Cost

Evaluation volume, public availability, API pricing, and local hardware requirements.
Total Dataset Size2,500Annotated evaluation items
Public Test SetPublicOpenly mirrored on repositories
Access GatingOpen AccessUnrestricted download
Evaluation LicenseOpen AccessDataset usage and redistribution terms
Frontier API Compute Cost:

$5 – $20 USD for full benchmark evaluation run on frontier APIs.

Recommended Local GPU Setup:

1x NVIDIA RTX 4090 (24GB) or A100 (40GB/80GB) via vLLM / SGLang

Official Dataset & Benchmark Files:Download / View Dataset Repository ↗

How to Run & Reproduce

Standardized evaluation protocols, CLI commands, and reproducible runner templates.
Prompt Regimezero-shot
Reasoning Modedirect
Sampling Temp0
Pass@k Budgetk = 1
Tools & SandboxPure Text
Scoring Verifierllm-judge
Option AEleutherAI LM-Evaluation-Harness (Open-Weight Models)
lm_eval --model hf --model_args pretrained=<model_path> --tasks humanitys-last-exam --batch_size auto
Option BOpenCompass Evaluation Framework
opencompass --datasets humanitys-last-exam --models <model_config>
Python APIDeterministic Inference Loop Snippet
# Standard API Evaluation Loop
from openai import OpenAI

client = OpenAI()
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": prompt}],
    temperature=0.0,
)
Standardized Reporting Requirement:

When publishing results for Humanity's Last Exam, always report the exact prompt template, few-shot exemplar ordering, sampling temperature (temperature=0), maximum reasoning budget tokens, and the precise timestamped model snapshot ID.

Contamination & Memorization Analysis

Audit of pretraining exposure risks, memorization vectors, and refresh policies.
Overall Contamination Risk:MEDIUM
Refresh Cadence:

Static fixed snapshot

Test Set Exposure:

Public on web / HuggingFace