Needle-in-a-Haystack
Synthetic retrieval stress test that hides facts at different depths and context lengths, then asks the model to recover them.
92 AI benchmarks with their saturation status, contamination risk and the source of every score. Filter the list, or scan the trust matrix below to see which capabilities still have benchmarks that separate frontier models.
Synthetic retrieval stress test that hides facts at different depths and context lengths, then asks the model to recover them.
Continuously updated contest problems for time-windowed code generation, execution, test prediction, and self-repair evaluation.
Binary fill-in-the-blank commonsense problems, scaled from Winograd schemas and adversarially filtered to reduce dataset shortcuts.
Choose the plausible continuation of an everyday event from one real ending and three adversarially filtered machine generations.
Five-choice questions derived from ConceptNet relations, designed to require everyday knowledge beyond a supplied passage.
Nine-task English NLU suite combining acceptability, sentiment, similarity, paraphrase, inference and coreference into one score.
A 500-task, engineer-reviewed subset of SWE-bench that became the standard repository-level coding-agent test.
A manual error audit of MMLU: experts re-annotated thousands of questions and found ~6.5% are broken — the evidence base for MMLU's label-noise ceiling.
HumanEval's 164 Python tasks re-evaluated with roughly 80 times more tests to expose plausible but incorrect programs.
Standardized human exams — SATs, LSATs, China's Gaokao and civil-service tests — repurposed to grade models against real human test-takers.
23 BIG-Bench tasks where models trailed humans — the benchmark that proved chain-of-thought prompting works, then fell to the reasoning-model era.
Grade-level math word problems that require combining a question with structured, textual, or image-rendered tables.
8.5K grade-school math word problems (2–8 arithmetic steps) — the default elementary-math benchmark and the classic chain-of-thought demonstration.
817 adversarial questions probing whether models repeat common human misconceptions — famous for finding that bigger models were often less truthful.
Mostly Basic Programming Problems: 974 crowd-sourced Python synthesis tasks intended for entry-level programmers.
OpenAI's 164 hand-written Python function-synthesis tasks, graded by executing generated completions against unit tests.
12,500 AMC/AIME-level competition problems with worked solutions — usually evaluated on its 500-problem MATH-500 subset, the reasoning-model math standard.
57-subject multiple-choice exam spanning STEM, humanities, and social sciences — the defining knowledge benchmark of the 2020–2024 era.
Colored-grid abstraction tasks that test few-shot rule induction, not memorized facts, language fluency, or school math.
Eight English NLU tasks spanning question answering, inference, causal reasoning, word sense, and coreference.
Every benchmark by capability and status. Click a benchmark for its evidence and scores.