RULER
Configurable synthetic benchmark that expands needle retrieval into multi-needle, multi-hop tracing, and aggregation tasks.
92 AI benchmarks with their saturation status, contamination risk and the source of every score. Filter the list, or scan the trust matrix below to see which capabilities still have benchmarks that separate frontier models.
Configurable synthetic benchmark that expands needle retrieval into multi-needle, multi-hop tracing, and aggregation tasks.
707 instructions from 50 real-world agents, averaging 1,723 words and 11.9 constraints — whether instruction following survives deployment-scale prompts.
Bilingual benchmark of how models follow instructions when several constraints must hold at once — where stacking constraints is not linear.
PhD-written science questions so hard that skilled non-experts with Google score 34% — the 'Google-proof' exam, reported on its 198-question Diamond subset.
Zero-shot long-text suite derived from SCROLLS with test-only tasks and new aggregation-style long-context challenges.
Multiple-choice long-document QA where writers and validators read the full article, with a hard subset designed to defeat skimming.
The harder static ARC successor: human-calibrated colored-grid abstraction tasks, now pressured by 2026 frontier systems.
155 multi-turn dialogues scored on fine-grained constraints plus six inter-turn dialogue relationships (recall, refinement, expansion, follow-up, summary).
MMLU rebuilt to fix its flaws: 10 options instead of 4, trivia and label noise filtered out, and reasoning-heavy questions that reward chain-of-thought.
The MAA's competition exam — 15 integer-answer problems per test, refreshed yearly so each new exam is briefly contamination-free.
80 curated multi-turn questions across 8 skills, each a two-turn conversation scored by a strong LLM judge on a 1–10 scale.
100k+ token benchmark mixing realistic and synthetic tasks across English, Chinese, code, math, and retrieval.
A cleaned 378-task MBPP evaluation with roughly 35 times more tests for stricter Python functional correctness.
Bilingual multitask suite for long-context understanding across QA, summarization, few-shot learning, synthetic tasks, and code completion.
A human-translated 250-problem GSM8K subset for comparing grade-school mathematical reasoning across ten languages.
Seven-task long-sequence suite covering summarization, QA, and NLI over naturally long English texts.
Scientific-paper QA over full NLP papers, with abstractive, extractive, yes/no, and unanswerable answers plus supporting evidence.
Query-focused meeting summarization over long transcripts from academic, product, and committee meetings.
Every benchmark by capability and status. Click a benchmark for its evidence and scores.