KnowledgeActive
Trust Score:
93

HLE-Diamond

Curated 'diamond' subset of Humanity's Last Exam evaluated by Artificial Analysis.

Launched: —Refresh: static
Status Assessment (active):

Actively evaluated by Artificial Analysis on current frontier models; no saturation evidence recorded.

Performance Timeline

Longitudinal progression of model scores against human baselines.

Performance & Historical Trajectory

Empirical score progression across model release dates and evaluation rounds.

Independent Vendor-Reported
ModelScoreDateSource TypeProvenance
Muse Ember25.1%2026-11-15independentSource ↗
Qwen3.7 Plus13.9%2026-11-15independentSource ↗
NVIDIA Nemotron 3 Nano 4B2.8%2026-11-15independentSource ↗
Qwen3.8-Flash-Next16.3%2026-11-15independentSource ↗
Ministral 3 8B4.4%2026-11-15independentSource ↗
Qwen3.8 2.4T A95B20.1%2026-11-15independentSource ↗
Kimi K2.7 Code14.8%2026-11-15independentSource ↗
Gemini 4 Hydrogen58.8%2026-11-15independentSource ↗
DeepSeek V4-Pro19.5%2026-11-15independentSource ↗
Celeris-15.7%2026-11-15independentSource ↗
Granite 4.2 8B4%2026-11-15independentSource ↗
DeepSeek V4 Flash Vision21.1%2026-11-15independentSource ↗
Mercury 2.55.1%2026-11-15independentSource ↗
Gemini 3.8 Flash33.1%2026-11-15independentSource ↗
Inkling Small12.1%2026-11-15independentSource ↗
Ministral 3 14B4.4%2026-11-15independentSource ↗
Gemma 4 26B A4B6.2%2026-11-15independentSource ↗
Ling-3.0-flash-Fin5.4%2026-11-15independentSource ↗
Beam 1.012.8%2026-11-15independentSource ↗
Ministral 3 3B5.1%2026-11-15independentSource ↗
Gemma 4 31B7.2%2026-11-15independentSource ↗
Qwen3 Coder Next4.3%2026-11-15independentSource ↗
Quasar 438B13.1%2026-11-15independentSource ↗
Muse Wonder Full48.5%2026-11-15independentSource ↗
Inkling 2 Small28.7%2026-11-15independentSource ↗
Nemotron 3.5 Super VL8.8%2026-11-15independentSource ↗
Mistral Medium 3.54.7%2026-11-15independentSource ↗
Agnes 2.5 Pro Theta16.8%2026-11-15independentSource ↗
MiniMax M3.1 Flash28.1%2026-11-15independentSource ↗
Hy432.7%2026-11-15independentSource ↗
Kimi K3.1 Pro37.3%2026-11-15independentSource ↗
GLM 5.317.9%2026-11-15independentSource ↗
gpt-oss-120b3.8%2026-11-15independentSource ↗
GPT-5.5 Instant14.6%2026-11-15independentSource ↗
Gemini 3.1 Pro31.4%2026-11-15independentSource ↗
K2 Horizon 375B A23B10.2%2026-11-15independentSource ↗
Trinity Large Thinking6%2026-11-15independentSource ↗
JT-4.1 Flash 236B A21B16.4%2026-11-15independentSource ↗
Hy311.6%2026-11-15independentSource ↗
Qwen3.8 Max (0902)24%2026-11-15independentSource ↗
Qwen3.5 397B A17B12.4%2026-11-15independentSource ↗
GPT-6 Astra63.8%2026-11-15independentSource ↗
North Mini Code4.2%2026-11-15independentSource ↗
GPT-6 Luna22.2%2026-11-15independentSource ↗
GLM 5.3 Flash18.2%2026-11-15independentSource ↗
K-EXAONE 2.05.7%2026-11-15independentSource ↗
Solar Pro 44.7%2026-11-15independentSource ↗
Mistral Small 44.4%2026-11-15independentSource ↗
MiMo-V2.6-Flash13.7%2026-11-15independentSource ↗
Granite 4.2 30B4.3%2026-11-15independentSource ↗
Claude Haiku 5.529.6%2026-11-15independentSource ↗
Inkling10.6%2026-11-15independentSource ↗
Gemini 4 Carbon69.1%2026-11-15independentSource ↗
GPT-6.1 Terra47.3%2026-11-15independentSource ↗
Claude Fable 5.280.2%2026-11-15independentSource ↗
Granite 4.2 3B2.9%2026-11-15independentSource ↗
Step 524.3%2026-11-15independentSource ↗
Gemini 3.5 Flash-Lite9.9%2026-11-15independentSource ↗
GPT-6.1 Sol58.8%2026-11-15independentSource ↗
Apodex 1.111.6%2026-11-15independentSource ↗
Muse Wonder39%2026-11-15independentSource ↗
MiniCPM5-2B3.8%2026-11-15independentSource ↗
LongCat 2.011.5%2026-11-15independentSource ↗
Nemotron 3 Ultra8.5%2026-11-15independentSource ↗
Muse Glimmer9.1%2026-11-15independentSource ↗
DeepSeek V4.1 Flash 101725.3%2026-11-15independentSource ↗
Ling 3.1 Flash 101917.2%2026-11-15independentSource ↗
Nanbeige4.3-33B6.9%2026-11-15independentSource ↗
Gemini 4 Argon56.7%2026-11-15independentSource ↗
Ling-3.0-flash-VL7.6%2026-11-15independentSource ↗
Claude Sonnet 5.553.3%2026-11-15independentSource ↗
Muse Spark Open26.3%2026-11-15independentSource ↗
Qwen3.8 27B15.3%2026-11-15independentSource ↗
Claude Opus 5.567.9%2026-11-15independentSource ↗
Command A+5.1%2026-11-15independentSource ↗
Nemotron 3.5 Lightning4.5%2026-11-15independentSource ↗
Mistral Large 417.3%2026-11-15independentSource ↗
MiniMax M316%2026-11-15independentSource ↗
MiMo-V2.6-Pro15.2%2026-11-15independentSource ↗
Grok 4.723.2%2026-11-15independentSource ↗

Human Baseline & Difficulty Horizon

Calibrated human reference points, specialist benchmarks, and ceiling thresholds.
Measured Human Scoren/aNot established
Baseline Protocol & Interpretation

No peer-reviewed human baseline has been published for this benchmark.

Metric & Scoring Methodology

Verification protocols, aggregation formulas, and specialized metric variants.
Primary Metric:accuracy (%)
Scoring Engine:llm-judge

Dataset & Compute Cost

Evaluation volume, public availability, API pricing, and local hardware requirements.
Total Dataset Size—Annotated evaluation items
Public Test SetPublicOpenly mirrored on repositories
Access GatingOpen AccessUnrestricted download
Evaluation LicenseOpen AccessDataset usage and redistribution terms
Frontier API Compute Cost:

$5 – $20 USD for full benchmark evaluation run on frontier APIs.

Recommended Local GPU Setup:

1x NVIDIA RTX 4090 (24GB) or A100 (40GB/80GB) via vLLM / SGLang

How to Run & Reproduce

Standardized evaluation protocols, CLI commands, and reproducible runner templates.
Prompt Regimezero-shot
Reasoning Modedirect
Sampling Temp0
Pass@k Budgetk = 1
Tools & SandboxPure Text
Scoring Verifierllm-judge
Option AEleutherAI LM-Evaluation-Harness (Open-Weight Models)
lm_eval --model hf --model_args pretrained=<model_path> --tasks hle-diamond --batch_size auto
Option BOpenCompass Evaluation Framework
opencompass --datasets hle-diamond --models <model_config>
Python APIDeterministic Inference Loop Snippet
# Standard API Evaluation Loop
from openai import OpenAI

client = OpenAI()
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": prompt}],
    temperature=0.0,
)
Standardized Reporting Requirement:

When publishing results for HLE-Diamond, always report the exact prompt template, few-shot exemplar ordering, sampling temperature (temperature=0), maximum reasoning budget tokens, and the precise timestamped model snapshot ID.

Contamination & Memorization Analysis

Audit of pretraining exposure risks, memorization vectors, and refresh policies.
Overall Contamination Risk:MEDIUM
Refresh Cadence:

Static fixed snapshot

Test Set Exposure:

Public on web / HuggingFace