Long ContextActive
Trust Score:
93

AA-LCR (Long Context Reasoning)

Artificial Analysis long-context reasoning benchmark: questions that require reasoning across multiple long documents.

Launched: —Refresh: static
Status Assessment (active):

Actively evaluated by Artificial Analysis on current frontier models; no saturation evidence recorded.

Performance Timeline

Longitudinal progression of model scores against human baselines.

Performance & Historical Trajectory

Empirical score progression across model release dates and evaluation rounds.

Independent Vendor-Reported
ModelScoreDateSource TypeProvenance
JT-35B-Flash65.7%2026-11-15independentSource ↗
Qwen3.5 Omni Plus63.7%2026-11-15independentSource ↗
Nova 2.0 Omni59.7%2026-11-15independentSource ↗
Qwen3.5-122B-A10B61.3%2026-11-15independentSource ↗
Gemma 4 12B35%2026-11-15independentSource ↗
Claude Sonnet 5.582.7%2026-11-15independentSource ↗
Gemma 3 270M0%2026-11-15independentSource ↗
Apodex 1.179.3%2026-11-15independentSource ↗
GPT-6.1 Sol83%2026-11-15independentSource ↗
Gemini 3.5 Flash-Lite76%2026-11-15independentSource ↗
Step 588.3%2026-11-15independentSource ↗
Qwen3.5 2B14%2026-11-15independentSource ↗
Qwen3 Omni 30B A3B Instruct0%2026-11-15independentSource ↗
Granite 4.2 3B24.3%2026-11-15independentSource ↗
Claude Fable 5.283.3%2026-11-15independentSource ↗
GPT-6.1 Terra82.7%2026-11-15independentSource ↗
Gemini 4 Carbon79.3%2026-11-15independentSource ↗
Nova 2.0 Pro64%2026-11-15independentSource ↗
Inkling77.3%2026-11-15independentSource ↗
Qwen3.5 397B A17B77.3%2026-11-15independentSource ↗
Granite 4.2 30B49%2026-11-15independentSource ↗
K2 Horizon 7B69.7%2026-11-15independentSource ↗
MiMo-V2.6-Flash74.3%2026-11-15independentSource ↗
Mistral Small 449.7%2026-11-15independentSource ↗
Solar Pro 470.7%2026-11-15independentSource ↗
gpt-oss-20b31%2026-11-15independentSource ↗
o374.7%2026-11-15independentSource ↗
K-EXAONE 2.060%2026-11-15independentSource ↗
GLM 5.3 Flash80%2026-11-15independentSource ↗
GPT-6 Luna83.3%2026-11-15independentSource ↗
North Mini Code37.3%2026-11-15independentSource ↗
GPT-6 Astra80.7%2026-11-15independentSource ↗
Granite 4.0 Micro7%2026-11-15independentSource ↗
Claude Haiku 5.582.7%2026-11-15independentSource ↗
Grok 4.776.7%2026-11-15independentSource ↗
Qwen3.5-9B46%2026-11-15independentSource ↗
Hermes 4 - Llama-3.1 70B4.7%2026-11-15independentSource ↗
Nemotron 3 Nano 30B A3B10.7%2026-11-15independentSource ↗
MiMo-V2.6-Pro86.3%2026-11-15independentSource ↗
Qwen3 Next 80B A3B Instruct52.7%2026-11-15independentSource ↗
MiniMax M383%2026-11-15independentSource ↗
Nemotron 3.5 Lightning60.3%2026-11-15independentSource ↗
NVIDIA Nemotron Nano 9B V224%2026-11-15independentSource ↗
Command A+52.7%2026-11-15independentSource ↗
Claude Opus 5.584.7%2026-11-15independentSource ↗
G9v3-3B41.3%2026-11-15independentSource ↗
Qwen3.5 4B34.7%2026-11-15independentSource ↗
K2 Horizon 3.7B62.3%2026-11-15independentSource ↗
Olmo 3 7B Think0%2026-11-15independentSource ↗
HyperCLOVA X SEED Think (32B)13.7%2026-11-15independentSource ↗
HyperNova 60B 260536.3%2026-11-15independentSource ↗
Nova Micro13%2026-11-15independentSource ↗
Qwen3.8 27B82%2026-11-15independentSource ↗
Muse Spark Open78.3%2026-11-15independentSource ↗
INTELLECT-339.7%2026-11-15independentSource ↗
Muse Wonder80.7%2026-11-15independentSource ↗
Olmo 3.1 32B Think0%2026-11-15independentSource ↗
Nex-N2-Pro81.7%2026-11-15independentSource ↗
LFM2.5-VL-1.6B0%2026-11-15independentSource ↗
JT-MINI13%2026-11-15independentSource ↗
Molmo2-8B0%2026-11-15independentSource ↗
Qwen3.5 0.8B8%2026-11-15independentSource ↗
Ling-3.0-flash-VL78.3%2026-11-15independentSource ↗
MiniCPM-V 4.6 1.3B7%2026-11-15independentSource ↗
Sarvam 105B0%2026-11-15independentSource ↗
Llama 3.1 Instruct 405B25.3%2026-11-15independentSource ↗
Ring-flash-2.021%2026-11-15independentSource ↗
LFM2.5-1.2B-Instruct0%2026-11-15independentSource ↗
Gemini 4 Argon79.7%2026-11-15independentSource ↗
Apertus 70B Instruct0%2026-11-15independentSource ↗
Doubao Seed Code71.7%2026-11-15independentSource ↗
Gemma 4 E4B24%2026-11-15independentSource ↗
Ling 3.1 Flash 101982.3%2026-11-15independentSource ↗
K2 Horizon MoVA 36B A4B71.7%2026-11-15independentSource ↗
Granite 4.0 350M0%2026-11-15independentSource ↗
DeepSeek V4.1 Flash 101782.7%2026-11-15independentSource ↗
Muse Glimmer83.3%2026-11-15independentSource ↗
Kimi Linear 48B A3B Instruct28%2026-11-15independentSource ↗
Solar Open 100B41%2026-11-15independentSource ↗
GPT-5.3-Codex83.3%2026-11-15independentSource ↗
LFM2.5-1.2B-Thinking0%2026-11-15independentSource ↗
K2 Think V257%2026-11-15independentSource ↗
Nemotron 3 Ultra79.3%2026-11-15independentSource ↗
LongCat 2.065%2026-11-15independentSource ↗
Tri-21B-Think15.7%2026-11-15independentSource ↗
Granite 4.0 H Small11.3%2026-11-15independentSource ↗
Llama 3.2 Instruct 11B (Vision)12.7%2026-11-15independentSource ↗
LongCat Flash Lite28.3%2026-11-15independentSource ↗
Nova 2.0 Lite60.3%2026-11-15independentSource ↗
Solar Mini 483.3%2026-11-15independentSource ↗
MiniCPM5-2B59.3%2026-11-15independentSource ↗
Kimi K3.1 Pro86.3%2026-11-15independentSource ↗
Motif-2-12.7B-Reasoning14.7%2026-11-15independentSource ↗
Qwen3.6 35B A3B64.3%2026-11-15independentSource ↗
Ministral 3 3B17%2026-11-15independentSource ↗
Beam 1.078%2026-11-15independentSource ↗
Ling-3.0-flash-Fin73.7%2026-11-15independentSource ↗
Gemma 4 26B A4B65.7%2026-11-15independentSource ↗
NVIDIA Nemotron Nano 12B v2 VL41%2026-11-15independentSource ↗
Qwen3.5 Omni Flash52%2026-11-15independentSource ↗
Magistral Small 1.219.3%2026-11-15independentSource ↗
Ministral 3 14B26.3%2026-11-15independentSource ↗
Inkling Small75.7%2026-11-15independentSource ↗
Gemini 3.8 Flash81.3%2026-11-15independentSource ↗
Gemma 4 E2B16.3%2026-11-15independentSource ↗
Sarvam 30B0%2026-11-15independentSource ↗
Mercury 2.571.7%2026-11-15independentSource ↗
DeepSeek V4 Flash Vision81.3%2026-11-15independentSource ↗
MiniCPM5-1B6%2026-11-15independentSource ↗
Jamba 1.7 Large19.3%2026-11-15independentSource ↗
Nemotron 3.5 Bolt 74B-A9B69.7%2026-11-15independentSource ↗
Nanbeige4.1-3B0%2026-11-15independentSource ↗
Solar Open2 250B71.7%2026-11-15independentSource ↗
Ling 3.0 Tiny60.3%2026-11-15independentSource ↗
Celeris-138%2026-11-15independentSource ↗
Tiny Aya Global0%2026-11-15independentSource ↗
A.X-K270%2026-11-15independentSource ↗
Exaone 4.0 1.2B0%2026-11-15independentSource ↗
Hermes 4 - Llama-3.1 405B22.3%2026-11-15independentSource ↗
DeepSeek V4-Pro80.3%2026-11-15independentSource ↗
Magistral Medium 1.253%2026-11-15independentSource ↗
Apertus 8B Instruct0%2026-11-15independentSource ↗
Nemotron 3 Nano Omni 30B A3B Reasoning39.7%2026-11-15independentSource ↗
Kimi K2.7 Code79.3%2026-11-15independentSource ↗
Qwen3.8 2.4T A95B80.3%2026-11-15independentSource ↗
Olmo 3 7B Instruct0%2026-11-15independentSource ↗
Llama 4 Maverick50%2026-11-15independentSource ↗
Olmo 3.1 32B Instruct0%2026-11-15independentSource ↗
Ministral 3 8B25.7%2026-11-15independentSource ↗
Llama 3.3 Instruct 70B15.7%2026-11-15independentSource ↗
Nemotron Cascade 2 30B A3B39.3%2026-11-15independentSource ↗
Qwen3.8-Flash-Next79.7%2026-11-15independentSource ↗
K2 Horizon 0.9B6.3%2026-11-15independentSource ↗
NVIDIA Nemotron 3 Nano 4B21.7%2026-11-15independentSource ↗
Phi-4-mini15.3%2026-11-15independentSource ↗
K2-V235%2026-11-15independentSource ↗
Ling 3.1 Flash83%2026-11-15independentSource ↗
Granite 4.0 H 350M0%2026-11-15independentSource ↗
Qwen3.7 Plus73%2026-11-15independentSource ↗
Step3 VL 10B0%2026-11-15independentSource ↗
Llama 4 Scout27.7%2026-11-15independentSource ↗
Granite 4.2 8B45%2026-11-15independentSource ↗
Molmo 7B-D0%2026-11-15independentSource ↗
Motif 376%2026-11-15independentSource ↗
Qwen3.8 Max (0902)80.3%2026-11-15independentSource ↗
Hy379%2026-11-15independentSource ↗
Jamba 1.7 Mini16%2026-11-15independentSource ↗
Trinity Large Thinking38%2026-11-15independentSource ↗
Qwen3 Omni 30B A3B (Reasoning)0%2026-11-15independentSource ↗
K2 Horizon 375B A23B80%2026-11-15independentSource ↗
Gemini 3.1 Pro82%2026-11-15independentSource ↗
G9v3-39A5B65.3%2026-11-15independentSource ↗
DiffusionGemma 26B A4B19.7%2026-11-15independentSource ↗
GPT-5.5 Instant70%2026-11-15independentSource ↗
Falcon-H1R-7B11%2026-11-15independentSource ↗
Qwen3 Next 80B A3B (Reasoning)63.7%2026-11-15independentSource ↗
gpt-oss-120b52%2026-11-15independentSource ↗
GLM 5.379.7%2026-11-15independentSource ↗
Gemma 4 31B69.7%2026-11-15independentSource ↗
MiniMax M3.1 Flash79.7%2026-11-15independentSource ↗
Agnes 2.5 Pro Theta84.7%2026-11-15independentSource ↗
Granite 4.0 H 1B7%2026-11-15independentSource ↗
Reka Flash 30%2026-11-15independentSource ↗
Mistral Medium 3.569.3%2026-11-15independentSource ↗
Mi:dm K 2.5 Pro9%2026-11-15independentSource ↗
LFM2.5-2.6B5.7%2026-11-15independentSource ↗
LFM2.5-8B-A1B0%2026-11-15independentSource ↗
Nemotron 3.5 Super VL69%2026-11-15independentSource ↗
Inkling 2 Small83.3%2026-11-15independentSource ↗
Nova Premier34.3%2026-11-15independentSource ↗
Muse Wonder Full83.7%2026-11-15independentSource ↗
Cogito v2.122.7%2026-11-15independentSource ↗
Ring-2.6-1T70%2026-11-15independentSource ↗
Phi-40%2026-11-15independentSource ↗
Agnes 2.5 Pro Beta83%2026-11-15independentSource ↗
Jamba Reasoning 3B6.3%2026-11-15independentSource ↗
Quasar 438B76.3%2026-11-15independentSource ↗
LFM2 24B A2B0%2026-11-15independentSource ↗
KAT Coder Pro V273%2026-11-15independentSource ↗
Qwen3 Coder Next47%2026-11-15independentSource ↗

Human Baseline & Difficulty Horizon

Calibrated human reference points, specialist benchmarks, and ceiling thresholds.
Measured Human Scoren/aNot established
Baseline Protocol & Interpretation

No peer-reviewed human baseline has been published for this benchmark.

Metric & Scoring Methodology

Verification protocols, aggregation formulas, and specialized metric variants.
Primary Metric:accuracy (%)
Scoring Engine:llm-judge

Dataset & Compute Cost

Evaluation volume, public availability, API pricing, and local hardware requirements.
Total Dataset Size—Annotated evaluation items
Public Test SetPublicOpenly mirrored on repositories
Access GatingOpen AccessUnrestricted download
Evaluation LicenseOpen AccessDataset usage and redistribution terms
Frontier API Compute Cost:

$5 – $20 USD for full benchmark evaluation run on frontier APIs.

Recommended Local GPU Setup:

1x NVIDIA RTX 4090 (24GB) or A100 (40GB/80GB) via vLLM / SGLang

How to Run & Reproduce

Standardized evaluation protocols, CLI commands, and reproducible runner templates.
Prompt Regimezero-shot
Reasoning Modedirect
Sampling Temp0
Pass@k Budgetk = 1
Tools & SandboxPure Text
Scoring Verifierllm-judge
Option AEleutherAI LM-Evaluation-Harness (Open-Weight Models)
lm_eval --model hf --model_args pretrained=<model_path> --tasks aa-lcr --batch_size auto
Option BOpenCompass Evaluation Framework
opencompass --datasets aa-lcr --models <model_config>
Python APIDeterministic Inference Loop Snippet
# Standard API Evaluation Loop
from openai import OpenAI

client = OpenAI()
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": prompt}],
    temperature=0.0,
)
Standardized Reporting Requirement:

When publishing results for AA-LCR (Long Context Reasoning), always report the exact prompt template, few-shot exemplar ordering, sampling temperature (temperature=0), maximum reasoning budget tokens, and the precise timestamped model snapshot ID.

Contamination & Memorization Analysis

Audit of pretraining exposure risks, memorization vectors, and refresh policies.
Overall Contamination Risk:MEDIUM
Refresh Cadence:

Static fixed snapshot

Test Set Exposure:

Public on web / HuggingFace