Agentic & Tool UseActive
Trust Score:
93

Terminal-Bench 2.1

Version 2.1 of Terminal-Bench, which tests agents on realistic command-line tasks inside sandboxed environments.

Launched: —Refresh: static
Status Assessment (active):

Actively evaluated by Artificial Analysis on current frontier models; no saturation evidence recorded.

Performance Timeline

Longitudinal progression of model scores against human baselines.

Performance & Historical Trajectory

Empirical score progression across model release dates and evaluation rounds.

Independent Vendor-Reported
ModelScoreDateSource TypeProvenance
Qwen3.8 2.4T A95B82%2026-11-15independentSource ↗
Llama 4 Scout3.7%2026-11-15independentSource ↗
Qwen3.7 Plus61%2026-11-15independentSource ↗
Phi-4-mini0.4%2026-11-15independentSource ↗
NVIDIA Nemotron 3 Nano 4B3.7%2026-11-15independentSource ↗
K2 Horizon 0.9B1.1%2026-11-15independentSource ↗
Qwen3.8-Flash-Next86.1%2026-11-15independentSource ↗
Nemotron Cascade 2 30B A3B20.6%2026-11-15independentSource ↗
Llama 3.3 Instruct 70B4.9%2026-11-15independentSource ↗
Ministral 3 8B4.1%2026-11-15independentSource ↗
Llama 4 Maverick7.9%2026-11-15independentSource ↗
Granite 4.2 8B18.4%2026-11-15independentSource ↗
Kimi K2.7 Code67.4%2026-11-15independentSource ↗
Nemotron 3 Nano Omni 30B A3B Reasoning6.7%2026-11-15independentSource ↗
Magistral Medium 1.212.4%2026-11-15independentSource ↗
DeepSeek V4-Pro78.7%2026-11-15independentSource ↗
A.X-K239%2026-11-15independentSource ↗
Celeris-111.2%2026-11-15independentSource ↗
Ling 3.0 Tiny27.7%2026-11-15independentSource ↗
Solar Open2 250B44.2%2026-11-15independentSource ↗
Nanbeige4.1-3B1.1%2026-11-15independentSource ↗
DeepSeek V4 Flash Vision74.2%2026-11-15independentSource ↗
gpt-oss-120b26.2%2026-11-15independentSource ↗
Qwen3 Coder Next38.2%2026-11-15independentSource ↗
KAT Coder Pro V270%2026-11-15independentSource ↗
Quasar 438B69.3%2026-11-15independentSource ↗
Agnes 2.5 Pro Beta69.7%2026-11-15independentSource ↗
Ring-2.6-1T43.1%2026-11-15independentSource ↗
Inkling 2 Small81.3%2026-11-15independentSource ↗
LFM2.5-2.6B4.5%2026-11-15independentSource ↗
Mistral Medium 3.550.6%2026-11-15independentSource ↗
Gemma 4 31B43.4%2026-11-15independentSource ↗
GLM 5.383.9%2026-11-15independentSource ↗
Gemini 3.8 Flash87.6%2026-11-15independentSource ↗
Qwen3 Next 80B A3B (Reasoning)6.7%2026-11-15independentSource ↗
GPT-5.5 Instant34.8%2026-11-15independentSource ↗
DiffusionGemma 26B A4B12.4%2026-11-15independentSource ↗
G9v3-39A5B32.6%2026-11-15independentSource ↗
Gemini 3.1 Pro73.8%2026-11-15independentSource ↗
K2 Horizon 375B A23B71.9%2026-11-15independentSource ↗
Trinity Large Thinking20.6%2026-11-15independentSource ↗
Hy364.4%2026-11-15independentSource ↗
Qwen3.8 Max (0902)88.8%2026-11-15independentSource ↗
Motif 374.9%2026-11-15independentSource ↗
Solar Pro 457.3%2026-11-15independentSource ↗
Qwen3.5 4B21.3%2026-11-15independentSource ↗
G9v3-3B6%2026-11-15independentSource ↗
Command A+22.8%2026-11-15independentSource ↗
Nemotron 3.5 Lightning24.3%2026-11-15independentSource ↗
MiniMax M365.2%2026-11-15independentSource ↗
Qwen3.5-9B21.3%2026-11-15independentSource ↗
GPT-6 Astra88.4%2026-11-15independentSource ↗
North Mini Code35.6%2026-11-15independentSource ↗
GLM 5.3 Flash84.3%2026-11-15independentSource ↗
K-EXAONE 2.040.4%2026-11-15independentSource ↗
K2 Horizon 3.7B28.1%2026-11-15independentSource ↗
Mistral Small 421%2026-11-15independentSource ↗
K2 Horizon 7B44.2%2026-11-15independentSource ↗
Granite 4.2 30B26.6%2026-11-15independentSource ↗
Qwen3.5 397B A17B51.3%2026-11-15independentSource ↗
Inkling55.1%2026-11-15independentSource ↗
Nova 2.0 Pro29.6%2026-11-15independentSource ↗
Granite 4.2 3B13.9%2026-11-15independentSource ↗
Qwen3.5 2B0%2026-11-15independentSource ↗
Gemini 3.5 Flash-Lite53.6%2026-11-15independentSource ↗
Apodex 1.169.7%2026-11-15independentSource ↗
Nemotron 3 Ultra53.9%2026-11-15independentSource ↗
Inkling Small55.1%2026-11-15independentSource ↗
Ministral 3 14B9.7%2026-11-15independentSource ↗
Magistral Small 1.24.5%2026-11-15independentSource ↗
Gemma 4 26B A4B39%2026-11-15independentSource ↗
Ling-3.0-flash-Fin62.5%2026-11-15independentSource ↗
Ministral 3 3B0%2026-11-15independentSource ↗
Qwen3.6 35B A3B41.6%2026-11-15independentSource ↗
MiniCPM5-2B8.6%2026-11-15independentSource ↗
Nova 2.0 Lite16.1%2026-11-15independentSource ↗
LongCat 2.050.2%2026-11-15independentSource ↗
Qwen3.5-122B-A10B47.2%2026-11-15independentSource ↗
K2 Think V215%2026-11-15independentSource ↗
Muse Glimmer51.7%2026-11-15independentSource ↗
K2 Horizon MoVA 36B A4B56.6%2026-11-15independentSource ↗
MiniCPM-V 4.6 1.3B0%2026-11-15independentSource ↗
Ling-3.0-flash-VL64.4%2026-11-15independentSource ↗
Qwen3.5 0.8B0.4%2026-11-15independentSource ↗
Nex-N2-Pro67.8%2026-11-15independentSource ↗
Muse Spark Open76.4%2026-11-15independentSource ↗
Qwen3.8 27B79.8%2026-11-15independentSource ↗
HyperNova 60B 260518.4%2026-11-15independentSource ↗

Human Baseline & Difficulty Horizon

Calibrated human reference points, specialist benchmarks, and ceiling thresholds.
Measured Human Scoren/aNot established
Baseline Protocol & Interpretation

No peer-reviewed human baseline has been published for this benchmark.

Metric & Scoring Methodology

Verification protocols, aggregation formulas, and specialized metric variants.
Primary Metric:task success (%)
Scoring Engine:unit-tests

Dataset & Compute Cost

Evaluation volume, public availability, API pricing, and local hardware requirements.
Total Dataset Size—Annotated evaluation items
Public Test SetPublicOpenly mirrored on repositories
Access GatingOpen AccessUnrestricted download
Evaluation LicenseOpen AccessDataset usage and redistribution terms
Frontier API Compute Cost:

$5 – $20 USD for full benchmark evaluation run on frontier APIs.

Recommended Local GPU Setup:

1x NVIDIA RTX 4090 (24GB) or A100 (40GB/80GB) via vLLM / SGLang

How to Run & Reproduce

Standardized evaluation protocols, CLI commands, and reproducible runner templates.
Prompt Regimezero-shot
Reasoning Modedirect
Sampling Temp0
Pass@k Budgetk = 1
Tools & SandboxPure Text
Scoring Verifierunit-tests
Option AEleutherAI LM-Evaluation-Harness (Open-Weight Models)
lm_eval --model hf --model_args pretrained=<model_path> --tasks terminal-bench-2-1 --batch_size auto
Option BOpenCompass Evaluation Framework
opencompass --datasets terminal-bench-2-1 --models <model_config>
Python APIDeterministic Inference Loop Snippet
# Standard API Evaluation Loop
from openai import OpenAI

client = OpenAI()
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": prompt}],
    temperature=0.0,
)
Standardized Reporting Requirement:

When publishing results for Terminal-Bench 2.1, always report the exact prompt template, few-shot exemplar ordering, sampling temperature (temperature=0), maximum reasoning budget tokens, and the precise timestamped model snapshot ID.

Contamination & Memorization Analysis

Audit of pretraining exposure risks, memorization vectors, and refresh policies.
Overall Contamination Risk:MEDIUM
Refresh Cadence:

Static fixed snapshot

Test Set Exposure:

Public on web / HuggingFace