Agentic & Tool UseActive
Trust Score:
93

τ²-Bench Telecom

Customer-service agent benchmark in which both the agent and a simulated user act on a shared environment; this entry is the telecom domain.

Launched: 2025-06-09Refresh: static
Status Assessment (active):

Actively evaluated by Artificial Analysis on current frontier models; no saturation evidence recorded.

Performance Timeline

Longitudinal progression of model scores against human baselines.

Performance & Historical Trajectory

Empirical score progression across model release dates and evaluation rounds.

Independent Vendor-Reported
ModelScoreDateSource TypeProvenance
JT-35B-Flash99.1%2026-11-15independentSource ↗
Qwen3.5 Omni Plus88.3%2026-11-15independentSource ↗
Nova 2.0 Omni80.4%2026-11-15independentSource ↗
Qwen3.5-122B-A10B84.5%2026-11-15independentSource ↗
Gemma 4 12B31.9%2026-11-15independentSource ↗
Gemma 3 270M9.1%2026-11-15independentSource ↗
Qwen3.5 2B81.6%2026-11-15independentSource ↗
Qwen3 Omni 30B A3B Instruct16.4%2026-11-15independentSource ↗
Nova 2.0 Pro92.7%2026-11-15independentSource ↗
Qwen3.5 397B A17B95.6%2026-11-15independentSource ↗
Mistral Small 441.2%2026-11-15independentSource ↗
gpt-oss-20b50.3%2026-11-15independentSource ↗
o380.7%2026-11-15independentSource ↗
North Mini Code37.4%2026-11-15independentSource ↗
Granite 4.0 Micro12.6%2026-11-15independentSource ↗
Qwen3.5-9B85.1%2026-11-15independentSource ↗
Hermes 4 - Llama-3.1 70B21.6%2026-11-15independentSource ↗
Nemotron 3 Nano 30B A3B25.4%2026-11-15independentSource ↗
Qwen3 Next 80B A3B Instruct21.6%2026-11-15independentSource ↗
MiniMax M388.9%2026-11-15independentSource ↗
NVIDIA Nemotron Nano 9B V223.4%2026-11-15independentSource ↗
Command A+80.7%2026-11-15independentSource ↗
EXAONE 4.0 32B4.1%2026-11-15independentSource ↗
Qwen3.5 4B87.7%2026-11-15independentSource ↗
Olmo 3 7B Think0%2026-11-15independentSource ↗
HyperCLOVA X SEED Think (32B)87.4%2026-11-15independentSource ↗
HyperNova 60B 260563.2%2026-11-15independentSource ↗
Nova Micro14%2026-11-15independentSource ↗
INTELLECT-326.6%2026-11-15independentSource ↗
Olmo 3.1 32B Think0%2026-11-15independentSource ↗
Nex-N2-Pro81.6%2026-11-15independentSource ↗
LFM2.5-VL-1.6B8.5%2026-11-15independentSource ↗
JT-MINI93%2026-11-15independentSource ↗
Molmo2-8B0%2026-11-15independentSource ↗
Qwen3.5 0.8B65.2%2026-11-15independentSource ↗
MiniCPM-V 4.6 1.3B87.7%2026-11-15independentSource ↗
Sarvam 105B46.8%2026-11-15independentSource ↗
Llama 3.1 Instruct 405B19%2026-11-15independentSource ↗
Ring-flash-2.00%2026-11-15independentSource ↗
LFM2.5-1.2B-Instruct10.8%2026-11-15independentSource ↗
Apertus 70B Instruct12.9%2026-11-15independentSource ↗
Doubao Seed Code58.2%2026-11-15independentSource ↗
Gemma 4 E4B26%2026-11-15independentSource ↗
Granite 4.0 350M13.2%2026-11-15independentSource ↗
Kimi Linear 48B A3B Instruct0%2026-11-15independentSource ↗
Solar Open 100B48.2%2026-11-15independentSource ↗
GPT-5.3-Codex86%2026-11-15independentSource ↗
LFM2.5-1.2B-Thinking19.6%2026-11-15independentSource ↗
K2 Think V225.4%2026-11-15independentSource ↗
Nemotron 3 Ultra83.3%2026-11-15independentSource ↗
Tri-21B-Think81%2026-11-15independentSource ↗
Granite 4.0 H Small17.3%2026-11-15independentSource ↗
Llama 3.2 Instruct 11B (Vision)14.6%2026-11-15independentSource ↗
LongCat Flash Lite79.5%2026-11-15independentSource ↗
Nova 2.0 Lite72.8%2026-11-15independentSource ↗
Motif-2-12.7B-Reasoning46.5%2026-11-15independentSource ↗
Qwen3.6 35B A3B85.1%2026-11-15independentSource ↗
Ministral 3 3B24.9%2026-11-15independentSource ↗
Gemma 4 26B A4B43.6%2026-11-15independentSource ↗
NVIDIA Nemotron Nano 12B v2 VL21.3%2026-11-15independentSource ↗
Qwen3.5 Omni Flash84.5%2026-11-15independentSource ↗
Magistral Small 1.227.8%2026-11-15independentSource ↗
Ministral 3 14B27.2%2026-11-15independentSource ↗
Gemma 4 E2B22.2%2026-11-15independentSource ↗
Sarvam 30B34.5%2026-11-15independentSource ↗
MiniCPM5-1B81%2026-11-15independentSource ↗
Jamba 1.7 Large13.5%2026-11-15independentSource ↗
Nanbeige4.1-3B21.6%2026-11-15independentSource ↗
Tiny Aya Global0%2026-11-15independentSource ↗
Exaone 4.0 1.2B20.5%2026-11-15independentSource ↗
Hermes 4 - Llama-3.1 405B22.2%2026-11-15independentSource ↗
Magistral Medium 1.252%2026-11-15independentSource ↗
Apertus 8B Instruct11.4%2026-11-15independentSource ↗
Nemotron 3 Nano Omni 30B A3B Reasoning45.3%2026-11-15independentSource ↗
Kimi K2.7 Code90.1%2026-11-15independentSource ↗
Olmo 3 7B Instruct12.6%2026-11-15independentSource ↗
Llama 4 Maverick17.8%2026-11-15independentSource ↗
Olmo 3.1 32B Instruct21.3%2026-11-15independentSource ↗
Ministral 3 8B26.6%2026-11-15independentSource ↗
Llama 3.3 Instruct 70B26.6%2026-11-15independentSource ↗
Nemotron Cascade 2 30B A3B53.2%2026-11-15independentSource ↗
NVIDIA Nemotron 3 Nano 4B28.1%2026-11-15independentSource ↗
Phi-4-mini8.2%2026-11-15independentSource ↗
K2-V227.8%2026-11-15independentSource ↗
ERNIE 5.0 Thinking83.9%2026-11-15independentSource ↗
Granite 4.0 H 350M14.6%2026-11-15independentSource ↗
Apriel-v1.6-15B-Thinker69.3%2026-11-15independentSource ↗
Qwen3.7 Plus93%2026-11-15independentSource ↗
Step3 VL 10B16.1%2026-11-15independentSource ↗
Llama 4 Scout15.5%2026-11-15independentSource ↗
ERNIE 4.5 300B A47B0%2026-11-15independentSource ↗
Molmo 7B-D0%2026-11-15independentSource ↗
Jamba 1.7 Mini12.6%2026-11-15independentSource ↗
Trinity Large Thinking90.1%2026-11-15independentSource ↗
Qwen3 Omni 30B A3B (Reasoning)21.3%2026-11-15independentSource ↗
Gemini 3.1 Pro95.6%2026-11-15independentSource ↗
Falcon-H1R-7B27.8%2026-11-15independentSource ↗
Qwen3 Next 80B A3B (Reasoning)41.5%2026-11-15independentSource ↗
gpt-oss-120b65.8%2026-11-15independentSource ↗
Gemma 4 31B59.9%2026-11-15independentSource ↗
Llama 3.1 Nemotron Ultra 253B v111.4%2026-11-15independentSource ↗
Granite 4.0 H 1B19.6%2026-11-15independentSource ↗
Reka Flash 30%2026-11-15independentSource ↗
Mistral Medium 3.594.2%2026-11-15independentSource ↗
Mi:dm K 2.5 Pro86.5%2026-11-15independentSource ↗
LFM2.5-8B-A1B16.1%2026-11-15independentSource ↗
Nova Premier38.3%2026-11-15independentSource ↗
Ring-2.6-1T92.4%2026-11-15independentSource ↗
Phi-40%2026-11-15independentSource ↗
Jamba Reasoning 3B15.8%2026-11-15independentSource ↗
LFM2 24B A2B11.1%2026-11-15independentSource ↗
KAT Coder Pro V289.5%2026-11-15independentSource ↗
Qwen3 Coder Next79.5%2026-11-15independentSource ↗

Human Baseline & Difficulty Horizon

Calibrated human reference points, specialist benchmarks, and ceiling thresholds.
Measured Human Scoren/aNot established
Baseline Protocol & Interpretation

No peer-reviewed human baseline has been published for this benchmark.

Metric & Scoring Methodology

Verification protocols, aggregation formulas, and specialized metric variants.
Primary Metric:task success (%)
Scoring Engine:composite

Dataset & Compute Cost

Evaluation volume, public availability, API pricing, and local hardware requirements.
Total Dataset Size—Annotated evaluation items
Public Test SetPublicOpenly mirrored on repositories
Access GatingOpen AccessUnrestricted download
Evaluation LicenseOpen AccessDataset usage and redistribution terms
Frontier API Compute Cost:

$5 – $20 USD for full benchmark evaluation run on frontier APIs.

Recommended Local GPU Setup:

1x NVIDIA RTX 4090 (24GB) or A100 (40GB/80GB) via vLLM / SGLang

How to Run & Reproduce

Standardized evaluation protocols, CLI commands, and reproducible runner templates.
Prompt Regimezero-shot
Reasoning Modedirect
Sampling Temp0
Pass@k Budgetk = 1
Tools & SandboxPure Text
Scoring Verifiercomposite
Option AEleutherAI LM-Evaluation-Harness (Open-Weight Models)
lm_eval --model hf --model_args pretrained=<model_path> --tasks tau2-bench-telecom --batch_size auto
Option BOpenCompass Evaluation Framework
opencompass --datasets tau2-bench-telecom --models <model_config>
Python APIDeterministic Inference Loop Snippet
# Standard API Evaluation Loop
from openai import OpenAI

client = OpenAI()
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": prompt}],
    temperature=0.0,
)
Standardized Reporting Requirement:

When publishing results for τ²-Bench Telecom, always report the exact prompt template, few-shot exemplar ordering, sampling temperature (temperature=0), maximum reasoning budget tokens, and the precise timestamped model snapshot ID.

Contamination & Memorization Analysis

Audit of pretraining exposure risks, memorization vectors, and refresh policies.
Overall Contamination Risk:MEDIUM
Refresh Cadence:

Static fixed snapshot

Test Set Exposure:

Public on web / HuggingFace