Agentic & Tool UseActive
Trust Score:
93
τ³-Banking
Banking customer-service agent benchmark from the τ-bench family, evaluated by Artificial Analysis.
Launched: —Refresh: static
Status Assessment (active):
Actively evaluated by Artificial Analysis on current frontier models; no saturation evidence recorded.
Performance Timeline
Longitudinal progression of model scores against human baselines.Performance & Historical Trajectory
Empirical score progression across model release dates and evaluation rounds.
Independent Vendor-Reported
| Model | Score | Date | Source Type | Provenance |
|---|---|---|---|---|
| Llama 4 Maverick | 3.7% | 2026-11-15 | independent | Source ↗ |
| Qwen3.8 Max (0902) | 47.8% | 2026-11-15 | independent | Source ↗ |
| Motif 3 | 35.3% | 2026-11-15 | independent | Source ↗ |
| Granite 4.2 8B | 7.6% | 2026-11-15 | independent | Source ↗ |
| Llama 4 Scout | 3.3% | 2026-11-15 | independent | Source ↗ |
| Qwen3.7 Plus | 17.5% | 2026-11-15 | independent | Source ↗ |
| K2 Horizon 0.9B | 4.9% | 2026-11-15 | independent | Source ↗ |
| Qwen3.8-Flash-Next | 45.4% | 2026-11-15 | independent | Source ↗ |
| Ministral 3 8B | 3.7% | 2026-11-15 | independent | Source ↗ |
| Hy3 | 22.9% | 2026-11-15 | independent | Source ↗ |
| Qwen3.8 2.4T A95B | 49.1% | 2026-11-15 | independent | Source ↗ |
| Kimi K2.7 Code | 20.2% | 2026-11-15 | independent | Source ↗ |
| Magistral Medium 1.2 | 5.6% | 2026-11-15 | independent | Source ↗ |
| DeepSeek V4-Pro | 39.6% | 2026-11-15 | independent | Source ↗ |
| A.X-K2 | 16.1% | 2026-11-15 | independent | Source ↗ |
| Celeris-1 | 3.9% | 2026-11-15 | independent | Source ↗ |
| Ling 3.0 Tiny | 20.8% | 2026-11-15 | independent | Source ↗ |
| Solar Open2 250B | 21.6% | 2026-11-15 | independent | Source ↗ |
| Gemma 4 31B | 14.8% | 2026-11-15 | independent | Source ↗ |
| Qwen3 Coder Next | 5.4% | 2026-11-15 | independent | Source ↗ |
| KAT Coder Pro V2 | 4.5% | 2026-11-15 | independent | Source ↗ |
| Quasar 438B | 28.7% | 2026-11-15 | independent | Source ↗ |
| Agnes 2.5 Pro Beta | 35.7% | 2026-11-15 | independent | Source ↗ |
| Ring-2.6-1T | 17.9% | 2026-11-15 | independent | Source ↗ |
| Inkling 2 Small | 40.4% | 2026-11-15 | independent | Source ↗ |
| LFM2.5-2.6B | 7.2% | 2026-11-15 | independent | Source ↗ |
| Mistral Medium 3.5 | 15.1% | 2026-11-15 | independent | Source ↗ |
| DeepSeek V4 Flash Vision | 41% | 2026-11-15 | independent | Source ↗ |
| GLM 5.3 | 50.3% | 2026-11-15 | independent | Source ↗ |
| gpt-oss-120b | 12.8% | 2026-11-15 | independent | Source ↗ |
| Qwen3 Next 80B A3B (Reasoning) | 6.2% | 2026-11-15 | independent | Source ↗ |
| GPT-5.5 Instant | 11.8% | 2026-11-15 | independent | Source ↗ |
| G9v3-39A5B | 22.1% | 2026-11-15 | independent | Source ↗ |
| Gemini 3.1 Pro | 21.4% | 2026-11-15 | independent | Source ↗ |
| K2 Horizon 375B A23B | 34.2% | 2026-11-15 | independent | Source ↗ |
| Trinity Large Thinking | 5.8% | 2026-11-15 | independent | Source ↗ |
| Solar Pro 4 | 23.3% | 2026-11-15 | independent | Source ↗ |
| Qwen3.5 4B | 4.3% | 2026-11-15 | independent | Source ↗ |
| Command A+ | 6% | 2026-11-15 | independent | Source ↗ |
| Nemotron 3.5 Lightning | 8.9% | 2026-11-15 | independent | Source ↗ |
| MiniMax M3 | 15.3% | 2026-11-15 | independent | Source ↗ |
| GPT-6 Astra | 41.4% | 2026-11-15 | independent | Source ↗ |
| North Mini Code | 6.4% | 2026-11-15 | independent | Source ↗ |
| GLM 5.3 Flash | 47.2% | 2026-11-15 | independent | Source ↗ |
| K-EXAONE 2.0 | 11.5% | 2026-11-15 | independent | Source ↗ |
| K2 Horizon 3.7B | 19.6% | 2026-11-15 | independent | Source ↗ |
| Mistral Small 4 | 4.9% | 2026-11-15 | independent | Source ↗ |
| K2 Horizon 7B | 27.6% | 2026-11-15 | independent | Source ↗ |
| Granite 4.2 30B | 14.4% | 2026-11-15 | independent | Source ↗ |
| Qwen3.5 397B A17B | 13.4% | 2026-11-15 | independent | Source ↗ |
| Inkling | 29.1% | 2026-11-15 | independent | Source ↗ |
| Granite 4.2 3B | 5.6% | 2026-11-15 | independent | Source ↗ |
| Gemini 3.5 Flash-Lite | 17.5% | 2026-11-15 | independent | Source ↗ |
| Apodex 1.1 | 25.2% | 2026-11-15 | independent | Source ↗ |
| MiniCPM5-2B | 20.8% | 2026-11-15 | independent | Source ↗ |
| Gemini 3.8 Flash | 44.9% | 2026-11-15 | independent | Source ↗ |
| Inkling Small | 18.8% | 2026-11-15 | independent | Source ↗ |
| Ministral 3 14B | 6.6% | 2026-11-15 | independent | Source ↗ |
| Magistral Small 1.2 | 4.5% | 2026-11-15 | independent | Source ↗ |
| Gemma 4 26B A4B | 12% | 2026-11-15 | independent | Source ↗ |
| Ling-3.0-flash-Fin | 38.6% | 2026-11-15 | independent | Source ↗ |
| Ministral 3 3B | 4.7% | 2026-11-15 | independent | Source ↗ |
| Qwen3.6 35B A3B | 5.4% | 2026-11-15 | independent | Source ↗ |
| Qwen3.5-122B-A10B | 10.3% | 2026-11-15 | independent | Source ↗ |
| LongCat 2.0 | 13.2% | 2026-11-15 | independent | Source ↗ |
| Nemotron 3 Ultra | 14.2% | 2026-11-15 | independent | Source ↗ |
| Muse Glimmer | 23.5% | 2026-11-15 | independent | Source ↗ |
| K2 Horizon MoVA 36B A4B | 32% | 2026-11-15 | independent | Source ↗ |
| Ling-3.0-flash-VL | 34.4% | 2026-11-15 | independent | Source ↗ |
| Nex-N2-Pro | 17.5% | 2026-11-15 | independent | Source ↗ |
| Qwen3.8 27B | 48% | 2026-11-15 | independent | Source ↗ |
| HyperNova 60B 2605 | 5.6% | 2026-11-15 | independent | Source ↗ |
Human Baseline & Difficulty Horizon
Calibrated human reference points, specialist benchmarks, and ceiling thresholds.Measured Human Scoren/aNot established
Baseline Protocol & Interpretation
No peer-reviewed human baseline has been published for this benchmark.
Metric & Scoring Methodology
Verification protocols, aggregation formulas, and specialized metric variants.Primary Metric:
task success (%)Scoring Engine:composite
Dataset & Compute Cost
Evaluation volume, public availability, API pricing, and local hardware requirements.Total Dataset Size—Annotated evaluation items
Public Test SetPublicOpenly mirrored on repositories
Access GatingOpen AccessUnrestricted download
Evaluation LicenseOpen AccessDataset usage and redistribution terms
Frontier API Compute Cost:
$5 – $20 USD for full benchmark evaluation run on frontier APIs.
Recommended Local GPU Setup:
1x NVIDIA RTX 4090 (24GB) or A100 (40GB/80GB) via vLLM / SGLang
How to Run & Reproduce
Standardized evaluation protocols, CLI commands, and reproducible runner templates.Prompt Regimezero-shot
Reasoning Modedirect
Sampling Temp0
Pass@k Budgetk = 1
Tools & SandboxPure Text
Scoring Verifiercomposite
Option AEleutherAI LM-Evaluation-Harness (Open-Weight Models)
lm_eval --model hf --model_args pretrained=<model_path> --tasks tau3-banking --batch_size autoOption BOpenCompass Evaluation Framework
opencompass --datasets tau3-banking --models <model_config>Python APIDeterministic Inference Loop Snippet
# Standard API Evaluation Loop
from openai import OpenAI
client = OpenAI()
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": prompt}],
temperature=0.0,
)Standardized Reporting Requirement:
When publishing results for τ³-Banking, always report the exact prompt template, few-shot exemplar ordering, sampling temperature (temperature=0), maximum reasoning budget tokens, and the precise timestamped model snapshot ID.
Contamination & Memorization Analysis
Audit of pretraining exposure risks, memorization vectors, and refresh policies.Overall Contamination Risk:MEDIUM
Refresh Cadence:
Static fixed snapshot
Test Set Exposure:
Public on web / HuggingFace