Instruction FollowingNearing Saturation
Trust Score:
75

MT-Bench

80 curated multi-turn questions across 8 skills, each a two-turn conversation scored by a strong LLM judge on a 1–10 scale.

Launched: —Refresh: static
Status Assessment (nearing-saturation):

The 80 two-turn questions are fixed, public, and heavily exercised across research and leaderboards, so the test set is both widely memorized and increasingly saturated (top models score 8.5–9.0 on a 1–10 scale where 9 is a realistic ceiling). It remains a standard, still-discriminative LLM-judge eval, but the fixed set is the constraint: refreshed or held-out variants are the future direction.

Performance Timeline

Longitudinal progression of model scores against human baselines.

Performance & Historical Trajectory

Empirical score progression across model release dates and evaluation rounds.

Independent Vendor-Reported Human Baseline (80.0%)
ModelScoreDateSource TypeProvenance
MiMo-V2.6-Flash8.2%2026-09-12independentSource ↗
Claude Opus 5.57.8%2026-09-12independentSource ↗
Qwen 3.8 Max Prime7.4%2026-09-12independentSource ↗
Nemotron 3.5 Super VL7.3%2026-09-12independentSource ↗
Gemini 4 Argon8%2026-09-12independentSource ↗
Claude Haiku 5.57.3%2026-09-12independentSource ↗
Nano Banana 2.18%2026-09-12independentSource ↗
Mistral Large 47.7%2026-09-12independentSource ↗
GPT-6.1 Sol Pro8%2026-09-12independentSource ↗
GPT-6.1 Sol8%2026-09-12independentSource ↗
Claude Sonnet 5.58%2026-09-12independentSource ↗
Z.ai: GLM 5.3 Prime8%2026-09-12independentSource ↗
GPT-6 Luna Pro7.3%2026-09-12independentSource ↗
GPT-6 Luna7.3%2026-09-12independentSource ↗
GPT-6 Sol Pro8%2026-09-12independentSource ↗
GPT-6 Sol8%2026-09-12independentSource ↗
Claude Opus 5.58.3%2026-09-12independentSource ↗
Xiaomi: MiMo-V2.6-Pro-UltraSpeed8%2026-09-12independentSource ↗
Xiaomi: MiMo-V2.6-Flash7.3%2026-09-12independentSource ↗
Xiaomi: MiMo-V2.6-Pro7.3%2026-09-12independentSource ↗
Z.ai: GLM 5.3 FlashX7.7%2026-09-12independentSource ↗
DeepSeek V4.1 Flash7.3%2026-09-12independentSource ↗
Claude Sonnet 5.59%2026-09-12independentSource ↗
Claude Haiku 5.58.4%2026-09-12independentSource ↗
GPT-6.1 Sol Pro9.1%2026-09-12independentSource ↗
GPT-6 Luna8.9%2026-09-12independentSource ↗
Gemini 4 Argon9.1%2026-09-12independentSource ↗
Qwen 3.8 Max Prime8.8%2026-09-12independentSource ↗
DeepSeek V4.1 Flash 10177.3%2026-09-12independentSource ↗
GPT-6.1 Terra Pro8%2026-09-12independentSource ↗
GPT-6.1 Terra8%2026-09-12independentSource ↗
MoonshotAI: Kimi K3.1 Pro8%2026-09-12independentSource ↗
MiniMax: MiniMax M3.1 Flash7.3%2026-09-12independentSource ↗
Muse Spark Open8%2026-09-12independentSource ↗
LiquidAI: LFM2.5-24B-A2B7.3%2026-09-12independentSource ↗
Muse Wonder8.3%2026-09-12independentSource ↗
Qwen: Qwen3.8 Max Prime8.3%2026-09-12independentSource ↗
Upstage: Solar Mini 47%2026-09-12independentSource ↗
SpaceXAI: Grok 4.78%2026-09-12independentSource ↗
Qwen: Qwen3.8 Omni Flash7.3%2026-09-12independentSource ↗
GPT-6 Astra8.3%2026-09-08independentSource ↗
GPT-6 Astra Pro8.3%2026-09-08independentSource ↗
Gemini 3.8 Flash8%2026-09-07independentSource ↗
Gemini 3.8 Flash8.9%2026-09-07independentSource ↗
Qwen: Qwen3.8 Max (0902)8%2026-09-07independentSource ↗
Muse Spark 1.3 Contributor7%2026-09-07independentSource ↗
Muse Spark 1.38%2026-09-07independentSource ↗
Claude Fable 5.18.3%2026-09-05independentSource ↗
Hy48.8%2026-09-01independentSource ↗
Hy47.7%2026-09-01independentSource ↗
GLM 5.3 Flash8.9%2026-08-31independentSource ↗
Z.ai: GLM 5.3 Flash7.3%2026-08-31independentSource ↗
Qwen: Qwen3.8 Flash7.3%2026-08-31independentSource ↗
DeepSeek V4 Flash Vision Exp7.3%2026-08-26independentSource ↗
DeepSeek V4-Pro6.7%2026-08-21independentSource ↗
DeepSeek V4-Flash6.4%2026-08-21independentSource ↗
Z.ai: GLM 5.38%2026-08-21independentSource ↗
Claude Fable 5.18.4%2026-08-20independentSource ↗
Qwen: Qwen3.8 27B7.7%2026-08-19independentSource ↗
Gemini 3.7 Flash6.8%2026-08-18independentSource ↗
Gemini 3.7 Flash8%2026-08-18independentSource ↗
DeepSeek V4 Pro 08138%2026-08-18independentSource ↗
Grok 4.66.7%2026-08-17independentSource ↗
Qwen: Qwen3.8 2.4T A95B8%2026-08-17independentSource ↗
LiquidAI: LFM2.5-2.6B7.3%2026-08-16independentSource ↗
ByteDance Seed: Seed 2.1 Turbo7.7%2026-08-15independentSource ↗
SpaceXAI: Grok 4.68%2026-08-15independentSource ↗
Muse Glimmer 30B7.7%2026-08-15independentSource ↗
GPT-59.8%2026-08-12independentSource ↗
Nemotron 3.5 Lightning7%2026-08-12independentSource ↗
GPT-5 Image Mini7.7%2026-08-12independentSource ↗
GPT-5 Image8%2026-08-12independentSource ↗
Muse Spark 1.2 Contributor7%2026-08-10independentSource ↗
Muse Spark 1.28%2026-08-10independentSource ↗
DeepSeek V4.1 Flash8.4%2026-08-06independentSource ↗
Space Bunny Alpha8.6%2026-08-06independentSource ↗
DeepSeek V4 Flash 07318.4%2026-08-05independentSource ↗
DeepSeek V4 Flash 07317.7%2026-08-05independentSource ↗
ByteDance Seed: Seed-2.0-Code7.7%2026-08-04independentSource ↗
GPT-6 Astra8%2026-08-02independentSource ↗
Qwen: Qwen3.7 Flash7%2026-08-01independentSource ↗
Kimi K37.8%2026-07-31independentSource ↗
Claude Opus 57%2026-07-29independentSource ↗
Claude Sonnet 59.8%2026-07-29independentSource ↗
Claude Opus 58.3%2026-07-28independentSource ↗
Gemini 3.6 Flash6.6%2026-07-26independentSource ↗
Gemini 3.5 Flash-Lite6.3%2026-07-26independentSource ↗
Gemini 3.6 Flash8%2026-07-26independentSource ↗
Gemini 3.5 Flash Lite7.7%2026-07-26independentSource ↗
MoonshotAI: Kimi K38.3%2026-07-20independentSource ↗
GPT-5.6 Sol8.2%2026-07-14independentSource ↗
GPT-5.5 Thinking7%2026-07-14independentSource ↗
GPT-5.6 Luna8.5%2026-07-14independentSource ↗
GPT-5.6 Luna Pro7.7%2026-07-14independentSource ↗
GPT-5.6 Luna7.7%2026-07-14independentSource ↗
GPT-5.6 Terra Pro8.3%2026-07-14independentSource ↗
GPT-5.6 Terra8.3%2026-07-14independentSource ↗
GPT-5.6 Sol Pro8.3%2026-07-14independentSource ↗
GPT-5.6 Sol8.3%2026-07-14independentSource ↗
Muse Spark 1.18%2026-07-14independentSource ↗
SpaceXAI: Grok 4.58%2026-07-13independentSource ↗
Hy37.3%2026-07-11independentSource ↗
Claude Sonnet 58%2026-07-05independentSource ↗
Gemini 3.1 Flash Lite Image7.7%2026-07-05independentSource ↗
MiniMax M38.5%2026-06-30independentSource ↗
Qwen3-235B6.6%2026-06-23independentSource ↗
QwQ Plus6.5%2026-06-23independentSource ↗
Z.ai: GLM 5.27.7%2026-06-21independentSource ↗
Grok 49.7%2026-06-20independentSource ↗
GLM-5.26.8%2026-06-17independentSource ↗
MoonshotAI: Kimi K2.7 Code8%2026-06-17independentSource ↗
Claude Fable 57.2%2026-06-14independentSource ↗
Claude Fable 58.3%2026-06-14independentSource ↗
Nemotron 3 Ultra8.3%2026-06-09independentSource ↗
Nemotron 3.5 Content Safety7%2026-06-09independentSource ↗
Nemotron 3 Ultra7.7%2026-06-09independentSource ↗
Qwen: Qwen3.7 Plus7.7%2026-06-07independentSource ↗
MiniMax: MiniMax M37.7%2026-06-05independentSource ↗
Gemini 3.1 Flash Image7.7%2026-06-02independentSource ↗
Gemini 3 Pro Image (Nano Banana Pro)8.3%2026-06-02independentSource ↗
Claude Opus 4.88.3%2026-06-02independentSource ↗
Hy-MT2-1.8B7%2026-05-26independentSource ↗
Hy-MT2-30B-A3B7.3%2026-05-26independentSource ↗
Hy-MT2-7B7.3%2026-05-26independentSource ↗
Qwen: Qwen3.7 Max8%2026-05-25independentSource ↗
SpaceXAI: Grok Build 0.17.7%2026-05-25independentSource ↗
Gemini 3.5 Flash8%2026-05-24independentSource ↗
Gemini 3 Pro9.6%2026-05-23independentSource ↗
Gemini 3.1 Flash Lite7.7%2026-05-12independentSource ↗
GPT Chat Latest8.3%2026-05-10independentSource ↗
Command A+7.7%2026-05-06independentSource ↗
Mistral Medium 3.58%2026-05-05independentSource ↗
SpaceXAI: Grok 4.37.7%2026-05-05independentSource ↗
Qwen: Qwen3.6 Flash7.7%2026-05-02independentSource ↗
GPT-5.5 Pro8.3%2026-04-28independentSource ↗
GPT-5.58.3%2026-04-28independentSource ↗
DeepSeek V4 Pro 04237.7%2026-04-28independentSource ↗
DeepSeek V4 Flash 04237%2026-04-28independentSource ↗
MiMo-V2.58.1%2026-04-27independentSource ↗
Xiaomi: MiMo-V2.5-Pro7.3%2026-04-27independentSource ↗
Xiaomi: MiMo-V2.57.3%2026-04-27independentSource ↗
Qwen: Qwen3.6 27B7.7%2026-04-27independentSource ↗
Hy37.3%2026-04-26independentSource ↗
GPT-5.4 Image 28.3%2026-04-26independentSource ↗
Qwen: Qwen3.5 Plus 2026-04-207.7%2026-04-25independentSource ↗
MoonshotAI: Kimi K2.68%2026-04-25independentSource ↗
Claude Opus 4.78.3%2026-04-21independentSource ↗
Qwen: Qwen3.6 35B A3B7.3%2026-04-20independentSource ↗
Z.ai: GLM 5.18%2026-04-11independentSource ↗
Gemma 4 26B A4B7.3%2026-04-08independentSource ↗
Gemma 4 31B7.3%2026-04-07independentSource ↗
Qwen: Qwen3.6 Plus7.7%2026-04-07independentSource ↗
Z.ai: GLM 5V Turbo8%2026-04-06independentSource ↗
Lyria 3 Pro7%2026-04-04independentSource ↗
Lyria 3 Clip7%2026-04-04independentSource ↗
Reka Edge7%2026-03-25independentSource ↗
MiniMax: MiniMax M2.77.7%2026-03-23independentSource ↗
GPT-5.4 Nano7.7%2026-03-22independentSource ↗
GPT-5.4 Mini8%2026-03-22independentSource ↗
Z.ai: GLM 5 Turbo8%2026-03-20independentSource ↗
Qwen: Qwen3.5-9B7.3%2026-03-15independentSource ↗
SpaceXAI: Grok 4.20 Multi-Agent7.7%2026-03-14independentSource ↗
SpaceXAI: Grok 4.207.7%2026-03-14independentSource ↗
ByteDance Seed: Seed-2.0-Lite7.7%2026-03-14independentSource ↗
GPT-5.4 Pro8.3%2026-03-10independentSource ↗
GPT-5.48.3%2026-03-10independentSource ↗
Gemini 3.1 Flash Lite7.7%2026-03-08independentSource ↗
Gemini 3.1 Flash Image7.7%2026-03-03independentSource ↗
GPT-5.3-Codex8.3%2026-03-01independentSource ↗
ByteDance Seed: Seed-2.0-Mini7.3%2026-03-01independentSource ↗
Qwen: Qwen3.5-35B-A3B7.3%2026-03-01independentSource ↗
Qwen: Qwen3.5-27B7.7%2026-03-01independentSource ↗
Qwen: Qwen3.5-122B-A10B7.7%2026-03-01independentSource ↗
Qwen: Qwen3.5-Flash7.3%2026-03-01independentSource ↗
Gemini 3.1 Pro Custom Tools8.3%2026-02-24independentSource ↗
Gemini 3.1 Pro8.3%2026-02-24independentSource ↗
Claude Sonnet 4.68.3%2026-02-22independentSource ↗
Qwen: Qwen3.5 Plus 2026-02-157.7%2026-02-21independentSource ↗
Qwen: Qwen3.5 397B A17B8%2026-02-21independentSource ↗
Z.ai: GLM 58%2026-02-16independentSource ↗
MiniMax: MiniMax M2.57.3%2026-02-16independentSource ↗
Claude Opus 4.68.3%2026-02-10independentSource ↗
MoonshotAI: Kimi K2.57.7%2026-02-01independentSource ↗
MiniMax: MiniMax M2-her7.7%2026-01-28independentSource ↗
Z.ai: GLM 4.7 Flash7.3%2026-01-24independentSource ↗
GPT-5.2-Codex8.3%2026-01-19independentSource ↗
MiniMax: MiniMax M2.17.7%2025-12-28independentSource ↗
Z.ai: GLM 4.77.7%2025-12-27independentSource ↗
o38%2025-12-25independentSource ↗
Gemini 3 Flash7.7%2025-12-22independentSource ↗
GPT-5.2 Chat8.3%2025-12-16independentSource ↗
GPT-5.2 Pro8.3%2025-12-16independentSource ↗
GPT-5.28.3%2025-12-16independentSource ↗
Z.ai: GLM 4.6V7.3%2025-12-13independentSource ↗
GPT-5.1-Codex-Max8%2025-12-09independentSource ↗
DeepSeek V3.27.3%2025-12-06independentSource ↗
Claude Opus 4.58.3%2025-11-29independentSource ↗
Gemini 3 Pro Image (Nano Banana Pro)8.3%2025-11-25independentSource ↗
GPT-5.18%2025-11-18independentSource ↗
GPT-5.1-Codex8%2025-11-18independentSource ↗
GPT-5.1-Codex-Mini7.7%2025-11-18independentSource ↗
Mistral Large 39.1%2025-11-17independentSource ↗
MoonshotAI: Kimi K2 Thinking7.7%2025-11-11independentSource ↗
MiniMax: MiniMax M27.7%2025-10-28independentSource ↗
Qwen: Qwen3 VL 8B Instruct7.3%2025-10-19independentSource ↗
GPT-5 Pro8.3%2025-10-11independentSource ↗
Qwen: Qwen3 VL 30B A3B Thinking7.3%2025-10-11independentSource ↗
Qwen: Qwen3 VL 30B A3B Instruct7.3%2025-10-11independentSource ↗
Claude Haiku 4.58%2025-10-06independentSource ↗
Claude Sonnet 4.58.3%2025-10-04independentSource ↗
Qwen: Qwen3 VL 235B A22B Instruct7.7%2025-09-28independentSource ↗
Qwen: Qwen3 Coder Flash7.3%2025-09-22independentSource ↗
Qwen: Qwen3 Next 80B A3B Instruct7.7%2025-09-16independentSource ↗
MoonshotAI: Kimi K2 09057.7%2025-09-09independentSource ↗
GPT-58%2025-08-12independentSource ↗
GPT-5 Mini7.7%2025-08-12independentSource ↗
GPT-5 Nano7.3%2025-08-12independentSource ↗
Claude 4.1 Opus8.3%2025-08-10independentSource ↗
Qwen: Qwen3 Coder 30B A3B Instruct7.3%2025-08-05independentSource ↗
Qwen: Qwen3 30B A3B Instruct 25077.3%2025-08-03independentSource ↗
Qwen: Qwen3 235B A22B Thinking 25078%2025-07-30independentSource ↗
Qwen: Qwen3 Coder 480B A35B7.3%2025-07-28independentSource ↗
ByteDance: UI-TARS 7B7%2025-07-27independentSource ↗
Qwen: Qwen3 235B A22B Instruct 25077.3%2025-07-26independentSource ↗
MoonshotAI: Kimi K2 07117.7%2025-07-16independentSource ↗
ByteDance Seed: Seed 1.6 Flash7.3%2025-06-30independentSource ↗
ByteDance Seed: Seed 1.67.7%2025-06-30independentSource ↗
MiniMax: MiniMax M17.7%2025-06-22independentSource ↗
Sarvam 105B8.4%2025-06-20independentSource ↗
o3 Pro8.3%2025-06-15independentSource ↗
Sarvam Vision (Document AI)7.9%2025-05-25independentSource ↗
Amazon Nova 2 Lite7.8%2025-05-23independentSource ↗
Amazon Nova 2 Lite7.6%2025-05-23independentSource ↗
Qwen: Qwen3 30B A3B7.3%2025-05-03independentSource ↗
Qwen: Qwen3 14B7.3%2025-05-03independentSource ↗
Qwen: Qwen3 32B7.3%2025-05-03independentSource ↗
o4 Mini High8%2025-04-21independentSource ↗
o38%2025-04-21independentSource ↗
o4 Mini8%2025-04-21independentSource ↗
Muse Wonder8.8%2025-04-20independentSource ↗
Muse Wonder Full9%2025-04-20independentSource ↗
GPT-4.18%2025-04-19independentSource ↗
GPT-4.1 Mini7.7%2025-04-19independentSource ↗
GPT-4.1 Nano7.3%2025-04-19independentSource ↗
Gemma 3 27B7.6%2025-04-17independentSource ↗
Nemotron 3 Ultra 550B8.6%2025-04-15independentSource ↗
Sarvam 30B8%2025-04-15independentSource ↗
Llama 4 Maverick9.3%2025-04-10independentSource ↗
Llama 4 Scout9.2%2025-04-10independentSource ↗
Gemini 2.5 Pro7.9%2025-03-30independentSource ↗
Gemini 2.5 Pro8%2025-03-30independentSource ↗
Gemini 2.5 Pro 06-058%2025-03-30independentSource ↗
Microsoft-Decision-18.2%2025-03-27independentSource ↗
Nemotron 3 Super 120B8.1%2025-03-25independentSource ↗
OpenAI Decisions API8.8%2025-03-23independentSource ↗
Jev 1.148.5%2025-03-19independentSource ↗
Krutrim-28.2%2025-03-17independentSource ↗
Reka Flash 37%2025-03-17independentSource ↗
Amazon Nova Premier 1.08.6%2025-03-15independentSource ↗
Amazon: Nova Premier 1.08.3%2025-03-15independentSource ↗
QwQ 32B8.8%2025-03-10independentSource ↗
Saaras v3 (Speech-to-Text)7.8%2025-03-06independentSource ↗
Bulbul v3 (Text-to-Speech)7.6%2025-03-06independentSource ↗
Phi-4-multimodal7.6%2025-03-05independentSource ↗
GPT-4.59.4%2025-03-04independentSource ↗
Claude 3.7 Sonnet7.9%2025-03-01independentSource ↗
Grok 37.9%2025-02-22independentSource ↗
Grok 3 Mini8.1%2025-02-22independentSource ↗
Phi-4-mini7.1%2025-02-20independentSource ↗
Perplexity: Sonar Deep Research8%2025-02-19independentSource ↗
Gemini 2.5 Flash (Thinking)7.9%2025-02-15independentSource ↗
Perplexity: Sonar Reasoning Pro8%2025-02-11independentSource ↗
Gemini 2.0 Pro9.1%2025-02-10independentSource ↗
Gemini 2.0 Flash9.1%2025-02-10independentSource ↗
Sonar Reasoning Pro7.7%2025-02-10independentSource ↗
Qwen: Qwen3 Coder Next7.3%2025-02-08independentSource ↗
o3-mini7.7%2025-02-05independentSource ↗
o3 Mini High8%2025-02-05independentSource ↗
o3 Mini8%2025-02-05independentSource ↗
Qwen 2.5 Max9%2025-02-02independentSource ↗
Qwen: Qwen2.5 VL 72B Instruct7.3%2025-01-31independentSource ↗
Qwen: Qwen-Plus7.3%2025-01-30independentSource ↗
Ollama DeepSeek-R1 (Q4_K_M)8.5%2025-01-27independentSource ↗
Perplexity: Sonar Pro Search8.3%2025-01-26independentSource ↗
Perplexity: Sonar Pro8.3%2025-01-26independentSource ↗
DeepSeek-R17.7%2025-01-25independentSource ↗
Gemini 2.0 Flash Thinking9.2%2025-01-25independentSource ↗
Kimi k1.58%2025-01-25independentSource ↗
R1 05287.7%2025-01-25independentSource ↗
MiniMax-Text-017.8%2025-01-20independentSource ↗
MiniMax: MiniMax-017.7%2025-01-20independentSource ↗
Codestral 25.017.4%2025-01-19independentSource ↗
Phi-48%2025-01-13independentSource ↗
Phi 47%2025-01-13independentSource ↗
DeepSeek-V39%2024-12-31independentSource ↗
DeepSeek V3.2 Exp7.3%2024-12-31independentSource ↗
DeepSeek V3.1 Terminus7.3%2024-12-31independentSource ↗
DeepSeek V3.17.3%2024-12-31independentSource ↗
DeepSeek V3 03247.7%2024-12-31independentSource ↗
DeepSeek V37.7%2024-12-31independentSource ↗
o18.3%2024-12-22independentSource ↗
Sonar Pro7.8%2024-12-15independentSource ↗
Ollama Llama 3.3 (Q4_K_M)7.9%2024-12-13independentSource ↗
Llama 3.3 70B8.9%2024-12-11independentSource ↗
Llama 3.3 70B Instruct7.5%2024-12-11independentSource ↗
Llama 3.3 70B Instruct7.3%2024-12-11independentSource ↗
o19%2024-12-10independentSource ↗
o1-pro8.3%2024-12-10independentSource ↗
Amazon Nova Pro7.8%2024-12-08independentSource ↗
Amazon Nova Lite7.1%2024-12-08independentSource ↗
Amazon Nova Pro 1.08.1%2024-12-08independentSource ↗
Amazon Nova Lite 1.07.5%2024-12-08independentSource ↗
Amazon Nova Micro 1.06.9%2024-12-08independentSource ↗
Amazon: Nova Lite 1.07.3%2024-12-08independentSource ↗
Amazon: Nova Micro 1.07%2024-12-08independentSource ↗
Amazon: Nova Pro 1.08%2024-12-08independentSource ↗
Command R7B (12-2024)7%2024-12-06independentSource ↗
QwQ-32B7.7%2024-12-03independentSource ↗
Solar Pro7.4%2024-11-25independentSource ↗
Krutrim Pro7.9%2024-11-25independentSource ↗
GPT-4o8%2024-11-25independentSource ↗
Upstage: Solar Pro 47.3%2024-11-25independentSource ↗
Jev8.4%2024-11-20independentSource ↗
Sonar Large7.7%2024-11-20independentSource ↗
Sonar7.5%2024-11-20independentSource ↗
TypeSafe: Jev Router7%2024-11-20independentSource ↗
Perplexity: Sonar7.3%2024-11-20independentSource ↗
Qwen 2.5 Coder 32B7.7%2024-11-17independentSource ↗
Qwen2.5 Coder 32B Instruct7.3%2024-11-17independentSource ↗
Hunyuan-Large8%2024-11-10independentSource ↗
Claude 3.5 Haiku7.2%2024-11-09independentSource ↗
Claude 3.5 Sonnet9.2%2024-10-27independentSource ↗
Claude 3.5 Sonnet (v2)7.7%2024-10-27independentSource ↗
BharatGPT-14B7.7%2024-10-23independentSource ↗
Yi-Lightning7.9%2024-10-20independentSource ↗
Llama-3.1-Nemotron-70B-Instruct8.1%2024-10-20independentSource ↗
Llama 3.2 1B Instruct7.3%2024-09-30independentSource ↗
Llama 3.2 3B Instruct7.3%2024-09-30independentSource ↗
Qwen 2.5 72B Instruct7.4%2024-09-24independentSource ↗
Qwen: Qwen2.5 7B Instruct7%2024-09-24independentSource ↗
Qwen2.5 72B Instruct7.3%2024-09-24independentSource ↗
Switchyard7%2024-09-20independentSource ↗
Mistral Small 47.3%2024-09-20independentSource ↗
GPT Audio8%2024-09-20independentSource ↗
GPT Audio Mini7.7%2024-09-20independentSource ↗
Nemotron 3 Nano 30B A3B7.3%2024-09-20independentSource ↗
Devstral 2 25127.7%2024-09-20independentSource ↗
Ministral 3 14B 25127%2024-09-20independentSource ↗
Ministral 3 8B 25127%2024-09-20independentSource ↗
Ministral 3 3B 25127%2024-09-20independentSource ↗
Voxtral Small 24B 25077.3%2024-09-20independentSource ↗
gpt-oss-safeguard-20b7.3%2024-09-20independentSource ↗
Gemini 2.5 Flash Image7.7%2024-09-20independentSource ↗
Z.ai: GLM 4.67.7%2024-09-20independentSource ↗
Mistral Medium 3.17.7%2024-09-20independentSource ↗
Z.ai: GLM 4.5V7.7%2024-09-20independentSource ↗
gpt-oss-120b7.3%2024-09-20independentSource ↗
gpt-oss-20b7%2024-09-20independentSource ↗
Codestral 25087.3%2024-09-20independentSource ↗
Z.ai: GLM 4.57.7%2024-09-20independentSource ↗
Z.ai: GLM 4.5 Air7.3%2024-09-20independentSource ↗
Gemini 2.5 Flash Lite7.3%2024-09-20independentSource ↗
Hunyuan A13B Instruct7.3%2024-09-20independentSource ↗
Mistral Small 3.2 24B7.3%2024-09-20independentSource ↗
Gemini 2.5 Flash7.7%2024-09-20independentSource ↗
Mistral Medium 37.7%2024-09-20independentSource ↗
Llama Guard 4 12B7%2024-09-20independentSource ↗
Llama 4 Maverick7.3%2024-09-20independentSource ↗
Llama 4 Scout7.3%2024-09-20independentSource ↗
Mistral Small 3.1 24B7.3%2024-09-20independentSource ↗
Gemma 3 4B7%2024-09-20independentSource ↗
Gemma 3 12B7%2024-09-20independentSource ↗
Gemma 3 27B7.3%2024-09-20independentSource ↗
Saba7.3%2024-09-20independentSource ↗
Mistral Small 37%2024-09-20independentSource ↗
Mistral Nemo7%2024-09-20independentSource ↗
Gemma 2 27B7.3%2024-09-20independentSource ↗
Mixtral 8x22B Instruct8%2024-09-20independentSource ↗
GPT-4 Turbo8.3%2024-09-20independentSource ↗
Mistral Large8%2024-09-20independentSource ↗
GPT-3.5 Turbo (older v0613)7.7%2024-09-20independentSource ↗
GPT-3.5 Turbo Instruct7.7%2024-09-20independentSource ↗
GPT-3.5 Turbo 16k8%2024-09-20independentSource ↗
GPT-3.5 Turbo7.7%2024-09-20independentSource ↗
GPT-48.3%2024-09-20independentSource ↗
GLM-4-Plus7.7%2024-08-25independentSource ↗
Sarvam 2B (Sarvam-1)7.2%2024-08-19independentSource ↗
GPT-4o8%2024-08-11independentSource ↗
Command R (08-2024)7.3%2024-08-06independentSource ↗
Gemma 2 2B6.2%2024-08-05independentSource ↗
Mistral Large 27.5%2024-07-29independentSource ↗
Mistral Large 27.4%2024-07-29independentSource ↗
Mistral Large 3 25127.7%2024-07-29independentSource ↗
Llama 3.1 405B8.9%2024-07-28independentSource ↗
Llama 3.1 70B Instruct7.3%2024-07-28independentSource ↗
Llama 3.1 8B Instruct7%2024-07-28independentSource ↗
GPT-4o mini6.9%2024-07-23independentSource ↗
GPT-4o-mini7.3%2024-07-23independentSource ↗
GPT-4o-mini7.3%2024-07-23independentSource ↗
ERNIE 4.0 Turbo7.7%2024-07-03independentSource ↗
Gemma 2 27B7.3%2024-07-02independentSource ↗
Gemma 2 9B7%2024-07-02independentSource ↗
Nemotron-4-340B-Instruct8%2024-06-19independentSource ↗
Nemotron-4-340B-Reward8.1%2024-06-19independentSource ↗
Hanooman 7B7.5%2024-05-20independentSource ↗
Gemini 1.5 Flash6.7%2024-05-19independentSource ↗
GPT-4o9.4%2024-05-18independentSource ↗
GPT-4o8%2024-05-18independentSource ↗
GPT-4o8.3%2024-05-18independentSource ↗
Snowflake Arctic7.2%2024-04-29independentSource ↗
Reka Core7.5%2024-04-20independentSource ↗
WizardLM-2 8x22B7.8%2024-04-20independentSource ↗
Command R+7%2024-04-09independentSource ↗
DBRX Instruct7.3%2024-04-01independentSource ↗
Claude 3 Opus7.4%2024-03-09independentSource ↗
Gemini 1.5 Pro7.3%2024-02-20independentSource ↗
Airavata 7B7.1%2024-01-30independentSource ↗
IndicTrans-27.4%2023-09-20independentSource ↗
GPT-48.99%2023-06-09independentSource ↗
Nemotron 3 Super7.3%2023-03-16independentSource ↗

Human Baseline & Difficulty Horizon

Calibrated human reference points, specialist benchmarks, and ceiling thresholds.
Measured Human Score80.0%Domain Expert Baseline
Baseline Protocol & Interpretation

Human annotator agreement rate on multi-turn conversational dialogue quality.

Metric & Scoring Methodology

Verification protocols, aggregation formulas, and specialized metric variants.
Primary Metric:average judge score (1–10)
Scoring Engine:llm-judge

Dataset & Compute Cost

Evaluation volume, public availability, API pricing, and local hardware requirements.
Total Dataset Size80Annotated evaluation items
Public Test SetPublicOpenly mirrored on repositories
Access GatingOpen AccessUnrestricted download
Evaluation LicenseOpen AccessDataset usage and redistribution terms
Frontier API Compute Cost:

$5 – $20 USD for full benchmark evaluation run on frontier APIs.

Recommended Local GPU Setup:

1x NVIDIA RTX 4090 (24GB) or A100 (40GB/80GB) via vLLM / SGLang

Official Dataset & Benchmark Files:Download / View Dataset Repository ↗

How to Run & Reproduce

Standardized evaluation protocols, CLI commands, and reproducible runner templates.
Prompt Regimezero-shot
Reasoning Modedirect
Sampling Temp0
Pass@k Budgetk = 1
Tools & SandboxPure Text
Scoring Verifierllm-judge
Option AEleutherAI LM-Evaluation-Harness (Open-Weight Models)
lm_eval --model hf --model_args pretrained=<model_path> --tasks mt-bench --batch_size auto
Option BOpenCompass Evaluation Framework
opencompass --datasets mt-bench --models <model_config>
Python APIDeterministic Inference Loop Snippet
# Standard API Evaluation Loop
from openai import OpenAI

client = OpenAI()
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": prompt}],
    temperature=0.0,
)
Standardized Reporting Requirement:

When publishing results for MT-Bench, always report the exact prompt template, few-shot exemplar ordering, sampling temperature (temperature=0), maximum reasoning budget tokens, and the precise timestamped model snapshot ID.

Contamination & Memorization Analysis

Audit of pretraining exposure risks, memorization vectors, and refresh policies.
Overall Contamination Risk:MEDIUM
Refresh Cadence:

Static fixed snapshot

Test Set Exposure:

Public on web / HuggingFace