Instruction FollowingActive
Trust Score:
93

IFBench

Verifiable instruction-following benchmark built from out-of-distribution constraints, designed to test generalisation beyond IFEval-style prompts.

Launched: 2025-07-03Refresh: static
Status Assessment (active):

Actively evaluated by Artificial Analysis on current frontier models; no saturation evidence recorded.

Performance Timeline

Longitudinal progression of model scores against human baselines.

Performance & Historical Trajectory

Empirical score progression across model release dates and evaluation rounds.

Independent Vendor-Reported
ModelScoreDateSource TypeProvenance
JT-35B-Flash42%2026-11-15independentSource ↗
Qwen3.5 Omni Plus51.2%2026-11-15independentSource ↗
Nova 2.0 Omni66.2%2026-11-15independentSource ↗
Qwen3.5-122B-A10B50.8%2026-11-15independentSource ↗
Gemma 4 12B45.2%2026-11-15independentSource ↗
Gemma 3 270M12.1%2026-11-15independentSource ↗
Qwen3.5 2B29.1%2026-11-15independentSource ↗
Qwen3 Omni 30B A3B Instruct31.2%2026-11-15independentSource ↗
Nova 2.0 Pro79%2026-11-15independentSource ↗
Qwen3.5 397B A17B78.8%2026-11-15independentSource ↗
Mistral Small 448.2%2026-11-15independentSource ↗
gpt-oss-20b57.8%2026-11-15independentSource ↗
o371.4%2026-11-15independentSource ↗
North Mini Code57.6%2026-11-15independentSource ↗
Granite 4.0 Micro24.8%2026-11-15independentSource ↗
Qwen3.5-9B37.8%2026-11-15independentSource ↗
Hermes 4 - Llama-3.1 70B29%2026-11-15independentSource ↗
Nemotron 3 Nano 30B A3B37.5%2026-11-15independentSource ↗
Qwen3 Next 80B A3B Instruct39.7%2026-11-15independentSource ↗
MiniMax M382.9%2026-11-15independentSource ↗
NVIDIA Nemotron Nano 9B V227.1%2026-11-15independentSource ↗
Command A+73.9%2026-11-15independentSource ↗
EXAONE 4.0 32B33.5%2026-11-15independentSource ↗
Qwen3.5 4B33.3%2026-11-15independentSource ↗
Olmo 3 7B Think41.5%2026-11-15independentSource ↗
HyperCLOVA X SEED Think (32B)37.9%2026-11-15independentSource ↗
HyperNova 60B 260566.5%2026-11-15independentSource ↗
Nova Micro29.4%2026-11-15independentSource ↗
INTELLECT-334%2026-11-15independentSource ↗
Olmo 3.1 32B Think66%2026-11-15independentSource ↗
Nex-N2-Pro66.2%2026-11-15independentSource ↗
LFM2.5-VL-1.6B33.1%2026-11-15independentSource ↗
JT-MINI36.7%2026-11-15independentSource ↗
Molmo2-8B26.9%2026-11-15independentSource ↗
Qwen3.5 0.8B21.6%2026-11-15independentSource ↗
MiniCPM-V 4.6 1.3B26.7%2026-11-15independentSource ↗
Sarvam 105B34.4%2026-11-15independentSource ↗
Llama 3.1 Instruct 405B39%2026-11-15independentSource ↗
Ring-flash-2.043.3%2026-11-15independentSource ↗
LFM2.5-1.2B-Instruct43.8%2026-11-15independentSource ↗
Apertus 70B Instruct25.9%2026-11-15independentSource ↗
Doubao Seed Code51.4%2026-11-15independentSource ↗
Gemma 4 E4B40.5%2026-11-15independentSource ↗
Granite 4.0 350M15.9%2026-11-15independentSource ↗
Kimi Linear 48B A3B Instruct28.1%2026-11-15independentSource ↗
Solar Open 100B57.7%2026-11-15independentSource ↗
GPT-5.3-Codex75.4%2026-11-15independentSource ↗
LFM2.5-1.2B-Thinking41.8%2026-11-15independentSource ↗
K2 Think V262.8%2026-11-15independentSource ↗
Nemotron 3 Ultra81.4%2026-11-15independentSource ↗
Tri-21B-Think54.6%2026-11-15independentSource ↗
Granite 4.0 H Small31.5%2026-11-15independentSource ↗
Llama 3.2 Instruct 11B (Vision)30.4%2026-11-15independentSource ↗
LongCat Flash Lite43.1%2026-11-15independentSource ↗
Nemotron 3.5 Bolt E4B 4GB61.1%2026-11-15independentSource ↗
Nova 2.0 Lite70.7%2026-11-15independentSource ↗
Motif-2-12.7B-Reasoning57%2026-11-15independentSource ↗
Qwen3.6 35B A3B36.2%2026-11-15independentSource ↗
Ministral 3 3B26.8%2026-11-15independentSource ↗
Gemma 4 26B A4B72.4%2026-11-15independentSource ↗
NVIDIA Nemotron Nano 12B v2 VL31.9%2026-11-15independentSource ↗
Qwen3.5 Omni Flash38%2026-11-15independentSource ↗
Magistral Small 1.244.4%2026-11-15independentSource ↗
Ministral 3 14B32%2026-11-15independentSource ↗
Gemma 4 E2B33.6%2026-11-15independentSource ↗
Sarvam 30B26.5%2026-11-15independentSource ↗
MiniCPM5-1B49.3%2026-11-15independentSource ↗
Jamba 1.7 Large35.2%2026-11-15independentSource ↗
Nanbeige4.1-3B35.4%2026-11-15independentSource ↗
Tiny Aya Global20.1%2026-11-15independentSource ↗
Exaone 4.0 1.2B25.3%2026-11-15independentSource ↗
Hermes 4 - Llama-3.1 405B32.7%2026-11-15independentSource ↗
Magistral Medium 1.243%2026-11-15independentSource ↗
Apertus 8B Instruct22.4%2026-11-15independentSource ↗
Nemotron 3 Nano Omni 30B A3B Reasoning63.2%2026-11-15independentSource ↗
Kimi K2.7 Code63.1%2026-11-15independentSource ↗
Olmo 3 7B Instruct32.8%2026-11-15independentSource ↗
Llama 4 Maverick43%2026-11-15independentSource ↗
Olmo 3.1 32B Instruct39.2%2026-11-15independentSource ↗
Ministral 3 8B29.1%2026-11-15independentSource ↗
Llama 3.3 Instruct 70B47.1%2026-11-15independentSource ↗
Nemotron Cascade 2 30B A3B80.4%2026-11-15independentSource ↗
NVIDIA Nemotron 3 Nano 4B58.2%2026-11-15independentSource ↗
Phi-4-mini21.1%2026-11-15independentSource ↗
K2-V260.1%2026-11-15independentSource ↗
ERNIE 5.0 Thinking41.4%2026-11-15independentSource ↗
Granite 4.0 H 350M17.6%2026-11-15independentSource ↗
Apriel-v1.6-15B-Thinker69.1%2026-11-15independentSource ↗
Qwen3.7 Plus78%2026-11-15independentSource ↗
Step3 VL 10B50.2%2026-11-15independentSource ↗
Llama 4 Scout39.5%2026-11-15independentSource ↗
ERNIE 4.5 300B A47B39.1%2026-11-15independentSource ↗
Molmo 7B-D19.7%2026-11-15independentSource ↗
Jamba 1.7 Mini31.4%2026-11-15independentSource ↗
Trinity Large Thinking56.3%2026-11-15independentSource ↗
Qwen3 Omni 30B A3B (Reasoning)43.4%2026-11-15independentSource ↗
Gemini 3.1 Pro77.1%2026-11-15independentSource ↗
DiffusionGemma 26B A4B59.5%2026-11-15independentSource ↗
Falcon-H1R-7B54.4%2026-11-15independentSource ↗
Qwen3 Next 80B A3B (Reasoning)60.7%2026-11-15independentSource ↗
gpt-oss-120b69%2026-11-15independentSource ↗
Gemma 4 31B75.6%2026-11-15independentSource ↗
Llama 3.1 Nemotron Ultra 253B v138.2%2026-11-15independentSource ↗
Granite 4.0 H 1B26.2%2026-11-15independentSource ↗
Reka Flash 330.4%2026-11-15independentSource ↗
Mistral Medium 3.568.8%2026-11-15independentSource ↗
Mi:dm K 2.5 Pro49.3%2026-11-15independentSource ↗
LFM2.5-8B-A1B55.6%2026-11-15independentSource ↗
Nova Premier36.2%2026-11-15independentSource ↗
Cogito v2.146.3%2026-11-15independentSource ↗
Ring-2.6-1T44.6%2026-11-15independentSource ↗
Phi-423.5%2026-11-15independentSource ↗
Jamba Reasoning 3B52.4%2026-11-15independentSource ↗
LFM2 24B A2B45.9%2026-11-15independentSource ↗
KAT Coder Pro V266.7%2026-11-15independentSource ↗
Qwen3 Coder Next35.2%2026-11-15independentSource ↗

Human Baseline & Difficulty Horizon

Calibrated human reference points, specialist benchmarks, and ceiling thresholds.
Measured Human Scoren/aNot established
Baseline Protocol & Interpretation

No peer-reviewed human baseline has been published for this benchmark.

Metric & Scoring Methodology

Verification protocols, aggregation formulas, and specialized metric variants.
Primary Metric:constraint accuracy (%)
Scoring Engine:unit-tests

Dataset & Compute Cost

Evaluation volume, public availability, API pricing, and local hardware requirements.
Total Dataset Size—Annotated evaluation items
Public Test SetPublicOpenly mirrored on repositories
Access GatingOpen AccessUnrestricted download
Evaluation LicenseOpen AccessDataset usage and redistribution terms
Frontier API Compute Cost:

$5 – $20 USD for full benchmark evaluation run on frontier APIs.

Recommended Local GPU Setup:

1x NVIDIA RTX 4090 (24GB) or A100 (40GB/80GB) via vLLM / SGLang

How to Run & Reproduce

Standardized evaluation protocols, CLI commands, and reproducible runner templates.
Prompt Regimezero-shot
Reasoning Modedirect
Sampling Temp0
Pass@k Budgetk = 1
Tools & SandboxPure Text
Scoring Verifierunit-tests
Option AEleutherAI LM-Evaluation-Harness (Open-Weight Models)
lm_eval --model hf --model_args pretrained=<model_path> --tasks ifbench --batch_size auto
Option BOpenCompass Evaluation Framework
opencompass --datasets ifbench --models <model_config>
Python APIDeterministic Inference Loop Snippet
# Standard API Evaluation Loop
from openai import OpenAI

client = OpenAI()
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": prompt}],
    temperature=0.0,
)
Standardized Reporting Requirement:

When publishing results for IFBench, always report the exact prompt template, few-shot exemplar ordering, sampling temperature (temperature=0), maximum reasoning budget tokens, and the precise timestamped model snapshot ID.

Contamination & Memorization Analysis

Audit of pretraining exposure risks, memorization vectors, and refresh policies.
Overall Contamination Risk:MEDIUM
Refresh Cadence:

Static fixed snapshot

Test Set Exposure:

Public on web / HuggingFace