BenchMIRT reveals hidden dimensions behind LLM benchmarks
BenchMIRT is a new method for auditing large language model benchmarks at the level of individual prompts, allowing researchers to see which capabilities drive each score.
Benchmarks are built to measure specific abilities such as safety or reasoning, yet the items inside often require additional skills like tracking context or reasoning from evidence rather than relying on the stated goal.
Within a single benchmark, groups of prompts can reflect different traits; for example WildJailbreak mixes harmful jailbreak items with benign prompts that test refusal behavior, so averaging scores can hide important distinctions.
BenchMIRT applies multidimensional Item Response Theory to separate underlying capabilities, estimating how difficult each question is and how well it distinguishes models that differ in specific abilities.
The analysis was trained on results from one hundred large language models across sixteen benchmarks and more than thirty four thousand questions, covering six reasoning benchmarks and ten safety benchmarks.
Without being told which benchmarks measured which traits, BenchMIRT recovered two dominant dimensions, safety and general reasoning, and repeated analyses confirmed the stability of these dimensions.
Many benchmarks align with their intended focus, but BBQ shows a stronger link to reasoning than to safety, and WMDP associates more with reasoning because refusing dangerous knowledge is the expected response.
HarmBench contains both safety oriented prompts and reasoning oriented copyright questions, illustrating how a single benchmark can combine multiple signals.
Bar charts display Pearson correlation across models, with pink indicating general reasoning and teal indicating safety, while bold and underline highlight the strongest relationships.
Com informações de: Hugging Face Blog