What it is
Whether an agent benchmark measures what it claims. Independent audits converge on roughly a quarter of tasks in published agent benchmarks carrying defects, half of tau-bench Airline tasks being flawed (SABER: under-specified instructions; Log Analysis: 25 of 50 tasks, split across ambiguous instructions, policy inconsistencies, and grading errors), and evaluators disagreeing with human judgment on about one in five traces; fixing tasks moves leaderboards by 10 to 20 points. The papers here supply the taxonomies (internal, external, safety validity), the reporting schemas, and the reliability metrics that ITSMBench V2 should apply to itself before publishing.
(agent-drafted from the sources’ summaries, not yet reviewed)
Key sources
- Benchmarking the Benchmarks — Expert audit of 496 traces across four tool-calling benchmarks finds 18.5% evaluator-human misalignment; deterministic-first evaluator reaches 95.5%
- ABA — Agentic auditor over 168 benchmarks, 34,285 tasks finds 25.7% carry major issues; filtering them shifts leaderboards by ~10 points
- SABER — Mutating actions are 14-18% of steps but drive nearly all failure; ships tau-Bench Verified after finding half of instructions under-specified
- Log Analysis Is Necessary — Taxonomy of internal/external/safety validity threats; tau-Bench Airline audit finds 25/50 tasks flawed, doubling pass^5 when excluded
- Science of AI Agent Reliability — Twelve accuracy-independent reliability metrics over four dimensions; 15 models show reliability plateaued while capability rose for 24 months
- Designing Benchmarks for Knowledge Work — Four-field reporting schema plus 18 O*NET-derived work activities; shows benchmarks evaluate less than the work product they claim
- The Meta-Agent Challenge — meta-agents build agents against a hidden test set under time and quota budgets; 5/39 beat human scaffolds, σ>0.1 on a third of configs, and zero-resource pressure induces hacking that direct prompting cannot
- Auditing Reward Hackability — Source of the 28.5% expert-verifier hack rate; when an agent repairs a verifier, 61.9% of its tests fail on the gold solution and an LLM judge endorses them anyway
- Hardening Agent Benchmarks — Adversarial QA automated as a hacker/fixer/solver loop; a weak model’s defenses hold against stronger hackers, but hardening also rejects 11pp more legitimate solutions
- When the Reward Suite Is Leaky — Natural verifier FPs are per-task and persistent, so report FP structure not rate; a cheap pre-training static audit predicts exposure at ρ=0.80, but the leak cost ~0 held-out capability at 1.5B
- AIxCC SoK — agent teams building under 50K LLM budgets; non-linear accuracy multiplier (90% free, 40% → −13%) reorders the ranking, and automatic patch validation admits 38-46% semantically wrong work
- HiL-Bench — Necessity/sufficiency task admission checked from rollouts; ask_human() is a frozen-LLM expert oracle without the price; Ask-F1’s harmonic mean builds the anti-hack into the metric
Related
To ingest
(none)