Automated Benchmark Auditing for AI Agents and Large Language Models

Summary

ABA (Auto Benchmark Audit) is an agentic pipeline that audits benchmark tasks rather than models: an Evidence Collector agent normalises heterogeneous artifacts (a Terminal-Bench directory, a SWE-bench task_patch inside JSON, a bare (instruction, gold_answer) pair) into one manifest, then an Auditor agent — Claude Code with Opus 4.7, filesystem and bash tools — reads the task, tests and harness against a fixed rubric and emits structured findings with category, severity, evidence paths, claim, and a suggested fix. Run over 168 benchmarks and 34,285 tasks across nine domains, it flags 25.7% of tasks with a major issue (severity 2, task effectively unsolvable) and a further 15.1% with only minor issues — under 60% of tasks are clean. Issues fall on three axes: Instruction (prompt underdetermines what the rubric demands), Environment (container lacks what the prompt assumes), Evaluation (tests too narrow, too broad, or the gold answer is simply wrong). Validation is layered rather than assumed: against a Terminal-Bench 2 maintainer fix PR written independently, ABA recovers 66.7% of the 21 issues strictly (81.0% partial) and flags real defects the PR missed; it independently flags both SWE-bench Verified tasks OpenAI named when retiring that benchmark; hand review of sampled findings gives 73% strict / 91% partial precision on major issues (92%/96% on SWE-bench Verified in trajectory mode). Filtering flagged tasks moves leaderboards: average score rises 9.9% on SWE-bench Verified and 9.6% on Terminal-Bench 2, up to 12.8 points on Dabstep, with model rankings shifting.

Key points

• The three-axis taxonomy (Instruction / Environment / Evaluation) plus a 3-point severity scale is the transferable artifact, and each finding is required to cite the file path it came from — an audit record, not prose criticism. Prior validity work (the Agentic Benchmark Checklist and similar) is expert prose over a fixed benchmark set; ABA’s contribution is making the audit executable and uniform across benchmark families.
• Issue rate tracks environment and grading complexity, not author quality. Math 13.2% major (numeric answers, deterministic graders, no environment) and Coding 14.1% at the low end; Safety/Alignment 42.2%, Medical 41.5%, Professional 38.4% at the high end. Agentic/Tool Use sits at 20.6% major plus 20.3% minor. The authors are explicit that this is “the natural residue of complexity”, not an indictment.
• Where the issues land depends on how you grade. Deterministic-checker domains are instruction-dominant (Math 65% of major issues are instruction, Science 69%); domains needing multi-faceted answers and LLM judges or rubrics are evaluation-dominant (Safety 51%, Retrieval 57%); complex runtimes push environment issues up (Agentic, Professional, Multimodal).
• Trajectory mode beats static mode, and the gap says what execution evidence buys. Replaying recorded agent runs flags 8.5% more major tasks (+6.1% tasks flagged overall) than reading artifacts alone; max-severity agreement between modes is only 29–63% (mean 53%). Runtime contamination and environment drift appear only in trajectories; metadata ambiguity and test-suite leakage appear mainly in static inspection. Neither subsumes the other.
• The validation hierarchy is a methodological point in itself: “fixes in the wild” (a maintainer PR, a benchmark retirement) rank above constructed gold-issue sets, because the actors had real stakes and no evaluative motive. Ground truth here is genuinely scarce — the only people qualified to build it are the benchmark authors, who are precisely the ones who missed the issues.
• Precision is real but not clean: 58.3% strict against the Terminal-Bench PR (a lower bound, since the PR is not exhaustive), 50.0% strict on BixBench, 40.7% on ScienceAgentBench, 73% on hand-reviewed major findings. An auditor agent produces a triage queue, not verdicts.
• Worked examples of each axis: FinanceAgent asks to rank “competitors” against a rubric-pinned peer set that omits KO’s canonical competitor; a Terminal-Bench task needs a YouTube video on bare ubuntu:24.04 and YouTube 403s cloud IPs, so 4 of 5 runs fail before writing an answer; an OSWorld reference file mixes /pdf/ and /abs/ URL forms so agents that follow the stated format fail exact match.
• Limitations: findings are LLM judgments validated on small samples; scope excludes subjective evaluation and meta-benchmarks by construction; the leaderboard-lift experiment removes flagged tasks, which is the optimistic direction (an agent penalised by a broken task gains) and the authors run a hypothesis test to show the extra trajectory findings are not just agent failures; per-domain audits vary in task coverage under budget.

Relevance

• ITSMBench — the base-rate paper. Before V2 claims validity, this says roughly a quarter of tasks in published, expert-authored, peer-reviewed benchmarks are broken enough to be unsolvable, and 20.6% of tasks in the agentic/tool-use domain specifically. That reframes the yardstick: the question is not whether V2 has zero bad tasks but whether its rate beats ~25%, and whether it is measured at all. The auditor is directly runnable on ITSMBench’s Harbor task packages (repo + task dirs + recorded trajectories is exactly the input shape it takes), and trajectory mode fits V1’s existing adversarial-run artifacts. What it does not cover: nothing about generating tasks, nothing about coverage or realism, nothing about whether a valid task is a worthwhile task, and no user-simulator or multi-agent dimension.
• WorldSmith — the strongest evidence in this cluster that an agent can do the auditing half of environment work at scale, with numbers: 34,285 tasks audited, 73–92% precision on major findings, 66.7% recall against an independent human fix set. Note the asymmetry with agent task generation: auditing is verified against external ground truth here, and it still only reaches triage-grade precision, so the automated-audit loop needs a human at the end.
• Benchmark Research — the Instruction/Environment/Evaluation × severity schema, the “audit-then-score” stance that benchmarks are living artifacts needing CI-style hardening, and the demonstration that filtering broken tasks changes rankings by ~10 points, which is larger than most published model deltas.
• Benchmark Validity — home concept since 2026-09-10

Questions it raises for ITSMBench:

• Run ABA (it is open source) over itsmbench-core as a pre-publication gate: what is V2’s major-issue rate, and do the findings cluster on Instruction (ambiguous ticket text) or Evaluation (over-specified DB-state assertions)? That number would be a cheap, credible line in the paper.
• Their environment axis barely applies to a mocked-API benchmark with a frozen clock and deterministic ids — arguably V2’s design already removes a whole failure class. Is that worth claiming explicitly, and does the mocked layer introduce a different class (drifting vendor semantics) that no existing taxonomy names?
• Static vs trajectory agreement is only ~53%, so an audit without recorded runs misses half the major issues. Does that mean V2’s task-authoring loop must generate reference trajectories for every task before the task can be considered audited?

Notes

(Our take: to add after reading; drafted 2026-09-09, not yet discussed)