Summary
Many apparent agent failures are failures of the benchmark: broken specifications, implicit assumptions and rigid evaluation scripts that reject valid alternatives. BenchGuard, from the University of Washington’s Allen School and Phylo, uses frontier LLMs as auditors of execution-based agent benchmarks. It takes each task’s four artifacts (instruction, ground-truth program, evaluation script, environment configuration), cross-verifies them in one structured LLM call plus deterministic static checks, and optionally adds agent solutions and execution logs as evidence. Findings are classified into a four-category taxonomy with severity and confidence, then handed to experts to adjudicate rather than auto-applied. On ScienceAgentBench it recovered 12 author-confirmed defects, including fatal errors that made tasks unsolvable; on the BIXBench Verified-50 subset a five-model ensemble exactly matched 83.3% of expert-identified issues for under $15 and under 12 minutes.
Key points
- Four defect categories (14 subcategories): INST (underspecified, contradictory or infeasible instructions), GT (wrong logic, data handling or format in the reference solution), EVAL (judge bias, spec–eval mismatch, incomplete coverage, wrong tolerances, unhandled non-determinism), ENV (missing dependencies, path errors, resource limits).
- Six-phase audit protocol in a single LLM call: task understanding, ground-truth correctness, evaluation logic, task specification, environment, then consolidation with a “one-fix test” (report a root cause once) and a “split test” (findings must be atomic).
- Instruction first: ambiguous cases default to the instruction category, because gold programs come from complete codebases and the instruction is where implicit assumptions go undocumented (“solution fixation”).
- The most frequent defect is underspecified instructions with rubrics anchored to a single reference implementation, penalizing valid alternative approaches (e.g. a gold program silently filters genes or adds a sex covariate the instruction never mentions).
- Execution-level evidence helps: adding agent programs and logs raised one model’s recall on ScienceAgentBench from 83.3% to 91.7%.
- Ensembles beat single auditors: five models together match 20/24 BIXBench expert issues exactly (83.3%) vs 54.2% for the best single model; each contributes unique detections.
- Human–AI collaboration boundary runs both ways: experts sometimes over-correct (adding format constraints a judge already handled), and LLM auditors catch structural inconsistencies experts missed. Scope excludes open-ended conversational tasks and agents with irreversible side effects.
Relevance
An automated version of the human audits behind SWE-Bench Verified and OSWorld-Verified, and direct evidence for Benchmark Validity that human-reviewed agent benchmarks still carry fatal defects. For WorldSmith it is a ready-made inspection step: the four-artifact cross-check maps onto the verifier-soundness gate, at a cost (under $15 per 50 tasks) that makes running it on every generated environment plausible. Its own caveat matters for generated tasks too: fixing an instruction until the evaluator passes can over-specify it.
Notes
From paper club, 19 May 2026:
It reaffirms the notion that there are false failures in many benchmarks that are actually the result of a benchmark issue rather than an agent issue. While this makes sense, benchmark repair can lead to overspecifying tasks. If you keep adding context until the verifier passes, you may leak information that the agent should have found out on its own. In this realm, we’re also exploring atomic fact decomposition: breaking every task instruction into individual facts, then classifying each one based on whether or not the agent can look it up in the environment.