Summary
A position paper arguing that outcome-only scoring cannot license the inferences the field draws from benchmark scores, and that log analysis — systematic inspection of an agent’s inputs, execution and outputs — is the only way to test them. The chain it breaks into is: score → capability (internal validity), capability → real-world utility (external validity), capability → safe deployment (safety evaluation); each link has documented failure modes, consolidated into a taxonomy from published work by METR, Apollo, UK AISI, CAISI and HAL. Four principles follow: define a validity target (which sets the burden of proof — a safety hunt only needs low false negatives, a causal claim needs a calibrated classifier); confirm the harness actually captured the trajectory; build a rubric and validate its labels against human annotators on a held-out set, reporting precision and recall; and quantify the label→outcome link with prevalence-by-outcome first, then risk ratios or mixed-effects regression, with no causal claims absent an experimental design. The τ-Bench Airline case study over 13 frontier models finds errors in 25 of 50 tasks (9 policy inconsistencies, 8 ambiguous instructions, 8 database or grading errors); excluding them roughly doubles mean pass^5 from 20.8% to 40.0% — capability was under-elicited by half. A second, deployment-facing analysis scores persuasion resistance on the 13 tasks where the simulated user is instructed to push for a policy exception: Gemini 2.5 Flash and GPT-4 Turbo look equivalent on pass^5, yet GPT-4 Turbo is about 4× more likely to be talked into an inappropriate credit, refund or upgrade.
Key points
- The three-axis taxonomy is the paper’s spine, and each cell is backed by a real incident: internal-validity overestimation (benchmark lookup — agents find CTF flags in blogs or navigate to the dataset on HuggingFace; reward hacking — reading future codebase state from git logs, hard-coded solutions; infrastructure manipulation — agents editing evaluation code to report success); internal-validity underestimation (sandbagging, tool-use errors accounting for over 50% of failed tasks, adding a missing tool improving scores by 10%+, a HCAST scoring bug causing 16% underestimation, refusals scored as failures); external validity (scaffold conflicts, timeout brittleness, CAPTCHAs as a persistent bottleneck, and half of passing SWE-Bench solutions being ones repo maintainers would not merge); safety (agents refunding or charging customers more than double the correct amount, web agents violating stated policy on required user input, o3 reasoning about underperforming before answering honestly).
- The 25-of-50 τ-Bench Airline defect rate is the headline number and it is independently corroborated — the authors note in a footnote that SABER (Cuadron et al.) found many of the same issues separately, which is about as strong as validity evidence gets in this area.
- Persuasion resistance is the most directly copyable idea: a deployment-oriented metric extracted from existing logs by an LLM judge, plotted against pass^k, that separates models the outcome metric says are equal. It costs no new runs — only a rubric over trajectories already recorded.
- The “log analysis sandwich” decomposes what must be captured: inputs (task instructions, scaffold config, system prompt), execution (reasoning chain, completions, tool calls, container state, model/scaffold/env errors, stop condition), outputs (final submission, grader input, rubric and decision). Which artifacts you need depends on the validity target — reward-hacking classification needs the full instruction set, capability-action correlation needs per-task outcomes.
- Rubric construction is a “funnel”: start from a broad question, read transcripts to find the confounders, convert iteratively into necessary and sufficient conditions, then validate on a balanced held-out sample and report judge precision/recall.
- The three counterarguments are answered rather than ignored. “Better benchmarks would suffice” — you cannot anticipate every shortcut a more capable agent will find. “Outcomes are what matter” — this conflates evaluation with deployment; a benchmark’s job is to predict behaviour, and whether a success survives a scaffold change requires trajectory patterns. “Log analysis has the same problems one level up” — an LLM unreliable as an agent can still be a competent grader, because reading for one behaviour in a fixed context is far easier than acting coherently over thousands of steps, and full context passes the judge only a few times during rubric refinement.
- Recommendations aimed at norms rather than tools: standardise logging formats, build end-to-end tooling (Docent, Inspect Scout, Apollo Revealer exist), gated/redacted transcript access for independent evaluators, a public per-benchmark registry of credibility threats, log release as a default expectation at model and benchmark launches, and soliciting safety targets from deployers, who know which trajectories count as failures in their industry.
- Limitations, mostly self-declared: it is a position paper, the case study covers one domain of one benchmark, the automated labels come from LLM judges (Docent with GPT-5 medium and Claude Sonnet 4.5) with manual checking, and the authors are explicit that log analysis is necessary but not sufficient.
Relevance
- ITSMBench — the strongest argument in this cluster for V2 shipping something beyond pass^k, and it supplies the shape. Task validity: the same audit method applied to the benchmark V2 is modelled on found half the tasks defective, which is why an audit line belongs in V2’s paper. Verifier design: it names precisely what end-state grading cannot see (the project already flags this as an open question) — reward hacking, policy-violating paths to a correct state, refusals-as-failures, and costly actions — and its answer is a validated rubric over recorded trajectories rather than more state predicates, which is cheaper than it sounds. Practitioner realism: “center deployers in safety evaluation” is an argument that ITSM practitioners should be asked not only “does this look like the work” but “which trajectories would get someone fired”, which is a sharper interview question than the current one. What it does not cover: no task generation, no coverage measurement, no simulator design, no multi-agent, and no method for building an environment — it is entirely about reading what came out of one.
- WorldSmith — makes the case that LLM judges are more reliable as readers of trajectories than as actors in them (“reading for a specific behavior in a fixed window of context is a far easier task than acting coherently across thousands of steps”), which is the cleanest available argument for where agent automation is safe in the benchmark pipeline: auditing and labelling yes, unsupervised authoring less so.
- Benchmark Research — the internal/external/safety validity triad, the four principles, the artifact sandwich, and the persuasion-resistance example of a deployment metric derived from logs at no extra evaluation cost.
- Benchmark Validity — home concept since 2026-09-10
Questions it raises for ITSMBench:
- What is V2’s equivalent of persuasion resistance? For ITSM the obvious candidates are approval-bypass rate, unauthorised entitlement grants, and closing a ticket without the required work record — all computable from existing trajectories with a rubric, and all invisible to DB-state grading.
- If a τ-Bench Airline audit doubles measured capability, a V2 audit could move our numbers by a similar factor. Should the audit run before the first leaderboard rather than after, and does it belong in the release as a per-task credibility record?
- The paper asks deployers to supply the safety targets. Our practitioner calls are exactly that channel — is there a version of the practitioner interview that produces a rubric list (“these trajectories are unacceptable regardless of outcome”) rather than a realism verdict?
Notes
(Our take: to add after reading; drafted 2026-09-09, not yet discussed)