Summary
Three expert annotators re-adjudicated 496 execution traces from four tool-calling benchmarks (BFCL v4, τ²-Bench Retail, LiveMCPBench, MCP-Atlas) against the benchmarks’ own verdicts, at 89 annotator-hours or ~10.8 minutes per trace. The official label disagreed with the human judgment 92 times, an 18.5% aggregate misalignment rate, ranging from 9.8% on τ²-Bench Retail to 20.0% on BFCL v4 and 30.5% on LiveMCPBench. Neither scoring paradigm is safe: deterministic evaluators fail through brittle state matching, trajectory lock-in, wrong ground-truth annotations, substring communication checks, and reward-basis mismatch, while LLM judges fail through rubric drift, hallucinated completion, answer-only scoring, and variance — 23 identical reruns of LiveMCPBench span 57.9% to 76.8%, an 18.9-point spread from evaluator and agent sampling combined, larger than most leaderboard margins. The authors propose decomposing the single binary score into (tool invocation, task completion, outcome verification), and ship two artifacts: Tool-Veritas, a deterministic-first evaluator where state predicates gate the pass and a restricted LLM judge may only add qualitative criteria on top, never override a failed gate; and Harness Lab, a benchmark execution and debugging layer with per-turn traces, repeated-run comparison, selective retry, and human adjudications stored separately from official labels. Tool-Veritas reaches 95.5% agreement with experts over 420 model–task evaluations, and all 19 of its disagreements are false negatives — zero unsupported passes.
Key points
- The headline number for anyone building a DB-state verifier: τ²-Bench Retail, the closest design to ITSMBench V1’s grading, still misfires on 11 of 112 tasks (9.8%) — 8 false negatives, 3 false positives. Being deterministic buys reproducibility, not validity.
- The two canonical failures are worth naming exactly. False negative (Task 7): DB state and all five expected tool actions match, but the task fails because the final message lacks the substring “1628” — the agent correctly excluded two cancelled reservations and said $708, and the evaluator’s semantic target was wrong. False positive (Task 10): the user asks to return two orders, the agent punts to a human without calling the return tool, and because the expected DB state was unchanged anyway, it passes. Final-state scoring rewards inaction whenever the target state equals the initial state.
- BFCL v4’s failure mode is over-specification: 20 of 25 official failures in the inspected 50-task multi-turn export are
instance_state_mismatch, which conflates real failures with task-irrelevant state deltas, punctuation-level differences, whole-object comparison where one field matters, and premature termination before a later corrective action. - LLM-judge instability is the most quantified result: 23 valid reruns of the same 95-task LiveMCPBench configuration give mean 69.4%, σ 5.4 pp, spread 18.9 pp — best and worst runs differ by 18 of 95 tasks. The evaluator itself regenerates task-specific key points each run, so two evaluations score the same trace against different criteria. Swapping a deterministic evaluator for an LLM judge makes things worse, not better (30.5% vs 9.8% misalignment).
- The taxonomy is the reusable artifact. Deterministic: exact-match constraints, state over-specification, trajectory lock-in, annotation errors, reward-basis mismatch. LLM-judge: rubric drift, judge variance, hallucinated completion, implementation-specification mismatch.
- Tool-Veritas’s design rule is the paper’s recommendation in one line: deterministic state predicates are a gate (
C_state = ∏ 1[g(s_t)=1]), and the LLM judge is invoked only for communication quality, policy adherence, and confirmation adequacy — it cannot rescue a failed gate. It also records a bounded repair window, scoring first-attempt completion separately from completion-after-repair. - Result asymmetry worth copying: Tool-Veritas 95.5% agreement with zero lenient disagreements. A verifier that is too strict costs you signal; one that is too lenient costs you the benchmark’s meaning, and the authors deliberately tune to the former.
- Limitations: one agent model per benchmark (Kimi-K2.6 on τ², MiniMax-M2.7 on the rest), so the misalignment rate is entangled with that model’s failure modes; audit sizes are modest per benchmark (89–200); Tool-Veritas is evaluated on its own 70-task suite with a different task distribution, which the authors flag as evidence for the design rather than a controlled comparison; artifacts are promised, not yet released.
Relevance
- ITSMBench — the most operationally useful of this cluster for V2’s verifier design, because it audits the exact grading scheme V1 already uses (DB state + NL assertion) and gives the two failure modes to defend against: brittle NL/substring assertions producing false negatives, and unchanged-target-state tasks producing false positives that reward doing nothing. Concrete rules to adopt: never let an assertion be a substring check; audit every task whose target state equals its initial state (in ITSM, “no action needed / close as duplicate” tasks are exactly this shape); use LLM judging only above a passed deterministic gate; and separate first-attempt from after-repair success, which maps onto ITSM’s escalate-and-retry reality. It does not cover task generation, coverage, realism, multi-agent attribution, or the user simulator — it is purely about whether the scorer is right.
- WorldSmith — sets the bar an automated pipeline must clear: human-authored, human-verified benchmarks still misalign with expert judgment 9.8–30.5% of the time, so “agent-generated tasks are 90% valid” is not obviously worse than the status quo — but it also shows the audit itself costs ~11 expert-minutes per trace, which is the cost an automated auditor would have to displace. The zero-false-positive result for a gated evaluator is the design an agent-built verifier should be held to.
- Benchmark Research — supplies the failure taxonomy, the misalignment-rate metric (Err(B) = mean 1[y≠h]), the reproducibility metric (score spread Δ over K identical reruns), and the argument for decomposing one binary score into tool invocation / task completion / outcome verification.
- Benchmark Validity — home concept since 2026-09-10
Questions it raises for ITSMBench:
- What is V2’s own misalignment rate? The method is cheap to copy — sample ~100 trajectories across models, have an ITSM practitioner adjudicate, report Err(B) next to the leaderboard. Doing this before publication would be a differentiator; V1’s adversarial scripts already found two cheats at 1.0 and one correct trajectory at 0.0, which is the same phenomenon found by hand.
- Their false-positive case (agent transfers to human, state unchanged, passes) is a live risk for ITSM, where escalation is sometimes the correct action. How do we distinguish “correctly escalated” from “gave up”, and does that require a trajectory check the end-state verifier cannot provide?
- If our grading stays partly NL-assertion based, should every assertion be re-expressed as a state predicate plus an optional judged criterion, and what fraction of V1’s 53 tasks survive that translation?
Notes
(Our take: to add after reading; drafted 2026-09-09, not yet discussed)