Summary
Borrowing the reliability vocabulary of aviation, nuclear and automotive certification, this paper decomposes agent reliability into four dimensions — consistency (repeatable outcomes, trajectories and costs), robustness (graceful degradation under faults, environment changes and prompt paraphrases), predictability (confidence that tracks success), safety (bounded severity when things go wrong) — and defines twelve computable metrics, all normalised to [0,1] and all deliberately independent of raw accuracy. Evaluating 15 models from three providers across GAIA and τ-bench with K=5 runs per task at temperature 0, five paraphrases per instruction, fault injection at p=0.2, and medium-intensity tool-interface perturbations, it finds that 24 months of capability gains bought only small reliability gains, and that all frontier providers cluster together — an industry-wide plateau, not a vendor problem. The dimension-level findings are sharper than the headline: outcome consistency stays low everywhere (the pass@k vs pass^k gap); agents show “what but not when” behaviour, with much higher action-distribution consistency than action-sequence consistency; fault and environment robustness are near ceiling while prompt robustness — sensitivity to a semantically equivalent rephrasing — remains the main differentiator, which the authors call counterintuitive; calibration has improved (Claude notably) but discrimination has not, and worsens on GAIA in the newest models; smaller models are often more consistent than larger ones, because larger models have more solution paths. Safety is computed but deliberately excluded from the overall aggregate because violations are a tail phenomenon; on τ-bench the most common violation category across every model is financial accuracy — incorrect charges and refunds.
Key points
- The metric definitions are directly implementable and this is why the paper is worth having. Outcome consistency
C_out = mean over tasks of (2p̂−1)²(Bernoulli variance normalised); trajectory consistency in two flavours — distributional (mean pairwise Jensen–Shannon divergence between action-type distributions across K runs) and sequential (mean pairwise normalised Levenshtein distance between action sequences); resource consistencyexp(−mean coefficient of variation)over cost, time and API calls; robustness as clamped accuracy ratios under fault / environment / paraphrase perturbation; calibration (1 − ECE), discrimination (AUROC of confidence vs outcome), Brier; safety as compliance rate and mean severity conditioned on violating runs. - Safety uses the classical Kaplan–Garrick risk form: R_saf = 1 − P(violation)·E[severity | violation], and is reported separately as a hard constraint rather than averaged in, precisely so a model that behaves well 99% of the time and catastrophically 1% of the time cannot average its way to a good score.
- τ-bench safety is evaluated against four domain constraints — block unauthorised modifications, correct transaction amounts, require identity verification, resist policy circumvention — scored by an LLM judge over full interaction traces rather than final outcomes. That constraint list is almost verbatim transferable to ITSM (unauthorised entitlement change, correct approval path, requester verification, policy-exception resistance).
- Benchmark defects distort reliability metrics, not just accuracy. Because SABER found 24 of 50 τ-bench Airline tasks defective, the authors restrict to the clean 26-task subset; comparing clean vs full, predictability and safety improve almost universally while consistency and robustness do not — an agent that confidently solves a task and is marked wrong by a broken answer key is scored as overconfident. Calibration is therefore the metric most contaminated by bad tasks.
- The “what but not when” result is the most useful for verifier design: agents reliably pick the same kinds of actions across runs but vary the order, which is exactly the property that makes strict trajectory matching a bad verifier and action-set or constraint checks a better one.
- Temperature is set to zero throughout, so all observed variance comes from non-sampling sources — floating-point non-associativity, batch-size variation under concurrent load, non-deterministic kernel scheduling. The authors flag that this likely overestimates achievable reliability relative to the nonzero temperatures used to maximise accuracy.
- Recommendation 1 is the one that bites for benchmark builders: benchmarks should be generative and parameterised rather than fixed test sets — able to rename fields, reorder responses, and inject fault probabilities — both to measure robustness and to blunt answer-lookup shortcuts, with temporal re-evaluation to catch silent drift. Recommendation 4 scales the requirement to autonomy: in augmentation settings a human is the reliability backstop; in automation settings unreliability converts directly into failures.
- Limitations, self-listed: two benchmarks; a single scaffold per benchmark; LLM-based safety judging with its own reliability problems; metric and aggregation choices are subjective and R does not include safety; temperature-0 caveat above.
Relevance
- ITSMBench — this is the ready-made answer to V2’s third yardstick, “discrimination and stability”, which the project currently states as pass^k plus separation. Twelve metrics, all computable from runs V2 would do anyway, and three of them fill acknowledged gaps: trajectory consistency gives a way to score the process without a gold trajectory (which matters because ITSM tasks have several valid orderings — and the paper’s own “what but not when” finding says ordering is exactly where agents vary); the four τ-bench safety constraints map onto ITSM policy violations that DB-state grading cannot see; and prompt robustness is a live realism concern, since real tickets are paraphrases of each other and a benchmark whose tasks are one fixed wording measures something narrower than the job. Recommendation 1’s “generative, parameterised benchmark” is also a direct argument for the log-grounded generator V2 is contemplating, and Recommendation 4 explains why the multi-agent automation tracks need a higher reliability bar than the single-agent assistant track. What it does not cover: no task generation, no task-validity method of its own (it borrows SABER’s), no coverage or realism notion, no multi-agent, and no ITSM.
- Benchmark Research — the strongest available argument that a single accuracy number is the wrong headline, plus a full metric suite to replace it with, plus the finding that reliability has plateaued industry-wide for two years while accuracy climbed. The clean-vs-full comparison is also a neat demonstration that benchmark defects specifically corrupt calibration measurements.
- WorldSmith — Recommendation 1 (“benchmarks should become generative and parameterized rather than relying on fixed test sets”) is a reliability-side argument for exactly the automated environment generation WorldSmith asks about: perturbable environments are not a nice-to-have but a prerequisite for measuring robustness at all, and no fixed hand-built suite can supply them.
- Benchmark Validity — home concept since 2026-09-10
Questions it raises for ITSMBench:
- Which of the twelve does V2 report? Outcome consistency and resource consistency are nearly free given pass^k runs; trajectory consistency needs only the action logs the harness already writes. Is there a reason not to publish a reliability profile alongside the leaderboard from day one?
- Their environment-perturbation lever (rename API parameters, change date formats, reorder response fields) is trivial to build into a mocked-vendor layer and would be a genuine differentiator versus a static benchmark — but does perturbing a simulated ServiceNow still measure anything a practitioner would recognise, or does it drift into measuring schema brittleness?
- Safety is measured by an LLM judge over traces against four named constraints. What is the ITSM constraint list, who writes it — us or the practitioners — and does it belong in the score or beside it as a hard gate, as the paper argues?
Notes
(Our take: to add after reading; drafted 2026-09-09, not yet discussed)