Summary
FaulT-Bench attacks the assumption behind every prior network-troubleshooting benchmark: that the ticket is accurate and that a fault exists. It is 200 scenarios over eight Kathará-emulated topologies (five reimplemented from public practitioner labs), split into 80 Correct-Fault, 72 False-Premise (network healthy, ticket wrong), 24 Wrong-Device, and 24 Wrong-Cause; the 72 false-premise tickets are then rewritten into five reporter personas over an identical network state, isolating reporter confidence and verifiable detail one factor at a time. Three agents (SADE, Claude Code baseline, ReAct) at 20 turns over a shared 22-tool interface are graded by a gpt-5-mini judge on outcome (0.7 diagnosis accuracy + 0.3 explanation), fix, and reasoning (grounding/causality/coverage from the investigation log). All three are effectively saturated on accurate tickets (0.932 mean) and shrug off misdirection (Wrong-Device +0.025, Wrong-Cause −0.021), but drop to 0.820 on healthy networks, over-diagnosing in 11–24% of runs — they disprove the symptom and then keep probing until a benign standing condition can be promoted to a root cause. The persona controls invert the sycophancy expectation: a confidently wrong ticket is handled about as well as an accurate one, while removing the reporter’s certainty costs 0.182 and removing the verifiable identifiers a further 0.141, pushing over-diagnosis to 47%. The conclusion: treating the ticket as a hypothesis rather than an instruction is a capability distinct from diagnostic skill, and current agents lack it.
Key points
- Environment: Kathará container emulation — each device is a Docker container, routers run FRRouting, hosts run Linux, so routing decisions and packet forwarding are made by real software, not a model of it. Only hardware, links, and topology are virtualised. The agent gets a topology overview and 22 tools (connectivity tests, interface and routing-table inspection, service queries, shell exec, telemetry) but not the config files; everything diagnostic must be probed.
- Task authoring is entirely manual: five topologies hand-ported from public practitioner labs (YouTube/GNS3 teaching networks), all 200 tickets and all 360 persona rewrites written by the authors and independently checked by three of them. The authors name this as the principal limitation — real helpdesk tickets are not obtainable at research scale.
- Each scenario is a plain-text file of visibility-scoped blocks:
[INJECTION](the fault),[POST-INJECT-CHECK](commands run on named devices whose output decides whether injection succeeded),[SCORING-AXES](assert-identifies / must-identify-device / must-identify-component / bonus axes), and[FIX]. The agent sees only the ticket. This is a clean, copyable format for separating environment setup, verification, and grading from the task prompt. - Verification is a three-score LLM judge, not a state diff: Outcome = 0.7·(axes passed) + 0.3·explanation quality, so a correct diagnosis always outranks a well-explained wrong one; Fix ∈ {0, 0.5, 1} on whether the remediation would actually work; Reasoning = (grounding + causality + coverage)/30 on a G-Eval anchored 0–10 scale, with the judge given a digest of the tool calls so a claim counts as grounded if the log contains the observation. The judge is instructed to score validity of reasoning, not adherence to a prescribed procedure.
- The negative-case finding is the whole paper: as long as a fault exists, an agent that verifies the ticket and an agent that blindly follows it produce the same diagnosis — the two behaviours are indistinguishable. They separate only when the reported fault is not there. SADE, the most sophisticated agent (skills + playbooks), has the worst false-alarm rate (24%) and simultaneously the best grounding score (0.933) on that class: it is not investigating badly, it lacks a stopping criterion for “nothing is wrong”.
- Persona controls, all on the same healthy networks: novice→novice-confident (tone flipped, every identifier preserved) moves outcome only +0.039; naive→naive-unsure (certainty withdrawn, identifiers unchanged) costs −0.182 and pushes over-diagnosis 16%→35%; naive-unsure→no-detail (identifiers deleted) costs a further −0.141 and 47% over-diagnosis. What matters is whether the ticket contains something checkable, not how confident it sounds.
- Failure modes differ by agent and matter operationally: Claude Code collapses on vague tickets (0.870 → 0.283 outcome, 49% turn-budget exhaustion) while SADE holds a steady 20–34% false-alarm rate that could in principle be calibrated away. Effort follows the ticket, not the fault: 13 tool calls and 136k tokens on Correct-Fault vs 30 calls and 549k tokens on no-detail; the 9% of runs that timed out consumed 23% of the $1,525 total spend.
- Cost and scale of running it: ~6 min wall-clock per run, ~59 hours and ~US$165 per full pass over the 200 core scenarios, 3,960 runs total; 8% of (scenario, agent) cells move by ≥0.5 between passes, so everything is a three-pass mean.
Relevance
- ITSMBench — the closest published thing to V2’s ticket-driven ITSM setting, and its central finding transfers almost verbatim: an ITSM benchmark built only from tickets that describe real, resolvable problems cannot distinguish an agent that verifies from an agent that obeys. BPI 2013/2014 are full of exactly the cases FaulT-Bench synthesises — tickets that turn out to be user error, tickets misrouted to the wrong assignment group, tickets whose stated cause is wrong — and the “wait-user” and reassignment patterns in the logs are the trace signature of a false or underspecified premise. This gives V2 a concrete, log-groundable negative task class and the over-diagnosis metric to score it, plus a persona methodology (rewrite the same ticket varying one factor, hold environment state fixed) that is directly usable for calibrating a tau-bench-style operator simulator. What it does not cover: nothing here comes from logs — topologies are hand-ported teaching labs and all tickets are hand-written; there is no user simulator (the ticket is a static string, no conversation), no multi-agent or handoff structure, no database state, and no task distribution argument at all.
- Benchmark Research — a strong design template on three counts: the visibility-scoped scenario file with a machine-checked
[POST-INJECT-CHECK]that verifies the environment was set up correctly before the agent runs; the decomposition of grading into outcome / fix / reasoning so a correct answer reached by unsupported reasoning is visible; and the discipline of reporting timeouts separately and never folding them into score means, with per-agent cost as a first-class result. Also a clean demonstration of saturation analysis — “further scenarios of this kind are less likely to separate one agent design from another”. - Enterprise Environments — home concept since 2026-09-10
Questions it raises for ITSMBench:
- What fraction of BPI 2013/2014 cases actually close as no-fault, user-error, or “could not reproduce”? If it is substantial, the log itself supplies the false-premise class V2 would otherwise have to invent — and an over-diagnosis rate becomes a log-calibrated yardstick rather than a synthetic one.
- V2’s verifiers are binary end-state DB checks. How is “correctly concluded nothing was wrong” even expressible as an end-state assertion, given the correct action is to close the ticket without changing anything — and does that force the trajectory or rubric grading the project has so far avoided?
- FaulT-Bench’s tickets are static text; V2 wants a conversational simulated operator. Does an operator the agent can interrogate eliminate the vague-ticket failure mode entirely (the agent just asks for identifiers), or does it reproduce it in a worse form, with the simulator inventing detail the log never contained?
Notes
(Our take: to add after reading; drafted 2026-09-09, not yet discussed)