Summary
ITBench (IBM + UIUC) benchmarks agents on live IT infrastructure rather than on records about it: 94 scenarios across three personas — Site Reliability Engineering, Compliance/Security (CISO), and FinOps — each instantiated as a real Kubernetes testbed running the OpenTelemetry Astronomy Shop with a full observability stack (Grafana, Loki, Jaeger, Prometheus) plus a fault-injection mechanism. A scenario is a tuple ⟨metadata, environment, triggering events, desired outcome⟩ where the desired outcome for SRE is literally “clear the alert in the environment”, so grading reads the final system state rather than a database record. Results are near the floor: 13.8% pass@1 diagnosis and 11.4% mitigation for GPT-4o on SRE, ~25% on CISO, 0% on FinOps mitigation, and no model mitigated a single Hard scenario in any run. Two findings carry beyond the numbers: removing distributed traces cuts GPT-4o’s diagnosis from 13.8% to 9.5% and mitigation to 2.9%, and the same model diagnoses the same scenario in 6/10, 1/10 and 8/10 runs on different scenarios — non-determinism the authors attribute to live telemetry perturbing token generation.
Key points
- This is the ops-side sibling of ITSM, not the ticket side: incidents are induced in a running cluster (cache failure, memory leak, corrupt image, network partition, HTTP body tamper) and cleared by acting on the cluster. There is no ticket, no requester, no ITIL record; the “state” is infrastructure, not a CMDB row.
- Ground truth is richer than pass/fail: for diagnosis it records the entities in the fault propagation chain, the chain(s) themselves, and the fault conditions; for mitigation, the plausible mitigation actions. This lets them score partial credit with NTAM (Normalized Topology-Aware Metric), which grades a predicted root cause and propagation chain against the real system topology instead of demanding an exact string.
- Scenario complexity is defined as a formula over scenario properties — fault propagation chain length, number of resolution steps, diversity of technologies involved — and explicitly declared independent of agent capability. Results degrade monotonically along it (GPT-4o diagnosis 36% / 7.7% / 5.0% for Easy / Medium / Hard), so it is a validated difficulty axis rather than a post-hoc label.
- Observability is an ablatable input: 21 scenarios run with traces, 21 without. The gap quantifies how much of agent performance is diagnostic reasoning versus having the evidence, and the authors name reasoning under incomplete observability as the open problem.
- Scenario sources differ by persona and the paper is candid about it: SRE scenarios come from real incidents observed in IBM’s own SaaS products, CISO scenarios from the CIS benchmarks, and FinOps from FinOps Foundation domains — with only two FinOps scenarios “due to the lack of standard benchmarks.”
- Scenarios are parameterizable rather than one-off: the same PodFailure fault applied to any of 18 pods in HotelReservation yields 18 scenarios, so the authors claim hundreds to thousands of variations from 21 unique patterns. This is a cheap combinatorial task-generation route that does not need per-task authoring.
- Non-determinism is measured and reported (10 runs per scenario per model) rather than averaged away — the closest thing here to a pass^k discipline, and their justification is environmental: live telemetry differs run to run.
- Metrics are operational, not just correctness: mean time to diagnosis and mean time to repair alongside pass@1, hourly infra cost and CPU efficiency for FinOps, time to process for CISO.
- Only 11 of 94 scenarios are open-sourced; the rest are held back to keep the leaderboard uncontaminated. The paper closes by naming multi-agent orchestration across specialisms, agent safety, and human-in-the-loop as open problems it does not address.
Relevance
- ITSMBench — the other half of IT work, and useful mainly as contrast: it shows what verification looks like when the environment is real (clear the alert) rather than a mocked API, and it supplies two things V2 lacks — a graded partial-credit metric grounded in system topology (NTAM) and a scenario-complexity formula computed from structural properties rather than assigned by a reviewer. The complexity formula (chain length × resolution steps × technology diversity) is a template for deriving difficulty from BPI case traces. It does not cover log grounding in the ITSM sense (fault injection into a cluster, not process data), has no user simulator or requester at all, and no multi-agent track — the paper lists agent orchestration as future work.
- Benchmark Research — measuring run-to-run variance explicitly (10 runs per scenario) and attributing it to the environment, an observability ablation that separates “lacked evidence” from “could not reason”, topology-aware partial credit instead of exact match, and holding back most scenarios from public release to protect the leaderboard.
- Enterprise Environments — home concept since 2026-09-10
Questions it raises for ITSMBench:
- Their scenario complexity is a computed function of fault-chain length and resolution steps. BPI case traces have exactly those quantities (variant length, number of activities, number of reassignments). Is that the bridge from log to difficulty stratification that V2’s task distribution needs?
- Removing traces halves mitigation success. ITSM tickets carry an analogous evidence channel — work notes, prior incident history, CI relationships. Is “what evidence is in the ticket” the ITSM version of the observability ablation, and does the log tell us what evidence was actually available to the human?
- ITBench accepts non-determinism as a property of a live environment and reports the spread. V1 uses a frozen clock and deterministic ids to remove it. Which is right for V2’s stability yardstick — and does a log-grounded world have irreducible non-determinism of its own once simulated actors are in it?
Notes
(Our take: to add after reading; drafted 2026-09-09, not yet discussed)