Vibrant Labs Research

ITSMBench

12 min read

Question

I am studying the transformation of enterprise operational data into benchmarks, evals, and RL environments, because I want to find out how the process structure in ITSM event logs (BPI 2013/2014) becomes a grounded world, a task distribution, calibrated simulators, and verifiers, and which parts of that generalize to other domains, in order to help frontier labs and ITSM vendors understand whether years of operational data can become representative evals and training environments, so that labs can track ITSM-agent capability on real work and enterprises can benchmark any ITSM company against its own workflows.

Vehicle: ITSMBench V2, the successor to V1 = EnterpriseWorlds ITSMBench (Vibrant Labs; 53 tasks, conversational simulated operator, hybrid DB-state × NL-assertion grading, pass@k vs pass^k leaderboard). V2 adds a tau-bench-style user simulator and multi-agent workflows, grounded in real event logs. Two multi-agent tracks: (1) one agent under test with simulated user and simulated colleagues or teams, (2) several agents under test collaborating on one workflow.

What counts as an answer: V2 exists, built from the logs, and scores on four yardsticks, with a record of what the logs supplied and what had to come from elsewhere.

Yardsticks (decided 2026-09-09):

  • Coverage of the process variants and roles present in the event logs, weighted by frequency.
  • Practitioner realism: ITSM domain experts, shown the benchmark, say it captures the work activities of the job persona. This is the primary check on tasks.
  • Discrimination and stability: separates models, stable under repeats (pass
  • Real-world tracking: scores follow real ITSM outcomes. Gold standard, hardest to get.

Benchmark vs RL environment: a benchmark must be representative of real work. An RL environment keeps tasks valid and realistic by the human standard but expands difficulty for the model under test so training lands in the GRPO sweet spot. Frontier-model evals on V2 locate that difficulty.

Why it matters

Practical origin: V2 is being built by hand now and the manual cost forces the question of what the logs can supply.

The core bet of the company: the years of operational data inside enterprises can be turned into benchmarks, evals, and RL environments that represent the tasks that happen there. An enterprise with a mandate to automate its work needs exactly this. Two audiences:

  • Enterprises and ITSM vendors: the workflows explored here should let us build and benchmark any ITSM company against its own operations.
  • Frontier labs: a benchmark to track agent capability as an ITSM agent, on real work rather than synthetic tasks.

So-what chain: if operational logs cannot become environments, enterprise agents keep being trained and bought against synthetic tasks and scores do not transfer; environment supply stays human-bound (the WorldSmith consequence); decades of process-mining data stay unused. If they can, the same “refiner” applies to other domains after ITSM. Applied research: the significance is a product and a method.

What I already know

In our words (interview, 2026-09-09):

  • Logs help you create the world in which the agents operate, a rich world. They give you a distribution of tasks based on the activities happening. They do not give you the task assets, but they give you something to check back on: if you come up with a distribution of tasks, you can see how much of it matters for day-to-day work and calibrate your distribution against the logs.
  • Task design takes inspiration from existing benchmarks and RL environments, but the task distribution always has to relate to the ground-truth distribution in the logs. The check I keep for tasks: give the benchmark to a domain expert and he should be able to say this captures all the work activities for this job persona. Expanding to frontier-hard is an option to explore, with evals against frontier models to see where they perform.
  • Verifier design is external. Good verifier design has a lot that generalizes; mostly it will be DB-state, and the science of building good verifiers can be taken from other benchmarks and RL environments and applied here to see if it works.
  • Simulator calibration: logs record what actually happened, so you can replay them, compare trajectories, and see where things change.
  • Source data is not high quality by default; you need source data that captures the world in as much detail as possible. Current sources: BPI Challenge 2013 (Volvo IT, VINST incidents and problems) and 2014 (Rabobank ICT, HP Service Manager). We want more of the same kind.
  • V2 will not use agents end to end; we are actively asking how much of it can be automated. That question is WorldSmith; V2 is its first live case.

State of the code (repo survey of itsmbench-core, 2026-09-09):

  • Harbor task packages; agent works through mocked vendor APIs (ServiceNow, Entra ID, Governance, Teams, SharePoint; about 309 operations); BM25 endpoint discovery; deterministic ids and frozen clock.
  • Verifiers are binary pytest end-state checks that read back over the same API the agent used, with mandatory preservation controls. No trajectory checks, no rubric or LLM judge, no user simulator, no multi-agent yet.
  • BPI 2014 currently seeds only the org structure: 242 anonymized assignment groups distilled into 20 teams and positions, with a reproducible analysis pipeline (Interaction 147,004 · Incident 46,607 · Incident_Activity 466,737 · Change 30,275 rows). Task seeds are the fictional “Vantage Pay” data ported from a partner ITSMBench corpus (89 tasks). A BPI-derived seed canon in the feat-seed-data worktree was deleted on 2026-09-07 and survives only in git history.
  • Adversarial scripts already exist and have caught two cheats scoring 1.0 and one correct trajectory scoring 0.0; differential and mutation testing of ports is in the multiservice worktree.
  • From the August all-hands: “our benchmark can be the same as tau-bench”; “we want it to be Terminal-bench”; ports planned to Prime Intellect, Harbor, OpenEnv.

Working claim

Provisional, no evidence yet. Logs are sufficient to build the world (state, roles, process variants), to define the ground-truth task distribution, and to calibrate simulators by replay. Task design and verifier design are external: inspired by other benchmarks and RL-environment practice, and validated by domain experts and adversarial testing rather than derived from the logs.

Rivals the work must rule out:

  • Logs are enough for everything, because case outcomes (closure codes, SLA breaches, reassignments) already define tasks and verifiers.
  • Logs are only a yardstick for realism; the environment itself must come from vendor schemas and policies.

Kind of claim: fact with a method. Evidence: build V2 from BPI 2013/2014 and score it on the four yardsticks; keep a ledger of which artifact came from logs, from external practice, or from experts.

What would overturn it: practitioners rate log-derived tasks and simulated actors as unrealistic or missing what matters, even at full coverage of the logs.

Open questions

  • First step: rebuild the BPI-grounded world properly (state, roles, process variants from 2013/2014) before the task distribution. What is the minimum world that a practitioner recognises?
  • How is the task distribution extracted from logs: by activity, by process variant, by role, by outcome? What does a “task” correspond to in a case trace?
  • No paper builds an agent environment from event logs (survey, 2026-09-09). Process-mining tooling (PM4Py, AgentSimulator) discovers models from logs; is that the bridge?
  • Simulator fidelity: published simulators are over-cooperative (Sim2Real Gap, RealUserSim). Can BPI ping-pong, wait-user, and reassignment patterns calibrate an ITSM operator simulator, and how is fidelity measured?
  • Verifiers: V1 grading is DB-state plus NL assertion; V2 tracks add handoffs and approvals. End-state only misses policy-violating paths (log-analysis literature). What trajectory checks are needed, and what do they cost?
  • Two tracks: what is scored in the team track, and how are failures attributed?
  • Contamination: BPI 2013/2014 are ten-year-old public CSVs. Re-sample and anonymise entities; synthesise tasks rather than copy cases.
  • More source data of the same kind (to vet): UCI incident-management event log (ServiceNow instance, 141,712 events, CC-BY-4.0); Helpdesk event log (Polato 2017); BPI 2012/2017 loan applications (customer-facing service workflow); BPI 2019 purchase-to-pay; BPI 2020 travel requests; Loghub (raw system logs, AIOps rather than tickets).
  • Publish target: end of September 2026, mid-October at worst. What is the minimum publishable V2?
  • Feedback pointers with no captured content yet: two partner calls (Aug 2026), annotator platform review (Sep 2026).
  • Time (survey 2026-09-10, see Time in Environments): V1 freezes one instant per task and never tells the agent what it is; SLAs cannot newly breach and multi-turn rows share a timestamp. Which tracks need an event-driven clock (operator replies, SLA breaches, reassignments at simulated times, fitted from BPI durations), and how does the Claude Code harness see world time (instruction plus get_current_time, libfaketime in the agent container)?

Reading list

Sources

  • Tau-Bench — the user-simulator template (instruction as system prompt, uncalibrated), unique-outcome task discipline, DB-state × output grading, pass^k; end-state grading admits policy-violating paths
  • Tau2-Bench — user with tools (dual control) cuts simulator error 40% to 16%; compositional verified-by-construction task generation; removing the user raises pass^1 by 18 to 25 points
  • Mind the Sim2Real Gap — 451-human reference: simulators are an easy mode (USI 76 vs human ceiling 93), prompting made fidelity worse, binary reward orthogonal to human-judged success
  • RealUserSim — Grounds simulators in 7,275 profiles mined from real WildChat logs; fidelity 24.2%→45.3%, and tau-bench’s own persona directives inflate difficulty
  • Simulated Customers Never Walk Away — Simulators match real buyers but inflate non-buyers (d=0.38) against verified purchase outcomes; disengagement cannot be prompted in
  • SAGE — Cold-start simulator grounded in ICPs plus the agent’s own catalogs and FAQs; finds 25-33% more bugs than baselines and than real traffic
  • VISTA — Six simulator-quality metrics (tool-transition and trajectory-distance coverage) plus a hybrid UI+API user; 42% more failures than UI-only
  • EnterpriseOps-Gym — ServiceNow sandbox, 164 tables, 512 tools, 1,150 SME-authored tasks, SQL end-state verifiers; best model 37.4%, ITSM hardest at 28.5%
  • EntCollabBench — 11 role-specialized agents across six departments with permission isolation; trace + DB-diff judging by three-model vote; best 62%
  • TheAgentCompany — Self-hosted software startup with Sotopia-backed simulated colleagues; weighted checkpoint scoring; best agent completes 30.3% of 175 tasks
  • World of Workflows — ServiceNow instance with 4.8K hidden business rules; success under constraints collapses to ~2% without audit-log visibility
  • CRMArena-Pro — Salesforce B2B/B2C Orgs, 19 tasks, four business skills; 58% single-turn falls to 35% multi-turn against an incremental-release user simulator
  • ITBench — 94 live Kubernetes SRE/CISO/FinOps scenarios with topology-aware partial credit; GPT-4o resolves 13.8% of SRE diagnoses, 0% hard mitigations
  • AppWorld — 9-app simulated world, 457 APIs; state-diff verification with expected/allowed change sets catches collateral damage; best 48.8% TGC
  • LOGIGEN — Triple-agent pipeline compiles policy into DB triggers, forward-explores to synthesize 20k verifiable tasks; state-diff verification, demonstrated simulator hacking
  • TASTE — Samples tool sequences from contrastive n-gram model to build harder, broader-coverage tau-bench extension; hint-assisted verifier validated at precision 1.0
  • Benchmarking the Benchmarks — Expert audit of 496 traces across four tool-calling benchmarks finds 18.5% evaluator-human misalignment; deterministic-first evaluator reaches 95.5%
  • ABA — Agentic auditor over 168 benchmarks, 34,285 tasks finds 25.7% carry major issues; filtering them shifts leaderboards by ~10 points
  • Designing Benchmarks for Knowledge Work — Four-field reporting schema plus 18 O*NET-derived work activities; shows benchmarks evaluate less than the work product they claim
  • SABER — Mutating actions are 14-18% of steps but drive nearly all failure; ships tau-Bench Verified after finding half of instructions under-specified
  • Log Analysis Is Necessary — Taxonomy of internal/external/safety validity threats; tau-Bench Airline audit finds 25/50 tasks flawed, doubling pass^5 when excluded
  • Science of AI Agent Reliability — Twelve accuracy-independent reliability metrics over four dimensions; 15 models show reliability plateaued while capability rose for 24 months
  • AgentSimulator — Discovers a multi-agent system from an event log — per-resource schedules, capabilities, timings, handover matrix — beating control-flow-first simulators 20-80x faster
  • From Event Logs to Governed Action — BlueSky agenda for mining logs into governed actions where act, defer, ask and refuse are all valid scored outputs; names four artifacts, no method
  • Agent Mentor — Mines an agent’s own trajectory logs by clustering outcomes and eliciting SVO features, then injects derived corrections into prompts; +37pp on ambiguous specs
  • FaulT-Bench — 200 network troubleshooting scenarios where 120 tickets are wrong; agents saturate real faults but over-diagnose healthy networks up to 47%
  • CHI-Bench — Hand-built 20-app healthcare simulator, 75 policy-dense multi-role tasks; best agent 28% pass@1, two-agent provider-payer arena collapses to 0%
  • AgentDojo — Framework crossing 97 user tasks with 27 injection goals into 629 prompt-injection cases; capable models are easier to hijack, tool filtering cuts ASR to 7.5%
  • MANTRA — Generates trace-level compliance checks and a symbolic world model independently from a manual, cross-validating them with Z3; 285 validated tasks, no LLM judge
  • STAGE-Claw — An authoring agent writes task, environment and verifier; an independent checker agent validates reproducibility and difficulty; output-only grading inflates scores 5-7 points
  • ClawEnvKit — Generates whole environments (task, tools, fixtures, grader) from one natural-language line; 1,040 environments for $80 versus 208 human-hours, but validity is LLM-judged

To ingest

All arxiv items ingested 2026-09-09. Remaining non-arxiv sources (need Web Clipper or manual capture):

  • BPI Challenge 2013 and 2014 challenge pages and 4TU dataset records
  • the partner ITSMBench repository and corpus
  • EnterpriseWorlds site and leaderboard

Graph View

Backlinks