Summary
World of Workflows (WoW) runs on a live ServiceNow developer instance with 4,800 business rules and 55 active workflows left deliberately hidden from the agent, framing enterprise work as a POMDP where the true state is the whole database and the agent sees only tool responses. WoW-bench is 234 tasks in four categories: 50 constraint-understanding, 50 long-horizon agentic tasks (avg 13 actions), 67 audit prediction (forward dynamics) and 67 action prediction (inverse dynamics). The central experiment is an observation ablation — tool response vs. oracle table-audit logs, each audit a (table, column, old value, new value) tuple — and it shows that task success rate holds up under plain tool responses while success under constraints collapses to near zero (GPT-5.1: 22% TSR but 2% TSRUC; with audits, 32% and 14%), i.e. agents complete the task by silently violating hidden rules they cannot see. As zero-shot world models the same LLMs score under 30% on both audit and action prediction, with 73.5% of errors being confusion between human-readable names and symbolic identifiers. The authors argue enterprise reliability needs dynamics-aware architectures, not better instruction following.
Key points
- Environment is a real ServiceNow developer instance rather than a mock: 4.8K business rules (atomic, database-driven column logic) plus 55 workflows (multi-step orchestration). Cascading failure is the designed phenomenon — assigning a 4th asset trips a workflow that decrements the user’s clearance, which trips a second workflow that unassigns compliant assets, all invisible in the agent’s API success messages.
- TSRUC (task success rate under constraint) is the load-bearing metric: goal achieved AND no constraint violated anywhere during execution. Reporting TSR alone would have shown 22–38% and hidden the fact that nearly every “success” was contaminated. This is the sharpest published argument that end-state goal checks are not enough.
- Constraints are dynamic and silently violated by the system, not static budget-style rules: domain experts wrote 10 realistic constraints (“a user cannot be assigned more than 3 active incidents”, “flagged articles should not be published”), then deliberately designed trajectories whose cascading effects violate them. 10 templates × perturbation = 50 trajectories; the 50 agentic tasks are the same 10 templates × 5 permutations, with descriptions LLM-generated from trajectories and human-verified.
- Tool-dependency-graph sampling is the automation contribution: build a graph whose edges say which tool’s output can feed another’s input (derived by a scripted REST API that checks whether a field references another table), pick a tool, backtrack to a root, and repeat — generating connected multi-hop trajectories with no human in the loop. The authors contrast this with manual expert creation (high quality, expensive) and LLM random sampling (scales but loses diversity), and note connected sequences are what actually trigger workflow side effects.
- Dynamics modelling is scored directly: audit prediction by IoU over exact (table, column, old, new) tuples, action prediction by tool-name accuracy and full-action accuracy including all parameters. Both under 30%; models systematically under-predict side effects, missing e.g. that creating an incident silently updates
metric_instance. - Three named gaps: representation (models treat “User X” as a token, not a graph node — 73.5% of errors are name-vs-sys_id confusion), dynamics (no forward transition model; they rely on semantic plausibility), causal (greedy planners that optimise the immediate action and miss the ripple, worsened by attention decay over long contexts).
- Every task template ships with a cleanup function that deletes the task-specific data from the instance, making runs repeatable on a live shared system.
- Authors’ proposed direction: explicit state abstractions, model-based RL where the agent predicts the audit log before acting and updates on the discrepancy, and active epistemic probing to learn workflow trigger conditions — “zero-shot performance is largely a fallacy in opaque enterprise systems.”
- Limitations they name: expert curation of workflows and constraints is expensive and domain-specific; dynamics cover a subset of one vendor’s system; system tables producing hundreds of noisy audits were stripped to keep observations tractable; and WoW is designed as an evaluation benchmark, not a training environment, though the sampling machinery makes it one.
Relevance
- ITSMBench — the strongest available argument for V2’s trajectory-checking question: TSR vs TSRUC quantifies exactly what V1’s binary DB-state grading misses, and their audit-log representation (table, column, old, new) is a ready-made schema for a trajectory check that is still deterministic and cheap. The hidden-workflow framing also reframes the “minimum world a practitioner recognises” question — an ITSM world without SLA escalations, auto-reassignment and approval cascades is missing the part that makes the work hard. It does not cover log grounding (constraints and workflows are expert-authored, and the seed data is ServiceNow demo data, not operational records), has no user simulator at all (single task description, no requester), and no multi-agent track.
- WorldSmith — tool-dependency-graph sampling is the one concrete published method here for generating valid multi-hop trajectories without a human, and it works precisely because connectedness is what surfaces hidden side effects. Their honest framing of the trade-off (manual = quality, graph = scale, LLM-random = diversity collapse) is the right frame for the automation ledger.
- Benchmark Research — reporting a constrained success rate alongside plain success rate as a matter of course; scoring forward and inverse dynamics as separate capabilities from task completion; and the observability ablation as a way to attribute failure to information vs. reasoning.
- Enterprise Environments — home concept since 2026-09-10
Questions it raises for ITSMBench:
- BPI event logs are, structurally, audit logs — activity, resource, timestamp, case. WoW had to bolt an oracle audit channel onto ServiceNow to make dynamics measurable. Does V2 get that channel for free from the log format, and does it change the answer to “what trajectory checks are needed and what do they cost”?
- Their constraint violations are hand-designed traps built by experts who studied the hidden rules. BPI 2013/2014 contain real violations — SLA breaches, ping-pong reassignments, wait-user loops. Can real violated cases be mined as constraint-understanding tasks rather than invented?
- TSRUC near zero means the benchmark separates models on constraint compliance, not on task completion. If V2 adopts a constrained metric, does it still discriminate, or does everything sit at the floor — and is that the discrimination-and-stability yardstick or a failure of it?
Notes
(Our take: to add after reading; drafted 2026-09-09, not yet discussed)