Summary
EnterpriseOps-Gym (ServiceNow Research) is a containerized enterprise sandbox of 164 relational tables and 512 tools spanning eight domains — CSM, HR, ITSM, Email, Calendar, Teams, Drive, and a cross-domain Hybrid category — with 1,150 expert-authored tasks including 30 infeasible ones. Every task was written by a domain SME who designed the scenario, executed a gold trajectory in the sandbox, wrote a natural-language oracle plan, and hand-wrote SQL verifiers over the final database state; the campaign ran four months with 160+ contributors at roughly 100 USD per task. Across 14 frontier models the best, Claude Opus 4.5, reaches only 37.4% pass@1, and ITSM is the hardest single domain (best 28.5%) alongside Hybrid (30.7%). The headline diagnosis is that planning, not tool use, is the bottleneck: adding distractor tools changes nothing (±1%), while conditioning on human-authored oracle plans gains 14–35 points, and multi-agent decomposition can regress below a plain ReAct loop. Refusal of infeasible tasks is weak (best 53.9%), and permission/process-compliance verifiers fail more often than task-completion verifiers.
Key points
- Environment is a Docker sandbox with a fresh DB instance per task run (no state leakage), 164 tables at mean 1.7 foreign keys per table, ~3,443 rows per task, and a per-task subgraph averaging 24.9 tables (up to 73 in Hybrid). Schemas were reverse-engineered from public vendor API docs and data models but kept deliberately platform-agnostic — no proprietary dataset.
- Task authoring is fully human and expensive: annotators design a scenario against complexity thresholds (tool count, verifier count, state dependencies, access constraints, policy conflicts), then extend the sandbox with whatever new tables and tools the task needs. Tasks are constrained to a unique final state though multiple valid paths may reach it — the same discipline as tau-bench.
- Verification is outcome-based only: hand-written SQL scripts over the final state, averaging 5.3 conditions per task and reaching 44. Verifiers split into three kinds — task completion, integrity constraints (foreign keys intact), and permission/process compliance — plus explicit side-effect checks. No trajectory grading, no LLM judge, no rubric.
- There is no user simulator. Tasks arrive as a single user prompt with a system-prompt policy, and the system prompt explicitly tells the agent it is “fully authorized” and need not ask for confirmation or clarification. Multi-turn interaction is out of scope.
- ITSM domain: agent plays an IT Service Desk Engineer over ITIL records (Incidents, Problems, Changes, CMDB), 93 tools — the largest toolset of any domain. Reasoning is relational and causal: link incidents to root causes, compute priority from impact × urgency, respect change-approval workflows, close a change out with linked incidents/problems plus caller notification.
- QA is multi-layered: draft tasks are stress-tested against GPT-5, Gemini, and Claude to surface bad task definitions, missing tools, invalid DB entries, and access-control conflicts, and to discard trivial tasks; then human reviewers score trajectory quality, prompt clarity, and verifier robustness against rubrics.
- Multi-agent findings are negative: a Claude-Sonnet planner + executor beats plain ReAct (+10.7 CSM, +8.8 HR), but adding task decomposition with per-subtask subagents regresses below the ReAct baseline in CSM (16.2 vs 16.7), which the authors attribute to strong sequential state dependencies that decomposition breaks.
- Pass@1 degrades monotonically with horizon (group mean ~35% at 4 steps to under 20% by 16); expert trajectories average 9.15 steps, up to 34. Thinking budget helps but plateaus domain-specifically — ITSM goes 1.1 → 6.1 → 6.1% and stops.
- Limitations: single-turn only, no colleagues or teams, no real operational data behind the seed (SME-invented records guided by API docs), pass@1 averaged over three runs rather than pass^k, and the whole thing is human-bound at ~100 USD per task.
Relevance
- ITSMBench — this is the paper ITSMBench V1 built on, and it sets the bar for what a hand-built enterprise benchmark costs and yields: SME-authored scenarios, gold trajectories, oracle plans, SQL end-state verifiers with integrity and permission classes, and infeasible tasks for refusal. Its verifier taxonomy (completion / integrity / permission-and-process) is directly reusable for V2 grading, and its ITSM domain is the closest published analogue to the V1 task set. It does not cover any of V2’s three additions: no log grounding (seed data is SME-invented from vendor docs, not event logs), no user simulator (single-turn, explicitly authorized agent), and no multi-agent track under test — its multi-agent experiments are orchestration ablations of one agent under test, and they come out negative.
- WorldSmith — the strongest published cost datum for the automation question: 4 months, 160+ contributors, ~100 USD per task for 1,150 tasks. The only automated step is LLM stress-testing of drafts to prune trivial tasks; scenario design, gold execution, plan authoring, and verifier writing are all human. If any of that is automatable, this is the ledger it has to beat.
- Benchmark Research — the oracle-plan ablation is a clean way to separate planning from execution (14–35 point ceiling), the distractor-tool ablation isolates retrieval as a non-bottleneck, and stratifying pass@1 by expert-trajectory horizon length gives a difficulty axis that does not need a new task set.
- Enterprise Environments — home concept since 2026-09-10
Questions it raises for ITSMBench:
- ITSM is the hardest domain here and also the one where test-time compute plateaus earliest. Is that a genuine property of policy-heavy service work, or an artefact of verifiers that check permission and process compliance more densely than other domains — and which is V2 measuring?
- The unique-final-state constraint plus SQL end-state verifiers is exactly V1’s design. Their own failure taxonomy (cascading state propagation, missing prerequisite lookup) describes path errors that an end-state check can only catch when they happen to corrupt the final state. What would BPI-derived trajectory checks add on top?
- Decomposition into subagents regressed because of sequential state dependencies. V2’s track 1 (agent plus simulated colleagues) and track 2 (several agents under test) both impose handoffs on the same kind of stateful workflow — is the negative result a warning about the tracks, or about naive decomposition of a task that was authored as one person’s job?
Notes
(Our take: to add after reading; drafted 2026-09-09, not yet discussed)