Summary
EntCollabBench simulates a permission-isolated organization of 11 role-specialized agents across six departments (IT, HR, Customer Service, Shared Services, Engineering, Approval Center) over 8 MCP-exposed enterprise services — ITSM, HR, CSM, Gitea plus Email, Calendar, Teams, Drive. Its 300 tasks split into a Workflow subset (160 single + 40 multi-step) graded on execution traces plus database state diffs, and an Approval subset (80 + 20) graded against a deterministic decision engine over a 290-rule policy schema extracted from the GitLab Handbook and GDPR. Every task requires at least two departments, and cross-role handoff is an explicit typed tool (ask_<agent>_by_http) with each agent holding isolated memory and its own per-service identity token, so out-of-scope calls are rejected server-side. Role-level pass rates exceed 80% for strong models while end-to-end task accuracy is far lower — best overall is DeepSeek-V4-Pro at 62.0%, and multi-step workflow drops from 78.3% subtask to 50.0% task. The authors locate the bottleneck in delegation, context transfer, parameter-level grounding, and workflow closure rather than in isolated tool use; its tool catalog and seed database are explicitly derived from EnterpriseOps-Gym.
Key points
- Multi-agent is the object under test, not an orchestration choice: the same LLM instantiates all 11 roles, each with a role-specific system prompt, an isolated memory, and a permission set enforced twice — by tool visibility and by per-service identity tokens rejected at the server. Delegation may recurse up to a depth cap.
- Grading is three-signal and per-agent: for each (agent, subtask) pair the judge sees the reference trajectory slice, the actual trace slice, and the normalized DB state diff as objective evidence, and tolerates equivalent implementations and reordering but fails missing key actions or arguments contradicted by state. A subtask passes only if every involved agent passes; a task passes only if every subtask passes.
- The LLM judge is a three-model majority vote (Gemini-3.1-Pro, GPT-5.4, Claude-Sonnet-4.6), validated against human annotation at 96% and 98% agreement on 50 cases each. This is the cleanest published recipe for making a trajectory-plus-state judge defensible.
- Workflow tasks are generated category-first from 20 hand-written domain templates crossed with 5 trigger types and 5 governance rules, instantiated over seed-database objects, then filtered by checks that the trajectory uses only cataloged tools, respects tool ownership, delegates every cross-department transition explicitly, and recovers all argument values from the instruction.
- Multi-step tasks are cut along a natural three-phase structure — record establishment, technical resolution, coordination and closure — each phase becoming a subtask with its own starting agent and ground-truth fragment, and each chain must contain at least three cross-agent delegations so it cannot collapse into one agent’s tool calls.
- Approval tasks are synthesized top-down: chunk → LLM classifier → LLM extractor into a strict JSON rule schema where each citation must be a verbatim contiguous substring of the source chunk (a falsifiable grounding check against hallucinated rules), then sample target rules, add non-applicable distractors, and apply perturbations (delete a predicate field to force a missing-information flag, withhold required evidence, span multiple roles).
- Failure taxonomy is a direct catalogue of what handoffs break: missing or premature delegation, insufficient context passed downstream, prefix decay across multi-step chains, and failure at the final closure step. Role difficulty tracks position in the delegation chain — the Knowledge Base Specialist scores far better as the starting agent than downstream.
- A recurring stateful failure: after failing to find a record (wrong identifier field), agents create a new record instead of updating the existing one — persistent, wrong enterprise state that a naive check might miss. Collaboration tools (Email, Teams, Calendar) cause disproportionate closure failures via sender-identity and payload-format errors.
- Limitations the authors name: fully simulated, no human preferences, no informal communication norms, no noisy or incomplete real-world records; 11 agents is finite coverage; policy sources are selected and may not reflect real jurisdictional ambiguity; model-based judges reduce but do not remove evaluator bias; multi-agent runs are expensive in tokens.
Relevance
- ITSMBench — the closest published model for V2’s track 2 (several agents under test collaborating on one workflow): typed delegation tools, permission isolation enforced at the server, per-agent judgment against a reference trajectory slice plus state diff, and a three-tier metric (agent / subtask / task) that makes failure attribution explicit — exactly the “what is scored in the team track, and how are failures attributed” open question. Its multi-step decomposition into establish / resolve / close is also a candidate mapping from a BPI case trace to subtasks. It does not cover log grounding (seed data is borrowed from EnterpriseOps-Gym, itself SME-invented) or a user simulator at all — the requester is a single instruction string, and there is no simulated colleague outside the fixed agent roster.
- WorldSmith — a mostly-automated construction pipeline: workflow tasks from 20 templates × 5 triggers × 5 governance rules over a seed DB, approval tasks generated deterministically from an LLM-extracted policy schema with verbatim-substring citation checks. The human cost sits in the 20 templates and the policy corpus choice, not in per-task authoring — a sharp contrast with EnterpriseOps-Gym’s ~100 USD per task.
- Benchmark Research — three-model majority-vote judging validated against human annotation, the agent/subtask/task metric ladder, and distractor rules plus predicate-deletion perturbations as a way to defeat shallow pattern matching while keeping ground truth deterministic.
- Enterprise Environments — home concept since 2026-09-10
Questions it raises for ITSMBench:
- Their reference trajectory is generated from a template, so the judge is comparing against a synthetic gold path. If V2’s gold paths come from replayed BPI traces instead, does the same “tolerate equivalent implementations” judge still work, or do real logs contain too much variance for trajectory comparison?
- Role-level accuracy above 80% with task accuracy near 50% says almost all the loss is in handoffs. If V2 scores the team track this way, is the interesting metric the delegation edges rather than the tasks — and can BPI reassignment and ping-pong patterns supply a ground-truth distribution over those edges?
- The Approval subset shows policy adjudication can be graded deterministically from an extracted rule schema, no DB state needed. ITSM change approvals and SLA rules look like the same shape. Is that a cheap third verifier class for V2 alongside DB-state and NL assertions?
Notes
(Our take: to add after reading; drafted 2026-09-09, not yet discussed)