Vibrant Labs Research

Enterprise Environments

3 min read

What it is

Benchmarks that stand in for a company: ServiceNow-style ITSM sandboxes (EnterpriseOps-Gym, World of Workflows), multi-department role collaboration (EntCollabBench, TheAgentCompany), live operations (ITBench), and adjacent regulated domains (CHI-Bench, FaulT-Bench). All of them invent their world from vendor schemas or author judgment; none is grounded in real operational logs, and none combines log grounding, a user simulator, and a multi-agent track. AgentSimulator and the BlueSky agenda paper are the closest work on turning event logs into agent systems, and Agent Mentor mines an agent’s own logs. This is the home concept for ITSMBench.

(agent-drafted from the sources’ summaries, not yet reviewed)

Key sources

  • EnterpriseOps-Gym — ServiceNow sandbox, 164 tables, 512 tools, 1,150 SME-authored tasks, SQL end-state verifiers; best model 37.4%, ITSM hardest at 28.5%
  • EntCollabBench — 11 role-specialized agents across six departments with permission isolation; trace + DB-diff judging by three-model vote; best 62%
  • World of Workflows — ServiceNow instance with 4.8K hidden business rules; success under constraints collapses to ~2% without audit-log visibility
  • ITBench — 94 live Kubernetes SRE/CISO/FinOps scenarios with topology-aware partial credit; GPT-4o resolves 13.8% of SRE diagnoses, 0% hard mitigations
  • CHI-Bench — Hand-built 20-app healthcare simulator, 75 policy-dense multi-role tasks; best agent 28% pass@1, two-agent provider-payer arena collapses to 0%
  • FaulT-Bench — 200 network troubleshooting scenarios where 120 tickets are wrong; agents saturate real faults but over-diagnose healthy networks up to 47%
  • TheAgentCompany — Self-hosted software startup with Sotopia-backed simulated colleagues; weighted checkpoint scoring; best agent completes 30.3% of 175 tasks
  • AgentSimulator — Discovers a multi-agent system from an event log — per-resource schedules, capabilities, timings, handover matrix — beating control-flow-first simulators 20-80x faster
  • From Event Logs to Governed Action — BlueSky agenda for mining logs into governed actions where act, defer, ask and refuse are all valid scored outputs; names four artifacts, no method
  • Agent Mentor — Mines an agent’s own trajectory logs by clustering outcomes and eliciting SVO features, then injects derived corrections into prompts; +37pp on ambiguous specs
  • Tau-Tau-Bench — scores a builder end-to-end on one artifact under a serving budget with the eval suite withheld; best 23.9% vs 82.2% expert ceiling, and developers weaken their own failing tests
  • EnterpriseBench Corecraft — Expert-built enterprise world whose rubrics are LLM-judged and never validated; supplies the transfer evidence and the three OOD benchmarks, and is the position WorldSmith argues against

To ingest

(none)

Graph View

Backlinks