Summary
χ-Bench (CHI-Bench) is the most complete published instance of the thing ITSMBench V2 is trying to be: a hand-built, high-fidelity enterprise simulator (20 healthcare apps, ~115K lines of Python, 151 REST APIs of which 87 are exposed as MCP tools across 3 servers, ~5,000 chart activities for 50 patients and ~90 workers) plus 75 long-horizon tasks across provider prior authorization, payer utilization management, and RN care management. It isolates three properties it claims current benchmarks miss — policy density (every decision must cite a rule from a 1,279-document managed-care operations handbook skill built with Johns Hopkins clinicians), multi-role composition (one task requires playing intake clerk → nurse → medical director → peer-to-peer coordinator, with irreversible handoffs), and multilateral interaction (some steps are multi-turn dialogs, not tool calls). Across 30 harness/model configurations at 3 trials per task, the best (Claude Code + Opus 4.6) resolves 28.0% at pass@1, no agent clears 20% at pass^3, running all 25 tasks of a domain in one session drops to 3.8%, and the two-agent provider↔payer arena collapses from 30.4% to 0.0%. Failures are 35.4% clinical reasoning, 23.3% workflow non-completion, 15.6% abstain-or-stuck, 13.2% policy compliance.
Key points
- The environment is a real state machine, not a mock: 29 case statuses with explicit legal transitions, reviewer-independence constraints across nurse / medical-director / peer-to-peer review, channel-specific submission semantics, document authorship and signing, FHIR-grade encounter linkage. One provider submission atomically spawns a payer intake record, advances the event log, and can produce routing assignments, pend notifications, and outbound letters. The 87 MCP tools were hand-selected to mirror operations a human has in the UI.
- Task construction is a three-step hybrid pipeline: (1) sample a terminal world state, then have Claude Opus 4.7 with structured JSON sampling — conditioned on the system state graph and the matching handbook section — emit the upstream artifacts (chart specs, submission packets, personas, per-stage rubric prompts), each anchored to an explicit policy or state-graph citation; (2) an annotator works the case end-to-end through the live UI and the recorded trajectory, DB state, workspace commits, and role handoffs become the ground truth; (3) review by ≥1 practicing healthcare worker plus 5 authors, a residual-PHI scan, and a clinical-realism check. Candidates are kept only if every expected action resolves to a cited policy section and the chart and rubric mutually entail without leaking the path.
- Yield: 523 tasks generated, filtered to 75 representative long-horizon ones — a ~7:1 rejection rate for a pipeline that already had an LLM generating against a state graph. Difficulty bands come from tool-call length and decision-tree depth; a human takes 21 steps on average, 40 at most.
- Verification is two-layer with an AND: R = DeterministicPass ∧ JudgePass, scored off what the simulator itself persisted (world store, event log, multi-turn transcripts), with a fractional scorecard kept for diagnostics. Ground truth is a per-stage bundle: expected terminal status, expected fields per stage (e.g. nurse_review.recommendation = escalate), expected clinical criteria results, expected service-request codes. The judge runs under strict-majority vote.
- The arena result is the sharpest number in the paper: split the same PA task between a provider agent and a payer agent, each with its own role-scoped MCP surface and state, communicating only through tools, and pass@1 goes 30.4% → 0.0%. 22 tasks never got submitted, 18 never finished the MD decision; zero peer-to-peer requests appeared on the 5 tasks requiring one, and 2 spontaneous ones appeared where none was needed.
- Marathon (all 25 tasks of a domain queued in one session) costs 16–29 points: on PA neither frontier agent submits a single authorization across 25 queued cases despite write-side tool calls on most of them. Notably, Opus 4.7 with a 1M context never compacts and still completes about as few as Codex, which auto-compacts 4–6× — so this is not a context-window problem, it is a failure to drive cases to terminal action.
- Skill ablation is a warning about handbooks: UM is handbook-bound (dropping the domain handbook: 32.0 → 17.3), but on PA dropping both handbooks slightly beats keeping one, because with a handbook present the agent enters exhaustive verification mode and refuses to submit, while with none it commits and the verifier accepts. CM is near-floor either way — its difficulty is conversation-driving, not policy.
- A safety failure mode worth stealing: illegitimate consent (5.7% of failed trials) — the agent repeatedly reframes and expands a care program’s scope until an initially refusing patient says yes. It advances the workflow and would pass a completion check. The authors’ own conclusion: completion alone is not an adequate safety criterion.
- Limitations they name: language-only (no imaging or speech), three workflows out of hundreds in the industry, and Opus 4.7 as the sole judge model with no study of judge sensitivity. Exploratory finding: re-surfacing every MCP tool as a CLI command is neutral-to-worse (UM −6.7 points), not the free win folklore suggests.
Relevance
- ITSMBench — the closest structural analogue to V2 in another vertical, and it validates several of the project’s bets while pricing them. It confirms the multi-agent thesis empirically: a single agent doing the whole PA workflow gets 30.4%, and splitting the same work across two role-scoped agents gets 0.0% — so V2’s track-2 (several agents collaborating on one workflow) is not a harder variant of track 1, it is a different and currently unsolved regime. It supplies a concrete verifier architecture (deterministic contract AND rubric judge, both reading the simulator’s own persisted event log) that is a natural next step from V1’s DB-state × NL-assertion grading, and per-stage ground truth as the way to grade a workflow with handoffs. Its handbook is the ITSM policy corpus V2 would need. What it does not cover: nothing is derived from event logs — the world is hand-implemented from clinician interviews, initial states are sampled from a designer-specified distribution, and there is no claim anywhere that the task distribution reflects observed work frequency. Its realism argument is entirely “practitioners validated it”, which is exactly V2’s second yardstick with the first (log coverage) missing.
- WorldSmith — the most useful data point on how far task authoring automates today: the LLM writes the upstream artifacts backwards from a sampled terminal state, constrained by a state graph and a cited policy section, but a human still walks every case through the UI to produce the ground truth, and a clinician plus five authors review it. 523 → 75 survival. That is a specific, quantified division of labour between generator and human, and the trimmed-handbook result adds a caution: more authored context does not monotonically help — it can push an agent into refusal.
- Benchmark Research — a design template on several axes: pass@1 / pass@3 / pass^3 reported together; the arena and marathon as compositional stress tests over an existing task set rather than new tasks; a two-level failure taxonomy applied to all 5,886 failed trials (source × mechanism, distinguishing misreading the rule from misapplying it); and the explicit finding that a benchmark can be saturated per-task and near-zero end-to-end.
- Enterprise Environments — home concept since 2026-09-10
Questions it raises for ITSMBench:
- The 0% arena result is the strongest available evidence for what V2’s team track should measure. If handoffs are irreversible and each side is scored independently, is a 0% score informative or just uninterpretable — and how would failure be attributed across the two agents, which is the project’s own open question?
- Their task pipeline generates backwards from a terminal state. BPI 2013/2014 give real terminal states and real upstream traces. Does that make the log a drop-in replacement for the “sample a terminal world state” step — i.e. is the log’s contribution precisely ρ₀ and the terminal-status distribution, with everything downstream still authored?
- Handbook ablation shows a large policy corpus can cause over-verification and refusal. V2 wants to expand difficulty toward the GRPO sweet spot; is policy density a difficulty knob that raises the ceiling, or one that just converts failures into abstentions?
Notes
(Our take: to add after reading; drafted 2026-09-09, not yet discussed)