Summary
tau²-bench (Sierra) extends tau-bench to a dual-control setting: a telecom troubleshooting domain in which both the agent (CRM tools) and the simulated user (a mocked phone with 15 read and 15 write tools) act on a shared world, formalised as a Dec-POMDP. Tasks are generated compositionally from 15 atomic subtask groups (init, solution, assertion functions) into 2,285 auto-verified tasks, subsampled to 114; retail and airline are re-verified and kept. Frontier models score pass^1 of 34% (gpt-4.1), 42% (o4-mini), 49% (claude-3.7-sonnet) on telecom, and removing the user (agent holds all tools) raises pass^1 by 18 to 25 points, so the authors attribute the gap to communication and coordination rather than reasoning. Tool-grounding the user simulator cuts its annotated error rate from 40-47% to 16%. They conclude that guiding an active user is the bottleneck, and that domain curation still depends on human experts.
Key points
- Dual control as a Dec-POMDP: state = agent DB × user DB × history; each turn one player either calls a tool on its own DB or sends a message; reward is a function of final state. Agent actions can change the user’s world.
- The user simulator has tools: its behaviour is constrained by a mocked device with observable state rather than by long natural-language prompting, which the authors credit for the reliability gain (16% error, 6% critical, vs 40-47% and 12-13%).
- Compositional programmatic task generation: each atomic subtask = (init functions, solution functions, assertion functions); mutually exclusive subtasks share a group; a task picks at most one per group and concatenates. Correctness is checked by applying init then solution and testing asserts, plus a check that the task is not solved before all solution calls. Subtask count is the difficulty knob.
- Domain build pipeline: LLM writes a PRD, then schema, tools, mock DB, unit tests, then manual refinement; the same for the user-side device; policy document generated from tasks, then everything manually refined. Personas (None/Easy/Hard) per task; None performs as badly as Hard.
- Simulator validation is by manual annotation: two annotators per trace, 50 to 100 traces per domain, four criteria, errors split critical vs benign. No comparison to real user traces.
- Grading: five criteria available (DB check, state assertions, NL assertions on history, communication-info check, action matching) but telecom uses assertion functions only.
- Single vs dual control: No-User mode (agent gets a ticket and all tools) vs Default: minus 18 (gpt-4.1) and minus 25 (o4-mini) pass^1; pass^1 goes to about 0 beyond 7 actions in either mode, so horizon length is a second bottleneck.
- Limitations the authors name: user tools not yet applied to retail and airline; domain extension still needs human experts and curation is not automated; the expert-novice gap is not modelled; tiny telecom DB (4 customers, 9 lines).
Relevance
- ITSMBench — gives V2 the pattern for a requester who can act (reboot, re-run an installer, approve, confirm a fix) and a verified-by-construction task generator whose subtask groups could be seeded from BPI activity and variant frequencies to make the sampled distribution log-shaped. It does not cover multi-agent tracks (asymmetric agent-user only), log calibration of the simulator (annotation only), or trajectory-level grading beyond action matching.
- WorldSmith — the five-stage LLM-drafted PRD, tools, tests, tasks, policy pipeline is a partial automation of world-building, and the authors flag domain curation as the unsolved human-bound step.
- Benchmark Research — pass^k for stability, ablation modes (No-User, Oracle Plan) to decompose failure sources, and a simulator error taxonomy (critical vs benign).
- User Simulators — home concept since 2026-09-10
Questions it raises for ITSMBench:
- If tool-grounding the user cuts simulator errors that much, can BPI wait-user and ping-pong patterns be expressed as user tools and states rather than prompts, and would replaying logs against such a simulator give the calibration the paper only does by annotation?
- The atomic-subtask composition assumes issues are independent apart from exclusivity groups; BPI 2013 incidents have reassignment and escalation chains. Do these compose, or does the log-grounded distribution need dependencies the generator cannot express?
- The 18 to 25 point dual-control drop is measured with one user. What happens to pass^k when the second party is a simulated colleague or team (track 1), and how is failure attributed?
Notes
(Our take: to add after reading; drafted 2026-09-09, not yet discussed)