Summary
tau-bench (Sierra) evaluates agents that must talk to an LM-simulated user, call database APIs, and obey a Markdown domain policy, across two hand-built domains (retail: 115 tasks, 500 users, 1,000 orders, 7 write and 8 read APIs; airline: 50 tasks, 300 flights, 2,000 reservations). Each task is a POMDP whose reward is binary: the final DB must equal the annotated goal state and agent messages must contain the required output strings. It introduces pass^k, the probability that all k i.i.d. trials succeed. GPT-4o function calling is the best model at 61.2% retail and 35.2% airline pass^1, yet its retail pass^8 falls below 25%; open-weight models trail far behind. The authors conclude that LM function-calling agents lack the consistency and rule-following needed for real deployment, and that a small set of carefully annotated tasks run many times tells you more than a large noisy set.
Key points
- User simulator = gpt-4-0613 at temperature 1.0 whose system prompt is the hidden task instruction (identity, intent, preferences, persona traits such as “in debt and sad today, but very brief”); it sees only the user-agent chat, never tool calls, and ends the episode with a STOP token. No fidelity calibration against real users is attempted.
- Task spec is two fields: a natural-language user instruction and a ground-truth list of DB write actions (plus optional output strings). The instruction is written to guarantee exactly one legal outcome under the policy; ambiguity is removed by running a gpt-4-turbo agent more than 40 times per task and editing until failures are agent faults, not spec faults.
- Grading: reward = action match × output match, in {0,1}. Action match compares end DB state with the goal (reads are free, write set must match); output match is substring match on agent messages. The authors admit reward 1 is necessary, not sufficient: an agent can reach the right state while skipping mandatory user confirmation. Trajectory checks are left as future work.
- pass^k = E_task[C(c,k)/C(n,k)] for c successes in n trials. Stochasticity comes only from LM sampling of user and agent, so the metric isolates conversational robustness. GPT-4o: over 60% pass^1 but under 25% pass^8 in retail.
- Construction in three stages: hand-designed schemas, APIs, and policies (kept minimal for logical consistency); gpt-4-written data-generation code sampling from random distributions and LM lists; manual task annotation via agent runs, with agent actions copied and edited as ground truth. Policy rules split between API-enforced checks and prose the agent must apply itself.
- Failure breakdown (36 GPT-4o retail failures): about 55% wrong argument or info (DB reasoning, arithmetic, hallucinated IDs), 25% wrong decision (rule violation), 19% partial completion of compound requests; more DB writes means harder. Removing the policy costs GPT-4o 4.4 points in retail but 22.4 in airline, so retail success is mostly common sense.
- Simulator limitations the authors name: instruction typos and ambiguity (4 of 40 initial failures were spec bugs), the user lacks domain knowledge and so authorises policy-violating actions, and the user LM fails at reasoning, arithmetic, and long-context adherence. They frame this partly as realism.
- Cost and bias: one trial over retail costs about $200; tuning instructions against gpt-4-turbo bakes that model’s behaviour into the task set.
Relevance
- ITSMBench — supplies V2’s user-simulator template (instruction as system prompt, persona traits, STOP token, sees no tool log), the unique-outcome task discipline with agent-run debugging, DB-state × output grading, and the pass^k stability yardstick. It does not cover multi-agent or colleague simulation, log-grounded task distributions, simulator fidelity measurement, trajectory or policy-path checks, or real-outcome validation.
- WorldSmith — stages I and II (schemas, policies, data generation) are already LM-assisted, but stage III (making each task uniquely resolvable) is manual and the hardest step, exactly the part WorldSmith would have to automate.
- Benchmark Research — origin of pass^k and of the small-tasks-many-trials design argument; end-state grading and its known blind spot.
- User Simulators — home concept since 2026-09-10
Questions it raises for ITSMBench:
- Can BPI case traces stand in for the hand-tuned unique-outcome guarantee? If the log shows several legitimate resolution paths for one incident type, does V2 grade the set of paths or force one?
- The simulator is deliberately uncalibrated and over-compliant; the ITSM operator in the logs waits, reassigns, and has incomplete information. Which log patterns become simulator behaviours, and does pass^k then measure agent robustness or simulator noise?
- End-state grading passes policy-violating paths. With V2 tracks adding approvals and handoffs, what minimum trajectory check (confirmation before write, correct assignee sequence) is needed, and how much does it change agreement with practitioners?
Notes
(Our take: to add after reading; drafted 2026-09-09, not yet discussed)