What it is
How agent benchmarks stand in for the human on the other side of the conversation. tau-bench set the pattern (hidden instruction as a system prompt, uncalibrated), tau²-bench made the user act through tools, and the 2026 fidelity studies measured the gap: simulators are too polite and too cooperative, inflate agent success, and prompting for realism made fidelity worse. Grounding in real interaction data (RealUserSim, SAGE) and replay against verified outcomes (Simulated Customers) are the working answers. For ITSMBench the open move is calibrating an operator and requester simulator from event logs rather than from conversations.
(agent-drafted from the sources’ summaries, not yet reviewed)
Key sources
- Tau-Bench — the template: instruction-as-system-prompt user, sees no tool calls, uncalibrated; ends episodes with a STOP token
- Tau2-Bench — user with its own tools (dual control); tool-grounding cuts simulator error from 40% to 16%; removing the user raises pass^1 by 18 to 25 points
- Mind the Sim2Real Gap — validity study of the tau-bench user sim and binary reward, with reusable fidelity metrics
- RealUserSim — Grounds simulators in 7,275 profiles mined from real WildChat logs; fidelity 24.2%→45.3%, and tau-bench’s own persona directives inflate difficulty
- Simulated Customers Never Walk Away — Simulators match real buyers but inflate non-buyers (d=0.38) against verified purchase outcomes; disengagement cannot be prompted in
- SAGE — Cold-start simulator grounded in ICPs plus the agent’s own catalogs and FAQs; finds 25-33% more bugs than baselines and than real traffic
- VISTA — Six simulator-quality metrics (tool-transition and trajectory-distance coverage) plus a hybrid UI+API user; 42% more failures than UI-only
- CRMArena-Pro — incremental-release persona simulator; 58% single-turn falls to 35% multi-turn
- Tau-Tau-Bench — scores a builder end-to-end on one artifact under a serving budget with the eval suite withheld; best 23.9% vs 82.2% expert ceiling, and developers weaken their own failing tests
- HiL-Bench — Necessity/sufficiency task admission checked from rollouts; ask_human() is a frozen-LLM expert oracle without the price; Ask-F1’s harmonic mean builds the anti-hack into the metric
Related
To ingest
(none)