Summary
Diagnoses two failure modes of LLM user simulators and offers a constructive fix: ungrounded simulators hit a Formalism Ceiling (6-8% style match against real users, everything normalised into polished prose), while hand-written behavioural directives (“you are an angry customer”) trigger Directive Amplification, where models hyper-interpret the instruction into caricature that differs wildly by model. RealUserSim mines 7,275 executable behavioural profiles from 14,000+ real human-LLM conversations in WildChat and injects them into the simulator as “Command + Example” pairs. On a new fidelity benchmark (PT3: 600 conversations, 71+ domains, LLM judge comparing a synthetic trajectory against the real one on five dimensions, with anti-leakage cleaning) grounding lifts match rate from 24.2% to 45.3%. On tau-bench with 6 simulator models and GPT-4.1 as agent, grounded personas cost the agent a mean 3.2 points (Airline) and 3.5 points (Retail) and surface three failure mechanisms invisible to cooperative simulators; stripping tau-bench’s own directives raises Retail success by up to 16.7 points, showing the published difficulty is partly an artefact of the directives.
Key points
- Grounding source is real interaction logs, not fresh human studies: WildChat 4.8M filtered to 21,637 multi-turn English trajectories over 7,311 users, one consolidated profile per user (linguistic style + demographics), 99.5% coverage.
- The profile format is the mechanism: an “Executable Persona Manual” of up to 15 Command + Example pairs, commands quoted with the user’s own messages. Commands say what pattern to reproduce, examples calibrate how far — commands alone overshoot into caricature.
- PT3 fidelity metric: paired audit of a real trajectory vs a synthetic one from the same profile on the same task, judged blind on five dimensions (persona/affect, linguistic style, tech competency, interaction and data flow, pacing and sequencing); Fidelity Index is the mean match rate.
- Grounding gains are style-only: persona/affect +31.8, interaction/data flow +28.3, pacing +25.5, linguistic style +20.0, while tech competency stays flat at ~93% (and drops 10 points in Technology & IT, where an averaged profile overshoots the expertise of a specific conversation).
- Linguistic style stays the hardest dimension (26.2% even with profiles): GPT-4o cannot reliably produce messy or idiosyncratic text; performance tracks how many examples anti-leakage cleaning stripped (single-conversation users lose all examples).
- Directive Amplification makes benchmark scores incomparable across simulator models: “you are extremely distraught” gives Llama-3-70b stage directions (8.6% roleplay markers) and GPT-4o nothing (0.0%); GPT-5-mini’s message length doubles to 1,218 chars. Without directives all six models default to cooperative.
- Three named failure mechanisms from realistic personas: information loss (terse user omits which reservations to cancel, agent over-cancels), agent misinterpretation (“GIMME A REFUND PLS” read as a cancellation), compliance failure (an agreeable persona caves on an adversarial scenario it was supposed to resist). Not all effects are negative — terse users sometimes give clearer instructions.
- Persona assignment is a first-order noise source: across 3 seeds the per-model range is 8.0 points (Airline) and 5.9 points (Retail), and the no-persona baseline itself moves 8 points between re-runs; single-run evaluation is statistically insufficient. A synthetic “Perfect User” upper bound adds up to +20.2 points, confirming real personas add real friction.
- Setup and limits: fidelity study is GPT-4o everywhere (simulator, agent, judge), match is judged by an LLM rather than humans, WildChat is generic chat rather than task-oriented enterprise dialogue, and the agent evaluation is tau-bench airline/retail only.
Relevance
- ITSMBench — the first paper in this cluster that grounds the simulator in logged interaction data rather than a fresh human study, which is exactly V2’s bet: profiles are mined from transcripts, cleaned for leakage, and injected as Command + Example pairs. It also supplies a cheap fidelity metric (PT3 paired audit, blind judge, five dimensions) that needs no new human participants, and a hard warning that V1’s hand-written operator directives make scores incomparable across simulator models. What it does not cover: its logs are conversational transcripts, so nothing here says how to build profiles from BPI-style activity logs with no utterances; the users are consumers on generic chat, not enterprise requesters or IT operators; and there is no multi-agent or colleague simulation.
- WorldSmith — the profile pipeline is fully automated end to end (extract, infer, consolidate, clean, inject), so simulator construction is a candidate for automation given a transcript corpus; the human input is the corpus itself, not per-simulator authoring.
- Benchmark Research — a validity finding about tau-bench specifically: part of its published difficulty comes from the behavioural directives, not the tasks, and single-run scores are unreliable at the 6-8 point scale that separates models.
- User Simulators — home concept since 2026-09-10
Questions it raises for ITSMBench:
- Extends “calibrate simulators by replay”: profiles come from logged conversations, not live humans. BPI 2013/2014 have no utterances, but V1’s own conversational operator transcripts and any vendor ticket-comment corpus would. Is a ticket-comment corpus the right grounding source for the V2 requester simulator, and who supplies one?
- Contradicts the current V1 design directly: hand-authored operator directives produce Directive Amplification and make cross-model comparison invalid. Should V2 drop directive-style personas entirely in favour of mined profiles, and re-run V1’s leaderboard without directives to see how much of its difficulty was artificial?
- Open question on the yardsticks: persona sampling alone moved scores by 6-8 points across seeds, larger than most model gaps. If V2 samples operator personas per run, does the discrimination-and-stability yardstick require fixing the persona-task assignment, or reporting over seeds as well as over k?
Notes
(Our take: to add after reading; drafted 2026-09-09, not yet discussed)