Summary
Argues that every existing simulator-fidelity study has a structural blind spot: ground truth comes from paid participants role-playing an assigned goal, so the user’s willingness is exogenous and no benchmark can test whether a simulator decides like a real person whose motivation is latent and decaying. Introduces decision fidelity and measures it on ZhenaiSales, 2,790 production conversations between a deployed LLM sales agent and real customers of a Chinese matchmaking service, 793 with a verified payment. Using a teacher-forced probe protocol (feed the simulator the real conversation prefix, ask only for the next user turn, score both real and simulated turns with the same fixed LLM instrument), it finds a disengagement deficit: simulators reproduce eventual buyers almost exactly (engagement-depth bias +0.09) but push eventual non-buyers toward the purchase frame (+0.40; contrast d=0.38, permutation p<0.001, n=374), halving expressed resistance (25.1% to 13.5%) and nearly doubling deliberation (21.9% to 40.1%) without fabricating any purchases. The deficit survives a model-family swap and an instrument swap, and explicitly telling the simulator it may disengage cuts marginal bias five-fold while leaving the outcome-conditioned contrast intact (d=0.34, p=0.008). The practical consequence: after a pitch, simulated users deliberate 75.2% of the time versus 45.3% for real ones, so the simulator over-rewards exactly the tactics that lose real customers.
Key points
- The framing move: communicative fidelity asks whether the simulator sounds human; decision fidelity asks whether it decides like one. Role-players handed a goal never genuinely decide not to act, so no assigned-goal study can measure the second.
- The protocol needs no new human study — only logged production conversations with verified outcomes. Context is held fixed (real prefix, next turn only) and the measurement instrument is held fixed (one LLM perceiver maps both real and simulated turns to the same schema), so its biases cancel in the difference.
- Decision-state schema: engagement stage (exploring / engaging / considering / deciding / resisting), emotion (positive / neutral / hesitant / negative), blocker (none / price / trust / capability / timing / external / prior-failure), scored causally from the prefix only so no future information leaks.
- The endpoint is outcome-conditioned, not marginal: Δ = (bias on non-buyers) − (bias on buyers). A simulator can look unbiased overall and still be wrong entirely on the population that walks away. Perfect fidelity means Δ→0 and the buyer stratum stays matched.
- Prompting is not the fix and this is the paper’s central claim: the “instructed” simulator, explicitly licensed to be disinterested, perfunctory, impatient, or silent, learns to perform disengagement but aims it at the wrong people — it over-corrects buyers to −0.187 while the stratum gap stays at +0.285. The cause is diagnosed as instruction tuning, which rewards being an engaged interlocutor; willingness-decay is largely absent from alignment data.
- Action-conditioned analysis (595 probes) shows the distortion is concentrated on pressure tactics: after a pitch, deliberation +29.9pp (p<0.001); after a push-close, resistance suppressed 18.9% to 10.7% (p=0.046); after a neutral probe the shift is negligible. This makes the simulator an actively misleading training signal, not merely a noisy one.
- Data discipline worth copying: converted conversations are truncated at the first payment timestamp (mean 19 turns dropped per buyer) so post-purchase service chat cannot leak the outcome; 20 internal test users excluded; 98% carry a structured demographic profile; median 16 turns.
- The prescription: decision fidelity must be learned from real disengagement trajectories or enforced by an external willingness model gating the simulator’s cooperativeness. Prompt engineering is ruled out by evidence, not by assumption.
- Limitations the author names: one domain, one language, and culturally specific refusal forms (“busy”, “no need”); LLM-assigned states with no human-validated labels yet; teacher-forced next-turn only, not free-running rollouts (expected to worsen the deficit); prompted persona simulators only, no retrieval-grounded or fine-tuned ones; payment is a coarse endpoint proxy for a latent willingness trajectory. Single author.
Relevance
- ITSMBench — the most directly transferable methodology in this cluster, because its ground truth is logged production conversations with recorded outcomes, which is the shape of BPI 2013/2014 and of any ITSM ticket corpus. The teacher-forced probe (replay the real prefix, ask the simulator for the next move, score both with a fixed instrument) is a concrete version of our “replay them, compare trajectories, and see where things change”, and the outcome-conditioned contrast gives it an endpoint instead of an eyeball comparison. The ITSM analogue of “walking away” is a requester who stops responding, closes the ticket themselves, escalates outside the process, or lets it age into a wait-user state — all visible in BPI activity codes. What it does not cover: no enterprise or IT domain, no operator-side simulation, no multi-agent or colleague simulation, and the decision states are sales-funnel stages that would need an ITSM re-definition.
- WorldSmith — a hard bound on automating the simulator step: the paper shows by experiment that the failure cannot be prompted away, so an agent-built simulator needs either a real trajectory corpus to learn from or an external willingness model to gate it. Neither is a prompt-authoring task.
- Benchmark Research — a validity result about a whole class of benchmarks: fidelity measured marginally can be near-zero while the simulator is systematically wrong on the outcome stratum that decides the score, and the error is concentrated on exactly the agent behaviours the benchmark is meant to grade.
- User Simulators — home concept since 2026-09-10
Questions it raises for ITSMBench:
- Extends the fidelity yardstick: Mind the Sim2Real Gap scores style against fresh humans; this scores decisions against logged outcomes. Can V2 define an ITSM engagement-depth schema (responsive / providing info / stalling / escalating / abandoning) over BPI activity sequences, and use outcome-conditioned Δ against reopened, breached, and abandoned cases as the simulator-calibration metric that does not need a human study?
- Contradicts the “just prompt the persona” approach twice over — here and in RealUserSim — but the two disagree on the remedy: RealUserSim mines profiles from transcripts and gets style, this paper says decision behaviour has to be learned or gated externally. Does V2 need both, a mined style profile and a willingness model driving when the simulated requester goes quiet?
- Open question on grading: if simulated requesters never disengage, V2 would systematically over-reward agents that keep pushing rather than agents that correctly close, defer, or hand off a stalled ticket. Should the verifier ledger include an “agent correctly recognised a disengaging requester” outcome, and can that be labelled from logs at all?
Notes
(Our take: to add after reading; drafted 2026-09-09, not yet discussed)