Summary
Prior ways of training user simulators reward matching one ground-truth reply, by log-probability or by an LLM-judged similarity score, even though a real user in the same context could have said many different things. Turing-RL instead rewards indistinguishability: an LLM judge sees the user’s history and two replies, the real one and the model’s, and rates on a 1–7 scale which the real user wrote, and that score (capped to stop the model becoming “more human than human”) is optimized with GRPO after an SFT warm start. On a Qwen3-8B simulator across two domains — multi-turn chat (PRISM) and Reddit threads (ConvoKit) — it beats a similarity reward and a log-probability reward on Turing scores from a different, stronger judge, matches the similarity-trained model on content similarity, and in a 360-person human Turing test is the only model picked over the real user above chance on chat (win rate .57). Its central finding is that content matching and human-likeness come apart. The authors flag that a simulator trained this way may learn the judge’s blind spots rather than real human-likeness.
Key points
- The Turing reward. The judge (Qwen3.5-397B-A17B at training time) is given the user’s history, the current context and both replies in random order, and scores 1–7 which is more likely the real user’s, across immediate target, human goal and communication style. The score is capped at 5 and normalized to [0, 1]: a reply judged more human than the real one is treated as likely reward hacking, which the authors say they saw in preliminary runs.
- Against matching rewards. Two baselines share the same SFT start, user representation and GRPO setup: Sim-RL, an LLM-judged similarity score adapted from HumanLM, and Logprob-RL, the per-token log-probability of the real reply after a chain-of-thought, from Gandhi et al. The similarity reward did not significantly improve the Turing score over SFT, so saying roughly what the user said does not make a reply indistinguishable.
- Setup. Qwen3-8B base with LoRA; SFT on real replies augmented with reasoning traces generated by Qwen3-8B, then GRPO with 4 samples per example on a disjoint split, plus a length penalty for replies outside a band around the real reply’s length. Users are represented by a held-out block of their past behaviour plus an LLM-induced persona; train and evaluation users do not overlap.
- Data. PRISM chat: 1,288 users with at least 6 conversations, 128 held out (880 target turns). ConvoKit Reddit: 1,282 users across 14 subreddits, with r/tifu and r/worldnews held out (102 users, 267 examples).
- Evaluation by a different judge. Claude Sonnet 4.6 scores Turing distinguishability, similarity to the real reply, and context/user specificity. Turing-RL leads on Turing score in both domains, by a wide margin on chat; GPT-5 and Qwen3.5-397B barely beat the 8B base model because they write “verbose, overly hedged responses that read as assistant-like”.
- Human Turing test. 360 Prolific participants make 600 binary choices per condition. Win rate against the real reply on chat: SFT .49, Sim-RL .50, Turing-RL .57; on Reddit: .41, .52, .50. Turing-RL beats SFT in both domains and Sim-RL on chat (p = 0.022); on Reddit it does not separate from Sim-RL, and the authors call the human study “a validation of broad trends rather than conclusive evidence”, since humans find Reddit harder to judge.
- The LLM judge is the better discriminator. On 50 users per domain, Sonnet 4.6 matches or beats human majority-vote accuracy at spotting the real reply in most conditions; both agree Turing-RL is the hardest model to tell apart.
- User representation barely matters for human-likeness. Turing scores are robust to history-only, persona-only or both; persona alone keeps style but loses content similarity, and larger persona-inducing models did not reliably help.
- Limitations the authors name: two open-ended domains only (no task-oriented dialogue, negotiation or collaboration), an 8B base model only, and dependence on the training judge’s biases (the simulator “may learn to exploit those blind spots rather than achieve genuine human-likeness”). The reward scores one reply at a time against the real one. They also flag impersonation risk.
Relevance
- Persimmon — the same idea at very different scale: both train a dedicated user model with RL against a discriminator instead of prompting an assistant to play the user. Turing-RL is the controlled, open version (8B, fixed judge, per-reply reward, ablated against matching rewards); Persimmon is the 550B product with an adaptive discriminator aimed at long-conversation drift. Read together they ask whether indistinguishability is the right objective, and how far it holds over many turns.
- Mind the Sim2Real Gap — that study found assistant simulators over-cooperative and verbose; Turing-RL’s frontier baselines fail the same way (“assistant-like rather than human-like”), and it reports a trained 8B model beating them. It does not report USI, so the two are not directly comparable.
- User Simulators — a fourth training signal next to prompting, profile grounding and log-likelihood: an explicit Turing reward. Its warning that the simulator can learn the judge’s blind spots is the same problem as a hackable verifier.
- ITSMBench — evaluated only on open chat and Reddit, not task-oriented service dialogue, so it is a method to try on a requester simulator rather than evidence it works there.
- WorldSmith — if simulators are trained against judges, the judge becomes part of the environment’s verification stack and inherits the same soundness questions.
Notes
(none)