The paper
A follow-up to Autoscaled Environments and the Sim2Real Gap, where Mind the Sim2Real Gap showed simulated users are an easy mode and we split on whether to use them at all. Both papers this time take the route the Sim2Real paper left open: instead of prompting an assistant to play a user, train a model whose reward is being hard to tell apart from real users.
Persimmon is the large version: a 550B “user model” mid-trained on public human conversations, then trained with RL against an adaptive discriminator to hold together over long conversations Persimmon. In a Multi-User Turing Test it fools an LLM judge 19.8% of the time (50% is parity) against 1.2% for its base model and under 1% for frontier models, but it stays coherent over 80 turns only 60.7% of the time, against 87.3% for humans Persimmon.
Turing-RL is the controlled version. An LLM judge sees a user’s history and two replies, the real one and the model’s, and rates which the real user wrote; that score, capped so the model cannot become “more human than human”, is optimized with GRPO on a Qwen3-8B simulator Turing-RL. On multi-turn chat and Reddit threads it beats similarity and log-probability rewards on human-likeness while matching the similarity-trained model on content, and in a 360-person human test it is the only model picked over the real user above chance on chat (win rate .57) Turing-RL.
Questions to bring:
- Same idea, different evidence. Turing-RL changes only the reward against matched baselines Turing-RL; Persimmon reports a product with no reward-only ablation but evaluates whole conversations, including coherence Persimmon. What would it take to trust either for an RL environment?
- The judge is the new verifier. Both find the LLM judge a sharper discriminator than people Turing-RL Persimmon, and the Turing-RL authors warn a simulator may learn the judge’s blind spots rather than human behaviour Turing-RL. Who checks the judge?
- Does this answer the Sim2Real result? The four specialized user-sim models in Mind the Sim2Real Gap all fell below the human baseline. Neither paper reports a USI comparison under the same contract — Persimmon’s own USI run uses a different prompting contract Persimmon.
- Control versus realism. Last time the case for simulators was tightly closed instructions. Both papers optimize for matching human variance instead Persimmon Turing-RL. Do the two approaches conflict, or answer different needs?
Discussion
(none)
What we took from it
(none)