Summary
humans& releases Persimmon (v0.1, 10 September 2026), a “user model” built to simulate how people behave in multi-turn, multi-user chat, and pitched against using assistant models as simulators: frontier models’ outputs “vary much less than those of real people, representing only a sliver of the wider human distribution.” It is Nemotron 3 Ultra (550B, 55B active) mid-trained on public human conversations, then post-trained with RL on its own generations against an adaptive discriminator to fix drift over long conversations. Rather than score fixed traits, it is evaluated by distribution matching — a Multi-User Turing Test where an LLM judge tries to tell real continuations from generated ones — plus the User-Sim Index from Mind the Sim2Real Gap, a Trickle Test for when facts get revealed, and an 80-turn coherence test. Persimmon fools the judge 19.8% of the time on average (50% is parity) against 1.2% for its own base model and well under 1% for frontier models; it is the most precise at not volunteering facts early, but its long-conversation coherence (60.7% at 80 turns) is below the human 87.3%. Source is a blog post and model card, not a paper; access is a gated playground and API.
Key points
- A user model is not an assistant playing a user. It conditions on a scenario, per-user profiles and conversation history and outputs the next user turn; the goal is to match the distribution of human behaviour across and within people, not one plausible persona. Assistant-style guardrails are deliberately not the standard, since a uniformly compliant persona “could distort the behavior it is intended to simulate.”
- Training: mid-train then RL against a discriminator. Mid-training on diverse chat and forum data gives the human-like model; pure next-token prediction drifted into incoherence over long horizons, so post-training uses RL against an adaptive discriminator. That improved long-horizon coherence but “did not change noticeably on general distribution matching.”
- Multi-User Turing Test (k = 8): Persimmon 21.1% TIDES, 18.6% internal workspace, 19.8% TutorMoments; mean 19.8% vs Nemotron Ultra Base 1.2%, GPT 6 (Astra) 0.22%, Fable 5 0.17%, OSim 8B 0.06%. The judge is an LLM, and it is the stronger discriminator: it picks the real human continuation 90.6% of the time against 75.2% for a human panel. Persimmon’s prompting is the best found by prompt optimization.
- Giving model and judge a user profile raises Persimmon’s fooled rate on TIDES from 21.1% to 22.5–25.6% across four profile formats; frontier models were not run here because they already score near zero. Pangram flags 2.3% of Persimmon transcripts as AI (humans 1.3%) against 54.3% for GPT 6 and 94.3% for Fable 5.
- On USI’s four behavioural dimensions Persimmon scores 66.0 / 91.8 / 74.2 / 79.3 (style, information, clarification, error reaction), leading on all but clarification (Grok 4.6, 82.9). The authors decline to rank overall USI: Persimmon’s score omits evaluation alignment and uses a different prompting contract from the comparison models. They also name USI’s limits — fixed dimensions, brittle hand-written rules, fixed turn counts that can penalise a user who takes longer.
- Trickle Test: Persimmon has the highest precision of 43 models (88.5%) but lower recall (77.0%) than most. Assistant-tuned models show the reverse — high recall, lower precision — because “their tendency to give thorough, helpful answers degrades into oversharing when they play the user.”
- Coherence over 80 turns: humans 87.3%, Persimmon 60.7%, frontier models 95–99%. Frontier models are too coherent over long runs; Persimmon declines in a human-like shape but not yet at the human level.
- Limits named: text in and out, no distribution-matching evals in other modalities or languages, may invent details, drift from a profile or echo human biases; access is restricted to avoid impersonation and consequential decisions about real people.
Relevance
- Mind the Sim2Real Gap — Persimmon reports its USI, the metric introduced there. That paper found the four specialized user-sim models all fell below the human baseline, blamed on weak instruction-following; Persimmon is a specialized model built on a 550B base and leads the frontier models on three of four dimensions, but its USI is not run under the same contract, so the head-to-head is open. It also echoes that paper’s “front-loading” finding with the Trickle Test.
- User Simulators — a third route next to prompting an assistant and grounding it in real profiles (RealUserSim): train a dedicated model on human conversation. Its evaluations (Turing-style judge, Trickle, coherence) are reusable on any simulator.
- ITSMBench — a candidate requester simulator that should not volunteer ticket details early. Its USI run is on tau-bench customer service, but nothing is reported on IT-service or operator conversations, and it is only reachable through a gated API.
- WorldSmith — if simulators are trained models, environment construction inherits a model dependency and its biases; the coherence gap at long horizons bounds how long an episode can lean on one.
Notes
(none)