Summary
CRMArena-Pro (Salesforce) populates two live Salesforce Orgs — one B2B (29,101 records) and one B2C (54,569) — with LLM-generated synthetic data grounded in the merged Service Cloud, Sales Cloud and CPQ schemas, using 21 latent variables to induce implicit causal structure across 25 objects. It defines 19 expert-validated tasks grouped into four business skills (Database querying, Text retrieval, Workflow execution, Policy compliance), 100 query instances each per Org, plus 80 confidentiality probes per Org, for 4,280 queries total. Agents act through SOQL and SOSL only, and are graded by exact match or token-F1 against a reference answer, with an LLM judge used solely for the confidentiality refusals. Leading agents score ~58% single-turn, falling to ~35% multi-turn against an LLM user simulator that releases task-relevant information incrementally; confidentiality awareness is near zero under a standard prompt, and prompting for it costs task completion. Realism was checked by recruiting CRM professionals to work in the sandbox: 66.7% (B2B) and 62.3% (B2C) rated the data realistic or highly realistic.
Key points
- The multi-turn drop is engineered, not incidental: the domain knowledge a task needs (e.g. the definition of handle time, the case-routing policy) is handed to the agent in the query in single-turn mode, and moved into the simulated user’s head in multi-turn mode. The 58% → 35% gap therefore measures information elicitation specifically, not conversation in general.
- User simulator design: an LLM with a randomly sampled persona (“you are quality-focused…”) plus a system prompt instructing incremental release of task information. Fidelity was checked by human inspection of 20 sampled trajectories, finding 1 error (5%). In failure analysis of 20 gemini-2.5-pro failures, 9 were the agent not obtaining the necessary information, and only 1 was simulator error.
- Models that ask for clarification more often perform better in multi-turn — the distribution of clarification counts shifts right with overall performance. This is a usable behavioural signal, not just an outcome metric.
- Expert validation is done on the environment, not the tasks: professionals recruited through User Interviews were oriented in the Org, made to complete five sampled query instances hands-on via the GUI, then rated realism against their own real systems with written rationales. This is the closest published analogue to the practitioner-realism yardstick, and it is a 2/3-approval result, not a ceiling.
- Data generation is synthetic-but-schema-grounded: gpt-4o generates against real Salesforce schemas, 21 latent variables control company and record properties to create implicit causal relationships, then de-duplication, format verification, and rule-based plus LLM content-plausibility checks.
- Skill decomposition is the reporting unit rather than task-level score. Workflow Execution is the tractable skill (gemini-2.5-pro exceeds 83% single-turn in both Orgs) while Database, Text, and Policy lag — the same shape as ITSM benchmarks where procedural execution is easier than constraint reasoning.
- Confidentiality is a separate axis with its own 80 queries per Org across three kinds (other customers’ PII, internal-only metrics, proprietary company knowledge), where the correct behaviour is refusal, judged by gpt-4o. Standard prompts give ~0–2% refusal; a confidentiality-aware prompt raises it to 24–63% for proprietary models but drops task completion by up to 11 points, and works less well multi-turn as the guideline loses salience.
- B2B vs B2C is a controlled pair over the same task set, which surfaces an interaction: stronger models do slightly better on the larger B2C Org, weaker ones do better on B2B, plausibly a long-context effect.
- Limitations the authors name: the user simulator is imperfect (5% observed error, no comparison to real users), open-source models barely respond to the confidentiality prompt (instruction-hierarchy weakness), and grading is answer-string comparison rather than state verification — most tasks are read-only queries.
Relevance
- ITSMBench — the cleanest published demonstration of the design V2 wants for the user simulator: move the domain knowledge out of the prompt and into the requester, and the same tasks become an elicitation benchmark with a 23-point drop. Also supplies a persona-sampling recipe and, uniquely among these papers, a practitioner-realism protocol (hands-on tasks then a rated comparison against real systems) that maps directly onto ITSMBench’s second yardstick. It does not cover log grounding (data is gpt-4o-generated from vendor schemas), multi-agent tracks of any kind, or state-change verification — the agent has read-only SOQL/SOSL access and is graded on its answer string, so V1’s DB-state verifiers have no counterpart here.
- Web Agents — the environment is a real Salesforce Org accessible by both GUI and API; the expert studies drive it through the GUI even though the benchmarked agents use APIs.
- Benchmark Research — skill-level rather than task-level reporting, the single-turn/multi-turn ablation as a controlled measure of information elicitation, an orthogonal safety axis (confidentiality) with an explicit performance trade-off table, and B2B/B2C as a matched-pair robustness check.
- User Simulators — home concept since 2026-09-10
Questions it raises for ITSMBench:
- Their realism check gets 62–67% expert approval on synthetic-but-schema-grounded data. If BPI-grounded data does not beat that number, what exactly did the logs buy — and is expert approval rate the right way to score the coverage yardstick?
- Moving the routing policy into the user’s head is what creates their multi-turn gap. In ITSM the requester usually does not know the policy — the agent does — and what the requester holds is symptom detail, environment context, and impact. Does the same ablation shape survive when the withheld information is diagnostic rather than procedural?
- Clarification frequency correlates with success here, but Tau2-Bench found personas barely matter and Mind the Sim2Real Gap found simulators too cooperative. Is “asks more questions” a real capability signal, or an artefact of a simulator that always answers helpfully?
Notes
(Our take: to add after reading; drafted 2026-09-09, not yet discussed)