Summary
Attacks the cold-start version of the simulator problem: how to build a realistic user simulator for a specific business agent when no historical interaction data exists. SAGE grounds the simulator in two knowledge sources the business already has — top-down business logic (Big Five traits, demographics, and Ideal Customer Profiles: firmographics, decision-making authority, budget) and bottom-up agent infrastructure (product catalogs, FAQs, knowledge bases, inventory and customer tables, the same artefacts the agent under test retrieves from). It turns each declared agent capability into a user goal, hierarchically selects a relevant-but-unused piece of infrastructure knowledge, generates a coherent user profile, and composes the three into a task-scenario system prompt. Evaluated on a production RAG sales agent for a robotics company and a Shopify tool-augmented shopping agent (13 APIs), SAGE beats both ablations on lexical diversity and on human-rated human-likeness (2.58 vs 2.20-2.38 on a 3-point scale, human reference 2.90) and finds 25-33% more unique agent bugs (38.6 vs 29.0 and 13.6 on the RAG agent). Notably it also finds far more bugs than 150 sampled real customer conversations do (14.0), suggesting simulated interaction can be a better bug-finder than production traffic.
Key points
- The grounding is documents, not conversations: infrastructure knowledge is the same catalog/FAQ/DB the agent retrieves from, given to the simulator so its information needs match a customer who already read the website. Nothing here requires a human study or a transcript corpus.
- Task-scenario construction is fully mechanical and coverage-driven: loop over every declared agent capability, LLM-translate each into a user goal, cluster infrastructure knowledge and sample an unused piece from the relevant cluster, then generate a profile whose fields are chosen jointly to stay coherent (job depends on industry and company).
- Grading is LLM-as-judge on five dimensions (helpfulness, coherence, verbosity, relevance, faithfulness), 0-4; only turns scoring below 2 count as errors. Threshold validated by hand: 94% of 50 low-scoring turns contained real bugs, 4/50 high-scoring turns did.
- Bug extraction is validated against three human experts: two-stage LLM summarisation into unique bug statements achieves precision 0.73, recall 0.74, F1 0.73 against human bug lists, with inter-annotator Fleiss’ κ = 0.85 among the experts. Bottom-up knowledge is what lets the judge check answers against ground truth instead of its own priors.
- Ablations disagree about which half matters: removing infrastructure knowledge hurts most (bugs drop 38.6 → 13.6 on the RAG agent, human-likeness 2.58 → 2.20), removing ICPs hurts less (→ 29.0). For RAG agents, knowing the domain content beats knowing the persona.
- Simulators still fall short of humans on every diversity metric (rootTTR 11.80 vs 20.62, vocab 1,034 vs 1,360, distinct-1 0.21 vs 0.30) and on human-likeness — the gap this cluster keeps reporting — yet they surface more bugs, because a scripted goal loop probes capabilities that real users never happened to ask about.
- Real-traffic baseline built with outcome-driven filtering: a year of production logs, English only, kept to 3-15 turns because that band converts above 20% while 1-2 turn greetings and >15 turn off-topic sessions do not.
- Errors concentrate in “unspecific response” and “inaccurate answer”, and the vast majority appear after the first turn, which is the paper’s argument for multi-turn evaluation over static benchmarks. Generalisation checked across gpt-4.1, gpt-5, gemini-2.5-flash, Qwen2.5-7B, Llama-3.3-70B.
- Limitations the authors name: attributes are customer-facing business logic and would need rework for internal or back-office agents; one goal and one knowledge piece per scenario, so no topic switching within a session; and no direct comparison against real logs as a fidelity reference, which they flag as the cold-start setting’s inherent weakness.
Relevance
- ITSMBench — the one paper in this cluster whose grounding source is the environment’s own artefacts rather than conversation data, which fits V2 directly: the mocked ServiceNow/Entra/Governance catalogs, the ~309 operations, and the BPI-derived org structure are exactly “agent infrastructure knowledge”, and V1’s task list is exactly “agent capabilities” to loop over for coverage. It also gives a coverage argument for free — enumerate capabilities, generate one goal each — which is a cheaper first pass than deriving the task distribution from log frequencies. The ICP half maps poorly, though: it is B2C purchase-intent modelling, and the authors themselves say internal-facing agents need different attributes, so ITSM requester/operator personas would have to be rebuilt from role and process-variant data. Does not cover log grounding (explicitly a cold-start method), enterprise operators, or multi-agent.
- WorldSmith — the strongest automation evidence in this cluster: the whole pipeline from declared capabilities and existing business documents to scenarios, interactions, judged turns, and a deduplicated bug report runs without a human in the loop, and the human input is only the attribute schema and the infrastructure dump. It also gives a validated recipe for automated grading (score-below-2 threshold, two-stage bug summarisation) with measured agreement against experts.
- Benchmark Research — two design contributions: an LLM-judge threshold and bug-extraction pipeline validated against expert annotation with reported precision/recall, and the counter-intuitive finding that simulated interactions surface more agent defects than a sample of real production conversations, because real traffic under-covers the capability space.
- User Simulators — home concept since 2026-09-10
Questions it raises for ITSMBench:
- Extends the “minimum world a practitioner recognises” question in a useful direction: SAGE says the simulator should read the same knowledge base as the agent. If the V2 requester simulator gets the ITSM knowledge articles, service catalog, and org chart, does it start asking the questions a real requester asks, without any log grounding at all?
- Contradicts the working claim’s ordering: SAGE derives full task coverage from declared agent capabilities, not from a ground-truth distribution. If capability-loop coverage is this cheap, is log-derived frequency weighting needed for the benchmark, or only for the realism yardstick that checks it?
- Open question on grading: the bug-report output (unique, deduplicated, turn-linked defects) is a different product from a pass/fail score, and it caught faithfulness errors that binary end-state checks would miss. Is there a V2 track that reports discovered defects rather than a leaderboard number, and would that be more useful to ITSM vendors than pass
Notes
(Our take: to add after reading; drafted 2026-09-09, not yet discussed)