Summary
The follow-up to SAGE from the same group, aimed at two gaps: simulation frameworks have almost no way to say whether a simulator sufficiently exercises the agent under test, and they are restricted to either UI actions or API actions but not both. VISTA contributes six metrics — three coverage measures computed from the product agent’s own tool-call traces (transition entropy, tool-distribution entropy, within-goal trajectory distance), an LLM-judge realism triple (utterance consistency with goal, profile, and domain knowledge), a cost pair (tokens and actions per turn), and a unique-failure count — plus a hybrid simulator that routes each step between a real browser UI action (via UXAgent’s DOM connector) and a user-side API tool. Across an e-commerce Shopify agent (13 APIs) and a production education support agent, 200 interactions per model per domain with gpt-5.4 and Qwen3.5-27B, the hybrid simulator beats UI-only on coverage by up to 10% and finds up to 42% more unique agent failures at comparable cost. The realism metrics correlate with human ratings at Spearman ρ=0.60, and higher coverage tracks more failures found, so the entropy metrics work as a cheap proxy for evaluation power without running an LLM judge.
Key points
- The coverage metrics are the novel part and they are measured on the agent under test, not the user: transition entropy (normalised Shannon entropy of the tool-to-tool transition matrix), tool-distribution entropy (how evenly the N tools are exercised overall), and trajectory distance (mean pairwise normalised Levenshtein distance between tool-call sequences within the same user goal, over names and serialised arguments).
- Trajectory distance is the one that catches redundancy: the first two entropies can look high while every run of the same goal follows an identical path. Reported values expose real differences — 0.94-0.97 in shopping vs 0.62-0.68 in education, where the agent’s troubleshooting flow is rigid.
- Coverage predicts failure discovery. The hybrid simulator scores higher on all three coverage metrics in nearly every setting and finds substantially more unique failures (education, gpt-5.4: 24 → 34; shopping: 69 → 79), so entropy can substitute for the expensive LLM-judge bug pipeline as a screening signal.
- Realism is measured reference-free as consistency of each utterance with its three prompt components (goal / profile / knowledge), scoring 96-100% — near ceiling, and far above what human annotators give the same conversations (human-likeness 2.04-2.26 on a 3-point scale). The automatic score tracks human ratings in rank order (ρ=0.60) but not in level.
- Human blind test on the education agent: 225 conversations from the hybrid simulator, the UI-only simulator, and real customers, annotators guessing human / bot / unsure. The hybrid simulator is 6% more likely to be taken for a human than UI-only. Annotator agreement is weak (Cohen’s κ = 0.45 on the Likert study).
- Task-specific user tools are implemented as mocked APIs with structured status fields, real database schemas in JSON, and error messages copied from the UI — explicitly so that large-scale testing does not need production databases, live interfaces, or hand-configured test accounts. Error paths are enumerated (rescheduling within an hour of the exam, unsupported country, ineligible test type).
- The case study is the sharpest argument: the UI-only simulator, having navigated login and account pages to find its order number, accumulates account details in context and volunteers them unprompted — so the shopping agent completes the return and a real failure (it should have asked for verification) is never triggered. The API-based simulator gives only what a real user would know, and the bug surfaces. A simulator with too much context is an easy mode in a second, mechanical way.
- Setup: scenarios follow SAGE (profile + goal + domain knowledge), goals designed to cover the product agent’s capabilities; action space is click, type, select, key_press, clear, scroll, navigate, terminate; loop runs observation → LLM router (API tool or UI action) → action until terminate or turn budget.
- Limitations the authors name: it inherits LLM behavioural biases and they cite the over-cooperativeness findings directly; two customer-facing domains only, with enterprise productivity named as untested; the metrics measure coverage and interaction quality, not deployment success; and hybrid routing adds real integration overhead.
Relevance
- ITSMBench — supplies the missing measurement for V2’s coverage yardstick in a form that needs no logs and no humans: run the benchmark, take the agent’s tool-call traces, and compute transition entropy, tool-distribution entropy, and within-goal trajectory distance over the ~309 mocked vendor operations. Trajectory distance in particular answers “are my 53 tasks actually 53 different tasks, or the same path 53 times”, which is the failure mode a hand-built task set is most prone to. The hybrid design also matches V2’s plan directly — an ITSM requester who both talks to the agent and acts in the portal (files a ticket, uploads a screenshot, approves a change) is a dual-control user in the Tau2-Bench sense, and VISTA gives an implementation. What it does not cover: no log grounding at all (coverage is measured against the tool inventory, not against process-variant frequency in BPI, which is the coverage V2 actually promised), no enterprise operator simulation, and no multi-agent.
- WorldSmith — the metric suite is fully automatic and cheaper than LLM judging, so it is a candidate automated stopping criterion for a generated benchmark: keep generating scenarios until coverage entropy and trajectory distance plateau. The mocked-API pattern (real schemas, real error strings, enumerated failure branches, no production system) is also a reusable recipe for building the user side of an environment.
- Benchmark Research — six general-purpose simulator-quality metrics that are not tied to one benchmark, with evidence that the cheap ones (entropy over tool traces) predict the expensive one (unique failures found), plus a mechanical demonstration that giving the simulated user more context than a real user would have silently removes failure modes from the benchmark.
- User Simulators — home concept since 2026-09-10
Questions it raises for ITSMBench:
- Extends the coverage yardstick with a concrete formula, but over the wrong space: VISTA’s coverage is over the agent’s tool inventory, V2’s promised coverage is over BPI process variants and roles weighted by frequency. Can trajectory distance be computed over log traces to get the empirical target — the real spread of activity sequences within a case type — so V2 measures its own coverage against that number rather than against a uniform ideal?
- Open question the case study forces: how much context does the V2 requester simulator get? If it holds the whole ticket record it will never fail to answer a verification question, and every “agent skipped verification” bug disappears from the benchmark. What is the principled information budget for a simulated ITSM requester, and does BPI say anything about what requesters actually know?
- Contradicts the automatic-realism story in this cluster: VISTA’s realism scores sit at 96-100% while its human ratings sit near 2/3 and its own limitations section cites Mind the Sim2Real Gap. Prompt-consistency judging measures obedience to the scenario, not human-likeness. If V2 uses an LLM judge for simulator quality, it needs a metric of the Sim2Real or decision-fidelity kind, not this one.
Notes
(Our take: to add after reading; drafted 2026-09-09, not yet discussed)