Summary
AppWorld builds a simulated digital world of 9 apps (Gmail, Venmo, Amazon, Spotify, phone, file system, …) exposing 457 REST APIs over a 101-table, 726-column database of ~370K rows, populated procedurally with 106 fictitious people whose relationships (roommates, siblings, coworkers) and two years of app activity are sampled consistently. Agents work in a stateful Jupyter-style shell writing real code against those APIs; time is frozen and the DB reset per task for reproducibility, and the engine ships 1,780 unit tests at 98% coverage so it responds correctly to arbitrary agent exploration. The benchmark is 250 task scenarios × 3 task variants = 750 tasks, each generated by a hand-written Task Generator with three programs — Setup, Evaluation, and a Validation Solution whose successful run is an end-to-end proof that the task is solvable and the evaluator correct. Grading is state-diff-based: the evaluator computes a hash-based diff of the final vs. initial DB and asserts it contains all expected changes and no changes beyond an explicitly allowed set — i.e. collateral damage is caught by construction rather than enumeration. The best agent, ReAct + GPT-4o, reaches 48.8% TGC on the normal test split and 30.2% on the challenge split, with scenario-level completion (all three variants of a scenario) 30–50% lower again.
Key points
- Two metrics, and the second is the interesting one: Task Goal Completion (passed all tests on a task) and Scenario Goal Completion (passed every task in the scenario). SGC is a consistency metric in the contrast-set sense — 48.8 TGC becomes 32.1 SGC — measuring whether an agent achieves the goal reliably across varying conditions, not once. It is a cheaper cousin of pass^k that varies the world instead of the seed.
- State-diff verification with an allow-set is the sharpest verifier design in this cluster: expected changes must be a subset of the diff, and the diff must be a subset of expected ∪ allowed. Everything else the agent touched is a failure automatically, which is how they check “collateral damage” (deleting a wishlist, initiating an unasked return) without enumerating it. Each assertion carries a paired natural-language statement, which the authors flag as a hook for future human-feedback simulation.
- Setup programs enforce four properties per generated task: well-defined (all presuppositions hold in the task DB), has distractors (multiple similar past orders so the agent must find the right one), has realistic hurdles (the default card has expired), and forms a contrast set with the scenario’s other variants (item available in preferred colour in one, not in another). Distractors and hurdles are designed in, not incidental.
- The Validation Solution is a programmatic gold solution that must pass the evaluator from the task’s start state — a regression test that keeps tasks solvable as the apps evolve, and simultaneously verifies the evaluator itself. It uses privileged internal knowledge (regexes over how a note is formatted) that the agent would have to obtain by interacting.
- Base DB population goes through the tested APIs rather than raw SQL, so consistency is enforced by the app logic (placing an order clears the cart, sends a confirmation email); the tooling can invoke APIs “in the past” for temporal realism. ChatGPT is used sparingly, only for entries where precise semantics do not matter.
- Test-C is a designed contamination and memorization guard: every task in it requires an API from a designated unseen app (Amazon, Gmail), so an agent cannot replay memorised action sequences and must read documentation. It is the paper’s primary target, and scores roughly halve on it.
- Oracle-API ablation shows retrieval is not the bottleneck (+5 to +10 TGC with oracle APIs, and the API predictor already scores 87/71 F1); difficulty comes from interactive multi-step code and error recovery. Difficulty also tracks hand-labelled level, lines of solution code, and unique API count — tasks needing 60+ lines score under 20 TGC.
- Error taxonomy: hallucinating instead of interacting to find a fact, misusing APIs (wrong field,
sent_payment_requestsvsreceived_payment_requests), partial instruction following (approving all requests when told only roommates’), commonsense slips, and forgetting prior actions and repeating work until the call budget is gone. - Cost and limitations: built entirely by the authors over five months, 40K+ lines of task-generator code; no UI modality (APIs only, though the authors note state-based evaluation would survive adding one); explicitly single-supervisor and single-agent, with multi-agent coordination named as possible-in-the-engine but unbuilt; and too few instances to train on.
Relevance
- ITSMBench — the verifier design to steal: expected-changes ⊆ diff ⊆ expected ∪ allowed turns V1’s “mandatory preservation controls” into a general principle, and the paired NL statement per assertion is a natural bridge to V1’s hybrid DB-state × NL-assertion grading. Its Task Generator triple (Setup / Evaluation / Validation Solution, with the solution proving both solvability and evaluator correctness) is a concrete template for authoring V2 tasks, and SGC — pass every variant of a scenario — is a discrimination-and-stability metric that varies the world rather than the sampling seed. It does not cover log grounding (the world is procedurally sampled fiction), has no user simulator (the Supervisor is an API you query for facts, not an interlocutor), and no multi-agent track — the authors state it is single-supervisor by design and leave coordination as future work.
- WorldSmith — populating the world through tested APIs rather than raw inserts is the key trick for keeping a generated world internally consistent, and the Validation Solution is the mechanism that makes generated tasks self-verifying. Against that, five months and 40K lines for 750 tasks is the human cost of doing it this way.
- Benchmark Research — contrast sets and a scenario-level consistency metric, a held-out unseen-app split as a memorization control, an oracle-retrieval ablation, and hardness indicators (solution LOC, unique APIs) computed from gold solutions rather than assigned by hand.
Questions it raises for ITSMBench:
- The allow-set makes “no unintended side effects” checkable without enumeration, but it requires knowing what a correct solution may touch. In ITSM, real resolutions touch work notes, SLA clocks, and related records unpredictably. Can BPI traces supply the allow-set empirically — everything real resolvers touched on that variant?
- SGC across three variants of one scenario is a cheap reliability measure. If V2 instantiates a task from several BPI cases of the same process variant, does variant-level completion become the natural analogue, and does it correlate with pass
- Their world is consistent because it was written through tested APIs. BPI logs are records of a world whose API we do not have. Does rebuilding the ITSM world from logs require first writing the API that would have produced them — and is that where most of the V2 effort actually goes?
Notes
(Our take: to add after reading; drafted 2026-09-09, not yet discussed)