Summary
A multi-turn tool-use benchmark from Meituan’s LongCat team built around the question “what constitutes task complexity for agents in real-world applications?” It splits complexity into reasoning (how much environmental information must be integrated), tool (tool sets modelled as dependency graphs) and interaction (user behaviour across a long conversation). Three life-service domains — food delivery, in-store consumption, online travel — share 66 tools whose dependency graph encodes the policy, so there is no τ-bench-style policy document and scenarios compose freely into 100 cross-scenario and 300 single-scenario tasks. Each task is derived from several real user requests and carries its own user profile, spatiotemporal context and service database; a rubric-based sliding-window evaluator scores long trajectories. The best model reaches 48.3% on single-scenario tasks and 30.0% cross-scenario.
Key points
- Complexity has three measurable axes: reasoning, tool (node count and edge density of the tool graph) and interaction; the authors report strong correlations between these metrics and task difficulty.
- Policy lives in the tool graph, not a document. Inter-tool dependencies encode the domain rules, which lets agents explore rather than follow a τ-bench policy and makes cross-scenario composition possible.
- Tasks are grounded in real requests and real profiles. Each task combines multiple authentic user requests; user profiles come from anonymised platform data (attributes such as age or dietary restrictions), and agents must infer some attributes from history or dialogue rather than read them.
- Distractors by construction: target options that satisfy every constraint are mixed with options that violate specific ones, giving large search spaces with only a handful of valid answers.
- Rubric-based sliding-window evaluator: hand-written atomic rubric items per task, a persistent satisfied-state vector carried across trajectory windows, all-or-nothing scoring for the benchmark, with the per-item rubric offered as dense signal for RL.
- The user simulator follows τ-bench and runs on gpt-4.1.
- Failure analysis: reasoning errors dominate (61.8%), then tool-use errors (21.1%) and interaction-management failures (7.9%); agents show poor self-awareness and limited error recovery.
Relevance
- User Simulators — a concrete recipe for grounding a simulator: full user profile and multi-request task written from the user’s side, with knowledge boundaries on what the agent may see.
- Benchmark Research — a tool-use benchmark where difficulty comes from reasoning over constraints rather than from tool count; a successor to Tau2-Bench as that saturates.
- ITSMBench — profiles, distractor values and rubric-per-task evaluation are directly reusable design choices for a requester-facing ITSM benchmark.
Notes
From paper club, 9 June 2026: two design choices we want to adopt for our tool-use work. (1) Write tasks from the user’s perspective with a full user profile attached; this additional context in world-building helps the agent follow realistic patterns, and if you give a generic task description to the user simulator it starts leaking the answer to the agent rather than behaving like a real user. (2) Inject distractor values into the env so agents can’t immediately identify the correct next step. Against WildToolBench: VitaBench uses rubric-based evals, and for our use case a hybrid could make sense, since rubric-only loses verifiability and state-matching alone can’t capture subgoals.