Summary
WildToolBench (Tencent) argues that “what truly challenges LLMs’ tool-use capabilities is not artificially constructed complex scenarios, but simple yet realistic user behaviors.” From a large-scale analysis of real user logs it extracts three: compositional tasks that need an efficient tool-call topology, intent hidden across earlier turns, and instruction transitions where task queries mix with clarifications, follow-ups and chat. The benchmark has 256 scenarios, each a multi-turn dialogue with four tasks (1,024 tasks), over a cleaned set of about 1,600 public APIs; trajectories are generated by a multi-agent user/assistant system and every tool call is manually annotated as ground truth. Across 57 evaluated LLMs, none exceeds 15% session accuracy and most fall below 60% task accuracy.
Key points
- Realism from behaviour, not complexity: seed scenarios are sampled to follow the distribution of real logs, using log samples as few-shot examples without leaking user data.
- Three challenges: tool orchestration for compositional tasks (sequential, parallel and mixed topologies), inferring hidden intention through dialogue, and instruction transition across task types.
- Evaluation is against ground-truth tool calls. An enumerate–match–score pipeline enumerates every legal execution path from expert-labelled tool dependencies, then incrementally matches the model’s calls against that decision tree; metrics include task accuracy, session accuracy (all four tasks right), optimal-path rate and progress.
- Results: no model above 15% session accuracy; proprietary beat open-source, reasoning models beat non-reasoning ones, and specialised tool-use models do worse than general models.
- Limitations named by the authors: human annotation limits scale, and keeping quality while covering all policy-transition types limits task length.
Relevance
- Benchmark Research — evidence that prior tool-use benchmarks are saturating and that realistic user behaviour, not tool count, is what still separates models.
- User Simulators — takes the opposite route to Tau2-Bench and VitaBench: no live LLM user at evaluation time, citing simulator flawlessness and instability.
- Benchmark Validity — matching against enumerated golden tool paths is precise but can only credit the paths the annotators anticipated.
Notes
From paper club, 9 June 2026: WildToolBench matches tool names against a golden tool-chain, where VitaBench uses rubrics; in our use case a hybrid could make sense, since rubric-only loses verifiability and state-matching alone can’t capture subgoals. A concern we have in scaling it for our purposes is that it uses pre-written user responses instead of a live simulator, which probably won’t scale as well for RL training. Matching tool names against a golden chain is also a limitation for anything with multiple valid solution paths.