Summary
Tongyi Lab (Alibaba) proposes a pipeline that builds simulated tool-use environments and hard, easily verifiable tasks starting from nothing but tool-description documents, plus an RL algorithm for training across many such environments. An LLM writes the state schema (a key–value database) and Python implementations of each tool; tools become nodes in a dependency graph, random walks over it yield tool sequences, and those are merged, de-duplicated and enriched with reasoning nodes and edges into a DAG that serves as the task blueprint. Tasks are verified by the final environment state rather than the exact tool sequence, since several call orders can be valid. ERPO, their GRPO variant, uses an LLM judge to mask trajectories that failed because the simulated user erred, and normalizes advantages per environment instead of per group. From 10 synthetic environments and 1,078 tasks, a 30B-A3B model beats its backbone across τ-bench, τ²-Bench and VitaBench and generalizes to an unseen Chinese benchmark.
Key points
- Input is only tool documentation. An LLM produces the state structure (attribute names), then Python tool code conforming to it; attribute values are filled in later, at task time, so tasks and environment states stay consistent.
- Tool graph → random walks → merged DAGs. A directed edge links two tools when an LLM judges one’s output a likely input to the other (motivated by AgentScaler). Random walks give thousands of sequences, which are merged, stripped of redundant calls, and augmented with reasoning nodes (e.g. computing a total price from earlier outputs) and reasoning edges to form a DAG blueprint for each task.
- Verification by final state. The task is executed on the instantiated initial state to produce a golden final state; reward compares the rollout’s final state with it, because “a single task may be solvable through multiple valid sequences of tool calls.” The task text is then refined to contain only the minimal information needed.
- ERPO: masking simulated-user errors. During RL an LLM judge flags trajectories where the simulated user hallucinated or omitted information, and these are excluded from advantage and loss; without masking, the reward curve trends downward late in training.
- ERPO: environment-level advantage. Advantages are normalized within each environment rather than each group, which gave a more stable and higher reward curve.
- Scale and results. Qwen3-Thinking-235B synthesizes 10 environments with 1,078 tasks; Qwen3-Thinking-30B-A3B is cold-started with SFT then RL. AutoForge-30B-A3B scores 74.8 / 62.0 / 76.3 on τ²-Bench retail / airline / telecom against the backbone’s 58.8 / 58.0 / 26.3, the best open model under 200B in their comparison.
- Out of domain. On ACEBench-zh — different multi-turn format, different tool-call format, unseen tools, and Chinese where all training data is English — both SFT and RL versions beat the backbone, RL more.
- The simulated user moves the score. Swapping the evaluation user from GPT-4.1 to GPT-5-thinking raised τ²-Bench results, most in telecom, where the user also calls tools.
- Limitations the authors name: still requires tool documents as input; only a few environments were used; ERPO is outcome-reward only.
Relevance
- WorldSmith — a full construct pipeline from tool docs to verified tasks, with state-diff verification, the design shape WorldSmith wants agents to run. It reports no inspection or verifier-soundness step beyond executing the blueprint, and only 10 environments.
- Environment Generation — tool-graph random walks plus reasoning-node augmentation as a recipe for task difficulty; a direct successor to AgentScaler and the baseline ScaleEnv compares against.
- User Simulators — masking trajectories ruined by simulated-user error is a training-side answer to the unreliable-user problem in Mind the Sim2Real Gap.
- Reinforcement Learning - RL — environment-level advantage normalization for multi-environment agentic RL.
Notes
From paper club, 7 April 2026: TongyiLab’s AutoForge and ScaleEnv directly address the scaling problem with Tau-Bench and EnterpriseOps-Gym by proposing that the environments and all of their data should be synthetic. Both follow roughly the same recipe: start from seed API docs or a tool schema, build a tool dependency graph, take random but valid walks across it, remove duplicate walks and merge ones that could form a longer toolchain, synthesize the DB state and variables that make the walk possible, and derive tasks from the trajectory.