Summary
AgentScaler scales function-calling ability by scaling environments. It starts from the abstraction that every tool call is a read or write over an environment database. More than 30,000 APIs (ToolBench, API-Gen and an internal repository) are embedded by their parameters, linked into a tool-dependency graph, and split into over 1,000 domains by Louvain community detection; each domain gets a generated database schema, and each tool is materialized as Python code that reads and writes it. Tasks come from directed walks over a domain’s tool graph: arguments are generated and actually executed against the database, so every task has a gold tool sequence and a gold final state. Simulated user–agent interplay produces trajectories, filtered by validity, final-state match and exact tool-sequence match, and the Qwen3-based AgentScaler models (4B, 8B, 30B-A3B) are trained with two-stage SFT (general domains, then vertical ones). AgentScaler-30B-A3B is competitive with trillion-parameter open models on τ-bench, τ²-Bench and ACEBench. There is no RL; the authors name it as the next step.
Key points
- Environment = tools + a database schema. Domains are found by clustering tools on parameter similarity, refined by an LLM checking pairwise dependencies; tools in a domain share a schema. On τ-bench domains, the generated schemas and code closely matched the official implementations on manual inspection.
- Task synthesis by graph walk gives two levels of verifiability: database-level state consistency and exact match of the tool sequence (used when a task has only reads, so state-based checks would pass trivially).
- Trajectories with tool errors are kept if they still reach the gold state, to teach recovery.
- Two-stage experience learning: broad tool-use skills first, then domain specialization; both stages help on ACEBench-en.
- Results: AgentScaler-30B-A3B scores 70.4/54.0 on τ-bench retail/airline and 70.2/60.0/55.3 on τ²-Bench retail/airline/telecom, near closed models in several domains; AgentScaler-4B matches 30B-class models. On out-of-distribution ACEBench-zh the 4B model’s agent score rises from 6.7 to 38.4.
- Long horizons stay hard: accuracy falls as the number of tool calls in a trajectory grows, for AgentScaler too; pass^k drops as k increases.
- Limitations named: SFT only (no RL), and validation only up to 30B scale.
Relevance
- WorldSmith — tool-graph clustering is a concrete way to get world and scenario diversity without hand-designing domains, and the gold-state plus exact-sequence filters are a verification design to compare against. It reports no cost and no measure of task difficulty beyond benchmark transfer.
- Environment Generation — tools → dependency graph → domains → schema → executable tools → graph-walk tasks; one of the first open descriptions of a tool-graph synthesis pipeline.
- Reinforcement Learning - RL — a negative space: the environments are built for RL, but training stops at SFT.
Notes
From paper club, 17 March 2026: AgentScaler creates tools, clusters the tools into a dependency graph, then creates scenarios accordingly. Their synthetic environment was capable of improving Qwen and other models, but it was still limited to SFT; we are expanding this to RL runs as well. We are taking world and scenario generation from AgentScaler into how we create synthetic post-training data at scale.