Summary
Stanford’s TRACE (Turning Recurrent Agent failures into Capability-targeted training Environments) improves an agent on a target environment by finding which capabilities it lacks and training each one separately. The base agent is rolled out; an LLM analysis agent first builds a dictionary of recurring capabilities from the trajectories, then labels each trajectory for each capability; capabilities are kept when their contrastive gap — how much more often they are lacking in failed than in successful trajectories — is at least 0.20 and they cover at least 10% of failures. For each retained capability an LLM generation agent synthesizes a verifiable environment that isolates it while keeping the target environment’s interface; a LoRA adapter is trained per capability with GRPO, and a mixture-of-experts router composes them. With Qwen3-30B-A3B-Instruct, TRACE reaches 48.2% on τ²-Bench (+15.3 over base) and 41% Pass@1 on SWE-bench Verified (+15), beating GEPA and GRPO baselines with under a quarter of the rollouts.
Key points
- A capability is one or more actions necessary to solve some subset of tasks — e.g. retrieving the correct customer record in customer service, or locating the relevant file in a coding environment.
- Two-phase identification for consistency. Discovery induces a canonical dictionary of candidate capabilities from tool calls, results and responses; labeling marks each trajectory per capability. Analysis (bar discovery) is repeated and only consistently selected capabilities are kept; one pass takes about 55 minutes.
- The contrastive gap beats failure-only analysis. It “filters out capabilities with uniformly low success rates, which often reflect task ambiguity or annotation noise rather than a trainable deficit,” and drops capabilities that fail occasionally but do not separate success from failure. Thresholds: gap δ = 0.20, failure coverage ρ = 0.10.
- One environment per capability. Each synthetic environment has a seeded task generator, transition logic and automatic evaluation whose reward depends primarily on exercising that capability, giving a denser, more attributable signal than the target environment’s outcome reward.
- Adapters, then routing. One LoRA per capability (2–4 adapters, each 5.3% of parameters), trained with GRPO; a learned mixture-of-experts over the adapters selects capabilities per task.
- Results. τ²-Bench 48.2% overall (+15.3 over base, +8.6 over GEPA); SWE-bench Verified 41% Pass@1 (+15 over base, +8.4 over SWE-RL); with Qwen3.6-27B, 73.2% on SWE-bench Verified.
- Training beats prompting. Training on the capability environments gives +8.6 on τ²-Bench and +10 on SWE-bench Verified over adding the same capabilities to the prompt.
Relevance
- WorldSmith — a closed loop from observed failures to synthesized environments: the environment-supply question asked from the demand side (which environments does this agent need?).
- Environment Generation — capability-targeted synthesis as an alternative to scaling untargeted synthetic data such as AutoForge and ScaleEnv, which the paper criticizes for spending compute on capabilities the model already has.
- Tau2-Bench — the main customer-service target.
Notes
From paper club, 22 April 2026: TRACE primarily focuses on capability-based task mining — run a model on a large set of existing tasks, label each trajectory with the capabilities it required, cluster the descriptions into categories, select capabilities that show up in 10%+ of failed trajectories, and compute the contrastive gap. The contrastive gap is an improvement over regular error analysis (just looking at failures) since it reduces the number of false-positive capabilities, and it helps distinguish model capability gaps from harness or environment problems. It’s an answer to one of the two counterpoints we hear most about autonomously scaling post-training data: how do you ensure you’re mining tasks the model genuinely can’t complete?