The paper
A deep dive into enterprise environments and benchmarks like τ²-Bench and CoreCraft. This type of high-fidelity RL environment is becoming increasingly popular as frontier labs push their models into more and more agentic capabilities.
Discussion
Right now, however, the largest bottleneck is human data operations.
For the majority of these environments, time-intensive human labour has been necessary so far to create the world, the synthetic data inside of it, the tasks, and the verifiers needed to do an RL run. CoreCraft, for one, credits domain experts with designing its tasks and rubrics, annotators with populating its entities, and expert evaluators with reviewing trajectories EnterpriseBench Corecraft.
While this has resulted in high-quality data, the human bottleneck is inherently limiting and prevents us from scaling up the number of environments and tasks that could train a model to the fullest extent.
Papers like Snowflake’s AWM and AutoEnv have explored creating synthetic data using LLMs to generate their own environments, tasks, and evaluation logic given some basic context Agent World Model AutoEnv. So far, these methods seem to be limited — the environments they produce are simpler than human-designed ones like τ²-Bench and CoreCraft. AutoEnv’s environments average about six actions and 470 lines of code AutoEnv, and frontier models pass 36–63% of AWM’s synthesized tasks Agent World Model against 30.8% at best on CoreCraft EnterpriseBench Corecraft.
What we took from it
- Our goal is to recreate something as complex and high-fidelity as CoreCraft, but completely synthetically (WorldSmith, Enterprise Environments).
- When we can successfully recreate the complexity of environments like CoreCraft without needing extensive human input, we’ll unlock the next stage of model improvement (Environment Generation).
