The paper
A deep dive into one of the hottest topics in our industry: world models.
In the past few months, companies like Simile (SiliconANGLE) and World Labs (Yahoo Finance) have raised huge sums of money to build out world models capable of complex simulations.
This week, we read and discussed APEX-Agents, CoreCraft, and Agent World Model (AWM) to better understand how we can leverage world models to automate environment and task creation.
Mercor’s APEX and Surge’s CoreCraft are benchmarks testing how well SOTA models can operate in realistic enterprise environments, doing human-created long-horizon tasks in fields like investment banking, consulting, corporate law and enterprise customer support APEX-Agents EnterpriseBench Corecraft.
Snowflake’s AWM takes a different approach: instead of handcrafting these environments and tasks, it attempts to mass-scale environment creation by feeding an LLM a high-level scenario (see examples below) and instructing the LLM to use the scenario to generate its own tasks, then to build out the necessary components of a world (a database, APIs, reward logic, etc.) from those tasks Agent World Model.
Discussion
A potential limitation with this approach is that if the initial scenario is too unsophisticated, the tasks the LLM generates will be similarly unexceptional. Even with large environment diversity, AWM’s tasks may not be complex enough to meaningfully challenge a state-of-the-art model.
To successfully automate the creation of high-quality post-training environments and tasks, we’ll make use of a combination of these approaches: we need to build a sufficiently complex world model so that it can create tasks as complex as the human-generated ones in APEX and CoreCraft.
In practice, making agents better at long-horizon tasks will depend highly on how well we can scale the worlds that the models act within.
What we took from it
- Human-built worlds set the bar for task complexity; LLM-built worlds set the bar for scale. Automating environment creation means getting the first from the second (World Models, Environment Generation).
- A synthetic world is only as rich as the scenario it starts from, so scenario quality is where an AWM-style pipeline needs the most work (Agent World Model, WorldSmith).
- Scaling long-horizon agent performance depends on scaling the worlds agents act in (Enterprise Environments).