Summary
AutoEnv treats an environment as a factorizable distribution over transitions, observations and rewards, and builds it in three code layers: BaseEnv (dynamics and reward), ObsEnv (what the agent can see) and SkinEnv (how it is rendered, as text or images). From a theme, an LLM writes a description, then a YAML DSL; coding agents implement the three layers, a level generator and a validator, and repair them in a loop until tests pass. Verification has three stages — execution, level generation, and a reliability check by differential model testing: if a weaker model (GPT-4o-mini) consistently beats a stronger one (DeepSeek-V3.1), the reward is treated as unreliable and the environment discarded. From 100 themes, 65 pass all stages at $4.12 per environment; 36 form AutoEnv-36 (358 levels), where seven models score 12–49% normalized reward. The second half of the paper studies cross-environment learning and finds that no fixed learning method scales across heterogeneous environments.
Key points
- Three layers let the same rules wear different skins. Changing ObsEnv varies observability; changing SkinEnv varies presentation while dynamics stay fixed. AutoEnv-36 includes “inverse” environments whose semantics deliberately contradict the rules (“poison restores health”).
- Generation is DSL first, code second. The YAML DSL fixes dynamics, observation policy, rendering and level generation before any code exists; coding agents then implement and self-repair.
- The reliability check is the transferable idea: a reward that ranks a weak model above a strong one is close to random. It is the stage that filters most (74.7% pass vs 90.0% execution and 96.7% level generation).
- Human review of the theme helps a lot: overall success 60% for purely LLM-written themes vs 80% for human-reviewed ones, mostly by removing rule inconsistencies.
- AutoEnv-36 separates models: o3 48.7%, GPT-5 46.8%, GPT-4o-mini 12.0%. Binary rewards score higher than accumulative ones, full observability higher than partial.
- Learning does not transfer for free: any single learning method’s gain shrinks as environments are added; choosing the method per environment helps, with diminishing returns as the method space grows.
- Environments are small (6.1 actions, ~470 lines of code on average) and game-like rather than enterprise tools.
Relevance
- WorldSmith — the differential-model reliability check is a cheap, automatable verifier-soundness gate worth comparing against WorldSmith’s own; the human-review result (60% → 80%) is a measured cost of keeping a person in the loop.
- Environment Generation — theme → DSL → layered code → three-stage verification, at a reported cost per environment.
- Open-Endedness — framed as a testbed for agents that learn across many heterogeneous worlds rather than within one.
Notes
From paper club, 17 March 2026: in AutoEnv, the LLM creates a YAML config, then uses it to create action spaces, observation spaces and rewards in a self-running loop. Their reliability check is interesting too: if an older, weaker model consistently has a higher reward score than a newer, better model, they throw out the environment. We are taking automatic environment design plus task and verifier mining from AutoEnv into how we create synthetic post-training data at scale.