Summary
This paper defines Unsupervised Environment Design (UED): given an underspecified environment with free parameters, automatically produce a distribution of fully specified, solvable environments tailored to the current policy. Domain randomization produces no structure and cannot adapt difficulty; minimax adversaries produce unsolvable levels. PAIRED (Protagonist Antagonist Induced Regret Environment Design) adds an antagonist agent allied with the environment generator and rewards the generator with regret — the antagonist’s return minus the protagonist’s — so it is pushed toward levels that are hard but achievable. In maze navigation PAIRED yields an emergent curriculum of increasingly complex levels and the best zero-shot transfer to novel mazes. The paper also connects UED to decision theory (“decisions under ignorance”), showing randomization, minimax and minimax-regret as the three natural choices.
Key points
- UED formalized: an underspecified POMDP plus a policy → a distribution over fully specified environments to train on next; iterating gives unsupervised curriculum design.
- Failure modes of prior approaches: domain randomization (uniform, structureless, difficulty-blind) and minimax adversarial training (unsolvable worst cases).
- PAIRED: the adversary maximizes regret = R(antagonist) − R(protagonist). Unsolvable levels give zero regret, so the generator is incentivized toward feasible-but-hard levels; as the protagonist improves, complexity rises automatically.
- At Nash equilibrium the generated environments are the maximum-regret ones, i.e., the protagonist plays a minimax-regret policy — a robustness guarantee.
- Results: PAIRED agents learn more complex behaviors and transfer zero-shot to held-out mazes (labyrinths, large mazes) where domain randomization and minimax fail; also gains on a continuous-control task.
- Decision-theory framing: domain randomization ↔ principle of insufficient reason, minimax ↔ maximin, PAIRED ↔ minimax regret.
Relevance
- Open-Endedness — the founding UED paper and the origin of regret as the objective every later method (PLR, Robust PLR, ACCEL) optimizes; it is the engineered version of the autocurricula argued for in Autocurricula and the Emergence of Innovation from Social Interaction.
- Reinforcement Learning - RL — environment generation as a training lever.
- Unsupervised Environment Design — home concept since 2026-09-10
Notes
(batch ingest; no notes yet)