Summary
This paper recasts Prioritized Level Replay as a UED method and unifies it with PAIRED under Dual Curriculum Design (DCD): a student faces two co-adapting teachers, one that generates new levels and one that curates existing ones for replay. All DCD methods inherit PAIRED’s minimax-regret guarantee at Nash equilibrium, which gives PLR its first theory. The theory yields a counterintuitive fix — Robust PLR (PLR⊥) updates the policy only on replayed (curated) levels and uses fresh random levels solely to score them, i.e., it trains on less data — and it also produces REPAIRED, PAIRED with a replay buffer. On mazes and a CarRacing UED domain, PLR⊥ and REPAIRED substantially improve zero-shot transfer over PLR and PAIRED; curated random levels even beat the learned PAIRED generator.
Key points
- DCD framework: generator teacher + curator teacher, both regret-maximizing; PLR (random generator, prioritized curator) and PAIRED (learned generator, no curator) are special cases.
- Robust PLR (PLR⊥): stop gradient updates on uncurated levels; only replayed high-regret levels train the policy. Provably minimax-regret at NE, and empirically better OOD transfer.
- REPAIRED = PAIRED generator + PLR⊥ curation; fixes PAIRED’s degenerate simple-track behaviour in CarRacing.
- Regret is estimated without an antagonist via positive value loss / max Monte-Carlo return minus value.
- Results: strong zero-shot transfer to human-designed mazes and Formula 1 tracks; PLR⊥ tracks look like human tracks.
- Limitations: guarantees hold only at NE with no convergence guarantee; replay methods are bounded by the generator’s proposals; choosing the free parameters of the UPOMDP is itself hard.
- Long-running UED in large spaces “closely resembles continual learning in open-ended domains.”
Relevance
- Open-Endedness — the paper that gave UED a unified theory (DCD) and the currently standard practical method (Robust PLR); connects Prioritized Level Replay to Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design.
- Unsupervised Environment Design — home concept since 2026-09-10
Notes
(batch ingest; no notes yet)