Summary
PLR is a drop-in change to how procedurally generated (PCG) RL environments sample training levels: instead of uniform sampling, keep a buffer of seen levels scored by estimated learning potential and replay the high-scoring ones more often. The score is the magnitude of the TD error (L1 value loss / GAE) on the last trajectory from that level, mixed with a staleness bonus so old scores get refreshed. It requires only that levels be identifiable by a seed and re-settable, not any control over the generator. On Procgen it improves sample efficiency and generalization on 10 of 16 games, matches the prior state of the art, and combined with that method (UCB-DrAC) sets a new one at >76% improvement in test return over PPO; on sparse-reward MiniGrid it induces an emergent easy-to-hard curriculum.
Key points
- Level scoring: learning potential ≈ average |TD error| (L1 value loss) over the latest trajectory on that level; sample next level from a rank-based distribution over scores mixed with a staleness prior.
- Blackbox assumption: only needs (i) a notion of level, (ii) seed-based sampling, (iii) reset-to-level; no generator access.
- Forward-view analogue of prioritized experience replay: past experience decides which future experience to collect.
- Results: better test return on 10/16 Procgen games and two hard MiniGrid tasks; combined with UCB-DrAC, >76% relative gain over baseline PPO on Procgen test.
- In MiniGrid the replay distribution shifts from few-room to many-room levels over training: an emergent curriculum without any difficulty labels.
- The scoring function is a free choice; L1 value loss was the best of those tried, learned scorers are future work.
Relevance
- Open-Endedness — the “curate randomly generated levels” branch of UED; cheap, generator-agnostic, and the base that Replay-Guided Adversarial Environment Design (Robust PLR) and Evolving Curricula with Regret-Based Environment Design (ACCEL) build on.
- Reinforcement Learning - RL — a practical lever on which experiences to collect.
- Unsupervised Environment Design — home concept since 2026-09-10
Notes
(batch ingest; no notes yet)