Summary
Open-ended learning needs a vast task space, and in a vast task space most tasks are either impossible or learnable-but-boring (minor variations on what the agent already does). OMNI combines two filters. A learning-progress (LP) curriculum samples tasks whose success rate is changing, so the agent trains at the frontier of its ability. A Model of Interestingness (MoI) — a foundation model prompted few-shot with what the agent has already mastered — predicts which candidate tasks humans would find interesting and worth learning next, and boring ones are down-weighted. In Crafter, LP + MoI learns 82 tasks against 55 for LP alone and 4 for uniform sampling, and performs almost identically to an oracle MoI; in AI2-THOR, where an FM also generates new tasks as code in an unbounded task space, OMNI learns 13 tasks against 2 for LP and 0 for uniform sampling. The authors name the open risk: by Goodhart’s law, any model of interestingness will develop pathologies once it is optimized against.
Key points
- Learnable is not enough. LP alone gets distracted by endless variations of uninteresting tasks; the MoI is what removes them.
- The MoI is a foundation model standing in for human judgment, prompted with directives (“learn as many interestingly different skills as possible”), an environment description, and the tasks the agent already does well.
- Results in finite task spaces: Crafter — OMNI 82 tasks learned (14 interesting), LP 55 (9), uniform 4; average success on interesting tasks 0.78 vs 0.52 for LP. BabyAI — 8 tasks vs 4. OMNI matches an oracle MoI across all metrics.
- In an infinite task space (AI2-THOR), the FM proposes new tasks as code, and OMNI learns 13 tasks to LP’s 2 (statistically significant at all checkpoints).
- Boring comes in several forms: repetitive tasks, compounds of interesting tasks, and synonyms were all tested as distractors.
- Goodhart risk is stated up front: a task generator tuned to please the MoI may eventually make the MoI a poor measure, motivating Open-Endedness with Human Feedback (OEHF).
Relevance
- Open-Endedness — the canonical paper for using foundation models as the notion of interestingness in auto-curricula.
- WorldSmith — a principled filter for which generated tasks are worth training on: learnable and interesting, rather than learnable alone. The Goodhart warning applies directly to any automated task-quality score.
- Unsupervised Environment Design — learning-progress curricula as the baseline OMNI improves on.
Notes
From paper club, 17 March 2026: OMNI goes further than environment generation by quantifying how interesting tasks are to humans (a Model of Interestingness) and how learnable they are (learning progress). For example, a task like “Go to Gmail, star an email, then delete it” is easily learnable and verifiable, but it is not interesting for humans, and should therefore not be included in RL runs. We are combining the notion of human interestingness with task learnability from OMNI to sample a subset of all the generated tasks.