Summary
A DeepMind manifesto arguing that single-agent RL has fallen into a “solipsistic trap”: an agent’s intelligence is bounded by the environments humans hand-design, and designing them does not scale (the “problem problem”). Nature solved this because life is a multi-agent system in which every innovation changes the environment others must adapt to. The paper names this an autocurriculum: a self-generated sequence of challenges arising from competition and cooperation among adaptive units, where solving one social task begets the next. It classifies autocurricula by their source (exogenous, from competition between units; endogenous, from miscoordination within a unit) and asks under what conditions they yield the open-ended accumulation of innovations seen in human culture.
Key points
- The problem problem: generating enough diverse, interesting environments is the bottleneck for scaling intelligence; hand-authoring cannot go on indefinitely.
- The Go analogy: a solitaire territory game is trivially solved; adding an opponent creates a game rich enough for millennia. Other agents are the cheapest source of endless challenge.
- Autocurriculum = the coupled dynamics of (a) strategic choices that shift as others adapt and (b) learning implementation policies for those strategies. Each strategic shift by one player reshapes the adaptive landscape for the others.
- Taxonomy: exogenous challenges (competition between units at the same level, e.g., arms races) vs endogenous challenges (coordination problems inside a collective, e.g., institutions).
- Innovations arise at every level of the hierarchy (cells, organisms, societies); the truly intelligent adaptive units may be collectives, not individuals.
- “Exploration by exploitation”: in a non-stationary multi-agent world, following the greedy policy already explores.
- Open question the paper leaves: which conditions turn an autocurriculum into sustained, cumulative innovation rather than cycling.
Relevance
- Open-Endedness — the multi-agent origin story for open-endedness and the term “autocurricula” the concept page already uses; it motivates UED as an engineered stand-in for the multi-agent dynamics described here.
- Reinforcement Learning - RL — argues environments, not algorithms, are the scarce input for training agents.
Notes
(batch ingest; no notes yet)