Autoscaling RL environments
Reinforcement learning on language models is bottlenecked by environments, not by algorithms. Every serious RL run needs tasks that are realistic, verifiers that cannot be gamed, and enough of both to train on. Today all three are made by hand, by people who understand the domain, one environment at a time. That is the constraint we work on.
We are a small lab doing this in the open. The notes, the benchmark designs, the experiments and the things that did not work are published here as we go, rather than saved up for a paper.
What we are working on
ITSMBench asks whether the operational data an enterprise already has, event logs, tickets, change records, can become a benchmark and an RL environment that represents the work actually done there. It is built on the public BPI Challenge IT service management logs. The open question is which parts of an environment the logs can supply, and which have to come from somewhere else.
WorldSmith turns the question on the builders. Give a team of coding agents a real company’s data and a fixed budget in dollars and expert-annotator minutes. How sound are the verifiers they ship, how many usable environments do they produce, and which function explains the shortfall? If environment supply stays human-bound, the data side of the self-improvement loop stays closed.
Paper club
We read one paper a fortnight on RL environments, benchmarks and simulators, and discuss it in public. Anyone working on this is welcome.
See the schedule and past sessions.
The wiki
Most of this site is a research wiki: one page per paper we have read, concept pages for the vocabulary, and project pages that hold a question and the evidence against it. Start with Enterprise Environments, User Simulators, Environment Generation or Open-Endedness.
It is written for us first, so it is uneven. Some pages carry a full reading, others are a summary with our take still to come. Pages we have not published are not missing on purpose; they simply are not ready.
Get in touch
We are opening a Discord for people working on this; the invite will appear in the footer with the first paper club session. Until then, the sign-up link on the paper club page reaches us. More on how this site works in about.