What it is
Reinforcement learning, with two threads so far: RL as the origin of machine learning, and what matters for training LLM agents with RL.
From Andy Barto - In the Beginning ML was RL - RLC 2024:
- this is actually the bitter lesson - instead of memory, it’s learning
From CS294-196 (Agentic AI MOOC) - Lecture 4, Weizhu Chen, the most important questions for training agents are:
- Data aka tasks
- today verifiable tasks are generating much of the gains
- but most situations in the real world are non-verifiable
- rubrics are one solution, people write them
- data synthesis
- rubrics are one solution, people write them
- Grader / eval, aka rewards
- Efficient environments
Key sources
- Agent2 RL-Bench — agent-driven online RL (rollouts, GRPO) is achievable but rare and brittle; SFT on collected trajectories is the default winning route
- PostTrainBench — what post-training agents reach for when free: SFT everywhere, GRPO with exact-match reward, one DPO attempt
- Curation-Bench — data policy as the object of optimisation; agent-search iterations behave like a compute axis
- When the Reward Suite Is Leaky — Natural verifier FPs are per-task and persistent, so report FP structure not rate; a cheap pre-training static audit predicts exposure at ρ=0.80, but the leak cost ~0 held-out capability at 1.5B
- Rate or Fate - RLVeR — verifier soundness reduces to one scalar, J = TPR − FPR; J > 0 costs only ~1/J compute, J < 0 destroys the policy (−12.6 pts at J = −0.1); assumes i.i.d. noise, which hackable verifiers violate
- EnterpriseBench Corecraft — Expert-built enterprise world whose rubrics are LLM-judged and never validated; supplies the transfer evidence and the three OOD benchmarks, and is the position WorldSmith argues against
Related
To ingest
- General Reasoning - Towards Universal Intelligence: scaling RL compute — blog, needs Web Clipper