Vibrant Labs Research

Reinforcement Learning - RL

3 min read

What it is

Reinforcement learning, with two threads so far: RL as the origin of machine learning, and what matters for training LLM agents with RL.

From Andy Barto - In the Beginning ML was RL - RLC 2024:

  • this is actually the bitter lesson - instead of memory, it’s learning

From CS294-196 (Agentic AI MOOC) - Lecture 4, Weizhu Chen, the most important questions for training agents are:

  • Data aka tasks
    • today verifiable tasks are generating much of the gains
    • but most situations in the real world are non-verifiable
      • rubrics are one solution, people write them
      • data synthesis
  • Grader / eval, aka rewards
  • Efficient environments

Key sources

  • Agent2 RL-Bench — agent-driven online RL (rollouts, GRPO) is achievable but rare and brittle; SFT on collected trajectories is the default winning route
  • PostTrainBench — what post-training agents reach for when free: SFT everywhere, GRPO with exact-match reward, one DPO attempt
  • Curation-Bench — data policy as the object of optimisation; agent-search iterations behave like a compute axis
  • When the Reward Suite Is Leaky — Natural verifier FPs are per-task and persistent, so report FP structure not rate; a cheap pre-training static audit predicts exposure at ρ=0.80, but the leak cost ~0 held-out capability at 1.5B
  • Rate or Fate - RLVeR — verifier soundness reduces to one scalar, J = TPR − FPR; J > 0 costs only ~1/J compute, J < 0 destroys the policy (−12.6 pts at J = −0.1); assumes i.i.d. noise, which hackable verifiers violate
  • EnterpriseBench Corecraft — Expert-built enterprise world whose rubrics are LLM-judged and never validated; supplies the transfer evidence and the three OOD benchmarks, and is the position WorldSmith argues against
  • DeepSeek-V4.1-Flash — a frontier lab stating it in print: stock SFT/RL/OPD, no algorithmic novelty, and the scale, diversity and verifiability of synthesized tasks and environments account for essentially all the gains
  • DSec — rollout infrastructure at frontier scale (~380K concurrent sandboxes); rollouts decoupled from preemptible GPU training; production reward-hacking catalogue
  • TMax — stability recipe for long-horizon agentic RL (DPPO, FP32 LM head, group size 32); collapse past 300 steps and beyond ~10 turns
  • Agent World Model — GRPO on 526 synthetic environments generalizes out of distribution; more environments help monotonically
  • AutoForge — ERPO: masking simulated-user errors and environment-level advantage normalization
  • ComputerRL — thousands of parallel desktop VMs, step-level GRPO, and Entropulse (SFT on successful rollouts between RL phases) against entropy collapse
  • T1 — long-horizon MoE agent RL: TITO and rollout routing replay close the train–inference gap; PPO with a warm-started critic beat GRPO; dense test-count reward beat binary

To ingest

Graph View

Backlinks