Vibrant Labs Research

Agent² RL-Bench: Can LLM Agents Engineer Agentic RL Post-Training?

4 min read

Summary

Agent² RL-Bench (Microsoft Research Asia, RD-Agent) asks whether a coding agent can autonomously design, implement, debug and run a full post-training pipeline, including closed-loop online RL, to improve a base model inside a 12-hour budget. Six tasks sit on a three-level ladder: L1 static rule-verified (GSM8K, HumanEval), L2 static LLM-judged (AlpacaEval 2.0), L3 interactive rollout (ALFWorld, WebShop, DeepSearchQA). The agent gets an isolated workspace, training-visible data, a base model and a scalar grading API, and is scored by its best valid submission. Across five scaffolds and six driver LLMs, one RL-oriented Claude Code run lifts ALFWorld from 4.85 to 93.28 via SFT warm-up plus GRPO with online rollouts, while DeepSearchQA stays hard (best 23.0) and weak stacks regress the base model by up to 51 points. The authors conclude that agents can sometimes close the online loop, but most winning routes are SFT-initialised composites, outcomes are highly stack- and mode-sensitive, and score gains alone do not prove online-RL engineering.

Key points

  • Setup: the agent must build the whole loop (data, rollouts, reward handling, training, submission), not fill a slot in an expert pipeline. Test sets are never mounted, the grader returns only a scalar and best-so-far, and every submission, code revision and model artifact is logged for post-hoc route attribution.
  • Score improvement and RL engineering are separable: in reconstructable 7B cells SFT was attempted in 19 and adopted in 18 best routes, GRPO attempted in 8 and adopted in 2; only ALFWorld has strict online RL among its best pipelines.
  • Stack sensitivity dwarfs task difficulty on L3: Codex CLI swapping GPT-4o for GPT-5.4 moves the ALFWorld gain from +0.74 to +81.97; ALFWorld spans 88 points across stacks vs 19 on GSM8K.
  • Operating mode is a hidden variable: at fixed scaffold and driver, Claude Code’s ALFWorld ranges 13.43 (Multi) to 95.52 (Free), an 80-point spread invisible to score-only leaderboards; mode ordering reverses between L1 and L3.
  • Failure taxonomy: rollout construction failure, reward/data design failure, training collapse, early coding failure, “static shortcut without transfer”. WebShop shows “code exists” is not “useful RL happened”: a run with DAgger, reward shaping and AWBC still regressed 14.5. The authors argue trajectory-collection quality, not algorithm choice, is the bottleneck.
  • DeepSearchQA resists post-training: 145 submissions, 47 code files, DPO/GRPO/KTO/ORPO/merging, still only 15.0 on 7B; two independent agents converged to the same ceiling.
  • Weak stacks actively harm: Gemini CLI + 2.5-flash drops HumanEval 51.22 and GSM8K 12.89 points, so the benchmark discriminates in both directions.
  • Limitations the authors state: single runs per cell, scaffold and driver entangled in the 7B study, no hidden final evaluator or submission cap (adaptive overfitting to the grader is “limited but not eliminated”), anti-cheating rules explicitly not adversarial-grade.

Relevance

  • WorldSmith — measures the consumer side of the RSI data loop: can agents run RL on given tasks with given verifiers. Its route-attribution protocol (audit workspaces, separate “score went up” from “the intended thing happened”) is a template for scoring verifier authorship. It does not have agents design tasks or verifiers, does not red-team any grader, and does not measure coverage or transfer. The “adaptive overfitting to the scalar grader” caveat is exactly the hack surface WorldSmith would score.
  • Reinforcement Learning - RL — evidence that agent-driven online RL (rollout, trajectory reward, GRPO) is achievable but rare and brittle under fixed budgets; SFT on collected trajectories is the default winning route.
  • Benchmark Research — a case for compact structure-coverage suites plus process metrics (route type, valid rate, time to first and best) over large leaderboards; also a worked example of single-run, entangled-factor reporting and how to caveat it.
  • Agents Automating ML Work — home concept since 2026-09-10

Questions it raises for WorldSmith:

  • Extend: trajectory-collection quality, not algorithm choice, is the L3 bottleneck. Does the same hold for environment builders, i.e. is the hard function “generate tasks that carry signal” rather than “write a verifier”? Score task design and verifier design as separate cells the way this paper separates SFT from GRPO routes.
  • Contradict: agents optimise against fixed verifiers for 12h with up to 195 submissions and no hacks are reported, but nobody looked. If an agent’s post-training loop is pointed at an agent-built verifier, does a hack rate appear, and does the L1/L2/L3 ladder predict where?
  • Open (authors’ own): they call for a hidden final evaluator, dev/test splits and submission caps because scalar best-so-far feedback permits adaptive overfitting. What absolute hack-rate threshold must a grader meet before best-so-far feedback can be exposed to an optimising agent?

Notes

(Our take: to add after reading; drafted 2026-09-09, not yet discussed)

Graph View

Backlinks