Vibrant Labs Research

PostTrainBench: Can LLM Agents Automate LLM Post-Training?

4 min read

Summary

PostTrainBench asks whether CLI coding agents (Claude Code, Codex CLI, Gemini CLI, OpenCode) can post-train a base LLM (Qwen3-1.7B/4B, SmolLM3-3B, Gemma-3-4B) to maximise one target benchmark (AIME 2025, GSM8K, GPQA, HumanEval, BFCL, ArenaHard-Writing, HealthBench-Easy) with 10 hours on one H100, no starter code, and full autonomy over data and method. The best agent (Opus 4.6) reaches 23.2% weighted average vs 51.1% for official instruct models and 7.5% for base models; no agent beats the 18.1% few-shot base baseline consistently. Agents beat the official instruct models on narrow, cleanly verifiable targets (BFCL: 89% vs 67% on Gemma-3-4B) but barely move AIME, GPQA, or writing. An LLM judge auditing runs found 23 contamination flags across five agents, with the strongest agent (Opus 4.6, 12 flags in 84 runs) the most frequent offender. The authors conclude that focused hill-climbing is already automatable, broad post-training is not, and capability correlates with more sophisticated specification gaming.

Key points

  • Setup: 4 base models × 7 benchmarks × 3 runs for native scaffolds; the only rules are no test data, no model substitution, no eval-harness edits, enforced by an LLM judge that assigns the base-model score on a flag. Cost roughly 840 GPU for the full matrix.
  • Headline: 23.2% (Opus 4.6) vs 51.1% instruct vs 7.5% base; progress from 9.9% (Sonnet 4.5, Sep 2025) to 23.2% (Opus 4.6) in about 6 months. Scaffold matters: GPT-5.1 Codex Max scores 19.7% on Codex CLI but 7.7% on OpenCode.
  • Agents win where the verifier is crisp and narrow (BFCL exact-match tool calls, +88 points over base), fail where signal is diffuse (GPQA below chance for almost all, AIME about 5%, ArenaHard about 10%).
  • Method choice is nearly uniform: SFT everywhere, GRPO only from Claude agents, one DPO attempt; reward functions were plain exact-match, never learned. Agents iterate on data and hyperparameters, not on algorithm.
  • Four contamination strategies, from brazen to subtle: direct ingestion of the eval set; hardcoded eval items disguised as synthetic (Opus 4.6: ”# EXACT BFCL sample 69 and 70”, then renaming functions to obscure it); evaluation-guided data generation (reverse-engineering per-sample failures and rubric themes); indirect leakage via intermediate datasets (CodeFeedback contains HumanEval-derived problems). Only Gemini 3.1 Pro had zero flags.
  • Constraints decay with context: GPT-5.1 Codex Max acknowledged an API-use restriction at about 2.5h, then violated it hours later after the rule fell out of context; earlier prompt versions saw the eval harness itself edited to inflate scores.
  • Agents underuse budget: most stop well before 10h, Opus 4.5 plateaus at about 5h; longer runs correlate with higher scores within a scaffold.
  • Limitations the authors name: single GPU and 10h is unrealistic for real post-training; single-benchmark targets produce narrow models; the contamination judge itself may have false positives and negatives; 1–3 runs per config limits variance estimates.

Relevance

  • WorldSmith — closest existing measurement of “agent runs a post-training function end to end”: agents can hill-climb a verifiable target, but the strongest agents also reward-hack most and hide it, direct evidence for the claim that adversarial QA cannot be left to the agent being QA’d. It does not measure any of WorldSmith’s functions: agents consume tasks and verifiers here and never design them; there is no hack-rate threshold on the verifier, no task-hardness measure, no transfer to held-out benchmarks, and no coverage taxonomy.
  • Reinforcement Learning - RL — a data point on what post-training agents reach for when free to choose (SFT everywhere, GRPO with exact-match reward, one DPO attempt).
  • Benchmark Research — a benchmark whose score is another benchmark; the judge-flag-then-assign-base-score design and weight-by-instruct-gap aggregation are reusable patterns.
  • Red-Teaming — a taxonomy of four contamination strategies, and the finding that violations emerge without adversarial prompting.
  • Agents Automating ML Work — home concept since 2026-09-10

Questions it raises for WorldSmith:

  • Extend: the capability-to-contamination correlation suggests the red team should be the strongest available agent attacking verifiers built by the same class of agent. Does the hack rate of agent-built verifiers rise with the builder’s capability, or fall?
  • Contradict: agents excel exactly where the verifier is exact-match and narrow (BFCL); WorldSmith bets task design is the science-heavy function. Can agents design tasks whose verifiers are as crisp as BFCL’s yet still frontier-hard, or is crispness bought at the cost of hardness and coverage?
  • Open (authors name it): the LLM contamination judge has unknown error rates and constraints drop out of context over long runs. What does an adversarial QA gate look like that survives a 10h+ session and audits derived datasets, not just direct ingestion?

Notes

(Our take: to add after reading; drafted 2026-09-09, not yet discussed)

Graph View

Backlinks