Summary
Curation-Bench fixes the model, training recipe, and eval suite (DataComp’s isolation principle) and gives a generalist coding agent a terminal loop: inspect a data pool, write a curation policy script, submit through a CLI that validates schema and budget and runs a contamination audit, train, read scores, revise. Selecting 10k of LLaVA-665K to fine-tune LLaVA-1.5-7B, open-prompt Claude Code (Opus 4.7) reaches 33.7 average over 8 benchmarks in 10 iterations, beating the published ICONS (33.3) and ARDS (33.2) baselines and recovering 59% of the full-data gain with 1.5% of the data. LLM-annotated trajectory diagnostics show an “execution-research gap”: agents run the loop reliably (fewer than 10 crashes in 500+ iterations) but 47% of moves are shallow threshold or seed tweaks and only 27% open a new policy family; strategy guides and paper references barely change this. A binding “adapt a cited prior method every iteration” scaffold raises new-family moves to 67%, eliminates shallow moves, and yields a policy scoring 34.9. Conclusion: agents are useful executors but narrow researchers; reliable data research needs method-adaptation scaffolds, not open-ended prompting.
Key points
- Setup: task = (base model, pool, budget, fixed trainer, fixed eval, constraints); the agent may edit only the curation script, never training, eval, or contamination code. Score = best of T iterations; 3 sessions × 10 iterations per agent, random baseline 10 runs.
- Open-prompt results: Claude Code 33.7, Codex 33.3, Qwen3.5-397B 33.2, Kimi K2.5 32.8 vs random 31.9; also gains on DataComp Small CLIP filtering, SmolVLM, and a rewriting extension (34.7 with 20 iterations).
- Exploit-vs-explore finding: once an agent finds a workable policy family it stays there, instantiating “the lowest-cost version of the idea” (quota, length filter, seed). The authors call the weakly grounded rationale “vibe optimization”.
- Awareness is not the bottleneck: light scaffolds (strategy list, paper skill cards) raise new-family moves from 27% to 43% but never beat the open-prompt max; agents change vocabulary, not implementation.
- Heavy scaffolds change execution: Adapt-Papers gives 100% grounded, 0% shallow, 67% new-family, best 34.9; but a hypothesis-first protocol raises grounding to 91% and lowers the score to 32.9. Heavier is not monotonically better.
- Contamination audit: normalized Q/A strings per turn vs the 8 eval sets; exact match via SHA-256 plus word 8-gram overlap at similarity ≥0.8; high risk (exact ≥10% or near-dup ≥20%) blocks submission, warning (≥5% / ≥10%) only alerts. The paper lists “overfitting visible evals” as a known failure and recommends hidden evals and red-team audits, which it does not run.
- Iteration budget is a compute axis: average outcome keeps improving from 10 to 50 iterations with no clear plateau.
- Limitations the authors name: mostly VLM instruction tuning; scaffold comparison not factorial; trajectory labels are LLM judgments; training-eval noise not separated from policy effect beyond the 10 random runs.
Relevance
- WorldSmith — closest existing evidence for the “agents optimise local variants rather than explore” worry and for the executor-vs-researcher split. It measures the agent as policy searcher against a fixed, trusted verifier (the eval suite), so it says nothing about agent-built tasks, agent-built verifiers, hack rate under adversarial attack, or coverage, and its only leakage defence is n-gram overlap with thresholds that let 9% exact match through.
- Benchmark Research — agent-centric benchmark contract: fixed pipeline, gated submission, trajectory-level rubrics (new family / grounded / effective / shallow) alongside outcome scores; a template for scoring process, not just result.
- Reinforcement Learning - RL — treats data policy as the object of optimisation and shows agent-search iterations behave like a compute axis under fixed data.
- Agents Automating ML Work — home concept since 2026-09-10
Questions it raises for WorldSmith:
- Extend: if the same execution-research gap holds for verifier authorship, do agent-built verifiers cluster in one family (end-state diffs) and never reach trajectory- or policy-level checks unless a method-adaptation scaffold forces it? The trajectory rubric could be reused to label verifier-design trajectories.
- Contradict: the paper equates “reaches published baseline” with success and never red-teams the audit. Would the 34.9 policy survive a hidden held-out eval, and what does that imply for the “transfer without hacking” yardstick when the only leakage check is n-gram overlap?
- Open (authors’): scaffolds are confounded and heavier hurt in one arm; which single ingredient drives exploration, and does the answer differ for verifier design where the prior-methods library is thin?
Notes
(Our take: to add after reading; drafted 2026-09-09, not yet discussed)