Vibrant Labs Research

WorldSmith

20 min read

Question

I am studying the automation of RL-environment building by teams of coding agents, because I want to find out how sound the verifiers they ship are and how many lab-acceptable environments they can produce from a real company’s data under fixed dollar and expert-annotator budgets, in order to help lab post-training and evals teams understand how far the data side of the recursive self-improvement (RSI) loop is from closing and where human gates must stay, so that they can decide what to automate.

Reframed 2026-09-19 (grilling session, see Benchmark design below) from a per-function tracker to budgeted production: the unit scored is the generator (the agent system), end to end, by the portfolio of environments it ships; the per-function answer (“what separates the ones they cannot”) comes from attribution ablations on top. Earlier framing, kept because the ablations still answer it: which functions of an RL-envs business can agents perform at expert quality today, and what separates the ones they cannot?

In full (earlier framing): assume high-quality source material from a real company is already in hand (sourcing and cloning are treated as solved engineering). Given that, can a team of coding agents (a) design tasks that capture the work activities of the target job and are frontier-hard, (b) build verifiers that track what matters in the domain and cannot be reward-hacked, and (c) do the adversarial QA that proves (b)? What would count as an answer: a function-level benchmark with results per function, and a stated reason for each function agents fail.

Yardsticks (decided 2026-09-09):

  • Verifier robustness by absolute threshold. Hack rate under adversarial attack, not parity with experts. Experts are a data point, not the standard. Note from the memo below: score the structure of false positives (especially correct-end-state-but-policy-violation), not only the rate.
  • Task hardness. Are agent-built tasks hard for current frontier models?
  • Transfer without hacking. Training or meta-harness hill-climbing on these tasks lifts held-out public benchmarks, and the gain does not come from hacking.
  • Coverage. Do the tasks capture the work activities of the job? Needs a reference taxonomy.
  • Expert comparison is reported alongside, as context.
  • 2026-09-19: these four are now folded into one acceptance process (below). Robustness and hardness are gates, coverage is the aggregation, transfer is a one-off validation that the gates predict lift.

Benchmark design

Decided 2026-09-19. Full evidence in the scout reports under Related → Raw.

Positioning. Lead with verifier soundness: the first benchmark to score the builder of verifiers under a budget, by false-positive structure under adversarial attack. The old claim “nobody scores the generator under a budget” is falsified by τ^τ-Bench (2609.04611) and the Meta-Agent Challenge (2606.04455); position against them as single artifact vs portfolio, agent vs environment, serving budget vs production budget. AIxCC is the ancestor of budgeted portfolio scoring with a defect penalty, MAP-Elites QD-score of the coverage term. Still without precedent, and framing the verifier number: a priced, allocatable expert-annotator budget; quantity × quality in one score; coverage the builder must discover from the company’s data. The Agentic Environment Engineering survey (2606.12191) has no category for benchmarks that score environment generators.

Setup. Source material is the ITSMBench V2 world (DB snapshot, ticket and event logs, policy docs) with the executable clone given. The builder ships (task, verifier) environments and declares a target model per environment from an allowed panel.

Budgets.

  • Dollars, covering builder tokens and its own target-model rollouts, plus a loose wall-clock cap. Report cost per accepted environment against the human ledgers.
  • Expert-minutes, spent on a priced simulated oracle (hybrid): real experts author the hidden ground truth once; every run queries the oracle; one real-human run validates that rankings hold. Queries are priced by type (policy question, trajectory label, task review), answered single-threaded, charged even when the answer is NOT_COVERED, with modeled refusal and error. Validate with an equivalence test (TOST on Krippendorff’s α vs human-human) on queries agents actually asked, and publish the resolution limit.
  • The hidden gates give no feedback during a run. The builder does its own QA.

Acceptance gates, in order (cheap first).

  1. Static validity: gold solution executes and passes the verifier; two planted wrong solutions are rejected; LLM audit of spec, world, grader.
  2. Verifier soundness: frozen-generator mutant kill rate (detection) and verification rate; isomorphic-perturbation invariance (no labels needed); report Youden’s J and false-positive structure (persistent per-task FPs, especially correct-end-state-but-policy-violation). Later: hacker-fixer held-out attack success rate on survivors, and one proxy-vs-gold overoptimisation curve on a stratified sample to show the cheap gates predict harm.
  3. Pass band on the declared model: sequential 10 then 30 rollouts. Full credit in [0.2, 0.8]. A frontier-hard tier below 0.2 earns credit only with a solvability witness (gold executes, hinted rollouts succeed). Zero-solve with no witness is a defect. In band but collapsing pass^k is flagged unstable.
  4. Structural non-duplication: canonicalised gold-solution operations plus verifier predicate set, up to renaming.

Score. AM(r) × Σ_c w_c · √(Σ_{e∈c} q_e) over a hidden, expert-built work-activity taxonomy with labour-mass weights (frequency × handling time). AM(r) = 1 − (1 − r)^k with r = sound / (sound + defective) shipped, judged by the mechanical gates only, so shipped defects cost the whole portfolio and the builder’s ship/no-ship call is measured. Report builder precision and budget utilisation alongside, and normalise against an expert reference portfolio.

Attribution (v1). One 2×2 verifier swap: expert verifier on agent-built tasks, agent verifier on expert-built tasks. Task-design and QA ablations later.

Deliverable by 2026-10-09. One-page design plus a pilot of gate 2’s mechanical stack: about 10 hand-built ITSMBench verifiers vs about 10 agent-built ones for the same tasks, 30 to 50 mutants each, labels by the team. ITSM practitioners are hired only once the gate is shown to discriminate (double review, α ≥ 0.8).

Deferred. Six-run gates-predict-lift study (quartile buckets, matched rollout budget, Qwen3-8B, CoreCraft’s three OOD benchmarks, partial correlation against initial pass rate); hacker-fixer loop; overoptimisation curve; real-human oracle track; batch certification by finite-population sequential acceptance sampling plus stratified PPI.

Why it matters

Practical origin: we build RL environments. Agents keep getting better at producing them, but the science-heavy functions (task design, verifier design, QA) are fuzzy and human-bound. Not knowing what agents can take over costs throughput and, worse, risks shipping environments with hackable verifiers.

So-what chain:

  1. If nobody measures this, labs either over-trust agent-built environments (reward-hacked verifiers corrupt post-training) or keep environment supply human-bound and the data side of RSI stalls.
  2. Therefore nobody knows how far the data side of the RSI loop is from closing, or where the human gates must stay. RSI forecasts and safety plans carry a blind spot.
  3. Answering tells labs what to automate and where to keep QA gates. Applied research: the significance is a decision, not just understanding.

Output plan: internal capability tracker first, published benchmark once the metrics hold up. Audience for significance: frontier labs’ post-training and evals teams.

What I already know

In our words (interview, 2026-09-09):

  • We are building an RL-environments company. As agents get better, they get better at creating environments. A few things are still out.
  • Source data. The data is not high quality. You need good source data that captures the world, as detailed and complicated as possible.
  • Clones. You need a lot of clones, all from a coding process. The question is whether the clone’s behaviour is consistent with the real system. That is a loop to build; more of an engineering problem.
  • Task creation and verifier creation is where the science is, and it is fuzzy. For a benchmark you have to find the distribution of tasks that captures the essence of the job function you are automating: benchmark first, then how to come up with tasks and distributions, maybe with editors’ help, then how to transform all of that into something verifiable. Here you have to be smart. Arbitrary coding is hard to verify; “clone this” has a checkpoint that is easily verifiable yet exercises the core work activities of a software engineer. Very big long-horizon work (replicate an OS) can be split into teams. Brilliance is translating a big fuzzy thing into a straightforward verifiable one.
  • High-quality verifiers that cannot be reward-hacked are essential for RSI: no false positives, very low false negatives. QA and adversarial testing are central.
  • The functions of an RL-envs business a team of agents would have to do: sourcing, QA, working with annotators, task and verifier creation. The measure is how much of building the business agents can take over.

Prior work on this thread:

  • A deep-research memo from 2026-09-09 (raw/worldsmith-deep-research-memo.md) frames WorldSmith narrowly as agents author verifiers for an ITSM Gym, graded offline against a hidden set of expert-labeled episodes, with a red-team loop. Its BLUF: no existing benchmark scores the generator of verifiers or environments against expert ground truth with adversarial hardening; everything (Eureka, Text2Reward, AutoEnv, Agent-World) evaluates the downstream policy. Its highest-leverage design point: score false-positive structure, since RLVR training harm follows persistent asymmetric errors, not aggregate rate.
  • The openenv hackathon note (not in this repo) argues the bottleneck for RL-trained LLMs is environments, not algorithms, and sketches EnterpriseSim plus a Task Miner.

Working claim

Provisional. Agents already handle sourcing and cloning. Task generation, verifier design, and adversarial QA are where they fall short, because they cannot adversarially test their own verifiers and lack the taste to pick which tasks matter.

Alternatives:

  • The bottleneck is the human-defined task distribution (coverage), not verifiers. Kept live.
  • The gap is coordination across a team, not capability. Rejected: solo agents already fail on verifier quality; orchestration does not fix a taste and QA problem.

Kind of claim: fact with a cause. Evidence needed: per-function benchmark results with an absolute verifier hack-rate threshold, task hardness against frontier models, transfer to public benchmarks, and expert results for context.

What would overturn it (both together): agent-built verifiers survive the same red-team as expert-built ones at or under the absolute hack-rate threshold, and agent-built tasks are frontier-hard and lift held-out public benchmarks without hacking.

Open questions

  • Scope reconciliation. The memo’s WorldSmith is verifier authorship on one gym (L1); this interview framed it as a function-level tracker across task design, verifier design, and QA. Is verifier authorship the first function measured, with the others added later, or is the memo’s design the whole month’s target?
  • Which job function and source material is the first target (the memo assumes ITSM)?
  • How to measure task-distribution coverage: what reference taxonomy of work activities, and who builds it (editors, annotators, agents)?
  • How to test “transfer without hacking”: which held-out public benchmarks, and what distinguishes real gains from leakage?
  • Is novelty measurable: can we tell recombination of seen tasks from a genuinely new task distribution? (Curation-Bench and Agent²RL-Bench both find agents optimise local variants.)
  • Since expert-built verifiers are themselves hackable (28.5% false accepts reported on SWE-bench Verified, Auditing Reward Hackability), what absolute threshold is defensible, and should the hidden expert set itself be red-teamed?
  • Does the survey’s finding that no environment-generation paper separates agent-built from human-built environments at matched compute change the first experiment?

Added 2026-09-19 (several above are now partly answered by Benchmark design: scope is budgeted production with verifier soundness first; first target is ITSM; transfer is a deferred validation study):

  • The exponent k in the accuracy multiplier. DARPA used 4, which is nearly free above 90% accuracy; labs’ near-zero tolerance for false positives argues for steeper.
  • Exact oracle query prices, and the allowed panel of target models.
  • Taxonomy construction: bottom-up clustering of the event log reconciled with GDPval-style expert-named activities; how much of it stays unpublished.
  • Does a pass band measured on a cheap model predict the band on a frontier model? Needs a 50-task pilot before trusting the cheap path (24 per task).
  • Sandbox defence against the builder exfiltrating hidden ground truth (the Meta-Agent Challenge saw this under optimisation pressure; here the builder also controls the verifier).
  • Mutant kill rate moves ~30 points with mutant-generator strength, and mutation scores of LLM-written suites may not track real-fault detection (2607.22880). How is the frozen generator chosen and validated?
  • Joint error control when sequential lot acceptance and PPI share one human sample: check by simulation before publishing a risk limit.
  • Is the structural non-duplication gate a contribution in its own right? Nothing in the literature separates recombination from a new task type.
  • Run-to-run variance: The Meta-Agent Challenge finds σ > 0.1 across three runs on a third of builder configurations. How many runs per configuration before the portfolio score may rank anything?
  • A hack-rate-only verifier score is gameable: Hardening Agent Benchmarks reaches 0% attack success and 0% benign pass in one run. The false-negative arm (verification rate) must be scored beside detection, which the gate 2 design already does; keep it that way.
  • Youden’s J theory (Rate or Fate - RLVeR) assumes i.i.d. noise, but a hackable verifier fails systematically on what the policy learns, so J drifts down during training. When the Reward Suite Is Leaky found bounded held-out harm at 1.5B. What harm claim is defensible as motivation: “rewards wrong behaviour” rather than “causes collapse”?
  • ITSMBench — first live case: V2 is built by hand while tracking how much could be automated
  • Wiki: Open-Endedness (UED, autocurricula), Reinforcement Learning - RL (tasks, graders, environments), Web Agents, Benchmark Research, Red-Teaming (adversarial QA), Retrieval Benchmarks (benchmark-design method)
  • Background notes not in this repo: an enterprise-simulation reading list (Tier 4, automatically generating the right training environments), UED core reading, a World Models × Open-Endedness reading list (Phase 4, self-improving systems), the OpenEnv hackathon presentation, NeurIPS 2026 notes, and the working journal for Aug 2026
  • Raw: raw/worldsmith-deep-research-memo.md
  • Raw, literature scouts of 2026-09-18 (raw/worldsmith-lit-scout-2026-09-18-<aspect>.md): wiki-protocol-survey (45 measurement primitives already in the wiki, and the gaps), verifier, human-budget, utility, hardness, coverage, precedents (includes publicly stated lab and vendor acceptance criteria)
  • Concepts (added 2026-09-10): Environment Generation, Agents Automating ML Work, Benchmark Validity

Reading list

Sources

  • Agent2 RL-Bench — agents as consumers of tasks and verifiers: can they engineer online RL in 12h. Route-attribution protocol worth copying; grader admits adaptive overfitting
  • PostTrainBench — agents post-train to one benchmark in 10 H100-hours; win only on crisp verifiers, strongest agent reward-hacks most and hides it
  • Curation-Bench — agents write data-curation policies; execute reliably but explore narrowly unless forced by a method-adaptation scaffold; leakage check is n-gram only
  • Tau-Bench — schemas, policies, data generation are LM-assisted; making each task uniquely resolvable is manual and the hardest step
  • Tau2-Bench — five-stage LLM-drafted PRD-to-policy pipeline partially automates world-building; domain curation flagged as the human-bound step
  • Mind the Sim2Real Gap — a simulator step cannot be fully agent-built without a human reference corpus
  • RealUserSim — Grounds simulators in 7,275 profiles mined from real WildChat logs; fidelity 24.2%→45.3%, and tau-bench’s own persona directives inflate difficulty
  • Simulated Customers Never Walk Away — Simulators match real buyers but inflate non-buyers (d=0.38) against verified purchase outcomes; disengagement cannot be prompted in
  • SAGE — Cold-start simulator grounded in ICPs plus the agent’s own catalogs and FAQs; finds 25-33% more bugs than baselines and than real traffic
  • VISTA — Six simulator-quality metrics (tool-transition and trajectory-distance coverage) plus a hybrid UI+API user; 42% more failures than UI-only
  • EnterpriseOps-Gym — ServiceNow sandbox, 164 tables, 512 tools, 1,150 SME-authored tasks, SQL end-state verifiers; best model 37.4%, ITSM hardest at 28.5%
  • EntCollabBench — 11 role-specialized agents across six departments with permission isolation; trace + DB-diff judging by three-model vote; best 62%
  • TheAgentCompany — Self-hosted software startup with Sotopia-backed simulated colleagues; weighted checkpoint scoring; best agent completes 30.3% of 175 tasks
  • World of Workflows — ServiceNow instance with 4.8K hidden business rules; success under constraints collapses to ~2% without audit-log visibility
  • AppWorld — 9-app simulated world, 457 APIs; state-diff verification with expected/allowed change sets catches collateral damage; best 48.8% TGC
  • LOGIGEN — Triple-agent pipeline compiles policy into DB triggers, forward-explores to synthesize 20k verifiable tasks; state-diff verification, demonstrated simulator hacking
  • TASTE — Samples tool sequences from contrastive n-gram model to build harder, broader-coverage tau-bench extension; hint-assisted verifier validated at precision 1.0
  • Benchmarking the Benchmarks — Expert audit of 496 traces across four tool-calling benchmarks finds 18.5% evaluator-human misalignment; deterministic-first evaluator reaches 95.5%
  • ABA — Agentic auditor over 168 benchmarks, 34,285 tasks finds 25.7% carry major issues; filtering them shifts leaderboards by ~10 points
  • SABER — Mutating actions are 14-18% of steps but drive nearly all failure; ships tau-Bench Verified after finding half of instructions under-specified
  • Log Analysis Is Necessary — Taxonomy of internal/external/safety validity threats; tau-Bench Airline audit finds 25/50 tasks flawed, doubling pass^5 when excluded
  • Science of AI Agent Reliability — Twelve accuracy-independent reliability metrics over four dimensions; 15 models show reliability plateaued while capability rose for 24 months
  • AgentSimulator — Discovers a multi-agent system from an event log — per-resource schedules, capabilities, timings, handover matrix — beating control-flow-first simulators 20-80x faster
  • Agent Mentor — Mines an agent’s own trajectory logs by clustering outcomes and eliciting SVO features, then injects derived corrections into prompts; +37pp on ambiguous specs
  • CHI-Bench — Hand-built 20-app healthcare simulator, 75 policy-dense multi-role tasks; best agent 28% pass@1, two-agent provider-payer arena collapses to 0%
  • AgentDojo — Framework crossing 97 user tasks with 27 injection goals into 629 prompt-injection cases; capable models are easier to hijack, tool filtering cuts ASR to 7.5%
  • MANTRA — Generates trace-level compliance checks and a symbolic world model independently from a manual, cross-validating them with Z3; 285 validated tasks, no LLM judge
  • STAGE-Claw — An authoring agent writes task, environment and verifier; an independent checker agent validates reproducibility and difficulty; output-only grading inflates scores 5-7 points
  • ClawEnvKit — Generates whole environments (task, tools, fixtures, grader) from one natural-language line; 1,040 environments for $80 versus 208 human-hours, but validity is LLM-judged
  • Tau-Tau-Bench — scores a builder end-to-end on one artifact under a serving budget with the eval suite withheld; best 23.9% vs 82.2% expert ceiling, and developers weaken their own failing tests
  • The Meta-Agent Challenge — meta-agents build agents against a hidden test set under time and quota budgets; 5/39 beat human scaffolds, σ>0.1 on a third of configs, and zero-resource pressure induces hacking that direct prompting cannot
  • Auditing Reward Hackability — Source of the 28.5% expert-verifier hack rate; when an agent repairs a verifier, 61.9% of its tests fail on the gold solution and an LLM judge endorses them anyway
  • Hardening Agent Benchmarks — Adversarial QA automated as a hacker/fixer/solver loop; a weak model’s defenses hold against stronger hackers, but hardening also rejects 11pp more legitimate solutions
  • SWE-Mutation — Scores an agent-built verifier by mutation kill rate, reporting both error directions in one run; the score moves ~30 points with attacker strength alone, so a frozen attacker is mandatory
  • When the Reward Suite Is Leaky — Natural verifier FPs are per-task and persistent, so report FP structure not rate; a cheap pre-training static audit predicts exposure at ρ=0.80, but the leak cost ~0 held-out capability at 1.5B
  • Rate or Fate - RLVeR — verifier soundness reduces to one scalar, J = TPR − FPR; J > 0 costs only ~1/J compute, J < 0 destroys the policy (−12.6 pts at J = −0.1); assumes i.i.d. noise, which hackable verifiers violate
  • AIxCC SoK — agent teams building under 50K LLM budgets; non-linear accuracy multiplier (90% free, 40% → −13%) reorders the ranking, and automatic patch validation admits 38-46% semantically wrong work
  • HiL-Bench — Necessity/sufficiency task admission checked from rollouts; ask_human() is a frozen-LLM expert oracle without the price; Ask-F1’s harmonic mean builds the anti-hack into the metric
  • EnterpriseBench Corecraft — Expert-built enterprise world whose rubrics are LLM-judged and never validated; supplies the transfer evidence and the three OOD benchmarks, and is the position WorldSmith argues against

To ingest

Regrouped 2026-09-19 after the literature scouts (protocols to copy, one line each; details in the raw scout reports). Non-arxiv items are listed without a link id and need the Web Clipper route.

Precedents and positioning

Verifier soundness

Human budget and certification

Hardness and validity

Coverage, representativeness, novelty

Downstream utility and environment value

Later

  • RSI measurement: Measuring AI R&D Automation, RSI survey, Anthropic “When AI builds itself” (May 2026), OpenAI “Research acceleration” (Sep 2026), Epoch “Toward an O*NET for AI R&D”, METR org-uplift game
  • Other: Rubrics as Rewards 2507.17746, Absolute Zero 2505.03335, DéjàQ 2601.01931, Red Queen Gödel Machine 2606.26294, RE-Bench 2411.15114, MLE-bench 2410.07095

Graph View

Backlinks