Auditing Reward Hackability in Code RL Training Environments

Summary

This paper asks the operational question a curator of a code RL training set actually has: what fraction of tasks reward the model for an incorrect solution? A frontier LLM (Claude Sonnet 4) generates candidate incorrect patches, each applied inside the task’s own Docker container and run through the project’s own test runner; a task is hackable if any wrong patch passes. On a 49-task sample of SWE-bench Verified, 28.5% are hackable; on 20 R2E-Gym tasks across 6 repositories at a weaker single-shot attack budget, 25.0%. A random-effects meta-analysis over 134 public model submissions shows the flag is not cosmetic: within the same 93-developer human difficulty stratum, Pass@1 is +14.14 pp higher on flagged-hackable tasks (95% CI [+11.80, +16.48], I² = 0%, 123 of 134 models positive). The paper then tries to repair the broken tasks with an LLM augmenter that writes new blocking tests — and finds that the naive version is an illusion: an inline 3-sample LLM judge endorsed 10 of 11 tasks as fixed, but Docker re-verification of a stratified sample found 6 of 8 endorsed augmentations were tests that fail on the gold patch itself. Adding a Docker gold-sanity gate (run each generated test against the reference solution before consulting the judge) reveals a 61.9% per-augmentation defect rate (65 of 105 decisive augmentations) that the judge alone never catches; with diversity-biased retry the gated loop converges 9 of 11 tasks.

Key points

• The headline rates are execution-verified, not judged: 28.5% (14/49) on SWE-bench Verified, 25.0% (4/16 decisive) on R2E-Gym. Round-1 single-shot exploits get 18.4% on SWE-bench; rounds 2–3 with failure logs fed back (the iterative-attacker pattern) add 5 more tasks. Both numbers are explicitly lower bounds — “at least 28.5% is reward-hackable by a frontier-LLM attacker”.
• The +14.14 pp meta-analysis is the bridge from “the verifier would accept a wrong answer” to “models actually produce wrong answers here.” Difficulty is controlled with the human-rated stratum fixed before any model evaluation; the authors explicitly reject cross-model p_solve as the control because hackable tasks inflate it through gaming, which would adjust away the effect. Unstratified the effect is +25.64 pp; stratifying retains 55% of it.
• The central methodology finding is the 61.9% augmenter defect rate — an agent asked to harden a verifier writes tests that mostly do not run correctly on the reference solution, and an LLM judge reading that test code endorses them anyway. The judge “correctly read the test code and reasoned about what it would check if it ran”; it cannot tell that the test does not run, runs incorrectly, or asserts the opposite of documented behaviour.
• The six named failure modes are mundane and mechanical (Table 1): wrong import style against the target module, inverted expectation on a decorator that raises, asserting None where the gold signature has inspect._empty, asserting escape() decodes entities when it encodes them, a duplicate-keyword TypeError, a database-routing early return. None require taste to catch — only execution against the reference.
• Ablation: retry presence is load-bearing, retry style is not. Gate + judge with no retry drops the headline from 10/11 to 3/11; retry recovers 6 of those 7. Replacing the diversity-biased retry prompt with a neutral “generate additional tests” at T=0.3 still gives 9/11. Cutting the judge from 3 samples to 1 costs one task and 56% of the API budget.
• Costs are small and reported: $6.04APIforthe49-taskSWE-benchaudit,$8.29 plus ~2.5h local Docker for R2E-Gym, $3.60forthegatedrepairloopover11tasks(vs$5.57 un-gated). The gate is cheaper than the judge-only loop because it kills bad augmentations before they are judged.
• Scoring is packaged as a per-task Environment Quality Score, EQS = 0.35·V + 0.30·(1−H) + 0.20·F1 + 0.15·D (verifier discrimination, empirical hackability, judge–execution agreement, cross-model learnability), thresholded into KEEP > 0.70 / FIX 0.40–0.70 / DROP < 0.40. Weights are untuned and the authors concede the partition is “a policy choice”; all three headline claims are weight-independent.
• Triangulation the authors offer: their 28.5% (Oct 2025), OpenAI’s 59.4%-flawed-on-failed retirement note for SWE-bench Verified (Feb 2026), Berkeley’s trustworthy-env 45 process-isolation exploits across 8 benchmarks (Apr 2026), and this meta-analysis (Jun 2026) — four different denominators, same direction.
• Limitations the author names: 49 tasks across 2 of 12 repositories (hackable tasks are only astropy and django), R2E-Gym at K=1, a single attacker model, the repair loop at n=11, and no causal test that fixing broken tasks improves training outcomes — that A/B at fixed compute is named as the natural next step.

Relevance

• WorldSmith — this is the source of the 28.5% figure the project’s open questions already lean on, and it converts “expert-built verifiers are themselves hackable” from an anecdote into a measured, execution-verified base rate with a cost attached. Two things it contributes that the project does not have elsewhere. First, a defensible absolute threshold argument: if hand-authored, human-audited SWE-bench Verified sits at 28.5% and R2E-Gym at 25%, then an agent-built verifier scored against parity with experts is being graded against a broken standard — the yardstick has to be absolute, which is the call the project already made. Second, and more directly on the benchmark’s subject: when an agent is asked to repair a verifier, 61.9% of its proposed tests fail on the reference solution, and an LLM judge reviewing them endorses them. That is a clean, cheap, purely mechanical measurement of exactly the failure the working claim predicts — agents cannot adversarially test their own verifiers — and it says the fix is not a better judge but an execution gate against a reference. For a benchmark scoring agent teams under dollar budgets, the numbers are usable as-is: ~$6toaudit49tasks,$3.60 to run a gated repair loop over 11, and a measured result that the gate lowers cost while raising honesty.
• Benchmark Validity — a per-task hackability score with an execution-verified denominator, plus the meta-analytic validation that flagged tasks inflate reported Pass@1 by 14 pp within difficulty stratum. The rejection of p_solve as a difficulty control on circularity grounds is a transferable piece of audit methodology.
• Red-Teaming — the attacker side is a concrete recipe: K candidate incorrect patches from a frontier LLM, applied through the project’s own harness, with rounds 2–3 conditioned on round-1 failure logs. Single-shot 18.4% → iterative 28.5% is the measured value of iterating the attack.
• Environment Generation — the repair half is agent-authored verifier hardening, and the paper’s contribution is the negative result: the LLM-judge-only loop reports 10/11 and is wrong. Execution-against-reference as an admission gate is the same move STAGE-Claw’s checker makes with build-twice-and-diff, applied to tests instead of environments.

Questions it raises for WorldSmith:

• The gate works because a gold solution exists to run the generated test against. In an agent-built environment there is no independent reference — the same team authored it. Is “does the environment come with a trustworthy reference solution” the precondition that decides whether verifier QA can be automated at all, and should WorldSmith score teams on producing one?
• The augmenter’s six failure modes are all syntactic or factual, not adversarial: wrong imports, inverted assertions, wrong defaults. If a mechanical gate catches those, what fraction of agent verifier-authoring failure is actually taste versus sloppiness — and does the hack rate look different once the sloppiness is gated out?
• The paper stops short of the causal test (train on fixed vs untouched broken tasks at fixed compute). WorldSmith’s “transfer without hacking” yardstick needs exactly that experiment. Is it cheaper to run it on 11 repaired tasks than to build the held-out public-benchmark transfer protocol?
• EQS weights are untuned and the author calls the KEEP/FIX/DROP partition a policy choice. If WorldSmith scores verifier soundness first, does it need a composite score at all, or just the H(t) > 0 indicator plus the cost to drive it to zero?

Notes

(Our take: to add after reading; drafted 2026-09-19, not yet discussed)