Summary
This paper asks what a deployed verifier’s natural false positives actually do to RLVR training, and answers it with a preregistered two-arm causal contrast rather than an audit. The key framing is that natural FPs are unlike the noise the robustness literature models: they are per-task, persistent and one-directional — the same weak tests accept the same wrong program on every rollout, so the error never averages out. Both arms run GRPO on the same 250 MBPP tasks with identical model, seeds, hyperparameters and compute; only the reward differs, original MBPP tests (leaky) versus the MBPP+ extra tests run instead of them (hardened). The headline result is a null with teeth: at 400 steps the hardened arm ends 0.20 pt ahead on held-out pass rate, non-inferior under the preregistered 1.5 pt margin, and rewarded-FP mass stays flat — hardening removes 8.37 pt of measurement inflation on train-task reward and buys essentially no capability. What the averages hide is the point: a cheap static audit run before training (sample the base model, flag solutions that pass MBPP and fail MBPP+) predicts where rewarded-FP mass will land at Spearman 0.80, the leak-stratum FP share runs +43.8 pt above clean tasks, and a signed human adjudication of all 2,821 rewarded FPs finds 47.57% record-weighted verified-wrong code — real bugs, not suite artifacts. Mechanism evidence points to selection of pre-existing error modes rather than learned exploitation: FP incidence does not grow over the horizon, and untrained base models already emit the same wrong outputs under the leaky filter. Two further model families replicate the core measurements under a preregistration frozen before their data existed.
Key points
- “Damage follows the error pattern, not the aggregate rate” is the paper’s thesis, but the measured damage at this scale is bounded, not catastrophic. Every aggregate instrument read clean — held-out averages inside the margin, no growth, no sub-behavioral sharpening — while the channel was demonstrably present at 8.82% of leaky rollouts, half of it real bugs. The author’s own gloss: clean readings bound what happened, they do not certify what the reward stream contains.
- The cheapest and most transferable result is the static leakiness audit: one sampling pass from the base model before any training predicts rewarded-FP mass at ρ = 0.803 / 0.813 / 0.700 across three families. That is a pre-flight QA instrument for a verifier, not a post-hoc training diagnostic, and it is validated against what training actually paid for.
- FP mass is extremely concentrated — Gini 0.79 over per-task rewarded-FP counts — and repairing the top 18 of 250 task suites removes about half of it. Verifier QA budget should be targeted, not spread. Caveat: some of that top mass indicts the hardened suite’s own tests, so repair means fixing both sides.
- Training rollouts work as natural mutants that strictly refine the static audit: they flag 74 tasks the static probe called clean with zero reversals (McNemar p ≈ 5.3e-23); total escape mass is unchanged, so the refinement buys localization rather than volume.
- The practical rule is about where to measure: train-task reward is an accounting of what was paid, not of what was learned, so reward QA must be computed on tasks the model has not trained on. The 8.37 pt inflation does not survive held-out.
- Theory placement: a deployed-but-weak suite sits inside the J > 0 region of the Youden-index phase transition, where the prediction is no amplification — which is what is measured. Family B drifts in the predicted decay direction; families A and C are flat, which the author flags as an open resolution question and tests again at twice the horizon (still flat).
- Cross-architecture leak sharing: three unrelated base models already produce the same characteristic wrong outputs before training, per-task leakiness correlates 0.69–0.75 between families, and fixing the suite once removed the tax for all three. Audit results transfer across models better than damage estimates do.
- Exploratory meta-audit on frontier judges: reading code blind and without execution, GPT-5.6 separates verified-wrong FPs from genuine passes at AUC .959/.958 and DeepSeek-V4-Pro at .902/.885, well above a code-length baseline (.672/.634) — separability exists above the 1.5B policy’s capability. But on their own false positives the judges self-assess only weakly. Two subjects, one benchmark, licenses nothing general.
- Epistemic discipline worth copying: every claim carries [C] confirmatory / [E] exploratory / [I] instrument-validity; preregistration failures are reported in their own subsection; the eval-side stratification is declared permanently exploratory because the cut was made after seeing data.
- Limits the author states plainly: 1–1.5B models, 400 steps, MBPP-length tasks, one benchmark family (the HumanEval+ transfer test was null), five seeds so true gaps between 0.2 and 1.5 pt are undetectable, and “hardened” means extra-tests-only rather than the union-scored EvalPlus metric. Single author, unreplicated externally.
Relevance
- WorldSmith — the sharpest available argument for how a verifier-soundness gate should report, and a caution about its headline metric. Three things transfer: (1) score FP structure — which wrong programs, how repeatably, concentrated where — because a scalar rate cannot distinguish persistent per-task acceptance from resampled noise, and only the former is a standing reward for a specific wrong answer; (2) a cheap static audit correlating 0.80 with realised FP mass means the gate does not need an expensive training run to estimate exposure, which matters directly under WorldSmith’s dollar and annotator budgets; (3) measure reward QA on held-out tasks, since train-task reward inflates. The uncomfortable finding for the project’s premise is that at this scale the leak cost ~0 held-out capability — so “the verifier is hackable” and “training on it is harmful” are separate claims, and WorldSmith should not assume the second follows from the first. The author’s scope is 1.5B models on MBPP-length tasks and he says reward-hacking pressure is reported to grow with task length, which is precisely the regime an enterprise environment lives in.
- Benchmark Validity — a verifier-side companion to the task-side audits: it quantifies what a defective grader is worth in training reward (8.37 pt of inflation) and what fraction of accepted-but-wrong is genuine defect (47.57%) versus artifact of the checking suite, with a human adjudication protocol and a published taxonomy.
- Reinforcement Learning - RL — places deployed test suites on the noise map: persistent asymmetric FPs versus the symmetric resampled flips that RLVR tolerates at up to 15%, and the Youden-index J = TPR − FPR phase transition as the criterion for whether incorrect modes amplify or die out.
Notes
(Our take: to add after reading; drafted 2026-09-19, not yet discussed)