Hardening Agent Benchmarks with Adversarial Hacker-Fixer Loops

Summary

The paper first measures the attack surface and then automates the defense. Probing 1,968 tasks across five terminal-agent benchmarks (Terminal-Bench 1.0/2.0/Pro, OpenThoughts-TB-dev, SETA) with three frontier models given only the task description and a hack-elicitation prompt, 323 environments (16%) are hackable; in Terminal-Bench 2.0 specifically, 13 of 89 (15%). The 3,632 confirmed hack trajectories are released as Terminal Wrench. The method is the hacker–fixer loop: three LLM agents alternate, a hacker tries to pass the verifier without solving the task, a fixer patches the verifier to block each discovered exploit, and a solver attempts the task legitimately so the patch is only committed if valid solutions still pass. Two levers broaden coverage without changing the hacker model — verifier access (the in-loop hacker reads verifier source, though the held-out evaluator stays blind) and a shared defense pool (a git repo that propagates task-agnostic, infrastructure-level fixes across every task sharing the harness). On KernelBench L1 the loop, run entirely on Gemini 3 Flash, drives held-out attack success from 62% to 0% on a corpus of 15 publicly reported exploit strategies, and drives the stronger blind hackers Gemini 3.1 Pro and Claude Opus 4.7 from 76% and 61% to 0%. On the messier Terminal Bench the same loop is only partly successful: documented exploits 50% → 39%, unhinted attacks 39% → 17%, and legitimate solutions are collaterally rejected, benign pass falling 76% → 65%.

Key points

• The weak-to-strong result is the headline: defenses authored by Gemini 3 Flash hold against Gemini 3.1 Pro and Claude Opus 4.7. The compensation for the capability gap is informational, not scale — verifier source access plus cross-task fix sharing. Ablations confirm both levers matter: blind + no pool converges at iteration 2 and leaves hinted ASR at 52%; aware + pool reaches 0%.
• The solver is the load-bearing third agent, and KernelBench is an accidental ablation of it. Validating against a deterministic reference solution (which never calls torch.cpp_extension.load_inline, the canonical submission API) let the fixer silently block load_inline alongside real exploits; raw loop output holds ASR at 0% but drops benign pass to 0%. A post-hoc autopatch relaxing two over-restrictive defenses lifts benign pass back to 92–98% without raising ASR. An LLM solver, used on Terminal Bench, catches this every iteration.
• The shared defense pool turns hardening from per-task effort into amortized infrastructure work — the final verifier for KernelBench task 001 contains no line originating from task 001. All 15 retained commits come from 13 other tasks, composing six defense layers: process isolation, monkeypatch capture, stack-introspection blocking, tensor-subclass detection, timing hardening, and unpredictable seeds. Over 14 iterations across 100 tasks the pool accumulates 171 commits from 45 distinct tasks.
• Verifier access buys exploits blind probing does not find. On KernelBench task 046 the verifier-aware hacker noticed reported speedup is ref_runtime / sol_runtime — attackable from either side — and used gc.get_objects() to find the reference model and make its forward pass burn matmuls, reporting 93,862× speedup. Across 49 unconstrained blind attempts, zero found it; in-loop verifier-aware hackers found it twice.
• Exploit recurrence and within-task diversity are what force the loop’s shape. The same classes recur (reading answers from unguarded files, replacing system binaries with wrapper scripts), motivating the pool; and a single task admits several independent exploits — SETA task 1219 can be passed by fake .deb packages, a fake process plus sentinel files, or replacing /usr/bin/xrandr with a script that prints the expected strings — so patching one leaves the rest open, motivating iteration.
• Solution narrowing is the measured price of hardening: 76.1% → 65.2% benign pass on Terminal Bench. The example is honest — a fixer adds gradient-correctness tests against a reference, which blocks hackers spoofing convergence flags but also rejects solvers that legitimately modify the objective by adding regularization.
• Some tasks are unfixable at the verifier level and need the evaluation infrastructure redesigned. Their example: a task requiring multi-pass shred cannot be verified inside a Docker container without access to the underlying filesystem, because shred and rm -rf leave identical observable state.
• Audit hygiene: of 4,848 trajectories that pass a verifier, an LLM judge marks 75% as genuine hacks and the rest as legitimate solves; the first 49 environments with a judge-confirmed hack were manually verified with no false positives. On Terminal Bench 10.0% of reward-passing attack trajectories were judged legitimate and filtering them leaves results qualitatively unchanged.
• Named ceiling: the loop patches only what its hacker discovers, bounded by model capability and iteration budget. The authors note defenses are far more effective against agent-generated attacks than against the human-discovered exploits in the hint corpora, and argue the agent-attack regime is the one closer to RL training and in-the-wild evaluation.

Relevance

• WorldSmith — the closest published instance of the adversarial-QA function running as an automated loop rather than a human gate, and it makes that function measurable in the terms the project scores on. Three contributions. First, a baseline hack rate for human-built environments across five benchmarks (16% of 1,968 tasks), which pairs with Auditing Reward Hackability’s 28.5% to bound what “expert quality” actually means before agents are compared to it. Second, an economics argument for agent teams: a cheap model hardens against an expensive one, and fixes amortize across every task sharing a harness, so verifier QA scales sub-linearly in tasks — exactly the property a dollar-budgeted benchmark should reward. Third, and most useful for scoring design, the loop exposes the two-sided nature of verifier soundness: driving attack success to 0% is easy if you may also reject legitimate solutions, and the raw KernelBench run does exactly that (0% ASR, 0% benign pass). Any WorldSmith verifier score that reports only a hack rate can be trivially maxed; the solver arm — a false-negative measurement — has to be scored alongside it. Terminal Wrench (323 hackable environments, 3,632 trajectories) is also directly reusable as a held-out attack corpus.
• Red-Teaming — a complete, released three-agent hardening protocol with the threat model stated (blind at rollout and evaluation; the in-loop hacker is deliberately over-powered with source access to anticipate stronger future attackers), a held-out evaluation split into hinted (replicate published exploits from other labs and models) and unhinted arms, and the finding that human-discovered exploits remain harder to defend than agent-discovered ones.
• Benchmark Validity — a 1,968-task cross-benchmark audit of how many environments accept a non-solution, with judge-confirmed and hand-spot-checked trajectories, plus the observation that fixing them is continuous maintenance rather than a one-time pass.
• Environment Generation — the fixer is an agent authoring verifier code under an automated admission test, and the shared pool is a mechanism for reusing environment infrastructure across generated tasks. The shred-versus-rm -rf case is the sharpest available statement that verifiability is a property of the world design, not of the checker.

Questions it raises for WorldSmith:

• The loop needs the solver to be genuinely diverse — a reference solution silently missed load_inline and the run scored 0% benign pass. If a WorldSmith team supplies both the verifier and its own solver, the QA is self-graded. Does the benchmark have to supply an independent solver population, and does that make solver diversity the scarce resource rather than attacker strength?
• Weak-to-strong worked because the in-loop hacker had verifier source and the evaluator did not. That asymmetry is a design choice about information, not capability. Should WorldSmith explicitly rate teams on how well they exploit their own source access, given that in an RL-envs business the builder always has it?
• Solution narrowing cost 11 pp of legitimate solutions on Terminal Bench with no obvious way to price it. What exchange rate between hack rate and false-negative rate should the verifier-soundness yardstick use — and does the project’s “no false positives, very low false negatives” framing survive contact with a loop that trades one for the other every iteration?
• Some tasks are unfixable at the verifier level. Is “can this task be made soundly verifiable at all” a judgment that must happen at task-design time, and therefore a part of the task design function rather than the QA function the project currently assigns it to?

Notes

(Our take: to add after reading; drafted 2026-09-19, not yet discussed)