Summary
SWE-Mutation is a benchmark for the verifier, not the solution: it asks an agent to write a repository-level test suite and then scores that suite by how many deliberately planted wrong solutions it rejects. The mutants are themselves agent-authored — a four-module pipeline (Locate, Mutation, Judge, Self-Play) built on Claude Sonnet 4 restricts edits to the files the golden patch touched, injects a semantic bug from a five-strategy menu of realistic human/model errors, validates that the mutant compiles and fails at least one Fail-to-Pass test, and then keeps only the top 50% of candidates that survived more than 3 of 10 sampled model-written test suites. The benchmark covers 800 instances (500 Python from SWE-bench Verified, 300 from SWE-bench-Multilingual across nine other languages) with 2,636 mutants, and defines two tasks (test generation from scratch, test repair of an existing suite) and three metrics: Pass@1 (applies and compiles), Verified Reproduction Rate (VRR: the suite fails on the bug and passes the golden solution), and Relative Detection Rate (RDR: share of surviving mutants killed). Seven models under Mini-Swe-Agent and Claude Code all do badly at generation — Claude-Sonnet-4.5 tops out at 40.40% VRR / 71.71% RDR and DeepSeek-V3.1 reaches only 10.20% VRR / 36.15% RDR. The load-bearing ablation is that the mutant generator’s strength moves the score by ~30 points: the same suites kill 71.04% of rule-based mutants but only 39.81% of agentic ones.
Key points
- The evaluation protocol is exactly a verifier-soundness gate: VRR is the false-negative side (does the suite wrongly reject the correct solution) and RDR is the false-positive side (does it accept a wrong one), measured in one run. RDR is a set difference — only mutants the original developer suite failed to kill count — so on the repair task it measures incremental value over human-written tests, and on generation it degenerates to plain mutation score.
- Mutant realism is the whole ballgame, and it is not free: rule-based 71.04% → few-shot LLM → agentic 39.81% average RDR. Under rule-based mutants the seven models cluster in a narrow band and the metric cannot separate them; under agentic mutants the spread is wide. A verifier-quality number is therefore only comparable against a frozen mutant generator.
- The five mutation strategies are a reusable taxonomy of “almost-right”: violation of API specifications and contracts (altered parameter defaults, swapped homogeneous parameters, substituted exception types), plus four further groups, all syntactically valid and semantically wrong. The Locate module uses Tree-sitter plus F2P execution traces to confine mutations to the defect’s triggering logic rather than scattering random bugs.
- The Self-Play filter is the move that makes the gate adversarial: candidate mutants are pre-screened against ten model-generated test suites and only the ones that evaded detection are admitted. The benchmark is tuned, by construction, to the blind spots of the generation it is evaluating — which sharpens discrimination and dates the benchmark at the same time.
- Generation is much harder than repair (Claude-Sonnet-4.5: 29.80% vs 42.60% VRR under Mini-Swe-Agent), and Pass@1 stays high (75–100%) while VRR stays low — models write syntactically valid tests that do not actually reproduce the bug.
- Scaffold buys formatting, not discrimination. Moving from Mini-Swe-Agent to Claude Code lifts Pass@1 and VRR across the board but shows no significant upward trend in RDR: killing semantic mutants needs repository understanding the harness cannot supply.
- Multilingual collapse: averaged over nine languages, Claude-Sonnet-4.5 falls from 42.60/79.30 (Python repair) to 33.33/58.33; C/C++ (memory management) and JS/TS (event-driven code) are worst. Global failure causes are issue misunderstanding and inconsistent environment/package versions.
- Generator-family bias was checked: swapping the Claude-4 mutation backbone for DeepSeek-V3.1 or Qwen3-Coder moves RDR within 1.5pp with evaluator rank correlations of 0.96 and 0.93.
- Caveats: mutants are synthesized errors, not observed agent failures; the v1 HTML is internally inconsistent about scale (intro says 1,664 mutants over 500 instances, §3 and the abstract say 2,636 over 800 — the latter are used here); and no cost or wall-clock figures are reported for the mutation pipeline.
Relevance
- WorldSmith — the closest published instance of the project’s core gate, one level down the stack. It scores an agent-built verifier against adversarially constructed near-miss solutions and reports both error directions in a single protocol, which is the shape WorldSmith needs for “verifier soundness first”. Two transfers are direct: the Self-Play admission filter (only keep attacks that already beat sampled verifiers) and the finding that the headline number moves ~30 points with attacker strength alone, which means a verifier-quality score is meaningless without a frozen attacker. What it does not give: it evaluates a single agent writing tests for an existing repo with a known golden patch, so there is no task authorship, no world to build, no annotator or dollar budget, and no team. The mutants are synthetic rather than drawn from real agent failures — for an ITSM-style environment the analogue would be trajectories that close the ticket with the wrong CI or the right fix on the wrong approval path, and nothing here says synthetic near-misses have that distribution.
- Environment Generation — a validity check for the verifier component specifically: mutation kill rate as the mechanical measure of whether an agent-authored grader discriminates, alongside the reproducibility and difficulty checks in STAGE-Claw.
- Red-Teaming — the mutation pipeline is an attack generator with an admission criterion (survive >3 of 10 sampled defenders), and the rule-based/few-shot/agentic ladder quantifies what attack sophistication is worth.
Notes
(Our take: to add after reading; drafted 2026-09-19, not yet discussed)