Summary
The first systematic analysis of DARPA’s AI Cyber Challenge (2023–2025), written from design documents, source code, execution traces, and interviews with the organizers and all seven finalist teams. In the August 2025 final, seven fully autonomous Cyber Reasoning Systems ran unattended for ~143 hours over seven phases against 53 challenge projects (48 scored, from 24 OSS-Fuzz C and Java repositories, containing 63 hand-crafted vulnerabilities plus 13 SARIF broadcasts), each team capped at 50,000 of LLM API credits. The paper reconstructs the scoring rules, taxonomises the CRS architectures, and then does the thing the scoreboard cannot: it independently re-annotates every vulnerability under ideal lab conditions to separate genuine capability from system failure. Two findings dominate. Ranking was driven mostly by system stability and submission accuracy, not technique — the winner scored 392.8 against 219.4 for second, three teams effectively stopped mid-competition, and the accuracy multiplier alone demoted one team from second to third. And automatic validation is not sound: patches that built, reproduced no PoV, and passed functional tests were semantically incorrect 37.7–45.6% of the time for the reference agents, and still 16–21% of the time for the best CRSs, a fact recoverable only by two security experts reading every patch.
Key points
- Budget rules. Per team: 50K LLM credits for the whole 143-hour final, i.e. budgets of hundreds of dollars per challenge, deliberately set so “resource availability would not limit technical exploration.” Teams self-provisioned and self-managed their clusters (the semifinal’s constrained shared sandbox was dropped once the field narrowed to seven). No team exhausted either quota. ~94% of LLM spend went to Anthropic and OpenAI; LLM spend rank tracked final score almost exactly, with only two teams flipping order; on score-per-$K, the leaders reorder.
- Scoring formula. Points per submission are set by developer time saved or wasted: Patch [3,6], PoV [1,2], SARIF assessment [0.5,1], Bundle [−7,+7] — bundles score both extremes because correct linkage saves the most triage effort and wrong linkage wastes the most. Two modifiers sit on top: time-decay (full points immediately, half at the deadline) and the accuracy multiplier, which scales a CRS’s per-challenge total by its accuracy rate on that challenge.
- The accuracy multiplier is deliberately non-linear, and it decided the podium. Its shape “balances technique exploration with practicality”: 90% accuracy → negligible penalty, 50% → 6% reduction, 40% → 13% reduction. High accuracy is nearly free, so the gate only bites on genuinely sloppy submitters. It cost the two worst-accuracy teams −16.3 and −13.5 points; for one of them the pre-penalty score beat second place and the penalty dropped it to third by 8.7 points. Best-in-class per-submission penalty was −0.1.
- The soundness result. Reference patch agents (MultiRetrieval and Claude Code) were validated automatically — build, PoV reproduction, functional tests — and then cross-validated by two security experts. CC 20/53 (37.7%) and MR 26/57 (45.6%) of patches passing all automatic checks were semantically incorrect; the best CRSs, which add multi-PoV validation, post-patch fuzzing, or LLM reflection, still land at 16–21%. Failure taxonomy: wrong root cause (suppressing the symptom, e.g. hard-coded iteration caps for timeout bugs), incomplete fix (patching the crashing path only), functionality deviation invisible to the functional tests, newly introduced bugs, and missing domain knowledge.
- Independent annotation as the way to read a leaderboard. Because a low score can mean a capability gap or an outage, the authors re-ran two representative techniques per vulnerability under competition-level resources and contamination-free models (parallel fuzzing, 16 cores × 6 h per harness, 8,906 CPU-hours; MR at 119.53 LLM + one person-week). Result: roughly half the challenges fall to a single off-the-shelf technique (PF 34/63 PoVs, MR 31/63 and CC 33/63 patches), and “a CRS that reliably applies annotation techniques in real-world conditions would rank among the top three.” The benchmark rewards real-world coverage and reliability, not difficulty.
- Reliability beat technique. Stability accounts for most of the score gap: one team’s master scheduler OOM’d and thereafter emitted 1,200+ PoVs against a single vulnerability; two teams plateaued after phase 3–4, plausibly on a single 1 TB-of-disk Wireshark challenge; even active teams lost points to submission bugs, misconfigured LLM budgets, and silent whole-language failures.
- Design fixes carried from semifinal to final, worth copying: the self-PoV requirement was dropped (patches were scored against all teams’ PoVs) because it penalised systems strong at patching but weak at discovery; per-repository challenge onboarding was replaced by OSS-Fuzz; three unscored exhibition rounds on a mirror challenge set let teams and organizers debug the process before scoring began.
- Bundling worked (91/99, 92% accurate) but only because PoV-based patch generation pairs evidence naturally; of the 8 wrong bundles, 7 were wrong because the patch failed, not because the linkage did. Patch quality is the bottleneck everywhere.
- Lessons the authors draw for future competitions: design telemetry for post-hoc analysis (intermediate outputs, decision traces, environment snapshots) rather than only live scoring; run organizer-built baseline CRSs alongside the teams as reference points; add a resource-limited track; add a sub-track for open-source models, since nothing in the current incentives rewards fine-tuning over prompting frontier models.
- Limitations the authors name: the taxonomy reads submission-version code as authoritative; the annotation is an approximation under one representative configuration (three runs, union taken as a lower bound); and there is no ablation, because migrating seven heterogeneous CRSs onto one platform was out of scope.
Relevance
- WorldSmith — the closest thing to a full-scale, real-money field test of the project’s own setup: agent teams building complex artefacts under a stated dollar budget and a stated expert-annotator budget, scored by a formula that has to punish wrong work without discouraging exploration. Three things transfer directly. (1) The accuracy multiplier is a ready-made design for the scoring function: penalise per-challenge accuracy non-linearly so that 90% is effectively free and 40% costs 13% of the total — it lets a team try things while making bursts of junk submissions expensive, and here it changed the ranking. (2) The soundness number is the headline for verifier-first design: a verifier consisting of build + reproduce + functional tests admits 38–46% semantically wrong work, and the best engineered pipelines only get that to 16–21%. Any WorldSmith claim that agent-built verifiers are sound has to be measured against a human semantic review, exactly as here, and the cost of that review is published (one person-week and ~$120–220 of LLM spend for 53–57 patches, two experts). (3) The independent-annotation protocol answers “was this a capability gap or an outage?” — re-run a fixed reference technique on every task under matched resources, and score agents against that reference rather than against each other. The caution for the project: at this scale, reliability and budget discipline dominated the leaderboard, so a WorldSmith harness that does not separate system failure from capability will measure ops engineering.
- Benchmark Research — a rich catalogue of competition-design decisions with their rationale and their observed effects: evidence weighted by developer time saved, symmetric ±7 bundling to price triage effort both ways, time-decay, the non-linear accuracy multiplier, unscored exhibition rounds, and the semifinal→final corrections (scoring patches against all teams’ PoVs; adopting OSS-Fuzz).
- Benchmark Validity — a first-class validity audit from inside a live competition: what scores conceal (stability and strategy confounded with capability), the false-positive rate of an automatic evaluator measured against expert review, and the telemetry gap that makes root-cause analysis impossible after the fact.
Notes
(Our take: to add after reading; drafted 2026-09-19, not yet discussed)