Summary
STAGE-Claw is a four-stage pipeline in which one LLM agent authors a benchmark task and a second, independent agent validates it, before a third agent is evaluated on it. Given only a human-written task hint, the authoring agent (Claude-Sonnet-4.6) emits a complete instance ℬ = (q, E₀, G, R, V): a task prompt, an environment-construction guide, hidden ground truth, and an executable verifier. A checker agent — which never solves or edits the task — scores it on structure, reproducibility (rebuild the environment twice from clean and diff snapshots across files, calendar, reminders, notes, email), verifiability, and difficulty calibration, aggregating Pass/Fail/Blocked/Warn into a 100-point score with an admission threshold of 80; failures go back to Stage 1 with diagnostics, averaging 2.67 repair iterations. The result is 40 tasks in real personal-computing environments (file system, browser, terminal, calendar, email, reminders, notes), graded purely on whether the final system state is right, with LLM adjudication used only when the executable verifier itself fails to run. Across 11 frontier models the best is Claude-Opus-4.7 at 77.1 (80% first-pass) and the worst MiniMax-M2.7 at 47.5. Two diagnostics carry the argument: replacing real tool-state changes with textual simulations of them inflates scores by 4.8–6.9 points, and perturbing the agent’s persistent memory with noise / misleading / conflicting entries costs 23–34 points.
Key points
- Task construction is agent-authored but human-seeded: 40 real assistant scenarios are curated by hand as “task hint words”, and the authoring agent explores each hint, imagines several scenarios, picks one with sufficient complexity, and builds the instance. Required properties baked into the authoring prompt: multi-step, multiple tools or information sources, reconstructable from a clean state, and objectively state-checkable.
- Difficulty is authored deliberately from a menu of types — cross-source conflicts, hidden dependencies, noisy data, entity alignment, tool state updates, cross-tool consistency — and the checker agent verifies the task actually exhibits them. Difficulty calibration as a validation criterion, not a post-hoc measurement, is the unusual move.
- Reproducibility is enforced mechanically: build the environment twice from clean and compare snapshots across every tool’s state. This is what makes an agent-authored environment trustworthy, and it is the check ITSMBench’s deterministic ids and frozen clock are already reaching for.
- Cost of the automation, stated plainly: ~0.17–6.55 and 3–15 minutes. The authors explicitly position 40 tasks as a “high-quality pilot” because of this cost, and run each model once per task.
- Human involvement survives at both ends: humans write the hints, and human annotators audit every accepted task for scenario realism, task completeness, instruction clarity, ground-truth correctness, and evaluator-rubric alignment. The claim is only that authoring, repair, and checker validation are automated in between.
- The state-vs-virtual ablation is the paper’s strongest result and a direct argument for building real environments: swap real tool-state changes for textual simulations of them and DeepSeek gains +6.93, Qwen +4.83, driven by 7 tasks with pure execution failure (avg Δ +76.6) and 13 with a “real-state gap” (avg Δ +14.1). Output-only evaluation systematically overestimates agents by hiding invalid writes, missing artifacts, and tool side effects.
- Failure analysis (multi-label over non-passing runs): Tool Failure 95.4% — missing, incomplete, or incorrectly routed writes to calendar/notes/reminders/email, i.e. the agent produces plausible intermediate output and never lands it in the environment; Invalid Format 75.0%; Trap and Reconciliation 69.7%; Verification 59.2%. By tool class, Shell/Session is 79.8% of calls and fails on command construction (53.3%), File I/O fails on missing paths (59.5%), and every observed State/Messaging failure is a tool runtime error.
- Memory perturbation (4 tasks, Qwen3.5-Plus, requirements moved from the prompt into persistent memory): Noise −23.3, Conflict −30.0, Misleading −33.5. Small sample, offered as diagnostic, but it is a ready-made protocol for stress-testing an agent that carries state between tickets.
- Limitations the authors name: 40 tasks, one run per model-task pair; construction remains expensive despite automation; and scores partly reflect the harness, OS configuration, permissions, and runtime stability rather than the model.
Relevance
- WorldSmith — the cleanest published instance of the full author→validate→admit loop for benchmark tasks, and it gives the missing numbers: 2.67 repair iterations per accepted task, an 80/100 admission bar, ~$35–40 and 1–2 hours each, with all accepted tasks passing executable verification without LLM adjudication. The load-bearing design choice is that the checker is a separate agent forbidden from solving or editing the task, and that its criteria are mechanical where they can be (rebuild twice and diff) rather than judged. It also shows exactly where the human stays: writing the seed hints, and auditing realism and ground-truth correctness at the end — the two places where the machine has no reference to check against.
- ITSMBench — supports V2’s decision to grade on DB state rather than text, with a measured number for what output-only grading would cost in validity (+5 to +7 points of illusory performance, and +76 on tasks where the agent never executed anything). Its reproducibility check (build twice, diff snapshots) is a concrete addition to V2’s determinism controls, and its difficulty menu — cross-source conflicts, hidden dependencies, noisy data, entity alignment, cross-tool consistency — reads like a specification for ITSM tasks that span ServiceNow, Entra ID, Teams, and SharePoint. What it does not cover: no event logs, no coverage or realism argument beyond “a human wrote the hint and audited the result”, no user simulator, no multi-agent workflow, and only 40 tasks run once each. Its realism claim is about the tools being real, not about the work being representative — which is precisely the gap the ITSMBench log question is meant to close.
- Benchmark Research — several transferable design ideas: benchmark instance formalised as (prompt, initial environment, target state, rubric, verifier) with the last three hidden from the agent; validation as a scored, thresholded, repairable gate rather than accept/reject; the virtual-state ablation as a way to measure how much a grading scheme overestimates; and reporting cost, tokens, wall-clock, and tool calls alongside score.
- Environment Generation — home concept since 2026-09-10
Questions it raises for ITSMBench:
- The authoring agent starts from a human-written hint. In V2 the log could supply the hints — process variants weighted by frequency, straight out of BPI. Does that make the log’s role precisely “the hint distribution”, with everything downstream (environment, ground truth, verifier) agent-authored and checker-validated?
- Their checker validates reproducibility, verifiability, and difficulty — but nothing about realism, which is left entirely to human audit. That is the same split ITSMBench’s yardsticks assume (practitioner realism is the primary check on tasks). Is realism genuinely unautomatable, or is the log the missing machine-checkable proxy for it?
- At $35–40 and 1–2 hours per accepted task, an agent-authored V2 of a few hundred tasks costs real money and does not obviously beat hand-authoring. What has to be true — reuse of the environment across tasks, a cheaper checker, higher first-pass yield — for the automated route to win?
Notes
(Our take: to add after reading; drafted 2026-09-09, not yet discussed)