Summary
This is a benchmark-reporting paper, not a benchmark: it argues that work-facing agent benchmarks describe themselves by tasks, environments and metrics while leaving four work-facing choices implicit, and proposes recording them explicitly as represented activity (what work the benchmark stands for), tested setting (materials, tools, role, workflow state under which it is performed), required work product (the object left for review, filing, execution or continuation), and evaluated result (which part of that product the metric actually checks). The gap between the last two is the whole point: a long tool-using episode can still be scored on a short final answer, so the score is evidence about a smaller thing than the work claim. To make the first field reportable across occupations, the authors distil 18 cross-occupation work activities from 18,796 O*NET 30.2 task statements (filtered to Job Zones 3–5 → 12,611, scope screen → 12,464 reporting statements, stricter atlas screen → 8,372, profession-neutral rewrite, embed, UMAP+HDBSCAN → 108 dense groups → 18 labels), validated for semantic coherence, algorithm sensitivity, ESCO legibility and human interpretability. Applying the four fields to GDPval, OfficeQA Pro and APEX-SWE shows all three “fit the representation but end at different points in the work”: GDPval stops at deliverable quality before any grant-workflow linkage or audit trail, OfficeQA Pro checks a final number with no reviewable source-and-calculation record, and APEX-SWE checks that an S3 object exists with the right path and schema while never checking that its rows correspond to the source file.
Key points
- The load-bearing distinction is required work product vs evaluated result. A metric provides evidence only for what it checks; if the benchmark scores a final answer, a rewritten paragraph, or a simulator success state, everything else the receiving workflow needs — provenance, traceability, handoff information, role compliance — is untested even though the episode looked like the whole job.
- The 18-activity inventory is the concrete reusable artifact, and ITSM work sits squarely inside it: troubleshooting (902 statements; “diagnose a problem and apply or recommend a remedy”; settings IT support and software debugging; common proxies log analysis, state repair), record-keeping (882; canonical organizational records; proxies extraction, database update), coordination (603; align actors, dependencies, timing; proxies scheduling, task routing), procedure-execution (735; carry out a prescribed procedure to a standard; proxies policy following, checklist execution), emergency-response (212; stabilize or triage an acute incident under time pressure; proxies triage classification, escalation decision), inspection (963), representation (615; act on behalf of an organization; proxies role-play dialogue, policy-constrained response), advising (403), rule-enforcement (30).
- Existing benchmark labels sit at three unhelpful levels and the activity label is proposed as the missing middle one. Occupation/domain labels span activities with different roles, products and standards, so a benchmark can sample a sliver of an occupation and keep the label; component labels (retrieval, summarization) isolate a behaviour without saying what activity it serves; agent-task labels bundle search, editing and calculation into one episode so an aggregate score cannot say which activity was required for the product, which was incidental to the environment, and which one failed.
- Tested setting is deliberately more than the execution environment: the environment says which services, files, tools, credentials and states exist; the tested setting adds which materials are authoritative, what role and authority the agent has, and which workflow stage the episode represents. The theoretical backing is situated action and distributed cognition — the same activity is a different task under different materials and role.
- The APEX-SWE finding is the sharpest concrete example of an under-checking verifier: the tests confirm execution, S3 path, schema and non-empty output, so a script that uploads the right shape of data from the wrong source passes. The paper calls this “source-to-state fidelity remains only partially checked” — a class of hole a state-diff verifier will not catch by construction.
- The worked contrast in Figure 2: the same activity (analysis + record-keeping) over the same corpus is either “answer-scored document analysis” (evaluated result = final number) or “work-product-aligned evaluation” (evaluated result = answer + sources + extracted values + calculations + assumptions). Same task, same environment, different claim supportable.
- Scope discipline the authors impose on themselves: the representation is about benchmark scope and the relation between a claim and its evidence; sampling, grader reliability, robustness and fairness are explicitly separate questions and not addressed here.
- Limitations named: one released instance per benchmark, no stratified sampling and no second independent coder, so the case analyses are demonstrations not measurements; O*NET task statements describe occupational work at a general level and inherit an occupational information system’s limits; the interpretability check used two validators over 18 blinded cards; procedure-execution and record-keeping have the weakest cluster boundaries, with procedure-execution sometimes read as navigation; the activity counts carry no labour-market prevalence or AI-deployment interpretation.
Relevance
- ITSMBench — this is the vocabulary for the practitioner-realism yardstick, which V2 currently states as “a domain expert says this captures the work activities for this job persona” without a way to enumerate those activities. The 18-label inventory gives an occupation-independent list to hold up against BPI 2013/2014, and the ITSM persona plausibly decomposes into troubleshooting, record-keeping, coordination, procedure-execution, emergency-response and representation — which is testable: map BPI activity codes to those labels, then map V2’s task set to them, and the coverage gap becomes a table rather than an opinion. The required-work-product field is also a direct challenge to V2’s grading: ITSM work leaves a ticket that a downstream human must be able to review (correct category, resolution notes, worklog, audit trail, correct handoff), and a DB-state verifier that checks status and assignment group is checking the state change but not the reviewable product. What it does not cover: no verifier mechanics, no task generation, no simulator, no automation, no numbers on any agent — it is a reporting schema, and its own case studies are single instances.
- Benchmark Research — the four-field record is the cleanest available answer to “what does this benchmark license you to claim”, usable both prospectively (design) and retrospectively (audit an existing suite), and it sits alongside the phenomenon→task→metric→claim chain it cites. The 18-activity inventory plus ESCO mapping is a reusable coverage vocabulary for the whole wiki.
- Benchmark Validity — home concept since 2026-09-10
Questions it raises for ITSMBench:
- If we score V2 against the 18 activities, which ones do the BPI logs actually contain and which does V2 test? My guess is V2 is heavy on record-keeping and procedure-execution, thin on coordination and emergency-response, and absent on representation — and the multi-agent tracks are precisely a coordination bet, so this is a way to justify them.
- What is V2’s required work product as opposed to its evaluated result? If the answer is “a ticket a service-desk lead would accept without rework”, the verifier is under-checking today in exactly the APEX-SWE way, and the gap (resolution notes, provenance, handoff completeness) is where a rubric or LLM judge earns its place on top of the state gate.
- Does the tested-setting field give us a cheap way to say what the logs supplied and what came from elsewhere? Materials, role, authority and workflow stage are exactly the four things BPI 2013/2014 can ground, which would make the project’s “ledger of what came from logs” concrete rather than narrative.
Notes
(Our take: to add after reading; drafted 2026-09-09, not yet discussed)