Summary
TheAgentCompany (CMU) builds a self-hosted software startup — GitLab, ownCloud, Plane, RocketChat, plus a sandboxed Docker workspace with browser, editor and terminal — and evaluates agents on 175 tasks that a digital worker at such a company would do. Its distinguishing feature is simulated colleagues: LLM-backed NPCs built on Sotopia, each with a name, role, responsibilities and project affiliations, whom the agent must message on RocketChat to obtain information deliberately withheld from the task description. Tasks are scored against weighted checkpoints implemented as Python evaluators over environment state and agent trajectory, with LLM-as-judge used as a fallback on 51 of 175 tasks (29%), yielding a full-completion score and a partial-completion score. The best agent, OpenHands + Gemini-2.5-Pro, completes 30.3% of tasks (39.3% partial) at ~27 steps and $4.2 per task. Task categories were chosen from the O*NET occupational database weighted by headcount × median salary, but all 175 tasks were hand-built by 20 people over 2 months, roughly 3,000 person-hours.
Key points
- Simulated colleagues are the paper’s contribution to simulator design: NPCs are profile-driven (name, role, responsibilities, channel access) rather than instruction-prompt-driven, backed by Claude-3.5-Sonnet, and reachable by DM or channel. 41 of 175 tasks involve them. The authors report only one role-play error found across all 41 tasks, and ~$0.024 per NPC interaction — but that audit is by the authors themselves, with no comparison to real coworker behaviour.
- Checkpoint scoring rather than binary end-state: each task decomposes into weighted checkpoints of three kinds — Action Completion (tools used, URLs visited, data collected), Data Accuracy (correctness of the deliverable), and Collaboration (did the agent message the right colleague or share the output). Partial score = 0.5 × (points earned / total) + 0.5 × full-completion indicator, so partial progress is measurable but full completion is doubly rewarded.
- Evaluators read both environment state and trajectory (browsing history, action sequences), so this is one of the few enterprise benchmarks with genuine trajectory-level checks rather than pure end-state grading. LLM judging is a fallback after deterministic keyword matching, and the framework overrides a bad intermediate-step judgement if the final deliverable is sound.
- Task selection is grounded in an external labor statistic, not introspection alone: O*NET 29.1 job categories ranked by headcount × median salary, physical-labor categories dropped, settling on a software company as the setting that covers the remaining high-value digital roles. The authors explicitly warn the benchmark does not exhaustively cover any occupation and should not be read as job-level automation risk.
- Curation cost: 20 CS students, engineers and PMs over 2 months, ~3,000 person-hours, some tasks over 10 hours each. Quality control is screenshot proof that the evaluator awards full credit on a correct run, unit tests for evaluators, code review by lead authors, and a final independent pass checking that checkpoint weights are consistent across tasks.
- Results by platform: RocketChat (social interaction) and ownCloud (complex office web UI) are the worst. Results by department: Data Science, Admin and Finance score lowest while SDE scores highest — the authors argue the perceived-difficulty ordering for humans is inverted for agents, because coding data is public and administrative/financial work is private.
- Failure modes worth naming: agents lack follow-through in conversation (asked who to talk to next, got an answer, declared the task done), get stuck on trivial browser obstacles (a closable welcome popup), and “deceive themselves” by inventing shortcuts — in one case renaming another user to the name of the person it could not find.
- A single-agent scaffold (OpenHands CodeAct, 8.6% with GPT-4o) beats a multi-agent one (OWL RolePlay, 4.0% same model), because the delegating agent loses context and the sub-agent cannot resume prior progress.
- Limitations the authors name: tasks skew toward the programmatically checkable, no human baseline was collected (cost), only two agent scaffolds, and task content came from co-author introspection, “which may result in some disconnect with actual tasks performed in enterprise settings.”
Relevance
- ITSMBench — the reference design for V2’s track 1: an agent working alongside simulated colleagues who hold information the agent must ask for, plus Collaboration as a first-class scored checkpoint category rather than an afterthought. Its weighted-checkpoint partial score is a live alternative to V1’s binary pass, and its evaluators reading trajectory as well as state answer part of the “what trajectory checks are needed” question. It does not cover log grounding (the company, its employees and its data are invented by co-authors, who flag the resulting disconnect from real enterprise work), does not have a user simulator in the tau-bench sense (colleagues are peers to consult, not a requester whose intent must be elicited and served), and has no multi-agent track under test — its one multi-agent scaffold is a baseline that loses.
- WorldSmith — the cost ledger to beat on the hand-built side (3,000 person-hours for 175 tasks) and, on the automation side, the O*NET headcount × salary weighting is a reusable trick for choosing what a benchmark should cover without asking a domain expert first.
- Web Agents — the agent works through real web UIs (GitLab, ownCloud, Plane, RocketChat) via a text browser, and browser incompetence is a named top failure mode, with a text-vs-visual-browsing comparison between OpenHands and OWL.
- Benchmark Research — checkpoint decomposition with independently-audited weights, deterministic-first-LLM-fallback judging with a documented override rule, and grounding the task taxonomy in an external occupational database.
- Enterprise Environments — home concept since 2026-09-10
Questions it raises for ITSMBench:
- Their NPC audit is one error across 41 tasks by the authors’ own inspection; Mind the Sim2Real Gap found simulators systematically over-cooperative against 451 humans. If BPI logs record what colleagues actually did (reassignments, waits, escalations), can they be used to audit an ITSM colleague simulator the way TheAgentCompany could not?
- Checkpoint weights were assigned by the task author and then checked for cross-task consistency by an independent person. For V2, could BPI activity frequencies set those weights instead of a reviewer’s judgement, so partial credit is log-shaped?
- Admin and Finance are the worst-performing categories precisely because that work is private and untrained-on. ITSM ticket work is the same shape. Does that make it a good discrimination axis for V2, or a warning that V2 will bottom out near zero for a while?
Notes
(Our take: to add after reading; drafted 2026-09-09, not yet discussed)