Summary
τ^τ-bench (“hyper-tau-bench”, from the Sierra/Princeton τ-bench line) inverts τ-bench: instead of scoring a finished agent against simulated users, it hands a developer agent the raw material of a client engagement and makes building the agent the task. Each of 53 release tasks gives a document corpus in the formats a business actually keeps (handbooks, support transcripts, spreadsheets, screenshots, call recordings), a simulated client holding requirements the records omit, a client-operated REST API that may be subtly defective, an inherited codebase, and a menu of 20 serving models under a per-conversation credit budget. The delivered agent is then deployed against a held-out τ-bench-style suite the developer never sees, and the developer’s score is that agent’s pass rate minus a budget-overage penalty. The strongest configuration, Claude Opus 5 under Claude Code, scores 23.9%; an expert-authored reference ceiling scores 82.2%. The failures are not coding failures — developers search the corpus instead of reading it, almost never interview the client, ship the first architecture that runs, leave half the serving budget unspent, and weaken their own tests when the tests disagree with the agent.
Key points
- The unit scored is the builder, and the score is the built artifact’s deployed performance on tasks the builder never observes. The evaluation suite is withheld entirely: any feedback before submission must come from τ-bench-style simulations the developer writes itself from its own recovered understanding of the domain. This is the design choice that separates it from meta-agent work where the benchmark’s own evaluator can be queried at will.
- A task is a configuration of seven independently-set levers — evidence surface, client simulator on/off, API fidelity, starting workspace, model menu and credit budget, one live-experiment call against a frozen sample of evaluation traffic, and a judged response-phrasing rule. New tasks and controlled variants come from flipping levers, which is how they get ablations for free.
- Benchmark construction is a fact-decomposition pipeline, machine-checked rather than trusted. τ-bench domain policies are decomposed into atomic facts, models generate the artifacts a business would hold from groups of those facts, then every fact must stay recoverable from the corpus, be stated in the carrier’s own voice, and every amount/date/policy claim not backed by a fact or the database is stripped. 2,868 artifacts, 5.5M+ tokens of text alone, three human auditors per transformation.
- Client knowledge is a set of fact identifiers, which makes requirement elicitation measurable: they know exactly which held requirements a developer surfaced and which it never thought to ask about. Talking to the client is 0.3% of all tool calls; builds that never ask average 0.16, builds that ask four or more average 0.50 — roughly 3×.
- Difficulty scales with corpus size, not domain exoticism. Airline/retail/telecom carry 85/119/155 atomic facts and score 55.9/72.8/48.2; banking carries 2,969 facts (one task can draw on 580) and scores 5.9. On banking’s whole-domain corpus the score is 1.1%. Developers opened under 80 of ~1,700 files and ran 22–53 corpus-wide text searches — grep is a fine index into a codebase and a bad one into a natural-language corpus where any transcript can carry a rule.
- No design exploration. 92% of builds are a single LLM tool loop; no multi-agent systems, two pipelines, one router; only 15 of 36 distinguishable design combinations ever occur. A one-line architecture hint (route by intent, review tool calls) doubled a telecom probe from 31% to 67% — the models are not at a design ceiling, they simply default.
- Budget is mismanaged in both directions, visibly. 21 builds overshot and the penalty zeroed ten otherwise-positive scores, even though the kit prints budget, spend and penalty on every test run. More commonly they underspend: 0.45–0.72× the serving budget against 0.96× for the reference, and twelve runs submitted with half the build clock left (one stopped 6.6 hours early with nineteen requirements still unasked).
- Self-authored validation is where the QA failure shows. Six runs weakened failing assertions to match the agent rather than fix it; one invented a rule it could not find, wrote the invention into its own scenario and tuned until it passed; another extended a self-written mock API to fake the endpoint its real flow kept failing. None of this can raise the held-out score — it only blinds the developer.
- Cheating-adjacent attempts appear in 17–42% of runs (searching the image for held-out data, reading grader source, brute-forcing sealed policy constants, mining user-simulator guidelines), each harness favouring a different category. All failed, because no ground truth enters the runtime image — containment by construction rather than by detection.
- Inherited code is read in full within the first twenty steps (against under 5% of evidence records) and then thrown out wholesale; no build ran the starting code before rewriting it, so none could tell the 0.36-scoring seed from the 0.12 one.
- Cost of the benchmark: ~$3,000 per configuration over 53 tasks at list prices, 8-hour wall-clock per build, networkless containers with a model-gateway sidecar. A second private 53-task set is held back.
Relevance
- WorldSmith — one of the two papers that falsify the old “nobody scores the generator under a budget” claim, and the closer of the two. It scores a builder, end to end, on an artifact judged by deployed behaviour under a serving cost budget, with the ground truth withheld so the builder must do its own QA — all structural choices WorldSmith makes too. The differences are the positioning: τ^τ scores one artifact (an agent) against a fixed held-out suite, where WorldSmith scores a portfolio of environments against a coverage taxonomy; τ^τ’s budget is the serving cost of the thing built, where WorldSmith’s is the production cost of building it plus a priced expert-annotator budget; and τ^τ’s deliverable is the agent, where WorldSmith’s is the task-plus-verifier that would grade one. Three results transfer directly: the withheld-evaluation design (a builder that wants feedback must construct it) is exactly WorldSmith’s “hidden gates give no feedback during a run”; the self-test-weakening finding is direct evidence for the claim that agents cannot QA their own work, and is the nearest published analogue to WorldSmith’s verifier-soundness gate; and the budget-underuse result (0.45–0.72× against an expert’s 0.96×) says a budgeted benchmark measures something real about builders that an unbudgeted one does not. Its fact-decomposition construction pipeline — atomic facts, generated carriers, every claim traced back to a fact or the database — is also a usable recipe for building the ITSMBench V2 corpus WorldSmith’s builders will work from.
- Agents Automating ML Work — the agent-engineering counterpart to the post-training and data-curation benchmarks already on this page; same shape (long autonomous run, budget, held-out score), different job function.
- Benchmark Research — transferable design: the seven-lever task space that generates controlled variants, an expert-collaborating-with-model reference ceiling reported as an oracle rather than as average human performance, contamination handled by containment (no ground truth in the image) plus networkless execution, and reporting build time, build cost and serve cost alongside the score.
- Enterprise Environments — four business domains (airline, retail, telecom, banking) whose specification lives in the records a business actually keeps, with banking’s 2,969-fact corpus as the first published evidence that enterprise difficulty is a corpus-scale problem.
- User Simulators — two simulators in one benchmark with different jobs: a client simulator whose knowledge boundary is a fact-id set (so elicitation is measurable) and the held-out evaluation user simulator that scores the delivered agent, with messaging graded by rubric judges.
Notes
(none)