Summary
APEX–Agents (the AI Productivity Index for Agents) asks whether agents can do long-horizon, cross-application work created by investment banking analysts, management consultants and corporate lawyers. It is built in three steps: experts on Mercor’s marketplace play roles on a 5–10 day project to build a data-rich “world” from a scenario, then write realistic tasks against that world’s files, then agents are given the world’s files and nine applications (63 tools) to execute the tasks. The benchmark has 480 tasks across 33 worlds (about 166 files per world), each graded by expert-written rubric criteria with a Gemini 3 Flash judge that agrees with human labels 98.5% of the time. The best agents are Gemini 3 Flash (24.0% Pass@1) and GPT-5.2 (23.0%), statistically tied; Pass@8 rises to about 40%, and every agent scores zero on at least 40% of its runs. The dataset is released under CC-BY.
Key points
- Worlds are built by humans doing the job, not by generating data. Each world starts from an expert-written project scenario; experts take in-world roles (partner, associate, analyst, one playing the customer), exchange email and chat, do research and produce real deliverables. 22 of 33 worlds are fully fictional, 9 put real companies in fictional scenarios.
- Tasks are adversarially iterated against frontier agents. Task writers could pull outputs from GPT-5, Claude Opus 4.5 and Gemini 3 Pro while drafting, and tasks went through multiple review rounds. 422 tasks want a console answer; 58 require creating or editing a spreadsheet, document or presentation.
- Grading is a rubric of 1–10 must-have criteria per task (mean 4.06), all required to pass. The judge sees the prompt, the output and a log of state changes, not the trajectory; on 747 human-labelled criteria it reaches 98.5% accuracy, 1.9% false negatives and 1.3% false positives.
- A baselining study had independent experts redo 20% of tasks from scratch, finding a minor problem with prompt, rubric or metadata in 10% of them and cascading the fixes.
- Results: Gemini 3 Flash 24.0%, GPT-5.2 23.0%, Claude Opus 4.5 and Gemini 3 Pro 18.4%, the two open models under 5%. Pass@8 is about 15 points above Pass@1 (agents can do the work but inconsistently); Pass^8 tops out at 13.4%. Mean criteria score (39.5% best) is a denser signal than Pass@1.
- Failure shape: all agents score zero on at least 40% of runs; Kimi K2 Thinking times out (250 steps) on 29.8% of trajectories; file-output tasks score lower than console tasks; 36 trajectories delete files nobody asked to delete, 21 of them from GPT-5.2.
- 256 experts contributed (mean 12.9 years’ experience); no cost or hours for building the worlds are reported.
Relevance
- WorldSmith — the human-built end of the spectrum WorldSmith wants to reach synthetically: a world, tasks and rubrics all authored by domain experts, with a measured judge and an independent re-execution study. Its frontier pass rate (~24%) sits inside WorldSmith’s [0.2, 0.8] band. It reports no cost of building a world, which is the number WorldSmith needs most.
- Enterprise Environments — professional-services worlds (files, mail, chat, spreadsheets, decks) rather than a single CRM or ticketing system; a different shape of enterprise environment from EnterpriseBench Corecraft.
- World Models — the “world” here is a static snapshot of files and apps produced by people, not a model of dynamics; useful contrast with the code-driven worlds of Agent World Model.
- Benchmark Research — Pass@1 / Pass@8 / Pass^8 / mean score reported together, and a judge validated against human labels with false-positive and false-negative rates.
Notes
From paper club, 27 February 2026: Mercor’s APEX and Surge’s CoreCraft are benchmarks testing how well SOTA models can operate in realistic enterprise environments, doing human-created long-horizon tasks in fields like investment banking, consulting and corporate law. To automate the creation of post-training environments and tasks at this quality, we need a world model complex enough to create tasks as complex as the human-generated ones in APEX and CoreCraft.