Summary
WorkArena++ extends WorkArena from 33 atomic tasks to 682 compositional ones built from the same ServiceNow building blocks, arranged in two new difficulty levels: L2 gives the agent an explicit multi-step workflow, L3 gives only a ticket describing the goal and requires the agent to work out the steps. Tasks are grouped by the skill they test — planning and problem solving, information retrieval, data-driven decision making, sophisticated memorisation, contextual understanding (including infeasible tasks) — and come with 10 fictitious companies and colour schemes for visual diversity, better database isolation for parallel evaluation, and oracle traces for fine-tuning. Results are stark: humans succeed on ~94% of L2 and L3 tasks while GPT-4o manages 3% on L2 and 0% on L3; open models score zero on both.
Key points
- Three levels: L1 = WorkArena atomic tasks (GPT-4o ~43%); L2 = compositional with explicit instructions (GPT-4o 3.0%, GPT-4o-vision 3.8%); L3 = compositional from a ticket, steps implicit (all agents 0%); humans 93.9% on both.
- Skills taxonomy: planning/problem-solving, information retrieval, data-driven decision making, sophisticated memorisation, contextual understanding (with infeasible-task detection).
- Task composition framework: workflows assembled from low-level building blocks with per-task oracle + validator, so thousands of ground-truth observation/action traces can be generated for fine-tuning.
- Visual diversity (10 companies, UI themes) to test generalisation; database isolation for parallel runs.
- Error analysis: agents lose track over long horizons, fail at retrieval from dashboards/lists, and struggle with memorisation across pages; all agents run through BrowserGym.
- Limitations named: no safety/security tasks, ServiceNow-only, no very-long-context models tested.
Relevance
- Web Agents — the hardest of the four benchmarks on the page and the one that separates UI manipulation (L1) from planning and reasoning (L2/L3); the near-zero agent scores are the gap the field is chasing.
- Benchmark Research — a model of skill-tagged compositional task design plus trace generation for training.
Notes
(batch ingest; no notes yet)