Summary
WorkArena (ServiceNow Research) is a benchmark of 33 task types (19,912 instances) on a real ServiceNow enterprise instance — filling forms, sorting and filtering lists, navigating menus, searching a knowledge base, using dashboards and the service catalog — each with a Playwright oracle and a validator that inspects the database. It also introduces BrowserGym, a gym environment for web agents that unifies MiniWoB, WebArena and WorkArena under one API with multimodal observations (HTML, accessibility tree, screenshot, set-of-marks) and a flexible action space, plus chat-based human–agent interaction. GPT-4o solves about 43% of WorkArena, GPT-3.5 6% and Llama-3 18%; the same BrowserGym agent nearly doubles GPT-4’s WebArena score (14.4% → 25.4%), showing how much observation and action design matters.
Key points
- Enterprise UIs are a distinct challenge: very large DOMs, non-standard HTML, complex widgets; tasks are simple for humans but hard for agents.
- BrowserGym: shared environment so results across MiniWoB/WebArena/WorkArena are comparable; observation/action ablations show accessibility tree + set-of-marks + high-level actions drive big gains.
- Results: GPT-4o 42.7% on WorkArena L1, GPT-3.5 6.1%, Llama-3 17.9%; MiniWoB ~71%; WebArena 23.5–25%.
- Large open/closed gap and strong scale dependence; long-context reasoning over HTML/AXTree “seems to emerge only in very large models.”
- Each task is certifiable (oracle + validator), which also makes it a source of ground-truth traces.
- Named future work: compositional workflows needing retrieval, memory, perception and reasoning — delivered as WorkArena++.
Relevance
- Web Agents — the enterprise-software counterpart to WebArena and the origin of BrowserGym, the environment most later web-agent work (including Agent-E’s peers and WorkArena++) runs on.
- Benchmark Research — an example of a benchmark whose backend is a real SaaS platform rather than a clone.
Notes
(batch ingest; no notes yet)