Summary
WebArena is a self-hosted, reproducible web environment with four fully functional sites built on the open-source software behind real ones — e-commerce (OneStopShop), a Reddit-style forum, GitLab, and a CMS admin panel — plus a map, calculator, scratchpad and documentation/Wikipedia as tools and knowledge bases. It ships 812 long-horizon tasks phrased as high-level natural-language intents, each with a programmatic evaluator that checks functional correctness (database state, page content, or fuzzy answer match) rather than matching a reference action sequence. Baseline few-shot LLM agents do poorly: the best GPT-4 agent reaches 14.41% end-to-end success against 78.24% for humans. The gap is attributed to missing exploration and failure-recovery abilities, and the benchmark became the standard hard test for web agents.
Key points
- Realism + reproducibility: Docker-hosted clones with real data, gym-style API, deterministic transitions; avoids CAPTCHAs and drift of live sites.
- Observation: accessibility tree (or DOM/screenshot); action space of browser primitives (click, type, scroll, tab ops, goto URL).
- 812 tasks from 241 templates across the sites, including cross-site tasks and infeasible tasks; outcome-based evaluation via
exact_match,must_include,fuzzy_matchand state checks. - Results: GPT-4 + CoT 14.41% success; GPT-3.5 ~6–9%; PaLM-2 lower; humans 78.24%. Later work (BrowserGym) raised GPT-4 to ~25% with richer observations, so the number is agent-design sensitive.
- Failure analysis: agents stop early, repeat actions, cannot recover from mistakes, and mis-ground intents to UI.
Relevance
- Web Agents — the canonical realistic web-agent benchmark; defines the functional-correctness evaluation style and the human-vs-agent gap that WorkArena and WorkArena++ port to enterprise software and that Agent-E tries to close.
- Benchmark Research — template for how to build a reproducible environment with programmatic validators.
Notes
(batch ingest; no notes yet)