Summary
Fifteen months after releasing OSWorld (April 2024), XLang Lab collected about 300 issues from institutions running the benchmark, despite more than 400 person-hours of checks before the first release. Their lesson: “providing reliable rewards consumes more human resources than we imagined.” The causes were uncontrollable environment uncertainty (websites changing structure, anti-crawling and IP blocks, fragile timing-dependent setup), incomplete task annotation (ambiguous instructions and multiple valid solution paths the evaluators did not accept), and decentralized evaluation that gave little incentive to report errors. A ~10-person team spent about two months fixing 300+ items, preferring evaluator fixes over task changes, and moved the infrastructure to AWS so a full run takes minutes instead of 10+ hours. On the rerun, the best system (CoACT-1) reaches 60.76% against an estimated ~72% for humans. Source is a blog post (28 July 2025), not a paper.
Key points
- Why the signal decays: real websites change structure and content, add CAPTCHAs, bot detection and IP blocks (e.g. Steam timeouts, NBA.com geo-restrictions); apps need specific start states and timing; files hosted on Google Drive got rate-limited.
- Why correct agents fail: ambiguous instructions (“purple background”, “fill all blank cells”) and valid alternative solutions the ground truth did not anticipate (CSV concatenation with or without headers, VS Code customization via extensions, multi-app workflows, equivalent spreadsheet formulas).
- Fixes favour the evaluator, not the task: “Minimal invasive changes: Prioritize evaluation fixes over task modifications.” Concretely: fuzzy URL matching (
is_expected_active_tab_approximate), 0–1 scoring for document comparison instead of binary, aproxyfield for anti-crawling sites, swapping heavily protected sites for functional equivalents, fixed single-IP assignment for tasks requiring human verification. - Infrastructure: from VMware images to QEMU-in-Docker to AWS; VM images compressed from 50 GB to 25 GB; evaluation parallelized from 10+ hours to minutes; task files moved from Google Drive to Hugging Face.
- What they would do differently: proactive health monitoring of tasks, simulated environments for tasks prone to external change, both centralized and decentralized evaluation, and far more budget for maintenance.
- Rerun results: agentic frameworks with reasoning models lead (CoACT-1 60.76%, Agent S2.5 w/ o3 56.0%, GTA1 w/ o3 53.1%); Claude 4 Sonnet 43.9% and UI-TARS 40.0% among models; human performance estimated at ~72%.
Relevance
The computer-use counterpart to SWE-Bench Verified: same diagnosis (benchmark-side false negatives), but the fixes land on the evaluator and the environment rather than on dropping tasks. For Benchmark Validity it is first-hand evidence that a live-web benchmark’s reward signal drifts over time unless it is maintained, and that drift-resistant environments (proxies, simulated pages) are part of verifier design. ComputerRL reports its results on OSWorld-Verified.
Notes
From paper club, 19 May 2026:
The team fixed the verifiers instead of the tasks, because changing a task changes the dataset. Nowadays, OSWorld-Verified is starting to get saturated as well. This is where we plan to provide better (“verified”) versions of existing CUA gyms and datasets, especially for issues that are known and remain hard.