Summary
Computer-use research has mostly stayed on short tasks over a few low-value applications, because building an environment around complex software takes a lot of human effort. Gym-Anything treats environment creation itself as a multi-agent task: a creation agent writes setup scripts, downloads real data and configures the software while producing evidence of correct setup, and an independent audit agent checks that evidence against a quality checklist, with learnings kept in a shared memory. Applied to 200 applications chosen from a GDP-grounded occupational taxonomy, it yields CUA-World — over 10K long-horizon tasks with train and test splits — plus CUA-World-Long, a benchmark of one task per application that often needs more than 500 steps. Tasks are scaled by propose-and-amplify: an agentic proposer writes a few hard seed tasks per application and a cheaper model amplifies them. Frontier models score low, especially on CUA-World-Long, and distilling trajectories into a 2B model beats models twice its size.
Key points
- Software selection is grounded in economic value: from U.S. occupational data, per-software GDP is estimated, candidates are filtered to what can be sandboxed, and 200 applications are picked across tiers balancing GDP and diversity (healthcare, education, engineering, enterprise and more).
- Creation–audit loop. Each agent is Claude Opus 4.5/4.6 run through Claude Code. The creation agent’s claims about what it did are not trusted on their own, so it must produce evidence (screenshots, logs) that the audit agent checks against a checklist; findings go back for correction over several iterations. A shared memory — a growing directory of files — lets parallel agents reuse each other’s discoveries.
- Propose-and-amplify task scaling: an agentic proposer (Opus 4.5/4.6 with computer-use tools) writes 5 difficult seed tasks per application, judged on realism, difficulty and diversity; a non-agentic Gemini 3 Pro then generates 75 more per application from those seeds, sequentially and with semantic deduplication.
- CUA-World-Test: Gemini 3 Flash passes 22.6% and Kimi K2.5 12.8%; Qwen3-VL-2B and -4B pass 1.6% and 3.9%. Distilling Kimi K2.5 trajectories into Qwen3-VL-2B lifts it to 4.4%, above the 4B model.
- CUA-World-Long, at 500 steps and a 18 a trajectory.
- The audit idea also works at test time: a separate VLM reviewing completed trajectories and saying what remains lifts Gemini 3 Flash on CUA-World-Long from 11.5% to 14.0%.
- Compared in the paper against prior CUA benchmarks including OSWorld (369 tasks over 9 applications).
Relevance
- Environment Generation — the creation agent plus evidence-checking audit agent is a construct → inspect loop for whole software environments, and propose-and-amplify is a cheap way to scale tasks once one environment is trusted.
- WorldSmith — the clearest open example of agent-built environments at scale, with the audit step as an explicit quality gate; it reports pass rates but not how often audited environments were later found wrong.
- Enterprise Environments — GDP-weighted software selection is an alternative to picking applications by convenience, and covers enterprise systems directly.
- Benchmark Validity — low pass rates on long tasks may reflect task difficulty or verifier and setup defects; the paper’s own audit loop is the main defence.
Notes
From paper club, 5 May 2026:
In short, real-world tasks are still way more difficult than the manually fabricated ones most RL env companies are building today. Simply scaling envs is necessary but insufficient. You need to be scaling specifically high-quality training signal data (with better QA, better verifiers, and better task specification). The most important piece of this pipeline isn’t the scaling; it’s rigorous QA.
Gym-Anything gives us a way to generate highly realistic envs and tasks at scale, but this is not enough. To actually produce large model gains, we need to build a fully self-improving training system (with embedded QA rather than QA afterwards).