Summary
ComputerRL, from Tsinghua and Z.AI, is a framework for training desktop agents with end-to-end online RL. It has three parts: an API-GUI paradigm that lets the agent call application APIs as well as click and type, with the APIs themselves generated by an LLM workflow from example tasks; a rebuilt OSWorld infrastructure (QEMU-in-Docker VMs, gRPC multi-node clustering) that runs several thousand concurrent desktop environments; and fully asynchronous RL training. Rewards are rule-based verification of each trajectory, and training uses a step-level extension of GRPO. To fight entropy collapse during long runs, they introduce Entropulse: SFT on the successful rollouts collected during RL, then a second RL phase. AutoGLM-OS-9B, trained this way on GLM-4 9B models, reaches 48.9% on OSWorld, state of the art at the time.
Key points
- API-GUI action space. GUI-only agents rely on human-oriented interactions; pure API control is efficient but costly to build. An LLM workflow (requirement analysis → API implementation → generated test cases) creates application APIs with minimal human work, and the agent mixes API calls with GUI actions.
- The API-GUI paradigm more than doubles GPT-4o’s success: 26.2% vs 11.2% GUI-only on OSWorld, with the largest gains in Office (27.9 vs 6.2) and Professional (41.6 vs 14.3) tasks. AutoGLM-OS completes tasks in at most a third of the steps of the strongest baselines.
- Environment at scale. OSWorld’s VMs were CPU-heavy and unstable under concurrency, lost IPs under load and had no multi-node support; the rebuild uses lightweight containerized Ubuntu VMs, an AgentBench-style decoupled interface and gRPC clustering to run thousands of environments.
- Reward design: a rule-based verification function per task; successful trajectories give reward 1 to every correctly formatted contributing action, failed ones 0. Step-level GRPO groups all steps of a task’s trajectories.
- Entropulse: RL plateaus after hundreds of steps as entropy falls; SFT on the accumulated successful rollouts keeps performance (41.5% vs 42.0%) while raising entropy, and a second RL phase reaches 45.8% in the ablation (Qwen2.5-14B). Workflow tasks improve most (10.8% → 27.2%).
- Also introduces OfficeWorld, 180 office tasks drawn from SpreadsheetBench, PPTC and in-house Writer tasks.
Relevance
For computer-use environments, the API-GUI split separates how the agent acts from what the environment can observe and grade: the structured API layer is a cheaper surface for state checks than pixels. That bears on Environment Generation and on verifier design in Benchmark Validity. The infrastructure section is another account, after OSWorld-Verified, of how fragile OSWorld’s VMs are under RL-scale concurrency. Entropulse is a practical recipe for long RL runs in Reinforcement Learning - RL.
Notes
From paper club, 19 May 2026:
GUIs are meant for humans, and insisting that agents interact with websites purely through the lens of a human-oriented paradigm has its limitations. In future work in the CUA space, we can take lessons from this research by having an agent still solve a task using the UI, but by having the benchmark use the API/structured state as a task/verifier-mining surface. This will help us create tasks that are hard for the agent to complete, but easy for the env to grade.