Summary
Collecting trajectory data for GUI agents usually means either paying humans to record demonstrations or having a model execute pre-defined tasks, which is expensive or gives no guarantee of quality. OS-Genesis inverts the order: an agent first explores Android and web environments with no task in mind (rule-based, with GPT-4o only filling input fields), then an annotation model turns each observed interaction into a low-level instruction and lifts those into high-level tasks — reverse task synthesis. The high-level tasks are executed to produce full trajectories, which a Trajectory Reward Model scores on completion and coherence rather than discarding incomplete ones. Trained on this data, Qwen2-VL-7B nearly doubles AndroidWorld success over task-driven synthesis, from 9.82% to 17.41%, with gains on AndroidControl and WebArena as well.
Key points
- Interaction first, task second. Exploration is human-free and task-free; tasks are inferred from what the exploration uncovered, so every synthesized task is grounded in functionality the environment actually has.
- Reverse task synthesis: an annotation model (e.g. GPT-4o) writes a low-level instruction for each (state, action, next state) triple, then turns these into high-level tasks; the whole process runs without human intervention.
- Graded, not binary, trajectory filtering. A Trajectory Reward Model scores trajectories on completion and coherence and samples them in proportion to reward, on the argument that incomplete trajectories still hold useful exploration of the GUI.
- AndroidWorld success: 15.18% (InternVL2-4B), 16.96% (InternVL2-8B) and 17.41% (Qwen2-VL-7B) with OS-Genesis data, against at most 9.82% for task-driven baselines; GPT-4o zero-shot with the M3A agent is 23.70%.
- On AndroidControl, OS-Genesis beats both task-driven baselines on high-level success rate for all three base models (e.g. 44.54% vs 39.36% for Qwen2-VL-7B); action-type match on low-level tasks is ~90% for every trained method, OS-Genesis included.
- On WebArena, overall success with OS-Genesis data is 7.88%, 9.96% and 10.79% for the three base models, above both task-driven baselines in each case but below GPT-4o zero-shot (16.25%). Training sets are small: 1K trajectories averaging 6.4 steps.
Relevance
- Environment Generation — an early, concrete instance of deriving tasks from an environment rather than the reverse; the inferred task is verifiable because the end state was observed before the task was written.
- Web Agents — trajectory synthesis for mobile and web GUI agents, evaluated on WebArena.
- WorldSmith — reverse synthesis is one way to get tasks with a known-reachable goal state out of an agent-built environment, which bears on task validity.
Notes
From paper club, 11 February 2026:
Most UI agent scaling is currently throttled by the cost of human time. OS-Genesis took a much more scalable path by using Reverse Task Synthesis. Instead of recording a user completing a task, they started from a terminal state and worked backwards to hypothesize the intent. The intuition here is that it’s easier to verify a goal if you already have the state.
For our work on CUAs, the reverse synthesis logic is a potential fix for the ground truth problem. If you can automate the trajectory, you can train on environments that humans haven’t manually mapped out yet.