Summary
A fully automated pipeline for terminal-agent training data that starts from human-authored agent skills (from ClawHub and SkillMP) rather than seed keywords or GitHub repos. A skill already says what to accomplish, when it applies and how to execute it, so the task instruction, the environment blueprint, the evaluation criteria and an execution guideline can all be derived from it together instead of retrofitting an environment around a task. Skills are paired with personas, composed into skill teams and skill graphs, built into sandboxes through generate–verify–repair loops with pytest verifiers, and solved by a DeepSeek-V3.2 teacher that sees the skill-derived guideline. The result is 5,723 environments at about $0.17 each; SFT on those 5.7K trajectories (1.2% of Nemotron-Terminal’s 490.5K) takes Qwen3-32B to 31.5 Pass@1 / 43.8 Pass@3 on Terminal-Bench 2.0, +4.5 over Nemotron-Terminal-32B with the same teacher. Training is SFT only; no RL is reported.
Key points
- Skills are the synthesis primitive. 10,000 skills are filtered by rules, an LLM score and download popularity to the top 1,000 (12 categories, 63 subcategories). SkillNet relations compose them into 76 skill teams (multi-role, within a subcategory) and 237 skill graphs (cross-subcategory pipelines). Tasks: 3,723 single-skill, 1,000 team, 1,000 graph.
- Task, environment and verifier are co-derived. Each skill is paired with a FinePersonas persona (dropped when the pair is incoherent) and an LLM writes a quadruple: instruction, environment blueprint (initial files + setup steps), evaluation criteria, execution guideline. An LLM judge keeps only tasks scoring ≥4 on all five of instruction quality, closed-world solvability, blueprint completeness, guideline quality and criteria quality.
- Generate–verify–repair for every artifact, up to 3 rounds, else discard. Initial files are routed to an LLM, a local-tool or a remote-fetch agent and checked for internal correctness and cross-file consistency. Setup runs as a task-specific script in one shared sandbox (no per-task Docker image), checked by an agent that writes diagnostic probes rather than trusting exit codes. Pytest verifiers must run cleanly and must fail on the initial state, to rule out vacuous passes. 6,884 accepted tasks yield 5,723 valid environments (83.1%).
- Teacher trajectories are guided, then the guideline is removed. The teacher (DeepSeek-V3.2 in Terminus 2) sees the execution guideline; on 500 tasks this lifts success from 27.6% to 39.6% and cuts successful trajectories from 14.27 to 12.66 steps. The guideline is stripped from SFT inputs — keeping it costs 2.2 Pass@1 at both 8B and 14B, read as prescriptive steps blocking autonomous planning.
- Failed trajectories are kept and they matter. Training only on successes drops 8B from 15.7 to 10.1 Pass@1, more than halving the data; penalising failures with a negative SFT loss is worse still (9.0). Four LLM judges rate 67.7% of verifier-failed trajectories as actually completing the task (Fleiss’ κ 0.742) — the failure label marks a narrow execution miss, not a wrong trajectory, which is also a measure of verifier strictness.
- Diversity and difficulty are measured. Personas multiply scenario clusters 1.96× (74 → 145) for fixed skills. Same-teacher pass rate is 39.8% on Terminal-World vs 38.2% on Terminal-Bench 2.0, against 79.8% on Nemotron-Terminal-Corpus and 58–59% on TermiGen and Endless-Terminal. LLM-judge-agent audits (three judge models with environment access) rate it highest on terminal nativeness, task–environment consistency, environment quality and verifier robustness.
- Results. Terminal-World-8B/14B/32B: 15.7 / 21.3 / 31.5 Pass@1 on TB 2.0, trained only on terminal trajectories. The authors say the 32B model matches or exceeds Nemotron-Terminal-32B (which adds 226.3K auxiliary math, code and SWE samples) on all five non-terminal benchmarks; on Pass@1 it is ahead on DABench, level on AIME and slightly behind on TableBench (71.5 vs 72.5) and BIRD (49.5 vs 52.5). On tasks both models solve, the 32B model needs 10.2 steps and 40.3 commands per task and beats its teacher on all four behavioural metrics. Cost: $999.59 total.
- Failure modes on TB 2.0 (manual analysis of 32B): context overflow forcing summarisation and repeated failed ideas, execution deadlock on interactive prompts, premature completion that ignores secondary constraints, and task substitution (e.g. grep instead of the required analysis).
Relevance
The closest counterpart to TMax: same problem (terminal-agent environments at scale), opposite bets. TMax samples tasks compositionally from nine axes, skips validation and trains with RL, letting all-zero groups filter broken tasks; Terminal-World grounds tasks in real human-written skills, validates every artifact with generate–verify–repair, and trains with SFT on guided teacher trajectories. Both pair tasks with personas; only Terminal-World measures what personas add.
For Environment Generation it is a clean example of co-deriving task, environment and verifier from one specification, and its “verifier must fail on the initial state” check is a cheap soundness gate. For WorldSmith it reports what DeepSeek did not: construction yield (83.1%), per-stage cost, and a difficulty comparison under a fixed solver. It does not measure verifier false positives directly — the 67.7% judged-complete-but-failed rate points at false negatives — and its quality audits are LLM-judged.
Notes
(none)