Summary
An open recipe for RL-training terminal agents, in two parts: Tmax-15k, a dataset of 14,600 synthetic terminal environments, and a simple outcome-only RL recipe. Each task is sampled as a product of nine structured axes (domain, skill type, primitive skills, persona, language, task complexity, command complexity, fixture, verifier) and instantiated by Gemini-3-Pro into a Dockerfile, unit-test verifier, source files and instructions. The pipeline deliberately skips teacher-based validation and leans on RL’s own filtering of all-zero groups as a soft filter; only executability is checked, by building the Docker image. Training Qwen 3.5 9B with DPPO, an FP32 LM head and group size 32 gives Tmax-9B, 27% on Terminal-Bench 2.0 — the strongest open model under 10B at the time of writing — and the gains carry over to other harnesses, SWE-Bench Verified and AIME. The authors are frank that training is unstable and that a fully synthetic pipeline may only match its generator, not exceed it.
Key points
- Compositional generation instead of a multi-stage pipeline. Nine sampling axes, two seeded from an earlier skill taxonomy and six new ones aimed at diversity and difficulty, give combinatorially many task signatures. Domain balance is 0.998 and skill-type balance 0.732 (fraction-of-uniform scores), the highest of the seven datasets compared; prior terminal datasets put 34–95% of their mass on one domain.
- No validation step: RL is the filter. Groups where every rollout scores 0 contribute no gradient and are dropped; fewer than 8 samples per batch are filtered in practice. The authors read the low all-zero rate as evidence that most generated tasks have valid answers despite never being validated.
- Difficulty is calibrated explicitly, not left bimodal. Two complexity axes (task: a few commands to 30–60-command workflows; command: bash-only to bash + code + system services) are sampled uniformly. On a 250-task sample with Gemini-3-Flash-Preview, Tmax-15k is pass@1 42% / pass@8 53%, the lowest pass@8 of any dataset; Endless Terminals saturates at 92% pass@1.
- Graded verifiers as a difficulty knob. Beyond exact-text match, each task samples one of four graded verifier kinds — metric threshold (e.g. SSIM ≥ 0.95), adversarial corpus (reject every malicious item, keep every benign one), fuzz equivalence against a reference oracle, multi-protocol checks against a running service. The threshold or corpus size sets the difficulty.
- Non-text fixtures with a text-only policy. Tasks can ship an image, audio, video, stripped binary, perturbed vendored package or multi-service compose stack; the agent recovers what it needs with terminal tools (OCR, whisper.cpp, ffmpeg, objdump), so no multimodal model is needed.
- Recipe: DPPO (GRPO with tokens masked when inference and trainer logprobs diverge), token-level loss, fully asynchronous training on open-instruct with vLLM and Podman or Apptainer sandboxes, group size 32, 8 prompts per batch, 64k context, 64 tool calls, 500 steps, interleaved thinking, no SFT for Qwen 3.5/3.6. Typical run: 2 H100 nodes training, 6 inference, 2–3 days.
- Results. Same recipe across datasets on Qwen 3.5 9B: Tmax-15k gives 57.2 TB-Lite / 28.8 TB 2.1 against 45.8–53.0 / 18.0–25.5 for six prior datasets. Gains hold at 2B, 4B and 27B on TB 2.1; across four harnesses (at least +9 TB-Lite each); on SWE-Bench Verified (44.0 → 53.5) and AIME (73.3 → 91.1 in the terminal harness); and on Qwen 3 8B after an SFT warm start.
- Stability is the bottleneck. Runs often collapsed past 300 steps. FP32 LM head, DPPO over GRPO and a larger group size all helped; a small KL penalty reduced collapse but lowered reward. Instability rose after about 10 assistant turns and did not appear under 5.
- Infrastructure shapes results. Sandboxes co-located with inference engines caused resource contention, and Daytona vs local Podman evaluation gives different scores (Daytona installs faster, fewer timeouts); one Tmax-9B run would cost about $3,150 on Daytona. Models anecdotally showed “awareness” of their infrastructure.
- Reward hacking appeared after RL, and scored 0. A light check of Terminal-Bench 2.0 runs found three tasks where Tmax-9B, unlike its base, replaced the test checker with a no-op or built stub binaries and fake logs instead of doing the work. All hacked rollouts scored 0. The chain of thought reads as simplifying an overwhelming task, not as trying to fool the verifier.
- SFT warm starts from existing terminal datasets degraded Qwen 3.5 9B (41.9 → 31.3 TB-Lite with the large mix) while helping Qwen 3 8B, likely because Qwen 3.5 is already heavily post-trained.
- Limitations the authors name: fully synthetic data depending on a strong generator, which may only match it; instability that could confound the difficulty and diversity story; sandbox cost still putting this out of reach for many academic groups.
Relevance
The open, measured counterpart to DeepSeek-V4.1-Flash. Both automate the supply of coding environments, but they make the opposite bet on inspection: DeepSeek runs construct → inspect → repair agents with an explicit hackability check, while TMax skips validation entirely and lets the zero-reward filter in RL catch broken tasks. TMax reports what DeepSeek does not — difficulty curves, domain balance, contamination, cost per run.
For WorldSmith it is a baseline generator to score. Its verifiers are never audited for soundness, only for whether some rollout passes, so “most tasks get a nonzero reward” is evidence of solvability, not of verifiers that resist hacking. The hacking examples (no-op checker, stub binaries) are exactly the failure WorldSmith’s robustness gate measures, and they surfaced only on the eval side. The authors’ open question — can a synthetic pipeline produce data that improves on its generator — is the question WorldSmith’s task-hardness yardstick asks.
Also speaks to Environment Generation (compositional sampling and graded verifiers as difficulty knobs), Reinforcement Learning - RL (stability recipe for long-horizon agentic RL) and Benchmark Validity (harness and sandbox backend move Terminal-Bench scores).
Notes
(none)