Summary
T1 post-trains Qwen3.5-122B-A10B with PPO to operate a real shell in a cloud sandbox for up to 300+ tool-call turns per task, rewarded by executing each task’s own verifier. The recipe has three parts: a dense reward that scores a trajectory by the absolute number of verifier assertions it passes on a fixed global scale, with a full-size critic warm-started before any policy step; training-inference consistency for the sparse MoE, via token-in-token-out (TITO) and rollout routing replay (R3), which cut the train–inference log-probability gap from 0.021 to 0.013 with zero token drift in the loss region; and a training pool (T1-15k) selected from recursively synthesized tasks by an LLM audit that hard-rejects hidden requirements, test leakage, solution shortcuts and weak verifiers. Three epochs of PPO lift an SFT checkpoint from 49.4% to 64.0% resolved on Terminal-Bench 2.1, 27.9 on Long-Horizon Terminal Bench and 38.0% on Terminal-Bench Hard. A binary-reward run on TMax-15k from the base model rose from 43.8% to 47.2% but never reached the SFT checkpoint.
Key points
- Binary reward was not enough; per-assertion reward was. “A rollout batch costs hundreds of sandbox-hours, yet a binary outcome yields one bit per trajectory; our first binary-reward campaign never exceeded its supervised baseline.” The production reward is passing assertions / S with S = 20 fixed globally (near the 90th percentile of assertion counts; median 4, max ~35), absolute count rather than pass ratio so harder tasks with more checks carry more signal. It falls back to the binary outcome if the per-assertion report is missing.
- Dense reward is a property of the data, not just the reward function. It works only because RST-synthesized tasks were built with per-assertion verifiers; TMax-15k’s verifiers “emit only a binary outcome”, so only binary reward was possible on it.
- Tasks: recursive synthesis plus a heavy audit. RST grows a curriculum from accepted seeds — extend the reference solution with more executable steps, then realign environment, verifier and instruction — and validates each candidate in a fresh sandbox where the reference solution must pass the held-out verifier. T1-15k is the audited subset: eight weighted dimensions (verifier 45%, solution 25%, instruction 20%, task value 10%; instruction–verifier alignment alone 20%), five selection stages over 15 rewrite rounds, 6,875 tasks given instruction-only repair. The audit runs on DeepSeek-V4-Pro.
- Training corpus is out of distribution by construction: isolated seeds and synthesized tasks disjoint from Terminal-Bench 2.1; all three evaluation suites contribute no training tasks.
- MoE stability stack. TITO trains on the exact sampled token IDs, repairing drift at turn boundaries; R3 records the sampler’s expert choices at every MoE layer and replays them in training. Both KL terms and the MoE load-balancing loss are disabled because they fight routing replay; symmetric clip ε = 0.2; weights published once per step so staleness is exactly one update.
- PPO with a warm critic, not GRPO. A full-size 122B critic time-shares the actor’s devices. Critic Warm-Up (one epoch over TMax-15k) keeps explained variance at 0.71–0.86 from the first update, against a cold start opening at −33.6. Critic learning rate 1.5×10⁻⁵ against 1.0×10⁻⁶ for the actor. A GRPO run on T1-15k under the same harness and reward stayed flat (51.7% at steps 10 and 20).
- Reward hacking: data-side filtering is the main defence. The audit’s hard rejects target exploitable tasks; on the reward side the test-count reward invited longer trajectories in 27B runs, but turn counts did not run away at 122B, so length shaping stays off. The fixed denominator stops one extreme sample from rescaling a batch’s rewards, and per-epoch shuffling matters because T1-15k is stored in quality-rank order.
- Sandbox scale and its bias. Daytona cloud sandboxes, each minimal (one core, small memory and disk); per-node concurrency caps of 70–94 at batch 512; creation paced at 6/s against a 600/min provider quota. The scheduler launches 560 trials for a batch of 512 and cancels the tail — but the tail concentrates the hardest task families, so oversampling biases training away from long tasks (48 of 561 trials cancelled in one measured step).
- More turns, not always better search. At evaluation T1 averages 94.4 turns against 31.5 (SFT) and 41.1 (base). On six unsolved tasks it spends 164–473 turns where GPT-5.6 Sol uses 6–40, under different resource settings.
- Lessons the authors flag: critic calibration improved without reward moving (the fix was reward granularity, not the optimizer); a context-compaction bug made 98.3% of full-summary attempts fail and turns rose from 22 to 30, so turn growth alone is an ambiguous diagnostic.
Relevance
The closest companion to TMax. T1 trains on three pools including TMax-15k itself, and its binary-reward run on TMax-15k is the reported reason it moved to dense per-assertion rewards on audited synthetic tasks — a direct, if indirect, comparison of the two bets: TMax skips validation and leans on RL’s all-zero filtering, while T1 spends its effort on a verifier-weighted audit and on verifiers rich enough to grade partial progress.
For WorldSmith it argues that verifier granularity is a design target for generated environments, not a by-product: a task’s verifier decides whether partial progress is even observable to RL. Its audit weights (verifier and instruction–verifier alignment dominate) are a concrete rubric for a verifier-soundness gate.
The sandbox story complements DSec and DeepSeek-V4.1-Flash: where DSec builds its own platform, T1 runs on a commercial provider and reports the quota, concurrency and straggler limits that come with it, plus a selection bias from oversampling that a richer reward cannot undo. Also speaks to Reinforcement Learning - RL (MoE train–inference mismatch, PPO vs GRPO at long horizons), Environment Generation (recursive task synthesis) and Benchmark Validity (hidden requirements and weak verifiers as hard rejects).
Notes
(none)