Summary
Corecraft is the first environment in EnterpriseBench, Surge AI’s suite of agentic RL environments: a stateful simulation of a customer-support organisation at a fictional PC-parts retailer, with 2,500+ entities across 14 types (customers, orders, products, builds, tickets, SLAs, shipping, compatibility rules, warranties, loyalty tiers, knowledgebase articles, promotions, inventory, policies) and 23 tools exposed over MCP, packaged as a self-contained Docker bundle. Every task carries an expert-authored rubric decomposed into verifiable criteria (completeness, correctness, constraint satisfaction, format compliance); a task counts as passed only when all criteria are satisfied, and the best frontier model (Claude Opus 4.6 adaptive) reaches 30.8%. The paper’s claim is that this environment is a training substrate, not just an eval: one epoch of GRPO with adaptive clipping on GLM 4.6, with reward = fraction of rubric criteria satisfied as scored by an LLM judge, lifts held-out Corecraft pass rate from 25.37% to 36.76% (+11.39pp, more than the Sonnet 4.5 → Opus 4.5 gap of 7.05pp). The result that carries the argument is out-of-distribution transfer: +4.5% BFCL Parallel, +7.4% τ²-Bench Retail, +6.8% Toolathlon Pass@1, which the authors attribute to three environment properties — task-centric world building, expert-authored rubrics, and realistic enterprise workflows.
Key points
- The design principle is task-centric, stated against entity-count maximalism: “every entity, tool, and data source exists to support diverse, challenging tasks rather than to maximize world complexity.” Worlds with fewer realistic entities supporting hard, diverse tasks beat worlds with many entities supporting redundant ones.
- Verification is an LLM judge over expert-authored rubric criteria, with reward the proportion satisfied — dense for RL, strict at task level (all criteria). The reliability argument is entirely a priori: criteria are “granular and objective”, each “a specific, verifiable assertion grounded in the environment state”. No agreement study, no human-judge calibration, no false-positive rate is reported. The quality bar is expert authorship, not measured verifier soundness.
- Transfer is the paper, and the three OOD benchmarks are its protocol: BFCL (Parallel 91.0→95.5, Simple 91.5→93.5), τ²-Bench Retail (68.7→76.1), Toolathlon (18.8→25.6 Pass@1 over 3 runs, private eval service). Toolathlon is the strongest evidence since its domains (Kubernetes, Canvas grading, inventory syncing) share no content with customer support.
- Reliability, not just peak capability, improves: Toolathlon Pass^3 nearly doubles, 9.3% → 17.6%, and the run-to-run standard deviation falls from ±2.2% to ±0.6%. Average turns per task rise 27.9 → 33.2, read as more deliberate exploration and verification.
- Gains are broadest in categories sharing structural properties with Corecraft (multi-step retrieval, constraint satisfaction, structured output), not domain content; Monitoring/Operations and Notion stay weak, which the authors read as motivating multi-domain curricula.
- What the trained model learned, from paired-trajectory review: multi-step workflow execution in the right order, constraint handling (time windows, before/after relations, joins), and response structure. Framed as professional patterns rather than environment-specific shortcuts.
- Frontier failures the environment exposes are about search discipline, not knowledge: broad keyword search instead of retrieving the customer’s actual order first; accepting exactly-10 results from a paginated tool as complete because truncation is not signalled; anchoring on the first plausible tool (
searchBuilds) and never checking the alternative (searchProducts). The environment deliberately carries the “noise” of real enterprise data — conflicting timestamps, incomplete records, silent pagination limits. - The human ledger is acknowledged and unpriced: domain experts designed tasks and rubrics, annotators populated entities and relationships, expert evaluators reviewed trajectories and failure patterns. No hours, headcount, or dollars are reported for any of it.
- Limitations the authors name: one epoch, one base model (GLM 4.6, 357B MoE / 32B active), and — the one that matters most here — no ablation against degraded environment variants (synthetic tasks, simplified rubrics, reduced entity complexity), so the three design principles are asserted as explanations of transfer, not demonstrated.
Relevance
- WorldSmith — the reference artifact for what “lab-acceptable environment” means, built the expensive way. It supplies the transfer half of WorldSmith’s argument with real numbers: high-fidelity enterprise environments produce OOD gains, so shipping more of them matters, and its three OOD benchmarks (BFCL Parallel, τ²-Bench Retail, Toolathlon) are a ready-made held-out suite for the deferred gates-predict-lift study. Its frontier pass rate of 30.8% sits inside WorldSmith’s [0.2, 0.8] pass band, so Corecraft-style tasks would clear gate 3.
- It is also the clearest statement of the position WorldSmith is arguing against. Corecraft’s verifier soundness rests on who wrote the rubric, and the paper offers no measurement of it — no kill rate, no false-positive structure, no judge-human agreement, only the claim that granular criteria reduce ambiguity. That is precisely the substitution WorldSmith exists to challenge: expert authorship as a proxy for verifier soundness, never measured. The mutant-kill and perturbation-invariance gates could be run against Corecraft’s own rubrics as an external validation of the gate.
- Two things it does not give, both load-bearing for WorldSmith: no cost (neither expert-minutes nor dollars for the environment, despite Surge being an annotation business) and no ablation separating task quality, rubric quality, and world realism, so nothing here attributes the transfer to any one function. Both are gaps WorldSmith’s budgeted-production framing and 2×2 verifier swap are designed to fill.
- Finally, a design argument worth carrying over: task-centric world building says the coverage term should be scored over tasks the world can support, not entities or tools counted, which is consistent with WorldSmith’s labour-mass-weighted work-activity taxonomy over entity counts.
- Enterprise Environments — a fully specified enterprise simulation with the entity, tool, and packaging details spelled out (Docker bundle, MCP server, tasks + rubrics as JSON), and the only one in the wiki so far shown to work as a training substrate rather than only an eval.
- Reinforcement Learning - RL — the concrete training loop: SGLang rollouts (16 per prompt) against per-rollout stateful Docker containers, LLM-judge rubric grading, Megatron GRPO with DAPO-style adaptive clipping, weight sync back to rollout.
- Benchmark Research — rubric-as-reward with all-criteria task pass, and the Pass@1 / Pass@3 / Pass^3 triple as a reliability report rather than a single score.
Notes
(Our take: to add after reading; drafted 2026-09-19, not yet discussed)