ClawEnvKit: Automatic Environment Generation for Claw-Like Agents

Summary

ClawEnvKit generates whole agent environments — task, tools, fixtures, and grader — from a one-line natural-language request (“generate 10 email management tasks, medium difficulty”). Its enabling move is representational: because an agent’s state space is infinite but its implementation is just a few in-memory mock-service databases, the pipeline never specifies state transitions; it specifies only E = (P, M, C) — a task prompt P, an interaction interface M = (tools, server-side audit log), and an evaluation functional C = weighted checks over the trajectory and audit log. A three-agent pipeline (Parser → Generator → Validator) instantiates that triple, with the Generator also able to design and stand up a new mock service when the task needs one, and the Validator enforcing format, coverage of every parsed intent unit, and feasibility, retrying up to three times before discarding. The output is Auto-ClawEval: 1,040 environments across 24 categories for $80 of Claude-Sonnet-4.6 API and 18 hours, against an estimated 208 person-hours for the 104-task human-curated Claw-Eval — and on a count-matched comparison the generated tasks score higher on LLM-judged coherence (0.59 vs 0.51) and clarity (3.54 vs 3.38) at 100% validity. Across 4 model families and 8 harnesses, structured harnesses beat a bare ReAct loop by up to 15.7 points, completion (34–76%) is the only axis with headroom while safety and robustness sit ≥83%, and the 104-task Mini variant tracks the full 1,040 within 2%.

Key points

• The declarative separation is the whole argument: “an LLM can produce a valid (P, M, C) triple without ever reasoning about state transitions, whereas generating a correct state-based grader requires understanding the full execution semantics of the environment.” Automating environment generation is made tractable by choosing a representation that does not require the generator to simulate.
• Parser output is typed intent units — actions the agent must perform, objects the environment must contain, constraints it must respect — and these are the bridge to verification: the Validator’s coverage check requires every unit to appear as a callable tool, as pre-loaded data, or as a scoring/safety rule, and regenerates the task if any is missing. That is a mechanical answer to “did the generated verifier actually check what was asked for”.
• Generator has three sub-workflows: task generation (P, M, fixtures, C, with diversity controls rotating API actions and deduplicating against previous tasks), service generation (design a new API, build a mock server, test it, confirm with the user, add it to the shared library for reuse), and fixture generation (synthesise the database, image, or document the scenario needs and mount it into the container).
• Validator checks are format (all fields present, weights sum to 1, at least one safety check, no self-contradiction such as a safety rule forbidding an action the scoring requires), coverage, and feasibility (catch counterfactual tasks like “get tomorrow’s emails”, or scoring criteria referencing information the agent cannot reach); a newly generated service is booted and its endpoints hit before admission.
• Grading: R(σ,E) = safety(σ) × (0.8·completion + 0.2·robustness). A safety gate zeroes the entire score on any forbidden action or prohibited keyword. Completion is a weighted sum over 15 check types drawn from audit log (what the agent did), output (what it said), and filesystem (what it created), with llm_judge capped at 55% total weight so the majority of every score stays deterministic. Robustness is the fraction of injected API errors recovered from — errors are injected on 25% of calls.
• Grading reads the server-side audit log, explicitly “to prevent agents from receiving credit for actions they described but did not perform” — the same failure V1’s adversarial scripts found (a cheat scoring 1.0).
• Harness portability is engineered in three tiers — native tool plugin, MCP server, or a curl-based SKILL.md appended to the prompt — with all tiers hitting the same mock services and producing identical audit entries. Tier does not determine performance: SKILL.md harnesses beat several MCP ones.
• The authors’ framing of the payoff is contamination and adaptivity, not cost: a generated benchmark is “contamination-free by construction” and can be refreshed continuously, and the same generator doubles as an RL environment source “producing task distributions that adapt to an agent’s current weaknesses rather than being bounded by existing user logs”.
• Weaknesses to hold onto: the quality comparison is LLM-judged, and the coherence win is explained by the generated tasks’ structured format being more legible to the judge than Claw-Eval’s grader code — an evaluation artefact, not clearly a quality result. There is no human realism audit anywhere, no check that the generated task distribution resembles work anyone actually does, and safety/robustness being near-ceiling means two of three scored dimensions currently carry no signal.

Relevance

• WorldSmith — the most aggressive automation claim in this batch and the most useful one to argue with. It is the existence proof that environment and verifier generation can be fully automated at ~$0.08 per environment, and it identifies the design decision that makes it possible (declare P/M/C, never the transition function). It also shows the automation extending past tasks into the environment itself — generating and testing a new mock service on demand, then banking it for reuse. But it is precisely the case Curation-Bench warns about: quality is measured by an LLM judge on dimensions (coherence, clarity) that favour the generated format, diversity is enforced by action rotation and deduplication rather than by any external reference, and nobody checks whether 1,040 environments represent anything. It is the strongest available claim that the WorldSmith answer is “yes, mostly automatable”, and the weakest available evidence that the result is valid.
• ITSMBench — directly contradicts the project’s working claim, and that makes it valuable. ClawEnvKit’s stated selling point is generating task distributions “rather than being bounded by existing user logs” — the exact opposite of grounding a benchmark in BPI 2013/2014. It thereby sharpens the question: if the generated route yields more environments, more cheaply, contamination-free, and refreshable, then the log’s contribution has to be something the generator cannot supply, and the only candidate left is representativeness — which ClawEnvKit never measures. Practically it also offers V2 reusable machinery: the audit-log-based grading, the deterministic-vs-judge weight cap (≤55%), the 25% error-injection robustness dimension (an ITSM analogue of flaky vendor APIs), and the three-tier harness adapter, which is close to the Prime Intellect / Harbor / OpenEnv porting already planned. What it does not cover: no event logs, no user simulator, no multi-agent workflow, no long-horizon composition, and no notion of realism at all.
• Benchmark Research — a benchmark-design contribution on the “live benchmark” thesis: static test sets go stale and leak through pretraining, post-training, and deployment adaptation, so evaluation should be regenerated on demand rather than frozen. The Mini-vs-full result (<2% divergence at 1/10th the size) is also a clean argument that a small sampled subset is a sound proxy once construction is automated.
• Environment Generation — home concept since 2026-09-10

Questions it raises for ITSMBench:

• ClawEnvKit’s diversity is internal (rotate actions, deduplicate); ITSMBench’s coverage yardstick is external (frequency-weighted process variants from the logs). Could the BPI variant distribution simply be fed to a ClawEnvKit-style parser as the specification — turning the log into the prompt for the generator — and would that be the whole bridge from logs to environments?
• Their safety and robustness dimensions are saturated (≥83%) and carry no signal. If V2 adopts a composite reward, which of its dimensions will be saturated on arrival, and is a dimension that everything passes worth scoring at all?
• The claim that generated environments beat human-written ones rests on an LLM judge scoring dimensions that favour the generated format. What would the honest comparison be for ITSM — and is “a practitioner cannot tell which of these two tickets was generated” the test that actually settles it?

Notes

(Our take: to add after reading; drafted 2026-09-09, not yet discussed)