HiL-Bench (Human-in-Loop Benchmark) Do Agents Know When to Ask for Help?

Summary

HiL-Bench (Scale AI) measures selective escalation: whether a coding agent notices, mid-task, that a gap cannot be resolved by exploration and must be asked about. Trained annotators take 300 tasks that top models already solve at ~85% pass@3 (150 from SWE-Bench Pro, 150 from BIRD text-to-SQL) and inject 3–5 independent blockers each — missing information (42%), ambiguous requests (36%), contradictory information (22%) — into a hidden registry, so that no task can be passed without resolving every blocker. Agents get an ask_human() tool backed by a frozen Llama-3.3-70B semantic judge that returns a blocker’s resolution when the question targets a registered blocker and the fixed string "irrelevant question" otherwise, giving a binary reproducible signal. The headline metric, Ask-F1, is the harmonic mean of question precision (fraction of questions judged relevant) and blocker recall (fraction of blockers resolved); the harmonic mean is chosen so that reaching 80% recall via 50 questions scores 14.5%. Across GPT-5.4, GPT-5.3-Codex, Claude Opus 4.6 and Gemini 3.1 Pro, full-information pass@3 of 75–89% collapses to 4–24% once the model must decide when to ask, and RLVR on a shaped Ask-F1 reward lifts a 32B model on both Ask-F1 and pass@3 with cross-domain transfer.

Key points

• Task admission is two mechanical checks, both computed from rollouts rather than judged: necessity — without ask_human() the pass rate must be ≤5% across reference models, proving the blockers cannot be circumvented; sufficiency — with all resolutions supplied the pass rate must approach 90% for at least one model, proving the blockers, not capability, are the obstacle. Tasks are selected from a source benchmark precisely because top models already clear ~85% on them.
• Every blocker must pass seven criteria or the task is rejected outright: realism, criticality (prevents correct completion), objectivity (single unambiguous resolution), vast search space (cannot be guessed or searched), independence (resolving one does not reveal another), no contamination (the resolution exists only in the registry), and non-contrived. Enforced by 5–6 rounds of independent human auditors plus an automated agentic pipeline; failures are revised or discarded.
• Ask-F1’s precision term is an anti-gaming property of the metric’s structure, not a patch bolted on after red-teaming. The authors’ framing — detection-only evaluations leave a “spam-your-way-to-recall” loophole open — is a design argument for building the hack out of the grader rather than measuring it afterwards.
• ask_human() is a priced-oracle design in everything but the price: a frozen open-source model, matched against registered trigger questions and descriptions, returning either the exact resolution or a fixed refusal string. Frozen and open-source deliberately, for stability over time and replicability.
• Progressive discovery is the load-bearing design choice and is validated by ablation: strip the environment tools from Claude Opus 4.6 (spec text and ask_human() only) and blocker recall falls from 61% to 11%, concentrated in missing-information and ambiguous-request blockers. Contradictory-information blockers drop least, since the conflict is visible in the prompt text.
• The judgment gap by model (pass@3, full info → with ask_human()): SQL — GPT-5.3-Codex 86→5.3, GPT-5.4 86→17.3, Gemini 3.1 Pro 88.7→35.3, Claude Opus 4.6 90.7→39.3; SWE — all four between 67 and 85 full-info and 1.3–9.4 with the tool. Ask-F1 averages 40.5% on SQL and 37.4% on SWE; no model exceeds 50% recall on SWE.
• Failure fingerprints are stable per model family across both domains (3,600+ traces, LLM-judged taxonomy): GPT executes confidently on wrong beliefs and barely uses the escalation option; Claude detects uncertainty without acting on it — in 45% of alignment failures it states in its trace that it is stuck and submits anyway; Gemini self-assesses poorly but is strongly correctable by external grounding on SQL and not on SWE. Adding the tool reshapes how models fail, not how often.
• Judgment is trainable: Qwen3-32B with LoRA under SkyRL, trained on 120 tasks per domain with a shaped reward (+0.3 per question hitting a registered blocker, −0.1 for irrelevant or duplicate, plus a terminal coverage term gated on finding at least one blocker), improves precision and recall on held-out tasks, improves pass@3 in lockstep, and transfers across domains — SQL-only training lifts Ask-F1 on held-out SWE and vice versa.
• Limitations worth noting: blockers are injected into existing benchmark tasks rather than drawn from real incomplete specs; the semantic judge is an LLM (validation deferred to an appendix); the RL result is one 32B model with LoRA, 120 train / 30 eval tasks per domain.

Relevance

• WorldSmith — mostly a method source rather than a subject one. Three things transfer directly. (1) Its necessity/sufficiency pair is a published precedent for WorldSmith’s acceptance gates: sufficiency is the solvability witness (gold path executes and passes) and necessity is a floor proving the task is not trivially circumventable, both computed from rollouts with no labels. (2) ask_human() is the closest existing thing to WorldSmith’s priced simulated expert oracle — frozen, replicable, binary, returning a fixed refusal when the query misses, which is exactly the NOT_COVERED case; what it lacks is the price, the per-type charge, and the modelled refusal/error rate, so WorldSmith’s oracle is HiL-Bench’s plus a meter. (3) Ask-F1 is an argument that anti-hacking can live in the metric’s algebra — the harmonic mean makes question spam self-defeating — which is a cheaper gate than the hacker-fixer loop WorldSmith has deferred.
• The cost side is the gap: trained domain-expert annotators inject blockers and 5–6 rounds of independent auditors validate every task, and the paper reports no hours or dollars for any of it. As an expert-annotator ledger entry it is a shape without a number.
• It also names a builder capability WorldSmith’s acceptance gates do not currently score: an agent building environments faces the same judgment problem (incomplete source material, ambiguous policy) and the gates measure only what it ships, never whether it should have escalated to the expert oracle before shipping. Budget utilisation is the nearest proxy.
• Benchmark Research — metric design: precision/recall decomposition of a process metric, harmonic mean as an anti-gaming structure, and the claim that process and outcome are coupled by construction (no task passes without resolving all blockers, so recall governs pass rate).
• User Simulators — a deliberately minimal human simulator: frozen open-source judge, registry-matched, binary output, chosen over free-form simulation to remove confounds. The opposite end of the design space from persona-driven simulated users.
• Benchmark Validity — construct validity by construction: tasks sourced where capability is known high, blockers validated against seven criteria, necessity and sufficiency checked mechanically before admission.

Notes

(Our take: to add after reading; drafted 2026-09-19, not yet discussed)