Summary
From MIT, Fulcrum and CMU: a framework for predicting whether a given agent will succeed on a given agentic coding task without running it. It extends Item Response Theory (IRT), which normally uses one-hot agent and task identifiers and so cannot generalise, by replacing task identifiers with features extracted by LLMs (embeddings or LLM-as-a-judge rubrics) from the problem statement, repository state, test cases and solution patch, and by decomposing agent ability into an LLM component and a scaffold component. Evaluated on SWE-bench Verified, SWE-bench Pro, Terminal-Bench 2.0 and GSO, the predictors beat an empirical-success-rate baseline on new tasks, approach the oracle on unseen LLM–scaffold combinations, and generalise decently but far from the oracle to unseen benchmarks.
Key points
- Agentic task artifacts predict difficulty beyond the problem statement: adding repository state, tests and gold patch improves held-out-task AUC over problem-statement-only features, and each source contributes.
- LLM and scaffold contribute additively to agent ability; the decomposition costs little against standard IRT on held-out responses (statistically indistinguishable on Terminal-Bench 2.0) while allowing prediction for unseen agents.
- Generalisation: near-oracle for new LLM–scaffold combinations (both parts seen in training), weaker for entirely new benchmarks — beats baseline, far from oracle.
- Practical use for benchmark designers: calibrate the difficulty of candidate tasks without expensive agent runs; it also quantifies scaffold effectiveness independently of the LLM, suggesting environment and grading setup substantially affect measured performance.
Relevance
- Benchmark Research and Benchmark Validity — a cheap difficulty yardstick for new tasks, and evidence that single pass rates hide task diversity.
- WorldSmith — task hardness is one of WorldSmith’s measurements; predicting it from task artifacts could cut rollout cost when scoring generated environments.
Notes
From paper club, 17 June 2026: we’ll likely take inspiration from this by running one rollout to calibrate, then using the model to avoid redundant rollouts on expensive runs. Given what a single run of a frontier model costs at scale, this kind of difficulty prediction could meaningfully cut eval costs.