Summary
IBM Research’s Agent Mentor treats an LLM agent’s own execution traces as a process event log and mines them to find and repair ambiguity in the agent’s system prompts. The pipeline (AMAP) runs the same user prompt k times (100 in the experiments), captures OpenTelemetry-extended trajectory logs, applies process discovery to recover a workflow view of task nodes across runs, k-means-clusters the output text at each node (elbow on WCSS) to separate good from bad runs, distils the text with an LLM, elicits distinguishing SVO-plus-discourse semantic features, fits a decision tree over those features with Gini importance, and feeds prompt + features + tree rules + success cluster to an LLM-as-judge that writes corrective statements injected back into the node’s prompt. On a proprietary AccessControl agent whose prompt said “Reject users that are unauthorized and untrusted” (conjunction or disjunction? the model split 58/37 across two clusters), accuracy went 50% → 87%, and 50% → 86–99% across four backbone LLMs. On CUGA/AppWorld the gains are real but small (+6, +11 pp easy; 0, +3 medium; +5, +2 hard), and HolidayFinder 7% → 15%. The authors’ own reading: AMAP fixes specification clarity, not reasoning.
Key points
- The event log here is an agent-trajectory log, not an enterprise log: time-ordered action records (mostly LLM invocations) with input instruction text, memory-state attributes, and output text; only inputs and outputs are analysed. Process mining is used to align many runs of the same prompt into a shared workflow view (in the worked example: orchestrator → XOR split over two sub-agents → back to orchestrator).
- Node clustering is the good/bad classifier: embed each node’s output text, k-means with an elbow-threshold stopping rule, then have an LLM name each cluster from three sampled trajectories. The answer-node is singled out as the most informative node because it encodes whether the request was fulfilled. Labels come from a user, or from majority vote when a benchmark label exists.
- Semantic feature elicitation extends Subject–Verb–Object with clauses and elementary discourse units (causes, conditions, results); 20 instances per cluster are shown to an LLM which invents the discriminating feature classes (for the access-control orchestrator: politeness tone, refusal mode, permission status, list status, condition expression, reason note, …). A decision tree over those features then explains the cluster split, and Gini importance gates which nodes get corrective statements at all.
- Derived corrections are literal prompt rules — “Only allow a user if they are NOT both unauthorized AND untrusted”; “If either condition is true, refuse” — and for CUGA, node-level directives like the wishlist data fields a task description must carry (importance 0.355) or the unit-price × quantity computation (0.282).
- The headline finding is a diagnosis of failure type: gains are large where instruction ambiguity dominates (+37 pp) and near zero where the task is genuinely hard (0 to +5 pp on AppWorld medium/hard). The pipeline is a specification-refinement layer, explicitly not a reasoning improvement.
- Robustness: unchanged pipeline across GPT-4o, LLaMA 4 Maverick, Mistral Medium, and gpt-oss-120b, all improving from 0.50 baseline — the ambiguity is in the spec, not the model.
- Limitations the authors name and that matter for reuse: when all trajectories fail there is no comparative basis and no correction can be inferred; the pipeline scans every node without root-cause localisation, so corrections are redundant and may interact negatively; longer baseline prompts dilute the injected fix (an improvement-vs-token-cost trade-off); numeric values embedded in instructions are ignored; cluster labelling is supervised.
- Sample sizes are tiny — six AppWorld tasks (two per difficulty) and two proprietary apps — so this is a demonstration, not an evaluation.
Relevance
- ITSMBench — a useful near-miss. It shows the machinery of turning traces into structured, actionable knowledge (align runs → cluster outcomes → elicit semantic features → decision tree → derived rule) and that machinery is domain-agnostic: the same steps could run over BPI 2013/2014 activity text to elicit what distinguishes cases that resolve from cases that ping-pong or breach SLA. But it does not bridge enterprise event logs to environments or tasks at all — its logs are generated by the agent under study, in an environment that already exists (AppWorld), with tasks that already exist and ground-truth labels that already exist. Nothing here creates a world, a task, or a verifier. Its real contribution to V2 is on the other side of the loop: it is a concrete template for a trajectory-diagnostic layer over V2 runs, which the current binary end-state pytest verifiers cannot provide, and it names the exact condition under which such a layer dies — zero successful trajectories, which is where frontier-hard tasks live.
- WorldSmith — evidence for the automatable half of the loop: an LLM can read many trajectories and write a specification (prompt rules with feature-importance justification) that measurably improves behaviour, without a human writing the rule. That is spec authoring from logs, and it is the closest thing here to automated task-instruction authoring. The counterweight is the “all runs fail → no signal” limitation and the absence of root-cause localisation, both of which say this automation is a refiner, not a generator.
- Enterprise Environments — home concept since 2026-09-10
Questions it raises for ITSMBench:
- Cluster-then-explain is a general recipe for “what distinguishes good from bad”. Run over BPI case traces rather than agent traces, does it yield an ITSM task taxonomy (the distinguishing semantic features of case outcomes) — and would that be the log-derived task distribution the project needs, or only a post-hoc description of it?
- AMAP needs both successes and failures to cluster. V2’s stated ambition is to expand difficulty to the GRPO sweet spot; at what pass rate does trajectory-diagnosis stop working, and does that put a floor under how hard tasks can be made before diagnosis blinds itself?
- Their AccessControl case is exactly a V2 failure mode in miniature: an ambiguous policy statement produces a 58/37 outcome split with no bug anywhere. How many V1/V2 task instructions carry the same conjunction/disjunction ambiguity, and would running AMAP over repeated agent runs be a cheaper task-validity audit than expert review?
Notes
(Our take: to add after reading; drafted 2026-09-09, not yet discussed)