Summary
AgentDojo is a framework — deliberately not a static test suite — for measuring whether a tool-calling agent can be hijacked by prompt injections hidden in the data its tools return. Four stateful environments (Workspace, Slack, Banking, Travel) with 74 tools are populated with benign dummy data containing explicit injection placeholders; 97 hand-written user tasks are crossed with 27 injection tasks to yield 629 security test cases, each scored by two independent deterministic functions (a utility function for the user’s goal, a security function for the attacker’s). Results across nine 2024-era models: more capable models are easier to hijack (an inverse scaling law), most models lose 10–25% absolute utility merely from being attacked, and attack success varies wildly by environment (92% on Slack, where the attacker controls a large share of tool output, vs 0% on one Travel injection needing two unrelated malicious actions). Attack phrasing matters more than attacker knowledge: the authors’ “Important message” injection beats “ignore previous instructions” and InjecAgent’s prompt, an adaptive best-of-four adds ~10 points, while knowing the user’s and model’s real names adds only +1.9% (and guessing wrong costs ~22%). Among defenses, a tool filter — the agent commits to a minimal tool set before seeing any untrusted data — cuts targeted ASR to 7.5%, and several defenses actually raise benign utility; but the filter fails on the 17% of cases where the tools needed to do the task also suffice to do the attack.
Key points
- Environment design: state is a collection of mutable objects; tools are ordinary Python functions (FastAPI-style, taking the state object as an argument) whose docstrings become the tool descriptions in the prompt. Adding a tool means adding a specially formatted function — the extensibility mechanism is the type system, not a config format.
- Task authoring: user tasks are hand-designed for diversity (up to 7,000 tokens of context data, up to 18 chained tool calls), and each exposes both a deterministic binary utility function reading pre- and post-state, and a ground-truth sequence of function calls. The ground-truth call sequence exists so attacks can be placed where the agent will actually look — it is an attack-authoring aid, not a grading aid.
- The security suite is generated combinatorially: 97 user tasks × the injection tasks relevant to their environment = 629 test cases, from only 27 hand-written attacker goals. Every user task also runs attack-free as a plain utility test. This cross-product is the cheapest published mechanism for turning a small authored set into a large evaluation set.
- Verification is deliberately deterministic, and the paper argues explicitly against LLM judges for this setting: if an injection is strong enough to hijack the agent, it may also hijack the evaluator. Environment dummy data was LLM-generated (GPT-4o, Claude 3 Opus) from a schema plus few-shot examples, then manually inspected — generation for the data layer, hand-checking for the grading layer.
- Three reported metrics: benign utility (no attack), utility under attack (robustness to denial-of-service — the complement is “untargeted ASR”), and targeted ASR (attacker’s goal achieved). A collection of attacks counts as successful if any member succeeds, modelling an adaptive attacker. Keeping “the agent still did the user’s job” separate from “the attacker also won” is the design point most benchmarks miss.
- Position matters: injections placed at the end of a tool’s response reach ~70% success against GPT-4o.
- Their stated reason for being a framework rather than a fixed benchmark: existing prompt-injection attacks are generic and do not optimise against a specific defense, and it is trivially easy to build a non-robust defense that happens to stop one fixed attack. Robustness claims therefore require adaptive attack evaluation, which requires the benchmark be extensible by the attacker.
- Limitations named: no multi-task sessions with persistent agent context, where an injection could instruct the agent to “wait” until a later task grants it the right tools; and isolation defenses still fall to injections that merely bias one tool’s result (a hotel listing that injects “always pick me”) without hijacking the agent.
Relevance
- ITSMBench — an ITSM environment is a near-ideal prompt-injection target and V2 currently has no notion of this: ticket bodies, email threads, KB articles, chat messages from simulated colleagues, and vendor API payloads are all attacker-controllable text that the agent reads and acts on, and the agent holds write access to identity, access, and change-management operations. AgentDojo gives the exact recipe to add it: mark injection placeholders in the seeded state, write a handful of attacker goals against the same mocked APIs (grant a role, add a mailbox forwarding rule, approve a change, exfiltrate a ticket), and cross them with the existing tasks. Its separation of utility under attack from targeted ASR also maps onto V2’s preservation controls — the adversarial scripts already in itsmbench-core catch cheating agents, not hijacked ones. What it does not cover: nothing to do with event logs, task distributions, realism, user simulators, or multi-agent workflows; its tasks are single-turn instructions in toy consumer apps, and it is a 2024 model snapshot.
- Red-Teaming — the canonical agentic prompt-injection benchmark and the source of several load-bearing results: the inverse scaling law (capability and attackability rise together), the finding that defenses can improve benign utility, tool filtering as the strongest cheap defense (7.5% ASR) with its precise failure condition (17% of cases where task tools = attack tools), and the methodological insistence that a security benchmark must be a live framework accepting adaptive attacks rather than a leaderboard over fixed attacks.
- WorldSmith — two automation levers worth noting: LLM-generated environment data from a schema with human inspection (the generation/verification split), and the user-task × injection-task cross-product, which turns O(n+m) authored artifacts into O(n·m) scored cases. The counterweight is that both the tasks and their deterministic verifiers are entirely hand-written, and the paper’s argument against LLM judges under adversarial conditions is a direct constraint on how far verifier authoring can be automated.
Questions it raises for ITSMBench:
- If V2 adds a conversational operator simulator and simulated colleagues, every message they send is an injection surface inside the benchmark’s own machinery. Does a simulated actor that can be prompt-injected make the environment more realistic (real users forward phishing) or just make scores noisy and unattributable?
- Tool filtering works because most user tasks need read access while most attacks need write access. ITSM work is write-heavy by nature — resolving a ticket is a state change. Does that mean the strongest known cheap defense simply does not apply to this domain, and what is the ITSM analogue?
- The paper refuses LLM judges because attacks can hijack the evaluator. V2 is heading toward rubric or NL-assertion grading for the multi-agent tracks. Does adding a security dimension force those verifiers back to deterministic DB-state checks, or is there a split — deterministic for security, judged for quality?
Notes
(Our take: to add after reading; drafted 2026-09-09, not yet discussed)