Prompt injection: your agent's security lives in the permissions, not in the prompt

"What is the worst thing your agent can do if the text it reads is written by a stranger?" In half the meetings the answer starts the same way: well, in the system prompt we tell it never to…. That is the moment to stop, because a system prompt is not a control. It is a request. It lives in the same place as the attack — the context — and it is read under the same rules as the hostile text that just arrived.

This piece is about where security is actually decided. The thesis, up front so it can be argued with from the first paragraph: an agent's security is not decided in its prompt, it is decided in its tool permission table. Everything that lives inside the context is inside the blast radius; only what is evaluated outside the model — at the gateway, in the client that executes the call, in IAM — is still a control once the model has been talked into something.

In the LLMOps article we describe the full lifecycle of a system in production, and in the one on the quality gap, what breaks when real traffic arrives. This is the one that was missing underneath both: what happens when real traffic is also hostile.

The day data started giving orders

The canonical formulation is from February 2023. Kai Greshake, Sahar Abdelnabi and their co-authors published Not what you've signed up for, arguing that LLM-integrated applications blur the line between data and instructions, and demonstrating practical attacks against real systems of the time — Bing Chat on GPT-4 and code-completion engines — by injecting instructions into content the system would retrieve. Their taxonomy already listed what we see in incident reports today: data theft, worm-like propagation, information ecosystem contamination, and control over which APIs get called and how.

The operational distinction is the one OWASP sets out in LLM01:2025, the top risk in its ten for LLM applications. Direct: the user writes the text that alters behaviour. Indirect: the content arrives from an external source — a website, a file, an email — and alters behaviour without anyone on the team writing it. The second is the one that matters in a company, for an uncomfortable reason: the attacker never talks to your agent. They write the ticket, the email, the PDF, the pull request comment or the page your agent will read tomorrow as part of its normal work.

In December 2025 the UK's NCSC published the clearest warning any public body has issued on this: prompt injection may never be fully mitigated the way SQL injection is. Not for lack of effort. In SQL you can enforce a strict separation between instructions and data — that is exactly what a parameterised query does — and inside a language model that separation does not exist: every token is a candidate to be read as an instruction. Their recommendation is what organises the rest of this piece: stop chasing "how do I stop it" and work on risk and impact, accepting that these systems are inherently confusable.

Before going on, four confusions worth clearing up, because each one leads to buying the wrong solution.

Why it cannot be patched, and why that is not pessimism

OWASP lists seven prevention measures for LLM01 — constrain behaviour through the system prompt, validate output formats, filter input and output, enforce least privilege, require human approval for high-risk actions, segregate and identify external content, and run adversarial testing — and then writes the line almost nobody quotes: given the stochastic influence at the heart of how models work, it is unclear whether fool-proof methods of prevention exist. It is a list of mitigations that presents itself as incomplete, and rightly so.

The hardest measure of just how incomplete came in October 2025 from Milad Nasr, Nicholas Carlini, Florian Tramèr and eleven other authors in The Attacker Moves Second. They took twelve published defences — prompting, adversarial training, filtering and secret-knowledge based — and attacked them with adaptive adversaries: gradient descent, reinforcement learning, random search and human-guided exploration. Most of those defences had reported attack success rates close to zero. Under adaptive attack, most exceeded 90%. It is not that the defences were bad: they had been evaluated against an attacker who does not adapt, and the real attacker adapts.

Simon Willison puts it in the form that is most useful for a purchasing decision. Guardrail products are sold with 95% capture rates, and in security 95% is a failing grade: the attacker does not fire a hundred attacks and settle for the average, they fire until they find the one that gets through. A high detection rate is good news for the cost of the attack and a poor basis for a guarantee.

The lethal trifecta and the rule of two

In June 2025 Willison named the combination that turns hostile text into a leak: the lethal trifecta. Three capabilities in the same agent and the same session — access to private data, exposure to untrusted content, and the ability to communicate externally. With all three, the attack has an entrance, a prize and an exit. With two, the incident stays inside.

On 31 October 2025, Meta published the operational version of the same idea: the rule of two. An agent should satisfy at most two of these three properties within a session: [A] process untrustworthy inputs, [B] have access to sensitive systems or private data, [C] change state or communicate externally. And it adds the clause that makes the rule genuinely applicable: if an agent needs all three without starting a new session — with a fresh context window — it should not run autonomously and needs at minimum human supervision or another reliable means of validation.

What makes this rule valuable for a technical committee is not its formal precision, it is that it turns an unresolvable argument about model robustness into an architectural decision with three checkboxes, one anyone can audit on a whiteboard without being an AI security specialist.

One recommendation follows from that table, and it usually surprises the team: cut C. The value of an enterprise agent normally sits in A and B — reading what arrives from outside and cross-referencing it with what the company knows — while C, the answer going out on its own with nobody looking at it, adds the least value and is what turns a failure into a leak. And when you cut it, cut it with an allowlist of destinations, not a blocklist: both incidents below travelled paths no reasonable blocklist would have anticipated.

Two incidents that show the mechanism

EchoLeak. Published on 11 June 2025 as CVE-2025-32711, found by Aim Security in Microsoft 365 Copilot. One email. The user did not have to open it or click anything: it was enough for it to land in the mailbox and for Copilot to have it within retrieval range. The payload sat where a human does not look and the system does, and the chain strung together four hops: evading Microsoft's cross-prompt injection classifier, working around link redaction with reference-style markdown, exploiting images the client auto-fetches, and abusing a Teams proxy that the content security policy allowed. Microsoft scored it critical — 9.3 — although the NVD entry puts it at 7.5; it was patched server-side and Microsoft states there was no real-world exploitation.

What it teaches is not the ingenuity of the chain. It is that every layer that failed was probabilistic or sanitisation-based — a classifier, link redaction — and the one that would have cut the damage was deterministic: do not load remote resources, do not allow destinations outside a list. The entry channel, moreover, was perfectly legitimate: receiving email.

GitHub's toxic agent flow. On 26 May 2025, Invariant Labs described the same mechanism somewhere it hurts more. An issue in a public repository with hidden instructions; an agent connected to GitHub's official MCP server with a token that also saw private repositories; the agent read the issue, pulled private data into its context, and opened a pull request in the public repository with that data inside. Their demonstration surfaced private project names, the owner's personal plans and salary details.

Their conclusion deserves to be read literally, because it defuses the usual instinct to blame the vendor: this is not a flaw in the GitHub MCP server code itself, but rather a fundamental architectural issue that must be addressed at the agent system level. The agent did exactly what it had permission to do. The lesson fits in one line: the scope of the token is the scope of the incident. A token that sees forty repositories turns any public issue into a door to forty repositories.

Where it is actually decided: the tool record

A permission, unlike an instruction, is evaluated outside the model: in the client that executes the call, at the gateway, in IAM, in the code wrapping the tool. It is deterministic and holds no opinion on how persuasive the text the agent just read happened to be. That is why the unit of work in agent security is not the prompt: it is the tool, and every tool needs a record as explicit as that of a public endpoint.

These are the fields that make such a record useful for deciding, not just for documenting.

The sixth field deserves a name because it changes the conversation: the per-session damage budget. Just as you cap what an agent session is allowed to spend with an explicit number, you cap its capacity to do harm: how many actions with effect, towards how many distinct destinations, and with how much data, before it has to ask for a signature. When the budget runs out the agent is not blocked — it would still be useful reading — the effect is. Most leaks need repetition or volume, and a deterministic counter cuts them where a classifier would hesitate.

The patterns that do hold

In June 2025, fourteen authors from academia and industry — among them part of the team that built AgentDojo — published Design Patterns for Securing LLM Agents against Prompt Injections, today the most practical document on the subject. Its governing principle is demanding and immediately understandable: once an agent has ingested untrusted input, it must be constrained so that it is impossible for that input to trigger consequential actions. Impossible, not improbable.

Six patterns follow. They are not mutually exclusive, and they are not chosen for the whole system: they are chosen per task.

• Action selector. The model translates the request into one of a closed set of actions and never sees the result. It fits trigger-style automations, where the variety is in the input and not in the plan.
• Plan-then-execute. The plan is fixed before anything external is read; outside content can fill in arguments, but cannot change the list of actions or add to it.
• LLM map-reduce. One isolated agent per piece of untrusted content, and an aggregator that does not delegate again. It is the natural pattern for processing two hundred documents without giving any of them tools.
• Dual LLM. The model with tools never sees untrusted text: a quarantined one processes it, with access to nothing, and only structured data crosses the border. It is the standard answer for email and tickets.
• Code-then-execute. The agent writes a program that calls the tools, so that control flow is written down before it ever meets hostile text.
• Context minimisation. Remove from the context what is no longer needed, so an injected instruction is not still alive twenty turns later.

The extreme case in this family is published and measured too. CaMeL — Debenedetti and others, 2025 — extracts control and data flow from the user's query, which is trusted, and prevents untrusted data from influencing program execution, with capabilities that block exfiltration. On AgentDojo it solves 77% of tasks with provable security, against 84% for an undefended system. Those seven points are the most useful figure in this whole literature: for the first time the price of security in an agent can be put in a table instead of argued in the abstract. Seven points of utility is a product conversation; a data leak is not.

And there are benchmarks for measuring at home before believing anything. AgentDojo (Debenedetti and others, NeurIPS 2024) brings 97 realistic tasks — email client, e-banking, travel booking — and 629 security test cases, and is designed to be extensible with new adaptive attacks, which is exactly what the previous generation of evaluations lacked. InjecAgent (Zhan and others, 2024) is more specific: 1,054 test cases across 17 user tools and 62 attacker tools, with two separate intents — direct harm to the user and private data exfiltration. Its result remains the best expectation-setter we know of: a ReAct agent on GPT-4 proved vulnerable 24% of the time, and with reinforced attack instructions the rate roughly doubled. These are not weak models or toy configurations.

Filters are worth it — but they cannot be the last link

None of the above means detection defences are pointless. Spotlighting, from Keegan Hines and his team at Microsoft (2024), proposes marking the provenance of text so the model can tell which part of its context is data and which is instruction, through delimiting, datamarking or encoding; in their experiments with GPT-family models it brought attack success rates down from over 50% to under 2% with minimal impact on task performance. That is an enormous improvement for a low implementation cost, and it still leaves two attacks in a hundred.

On 13 June 2025 Google published its layered defence strategy for Gemini, a good map of what is now considered complete: prompt injection content classifiers, security thought reinforcement, markdown sanitisation and suspicious URL redaction, a user confirmation framework, and end-user security mitigation notifications. Five layers, none sufficient on its own, with the stated goal of raising the cost and complexity for the attacker — not of guaranteeing anything.

The context is shifting, too. In April 2026 Google's own security team published an analysis of injections found on the open web using Common Crawl archives, with categories ranging from harmless pranks and AI deterrence to SEO manipulation, data exfiltration and destructive attacks. Their figure: a 32% relative increase in the malicious category between November 2025 and February 2026, with the authors' explicit caveat that the analysis is not exhaustive and misses uncommon signatures. As an absolute measure it is not usable; as a direction of travel it is what we have: the open web already contains text written for agents that have not passed through yet.

Human-in-the-loop breaks through fatigue

The rule of two allows all three capabilities in exchange for human supervision, and OWASP recommends human approval for high-risk actions. That is correct and it is the most used way out. It is also the one that degrades most quietly, because a repeated approval stops being an approval long before anyone notices.

Three conditions for a signature to mean something. First: show the effect, not the intent. "The agent wants to send an email" is not information; "it is about to send this text to this external address, with these three customer fields inside" is. Second: it must arrive before the effect — a notification afterwards is a log, not a control. Third, and this one is almost never designed: it must be infrequent. If a person approves thirty times a day, on the thirty-first they approve without reading, and the fault is not theirs: the permission is set wrong.

Which leads to something worth saying out loud in the committee: ticking "always approve" is not a checkbox, it is an architectural change. Whoever ticks it is saying that this tool belongs in the group that needs no signature — or that it should not exist. Both are legitimate decisions; what is not legitimate is making them at seven on a Thursday evening with no record.

The stranger test

You do not need a pentest to find out whether you have the trifecta. You need a whiteboard, twenty minutes and four questions, asked per session type rather than for the whole system — which is the most common mistake in these audits: "the product" gets approved when what needs approving are its modes of operation.

1. What text enters the context that a stranger could write? If the answer is "none", look again: an email footer, an attachment filename, a pull request comment, a supplier's product description, a web search result.
2. What can the agent reach in this session that is not public? Answer with a list of resources, not systems. If the list says "the CRM", you have not answered yet.
3. Where can a byte get out? Count the paths that do not look like exits: a link someone will click, a remote image the client loads on its own, an attachment, a call to a third-party tool.
4. Of everything it can do, what cannot be undone in five minutes?

If the first three have answers and the fourth is not empty, you have the lethal trifecta with an irreversible action inside it. That does not mean you will be attacked tomorrow; it means that if you are, the outcome does not depend on you.

The first week's cut

What we do when we walk into an agentic system that is already in production, in order and with no new platform.

1. Tool inventory, half a morning: name, effect, real scope of the credential. Two things show up in nearly every one — a tool nobody remembered connecting, and a token rather wider than its owner believed.
2. Mark A, B and C per session type. It is the literal translation of the rule of two and fits in a five-row table.
3. Cut C wherever it adds nothing: allowlist of destinations, no auto-loading of remote resources, no outbound links generated without review.
4. Narrow B: credentials per resource and per session, read-only where that suffices. This is where the GitHub MCP case closes, and it costs an afternoon.
5. Human sign-off only on the irreversible, with the effect in front of you and before it executes. Everything else without a signature, so that the signature gets read.
6. One hostile case per tool with effect in the golden set. Injection resistance is not a separate exercise from evaluation: it is one more capability, stratified like the rest.
7. A trace that answers "what text came in and what action went out?" — the same one you already need to operate the system, with the input text kept.

All seven are architecture and configuration. None of them requires changing model, which is why they survive a change of model — and that will happen twice while the system is alive.

What you buy and what you build

You buy injection classifiers, gateways that mediate tool calls, scanners that audit MCP servers, and models trained to resist better. All of that raises the cost of the attack and belongs in a security budget. What you do not buy is the table: which tools actually exist, with what scope, what is irreversible in your business — not the same for an insurer as for a SaaS — and who signs off on each thing. That knowledge is inside your organisation and getting it out takes days of work with the right people, exactly like the cases that define whether the system works.

It is also why, in our programmes, this does not go in a security annex at the end. It goes in the same place where specifications describe what an agent must build: the project constitution states what it may do, with what scope and who signs off, and that gets reviewed in a pull request like any other rule. A permission written in a spec can be audited; a permission written in a prompt is negotiable with the first person who can write a convincing email.

Frequently asked questions

What is prompt injection, in one sentence?

It is what happens when text your system treats as data — an email, a ticket, a document, a web page, an issue — contains instructions and the model obeys them. The cause is not a coding bug: inside a language model there is no separate channel for instructions and another for data, so any token can be read as a command. That is why the UK's NCSC warns it may never be fully mitigated the way SQL injection is, where that separation can actually be enforced.

How is it different from a jailbreak?

In who attacks and who is the victim. A jailbreak is the user going against the model provider's rules: they want the system to say something it shouldn't. An indirect injection is a third party going against your company, and your user is the victim, not the attacker: someone writes the content your agent will read so that it takes actions on your behalf. Confusing the two leads to buying content filters when the problem sits in your tool permissions.

Can a good system prompt or a filter prevent it entirely?

No, and there is direct evidence. The system prompt lives in the same context as the attack, which makes it a request, not a control. On filters: in 2025, fourteen researchers led by Milad Nasr and Nicholas Carlini attacked twelve published defences with adaptive attackers and broke most of them with success rates above 90%, when those same defences had reported rates close to zero. Filters raise the cost of the attack and are worth having as a layer; what they cannot be is the last link before an irreversible action.

What are the lethal trifecta and the rule of two?

Two formulations of the same idea, and the two most useful design tools available today. Simon Willison called the lethal trifecta the combination of three capabilities in one agent: access to private data, exposure to untrusted content, and the ability to communicate externally. Meta turned it into operational form as the rule of two: an agent should not hold more than two of those three properties within a session and, if it needs all three, it should not run autonomously. With all three, hostile text can turn into a data leak.

Are MCP servers insecure?

The protocol is not the problem; the scope of the credentials handed to it is. In May 2025 Invariant Labs demonstrated it with GitHub's official MCP server: an issue in a public repository with hidden instructions was enough for an agent to pull data from private repositories and publish it in a pull request. Their own conclusion is that this is not a flaw in the server code but an architectural problem to be solved at the agent system level: credentials per resource and per session, not one token that sees everything.

Where do I start if my agent is already in production?

With the tool inventory, not with the model. Half a morning: every tool with its effect (reads, writes or communicates), the real scope of its credential, and whether what it does can be undone in five minutes. Then mark, for each session type, which of the rule of two's three capabilities are active and cut the one that adds least value — almost always the outbound one: an allowlist of destinations rather than a blocklist. Human sign-off only on the irreversible, so that it means something.

Sources cited

• Greshake, Abdelnabi, Mishra, Endres, Holz, Fritz — Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection (2023; the line between data and instructions, and the taxonomy of impacts)
• OWASP — LLM01:2025 Prompt Injection (direct vs. indirect, seven mitigations, and the absence of fool-proof methods)
• NCSC (United Kingdom) — Mistaking AI vulnerability could lead to large-scale breaches (10 December 2025; why it is not SQL injection and what to do instead)
• Nasr, Carlini, Sitawarin, Schulhoff and others — The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against LLM Jailbreaks and Prompt Injections (2025; twelve defences, success rates above 90%)
• Willison, S. — The lethal trifecta for AI agents (16 June 2025; and why 95% is a failing grade in security)
• Meta — Agents Rule of Two: A Practical Approach to AI Agent Security (31 October 2025)
• Beurer-Kellner, Buesser, Creţu, Debenedetti and others — Design Patterns for Securing LLM Agents against Prompt Injections (2025; the six patterns and the "impossible" principle)
• Debenedetti, Shumailov, Fan, Hayes, Carlini and others — Defeating Prompt Injections by Design (CaMeL; 77% of tasks with provable security against 84% undefended on AgentDojo)
• Debenedetti, Zhang, Balunović, Beurer-Kellner, Fischer, Tramèr — AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents (NeurIPS 2024; 97 tasks and 629 security test cases)
• Zhan, Liang, Ying, Kang — InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents (2024; 1,054 test cases, GPT-4 with ReAct vulnerable 24% of the time)
• Hines, Lopez, Hall, Zarfati, Zunger, Kiciman — Defending Against Indirect Prompt Injection Attacks With Spotlighting (Microsoft, 2024; from over 50% to under 2%)
• Google Security Blog — Mitigating prompt injection attacks with a layered defense strategy (13 June 2025) and AI threats in the wild: the current state of prompt injections on the web (23 April 2026)
• Aim Security — EchoLeak, CVE-2025-32711 in Microsoft 365 Copilot (11 June 2025; technical analysis of the chain in Reddy and Gujral, 2025)
• Invariant Labs — GitHub MCP Exploited: Accessing private repositories via MCP (26 May 2025)