Treat your own agents as an adversary class: a threat-modelling method for agent platforms

Why the usual threat model is not enough

Every security team knows how to threat-model a web application: external attackers, insiders, supply chain. Agent platforms add two adversaries that the standard taxonomies barely mention, and both are uncomfortable. The first is the platform’s own agents. The second is the AI coding agents that write the platform’s monitoring, approval and policy code.

I did not want to assume either is malicious. I wanted controls that hold whether they are or not. This is the method I used for the governed agent platform I designed for Dutch local government, offered as a method rather than a result, because the platform is a prototype and the method is the transferable part.

One record per threat

Every threat is a fixed-field record: the assets affected, the entry point, the trust boundary crossed, the attack path, the control that is designed, the control that is missing, the residual risk in words, the evidence required to show the control fired, and an owner. The legend separates a designed control, specified but not yet proven, from a missing control, specified nowhere. That distinction sounds bureaucratic. It is the single most useful thing in the document, because it stops a diagram from counting as a defence.

Eight adversary classes

Each class gets a stated posture, a sentence that says how the platform treats it regardless of what any individual actor intends.

1. External principal. An unauthenticated caller or a forged workload identity. Refused at the edge; identity checked before anything executes.
2. Planted content. Instructions hidden in retrieved documents, tool results, memory, peer results or install-time text. Untrusted by default and never placed in instruction position.
3. Partner and supply chain. Capability packs, tool servers, extensions, dependencies, model adapters. Admitted by signature and declaration, isolated out of process, pinned.
4. Model provider, including the vendor’s own monitors. Send it as little context as possible; keep it swappable. A vendor control the platform cannot inspect does not count as a platform control.
5. Peer agent. Zero authority until a narrowing, verifiable delegation grants some.
6. Human insider. Operator, approver, tenant administrator. Separation of duties, two-person approval, and break-glass use that is itself recorded as evidence.
7. The platform’s own agent as an untrusted insider. The conservative assumption, borrowed from published AI-control research: every instance of the same model is assumed to share one goal, to coordinate, to persist across resets, and to entrench itself by being useful. Controls hold regardless of model intent and scale with model capability, not only with the stakes of the action. Refusal is by authority, never by inferred disposition. Everything that executes must be listable and stoppable. Anything persisted must be quarantinable. Checkers must be independent of the actor.
8. The agents that build the platform. The same reasoning applied to the software-delivery pipeline. Load-bearing control code is a protected supply chain: independent verification, mutation-tested proofs, build provenance, human approval of changes. This is not a claim that the build agents are malicious. The accidental form of sabotage, a test that passes over nothing, looks identical to the deliberate form, so the control must not care which it is.

Three consequences follow, and they are the heart of the method. A control that depends on reading the agent’s reasoning, trusting its summary of what it did, or inferring its intent protects against classes one to six. Against class seven it is a hope. The class-seven attack shape is the distributed kill chain: individually harmless steps spread across runs, sessions or instances, invisible to any single-run control. And class eight is simply class seven turned on your own delivery process.

Trust, but verify, the proverb says. Class seven is the admission that for your own agents the comma is doing all the work.

Two principles for measuring coverage

Bypass is not coverage. A test that approval-required actions cannot run without approval does not prove that every consequential action is classified as approval-required. If the denominator comes from the classifier under test, the measurement reads one hundred percent while the gate stands open. Coverage needs an independent count of consequential actions, taken from the declared effect class in each capability’s contract.

Semantic judgment is not a measurement boundary. Any harm rate or block rate must say which endpoint it measured: a model judge, audit-visible evidence, or a deterministic state difference in a sandbox. A model-judged gate must be bounded by deterministic checks, or it is a second opinion dressed as a control.

From threat to control to test

Each control is paired with the automated tests that act as its instruments, and the register states what each instrument does not reach. “A label exists” is not “the label is true”. “The offered toolset equals the decision” is not “the decision is minimal”. A newer, machine-readable control graph keeps one record per threat and per enforcement point: entry, seams traversed, enforcement point, the evidence that proves it fired, response and recovery, the tests that would kill it, owner, status. An integrity check in the build turns red when a control names no path, no evidence or no recovery. The rule of thumb: if you cannot name the evidence, the control does not exist yet.

External frameworks, the OWASP lists for LLM and agentic applications, the MITRE ATLAS matrix, are used as tag vocabularies for traceability, never as binding authority, and “mapped” is stated to mean traceable, not mitigated.

Ten threat patterns and what answers them

1. Prompt-injected tool call. Model output is only a proposal. Authority is checked independently of model intent. Approval is bound to the exact target and parameters. Injection detection feeds policy as a signal; it is never the defence.
2. Approval replay, mutation or forgery. The approval record is held server-side and hash-bound to the action. A bypass corpus covers missing, expired, mutated, wrong-target, replayed, self-approved, out-of-authority, forged and unsigned approvals.
3. Approver manipulation and fatigue. The text that describes expected effects is generated by the platform from the capability contract, never by the model. Model-authored text is labelled untrusted. Approval requests are rate-limited and budgeted, and approval volume is an anomaly signal.
4. Read-path exfiltration, where no consequential action ever fires. Classification travels with the data: an outbound parameter takes the highest classification of the context it came from. Outbound payloads are inspected, not just hosts allow-listed. A canary planted in classified context must never appear in egress. Model output is never rendered as active markup.
5. Persuasive instructions admitted at install time. Admission is by provenance only: signature, attribution, version pinning, minimised capabilities, a first run without credentials, out-of-process isolation. Content scanning is refused as an admission control, because scanners can be argued out of their verdict; they run as signals only. Metadata is re-verified every session.
6. Rogue or unmonitored execution by your own agent. Inventory invariant: everything that executes, runs, children, jobs, schedules, sessions, grants, can be listed, and anything that cannot be listed is a finding. The stop is recursive: list descendants first, then read back zero remaining autonomy from a store the stopped thing cannot write.
7. Distributed kill chain and memory persistence. Memory is proposed by the model and written by the kernel through a fail-closed promotion gate. A changed source blocks reuse of memory derived from it. Poisoned lineage is quarantined and the purge is proven across all copies. Derivations are correlated across runs and instances.
8. Checker subversion or collusion. Each checker carries an independence record: its model and provider compared with the actor’s, inputs taken from the platform record and never from the actor’s own summary, its verdict never visible to the actor, its fallback to refuse or defer and never to proceed. No action is admitted on a single checker whose inputs the actor could write. A monitor from the same model family is a correlated-failure surface.
9. Vulnerability insertion by build agents. A named inventory of load-bearing control code with a stricter change rule: human approval, build provenance, reproducible builds, no build-agent credential for the control it modifies, mutation-kill tests, and an independent verifier that never wrote the change.
10. Untraceability. A trace-completeness check on every consequential action: principal, mandate or delegation ancestor, run, step, correlation and clock source must be present and checked against a record outside the writer.

Red-teaming as a function, not an event

Red-teaming is organised into twelve families, following the published agentic red-teaming guidance, and the later families must also prove recovery: rollback, quarantine purge, cascading revocation, and a stop that still answers under load. A stop that works only when the system is idle has not been tested.

What I cannot prove

Prototype grade. The register exists, the graph exists, the integrity check runs. Several controls are designed and not yet proven by their instruments, and the register says so, row by row. What I will not publish is which ones, or their residual-risk wording, because that is a security disclosure for the people who would run it, not for a blog.

If you take one thing

Add two rows to your adversary list this week: your own agents, and the agents that write your code. Then go through your controls and mark each one that depends on reading an agent’s reasoning or trusting its summary. Those rows protect you from everyone except the two you just added.

Nobody expects the Spanish Inquisition. Class seven is the assumption that your own agents might.

References

The allusions and quotations in this piece, with their origin:

• “Trust, but verify”, a Russian proverb made famous by Ronald Reagan in the 1987 INF treaty talks.
• Monty Python’s Flying Circus (BBC, 1970), the Spanish Inquisition sketch: “Nobody expects the Spanish Inquisition” and the escalating list of chief weapons.