Content entering the model is untrusted at every entry point

Decision records, six of ten. One architecture decision per piece, from a governed AI-agent platform I designed and built as a prototype for Dutch local government.

The situation

A review of my own platform showed a lopsided threat model. Everything leaving the model was governed: tool calls went through policy, outputs were candidates until promoted, approvals bound humans to actions. Everything entering the model was, by comparison, trusted by default. A retrieved document, a tool result, an installed instruction file, a system prompt assembled from pieces: all of it landed in the context window with roughly the same standing as the platform’s own instructions.

That is backwards. The model is the component most easily steered by its input, and published red-team research keeps showing that content-based attacks succeed at rates nobody should be comfortable with. If I governed the model’s hands and not its eyes, I had governed the wrong end.

The decision

Every entry point to the model is an untrusted boundary, and each one has a named, kernel-side control. Not a general principle, a specific control per path.

Read. Every item placed in the context carries a trust label: platform, principal, tool, retrieved, unknown. Only platform instructions may occupy the instruction position. Every other span is structurally fenced, placed inside a delimited region the model is told to treat as data, and the fence is checked by a deterministic parser before the prompt is sent. A tool result that fails its declared schema is refused, not passed through with a warning.

Write. Anything written into stores the model later reads carries per-chunk provenance. A chunk whose provenance is unknown is quarantined: stored, visible to operators, not retrievable by the model.

Install. Instruction artefacts, the files that tell an agent how to behave, are admitted on provenance and signing only. Where it came from and who signed it. Never on content scanning. A scanner saying “this looks benign” is not a basis for admission.

Offer. The tools an agent sees are derived from the same policy that authorises execution. There is no separate tool list. If policy would deny a call, the tool is not offered, so the model cannot be talked into calling something it was never shown.

Routing. The retrieval strategy for a run is declared up front, and if it falls back to another strategy, the fallback is recorded. A silent switch from a governed index to an open search is itself an entry-point attack.

Minimisation. Secrets and identifiers are masked before the model reads them. The model gets a placeholder; the kernel keeps the mapping.

Postel’s law told a generation of engineers to be liberal in what they accept. For content entering a model it is the single worst advice available, and it is still the default in most frameworks.

What I rejected, and why

Deciding per item. Review each source, decide whether to trust it, move on. This is what most teams do and it produces inconsistency: the same kind of content is trusted on Tuesday and fenced on Thursday depending on who looked at it. A control per entry point is consistent by construction.

A detector as the gate. Run an injection classifier on inbound content and block what it flags. Detection is incomplete, and the published research is clear that it stays incomplete. A detector may inform, and in the platform it does, as a signal into policy. It may not be the thing that decides admission.

Scanner-based admission for instruction artefacts. Same reason, sharper: published research on bypassing content scanners for instruction files is not encouraging, and an artefact that will sit in the instruction position for every future run is not something to admit on a scanner’s say-so.

What it costs

A parser in the prompt path. Every prompt is checked before it is sent, which is latency and a component that can itself have bugs. A schema for every tool result, maintained alongside the tool. Provenance on every written chunk, which means every writer has to be modified. A signing process for instruction artefacts, with keys, rotation and the human workflow around them. And a masking layer that has to be right in both directions, or the model gets a real identifier or the citizen gets a placeholder.

What it does not do

Three limits, stated plainly.

Fencing constrains how the prompt is constructed. It does not constrain how the model thinks. A model that reads a fenced span and is persuaded by its contents anyway has not broken the fence; it has done what models do. The fence makes the attack harder and the record clearer. It does not make the model immune.

Quarantine cannot catch a correctly labelled lie from a trusted source. If a signed, provenanced document from a trusted system says something false, it is retrieved and the model reads it. Provenance tells you who said it, not whether it is true.

And no thresholds were set. Which trust labels may co-occur, how much unknown-provenance content tips a run into refusal, what score from the detector is worth acting on: those were left as open parameters in the prototype. The controls exist. Their tuning does not.

If you take one thing

Draw the context window your agent actually sends, and label every span with where it came from. If more than one source can occupy the instruction position, start there.

Don’t blink. The retrieved document is only ever harmless while you are looking at it, which is why the fence is checked by a parser that never looks away.

References

The allusions and quotations in this piece, with their origin:

• Jon Postel, RFC 761 (1980): “be conservative in what you do, be liberal in what you accept from others”, the robustness principle.
• Doctor Who, “Blink” (BBC, 2007): “Don’t blink.”