The controls an extension can never disable

Decision records, three of ten. One architecture decision per piece, from a governed AI-agent platform I designed and built as a prototype for Dutch local government.

The situation

A platform that only runs the agents its own team wrote is not a platform. The whole point was that a municipality, or a vendor serving several of them, could add a component: a retriever over their own document store, a tool server for their case system, a specialised agent. Extension is the value.

Extension is also the hole. Every component you let in is code you did not write running inside a boundary you are responsible for. The usual answer is vetting: review the component, sign it, admit it. I did that too. But vetting lowers the probability of a bad component. It does not make the platform’s guarantees independent of the component’s behaviour, and independence is what a guarantee means.

I found out how easy it is to get this wrong on my own design. The retrieval authorisation filter, the thing that stops a retriever returning documents the current principal is not allowed to see, was listed in an early version of the component model as something a retriever could replace with its own implementation. That is reasonable if you trust the retriever. It is a bypass if you do not.

The decision

A fixed set of controls lives in the kernel and is out of reach of every extension. An extension may add controls. It may never replace, disable, or wrap one of these.

The set: identity, policy decision and enforcement, approval, the evidence ledger, secret authority, signing and trust, isolation, emergency revocation, and the run and correlation model. Those nine are the platform. Everything else is a component.

The retrieval filter became the worked example of what “out of reach” means in practice. The fix was three rules. The kernel applies the authorisation filter after any component has run, on the returned items, not before. A component may narrow the result set, never widen it; if a retriever returns something the kernel’s filter would have excluded, the kernel excludes it anyway. And items without authority metadata are excluded. A document that cannot say who may read it is not returned to anyone.

That third rule is the one that bites. It means a retriever that forgets to attach authority metadata returns nothing, which is a loud failure during development and exactly the point.

What I rejected, and why

Relying on vetted components. Review and signing are in the platform and I would not remove them. But a vetting process is a probability reducer. A reviewer misses things, a component gets updated after review, a signing key leaks. The controls in the fixed set have to hold even when a component is actively malicious, and vetting cannot promise that. Non-bypassable is a structural property, not a process outcome.

A pre-filter alone. The tempting design is to constrain what the retriever is asked: pass it the principal’s authority scope and let it query only within that. That is worth doing for efficiency, and the prototype does it. But a pre-filter constrains the ask, not the return. A retriever that ignores the scope, or has a bug, or indexes documents under the wrong authority, still returns whatever it returns. Only a post-filter on the actual items closes that.

Chesterton would have you ask why the fence is there before removing it. This ring is the set of fences where the answer is written on the fence, and the gate has no hinges.

What it costs

Re-checking authority on every returned item, every time. That is a second pass over every retrieval result, with a lookup against the authority metadata for each item, before anything reaches the model. In the prototype the overhead was small because the corpus was small. At municipal scale it is a real latency and compute cost that has to be designed for rather than hoped away.

It also costs a specific kind of test: a deliberately malicious retriever. To prove the post-filter holds, you have to write a component that tries to return documents it should not, and run it against the kernel. I built one. It is an uncomfortable thing to have in a repository and it is the only test that actually demonstrates the property.

And the fixed set is a constraint on the platform’s own evolution. Adding a tenth control to the list is a bigger decision than adding a feature, because it changes the contract with every existing extension.

What it does not do

It does not make the authority metadata correct. If a document is labelled as readable by everyone when it should not be, the post-filter faithfully returns it. The control enforces the labels; it does not audit them.

It does not cover channels the kernel does not see. A component that fetches data over its own network connection and feeds it to the model through a side path bypasses the retrieval filter because it bypassed retrieval. Isolation, another member of the fixed set, is what closes that, and in the prototype isolation was designed more thoroughly than it was proven.

If you take one thing

Write down the controls in your platform that must hold even if a plugin is hostile. Then, for each one, find where it is applied. If any of them runs before the plugin rather than after it, or inside the plugin rather than around it, you have found your version of my retrieval filter.

Standards and practices, as the IT Crowd basement knew, are what the fire extinguisher is for. The ring is the extinguisher that is not also a fire.

References

The allusions and quotations in this piece, with their origin:

• G. K. Chesterton, The Thing (1929): Chesterton’s fence, do not remove a fence until you know why it was put there.
• The IT Crowd, “Calamity Jen” (2006): the fire extinguisher that is itself on fire, and the fire-safety training video.