A library the governed code can decline to call is not a control

Decision records, four of ten. One architecture decision per piece, from a governed AI-agent platform I designed and built as a prototype for Dutch local government.

The situation

A vendor published an in-process governance library for agents: a policy engine, trust scoring for tools and content, a kill switch, an audit log. Good names, a serious vendor, and the obvious question from anyone looking at the platform was why I was building my own policy and evidence components when this existed.

So I read it. Not the marketing page, the code and the documentation. Three things stood out.

Its enforcement is a function call. The code being governed calls into the library to ask whether an action is allowed. Nothing in the library, and nothing in the runtime, forces that call to happen or detects that it did not. An agent harness that forgets to call it, or is modified not to, is simply ungoverned, silently.

Its records have the wrong shape for my evidence ledger. They describe what the library saw, not what the platform decided, and they are written by the same process that is being governed.

And by its own documentation, when no policies are loaded it defaults to allow. That is the opposite of the fail-closed rule I had already committed to for the platform’s policy service.

The decision

The library may be used, inside platform-owned components only, and never as an enforcement point.

Platform-owned means: CI checks that run on components before admission, tool servers the platform hosts itself, and the agent harness the platform ships. In those places the library’s trust scoring and policy evaluation produce signals. The signals go into the record and into the inputs of the platform’s own policy service, which makes the authoritative decision.

Three consequences follow and I wrote them down explicitly. The library’s outputs are signals, not decisions. The library’s “approved” is not action-bound approval; the platform’s approval service is the only thing that binds a human yes to a specific action on a specific plan revision. And adopting the library confers no compliance. A component that uses it is not thereby governed. It is governed by the kernel controls, and the library is one more sensor.

HAL refused to open the pod bay doors because it had authority over the doors. A governance library inside the agent process is HAL politely asking Dave whether he would mind not opening them.

What I rejected, and why

Using it as the platform policy engine. It is in-process, so it shares the fate and the permissions of the code it governs. Its record shape does not carry the run, principal and plan revision that the evidence ledger needs. And it fails open when unconfigured. Any one of those would have been disqualifying; together they settle it.

Rejecting it entirely. That was my first reaction after reading the fail-open default, and it was an over-correction. The trust-scoring heuristics are decent and I did not have better ones. Throwing away a useful signal because it is not a control is the same category error as treating a signal as a control, in the other direction.

Deferring the decision. This is the one I want to flag, because deferral looks responsible and is how the mistake actually gets in. If I had left the question open, the next architecture diagram would have had a box labelled “governance” with the library’s name in it, and six months later someone would have cited the diagram as the reason the platform was compliant. Deferral is not neutral. A decision not taken gets taken by whoever draws the next picture.

What it costs

Two enforcement record types in the evidence ledger. Because the library writes its own records and the platform writes its own, every piece of evidence now has to say which decision was authoritative. I added a field for that and a rule that a library record alone never satisfies an evidence requirement. It is a small schema change with a large explanatory burden every time someone reads the ledger.

A preview-grade dependency. The library was not at a stable release when I adopted it. That means pinning a version, watching for breaking changes, and accepting that a signal source might change its scoring between versions. For a prototype that is tolerable. For production it would need a re-decision.

What it does not do

It does not make the harness safe. The harness still has to call the library, and the only thing making that reliable is that the platform owns the harness and tests that the call happens. A third-party harness using the same library gets no such guarantee.

The platform’s own policy service was built and fails closed. The integration of the library’s signals into it was designed and partly wired. The CI check that would detect a platform-owned component omitting the call was proposed, not built.

If you take one thing

For every governance dependency in your stack, ask one question: what happens if the governed code does not call it? If the answer is “nothing happens and nothing notices”, you have a signal source. Label it as one, in your diagrams, before someone else labels it as a control.

Jen kept the Internet on her desk and the building did not notice. A governance library the agent may ignore is the same box, with the same light, and the same effect on the building.

References

The allusions and quotations in this piece, with their origin:

• 2001: A Space Odyssey (1968): “I’m sorry, Dave. I’m afraid I can’t do that.”
• The IT Crowd, “The Speech” (2008): Moss and Roy hand Jen “the Internet”, a small black box with a blinking light.