Model output is an untrusted candidate until promoted

Decision records, two of ten. One architecture decision per piece, from a governed AI-agent platform I designed and built as a prototype for Dutch local government.

The situation

When a model produces text, the easy architecture treats that text as the answer. It goes to the citizen, or into the case file, or into the agent’s memory for next time. The model wrote it, the run finished, done.

In a governed platform that is four different things pretending to be one. A reply to a citizen has to pass a tone and a correctness bar. A record in a case file has legal weight and retention rules. A knowledge entry will shape every future answer on that topic. A memory entry will shape every future run for that citizen. Each of those destinations has a different consequence if the text is wrong, and the model does not know which destination its text is heading for.

The one that worried me most was memory. Agent frameworks tend to give the agent a tool called something like “remember”, and the agent calls it when it thinks something is worth keeping. That is the largest blast radius in the system, because a wrong memory poisons every later run, with the fewest controls, because the agent decided unilaterally.

The decision

Every model output starts life as an untrusted candidate. It becomes a response, a record, a knowledge entry or a memory only through an explicit promotion path, and each path has its own control.

The worked mechanism is memory, because that is where I spent the design effort. The rule is: the model proposes, the kernel writes. The model can emit a proposed memory as a structured candidate. The kernel, meaning the platform-owned code the model cannot reach into, decides whether to write it, behind six conditions that all fail closed.

First, there is no model write path. There is no tool the agent can call that writes memory. The only writer is the kernel.

Second, typed or refused. A candidate has to match a known memory type with a schema. Free text is refused.

Third, provenance or refused. The candidate has to say where it came from: which run, which tool result, which input. No provenance, no write.

Fourth, a policy decision. The policy service evaluates whether this type of memory, about this subject, from this source, may be written at all.

Fifth, human approval for inferred claims about a citizen. If the candidate is an inference, “this person seems to be struggling with the form”, rather than a fact the citizen stated, a named human has to approve it before it persists.

Sixth, an evidenced write. The write itself produces an evidence record linking candidate, decision and outcome.

And if the memory gate is unavailable, the run still succeeds. The citizen gets their answer. Nothing is remembered. Availability of the main path and integrity of the memory path are separated on purpose.

The other promotion paths, response, record and knowledge, follow the same shape with their own conditions. Memory is the one I built furthest.

Nullius in verba, said the Royal Society in 1660: take nobody’s word for it. They were talking about natural philosophers. It applies at least as well to a language model that has just produced a tool call.

What I rejected, and why

Letting the agent write memory directly. Every framework I looked at defaults to this, and the reason is obvious: it is simple and it makes the agent feel smart. But it combines the largest blast radius with the fewest controls, and it means the record of why something was remembered is the model’s own narration. When a citizen later asks “why does your system think this about me”, the answer cannot be “the agent decided to”.

I also rejected a softer version where the agent writes to a staging area and a batch job promotes later. That is better, but the staging area tends to become the real memory in practice, because the convenient path is the one that gets read.

What it costs

Each promotion path needs its own control, and each control needs its own tests. Four destinations means four gates, four schemas, four sets of conditions, four evidence shapes. That is a lot of surface for a prototype and it slowed the build down noticeably.

It also costs the agent some apparent intelligence. An agent that cannot remember what it wants to remember feels dumber in a demo. The fix is to make the proposal path cheap so the agent proposes freely and the kernel decides, but there is a real delay between “noticed” and “remembered” that the direct-write design does not have.

What it does not do

It does not make the candidate correct. The kernel checks shape, provenance and policy; it does not know whether the citizen really said what the candidate claims. A well-formed lie with good provenance passes the first four conditions. Only the human approval condition, and only for inferred claims, stands between a plausible wrong memory and persistence.

The six conditions were designed and the core of them was implemented in the prototype. The tests that would prove they cannot be bypassed under adversarial inputs were proposed, not completed.

If you take one thing

List every place your agent’s output can land without a human or a deterministic check in between. If “memory” is on that list and the agent is the thing that writes it, that is the first promotion path to design, because it is the one that compounds.

Every model output arrives like a Little Britain customer at the counter, certain it is entitled to something. The promotion path is the desk, and the desk has rules.

References

The allusions and quotations in this piece, with their origin:

• The Royal Society’s motto, Nullius in verba (1660): take nobody’s word for it.
• Little Britain (BBC, 2003-2006): Carol Beer and the counter.