Business rules are enforced outside the model

Decision records, one of ten. Each piece in this series takes one architecture decision from a governed AI-agent platform I designed and built as a prototype for Dutch local government, on my own initiative, and explains what I chose, what I rejected, and what it cost.

The situation

A municipality runs on rules. Who qualifies for a parking permit, which documents a benefit application needs, how long a complaint may sit before someone has to respond. These rules are written down, they change on a schedule, and when they are applied wrongly a citizen can appeal and a court can look at the reasoning.

The first instinct with a language model is to put the rules in the prompt. It works in a demo. You write “only grant the permit if the applicant lives in zone B” and the model mostly does that. Mostly is the problem. A prompt is a suggestion to a probabilistic system, and the platform I was building had to be able to say, to an auditor, exactly which version of which rule was applied to which case and why.

Alongside that sat a second issue. The platform also had to defend against prompt injection, and I had read enough published red-team research to know that injection detection is incomplete. Any detector I could put in front of the model would catch some attacks and miss others. If the model was also the thing enforcing the rules, a missed injection was a missed rule.

The decision

Business rules are executable, versioned, testable artefacts, and they are enforced by a policy service that sits outside the model.

Concretely: a rule is a file in a repository, with a version, a test suite, and a review history. The policy service loads a specific bundle of those rules and evaluates them against typed inputs. The model never evaluates the rule. When an agent run reaches a point where a rule applies, the harness calls the policy service with the facts of the case, gets a decision back, and that decision is what happens.

The model may receive an explanation of the rule. It is often useful for the model to know that a permit needs proof of residence so it can ask the citizen for the right document. But the model receiving the explanation and the model enforcing the rule are two different things. The model gets to be helpful about the rule. It does not get to decide whether the rule was met.

This pairs with how I treat injection. Detection runs, and its output goes into the record and into the policy inputs as a signal. But detection informs and policy decides. A detector saying “this looks fine” never grants anything, and a detector saying “this looks suspicious” is one more fact the policy can weigh, not a verdict on its own.

What I rejected, and why

Prompt-based enforcement. The argument for it is speed: you can change a rule in minutes by editing text. The argument against it is that it is not enforcement at all. Prompts configure behaviour. They shape what the model is likely to do. They do not constrain what it can do, and they leave no evaluable trace of the decision beyond the model’s own narration of it. For a system that has to survive an appeal, “the model said it checked” is not evidence that it checked.

I also looked at a hybrid where the model enforces and a second model audits. That doubles the cost and keeps the fundamental problem: two probabilistic components agreeing is still not a deterministic decision with a version number on it.

Asking a prompt to enforce a rule is asking the defendant to also be the judge, on the grounds that he has read the law and seems sincere.

What it costs

A dedicated policy service. That is a real component with its own deployment, its own availability concerns, and its own test suite, and in the prototype it is the single most load-bearing piece of infrastructure after identity.

It fails closed. When the policy service has no bundle loaded, or the bundle fails to load, the engine refuses every decision rather than defaulting to allow. In practice that meant a misconfigured environment stopped all agent work until someone fixed it. That is the intended behaviour and it is still annoying at two in the afternoon.

And there is a translation cost. Someone has to turn a paragraph of municipal regulation into a testable rule, and that person has to be comfortable with both the law and the format. That skill is scarcer than either on its own.

What it does not do

It does not make the model’s reasoning correct. The model can still misunderstand a citizen, pick the wrong facts, or ask for the wrong document. The policy service catches a wrong decision; it does not catch a wrong conversation.

It does not cover rules nobody has written down yet. If a municipality applies a rule by habit and never codifies it, the policy service does not know it exists. The prototype makes the written rules enforceable. It does not discover unwritten ones.

And the verification instruments around it, the golden tests of the rule bundles against known cases, were proposed in the design and only partially built. I would not claim they are proven.

If you take one thing

Open one of your agent prompts and find a sentence that starts with “always” or “never”. Ask whether anything other than the model’s cooperation makes it true. If the answer is nothing, that line is a preference, not a control, and you should decide today whether it needs to become one.

Asking the prompt to enforce the rule is the Monty Python argument sketch: you pay for an argument and get contradiction, delivered with great confidence.

References

The allusions and quotations in this piece, with their origin:

• An old legal maxim, nemo iudex in causa sua: no one may be judge in their own cause.
• Monty Python’s Flying Circus (BBC, 1972), the Argument Clinic: “No it isn’t.”