Three release gates, and only a human releases

Decision records, seven of ten. One architecture decision per piece, from a governed AI-agent platform I designed and built as a prototype for Dutch local government.

The situation

The architecture said evaluation gates releases. I believed it until I asked what evaluation meant. For every dimension I cared about, correctness, safety, usefulness, conformance, I needed a method and an independent oracle, something outside the system that could say what the right answer was. Not one dimension had both. Where I had a method, the oracle was a model. Where I had an oracle, I had no method to apply it at scale.

And the model-as-oracle option has known defects. Published research shows judge models prefer outputs that resemble their own. Position bias flips verdicts depending on which candidate is shown first. A gate built on that is a gate that opens for the wrong reasons.

So “evaluation gates releases” was a claim without a mechanism. I had to either find the mechanism or stop making the claim.

The decision

Three gates, with different powers.

G1 is deterministic. Conformance tests against the component contract. A denial corpus: inputs that must be refused, checked that they are. An approval-bypass corpus: attempts to act without approval, checked that they fail. Isolation checks. Regression tests. Final-state checks on what the run left behind. G1 may block a release. It may never release one.

G2 is statistical. A paired non-inferiority test against the incumbent version, on the same inputs, with a reported confidence interval. The question is not “is this good” but “is this no worse than what we have, within a stated margin”. Inconclusive is a first-class verdict: if the interval straddles the margin, the answer is inconclusive, and inconclusive is never rounded up to pass. A judge model may contribute to G2, but only if it is independent of the model under test, only with positions swapped and both orderings scored, and only if calibrated against a human-labelled sample. G2 may block. It may never release.

G3 is a named, accountable human. The only gate that releases. And it may do so only when G1 is green. A human cannot override a deterministic failure; they can only decide whether to proceed when the deterministic checks pass and the statistical ones are either green or inconclusive.

The evaluation result is a signed record produced by a platform service. Never by the agent, never by the harness that ran the agent. It is bound to the full configuration: model version, prompt bundle, policy bundle, tool set. There is no single overall score, because a single number hides which dimension failed. And the record must declare its limitations, including which dimensions had no oracle.

Golden sets are synthetic-first. Real citizen data does not go into an evaluation corpus by default; synthetic cases that exercise the same rules do.

Goodhart’s law in one line: the moment a score releases software, the software optimises the score. So nothing releases on a score, and the human who releases is the one who cannot be optimised against.

What I rejected, and why

A single score with a threshold. Everyone wants one. It compresses incomparable dimensions into a number that looks like a decision, and it hides the fact that the number came from a judge with known biases.

Releases gated by a judge model. If a model releases, a model is the sole approver, and that violates a principle the platform enforces on everyone else. Beyond principle, it is unreliable in documented ways.

Evaluation inside the harness. It is convenient to have the agent runtime evaluate its own run and report. But then the subject produces its own evidence. The point of a signed record from a separate service is that the thing being judged cannot write the verdict.

What it costs

Three corpora to build and maintain: conformance, denial, approval-bypass. A paired evaluation harness that can run the incumbent and the candidate on identical inputs. A calibration process for any judge model, which means a human-labelled sample that itself has to be maintained. A signing service. And the organisational cost of a named human on every release with no way to delegate it downward to a dashboard.

What it does not do

Naming gates does not create oracles. G1 proves conformance and the absence of known failures. G2 proves non-inferiority on the dimensions a judge can see. Neither tells you whether the agent’s output had the intended effect in the world. That effect oracle, the thing that would say “the citizen got the right outcome”, is still missing, and I do not have a design for it.

In the prototype, G1 was built. G2 was designed and the paired harness was partly built. The signed-record service was proposed. G3 is a role description, not software.

If you take one thing

Find the number your release process uses to say an AI change is ready. Ask who produced it and whether they could have produced a different one. If the answer is “the same system we are releasing”, your gate is a mirror.

A statistical gate that rounds inconclusive to pass is a Weeping Angel. Perfectly safe, as long as nobody ever stops looking at it.

References

The allusions and quotations in this piece, with their origin:

• Charles Goodhart (1975), as paraphrased by Marilyn Strathern: when a measure becomes a target, it ceases to be a good measure.
• Doctor Who, “Blink” (BBC, 2007): the Weeping Angels, which move only when unobserved.