Prove or hope: what evaluation and guardrail tooling can actually establish

Agent platform field notes, six of eight. Every governance conversation eventually reaches “how do we know it works”. This note sorts the tools that claim to answer that into three questions they actually answer, and one they do not.

The question

When a municipality asks whether an agent is safe to use on citizens’ cases, the engineering team usually points at three things: an evaluation suite, a guardrails layer, and an observability dashboard. All three are useful. They are also routinely confused with each other, and the confusion has a shape: a measurement gets treated as a control, and a control gets treated as proof.

So the question for this note is narrow. For each class of tool, what does it establish, and what does it only suggest?

Three questions, three tools

“Did it behave correctly on cases we chose?” That is evaluation. The tool I rate highest here is Inspect AI, originally from the UK AI Security Institute and now maintained by a non-profit. It composes datasets, solvers, tools and scorers; runs each sample’s tool calls in its own sandbox with no network by default; and writes a compact, versioned log file that has its own viewer and can live in storage you control. Three ideas from it went straight into my platform: the per-sample sandbox, the chained approval policy where a tool call that no approver claims is rejected and every decision lands in the transcript, and the eval log as the audit artefact.

What evaluation establishes: behaviour on the distribution you tested, at the time you tested, with the model snapshot you pinned. What it does not establish: behaviour on the next case, or after the provider updates the model. That is why, in my design, nothing ships without passing its evaluation set, and why a qualification expires.

“Did it stay within bounds on this request?” That is a guardrail. NeMo Guardrails is the clearest example: input, dialog, retrieval, execution and output rails, written in a small flow language, sitting between the application and the model. Execution rails in particular, which guard tool and action calls, are the closest a guardrails product comes to the policy gate of note one.

What a guardrail establishes: that a specific check ran on a specific request and returned a specific verdict. What it does not establish: that the check was correct, or that it could not be bypassed. Many rails are themselves model calls. A model judging a model is a useful second opinion; it is not a control in the sense a security engineer means, because its failure mode is the same as the thing it guards. I use rails as a defence in depth and never as the enforcement point. Enforcement is deterministic and lives in the governor.

“What happened, in production, and what did it cost?” That is observability. Langfuse is the one most teams I meet already run: tracing of prompts, completions, tool calls and retrieval, with evaluation, prompt versioning and cost metrics on top, self-hostable. The same role is filled by Phoenix, Opik and the OpenTelemetry-native stacks.

What observability establishes: what the system did, as seen by the system. What it does not establish: whether it was allowed to, and whether the record has been tampered with. A trace is not evidence. It is the raw material of evidence until something you own has signed and chained it.

Dijkstra said testing shows the presence of bugs, never their absence. He said it in 1969 about programs. It is still the whole evaluation story, only now the program can also argue back.

The question none of them answer

“Is this safe to use on a citizen’s case?” is not a tooling question. It is a decision, taken by a named person, on the basis of the three measurements above plus the legal and organisational facts around them: the data classes, the transfer decision, the oversight design, the contestability path.

That is why my platform has a qualification record and not a test score. The record pins the evaluation contract, its result, the model route, the policy version and the approver, with an expiry. The tools feed it. They do not replace it. A green evaluation closes nothing on its own; a human closes it, and the record says who.

What I learned from reading the code

Two findings tempered my enthusiasm for the evaluation tooling, and both are about defaults.

The sandbox isolation in the evaluation harness is strongest when it generates its own container configuration. Supply your own and the no-network default disappears. The approval chain is off unless you configure it. And there is no score gate: the harness reports, it does not stop a pipeline. Each of these is reasonable for a research tool. Each is a trap for a team that reads the feature list and assumes the governance is on. The lesson generalises: in evaluation and guardrail tooling, the safe behaviour is usually available and rarely the default.

The second finding is about evidence. The evaluation log is excellent as an artefact, but it is not signed, and metadata edits carry whatever author name the editor supplies. So it is auditable in the sense of readable, not in the sense of tamper-evident. For the municipal use case I treat the log as an input and hash it into the evidence chain at admission time, which closes the gap without changing the tool.

What I cannot prove yet

Prototype grade, with one specific caveat. I have evaluation suites for the platform, including a model qualification for one frontier model, and they gate changes in my pipeline. I do not yet have the thing that would make this note more than argument: a long enough production history to show that a qualification expiring, and the evaluation being re-run, caught a regression that would otherwise have reached a case. That is the evidence I am waiting for.

What to do with this

Draw three columns on the whiteboard: prove, control, observe. Put each tool in your stack in exactly one column. Then look for the row that says “safe to use”. If it is in any column, move it out. It belongs to a person, with a date next to their name.

Next note: build versus buy, and the evidence ledger I built so the decision could survive my own enthusiasm.

An evaluation that passed last quarter on a model the provider has since replaced is not resting. It is not pining for the fjords. It has ceased to be.

References

The allusions and quotations in this piece, with their origin:

• Edsger Dijkstra, “Notes on Structured Programming” (1969): testing shows the presence of bugs, never their absence.
• Monty Python’s Flying Circus (BBC, 1969), the Dead Parrot sketch: “pining for the fjords”, “it has ceased to be”.