Evidence against enthusiasm: the ledger behind a build-versus-buy decision

Agent platform field notes, seven of eight. The first six notes contain opinions. This one is about the discipline that keeps those opinions from becoming a procurement decision on their own.

The question

Build or buy is the question a board asks once. Underneath it sit a hundred smaller questions, each of the form “does this capability exist in the field, do we need it, and do we have it?” The failure mode is familiar to anyone who has sat through a vendor evaluation: evidence that something exists somewhere quietly becomes evidence that we have it, or that we are ahead, or that we could never buy it.

So the question for this note is: how do you keep an evidence base honest when the people maintaining it, including you, want a particular answer?

What I built

For the agent platform I kept a ledger of capabilities. By the time it stabilised it held over two hundred rows, each a capability that a governed agent platform might need, derived from more than a hundred external projects and several hundred research and practitioner sources, with each row classified by strategic posture: core, commodity, differentiator, open window, disrupter, or moat.

The classification drove decisions. “Commodity” meant buy or adopt. “Core” meant build or own the contract. “Differentiator” and “moat” meant invest. Those words carry money, so the rules for assigning them had to be stricter than the rules for an essay.

The rule that did the work

I wrote a short evidence standard with one central rule:

Evidence that another project implements a capability proves external feasibility. It does not prove that we built, shipped, operate, outperform with, uniquely hold, or compound value from that capability.

Everything else follows from it. Every claim names its subject: the external field, a named external party, the target platform, or our own organisation. Every claim names its time scope: historical, current as of a date, target state, forecast, or conditional. “The field implements signed receipts”, “the platform should emit signed receipts” and “we operate signed receipts in production” are three different claims, and the ledger will not let you write the third with the evidence for the first.

A few other rules turned out to matter more than I expected:

• A source occurrence is not an independent source. Twenty articles repeating one vendor announcement count as one source family. Convergence counts families.
• Reasoning is not evidence. Panels of models, reviewer votes and confidence averages can organise and challenge evidence. They cannot create it, and they do not count as confirmation.
• No pseudo-probabilities. A number is a probability only if its calibration method is recorded. Otherwise it is low, medium or high, with the gap stated.
• Absence defaults conservatively. If ownership, deployment or currency cannot be verified, the field says unknown. It does not say the favourable thing.
• Caveats survive every transformation. Abstract-only access, vendor authorship, a partial read: the caveat stays attached to every claim derived from it.

The first thing it did

When I applied the standard to the existing ledger, it performed what I now call the first-party regrade. Every row that asserted a present first-party asset, every “edge”, every held “differentiator”, every “moat”, was downgraded, because the evidence behind it was external. We had read a hundred projects. We had not built a hundred capabilities. The ledger’s decision grade went to “blocked”, and stayed there until first-party proof existed.

That was the correct outcome and it was unpleasant. It meant the architecture could no longer claim advantage on the strength of research. It could claim need, which is a different and more defensible thing, and it could claim the few capabilities that actually had code, tests and a running demonstrator behind them.

I mention this because the same inflation happens in every organisation evaluating AI right now. A pilot becomes “deployed”. A vendor’s benchmark becomes “our capability”. A capability on a roadmap becomes a capability in a tender response. The regrade is the only antidote I know.

Richard Feynman put it better than any evidence standard: the first principle is that you must not fool yourself, and you are the easiest person to fool. The regrade is that principle with a schema.

How it feeds the decision

The ledger does not decide. It supports a decision record, with requirements, alternatives, trade-offs, an owner, acceptance tests and a review date. The platform has more than forty of those. The ledger’s job is to make sure that when a decision record says “commodity, adopt”, the evidence behind it says the same thing, with the same subject and time scope, and that anyone can follow the citation to the supporting passage rather than to a repository name.

For the build-versus-buy answer itself, the ledger converged on what notes one and three argued from architecture: the runtime, the models, the gateways, the durable-execution engine and the observability stack are commodity or adopt. The governance spine is core, and nothing in the field could be bought that would own it on our behalf without becoming the trust root.

What I cannot prove yet

The ledger is a decision tool, not a result. It has made the architecture more honest and slower. It has not yet been tested by the thing it exists for: a procurement challenge, a vendor claiming we were wrong about a classification, or an auditor following a citation and finding it does not resolve. The validators check structure, subjects, time scopes and locators. They do not check that I read the passage correctly.

What to do with this

If your organisation has an AI capability list, run one query against it: for every row that says “we have”, find the first-party artefact. If you cannot, change the row to “the field has” and note the gap. Then look at how many of your strategic claims survived. That number is more useful than any roadmap.

Final note next: the written rules under which AI coding agents are allowed to work on the platform, and why validators detect but never prevent.

Apart from the runtime, the models, the gateways, the execution engine and the observability stack, what has the field ever built for us? The spine. The field did not build the spine.

References

The allusions and quotations in this piece, with their origin:

• Richard Feynman, “Cargo Cult Science”, Caltech commencement address (1974): the first principle is that you must not fool yourself.
• Monty Python’s Life of Brian (1979): “What have the Romans ever done for us?”