Confidence Is Maintenance, Not a Purchase

While researching buy-versus-build for enterprise AI, we kept meeting the same encouraging statistic: vendor-led AI efforts succeed about 67% of the time, internal builds about 33%. It appeared in article after article, which looked like independent corroboration. We traced every instance. All of them resolve to one non-peer-reviewed working paper, the same MIT NANDA study behind the famous 95%-of-pilots-fail number, which we had already publicly flagged as a pessimistic bound rather than settled fact. One weak source, echoed until it sounded like a chorus.

That was the loudest kill in three rounds of research. The full count: for one piece we surfaced 125 claims across 29 sources, adversarially verified the 25 that would bear weight, and killed 3. The next two rounds killed 7 each. Our do-not-use ledger now holds 17 dead claims, each with the reason it died. The pattern across all of them is the tell we now check first: a precise number with no findable primary is the signature of laundered content marketing. And the noisiest topics are the ones where both sides of the argument are selling something.

The lesson we expected to draw was about other people’s numbers. Then the same week handed us the uncomfortable sequel.

Our workspace believed things that had stopped being true

While auditing everyone else’s claims, we ran a drift check on our own content factory. Two of its skill files confidently asserted that certain reference documents “do not exist.” The documents existed. They had existed for three weeks. The assertions were true when written, then the world moved and the text did not. Nothing was lying. Things had simply expired.

That is the same disease we were diagnosing in the wild. A claim in a published article and a claim in your own docs both decay the same way: true at the time of writing, load-bearing forever after, verified never again. You cannot outsource confidence to a survey, and you also cannot outsource it to your own past self.

So the question the whole piece turns on: what keeps confidence alive after the day it was earned?

The discipline we stand on, credited

Our workspaces are built on the Interpretable Context Methodology, ICM, developed by Jake Van Clief. It is the best workspace framework we have used, and it now has a public, MIT-licensed write-up: Van Clief and McDermott, “Interpretable Context Methodology: Folder Structure as Agent Architecture”, a preprint, not peer-reviewed, and we cite it with exactly that label.

ICM’s core move is organizing an AI workspace as five layers of context, from a global identity file down to working artifacts, with a single agent reading different files at each stage. Its thesis is that filesystem structure can replace multi-agent orchestration. Readers of our agents piece will recognize that conclusion: structure over agency, arrived at independently from the architecture side. The paper also states its own limits plainly, informal data collection, a self-selected community, no controlled comparison, which is part of why we trust it.

What the five layers give you is space: where things live. What they do not give you is time: what happens as the workspace ages. That is where our week pointed.

The escalator

Work in an AI workspace sits in one of three layers. Code, which runs the same way every time and costs nothing. Rules, written instructions the model follows. Judgment, the model thinking fresh, which is powerful, metered, and able to decide differently on Tuesday than it did on Monday.

The maintenance law we have converged on is a movement rule between those layers: the second time judgment makes the same call, the call becomes a rule. The second time a rule is broken, the rule becomes code. Two strikes, demoted a layer.

Credit where the record demands it: the first rung is in Van Clief’s paper. Section 6.3 argues that if a practitioner keeps making the same edit, “that is a signal that the stage contract should say” what the edit keeps saying, and the future-work section proposes tracking repeated edits to surface exactly such promotions. Repeated judgment becoming a rule is his, in print, before us.

Our extension is the rest of the ladder. Rules decay too, and a rule that gets violated twice earns enforcement in code, where violating it is no longer possible. Last week our publish pipeline would have shipped any stray markdown file in the final folder to this live site. The guard was a convention, held in memory and commit messages. It nearly fired twice. It is now a filter in the script, and the mistake is not unlikely anymore, it is impossible. That incident trail is one shop’s logbook, not a study, and we offer it as exactly that.

There is also a reason nobody teaches the descent. The judgment layer is the only metered one. Every demotion moves work off a vendor’s meter and into code you own, so no one selling metered inference has an incentive to walk you down the ladder. To be fair where fairness is due: Anthropic’s own engineering guidance says to build the simplest thing that works, directly against their meter. One vendor arguing against its margin does not cancel the gradient everyone else rides.

The same ladder runs on beliefs

Here is the unification the drift check forced on us. Everything in a workspace is either work being done or claims being believed, and both rot on the same schedule. Our stale skill files were not a documentation problem. They were judgment-layer beliefs that had never been demoted.

So the escalator applies unchanged. A belief starts as prose, which is judgment: “the reference docs exist.” Demote it once and it becomes a dated claim with a check attached: verified on this date, re-verify with this command. That is a rule, and it can be audited. Demote it again and the check runs on a schedule, so the doc cannot silently rot. That is code. Our own first rung is already in place: after the drift repair, our audit process now carries the instruction to treat every recorded fact as point-in-time and re-test it against disk. The scheduled rung is designed and not yet running, and we say so.

The convention is small enough to adopt in an afternoon: every load-bearing assertion in your docs gets a date and a re-verification command. Docs stop being prose that can lie and become claims that expire.

The bet

A method that only audits the past is a museum. So we close with a prediction, registered before we look.

We have frozen a six-arm experiment: three model tiers, each run plain and wrapped in structure, 100 extraction tasks, three runs each, graded by exact match with no judge anywhere. The protocol, the arms, the sample size, and the analysis were committed to git before any API call, and the commit timestamp is the notary. Our registered prediction: the cheap model wrapped in structure beats the frontier model running plain, on cost per successful task. If the data comes back the other way, that is the finding, and it publishes exactly as our scaffolding reversal did.

Confidence is not a thing you buy once, from a vendor, a survey, or your own earlier certainty. It is a thing you maintain: verify what you cite, credit what you adopt, demote what repeats, date what you believe, and bet in public before you look. The results piece will keep the appointment either way.