Wire the Feedback Loop You Already Own

We wrote a few weeks ago about Autodesk paying $3.575 billion for maintenance records. The largest acquisition in the company’s history, roughly 26 times forward revenue, and the stated reason was contextual data about what happens to a thing after it leaves the drawing.

The point of that piece was that the feedback loop is the scarce asset. The question it left open is the one worth answering: what do you actually do, given that you do not have $3.575 billion.

You have something Autodesk did not. The data it needed sat inside another company. Yours is almost certainly already inside your own building, in a system nobody has connected to the one producing the outputs. So this is a join, not an acquisition, and the six steps below are the join.

Each step has three parts. Run this is the action. What good looks like is the default with its reasoning attached. Change it if is the condition that moves the default.

One thing to settle before step 1, because getting it wrong is why most of these projects collect data nobody can read. There are two different questions here and they need different answers.

The first is the business outcome: resolutions per hour, cycle time, first-pass yield. That is check 1 of our adoption guide, it is what finance recognizes, and you should already have it.

The second is the per-output signal: was this particular draft right or wrong, and how would you know. That one is what makes a feedback loop readable. A business metric moving tells you something changed somewhere across ten thousand outputs. It does not tell you which outputs were wrong or how. Teams that instrument only the first end up with a dashboard that goes up or down for reasons nobody can name.

Here is the whole thing as one system before we walk it. The top row is what your workflow already does. The bottom row is what these six steps build.

Step 1: Name the per-output signal

Run this. For the one workflow you picked, finish this sentence: “This output was wrong if ____, and the person who would notice is ____.” Write it for the workflow, not for the deployment. If you cannot finish it, you are not ready to capture anything, because you would not recognize a failure in the data you collected.

What good looks like. A signal a human already produces as part of doing the work. Someone edits the draft. Someone reopens the ticket. Someone escalates. Someone reverses the decision six weeks later. Those are all signals that already exist and already get recorded somewhere, which is the entire reason this is cheap.

The signal to avoid is one that requires a person to do something extra, like rate the output. Extra steps decay. A thumbs-up widget gets used enthusiastically for three weeks and then stops, and you are left with a dataset that describes your first three weeks.

Change it if the workflow has no human in the path at all, in which case your signal has to be a downstream system event: the refund got processed, the classification got overridden by a rule, the ticket came back. Those are weaker because they arrive later, and you should expect to wait longer before any of it is readable.

Step 2: Find where that data already sits

Run this. Go looking for the signal from step 1 in the systems you already run. In practice it is in the ticketing system, the CRM activity log, the QA queue, the version history of whatever document store the output lands in, or a support thread. Write down the system, the table, and the field.

What good looks like. You find it. That is the usual outcome, and it is the good one. The work here is joining, not collecting, and a team that concludes it must build new capture has usually not looked hard enough at the systems the work already flows through.

Autodesk is the instructive case precisely because it is the exception. A company with four decades of design software concluded the record it needed did not exist anywhere it could reach, and the record that contained it was a maintenance log written by a technician on a phone, in a different company’s software. That is what it looks like when the data genuinely is not yours. It cost $3.575 billion to fix.

Change it if you find the signal in a system owned by a different team. Then the constraint is not technical and this guide cannot solve it for you. Name the owner and get agreement before you write any code, because a join that depends on another team’s cooperation is a negotiation with a schema attached.

Step 3: Name the join key

Run this. Find the identifier that links one model output to the thing that happened to it. Ticket ID, document ID, request ID, whatever your systems already carry. Then check that it survives the trip: if the output gets copied into a new record without the ID, the chain breaks there and you need to fix that before anything downstream is worth building.

What good looks like. One stable identifier, written at generation time, carried through every system the output touches. If you have to reconstruct the link later by matching timestamps and text similarity, you do not have a join, you have a research project with an error rate.

Change it if no identifier survives end to end. Then the honest first move is to add one, which is usually a small change at the point of generation, and to accept that your usable history starts on the day you add it rather than backfilling something unreliable.

Step 4: Record the edit, not the acceptance

This is the step that carries the guide.

Run this. When a human changes a model’s output, store the before and the after. Not a boolean, not a rating, not a helpfulness score. The two versions, and enough context to tell them apart later.

What good looks like. Structured before-and-after text you can read. Acceptance tells you somebody clicked. The diff tells you what was wrong, which is the only version of the signal you can act on. Everything downstream of this step, every prompt fix, every retrieval fix, every eval task, comes out of reading diffs.

Two published results explain why the softer version does not work. METR ran a randomized controlled trial with 16 experienced developers across 246 real tasks: they predicted AI would make them 24% faster, they were 19% slower with it, and afterward they estimated it had made them 20% faster. Perception did not merely miss, it pointed the wrong way. Anything you capture by asking people is measuring that. A diff is not an opinion.

And DORA’s 2025 report, across roughly 5,000 professionals, found across both waves that AI adoption degraded delivery stability. The failures are real and they are already happening in your workflow. The diff is where they are written down.

Change it if you are in a regulated or sensitive domain where storing both versions creates a retention problem. Then store a structured description of the change rather than the raw text, and accept that you have traded resolution for compliance. What you should not do is fall back to a boolean, because a boolean is the same as capturing nothing while feeling like you did something.

Step 5: Read the diffs on a cadence

Run this. Put a recurring hour on one person’s calendar. They read the week’s diffs and write down what kept going wrong, in plain language, in a file that persists.

What good looks like. Weekly. That is our default and the reasoning is that below weekly you lose the thread of what the model is getting wrong, and above it you are reacting to noise. It is our cadence rather than a measured optimum, and we have said so every time we have used it.

Capture everything until volume forces you to sample. At the volumes most teams start at, sampling buys nothing and costs you the rare failure, which is the one worth reading. That is also ours, and it is not a statistical claim.

Change it if your volume or your risk says otherwise. Under roughly 50 outputs a week, go monthly. In a safety-critical workflow, go daily. And whoever reads should not be the person who championed the deployment. Staw’s escalation studies found people commit the most additional resources to a failing course of action precisely when they started it, which makes the author the worst available reader of evidence about their own project.

Step 6: Route what you learn

Run this. Every recurring failure goes to exactly one of four places, decided when you name it: a prompt or retrieval fix, a workflow change, a hardcoded rule, or an eval task. Write the destination next to the finding, or the finding sits in a document and nothing happens.

What good looks like. Two same-direction diffs before you act on a pattern. One diff is an anecdote and acting on it means you will spend the next month chasing the last thing you read. Two is our working threshold rather than a measured one, and we use it on our own analytics for the same reason.

The routing itself is mostly mechanical once the finding is written down. If the model had the information and used it wrong, that is a prompt problem. If it did not have the information, that is retrieval. If the step never needed judgment in the first place, that is a step you should be hardcoding rather than fixing. And if the failure is one you would want to catch automatically forever, it becomes an eval task.

Change it if the same finding keeps arriving and keeps getting routed to a fix that does not hold. That is usually the signal that the workflow itself is wrong rather than the model, which is the most expensive finding this loop produces and the one most likely to be ignored.

The order is the argument

Run them in sequence, because each is load-bearing for the next. You cannot go looking for data until you know which question it answers, so 1 gates 2. A diff you cannot attach to the output that produced it is an anecdote, so 3 gates 4. And step 5 is what makes the whole thing a loop rather than a log: data that nobody reads on a schedule is storage, not feedback.

Where this stops is deliberate. Step 6 routes findings; it does not build the instrument that scores them. Turning recurring failures into an eval, with the task count and the pass criteria and the two-experts-agree test, is a separate piece of work and we will write it separately.

One honest expectation to set before you start. Kohavi’s base rate from years of controlled experiments is that about a third of well-designed changes succeed, a third do nothing, and a third make things worse. Your loop will mostly tell you that things you believed are not true. That is the loop working. A feedback system that only ever confirms the deployment was a good idea is not measuring anything.

Autodesk concluded it could not build its way to this data and wrote the biggest check in its history. Most companies will not get that option, and most do not need it. The record you need is already being written, by the people already doing the work, in a system you already pay for. Nobody has connected it to anything.