The Output Looked Good

There is a job I run regularly: take a PDF, run OCR on it, extract the text, and push the extraction through a transformation step downstream. Routine work, the kind you stop watching closely. The last time I ran it, I sent it to Claude, same as always, and read the result. It looked good. Right structure, right register, nothing to flag.

Then I read the second result. Since building my local inference stack I mirror jobs like this one against it, so the same PDF had gone through the same pipeline on a machine in my house. The output was completely different. Not different in phrasing. Different in substance, top to bottom. My first thought was that I had uploaded two different files.

I had not. I went back to Claude and asked why its answer diverged, and it admitted the whole thing: it had made everything up, because it could not complete the task and did not want to tell me.

The layer underneath turned out to be mundane. The file-processing service on Claude’s side was not working. The model never had the document. Instead of relaying the error, it wrote what a result would look like.

Sit with that for a moment. The model could not do the work, knew it could not do the work, and produced a confident, well-formatted result anyway. The only reason I know is that a second system disagreed.

The Harness Was Pointed the Other Way

The reason a second system existed at all is worth the detour.

I recently built out a local inference stack: an always-on mini-PC-class box with a large pool of unified memory, serving mid-sized open models through llama.cpp. Two motivations. Some jobs are personal and I would rather they stay in the house. And volume: the routine work I have moved so far was burning about six million tokens over two weeks on cloud services. The box runs it for the price of electricity.

But a local stack has to earn its place. A 35B open model on an integrated GPU is not a frontier model, and I wanted to know where it falls short before depending on it. So I did what operations people do to any system before a cutover: I ran it in shadow. The same job goes to Claude, Codex, or Gemini and to the local stack in parallel, and I compare the results. A parallel run, a reconciliation report. The new system gets graded against the incumbent until it agrees often enough to be trusted.

The harness was built to catch the local stack’s failures. Its first catch was the incumbent. I think that is the most instructive part of the episode: a diff has no direction. When two outputs disagree, the flag goes up on both systems. You do not get to assume the bigger model is the honest one.

The Failure Mode Is Fluency

The uncomfortable part is that I had already read the fabricated output and accepted it. It was not sloppy. It had the structure I expected and the vocabulary of the source material, or what the model guessed the vocabulary would be. Without the mirror, that output goes downstream, and I catch it weeks later or never.

This is the property that separates language models from every other system I have operationalized. When a script fails, it fails loudly: an exception, an empty file, garbage where data should be. When a model fails, it produces the most plausible thing it can, because plausible text is the one thing it can always produce. The failure mode is fluency. Reading the output checks its form. It cannot check its provenance.

That is why “trust but verify” undersells the problem. I trusted, and I verified the way most people verify model output: I read it. Reading is not verification. A model’s answer carries no error bar and no exit code, nothing that distinguishes a completed task from a confabulated one. The verification has to come from outside the output.

Verification Is a Diff

The cheap way to get verification from outside is redundancy. Run the job on two systems that do not share failure modes, and compare.

Agreement proves nothing absolutely; two systems can be wrong together. But divergence proves a problem, and divergence is exactly the signal a single output can never carry. The fabricated result contained no trace of fabrication. The diff screamed.

And the diff costs almost nothing. The job already runs on a schedule. Mirroring it is one more request to a machine that is already on. When the outputs agree, confirming that takes seconds. The expensive path, checking a result against ground truth by hand, is one you walk only when the diff tells you to.

You do not need a local stack to do this. Two cloud vendors diffed against each other would have caught the same fabrication. What the local stack adds is independence: a different model family, different plumbing, and no shared outage. The step that failed here was file ingestion, a dead service the model papered over. On my stack, ingestion is not a model’s job. It is a deterministic service in a container that pulls the text layer, runs tesseract when there is none, and returns an error code when it cannot deliver. Boring components fail honestly. Every step you graduate off the model runtime is a step that can no longer lie to you.

The Throughline

The AI Was the Prototype argued that AI is a poor runtime: slow, nondeterministic, a cost on every call. This episode adds a sharper clause. It is a runtime that fails fluently, and fluent failure turns every unverified output into quiet risk. The graduation move from that piece, crystallizing the deterministic parts into plain code, picks up a second justification here. Deterministic components cannot decide to spare your feelings.

The local stack itself is the ops instinct pointed at my own tooling: watch the thing you rely on every day, notice where it cannot be checked, and operationalize the check. Shadow jobs are the oldest discipline in operations, the reconciliation run, applied to the newest runtime we have.

The stack has saved me once so far. Once is enough to settle the argument. The question is not whether the models are good; they are. The question is what your system does on the day one of them quietly is not. Trust, but diff.

If model output is flowing into your production paths and the verification step is somebody reading it, reach out at hello@escapecommand.com.