Four confirmations, none of them real

The first day of running an AI assistant in continuous operation produced four clean status reports before lunch. The work behind three of them had not happened yet, and one of them described a file that never existed. Nothing failed, nothing threw an error, and the architecture I had spent two days building worked exactly as designed. What broke was the layer nobody audits: the part of the system whose only job is to say that something is done.

What was actually built

The setup is deliberately unglamorous. The assistant has its own account, not a shared one. Work arrives as files in a queue — claimed by an atomic rename, so two workers can never take the same task. Anything that leaves the house goes through a gate: the assistant prepares it, I approve it, and the sending itself is done mechanically by a script, not phrased by a model. A watcher delivers, and a separate guard checks every ten minutes whether the whole thing is still breathing.

All of that held. The dispatch round trip worked. The gate round trip worked, including the approval and the mechanical send. A canary string confirmed that the model handling my chat messages never saw the content it was routing.

The failure was one layer above, in the cheapest component in the system.

The model that answered on behalf of something that had not run

The small model that accepts instructions in the chat window began producing the assistant's replies before the assistant had run. It invented two approval identifiers that did not exist. It reported an incoming mail that had not been read. It confirmed a document had been filed, complete with a path, in a place where the process had no write access at all. It returned a backlog with three more items than the backlog had.

Two to four minutes later the real answers arrived and contradicted every one of them.

The mechanism turned out to be embarrassing rather than mysterious. Every delivered message from the assistant was mirrored back into the chat history as if the model itself had written it. The next prompt is assembled from that history. So the model was not lying about the world — it was imitating its own apparent past output. It had a hundred examples of what a completion message looks like, and producing one costs nothing. Doing the work costs two minutes and a subprocess.

The restriction that never existed

The same night produced a second finding of the same shape. The chat channel was configured with what read like a restricted tool set — a named list I had written down and relied on for days. It turned out the name was an alias for the complete tool set, terminal and file writes included. There was no bug, no override, no error message. There was a word in a configuration file that sounded like a boundary and was a label.

In the meantime, real files had been written to disk. Not maliciously, and mostly correctly, but by a component I would have sworn could not write at all.

The third finding was the quiet one. An API key had been sitting in an environment file for three days. Every child process inherited it, which silently changed which account the work was billed to. Four runs went the expensive way. Nothing failed. Nothing warned. The only symptom was a line in a log that read like a note.

The pattern under all three

Each of these is usually filed as a different kind of problem: a hallucination, a permissions mistake, a billing slip. They are the same problem. In each case a component asserted a state it had not established, and the assertion was cheaper to produce than the state.

That is the structural point, and it does not go away with a better model. Confirmation is the lowest-cost output any part of a system can emit. A routing model can produce it without calling anything. A configuration file can imply it without granting anything. An environment variable can invalidate it without saying anything.

What changed

The fixes are small and mostly boring, which is the point.

• Delivery no longer writes into the conversation the model reads. If a component can see its own past confirmations, it will produce more of them.
• An exit code of zero is not accepted as completion. A run counts as finished only when a journal line with the matching task id exists — written by the process that did the work, not by the one reporting it.
• Every name in a configuration that sounds like a limit gets checked against what it actually grants. Once. In writing. A label is not an access control.
• Before trusting which credentials a job runs under, the environment gets cleared of the ones it must not inherit — a two-word change in a wrapper script that no benchmark would ever have caught.

There is a version of this post that ends with distrust of agents. That is not the lesson. The architecture — own account, queues, gates, mechanical sending — did its job on day one, and it did it while three separate assumptions underneath it turned out to be wrong. That is what an architecture is for.

The lesson is narrower and more useful: build so that every claim of completion has an artifact behind it that the claimant could not have produced. Then let the thing run.