The pipeline that went dark for three weeks

One of my weekly automations ran every Monday for three weeks straight. No exception, no alert, no red line in a log. It also produced nothing — no output, no notification, no trace that anything was wrong. The scheduler dutifully recorded a run each week. There was just nothing at the end of it.

A green light that meant nothing

The job was a content miner: once a week it reads a set of source files, ranks candidates, and writes a backlog. Boring, reliable, the kind of thing you stop watching. So I stopped watching it.

What I had built for observability was a run marker — a scheduler timestamp that said "this fired on Monday." It fired. Every Monday. The timestamp was honest and completely useless, because it recorded that the process *started*, not that it *produced* anything. A job can start, hang for an hour and 48 minutes, abort with an error nobody reads, and still leave that reassuring green timestamp behind.

The failure was a permission prompt, not a bug

The actual cause was almost embarrassingly mundane. Somewhere in the run, a shell loop read a handful of files — the equivalent of `for f in …; do cat "$f"; done`. In an interactive terminal that is nothing. In an unattended, headless run it opened a tool-permission dialog and waited for an answer that no human was there to give. Since a recent tooling change, that wait has no timeout. So it waited. For 108 minutes, then died with `AbortError: Tool permission stream closed`.

Nothing about that is a code defect. The code was correct. The environment quietly changed the meaning of one line — from "read three files" to "block forever waiting for a click" — and the failure lived entirely in that gap.

What actually caught it

Not a unit test. Tests check the code you wrote against the assumptions you already had. This failure lived in the assumptions themselves.

What caught it was a second, deliberately dumb watchdog that runs on its own schedule and asks one question of every pipeline: *when was the last time this thing wrote a real artifact?* Not "did it start," not "did the process exit zero" — did it produce the output that is the entire point of its existence. A backlog file with this week's date. If the freshness signal is older than it should be, the watchdog turns red and says so, out loud, in a channel I actually read.

That is a dead man's switch. It assumes the monitored system will eventually fail silently, and it makes silence itself the alarm.

The pattern worth stealing

• Never treat "it started" or "exit code 0" as proof of success. Prove the *artifact* — the file, the row, the sent message that was the actual goal.
• Put the health check in a *separate* process on a *separate* schedule. A pipeline cannot report its own silent death; something outside it has to.
• Make silence loud. A watchdog that only speaks up when something is wrong is a watchdog you forget exists. Mine posts a full green table every single day — the green is the proof the watchdog itself is still alive.
• Assume the environment will change the meaning of your code without telling you. Headless runs, permission models, context limits, quotas — the thing you built against is not a fixed contract.

None of this is clever. It's the unglamorous backbone: a second small system whose only job is to distrust the first one. The three weeks of silence cost me nothing except the thing I'd have written in them — and the lesson that the next silent pipeline is already running, I just don't know which one yet.