All insights
Operations

Half of them went through

Outages announce themselves. Partial success looks exactly like a working system.

Someone on the team at a client of mine noticed that a list of trial contacts looked short. Not empty, not broken, just short. Two dozen people had signed up; roughly half of them existed in the target system. There was no error log to look at, because nothing had gone wrong in the sense the job understood the word. It had processed what it accepted and reported success.

Three symptoms, one shape

The cause of the short list was mundane: a date field the form and the target system disagreed about. Records carrying the format the target rejected were dropped, silently, one by one. The rest went through. From the outside the run looked like every other run that week.

In the same week, two more things surfaced at the same client, and it took me a while to see that they are the same thing.

A new subscriber arrived with a misspelled first name, no tags, and no segment membership. Formally a valid record, because the email address is the only required field. Everything that would have made the record useful was optional, so it was empty. The person received a message that began "Hi," followed by nothing, and unsubscribed shortly after.

And a shared password list sits in a protected spreadsheet nobody can edit any more. Asking the person who created it did not resolve the protection. It still opens, it still shows the passwords, it is still technically fine.

None of these three produced an error. All three produced a system that is quietly worse than the one on the diagram.

Partial success has no alarm

An outage is a gift. It stops something a person cares about, so a person shows up. Partial success stops nothing. The dashboard is green because the job finished. The list is populated because most records are there. The record is valid because the required field is filled.

A failed job wakes someone up. A half-finished job gets promoted to normal.

The uncomfortable part is that automation makes this worse, not better. When the work was manual, the missing half was visible as work that had not been done: someone would have been sitting in front of the remaining twelve rows. Automating it converts "not done" into "done, but only partly", and that state has no natural place to complain from.

The person who fixes it is hiding it

Here is the bit that changed how I think about maintenance weeks.

Someone on the team fixed the missing half by hand. Correct move: the contacts needed to exist, the fix took less time than the diagnosis would have. They also corrected the misspelled name manually, and added the missing tags.

Both fixes worked. Both also removed the only evidence that anything was broken. After the patch, the target system contains exactly what a correctly working import would have produced. Nobody looking at it later can tell that the import is still dropping records with that date format, and it will drop them again on the next run.

Every manual repair that is not logged as an incident converts a system defect into a personal habit. The human becomes an undocumented error-correction layer: reliable, invisible, and completely absent from any diagram of how the process works. It only becomes visible when that person is on holiday, which is roughly when the client notices instead.

Rule

If a person had to touch it to make it right, that is an incident, not a task.

What would have caught all three

Nothing exotic. Three habits, each cheap:

  • **An import without a count reconciliation is not an import.** Read the expected number from the source, compare it to the number written, and fail loudly on a mismatch. "Processed 12 of 24" is a sentence a job can produce for free, and it is the whole difference between silence and an alarm.
  • **Required fields belong on what carries meaning, not on what carries identity.** Email is required because it is the key. Everything downstream (personalization, segmentation, routing) depends on fields that are optional. That is a guarantee of broken output, not a risk of it. Either make them required at the point of capture, or give every template a fallback that reads acceptably when they are missing.
  • **Ownership of any shared artefact is part of the artefact.** A file only one person can change is a countdown, not a resource. The protected spreadsheet is the same failure mode as the import: it works today, it just cannot be maintained, and nothing will tell you the day that starts to matter.

The honest definition of working

I have written before that the interesting question about an automation is not whether it runs but whether anyone would notice if it stopped. This week sharpened it. Stopping is the easy case.

The harder case is the one that keeps running at reduced capacity, produces plausible output, gets patched by a competent person, and reports green the whole time. That system is not working. It is being carried.

Hung Mai
Hung Mai

Hung Mai is a Germany-based freelance consultant for Digital Operations & Transformation, working remotely with international B2B clients.

Let's build something real.

Let's talkResponse within 24h.