The port nobody touched

On a Tuesday afternoon I changed one firewall rule on the server that runs my paid visibility scan. The goal was narrow: allow outgoing mail on one port, keep the classic spam port closed. I measured afterwards, exactly the ports I had touched, and every one of them behaved as intended. For the next three hours the server could not reach the payment provider, the certificate authority, or any of the AI providers the scan depends on. That port was not in the change, so it was not in the check.

A narrow change, verified narrowly

The problem the day before had been simple to state. The product needed to send customers their report by mail, and every outgoing connection to a mail server ended in a timeout. The local firewall on the machine was innocent; it allowed all outgoing traffic. The block sat one layer up, in the hosting provider's own firewall, which closes outgoing mail ports by default.

The fix was to replace that default with my own policy: two rules, drop port 25, accept port 465. Assign it, apply it, save it. Then measure: 25 closed, 465 open. Port 587 also turned out closed, which I noted and did not think about, because I had not written a rule for it and it was not the one I needed.

That note was the whole story, and I read past it. A port closed without a rule saying so means the policy is not a list of exceptions. It is a list of permissions, and everything not on it is denied. Including 443.

What stopped working

The certificate manager's last successful check ran just before the change; its first timeout came a few minutes after. From that point the server could not open a secure connection to anything outside. The payment API timed out, so any purchase would have ended in a gateway error. The scan itself calls four AI providers over the same port, so any scan would have failed. Certificate renewal hung.

Nothing announced this. There were no customers in that window, which is the only reason the incident is a lesson and not an apology. It was found about three hours later by a manual check — a single request to the payment API from the server, which answers with an authentication error when the path is open and with nothing at all when it is not.

The repair took minutes once the model was right: drop 25, drop 587 explicitly, accept 465, then accept everything else for TCP and UDP. Measured again, this time wider: mail ports as intended, and the payment provider, the certificate authority, the code host, the package registry and all four AI providers reachable. The explicit drop for 587 matters. Under a catch-all accept it would otherwise open, and it was only closed before by accident of the same default that closed 443.

The same shape, two days earlier

Looking for similar cases turned up one I had already had. Two working sessions had edited the web server's configuration file within three minutes of each other. The second one wrote out its version without the first one's change. The effect: the legal pages of a different product went back behind a password prompt they had just been freed from. Each session had verified its own edit, and each edit was fine. The lost one was nobody's diff. It stayed unnoticed for two days, until a comparison of backups showed it.

Both cases have the same structure. A change is made, the change is verified, and the verification is honest. What breaks is something adjacent that shared a mechanism with the change — a default policy, a file — and that nobody thought of as touched.

Check the money path, not the diff

The rule I now run is not "verify more carefully". I verified carefully. The rule is to verify something that does not depend on what I changed.

• After any change to network, firewall, proxy or DNS on a production machine, one request from that machine to the payment provider. Authentication error means open. Silence means closed. It takes eight seconds and does not care which rule I edited.
• The same probe runs before every deploy, from the server and not from my laptop, because from my laptop it always succeeds.
• For configuration files that more than one session can touch: build from the live file, and compare its checksum against the starting state immediately before copying. If they differ, someone else was here.