Our failover had never worked, and every dashboard was green

We were patching kernels. Boring, necessary work: our unattended upgrades install kernel security updates but never reboot, so the fleet reported itself fully patched while running an image months old. Reboot everything, in waves, carefully.

Somewhere in the middle of that, we found out our high-availability pair had never once worked.

Two layers, and only one of them was real

A floating IP on a cloud provider is not like a virtual IP on your own switch. There are two layers, and both have to happen.

The first is local: keepalived adds the address to the interface of whichever node holds the MASTER role. The second is at the provider: an API call tells their network to route that address to that machine. Without the second, the new MASTER has the address configured and receives nothing, because the provider is still sending packets to the old one.

Our hook did the first. It had never, not once, done the second.

Found by accident, which is the point

Rebooting the backup node produced a three-second window where it briefly claimed MASTER before hearing the primary’s advertisement. That blip ran the notification hook, and for the first time we were watching when it ran.

The log said:

Becoming MASTER, assigning floating IP to <node>
Resolved FIP ID: <id>

And then nothing. No assignment line. No error. The script simply stopped.

It resolved its own server by name:

GET /v1/servers?name=<short-hostname>

Our provider prefixes server names with the environment. The short hostname matched nothing, the API cheerfully returned HTTP 200 with an empty list, the inline parser raised, and set -euo pipefail killed the script one line before the call that mattered.

The log file showed the same truncated pair for every MASTER transition on both nodes going back two months. Six transitions. Silent every time.

Why nothing caught it

This is the part worth sitting with. We had monitoring. The pair looked healthy from every angle we had:

• keepalived was running on both nodes.
• Exactly one node held the address.
• ip addr on the MASTER showed the floating IP, correctly.
• No alert had ever fired.

Every one of those observations was true. None of them tested the thing that was broken. The local binding is what you naturally check, and the local binding always worked. The failure lived entirely in a layer nothing looked at.

And a failover would have exposed it in seconds. We had never done one. Not because anyone decided not to, but because nothing ever forced the question.

Fixing it nearly made it worse

The obvious fix is to stop resolving by name. The provider’s metadata service hands a machine its own id with no token and no naming convention to drift from:

curl -s http://169.254.169.254/<provider>/v1/metadata/instance-id

That part was easy. The problem was what the fix would have done on its own.

Remember the three-second blip. A backup node that boots next to a live primary waits about three advertisement intervals, hears nothing because the network stack is still coming up, and promotes itself. It demotes again a moment later. Harmless, historically, because the hook was broken.

With a working hook, that blip hands the floating IP to a node that is about to step down. And nothing hands it back, because the real MASTER never changes state and therefore never re-runs the hook. Our service address would have ended up parked on the standby, permanently, and the “fix” would have caused a worse outage than the bug.

So the hook now waits a few seconds and re-checks that it still holds the address before touching the provider API. Separately, we delay VRRP startup so the blip mostly stops happening. The guard stays anyway: it covers real but short-lived promotions that a startup delay cannot.

Then we found two more of the same shape

Same day, same class of problem, different mechanisms.

A check that could not fail. Our post-reboot verification asked whether WireGuard had come back with a key:

wg show wg0 public-key   # exit 0, prints "(none)"

The check tested for exit code zero and non-empty output. (none) satisfies both. It reported the VPN healthy on a host where the tunnel had no private key and every peer was at zero handshakes. The check had never been able to return a negative verdict about anything.

Rules that were never loaded. Two detection rules suppressing a known false positive sat in a rule group parsed before the group defining their parent rule. The engine resolves rule references in file order, could not find the parent, and silently ignored both. They were present in the file. They read correctly in review. They had never fired, and the false positives they exist to prevent had been firing for weeks.

The engine said so. Every single start-up:

Signature ID '<parent>' was not found ... in the 'if_sid' option of rule '<child>'
Empty 'if_sid' value. Rule '<child>' will be ignored.

Nothing in our deployment process required anyone to read that output. We wrote the file, restarted the service, and moved on.

Three ways to produce false assurance

Laid side by side, the pattern is clearer than any of the three individually:

	How it failed
The failover hook	Mute. Died without logging anything.
The VPN check	Lying. Structurally incapable of a negative verdict.
The detection rules	Ignored. Said exactly what was wrong, into a void.

Different mechanisms, identical outcome: a system that reports health it does not have. And a control that cannot fail is worse than no control, because it spends the attention you would otherwise have pointed at the problem.

What we actually changed

Not just the three bugs. The conditions that let them live.

Exercise the failure, on a schedule. We now have a playbook that performs a real failover and verifies it against the provider API, not the local interface, because the local interface is exactly what lied. It refuses to run against an already-degraded pair, drives from the node that stays up, and restarts keepalived in an always block so no failure can leave the primary without it.

We run it monthly, by hand, and deliberately did not put it on a timer. An unattended failover that goes wrong at three in the morning on infrastructure nobody is watching is a worse problem than the one it guards against. It takes about a minute. Continuous detection is a different mechanism, and that one does run on its own.

Make silence impossible. Both hooks now log a fatal line when they cannot do their job, and we added alert rules that fire on those lines. A script failing loudly into a file nobody reads is the same problem one layer further out.

Gate on the validator. Rule deployments now run the engine’s own validation before restarting anything, and roll back on rejection. It justified itself immediately by catching an invalid rule of ours on the first attempt. It is also how we found the rules that had never loaded: the warnings had been there all along, in output we had never been forced to look at.

Test checks against failure, not health. Every one of these was validated against a working system, where it returned the expected answer, which proves nothing. The question to ask of any check is not “does it pass when things are fine” but “have I ever seen this fail”.

The uncomfortable part

We found all of this while doing something else. Kernel patching, on a fleet that had been running fine.

We did not find it because our monitoring was good. Our monitoring was, on this specific point, worse than useless: it was actively reassuring. We found it because a reboot happened to create a three-second window where a broken thing became visible, and because somebody was reading the output at that moment.

That is not a repeatable detection strategy. Which is why the fixes we care about are not the three patches. They are the exercise, the validation gate, and the habit of asking, of every green indicator, what it would take for this to ever turn red.