fail2ban locked us out of the whole estate, and it was right

Our servers are hardened against the CIS benchmark. Two of the controls applied are entirely unremarkable, and turn up on more or less every serious estate:

MaxAuthTries 4          # sshd: authentication attempts per connection
maxretry  = 5           # fail2ban: failed connections before a ban
findtime  = 600         # ... counted over 10 minutes
bantime   = 3600        # ... banned for an hour

There is nothing wrong with those values. They nevertheless locked us out, across the entire estate simultaneously, with no wrong password and no invalid key.

What an SSH agent actually does

The starting point is client behaviour, not server behaviour, which is what makes the diagnosis counter-intuitive.

When an SSH agent holds several keys, the client does not know which one the server will accept. So it offers them one at a time, in the order the agent holds them, until one is recognised.

The server, though, does not see a negotiation. It sees a series of authentication attempts. Each one consumes an allowance from MaxAuthTries. With that value at 4 and six keys loaded in the agent, the first four keys offered exhaust the quota, and the server closes the connection before the correct key is ever presented.

The message returned to the client is Too many authentication failures. It is accurate and thoroughly misleading: there is no authentication failure in the sense anyone reads it. The right key is present, sitting in the agent, and was simply never offered.

How that becomes an estate-wide ban

A closed connection counts, as far as fail2ban is concerned, as a failed one. Five within ten minutes ban the source address for an hour.

The scaling factor is parallel execution. An Ansible playbook run with several forks does not open one connection, it opens as many as it targets machines, simultaneously, from the same source address. Each server then applies its own counter, independently of the others. They are hardened identically, they react identically, and they all ban the same address at the same moment.

The result has an unusual shape: the failure is not gradual, it is total and instantaneous. You lose the whole estate at once, including the bastion you would have used to recover.

And this needs saying plainly: fail2ban did exactly what it is installed to do. A single address producing dozens of failed authentication attempts across dozens of machines within seconds is the signature of a distributed brute-force attempt. The control worked. It simply identified the right pattern in the wrong actor.

The reaction not to have

The immediate temptation is to raise MaxAuthTries, or lengthen findtime, or allowlist your own address in fail2ban.

The first two genuinely weaken the protection: they increase the number of attempts an attacker gets per connection, or the window across which they can spread them. The third disarms the mechanism for the most sensitive address in the estate, the one everything is administered from.

Those are three ways of treating the symptom, and they share an assumption: that the server is wrong. The server is not wrong.

The fix is on the client

The defect is that the client offers irrelevant keys. The correction is to forbid it from doing so:

IdentitiesOnly=yes

This tells SSH to use only the explicitly named key and to ignore everything else the agent offers. A single authentication attempt reaches the server, it is the correct one, and MaxAuthTries is never approached.

One detail that matters in practice: the option has to be set on every hop. A connection bouncing through a bastion opens two successive SSH sessions, and the second starts afresh with the full agent unless it is constrained too:

ProxyCommand ssh -i ~/.ssh/key -o IdentitiesOnly=yes -W %h:%p user@bastion

Fixing only the final connection leaves the bastion to be banned, which produces exactly the same outage with one fewer component available to diagnose it.

What we take from it

1. A loaded SSH agent is a generator of authentication failures. The more keys it holds, the faster the server’s quota is reached. Workstation convenience becomes a production problem.
2. Too many authentication failures does not mean the key is wrong. It means the server stopped listening before it saw it.
3. Parallelism turns an annoyance into a total outage. A per-node control applied uniformly produces a uniform ban, bastion included.
4. IdentitiesOnly=yes on every hop, not only on the final one.
5. A security control that blocks you is not necessarily misconfigured. This one correctly recognised an attack pattern. The defect was in how we connected, not in its threshold.

The fifth point is the expensive one to learn, because the instinct under pressure, when you are locked out, is to loosen the control that locked you out.