Our NAT gateway triggered a port scan alert in our own SOC

Some of our servers have no public address. Message brokers and storage nodes have no business being reachable from the internet, but they do need to reach out, for package updates and DNS resolution. They go through a NAT gateway: a management server that does have a public address and routes their traffic.

One morning, after that gateway rebooted, the private nodes had lost internet access. And our SOC was showing a slow port scan alert originating from our own internal network.

Those two facts are the same fact.

Docker takes ownership of the FORWARD chain

The first point catches the most people, and it is not specific to our setup.

When Docker starts, it sets the FORWARD chain policy to DROP and inserts a jump to its own chain, DOCKER-USER, at the top. Any routing rule written directly into FORWARD is therefore either evaluated too late or never reached. The rule looks present, iptables -L shows it, and it does nothing.

DOCKER-USER exists for exactly this: it is the chain intended for custom forwarding rules, and it is evaluated before Docker’s own. That is where a gateway’s allowances belong:

- name: "FORWARD — allow SOC network through NAT (DOCKER-USER chain)"
  ansible.builtin.iptables:
    chain: DOCKER-USER
    source: 10.0.1.0/24
    jump: ACCEPT

Sharing a host with Docker, on a machine that does anything besides run containers, requires knowing that it modifies the host firewall without telling you. That is not a Docker defect, it follows from its networking model. But it is written nowhere at the moment you are configuring a gateway.

One layer below: the hypervisor filters too

Added after publication, because a layer was missing and anyone reproducing this setup would hit it before ever reaching the iptables rules.

At Hetzner, a private network is not a plain switch that carries whatever you hand it. Destination address filtering happens at the hypervisor, on the virtual switch. A packet sent by a private node to a public address is therefore dropped before it reaches the gateway’s kernel. No iptables rule will ever see it, and the diagnosis naturally heads the wrong way: you inspect the gateway’s counters, they sit at zero, and you conclude the traffic is not leaving.

The private network’s default route has to be declared to the hypervisor explicitly:

resource "hcloud_network_route" "soc_default" {
  network_id  = hcloud_network.soc.id
  destination = "0.0.0.0/0"
  gateway     = var.nat_gateway_soc_ip
}

Three layers therefore have to be correct for a private node to reach the internet, and each fails silently: the hypervisor route, the forwarding allowances in DOCKER-USER, and the outbound MASQUERADE. They do not belong to the same system, they do not log to the same place, and two of them produce no trace at all.

Rule order, and the asymmetry of return traffic

The second point is subtler, and it produces a particularly misleading failure.

Allowing forwarding from 10.0.1.0/24 looks sufficient. It is not, because a TCP connection is not symmetric from the firewall’s point of view. The outbound packet does carry a private source address. The return packet carries the remote server’s address, a package mirror for instance. A rule that only allows source 10.0.1.0/24 does not recognise it, and the chain’s DROP policy discards it.

The symptom is disorienting: the connection goes out, is never refused, and hangs. The SYN leaves, the SYN-ACK is destroyed on arrival. A ping may work, DNS resolution may work, and every download stalls indefinitely.

Hence the ordering enforced in the role, with established traffic allowed at position 1, ahead of any source-based rule:

- name: "FORWARD — allow established/related return traffic (DOCKER-USER chain)"
  ansible.builtin.iptables:
    chain: DOCKER-USER
    ctstate: [ESTABLISHED, RELATED]
    jump: ACCEPT
    action: insert
    rule_num: 1

The interesting part: the detection was right

Here is the core of this field note.

After the reboot those rules had not been restored. Every outbound attempt from a private node was therefore blocked, and every block logged. The result, as seen by the correlation engine: a single internal source address producing dozens of refused connections to varied destinations, steadily, spread over time.

That is precisely the signature of a slow port scan. The frequency rule did its job, escalated, and raised the alert.

This was not a detection defect. The rule was correct about what it observed. It was missing one thing only: that the source was one of our own nodes, failing to reach a package mirror.

The temptation in that situation is to disable the rule. That is the wrong answer: you remove detection of genuinely suspicious behaviour in order to silence one special case. The right answer is to supply the missing context, by scoping the suppression to the internal ranges concerned and keeping it at a level where it stays queryable:

<rule id="100043" level="3">
  <if_sid>100030</if_sid>
  <srcip>10.0.1.0/24</srcip>
  <description>UFW blocked NAT forward from SOC node — internal routing miss, not a scan</description>
</rule>

The event still exists and remains searchable. It simply stops feeding the frequency counter that led to the scan alert. An external source producing the same pattern still escalates.

What we take from it

1. Docker modifies the host firewall. On a machine that also routes, forwarding rules go in DOCKER-USER, never in FORWARD.
2. A source-based rule does not cover the return path. Allow ESTABLISHED,RELATED first, or connections hang instead of being refused, which takes far longer to diagnose.
3. Check what survives a reboot. An active firewall rule is not a persistent firewall rule, and the gap only shows at the next boot.
4. A false positive is not always a detection error. This one correctly flagged abnormal behaviour whose cause was our own infrastructure. The SOC observes your estate, not only your attackers, and it will tell you true things about yourself.

The fourth point is the one we keep. An alert pointing at an internal failure is not noise to suppress, it is information to read.