iptables configuration & logging of dropped packages due to missing ports when using masquerading

Phylu asked:

I am setting up iptables for natting of multiple hosts in a server cluster. There will be connection bursts from multiple servers usually to a single system. The natting is needed so that the receivers can whitelist the origin of the requests. The configuration is very basic:

echo 1 > /proc/sys/net/ipv4/ip_forward
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

Due to the burst of messages, I am worried that the number of open connections (or connections in a very short time) to a single destination will provoke the problem that no more ports on the nat host are available to open connections to the single target system.

  1. Is there a way to log messages for when such a case happens?

  2. If something like this happens, is there any way to recognise this when setting up logging for the post routing table like this?

iptables -t nat -I POSTROUTING -j LOG
Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target     prot opt in     out     source               destination
421K   25M LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            LOG flags 0 level 4
603K   36M MASQUERADE  all  --  *      eth0    0.0.0.0/0            0.0.0.0/0
  1. Can dropping packages be avoided by adding multiple static IP addresses and using SNAT instead of masquerading as follows?
iptables -t nat -A POSTROUTING -o eth0 -j SNAT --to 1.2.3.4,1.2.3.5,9.8.7.6

Any help is highly appreciated.

Phylu

My answer:


You’ve got two possible issues here:

  1. Will the connection initiator run out of source ports?
  2. Will the NAT device fill its connection tracking table?

In both cases it’s important to note what distinguishes one TCP connection from another, all of:

  • The source IP address
  • The source port
  • The destination IP address
  • The destination port

When a Linux machine initiates a TCP connection, it chooses a local port from the port range given in the sysctl net.ipv4.ip_local_port_range (which also applies to IPv6 TCP connections). By default this is 32000 61999, meaning the machine can establish 28232 simultaneous TCP connections to any particular destination host and port combination. It’s very unlikely you’ll have to raise this, but if necessary you can set it to its limit of 1024 65535, giving 64512 possible simultaneous connections.

[[email protected] ~]# sysctl net.ipv4.ip_local_port_range
net.ipv4.ip_local_port_range = 32768    60999
[[email protected] ~]# sysctl -w net.ipv4.ip_local_port_range="1024 65535"
net.ipv4.ip_local_port_range = 1024 65535

In the case of the NAT device, it keeps a connection tracking table that keeps track of each connection’s original addresses and ports and the NATted addresses and ports. The default size of this table varies depending on how much RAM the system has. Check the sysctl net.netfilter.nf_conntrack_max to find out what it is set to on your system.

[[email protected] ~]# sysctl net.netfilter.nf_conntrack_max
net.netfilter.nf_conntrack_max = 262144

If it is necessary to raise it, you should also raise the conntrack table hashsize, which is a hash table that speeds up lookups into the conntrack table. The rule of thumb here is that it should be a power of two, 1/4 of the conntrack table size set above. This is set as a module option, so you will need to put it in a file e.g. /etc/modprobe.d/nf_conntrack.conf. For example, if you set net.netfilter.nf_conntrack_max to 524288, you would also set:

options nf_conntrack expect_hashsize=131072 hashsize=131072

You can check the current size of the conntrack table in the read-only sysctl net.netfilter.nf_conntrack_count:

[[email protected] ~]# sysctl net.netfilter.nf_conntrack_count
net.netfilter.nf_conntrack_count = 3635

Note that if you are using MASQUERADE/SNAT then you also need to worry about the NAT device running out of source ports when making connections to the destination. NAT is best avoided if at all possible, not just here but everywhere (and it seems unlikely that you really need NAT here; you’re trading a small inconvenience of firewall rules for a huge problem). You should very strongly consider getting rid of NAT entirely and using normal routing.


View the full question and any other answers on Server Fault.

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.