Ipset with Iptables with large list of IP ranges (CIDR)

CvRChameleon asked:

I have read some answers here about blocking IP address ranges, and have already used iptables for this purpose before. It is suggested to use ipset in combination with iptables.

I have only installed ipset but have not configured it yet.

I found this site ip2location.com to generate a list of IPs to ban by country. I selected the 5 countries that target our sites regularly, but the list is huge, 256000 rows.

  • Would this massive list slow down my server when using ipset (before attempting this using only IPtables I questioned such a large file might slow performance.).
  • If that is the case, what is the way to do this? At the moment I use fail2ban but I do not think the configuration for nginx is correctly setup (I assume regex). In any case, I need a more robust way.
  • Finally, I do not claim to understand CIDR enough to make this list smaller (aggregate similar IP ranges if possible).

For instance, there are several /21 entries :

An online tool shows this resolves to : to

I do not think there is any easy way to make the entries less, so any advice regarding implementation and performance issues please.

My answer:

There is a command line utilty named aggregate. It takes a list of CIDR netblocks and aggregates consecutive blocks into the corresponding larger block. It also removes redundant netblocks.

For example:

$ aggregate -q << EOF

Feed it a text file containing only your CIDR blocks and it will attempt to aggregate them, reducing the size of the list.

From the man page:

       Takes  a list of prefixes in conventional format on stdin, and performs
       two optimisations to attempt to reduce the length of the prefix list.

       The first optimisation is to remove any supplied prefixes which are su‐
       perfluous because they are already included in another supplied prefix.
       For example, would be removed if  was  also

       The  second  optimisation identifies adjacent prefixes that can be com‐
       bined under a single, shorter-length prefix. For example,
       and can be combined into the single prefix

aggregate is packaged in most major Linux distributions, including Ubuntu.

(Note that I pulled a list from that web site and tried to aggregate them and nothing happened, so they may already be aggregated. You can certainly use more than one ipset, which is probably the best thing to do here.)

View the full question and any other answers on Server Fault.

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.