Route internet traffic through one nic and local traffic through a second nic

Matt Schramm asked:

I am trying to route traffic through a specific interface based on its destination.

OS is CentOS 7.

I am setting up a mini MPI farm where I have 2 slave workers and one master.
The master and slaves use the built-in nic (em1) to communicate locally through a switch (this way all of the workers have “local” ips 192.168.1.*).

I then have a secondary nic (p2p1) that is connected to my router for external internet traffic.

As things are currently setup, I am able to ssh into each worker without problem but I cannot communicate with the internet (ping 8.8.8.8 states that it cannot access google’s server).

From what I have searched for, I haven’t been able to find a solution for this specific problem but I feel it is a simple fix…

typing route yields

Kernel IP routing table
Destination     Gateway      Genmask       Flags Metric Ref Use Iface
default         Linksys01356 0.0.0.0       UG    100    0     0 em1
default         gateway      0.0.0.0       UG    101    0     0 p2p1
10.11.230.0     0.0.0.0      255.255.255.0 U     101    0     0 p2p1
192.168.1.0     0.0.0.0      255.255.255.0 U     100    0     0 em1
192.168.122.0   0.0.0.0      255.255.255.0 U     0      0     0 virbr0

and typying iptables –list gives

Chain INPUT (policy ACCEPT)
target     prot opt source            destination
ACCEPT     udp  --  anywhere          anywhere            udp dpt:domain
ACCEPT     tcp  --  anywhere          anywhere            tcp dpt:domain
ACCEPT     udp  --  anywhere          anywhere            udp dpt:bootps
ACCEPT     tcp  --  anywhere          anywhere            tcp dpt:bootps

Chain FORWARD (policy ACCEPT)
target     prot opt source            destination
ACCEPT     all  --  anywhere          192.168.122.0/24    ctstate RELATED,ESTABLISHED
ACCEPT     all  --  192.168.122.0/24  anywhere
ACCEPT     all  --  anywhere          anywhere
REJECT     all  --  anywhere          anywhere            reject-with icmp-port-unreachable
REJECT     all  --  anywhere          anywhere            reject-with icmp-port-unreachable

Chain OUTPUT (policy ACCEPT)
target     prot opt source            destination
ACCEPT     udp  --  anywhere          anywhere            udp dpt:bootpc

Ideally I would like all the MPI computers (master and slaves) to communicate using em1 and the master pull things from outside the local network on p2p1.

If it helps, the master IP is 10.11.230.43 for outside access and 192.168.1.143 on the local network, while one slave has the ip address 192.168.1.118.

If I am missing any other info please let me know.

Thank you

My answer:


You have set a “default” route on both NICs, but “default” means, among other things, the number one. It is where traffic goes that has no other route. If you accidentally set more than one of them, then the lower metric decides which is used. In this case, that means your Internet bound traffic is going to “Linksys01356” on interface em1.

Because this isn’t what you want, you should reconfigure the em1 interface to remove the gateway address. The only gateway that should be defined is the one which routes your traffic to the Internet.


View the full question and any other answers on Server Fault.

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.