correctly routed responses to NATed packages via OpenVPN are rejected by client

singinWhale asked:

I have the following setup:

  • Dedicated server S with static public IP address on eth0 running OpenVPN as server on tun0 and performing NAT from tun0 to eth0
  • Gateway router G that is the OpenVPN client on tun0 with IP 10.8.0.16

the issue is that I can apparently reach the internet just fine and the responses get back just fine only that the Gateway router just flat out does not accept them back.

See the following example:

TCPdump while pinging 1.1.1.1 on G yields

# ping 1.1.1.1 -I tun0
PING 1.1.1.1 (1.1.1.1) from 10.8.0.16 tun0: 56(84) bytes of data.
♥
--- 1.1.1.1 ping statistics ---
7 packets transmitted, 0 received, 100% packet loss, time 6005ms

# tcpdump -i tun0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tun0, link-type RAW (Raw IP), capture size 262144 bytes
05:23:35.781349 IP 10.8.0.16 > 1dot1dot1dot1.cloudflare-dns.com: ICMP echo request, id 6803, seq 15, length 64
05:23:35.814670 IP 1dot1dot1dot1.cloudflare-dns.com > 10.8.0.16: ICMP echo reply, id 6803, seq 15, length 64
05:23:36.781452 IP 10.8.0.16 > 1dot1dot1dot1.cloudflare-dns.com: ICMP echo request, id 6803, seq 16, length 64
05:23:36.826077 IP 1dot1dot1dot1.cloudflare-dns.com > 10.8.0.16: ICMP echo reply, id 6803, seq 16, length 64
05:23:37.781352 IP 10.8.0.16 > 1dot1dot1dot1.cloudflare-dns.com: ICMP echo request, id 6803, seq 17, length 64
05:23:37.819026 IP 1dot1dot1dot1.cloudflare-dns.com > 10.8.0.16: ICMP echo reply, id 6803, seq 17, length 64
05:23:38.781321 IP 10.8.0.16 > 1dot1dot1dot1.cloudflare-dns.com: ICMP echo request, id 6803, seq 18, length 64
05:23:38.818058 IP 1dot1dot1dot1.cloudflare-dns.com > 10.8.0.16: ICMP echo reply, id 6803, seq 18, length 64
05:23:39.781330 IP 10.8.0.16 > 1dot1dot1dot1.cloudflare-dns.com: ICMP echo request, id 6803, seq 19, length 64
05:23:39.813765 IP 1dot1dot1dot1.cloudflare-dns.com > 10.8.0.16: ICMP echo reply, id 6803, seq 19, length 64
05:23:40.781333 IP 10.8.0.16 > 1dot1dot1dot1.cloudflare-dns.com: ICMP echo request, id 6803, seq 20, length 64
05:23:40.813766 IP 1dot1dot1dot1.cloudflare-dns.com > 10.8.0.16: ICMP echo reply, id 6803, seq 20, length 64

as you can see the ping reaches 1.1.1.1 just fine and a reply gets back but the output of pingstays silent

Iptables for G:

# iptables-save | grep -v -e DOCKER -e docker
# Generated by iptables-save v1.6.0 on Sat Aug  4 06:00:09 2018
*filter
:INPUT ACCEPT [29125:29954508]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [28322:3971375]
-A FORWARD -i eth0 -j ACCEPT
-A FORWARD -o eth0 -j ACCEPT
COMMIT
# Completed on Sat Aug  4 06:00:09 2018
# Generated by iptables-save v1.6.0 on Sat Aug  4 06:00:09 2018
*nat
:PREROUTING ACCEPT [3840:167985]
:INPUT ACCEPT [110:29389]
:OUTPUT ACCEPT [198:66581]
:POSTROUTING ACCEPT [198:66581]
-A POSTROUTING -s 192.168.0.0/16 -o eth1 -j MASQUERADE
COMMIT
# Completed on Sat Aug  4 06:00:09 2018

tcpdump on S also shows that the ICMP request is forwarded properly:

22:39:44.983332  In ethertype IPv4 (0x0800), length 100: 10.8.0.16 > 1.1.1.1: ICMP echo request, id 5383, seq 3, length 64
22:39:44.983348 Out 00:25:90:82:12:92 ethertype IPv4 (0x0800), length 100: 91.121.133.159 > 1.1.1.1: ICMP echo request, id 5383, seq 3, length 64
22:39:44.989167  In 00:00:5e:00:01:01 ethertype IPv4 (0x0800), length 100: 1.1.1.1 > 91.121.133.159: ICMP echo reply, id 5383, seq 3, length 64
22:39:44.989184 Out ethertype IPv4 (0x0800), length 100: 1.1.1.1 > 10.8.0.16: ICMP echo reply, id 5383, seq 3, length 64

The iptables on S look alright, too:

singinwhale# iptables-save | grep -v -e DOCKER -e docker
# Generated by iptables-save v1.4.21 on Fri Aug  3 23:54:56 2018
*raw
:PREROUTING ACCEPT [134484:12309809]
:OUTPUT ACCEPT [143517:21443183]
COMMIT
# Completed on Fri Aug  3 23:54:56 2018
# Generated by iptables-save v1.4.21 on Fri Aug  3 23:54:56 2018
*nat
:PREROUTING ACCEPT [11432:503568]
:INPUT ACCEPT [3303:157660]
:OUTPUT ACCEPT [2083:317164]
:POSTROUTING ACCEPT [2083:317164]
-A POSTROUTING -s 10.8.0.0/16 -o eth0 -j MASQUERADE
COMMIT
# Completed on Fri Aug  3 23:54:56 2018
# Generated by iptables-save v1.4.21 on Fri Aug  3 23:54:56 2018
*mangle
:PREROUTING ACCEPT [134484:12309809]
:INPUT ACCEPT [112991:10911009]
:FORWARD ACCEPT [12383:1034220]
:OUTPUT ACCEPT [143517:21443183]
:POSTROUTING ACCEPT [155157:22421380]
COMMIT
# Completed on Fri Aug  3 23:54:56 2018
# Generated by iptables-save v1.4.21 on Fri Aug  3 23:54:56 2018
*filter
:INPUT ACCEPT [77764:7068235]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [107553:10768547]
-A INPUT -p tcp -m tcp --dport 45454 -j ACCEPT
-A INPUT ! -s 127.0.0.1/32 -p tcp -m tcp --dport 56845 -j DROP
-A FORWARD -i tun0 -o eth0 -j ACCEPT
-A FORWARD -i eth0 -o tun0 -j ACCEPT
COMMIT
# Completed on Fri Aug  3 23:54:56 2018

The same is happening for curl http requests btw so this is not only for ICMP. What is going on here?


EDIT

apparently rp_netfilter is enabled (set to 1) on all interfaces and thus the packets arriving from 1.1.1.1 on tun0 are classified as martians. Thats a start but I never dealt with netfilter before but at least its a start.

Here is my routing table:

My routing table looks like this:

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         server.frederix 0.0.0.0         UG    100    0        0 eth1
10.8.0.0        *               255.255.0.0     U     0      0        0 tun0
link-local      *               255.255.0.0     U     1000   0        0 eth0
172.29.8.236    *               255.255.255.252 U     100    0        0 eth1
server.frederix server.frederix 255.255.255.255 UGH   100    0        0 eth1
192.168.0.0     *               255.255.0.0     U     0      0        0 eth0

And here is the output of some commands that @A.B. requested:

[email protected] ~ # ip -br link
lo               UNKNOWN        00:00:00:00:00:00 <LOOPBACK,UP,LOWER_UP>
bond0            DOWN           8e:f6:77:ae:d0:f2 <BROADCAST,MULTICAST,MASTER>
[email protected]       DOWN           0.0.0.0 <NOARP>
[email protected]        DOWN           0.0.0.0 <NOARP>
eth0             UNKNOWN        c2:c5:04:7a:0c:ba <BROADCAST,MULTICAST,UP,LOWER_UP>
eth1             UNKNOWN        ba:0b:a9:79:40:5b <BROADCAST,MULTICAST,UP,LOWER_UP>
wlan0            DOWN           00:08:22:4c:b3:fb <BROADCAST,MULTICAST>
tun0             UNKNOWN        <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP>
[email protected] ~ # ip -4 -br address
lo               UNKNOWN        127.0.0.1/8
eth0             UNKNOWN        192.168.0.1/16
eth1             UNKNOWN        172.29.8.237/30
tun0             UNKNOWN        10.8.0.16/16
[email protected] ~ # ip route
default via 172.29.8.238 dev eth1  proto static  metric 100
10.8.0.0/16 dev tun0  proto kernel  scope link  src 10.8.0.16
169.254.0.0/16 dev eth0  scope link  metric 1000
172.29.8.236/30 dev eth1  proto kernel  scope link  src 172.29.8.237  metric 100
172.30.3.254 via 172.29.8.238 dev eth1  proto dhcp  metric 100
192.168.0.0/16 dev eth0  proto kernel  scope link  src 192.168.0.1
[email protected] ~ # ip route get 1.1.1.1
1.1.1.1 via 172.29.8.238 dev eth1  src 172.29.8.237

My answer:


Your routing table shows that you aren’t routing any traffic to the VPN, other than the VPN’s own 10.8.0.0/16 network.

If you want the VPN to handle traffic, you need to set up the appropriate routes for that traffic. Otherwise, nothing will go through the VPN.

For instance, to route all traffic through the VPN, your server config should have something like:

push "redirect-gateway ipv6 def1 bypass-dhcp"

View the full question and any other answers on Server Fault.

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.