Why NDP doesn't work on virtual ethernet interface for packets arriving from outside?

Malgond asked:

I am moving some of my services (SMTP, IMAP, etc.) into systemd-nspawn containers for better isolation, so I need to make some connections coming from outside redirected to containers. I got only 16 IPv6 addresses on this virtual machine from the provider, so I resolved to use iptables to redirect some ports into containers. For IPv4 it works ok, but I can’t get it to work for IPv6. I have nailed the problem down to loosing neighbour table entry for virtual ethernet’s peer MAC and I don’t know how to proceed from there. To simplify the problem I have made a test container with only socat in it, doing echo on port 5555. I have given it a virtual ethernet interface between the host and the container with a private IPv4 address and an ULA IPv6 one.

[email protected]:~# machinectl status listenbox
listenbox(8c403969837546a7ad6a342da35fdf49)
           Since: Thu 2020-09-17 10:42:50 CEST; 4s ago
          Leader: 7319 ((sd-stubinit))
         Service: systemd-nspawn; class container
            Root: /chroot/listenbox
           Iface: ve-listenbox
         Address: 10.30.0.1
                  fdc9:c654:8207:17cb::1%91
                  fe80::ecaa:95ff:fe9a:1cff%91
            Unit: [email protected]
                  ├─payload
                  │ ├─7319 (sd-stubinit)
                  │ └─7324 socat -ly -d -d -d TCP6-LISTEN:5555,fork PIPE
                  └─supervisor
                    └─7317 /usr/bin/systemd-nspawn --quiet --keep-unit --boot --link-journal=try-guest --network-veth -U --set

Sep 17 10:42:50 bonclok.hasiok.net systemd[1]: Started Container listenbox.
Sep 17 10:42:50 bonclok.hasiok.net socat[7324]: I socat by Gerhard Rieger and contributors - see www.dest-unreach.org
Sep 17 10:42:50 bonclok.hasiok.net socat[7324]: I This product includes software developed by the OpenSSL Project for use in t
Sep 17 10:42:50 bonclok.hasiok.net socat[7324]: I This product includes software written by Tim Hudson ([email protected])
Sep 17 10:42:50 bonclok.hasiok.net socat[7324]: I setting option "fork" to 1
Sep 17 10:42:50 bonclok.hasiok.net socat[7324]: I socket(10, 1, 6) -> 6
Sep 17 10:42:50 bonclok.hasiok.net socat[7324]: I starting accept loop
Sep 17 10:42:50 bonclok.hasiok.net socat[7324]: N listening on AF=10 [0000:0000:0000:0000:0000:0000:0000:0000]:5555

Here’s the configuration on both ends of the virtual interface

[email protected]:~# ip a sh dev ve-listenbox # on host
91: [email protected]: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 9e:81:a0:6a:93:be brd ff:ff:ff:ff:ff:ff link-netnsid 2
    inet 10.30.0.2/24 brd 10.30.0.255 scope global ve-listenbox
       valid_lft forever preferred_lft forever
    inet6 fdc9:c654:8207:17cb::2/64 scope global
       valid_lft forever preferred_lft forever
[email protected]:~# nsenter -a -t 7324 ip a sh dev host0 # in container
2: [email protected]: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether ee:aa:95:9a:1c:ff brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.30.0.1/24 scope global host0
       valid_lft forever preferred_lft forever
    inet6 fdc9:c654:8207:17cb::1/64 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::ecaa:95ff:fe9a:1cff/64 scope link

And route tables:

[email protected]:~# ip -6 r|grep ve-listenbox # on host
fdc9:c654:8207:17cb::/64 dev ve-listenbox proto kernel metric 256 pref medium
[email protected]:~# nsenter -a -t 7324 ip -6 r # in container
::1 dev lo proto kernel metric 256 pref medium
fdc9:c654:8207:17cb::/64 dev host0 proto kernel metric 256 pref medium
fe80::/64 dev host0 proto kernel metric 256 pref medium
default via fdc9:c654:8207:17cb::2 dev host0 metric 1024 pref medium

I have redirected the port 555 via iptables

[email protected]:~# ip6tables -n -L -t nat
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
DNAT       tcp     !fdc9:c654:8207:17cb::1  ::/0                 tcp dpt:5555 to:[fdc9:c654:8207:17cb::1]:5555

Now when I connect to fdc9:c654:8207:17cb::1 from the host it works. But if I connect from outside it only works if I recently connected from the host and there is a current neighbor table entry. I have run tcpdump on the ve-listenbox interface, and when I connect from the host:

[email protected]:~# socat - TCP:[fdc9:c654:8207:17cb::1]:5555

I see ICMPv6 packets going back and forth and TCP handshake:

[email protected]:~# tcpdump -t -n -vvv -i ve-listenbox
tcpdump: listening on ve-listenbox, link-type EN10MB (Ethernet), capture size 262144 bytes
IP6 (hlim 255, next-header ICMPv6 (58) payload length: 16) fe80::ecaa:95ff:fe9a:1cff > ff02::2: [icmp6 sum ok] ICMP6, router solicitation, length 16
          source link-address option (1), length 8 (1): ee:aa:95:9a:1c:ff
            0x0000:  eeaa 959a 1cff
IP6 (hlim 255, next-header ICMPv6 (58) payload length: 32) fdc9:c654:8207:17cb::2 > ff02::1:ff00:1: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has fdc9:c654:8207:17cb::1
          source link-address option (1), length 8 (1): 9e:81:a0:6a:93:be
            0x0000:  9e81 a06a 93be
IP6 (hlim 255, next-header ICMPv6 (58) payload length: 32) fdc9:c654:8207:17cb::1 > fdc9:c654:8207:17cb::2: [icmp6 sum ok] ICMP6, neighbor advertisement, length 32, tgt is fdc9:c654:8207:17cb::1, Flags [solicited, override]
          destination link-address option (2), length 8 (1): ee:aa:95:9a:1c:ff
            0x0000:  eeaa 959a 1cff
IP6 (flowlabel 0x2e8e4, hlim 64, next-header TCP (6) payload length: 40) fdc9:c654:8207:17cb::2.38176 > fdc9:c654:8207:17cb::1.5555: Flags [S], cksum 0xbc13 (incorrect -> 0xe27d), seq 3531392912, win 64800, options [mss 1440,sackOK,TS val 2219762528 ecr 0,nop,wscale 7], length 0
IP6 (flowlabel 0x78e11, hlim 64, next-header TCP (6) payload length: 40) fdc9:c654:8207:17cb::1.5555 > fdc9:c654:8207:17cb::2.38176: Flags [S.], cksum 0xbc13 (incorrect -> 0x151d), seq 3651211421, ack 3531392913, win 64260, options [mss 1440,sackOK,TS val 1555204218 ecr 2219762528,nop,wscale 7], length 0
IP6 (flowlabel 0x2e8e4, hlim 64, next-header TCP (6) payload length: 32) fdc9:c654:8207:17cb::2.38176 > fdc9:c654:8207:17cb::1.5555: Flags [.], cksum 0xbc0b (incorrect -> 0x3cdf), seq 1, ack 1, win 507, options [nop,nop,TS val 2219762528 ecr 1555204218], length 0

and I see the neighbor table entry.

[email protected]:~# ip ne sh dev ve-listenbox
fdc9:c654:8207:17cb::1 lladdr ee:aa:95:9a:1c:ff REACHABLE

Then, after I close the local connection and connect from outside:

[email protected]:~$ telnet -6 2a03:b0c0:3:f0::36:7000 5555
Trying 2a03:b0c0:3:f0::36:7000...
Connected to 2a03:b0c0:3:f0::36:7000.

I see original packets on eth0:

[email protected]:~# tcpdump -n -t -vvv -i eth0 tcp port 5555
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
IP6 (flowlabel 0xcad87, hlim 51, next-header TCP (6) payload length: 40) 2a02:2780:9000:50:fb:ff:fe02:2.43314 > 2a03:b0c0:3:f0::36:7000.5555: Flags [S], cksum 0x7dcc (correct), seq 3406297189, win 64800, options [mss 1440,sackOK,TS val 2466613775 ecr 0,nop,wscale 7], length 0
IP6 (flowlabel 0xcc342, hlim 63, next-header TCP (6) payload length: 40) 2a03:b0c0:3:f0::36:7000.5555 > 2a02:2780:9000:50:fb:ff:fe02:2.43314: Flags [S.], cksum 0x2ded (incorrect -> 0x701a), seq 1345388437, ack 3406297190, win 64260, options [mss 1440,sackOK,TS val 290303657 ecr 2466613775,nop,wscale 7], length 0
IP6 (flowlabel 0xcad87, hlim 51, next-header TCP (6) payload length: 32) 2a02:2780:9000:50:fb:ff:fe02:2.43314 > 2a03:b0c0:3:f0::36:7000.5555: Flags [.], cksum 0x97c6 (correct), seq 1, ack 1, win 507, options [nop,nop,TS val 2466613797 ecr 290303657], length 0

and translated packets on VE interface.

IP6 (flowlabel 0xcad87, hlim 50, next-header TCP (6) payload length: 40) 2a02:2780:9000:50:fb:ff:fe02:2.43314 > fdc9:c654:8207:17cb::1.5555: Flags [S], cksum 0x6bc7 (correct), seq 3406297189, win 64800, options [mss 1440,sackOK,TS val 2466613775 ecr 0,nop,wscale 7], length 0
IP6 (flowlabel 0xcc342, hlim 64, next-header TCP (6) payload length: 40) fdc9:c654:8207:17cb::1.5555 > 2a02:2780:9000:50:fb:ff:fe02:2.43314: Flags [S.], cksum 0x3ff2 (incorrect -> 0x5e15), seq 1345388437, ack 3406297190, win 64260, options [mss 1440,sackOK,TS val 290303657 ecr 2466613775,nop,wscale 7], length 0
IP6 (flowlabel 0xcad87, hlim 50, next-header TCP (6) payload length: 32) 2a02:2780:9000:50:fb:ff:fe02:2.43314 > fdc9:c654:8207:17cb::1.5555: Flags [.], cksum 0x85c1 (correct), seq 1, ack 1, win 507, options [nop,nop,TS val 2466613797 ecr 290303657], length 0

But after some time the neighbor table entry entry expires

[email protected]:~# ip ne sh dev ve-listenbox
fdc9:c654:8207:17cb::1  INCOMPLETE

and there’s no more traffic on ve-listenbox. When I try to reconnect, I see SYN packets on eth0 being responded by ICMP destination unreachable

IP6 (flowlabel 0x09321, hlim 51, next-header TCP (6) payload length: 40) 2a02:2780:9000:50:fb:ff:fe02:2.43532 > 2a03:b0c0:3:f0::36:7000.5555: Flags [S], cksum 0x53f1 (correct), seq 3900363577, win 64800, options [mss 1440,sackOK,TS val 2468133554 ecr 0,nop,wscale 7], length 0
IP6 (flowlabel 0xa154e, hlim 51, next-header TCP (6) payload length: 40) 2a02:2780:9000:50:fb:ff:fe02:2.43532 > 2a03:b0c0:3:f0::36:7000.5555: Flags [S], cksum 0x4feb (correct), seq 3900363577, win 64800, options [mss 1440,sackOK,TS val 2468134584 ecr 0,nop,wscale 7], length 0
IP6 (flowlabel 0x7c56a, hlim 51, next-header TCP (6) payload length: 40) 2a02:2780:9000:50:fb:ff:fe02:2.43532 > 2a03:b0c0:3:f0::36:7000.5555: Flags [S], cksum 0x480b (correct), seq 3900363577, win 64800, options [mss 1440,sackOK,TS val 2468136600 ecr 0,nop,wscale 7], length 0
IP6 (flowlabel 0xd1899, hlim 64, next-header ICMPv6 (58) payload length: 88) 2a03:b0c0:3:f0::36:7000 > 2a02:2780:9000:50:fb:ff:fe02:2: [icmp6 sum ok] ICMP6, destination unreachable, unreachable address 2a03:b0c0:3:f0::36:7000
IP6 (flowlabel 0xd1899, hlim 64, next-header ICMPv6 (58) payload length: 88) 2a03:b0c0:3:f0::36:7000 > 2a02:2780:9000:50:fb:ff:fe02:2: [icmp6 sum ok] ICMP6, destination unreachable, unreachable address 2a03:b0c0:3:f0::36:7000
IP6 (flowlabel 0xd1899, hlim 64, next-header ICMPv6 (58) payload length: 88) 2a03:b0c0:3:f0::36:7000 > 2a02:2780:9000:50:fb:ff:fe02:2: [icmp6 sum ok] ICMP6, destination unreachable, unreachable address 2a03:b0c0:3:f0::36:7000

but nothing on virtual ethernet – NDP query is not sent on VE interface. The iptables rule gets its count increased – so it is definitely NATing the new incoming connection to port 5555, but since there’s no neighbor table entry it does not know how to send the translated packet.

Why it does not try to re-query when there’s a packet coming from outside but re-queries happily when the connection is from the host? I don’t get it.

It works when I add a static entry to the neighbor table, but I don’t like this.

What am I doing wrong?

My answer:


It sounds like you need an NDP proxy. I use ndppd for this purpose. Its configuration is pretty straightforward and should just work once you’ve put in your external interface and a rule for your IPv6 block.


View the full question and any other answers on Server Fault.

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.