Stopping IPTables is the only way to make Kubernetes cluster work

user2405589 asked:

We have an inhouse 1.17.5 K8s cluster – 5 nodes. I cannot deploy, collect logs, anything on the cluster when IPTables is enabled.

[[email protected] ~]# kubectl run --generator=run-pod/v1  --rm utils -it --image quaytest.phx.aexp.com/sanupin/utils:1.0 bash

[[email protected] couchbase-autonomous-operator-kubernetes_2.0.1-linux-x86_64]# kubectl exec -it hello-0 bash

[[email protected] ~]# kubectl logs kube-proxy-kqs7m --tail 10 -n kube-system
[[email protected] ~]# kubectl logs couchbase-operator-d9696755c-tqx57

All the above operations just hang when IPTables is enabled.

The API server logs (I can get this since it’s on the same VM as my control plane and I am logged on to) show that there’s problems connecting to port 10250

Trace[1253082920]: [11.90845011s] [11.906664621s] Transformed response object
E0728 21:39:10.658466       1 status.go:71] apiserver received an error that is not an metav1.Status: &url.Error{Op:"Get", URL:"https://10.22.77.12:10250/containerLogs/default/couchbase-operator-d9696755c-tqx57/couchbase-operator", Err:(*errors.errorString)(0xc000098260)}
I0728 21:39:10.658761       1 trace.go:116] Trace[128874851]: "Get" url:/api/v1/namespaces/default/pods/couchbase-operator-d9696755c-tqx57/log,user-agent:kubectl/v1.17.5 (linux/amd64) kubernetes/e0fccaf,client:10.22.76.244 (started: 2020-07-28 21:39:06.353221799 +0000 UTC m=+80919.504899548) (total time: 4.30550213s):
Trace[128874851]: [4.305499605s] [4.303636525s] Transformed response object

I have configured 10250 on ALL my nodes:

[[email protected] ~]# iptables -L | grep 10250
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:10250
ACCEPT     tcp  --  anywhere             anywhere             tcp spt:10250

[[email protected] ~]# iptables -L | grep 10250
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:10250
ACCEPT     tcp  --  anywhere             anywhere             tcp spt:10250

[[email protected] ~]# iptables -L | grep 10250
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:10250
ACCEPT     tcp  --  anywhere             anywhere             tcp spt:10250

But no luck yet with accessing any logs.

I have calico pod network running:

[[email protected] couchbase-autonomous-operator-kubernetes_2.0.1-linux-x86_64]# kubectl get pods -n kube-system -owide
NAME                                                READY   STATUS    RESTARTS   AGE   IP               NODE                        NOMINATED NODE   READINESS GATES
calico-kube-controllers-58c67bc699-g2dzw            1/1     Running   0          47h   192.168.76.195   lpdkubpoc01a.phx.aexp.com   <none>           <none>
calico-node-9khc6                                   1/1     Running   0          47h   10.22.77.15      lpdkubpoc01e.phx.aexp.com   <none>           <none>
calico-node-fc9kp                                   1/1     Running   0          47h   10.22.77.12      lpdkubpoc01c.phx.aexp.com   <none>           <none>
calico-node-htxbh                                   1/1     Running   0          47h   10.22.76.245     lpdkubpoc01b.phx.aexp.com   <none>           <none>
calico-node-q59vd                                   1/1     Running   0          47h   10.22.77.13      lpdkubpoc01d.phx.aexp.com   <none>           <none>
calico-node-zkwtr                                   1/1     Running   0          47h   10.22.76.244     lpdkubpoc01a.phx.aexp.com   <none>           <none>
coredns-598947db54-dtsjk                            1/1     Running   0          47h   192.168.76.193   lpdkubpoc01a.phx.aexp.com   <none>           <none>
coredns-598947db54-mrjjl                            1/1     Running   0          47h   192.168.76.194   lpdkubpoc01a.phx.aexp.com   <none>           <none>
etcd-lpdkubpoc01a.phx.aexp.com                      1/1     Running   0          47h   10.22.76.244     lpdkubpoc01a.phx.aexp.com   <none>           <none>
kube-apiserver-lpdkubpoc01a.phx.aexp.com            1/1     Running   0          47h   10.22.76.244     lpdkubpoc01a.phx.aexp.com   <none>           <none>
kube-controller-manager-lpdkubpoc01a.phx.aexp.com   1/1     Running   0          47h   10.22.76.244     lpdkubpoc01a.phx.aexp.com   <none>           <none>
kube-proxy-2z5rx                                    1/1     Running   0          47h   10.22.76.245     lpdkubpoc01b.phx.aexp.com   <none>           <none>
kube-proxy-55jgf                                    1/1     Running   0          47h   10.22.77.15      lpdkubpoc01e.phx.aexp.com   <none>           <none>
kube-proxy-f5k5f                                    1/1     Running   0          47h   10.22.76.244     lpdkubpoc01a.phx.aexp.com   <none>           <none>
kube-proxy-gskwj                                    1/1     Running   0          47h   10.22.77.13      lpdkubpoc01d.phx.aexp.com   <none>           <none>
kube-proxy-kqs7m                                    1/1     Running   0          47h   10.22.77.12      lpdkubpoc01c.phx.aexp.com   <none>           <none>
kube-scheduler-lpdkubpoc01a.phx.aexp.com            1/1     Running   0          47h   10.22.76.244     lpdkubpoc01a.phx.aexp.com   <none>           <none>

Below is my IPTables configuration on the server which happens to host a hello world pod:

When IPTables is disabled, no problems doing any operation.

EDIT:

Below is the complete non-working firewall

[[email protected] ~]# iptables -nvL
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
1459K  268M cali-INPUT  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:Cz_u1IQiXIMmKD4c */
 790K   62M KUBE-SERVICES  all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate NEW /* kubernetes service portals */
 790K   62M KUBE-EXTERNAL-SERVICES  all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate NEW /* kubernetes externally-visible service portals */
    2   144 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0            /* 001 accept all icmp - Puppet Managed by fw_base */
 221K   16M ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0            /* 002 accept all to lo interface */
    0     0 REJECT     all  --  !lo    *       0.0.0.0/0            127.0.0.0/8          /* 003 reject local traffic not on loopback interface */ reject-with icmp-port-unreachable
 474K  193M ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* 004 accept related established rules */ state RELATED,ESTABLISHED
    2   128 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            multiport dports 22 /* 005 ssh - port 22 */
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            multiport dports 8089 /* 006 splunk client - port 8089 */
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            multiport dports 9898 /* 007 tripwire client - port 9898 */
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            multiport dports 5666 /* 009 nrpe/nagios client - port 5666 */
  789 47340 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            multiport dports 17472 /* 110 allow taniumclient access - port 17472 */
   10   600 LOGIT      tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp flags:0x17/0x02 /* 990 forward new SYN input to LOGIT chain */
 771K   61M DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* 999 drop everything else */
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:10250

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 cali-FORWARD  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:wUHhoiAYhphO9Mso */
    0     0 KUBE-FORWARD  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes forwarding rules */
    0     0 KUBE-SERVICES  all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate NEW /* kubernetes service portals */

Chain OUTPUT (policy ACCEPT 10956 packets, 1349K bytes)
 pkts bytes target     prot opt in     out     source               destination
 783K   84M cali-OUTPUT  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:tVnHkvAo15HuiPy0 */
 129K   25M KUBE-SERVICES  all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate NEW /* kubernetes service portals */
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp spt:10250

Chain KUBE-EXTERNAL-SERVICES (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain KUBE-FIREWALL (0 references)
 pkts bytes target     prot opt in     out     source               destination

Chain KUBE-FORWARD (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate INVALID
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes forwarding rules */ mark match 0x4000/0x4000
    0     0 ACCEPT     all  --  *      *       192.168.0.0/16       0.0.0.0/0            /* kubernetes forwarding conntrack pod source rule */ ctstate RELATED,ESTABLISHED
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            192.168.0.0/16       /* kubernetes forwarding conntrack pod destination rule */ ctstate RELATED,ESTABLISHED

Chain KUBE-KUBELET-CANARY (0 references)
 pkts bytes target     prot opt in     out     source               destination

Chain KUBE-PROXY-CANARY (0 references)
 pkts bytes target     prot opt in     out     source               destination

Chain KUBE-SERVICES (3 references)
 pkts bytes target     prot opt in     out     source               destination

Chain LOGIT (1 references)
 pkts bytes target     prot opt in     out     source               destination
   10   600 LOG        tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* 991 configure LOGIT chain to log everything as DROP INBOUND TCP */ LOG flags 0 level 4 prefix "DROP INBOUND TCP "

Chain cali-FORWARD (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:vjrMJCRpqwy5oRoX */ MARK and 0xfff1ffff
    0     0 cali-from-hep-forward  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:A_sPAO0mcxbT9mOV */ mark match 0x0/0x10000
    0     0 cali-from-wl-dispatch  all  --  cali+  *       0.0.0.0/0            0.0.0.0/0            /* cali:8ZoYfO5HKXWbB3pk */
    0     0 cali-to-wl-dispatch  all  --  *      cali+   0.0.0.0/0            0.0.0.0/0            /* cali:jdEuaPBe14V2hutn */
    0     0 cali-to-hep-forward  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:12bc6HljsMKsmfr- */
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:MH9kMp5aNICL-Olv */ /* Policy explicitly accepted packet. */ mark match 0x10000/0x10000

Chain cali-INPUT (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 ACCEPT     4    --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:PajejrV4aFdkZojI */ /* Allow IPIP packets from Calico hosts */ match-set cali40all-hosts-net src ADDRTYPE match dst-type LOCAL
    0     0 DROP       4    --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:_wjq-Yrma8Ly1Svo */ /* Drop IPIP packets from non-Calico hosts */
    0     0 cali-wl-to-host  all  --  cali+  *       0.0.0.0/0            0.0.0.0/0           [goto]  /* cali:8TZGxLWh_Eiz66wc */
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:6McIeIDvPdL6PE1T */ mark match 0x10000/0x10000
1466K  269M MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:YGPbrUms7NId8xVa */ MARK and 0xfff0ffff
1466K  269M cali-from-host-endpoint  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:2gmY7Bg2i0i84Wk_ */
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:q-Vz2ZT9iGE331LL */ /* Host endpoint policy accepted packet. */ mark match 0x10000/0x10000

Chain cali-OUTPUT (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:Mq1_rAdXXH3YkrzW */ mark match 0x10000/0x10000
    0     0 RETURN     all  --  *      cali+   0.0.0.0/0            0.0.0.0/0            /* cali:69FkRTJDvD5Vu6Vl */
    0     0 ACCEPT     4    --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:AnEsmO6bDZbQntWW */ /* Allow IPIP packets to other Calico hosts */ match-set cali40all-hosts-net dst ADDRTYPE match src-type LOCAL
 787K   85M MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:9e9Uf3GU5tX--Lxy */ MARK and 0xfff0ffff
 787K   85M cali-to-host-endpoint  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:OB2pzPrvQn6PC89t */
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:tvSSMDBWrme3CUqM */ /* Host endpoint policy accepted packet. */ mark match 0x10000/0x10000

Chain cali-failsafe-in (0 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:wWFQM43tJU7wwnFZ */ multiport dports 22
    0     0 ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:LwNV--R8MjeUYacw */ multiport dports 68
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:QOO5NUOqOSS1_Iw0 */ multiport dports 179
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:cwZWoBSwVeIAZmVN */ multiport dports 2379
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:7FbNXT91kugE_upR */ multiport dports 2380
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:ywE9WYUBEpve70WT */ multiport dports 6666
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:l-WQSVBf_lygPR0J */ multiport dports 6667

Chain cali-failsafe-out (0 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:82hjfji-wChFhAqL */ multiport dports 53
    0     0 ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:TNM3RfEjbNr72hgH */ multiport dports 67
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:ycxKitIl4u3dK0HR */ multiport dports 179
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:hxjEWyxdkXXkdvut */ multiport dports 2379
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:cA_GLtruuvG88KiO */ multiport dports 2380
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:Sb1hkLYFMrKS6r01 */ multiport dports 6666
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:UwLSebGONJUG4yG- */ multiport dports 6667

Chain cali-from-hep-forward (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain cali-from-host-endpoint (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain cali-from-wl-dispatch (2 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 cali-fw-cali5bdd8f7a3d4  all  --  cali5bdd8f7a3d4 *       0.0.0.0/0            0.0.0.0/0           [goto]  /* cali:Miz_dfm_OqFqStOj */
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:j9w09IE3nQzJkJrt */ /* Unknown interface */

Chain cali-fw-cali5bdd8f7a3d4 (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:RQw05lu9TEo6E9J7 */ ctstate RELATED,ESTABLISHED
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:01JI5c18EIipS498 */ ctstate INVALID
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:VyKbvKvpg3t6bqdZ */ MARK and 0xfffeffff
    0     0 DROP       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:rxc0MECUB43_dUKZ */ /* Drop VXLAN encapped packets originating in pods */ multiport dports 4789
    0     0 DROP       4    --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:IhtTke9qggxJCRvo */ /* Drop IPinIP encapped packets originating in pods */
    0     0 cali-pro-kns.default  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:V5EIVhFIRYorU3ee */
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:Gx-cKFSZy-GfPVMs */ /* Return if profile accepted */ mark match 0x10000/0x10000
    0     0 cali-pro-ksa.default.default  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:FSUAC6Xrp8hOklGS */
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:80aV6ux2xgqiIrzU */ /* Return if profile accepted */ mark match 0x10000/0x10000
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:dSsEIrLZxSINZx51 */ /* Drop if no profiles matched */

Chain cali-pri-kns.default (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:7Fnh7Pv3_98FtLW7 */ MARK or 0x10000
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:ZbV6bJXWSRefjK0u */ mark match 0x10000/0x10000

Chain cali-pri-ksa.default.default (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain cali-pro-kns.default (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:oLzzje5WExbgfib5 */ MARK or 0x10000
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:4goskqvxh5xcGw3s */ mark match 0x10000/0x10000

Chain cali-pro-ksa.default.default (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain cali-to-hep-forward (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain cali-to-host-endpoint (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain cali-to-wl-dispatch (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 cali-tw-cali5bdd8f7a3d4  all  --  *      cali5bdd8f7a3d4  0.0.0.0/0            0.0.0.0/0           [goto]  /* cali:SsXGJ85OfhKFm0ei */
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:pP7bpa3eH6NFcNsD */ /* Unknown interface */

Chain cali-tw-cali5bdd8f7a3d4 (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:uNKFNt79CGfLzpK9 */ ctstate RELATED,ESTABLISHED
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:ObEnKmaWBF0EWLU3 */ ctstate INVALID
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:JMSA5XB5i9j9eeav */ MARK and 0xfffeffff
    0     0 cali-pri-kns.default  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:ogNpbuRSm1qaUhka */
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:owLwcUjuD59m1Twf */ /* Return if profile accepted */ mark match 0x10000/0x10000
    0     0 cali-pri-ksa.default.default  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:89--cr3F1NqYju12 */
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:KbMS7iLpFxJMiJWe */ /* Return if profile accepted */ mark match 0x10000/0x10000
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:u5nFxLY0XdpnCUkZ */ /* Drop if no profiles matched */

Chain cali-wl-to-host (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 cali-from-wl-dispatch  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:Ee9Sbo10IpVujdIY */
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:nSZbcOoG1xPONxb8 */ /* Configured DefaultEndpointToHostAction */

Also, the host network is
10.22.76.0/23

Pod network is 192.168.0.0/16

Please help!

My answer:


Your rule to accept traffic to TCP port 10250 will never match, because it is at the end of the INPUT chain, and appears after the rule to DROP everything. It should be moved up, before the rules that log and drop traffic.


View the full question and any other answers on Server Fault.

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.