minikube/k8s/kubectl "failed to watch file [ … ]: no space left on device"

Ben asked:

I’m fairly new to k8s and minikube, and could use help understanding the frequent-but-not-ubiquitous error I get when running kubectl logs -f <POD NAME>

I get this, sometimes, and unpredictably, on pods in my own namespace, and in kube-system, e.g. when trying to tail the ingress controller logs. There is no apparent error in the pods themselves, and the output of the kubectl logs command contains normal-looking log lines right up to, e.g. failed to watch file "/var/log/pods/kube-system_nginx-ingress-controller-586cdc477c-95pgh_4b8310a4-5f9b-11e9-9666-0800270e7244/nginx-ingress-controller/1.log": no space left on device$

What’s the ultimate source of this, and how do I resolve it?

My answer:

It sounds like the node has run out of inotify watches.

You can check or set the number of available watches using the sysctl fs.inotify.max_user_watches on each Kubernetes node.

You will probably find it has been set to some ridiculously low number; by default it is autotuned based on the amount of memory the system has.

[[email protected] ~]# sysctl fs.inotify.max_user_watches
fs.inotify.max_user_watches = 8192

[[email protected] ~]# sysctl fs.inotify.max_user_watches
fs.inotify.max_user_watches = 1048576

Whatever the amount is currently, double it until the problem stops occurring.

(Use minikube ssh to access a Minikube node.)

View the full question and any other answers on Server Fault.

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.