I’m fairly new to k8s and minikube, and could use help understanding the frequent-but-not-ubiquitous error I get when running
kubectl logs -f <POD NAME>
I get this, sometimes, and unpredictably, on pods in my own namespace, and in
kube-system, e.g. when trying to tail the ingress controller logs. There is no apparent error in the pods themselves, and the output of the
kubectl logs command contains normal-looking log lines right up to, e.g.
failed to watch file "/var/log/pods/kube-system_nginx-ingress-controller-586cdc477c-95pgh_4b8310a4-5f9b-11e9-9666-0800270e7244/nginx-ingress-controller/1.log": no space left on device$
What’s the ultimate source of this, and how do I resolve it?
It sounds like the node has run out of inotify watches.
You can check or set the number of available watches using the sysctl
fs.inotify.max_user_watches on each Kubernetes node.
You will probably find it has been set to some ridiculously low number; by default it is autotuned based on the amount of memory the system has.
[[email protected] ~]# sysctl fs.inotify.max_user_watches fs.inotify.max_user_watches = 8192 [[email protected] ~]# sysctl fs.inotify.max_user_watches fs.inotify.max_user_watches = 1048576
Whatever the amount is currently, double it until the problem stops occurring.
minikube ssh to access a Minikube node.)
This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.