请教个 k8s 问题， node 貌似不定期会挂

V2EX = way to explore

V2EX 是一个关于分享和探索的地方

已注册用户请登录

这是一个创建于 1748 天前的主题，其中的信息可能已经有所发展或是发生改变。

症状就是 controller_manager 的 log 里面定期会有 nodenotready 的日志

然后 pod 对应的 endpoint ip 被清空，看了下 journalctl -f -u kubelet 输出如下，对这块不太熟，请教懂的同学帮忙看看

journalctl -f -u kubelet 
-- Logs begin at Tue 2019-07-30 00:44:14 CEST. --
Jul 30 03:35:34 server-m1 kubelet[14853]: W0730 03:35:34.257474   14853 watcher.go:87] Error while processing event ("/sys/fs/cgroup/devices/libcontainer_5686_systemd_test_default.slice": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/devices/libcontainer_5686_systemd_test_default.slice: no such file or directory
Jul 30 03:35:44 server-m1 kubelet[14853]: W0730 03:35:44.166927   14853 watcher.go:87] Error while processing event ("/sys/fs/cgroup/blkio/libcontainer_5704_systemd_test_default.slice": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/blkio/libcontainer_5704_systemd_test_default.slice: no such file or directory
Jul 30 03:35:44 server-m1 kubelet[14853]: W0730 03:35:44.166982   14853 watcher.go:87] Error while processing event ("/sys/fs/cgroup/memory/libcontainer_5704_systemd_test_default.slice": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/memory/libcontainer_5704_systemd_test_default.slice: no such file or directory
Jul 30 03:35:44 server-m1 kubelet[14853]: W0730 03:35:44.167008   14853 watcher.go:87] Error while processing event ("/sys/fs/cgroup/devices/libcontainer_5704_systemd_test_default.slice": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/devices/libcontainer_5704_systemd_test_default.slice: no such file or directory
Jul 30 03:35:54 server-m1 kubelet[14853]: W0730 03:35:54.165976   14853 watcher.go:87] Error while processing event ("/sys/fs/cgroup/memory/libcontainer_5761_systemd_test_default.slice": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/memory/libcontainer_5761_systemd_test_default.slice: no such file or directory
Jul 30 03:35:54 server-m1 kubelet[14853]: W0730 03:35:54.166075   14853 watcher.go:87] Error while processing event ("/sys/fs/cgroup/devices/libcontainer_5761_systemd_test_default.slice": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/devices/libcontainer_5761_systemd_test_default.slice: no such file or directory
Jul 30 03:35:54 server-m1 kubelet[14853]: W0730 03:35:54.184573   14853 watcher.go:87] Error while processing event ("/sys/fs/cgroup/memory/libcontainer_5767_systemd_test_default.slice": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/memory/libcontainer_5767_systemd_test_default.slice: no such file or directory
Jul 30 03:35:54 server-m1 kubelet[14853]: W0730 03:35:54.184612   14853 watcher.go:87] Error while processing event ("/sys/fs/cgroup/devices/libcontainer_5767_systemd_test_default.slice": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/devices/libcontainer_5767_systemd_test_default.slice: no such file or directory
Jul 30 03:35:54 server-m1 kubelet[14853]: W0730 03:35:54.253787   14853 watcher.go:87] Error while processing event ("/sys/fs/cgroup/memory/libcontainer_5809_systemd_test_default.slice": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/memory/libcontainer_5809_systemd_test_default.slice: no such file or directory
Jul 30 03:35:54 server-m1 kubelet[14853]: W0730 03:35:54.253812   14853 watcher.go:87] Error while processing event ("/sys/fs/cgroup/devices/libcontainer_5809_systemd_test_default.slice": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/devices/libcontainer_5809_systemd_test_default.slice: no such file or directory

1 条回复 • 2019-07-30 13:49:55 +08:00

snappyone

2019-07-30 13:49:55 +08:00

貌似找到问题了，参照以下 issue：
https://github.com/kubernetes/kubernetes/issues/76531