大佬们请教一个问题,公司 k8s 环境有两台 node 频繁的在 Ready 和 NotReady 状态切换,间隔大概 3 分钟。 版本信息:
Kernel Version: 3.10.0-1062.el7.x86_64
OS Image: CentOS Linux 7 (Core)
Operating System: linux
Architecture: amd64
Container Runtime Version: docker://18.6.1
Kubelet Version: v1.14.1
Kube-Proxy Version: v1.14.1
节点信息
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 3312m (28%) 370m (3%)
memory 24302Mi (19%) 270Mi (0%)
ephemeral-storage 0 (0%) 0 (0%)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal NodeNotReady 9m10s (x12084 over 45d) kubelet, dev-11 Node dev-11 status is now: NodeNotReady
Normal NodeReady 4m9s (x12086 over 65d) kubelet, dev-11 Node dev-11 status is now: NodeReady
日志信息
9 月 15 10:06:47 dev-11 kubelet[2016]: I0915 10:06:47.940194 2016 setters.go:521] Node became not ready: {Type:Ready Status:False LastHeartbeatTime:2023-09-15 10:06:47.940166803 +0800 CST m=+5667448.191374429 LastTransitionTime:2023-09-15 10:06:47.940166803 +0800 CST m=+5667448.191374429 Reason:KubeletNotReady Message:PLEG is not healthy: pleg was last seen active 3m4.847729429s ago; threshold is 3m0s.}
9 月 15 10:06:50 dev-11 kubelet[2016]: I0915 10:06:50.280321 2016 kubelet.go:1823] skipping pod synchronization - PLEG is not healthy: pleg was last seen active 3m7.187849858s ago; threshold is 3m0s.
.....
9 月 15 10:07:40 dev-11 kubelet[2016]: I0915 10:07:40.281597 2016 kubelet.go:1823] skipping pod synchronization - PLEG is not healthy: pleg was last seen active 3m57.189127759s ago; threshold is 3m0s.
9 月 15 10:07:43 dev-11 kubelet[2016]: E0915 10:07:43.124845 2016 remote_runtime.go:321] ContainerStatus "1f718a7646f7c8126e784*********************930620d33ab9bb" from runtime service failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded
9 月 15 10:07:43 dev-11 kubelet[2016]: E0915 10:07:43.124906 2016 kuberuntime_manager.go:917] getPodContainerStatuses for pod "test-jdk11-1-0_test1(13*****-1fe1-11ee-a143-f4******bb5)" failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded
9 月 15 10:08:42 dev-11 kubelet[2016]: E0915 10:08:42.995808 2016 kubelet_pods.go:1093] Failed killing the pod "test-jdk11-1-0": failed to "KillContainer" for "jdk11" with KillContainerError: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
9 月 15 10:09:01 dev-11 kubelet[2016]: E0915 10:09:01.488058 2016 remote_runtime.go:402] Exec 1f718a7646f7c8126e784*********************930620d33ab9bb '/bin/sh' from runtime service failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded
9 月 15 10:09:44 dev-11 kubelet[2016]: E0915 10:09:44.151795 2016 remote_runtime.go:321] ContainerStatus "1f718a7646f7c8126e784*********************930620d33ab9bb" from runtime service failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded
9 月 15 10:09:44 dev-11 kubelet[2016]: E0915 10:09:44.151843 2016 kuberuntime_manager.go:917] getPodContainerStatuses for pod "test-jdk11-1-0_test1(13*****-1fe1-11ee-a143-f4******bb5)" failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded
9 月 15 10:10:05 dev-11 kubelet[2016]: E0915 10:10:05.742413 2016 remote_runtime.go:402] Exec 1f718a7646f7c8126e784*********************930620d33ab9bb '/bin/sh' from runtime service failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded
这是一个专为移动设备优化的页面(即为了让你能够在 Google 搜索结果里秒开这个页面),如果你希望参与 V2EX 社区的讨论,你可以继续到 V2EX 上打开本讨论主题的完整版本。
V2EX 是创意工作者们的社区,是一个分享自己正在做的有趣事物、交流想法,可以遇见新朋友甚至新机会的地方。
V2EX is a community of developers, designers and creative people.