问题

监控报警了,top 命令查看

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                      
 1171 root      20   0 1155672 785112  77040 S 120.3  9.6 157:44.39 kube-apiserver                               
 7903 root      20   0 10.742g 777632  46784 S   5.3  9.5   8:23.43 etcd                                         
 8957 root      20   0 1365948 123764  73864 S   1.3  1.5   2:57.95 kubelet                                      
10369 root      20   0   44012  31584  20276 S   1.3  0.4   1:53.49 calico-felix                                 
 1147 root      20   0  451168  89944  68120 S   1.0  1.1   1:51.80 kube-scheduler

可以看到 CPU 飙到了 120%,不知道是什么原因导致的

HypoChen

2019-04-25 17:01:23 +08:00

先查日志，看看有啥异常，比如是不是啥服务 bug 了在 D 你的 api server

salamanderMH

2019-04-25 17:12:34 +08:00

@HypoChen
我看了下 apisever 的日志
```
E0425 09:11:11.383772 1 available_controller.go:311] v1beta1.admission.certmanager.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.admission.certmanager.k8s.io": the object has been modified; please apply your changes to the latest version and try again
E0425 09:11:14.341853 1 available_controller.go:311] v1beta1.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again
E0425 09:11:16.391080 1 available_controller.go:311] v1beta1.admission.certmanager.k8s.io failed with: Get https://10.43.42.227:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0425 09:11:19.349480 1 available_controller.go:311] v1beta1.metrics.k8s.io failed with: Get https://10.43.219.61:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0425 09:11:21.400839 1 available_controller.go:311] v1beta1.admission.certmanager.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.admission.certmanager.k8s.io": the object has been modified; please apply your changes to the latest version and try again
E0425 09:11:24.367592 1 available_controller.go:311] v1beta1.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again
```

HypoChen

2019-04-25 18:30:02 +08:00

@salamanderMH api server 的网络请求量如何？

salamanderMH

2019-04-25 20:09:50 +08:00

@HypoChen 我看到内网流出带宽有 1.29M bit/s，内网流入带宽是 400k bit/s

0312birdzhang

2019-04-26 08:16:42 +08:00

什么版本的？感觉你这个版本有 bug，重启 kubelet 可以缓解

salamanderMH

2019-04-26 12:31:18 +08:00

@0312birdzhang 1.11

0312birdzhang

2019-04-26 12:35:39 +08:00

@salamanderMH 具体到小版本号

salamanderMH

2019-04-26 12:46:06 +08:00

@0312birdzhang v1.11.6

0312birdzhang

2019-04-26 13:36:31 +08:00

@salamanderMH #8 可以直接升级到 1.11.7，有一个 bug 在 1.11.7 修复了。不过看到你的报错还跟我们的不完全一样，我们的是提示 version 已经更改了什么的

salamanderMH

2019-04-26 14:22:05 +08:00

@0312birdzhang 好的,我试试

这是一个专为移动设备优化的页面（即为了让你能够在 Google 搜索结果里秒开这个页面），如果你希望参与 V2EX 社区的讨论，你可以继续到 V2EX 上打开本讨论主题的完整版本。

https://www.v2ex.com/t/558667

k8s kube-apiserver CPU 负载很高

问题