踩到了 docker 一个 open 了 3 年+, 350+回复的 kennel 级神坑,如果是生产环境各位运维童鞋会怎么处理?

2017-06-25 13:18:25 +08:00
 fzinfz
unregister_netdevice: waiting for lo to become free. Usage count = #

https://github.com/moby/moby/issues/5618

貌似可能的原因很多,kernel 4.11 解决了一些:
https://github.com/coreos/bugs/issues/254
Another source of leakage has been fixed by torvalds/linux@f647821, which is part of 4.11.
torvalds/linux@b7c8487 is in the same batch of commits, is also in 4.11, and fixes another possible cause
16919 次点击
所在节点    Docker
23 条回复
choury
2017-06-25 13:50:28 +08:00
我们的做法是甩锅给内核组,让他们修
ryd994
2017-06-25 14:06:35 +08:00
几乎所有人都踩到的……
还好没影响我使用,除了 dmesg spamming 以外
既然有那么多内核大佬都在修,我们这些小菜只能等了
现在你知道运维找大师做法给服务器开光有啥用了吗?
哦,还有个办法是买 RH 订阅然后甩锅给 RH
xss
2017-06-25 14:29:14 +08:00
我也遇到了, 然而具体有什么影响?
cnnblike
2017-06-25 15:15:41 +08:00
我也遇到了,神烦
mritd
2017-06-25 16:07:30 +08:00
很正常
fuxkcsdn
2017-06-25 20:40:57 +08:00
遇到过+1
freestyle
2017-06-25 20:45:14 +08:00
遇到过 但不清楚有什么影响
htfy96
2017-06-25 21:34:56 +08:00
还有那个 userland-proxy,必须性能很差地在用户态搞 NAT,原因之一也是 kernel bug
dusheng
2017-06-25 23:33:10 +08:00
刚遇到过
SharkIng
2017-06-25 23:59:47 +08:00
这个问题把我们害惨了,不过据说最新 kernel 修复了?
fzinfz
2017-06-26 02:51:21 +08:00
@SharkIng 具体有啥影响?最新 kernel 有做了修复,不过不知是否完全修复。

这个错误我是刚重启的时候连不上 SSH 进 hyper-v console 才发现的,重启 VM 后 SSH 正常但是还是报了 2 条 unregister_netdevice,不知是否相关。

[以下有点乱,仅作个人记录]
看了下 dmesg,还伴随着很多“ docker0: port #(veth......) entered blocking/disabled state ”的 log,看了下 GCE/Vultr KVM 的 VM 也有 blocking/disabled log (但是没有 unregister_netdevice ),猜测是 packetbeat 抓包导致的。(没有 packetbeat 抓包的机子就没有这些 log。)
PS1:所有 container 都是--net host 运行
PS2:报 unregister_netdevice 的 VM 用的是 B150 主板板载 Realtek 网卡...
PS3:目前 kernel 4.10.0-19-generic,准备升 4.11 看下

贴下 unregister_netdevice 那台机器的 dmesg,当作记录:

[ 8.263327] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
[ 8.265021] Bridge firewalling registered
[ 8.270259] nf_conntrack version 0.5.0 (65536 buckets, 262144 max)
[ 8.352304] Initializing XFRM netlink socket
[ 8.395477] IPv6: ADDRCONF(NETDEV_UP): docker0: link is not ready
[ 8.934882] aufs au_opts_verify:1585:dockerd[1619]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 9.152098] docker0: port 1(veth41c7ef0) entered blocking state
[ 9.152098] docker0: port 1(veth41c7ef0) entered disabled state
[ 9.152267] device veth41c7ef0 entered promiscuous mode
[ 9.152695] IPv6: ADDRCONF(NETDEV_UP): veth41c7ef0: link is not ready
[ 9.152700] docker0: port 1(veth41c7ef0) entered blocking state
[ 9.152701] docker0: port 1(veth41c7ef0) entered forwarding state
[ 9.152771] docker0: port 1(veth41c7ef0) entered disabled state
[ 11.332410] eth0: renamed from vethe136227
[ 11.348824] IPv6: ADDRCONF(NETDEV_CHANGE): veth41c7ef0: link becomes ready
[ 11.348896] docker0: port 1(veth41c7ef0) entered blocking state
[ 11.348897] docker0: port 1(veth41c7ef0) entered forwarding state
[ 11.348985] IPv6: ADDRCONF(NETDEV_CHANGE): docker0: link becomes ready
[ 12.766843] docker0: port 2(veth49d7060) entered blocking state
[ 12.766844] docker0: port 2(veth49d7060) entered disabled state
[ 12.767074] device veth49d7060 entered promiscuous mode
[ 12.767203] IPv6: ADDRCONF(NETDEV_UP): veth49d7060: link is not ready
[ 12.767206] docker0: port 2(veth49d7060) entered blocking state
[ 12.767207] docker0: port 2(veth49d7060) entered forwarding state
[ 12.767329] docker0: port 2(veth49d7060) entered disabled state
[ 13.780687] eth0: renamed from vethc063ec5
[ 13.792810] IPv6: ADDRCONF(NETDEV_CHANGE): veth49d7060: link becomes ready
[ 13.792887] docker0: port 2(veth49d7060) entered blocking state
[ 13.792888] docker0: port 2(veth49d7060) entered forwarding state
[ 52.227899] hv_balloon: INFO_TYPE_MAX_PAGE_CNT = 3072000
[ 56.286544] hv_utils: KVP IC version 4.0
[ 910.342263] docker0: port 3(veth622db14) entered blocking state
[ 910.342265] docker0: port 3(veth622db14) entered disabled state
[ 910.342459] device veth622db14 entered promiscuous mode
[ 910.342573] IPv6: ADDRCONF(NETDEV_UP): veth622db14: link is not ready
[ 910.468949] eth0: renamed from vethf117385
[ 910.485201] IPv6: ADDRCONF(NETDEV_CHANGE): veth622db14: link becomes ready
[ 910.485297] docker0: port 3(veth622db14) entered blocking state
[ 910.485299] docker0: port 3(veth622db14) entered forwarding state
[ 924.891826] docker0: port 3(veth622db14) entered disabled state
[ 924.891832] vethf117385: renamed from eth0
[ 924.933207] docker0: port 3(veth622db14) entered disabled state
[ 924.935302] device veth622db14 left promiscuous mode
[ 924.935306] docker0: port 3(veth622db14) entered disabled state
[ 1088.348123] unregister_netdevice: waiting for lo to become free. Usage count = 1
[ 2182.977980] docker0: port 3(vethe1afdfb) entered blocking state
[ 2182.977981] docker0: port 3(vethe1afdfb) entered disabled state
[ 2182.978204] device vethe1afdfb entered promiscuous mode
[ 2182.978357] IPv6: ADDRCONF(NETDEV_UP): vethe1afdfb: link is not ready
[ 2183.076738] eth0: renamed from vethfcb6742
[ 2183.094282] IPv6: ADDRCONF(NETDEV_CHANGE): vethe1afdfb: link becomes ready
[ 2183.094481] docker0: port 3(vethe1afdfb) entered blocking state
[ 2183.094482] docker0: port 3(vethe1afdfb) entered forwarding state
[ 2194.980883] docker0: port 3(vethe1afdfb) entered disabled state
[ 2194.981195] vethfcb6742: renamed from eth0
[ 2195.018586] docker0: port 3(vethe1afdfb) entered disabled state
[ 2195.021027] device vethe1afdfb left promiscuous mode
[ 2195.021031] docker0: port 3(vethe1afdfb) entered disabled state
[ 3021.669760] docker0: port 3(vethb0d2857) entered blocking state
[ 3021.669760] docker0: port 3(vethb0d2857) entered disabled state
[ 3021.670990] device vethb0d2857 entered promiscuous mode
[ 3021.671312] IPv6: ADDRCONF(NETDEV_UP): vethb0d2857: link is not ready
[ 3021.784817] eth0: renamed from veth12e4511
[ 3021.804650] IPv6: ADDRCONF(NETDEV_CHANGE): vethb0d2857: link becomes ready
[ 3021.804739] docker0: port 3(vethb0d2857) entered blocking state
[ 3021.804740] docker0: port 3(vethb0d2857) entered forwarding state
[ 3022.613543] docker0: port 3(vethb0d2857) entered disabled state
[ 3022.613652] veth12e4511: renamed from eth0
[ 3022.651080] docker0: port 3(vethb0d2857) entered disabled state
[ 3022.656981] device vethb0d2857 left promiscuous mode
[ 3022.657016] docker0: port 3(vethb0d2857) entered disabled state
[ 3357.591305] docker0: port 3(vethb900f81) entered blocking state
[ 3357.591306] docker0: port 3(vethb900f81) entered disabled state
[ 3357.591425] device vethb900f81 entered promiscuous mode
[ 3357.591511] IPv6: ADDRCONF(NETDEV_UP): vethb900f81: link is not ready
[ 3357.696529] eth0: renamed from vethf79a1ab
[ 3357.716768] IPv6: ADDRCONF(NETDEV_CHANGE): vethb900f81: link becomes ready
[ 3357.716854] docker0: port 3(vethb900f81) entered blocking state
[ 3357.716855] docker0: port 3(vethb900f81) entered forwarding state
[ 3407.848335] vethf79a1ab: renamed from eth0
[ 3407.881076] docker0: port 3(vethb900f81) entered disabled state
[ 3407.898598] docker0: port 3(vethb900f81) entered disabled state
[ 3407.904083] device vethb900f81 left promiscuous mode
[ 3407.904104] docker0: port 3(vethb900f81) entered disabled state
[ 3603.292153] unregister_netdevice: waiting for lo to become free. Usage count = 1
SharkIng
2017-06-26 03:35:07 +08:00
@fzinfz #11 Kernel Panic, 表现类似于服务器断网,因为我们程序当初跑在 Docker 里面的,造成所有流出 Docker 的流量都无法流出。然后 Docker 的程序基本上就死了(主要是因为无法用了)最后我们就把整个程序做成 Dockerless 的了
Kernel 修复大概是这部分,但是我们没有测试: https://cdn.kernel.org/pub/linux/kernel/v4.x/ChangeLog-4.4.22
l142857
2017-06-26 09:48:00 +08:00
ixiaohei
2017-07-04 09:24:15 +08:00
貌似我们公司也碰到了,redhat 给我们的建议是升级为 RedHat 发行的 docker 及补充包来解决
jessynt
2017-07-22 19:09:08 +08:00
踩到
unregister_netdevice: waiting for lo to become free
0312birdzhang
2018-05-14 11:22:12 +08:00
kernel:unregister_netdevice: waiting for lo to become free. Usage count = 1

希望是踩到这个坑了,这样就可以甩锅了
caimaoy
2018-12-28 16:16:26 +08:00
踩锅 + 1
AngryPanda
2019-01-04 11:38:48 +08:00
踩锅 + 1
dozer47528
2019-02-22 17:52:26 +08:00
踩锅+1
1073850525
2019-04-28 19:11:06 +08:00
踩锅 + 1

这是一个专为移动设备优化的页面(即为了让你能够在 Google 搜索结果里秒开这个页面),如果你希望参与 V2EX 社区的讨论,你可以继续到 V2EX 上打开本讨论主题的完整版本。

https://www.v2ex.com/t/370924

V2EX 是创意工作者们的社区,是一个分享自己正在做的有趣事物、交流想法,可以遇见新朋友甚至新机会的地方。

V2EX is a community of developers, designers and creative people.

© 2021 V2EX