一个关于 Linux 进程的奇怪问题

2021-02-07 17:31:45 +08:00
 zhoudaiyu
是这样的,今天发现 k8s 有 2 台 master 节点 CPU 使用率有尖峰( k8s 1.11 Linux 3.10 1.13.1 ),由于没有当时进程状态快照,只能用 top 每秒抓一次进程扔日志里。果然过了一段时间,发现了几个叫 docker-containerd-current 的进程 cpu 飙升导致 CPU 使用率高达 90%,但是我们通过 ps pgrep 等等手段都查不到这个进程了,所以考虑是不是当时突然启动了这些进程然后又死了,后来想从 /proc 里查有没有啥信息,奇怪的事发生了:ll -arth /proc|grep $PID 通过 pid 号对应文件夹查不到这个进程,但是 cd $PID 居然可以进到进程的文件夹里,然后我不死心,退回到 /proc 里又肉眼看了一遍文件夹,还是没找到。进到 /proc/$PID 内查看了 exe 对应的二进制文件,然后通过 ps -ef|grep $EXE_FILE 发现对应的进程启动时间还是去年。

这里的问题就是 /proc 下面的以 pid 命名的文件夹不应该是正在运行着的进程的么?进程确实死了,通过文件名也查不到了,但是为什么可以进到一个根本不存在的文件夹呢?

重点是图里面圈中的部分。
2651 次点击
所在节点    Linux
24 条回复
mikeguan
2021-02-07 18:01:37 +08:00
ls 显示目录的参数是-d
zhoudaiyu
2021-02-07 18:02:50 +08:00
@mikeguan 不用加这个参数也可以看文件夹啊
lcdtyph
2021-02-07 18:07:49 +08:00
ll 是什么的 alias
zhoudaiyu
2021-02-07 18:12:01 +08:00
@lcdtyph ls -l
mikeguan
2021-02-07 18:15:08 +08:00
@zhoudaiyu 不加也可以,输出太多更乱
zhoudaiyu
2021-02-07 18:16:54 +08:00
@mikeguan 但是问题是查不到文件夹为啥可以进去,而且进程都死了,不应该在 proc 下了啊
mikeguan
2021-02-07 18:23:11 +08:00
@zhoudaiyu 具体原因不清楚,可以用 lsof -p 14546 看下有没有结果
zhoudaiyu
2021-02-07 18:23:30 +08:00
@mikeguan 看过 没有
Jirajine
2021-02-07 18:26:38 +08:00
可能是进程被隐藏了。
zhoudaiyu
2021-02-07 18:27:47 +08:00
@Jirajine 还有这种操作?那么有什么方法能看到被隐藏的进程吗?
lcdtyph
2021-02-07 18:29:43 +08:00
@zhoudaiyu
你看一下现在 procfs 的 mount options 是什么
Jirajine
2021-02-07 18:32:23 +08:00
lcdtyph
2021-02-07 18:38:12 +08:00
@zhoudaiyu
如果没有什么特殊的 mount options,那说明你这个不是 pid,而是某个 tid,你需要找到它的主进程再看。

/proc/<tid> 是不会出现在 getdents 调用中,所以不会被 ls 看到
barathrum
2021-02-07 18:50:52 +08:00
@lcdtyph 感谢兄弟,学到一个新知识。
zhoudaiyu
2021-02-07 19:03:02 +08:00
@lcdtyph 老哥 我看有 mountinfo mounts mountstats 截图里是 mountstats 的输出,tid 是线程 id 吗?

device rootfs mounted on / with fstype rootfsdevice sysfs mounted on /sys with fstype sysfs
device proc mounted on /proc with fstype procdevice devtmpfs mounted on /dev with fstype devtmpfs
device securityfs mounted on /sys/kernel/security with fstype securityfsdevice tmpfs mounted on /dev/shm with fstype tmpfsdevice devpts mounted on /dev/pts with fstype devptsdevice tmpfs mounted on /run with fstype tmpfs
device tmpfs mounted on /sys/fs/cgroup with fstype tmpfsdevice cgroup mounted on /sys/fs/cgroup/systemd with fstype cgroupdevice pstore mounted on /sys/fs/pstore with fstype pstoredevice cgroup mounted on /sys/fs/cgroup/pids with fstype cgroup
device cgroup mounted on /sys/fs/cgroup/blkio with fstype cgroupdevice cgroup mounted on /sys/fs/cgroup/perf_event with fstype cgroup
device cgroup mounted on /sys/fs/cgroup/freezer with fstype cgroupdevice cgroup mounted on /sys/fs/cgroup/net_cls,net_prio with fstype cgroupdevice cgroup mounted on /sys/fs/cgroup/devices with fstype cgroupdevice cgroup mounted on /sys/fs/cgroup/cpu,cpuacct with fstype cgroup
device cgroup mounted on /sys/fs/cgroup/hugetlb with fstype cgroup
device cgroup mounted on /sys/fs/cgroup/cpuset with fstype cgroupdevice cgroup mounted on /sys/fs/cgroup/memory with fstype cgroup
device configfs mounted on /sys/kernel/config with fstype configfsdevice /dev/mapper/vg_root-lv_root mounted on / with fstype xfsdevice selinuxfs mounted on /sys/fs/selinux with fstype selinuxfsdevice systemd-1 mounted on /proc/sys/fs/binfmt_misc with fstype autofs
device hugetlbfs mounted on /dev/hugepages with fstype hugetlbfsdevice debugfs mounted on /sys/kernel/debug with fstype debugfs
device mqueue mounted on /dev/mqueue with fstype mqueue
device /dev/sda1 mounted on /boot with fstype xfsdevice /dev/mapper/vg_root-lv_var mounted on /var with fstype xfsdevice /dev/mapper/vg_root-lv_opt mounted on /opt with fstype xfsdevice /dev/mapper/vg_app-lv_DATA mounted on /DATA with fstype xfs
device /dev/mapper/vg_root-lv_tmp mounted on /tmp with fstype xfs
device /dev/mapper/vg_root-lv_home mounted on /home with fstype xfs
device sunrpc mounted on /var/lib/nfs/rpc_pipefs with fstype rpc_pipefs
device /dev/mapper/vg_root-lv_var mounted on /var/lib/docker/containers with fstype xfs
device /dev/mapper/vg_root-lv_var mounted on /var/lib/docker/overlay2 with fstype xfs
device tmpfs mounted on /var/lib/origin/openshift.local.volumes/pods/206f7c4c-52cd-11ea-9f39-0050569a04bf/volumes/kubernetes.io~secret/sdn-token-d66w8 with fstype tmpfs
device tmpfs mounted on /var/lib/origin/openshift.local.volumes/pods/e3695fa7-2dce-11ea-97bb-005056a99505/volumes/kubernetes.io~secret/sdn-token-d66w8 with fstype tmpfs
device tmpfs mounted on /var/lib/origin/openshift.local.volumes/pods/5abfd92e-2dc8-11ea-97bb-005056a99505/volumes/kubernetes.io~secret/sync-token-tr7f8 with fstype tmpfs
device overlay mounted on /var/lib/docker/overlay2/c47caa96460e233c36b6790b1ca4ea450dee1d297f9e40bf199c6ab83bb1e690/merged with fstype overlay
device overlay mounted on /var/lib/docker/overlay2/edb026a3c373d3ddbd064b9d8e997ccb441d56eb883f19237374bdd839cd876a/merged with fstype overlay
device proc mounted on /run/docker/netns/default with fstype proc
device shm mounted on /var/lib/docker/containers/6d3e50e348e7bc4608b568fd43ad63670c57f0c53de71f446ba17e8ec5e47f4e/shm with fstype tmpfs
device shm mounted on /var/lib/docker/containers/f2c824ce425cfe15dda053af70d4fd02d64afdae9367ae78bcd6b5758338182c/shm with fstype tmpfs
device overlay mounted on /var/lib/docker/overlay2/60fffaf1c2c9cbe0d12504a67a0b603c9bdbf8b0f6322dc30b2befa9cbb87438/merged with fstype overlay
device overlay mounted on /var/lib/docker/overlay2/0045f1ad6a6556324b91b80892dcc4951ad32ec8dfe153b7c866b942fae74843/merged with fstype overlay
device overlay mounted on /var/lib/docker/overlay2/ec0204696ace63e3c1eb69bd81e029c13d7c1320b16b561ddceb3c56f4ae9172/merged with fstype overlay
device shm mounted on /var/lib/docker/containers/a774eddee040f246b9dea20a33781a1b8cb80a3bc668690d66baec33a0f56a1d/shm with fstype tmpfs
device overlay mounted on /var/lib/docker/overlay2/cfcfc406430dc094d2b2a7084f38ea44b1eb2200282ce5fa432d46c9fee9d1f6/merged with fstype overlay
device shm mounted on /var/lib/docker/containers/6d7a77c6a7dade24e619f63aa813962ae003e52f15dc146141af3c362564fd27/shm with fstype tmpfs
device overlay mounted on /var/lib/docker/overlay2/52230e78851c2d66d058e1fda52b3cb8b3493a86804354d234fe64895b3aed56/merged with fstype overlay
device tmpfs mounted on /var/lib/origin/openshift.local.volumes/pods/a684975d-9ef2-11ea-90f1-005056a9a751/volumes/kubernetes.io~secret/node-exporter-token-mttqc with fstype tmpfs
device overlay mounted on /var/lib/docker/overlay2/c07d51282d5db3eeaa6966bdb277d34fdab0707900b34822b5bb85c6f3dddc84/merged with fstype overlay
device shm mounted on /var/lib/docker/containers/14c0c7e261e57a3eac08de62225b5b33fe8a11d231b13e34ddbdec8231e98d24/shm with fstype tmpfs
device overlay mounted on /var/lib/docker/overlay2/7c510eedb0ebb50ace710478ceca3037011a99a24ded9b2862d2700064782b22/merged with fstype overlay
device overlay mounted on /var/lib/docker/overlay2/be29c4a1f9bbd28181851f27df38bf277a1984595490599db5d4297de8d0847f/merged with fstype overlay
device tmpfs mounted on /var/lib/origin/openshift.local.volumes/pods/42a3edd4-02be-11eb-854a-005056a958e5/volumes/kubernetes.io~secret/oauth-config with fstype tmpfs
device tmpfs mounted on /var/lib/origin/openshift.local.volumes/pods/42a3edd4-02be-11eb-854a-005056a958e5/volumes/kubernetes.io~secret/serving-cert with fstype tmpfs
device tmpfs mounted on /var/lib/origin/openshift.local.volumes/pods/42a3edd4-02be-11eb-854a-005056a958e5/volumes/kubernetes.io~secret/default-token-mlq5l with fstype tmpfs
device overlay mounted on /var/lib/docker/overlay2/2a50c3d9829ae039bbe3afb1108e2b131aca83febf40070b0423761aee576354/merged with fstype overlay
device shm mounted on /var/lib/docker/containers/4b04ef977813ab3b24704d433d09d56fd069ee9103a32c5d041a3a072eab4663/shm with fstype tmpfs
device proc mounted on /run/docker/netns/4dd48aec066f with fstype proc
device overlay mounted on /var/lib/docker/overlay2/e1ee93593bab0e24a711c4ce871534fefe651e8be5ee11cf64f544e0b3e2e86f/merged with fstype overlay
device binfmt_misc mounted on /proc/sys/fs/binfmt_misc with fstype binfmt_misc
device overlay mounted on /var/lib/docker/overlay2/e04f7635399c36249c8e9b90d5619ba593b7633275d1b2c2363d3c93d0e2f55b/merged with fstype overlay
device overlay mounted on /var/lib/docker/overlay2/0be6198515753a8a97b51d078ac64e1153325342221e8d77a956eecabd448d12/merged with fstype overlay
device shm mounted on /var/lib/docker/containers/f54abb6462617d5910b8d41e5fa4c131be18aa59d4d227040e3f3e2ecd69d70a/shm with fstype tmpfs
device overlay mounted on /var/lib/docker/overlay2/a177d06ddec1fb54a0b04725fabe944293ca72eb4aa0bdbf4e918f828d089466/merged with fstype overlay
device overlay mounted on /var/lib/docker/overlay2/0af1e5fe64c765d3e0a4d574b725829a6b6e3170b929ede21cdd5b5cb97b0fc4/merged with fstype overlay
device shm mounted on /var/lib/docker/containers/284f17f11532b9750a1f15adb3c2361796c42c4e0afc220c49c0cd394c477299/shm with fstype tmpfs
device overlay mounted on /var/lib/docker/overlay2/ce6e3d2eb2879de645014da2be3650c7f172634131d08718304ef106a3b4ea66/merged with fstype overlay
device tmpfs mounted on /var/lib/origin/openshift.local.volumes/pods/37a179b0-690c-11eb-b1fb-005056a9a751/volumes/kubernetes.io~secret/serving-cert with fstype tmpfs
device tmpfs mounted on /var/lib/origin/openshift.local.volumes/pods/37a179b0-690c-11eb-b1fb-005056a9a751/volumes/kubernetes.io~secret/webconsole-token-spxpx with fstype tmpfs
device overlay mounted on /var/lib/docker/overlay2/188242c0ee6187645f21d2f0195615af2d450337a461c4c197367bfaf794d5ed/merged with fstype overlay
device shm mounted on /var/lib/docker/containers/5b5fb40679472f61b4b5f37caf20666291fdb38f04648710dbd23a3e952c0845/shm with fstype tmpfs
device proc mounted on /run/docker/netns/feff0fd124b3 with fstype proc
device overlay mounted on /var/lib/docker/overlay2/d6d7732f83a71c8c3f33a257bab98a140b5099e0ba2b1bb789c057520cf75d95/merged with fstype overlay
device tmpfs mounted on /run/user/0 with fstype tmpfs
zhoudaiyu
2021-02-07 19:03:51 +08:00
@lcdtyph 错了,不是截图是是文本 🤣
lcdtyph
2021-02-07 19:13:09 +08:00
@zhoudaiyu #16
这里没有啊,你直接输入 mount 命令就会列出所有 mounted 和 options 了

mount | grep proc

输出括号里面的部分就是 options
lcdtyph
2021-02-07 19:17:37 +08:00
@zhoudaiyu #16
要找 tid 的话给 ps 加上-L 参数就行了
ps -efL | grep 14546
其中的 LWP 列就是 tid
zhoudaiyu
2021-02-07 19:30:26 +08:00
@lcdtyph mount|grep 14546 返回空,ps -efL | grep 14546 这个好使,多谢老哥,但是我还是不太明白为啥 /proc 看不到这个 14546 这个进程号的文件夹,但是可以 cd 到 14546 中来,老哥能解个惑不?
lcdtyph
2021-02-07 19:47:04 +08:00
@zhoudaiyu
mount 那条命令是看 procfs 的 options 的,而且不是 grep 14546,就是 grep proc 。如果设置了 hidepid=2 那么 /proc 下就看不到其他用户的 pid 了

你这里的 14546 不是进程号而是个线程号(tid),/proc/tid 下的内容和 /proc/pid/task/tid 下的内容是一模一样的,所以实现 procfs 的时候让这些入口不在 getdents 系统调用的结果中出现。而 ls 正是使用 getdents 实现的,所以 ls 自然看不到这些入口。

cd 依赖的是 chdir 系统调用,所以不受影响

这是一个专为移动设备优化的页面(即为了让你能够在 Google 搜索结果里秒开这个页面),如果你希望参与 V2EX 社区的讨论,你可以继续到 V2EX 上打开本讨论主题的完整版本。

https://www.v2ex.com/t/752138

V2EX 是创意工作者们的社区,是一个分享自己正在做的有趣事物、交流想法,可以遇见新朋友甚至新机会的地方。

V2EX is a community of developers, designers and creative people.

© 2021 V2EX