V2EX = way to explore
V2EX 是一个关于分享和探索的地方
现在注册
已注册用户请  登录
Distributions
Ubuntu
Fedora
CentOS
中文资源站
网易开源镜像站
zhoudaiyu
V2EX  ›  Linux

一个关于 Linux 进程的奇怪问题

  •  2
     
  •   zhoudaiyu · 127 天前 · 1609 次点击
    这是一个创建于 127 天前的主题,其中的信息可能已经有所发展或是发生改变。
    是这样的,今天发现 k8s 有 2 台 master 节点 CPU 使用率有尖峰( k8s 1.11 Linux 3.10 1.13.1 ),由于没有当时进程状态快照,只能用 top 每秒抓一次进程扔日志里。果然过了一段时间,发现了几个叫 docker-containerd-current 的进程 cpu 飙升导致 CPU 使用率高达 90%,但是我们通过 ps pgrep 等等手段都查不到这个进程了,所以考虑是不是当时突然启动了这些进程然后又死了,后来想从 /proc 里查有没有啥信息,奇怪的事发生了:ll -arth /proc|grep $PID 通过 pid 号对应文件夹查不到这个进程,但是 cd $PID 居然可以进到进程的文件夹里,然后我不死心,退回到 /proc 里又肉眼看了一遍文件夹,还是没找到。进到 /proc/$PID 内查看了 exe 对应的二进制文件,然后通过 ps -ef|grep $EXE_FILE 发现对应的进程启动时间还是去年。

    这里的问题就是 /proc 下面的以 pid 命名的文件夹不应该是正在运行着的进程的么?进程确实死了,通过文件名也查不到了,但是为什么可以进到一个根本不存在的文件夹呢?

    重点是图里面圈中的部分。
    24 条回复    2021-02-10 15:08:47 +08:00
    mikeguan
        1
    mikeguan   127 天前 via Android
    ls 显示目录的参数是-d
    zhoudaiyu
        2
    zhoudaiyu   127 天前 via iPhone
    @mikeguan 不用加这个参数也可以看文件夹啊
    lcdtyph
        3
    lcdtyph   127 天前 via iPhone
    ll 是什么的 alias
    zhoudaiyu
        4
    zhoudaiyu   127 天前 via iPhone
    @lcdtyph ls -l
    mikeguan
        5
    mikeguan   127 天前 via Android
    @zhoudaiyu 不加也可以,输出太多更乱
    zhoudaiyu
        6
    zhoudaiyu   127 天前 via iPhone
    @mikeguan 但是问题是查不到文件夹为啥可以进去,而且进程都死了,不应该在 proc 下了啊
    mikeguan
        7
    mikeguan   127 天前 via Android
    @zhoudaiyu 具体原因不清楚,可以用 lsof -p 14546 看下有没有结果
    zhoudaiyu
        8
    zhoudaiyu   127 天前 via iPhone
    @mikeguan 看过 没有
    Jirajine
        9
    Jirajine   127 天前 via Android
    可能是进程被隐藏了。
    zhoudaiyu
        10
    zhoudaiyu   127 天前 via iPhone
    @Jirajine 还有这种操作?那么有什么方法能看到被隐藏的进程吗?
    lcdtyph
        11
    lcdtyph   127 天前 via iPhone
    @zhoudaiyu
    你看一下现在 procfs 的 mount options 是什么
    lcdtyph
        13
    lcdtyph   127 天前 via iPhone   ❤️ 3
    @zhoudaiyu
    如果没有什么特殊的 mount options,那说明你这个不是 pid,而是某个 tid,你需要找到它的主进程再看。

    /proc/<tid> 是不会出现在 getdents 调用中,所以不会被 ls 看到
    barathrum
        14
    barathrum   127 天前
    @lcdtyph 感谢兄弟,学到一个新知识。
    zhoudaiyu
        15
    zhoudaiyu   127 天前
    @lcdtyph 老哥 我看有 mountinfo mounts mountstats 截图里是 mountstats 的输出,tid 是线程 id 吗?

    device rootfs mounted on / with fstype rootfsdevice sysfs mounted on /sys with fstype sysfs
    device proc mounted on /proc with fstype procdevice devtmpfs mounted on /dev with fstype devtmpfs
    device securityfs mounted on /sys/kernel/security with fstype securityfsdevice tmpfs mounted on /dev/shm with fstype tmpfsdevice devpts mounted on /dev/pts with fstype devptsdevice tmpfs mounted on /run with fstype tmpfs
    device tmpfs mounted on /sys/fs/cgroup with fstype tmpfsdevice cgroup mounted on /sys/fs/cgroup/systemd with fstype cgroupdevice pstore mounted on /sys/fs/pstore with fstype pstoredevice cgroup mounted on /sys/fs/cgroup/pids with fstype cgroup
    device cgroup mounted on /sys/fs/cgroup/blkio with fstype cgroupdevice cgroup mounted on /sys/fs/cgroup/perf_event with fstype cgroup
    device cgroup mounted on /sys/fs/cgroup/freezer with fstype cgroupdevice cgroup mounted on /sys/fs/cgroup/net_cls,net_prio with fstype cgroupdevice cgroup mounted on /sys/fs/cgroup/devices with fstype cgroupdevice cgroup mounted on /sys/fs/cgroup/cpu,cpuacct with fstype cgroup
    device cgroup mounted on /sys/fs/cgroup/hugetlb with fstype cgroup
    device cgroup mounted on /sys/fs/cgroup/cpuset with fstype cgroupdevice cgroup mounted on /sys/fs/cgroup/memory with fstype cgroup
    device configfs mounted on /sys/kernel/config with fstype configfsdevice /dev/mapper/vg_root-lv_root mounted on / with fstype xfsdevice selinuxfs mounted on /sys/fs/selinux with fstype selinuxfsdevice systemd-1 mounted on /proc/sys/fs/binfmt_misc with fstype autofs
    device hugetlbfs mounted on /dev/hugepages with fstype hugetlbfsdevice debugfs mounted on /sys/kernel/debug with fstype debugfs
    device mqueue mounted on /dev/mqueue with fstype mqueue
    device /dev/sda1 mounted on /boot with fstype xfsdevice /dev/mapper/vg_root-lv_var mounted on /var with fstype xfsdevice /dev/mapper/vg_root-lv_opt mounted on /opt with fstype xfsdevice /dev/mapper/vg_app-lv_DATA mounted on /DATA with fstype xfs
    device /dev/mapper/vg_root-lv_tmp mounted on /tmp with fstype xfs
    device /dev/mapper/vg_root-lv_home mounted on /home with fstype xfs
    device sunrpc mounted on /var/lib/nfs/rpc_pipefs with fstype rpc_pipefs
    device /dev/mapper/vg_root-lv_var mounted on /var/lib/docker/containers with fstype xfs
    device /dev/mapper/vg_root-lv_var mounted on /var/lib/docker/overlay2 with fstype xfs
    device tmpfs mounted on /var/lib/origin/openshift.local.volumes/pods/206f7c4c-52cd-11ea-9f39-0050569a04bf/volumes/kubernetes.io~secret/sdn-token-d66w8 with fstype tmpfs
    device tmpfs mounted on /var/lib/origin/openshift.local.volumes/pods/e3695fa7-2dce-11ea-97bb-005056a99505/volumes/kubernetes.io~secret/sdn-token-d66w8 with fstype tmpfs
    device tmpfs mounted on /var/lib/origin/openshift.local.volumes/pods/5abfd92e-2dc8-11ea-97bb-005056a99505/volumes/kubernetes.io~secret/sync-token-tr7f8 with fstype tmpfs
    device overlay mounted on /var/lib/docker/overlay2/c47caa96460e233c36b6790b1ca4ea450dee1d297f9e40bf199c6ab83bb1e690/merged with fstype overlay
    device overlay mounted on /var/lib/docker/overlay2/edb026a3c373d3ddbd064b9d8e997ccb441d56eb883f19237374bdd839cd876a/merged with fstype overlay
    device proc mounted on /run/docker/netns/default with fstype proc
    device shm mounted on /var/lib/docker/containers/6d3e50e348e7bc4608b568fd43ad63670c57f0c53de71f446ba17e8ec5e47f4e/shm with fstype tmpfs
    device shm mounted on /var/lib/docker/containers/f2c824ce425cfe15dda053af70d4fd02d64afdae9367ae78bcd6b5758338182c/shm with fstype tmpfs
    device overlay mounted on /var/lib/docker/overlay2/60fffaf1c2c9cbe0d12504a67a0b603c9bdbf8b0f6322dc30b2befa9cbb87438/merged with fstype overlay
    device overlay mounted on /var/lib/docker/overlay2/0045f1ad6a6556324b91b80892dcc4951ad32ec8dfe153b7c866b942fae74843/merged with fstype overlay
    device overlay mounted on /var/lib/docker/overlay2/ec0204696ace63e3c1eb69bd81e029c13d7c1320b16b561ddceb3c56f4ae9172/merged with fstype overlay
    device shm mounted on /var/lib/docker/containers/a774eddee040f246b9dea20a33781a1b8cb80a3bc668690d66baec33a0f56a1d/shm with fstype tmpfs
    device overlay mounted on /var/lib/docker/overlay2/cfcfc406430dc094d2b2a7084f38ea44b1eb2200282ce5fa432d46c9fee9d1f6/merged with fstype overlay
    device shm mounted on /var/lib/docker/containers/6d7a77c6a7dade24e619f63aa813962ae003e52f15dc146141af3c362564fd27/shm with fstype tmpfs
    device overlay mounted on /var/lib/docker/overlay2/52230e78851c2d66d058e1fda52b3cb8b3493a86804354d234fe64895b3aed56/merged with fstype overlay
    device tmpfs mounted on /var/lib/origin/openshift.local.volumes/pods/a684975d-9ef2-11ea-90f1-005056a9a751/volumes/kubernetes.io~secret/node-exporter-token-mttqc with fstype tmpfs
    device overlay mounted on /var/lib/docker/overlay2/c07d51282d5db3eeaa6966bdb277d34fdab0707900b34822b5bb85c6f3dddc84/merged with fstype overlay
    device shm mounted on /var/lib/docker/containers/14c0c7e261e57a3eac08de62225b5b33fe8a11d231b13e34ddbdec8231e98d24/shm with fstype tmpfs
    device overlay mounted on /var/lib/docker/overlay2/7c510eedb0ebb50ace710478ceca3037011a99a24ded9b2862d2700064782b22/merged with fstype overlay
    device overlay mounted on /var/lib/docker/overlay2/be29c4a1f9bbd28181851f27df38bf277a1984595490599db5d4297de8d0847f/merged with fstype overlay
    device tmpfs mounted on /var/lib/origin/openshift.local.volumes/pods/42a3edd4-02be-11eb-854a-005056a958e5/volumes/kubernetes.io~secret/oauth-config with fstype tmpfs
    device tmpfs mounted on /var/lib/origin/openshift.local.volumes/pods/42a3edd4-02be-11eb-854a-005056a958e5/volumes/kubernetes.io~secret/serving-cert with fstype tmpfs
    device tmpfs mounted on /var/lib/origin/openshift.local.volumes/pods/42a3edd4-02be-11eb-854a-005056a958e5/volumes/kubernetes.io~secret/default-token-mlq5l with fstype tmpfs
    device overlay mounted on /var/lib/docker/overlay2/2a50c3d9829ae039bbe3afb1108e2b131aca83febf40070b0423761aee576354/merged with fstype overlay
    device shm mounted on /var/lib/docker/containers/4b04ef977813ab3b24704d433d09d56fd069ee9103a32c5d041a3a072eab4663/shm with fstype tmpfs
    device proc mounted on /run/docker/netns/4dd48aec066f with fstype proc
    device overlay mounted on /var/lib/docker/overlay2/e1ee93593bab0e24a711c4ce871534fefe651e8be5ee11cf64f544e0b3e2e86f/merged with fstype overlay
    device binfmt_misc mounted on /proc/sys/fs/binfmt_misc with fstype binfmt_misc
    device overlay mounted on /var/lib/docker/overlay2/e04f7635399c36249c8e9b90d5619ba593b7633275d1b2c2363d3c93d0e2f55b/merged with fstype overlay
    device overlay mounted on /var/lib/docker/overlay2/0be6198515753a8a97b51d078ac64e1153325342221e8d77a956eecabd448d12/merged with fstype overlay
    device shm mounted on /var/lib/docker/containers/f54abb6462617d5910b8d41e5fa4c131be18aa59d4d227040e3f3e2ecd69d70a/shm with fstype tmpfs
    device overlay mounted on /var/lib/docker/overlay2/a177d06ddec1fb54a0b04725fabe944293ca72eb4aa0bdbf4e918f828d089466/merged with fstype overlay
    device overlay mounted on /var/lib/docker/overlay2/0af1e5fe64c765d3e0a4d574b725829a6b6e3170b929ede21cdd5b5cb97b0fc4/merged with fstype overlay
    device shm mounted on /var/lib/docker/containers/284f17f11532b9750a1f15adb3c2361796c42c4e0afc220c49c0cd394c477299/shm with fstype tmpfs
    device overlay mounted on /var/lib/docker/overlay2/ce6e3d2eb2879de645014da2be3650c7f172634131d08718304ef106a3b4ea66/merged with fstype overlay
    device tmpfs mounted on /var/lib/origin/openshift.local.volumes/pods/37a179b0-690c-11eb-b1fb-005056a9a751/volumes/kubernetes.io~secret/serving-cert with fstype tmpfs
    device tmpfs mounted on /var/lib/origin/openshift.local.volumes/pods/37a179b0-690c-11eb-b1fb-005056a9a751/volumes/kubernetes.io~secret/webconsole-token-spxpx with fstype tmpfs
    device overlay mounted on /var/lib/docker/overlay2/188242c0ee6187645f21d2f0195615af2d450337a461c4c197367bfaf794d5ed/merged with fstype overlay
    device shm mounted on /var/lib/docker/containers/5b5fb40679472f61b4b5f37caf20666291fdb38f04648710dbd23a3e952c0845/shm with fstype tmpfs
    device proc mounted on /run/docker/netns/feff0fd124b3 with fstype proc
    device overlay mounted on /var/lib/docker/overlay2/d6d7732f83a71c8c3f33a257bab98a140b5099e0ba2b1bb789c057520cf75d95/merged with fstype overlay
    device tmpfs mounted on /run/user/0 with fstype tmpfs
    zhoudaiyu
        16
    zhoudaiyu   127 天前
    @lcdtyph 错了,不是截图是是文本 🤣
    lcdtyph
        17
    lcdtyph   127 天前   ❤️ 1
    @zhoudaiyu #16
    这里没有啊,你直接输入 mount 命令就会列出所有 mounted 和 options 了

    mount | grep proc

    输出括号里面的部分就是 options
    lcdtyph
        18
    lcdtyph   127 天前   ❤️ 1
    @zhoudaiyu #16
    要找 tid 的话给 ps 加上-L 参数就行了
    ps -efL | grep 14546
    其中的 LWP 列就是 tid
    zhoudaiyu
        19
    zhoudaiyu   127 天前
    @lcdtyph mount|grep 14546 返回空,ps -efL | grep 14546 这个好使,多谢老哥,但是我还是不太明白为啥 /proc 看不到这个 14546 这个进程号的文件夹,但是可以 cd 到 14546 中来,老哥能解个惑不?
    lcdtyph
        20
    lcdtyph   127 天前 via iPhone   ❤️ 3
    @zhoudaiyu
    mount 那条命令是看 procfs 的 options 的,而且不是 grep 14546,就是 grep proc 。如果设置了 hidepid=2 那么 /proc 下就看不到其他用户的 pid 了

    你这里的 14546 不是进程号而是个线程号(tid),/proc/tid 下的内容和 /proc/pid/task/tid 下的内容是一模一样的,所以实现 procfs 的时候让这些入口不在 getdents 系统调用的结果中出现。而 ls 正是使用 getdents 实现的,所以 ls 自然看不到这些入口。

    cd 依赖的是 chdir 系统调用,所以不受影响
    howellz
        21
    howellz   127 天前
    没细看,不过 docker 不就是利用 namespace 来隔离进程和文件系统(包括 proc )的么?
    zhoudaiyu
        22
    zhoudaiyu   127 天前 via iPhone
    @lcdtyph 学习到了 十分感谢!!
    zhoudaiyu
        23
    zhoudaiyu   127 天前 via iPhone
    @howellz 但是宿主机上应该都能看到才对
    liuxu
        24
    liuxu   124 天前
    @howellz 学到了,多谢大佬,学习了


    $ man procfs
    /proc/[tid] subdirectories
    Each one of these subdirectories contains files and subdirectories exposing information about the thread with the corre‐
    sponding thread ID. The contents of these directories are the same as the corresponding /proc/[pid]/task/[tid] directo‐
    ries.
    The /proc/[tid] subdirectories are not visible when iterating through /proc with getdents(2) (and thus are not visible
    when one uses ls(1) to view the contents of /proc).


    然后用 sysdig 抓 ls /proc
    第一个命令行:$ sudo sysdig proc.name = ls
    第二个命令行:$ ls /proc
    然后第一个命令行得到:
    ...
    333156 14:54:19.153132412 4 ls (671251) > stat
    333159 14:54:19.153141644 4 ls (671251) < stat res=0 path=/proc
    333160 14:54:19.153149403 4 ls (671251) > openat
    333161 14:54:19.153159898 4 ls (671251) > fstat fd=-82524538700541
    333162 14:54:19.153161715 4 ls (671251) < fstat res=0
    333164 14:54:19.153169912 4 ls (671251) > getdents64 fd=137455746819
    333175 14:54:19.154214139 4 ls (671251) < getdents64 res=11840
    333176 14:54:19.154223263 4 ls (671251) > lstat
    333177 14:54:19.154232665 4 ls (671251) < lstat res=0 path=/proc/fb
    333178 14:54:19.154238761 4 ls (671251) > lstat
    333179 14:54:19.154242851 4 ls (671251) < lstat res=0 path=/proc/fs
    333180 14:54:19.154247207 4 ls (671251) > lstat
    333181 14:54:19.154250957 4 ls (671251) < lstat res=0 path=/proc/bus
    333182 14:54:19.154254649 4 ls (671251) > lstat
    333183 14:54:19.154257514 4 ls (671251) < lstat res=0 path=/proc/dma
    333184 14:54:19.154260911 4 ls (671251) > lstat
    333185 14:54:19.154263837 4 ls (671251) < lstat res=0 path=/proc/i8k
    ...
    关于   ·   帮助文档   ·   FAQ   ·   API   ·   我们的愿景   ·   广告投放   ·   感谢   ·   实用小工具   ·   3783 人在线   最高记录 5497   ·     Select Language
    创意工作者们的社区
    World is powered by solitude
    VERSION: 3.9.8.5 · 22ms · UTC 02:39 · PVG 10:39 · LAX 19:39 · JFK 22:39
    ♥ Do have faith in what you're doing.