@msg7086 #17 oracle 的，看上去可能稳定一些，我调研一下

@ruidoBlanco #19 已经是 1160 了，是红帽 7 的最新的内核了，再升就得用 elrepo 或者楼上说的 uek 内核了，领导估计不大支持升级

hang 住时候的内核日志截取了部分
Aug 15 09:33:53 node16 kernel: INFO: task jbd2/dm-2-8:1839 blocked for more than 120 seconds.
Aug 15 09:33:53 node16 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug 15 09:33:53 node16 kernel: jbd2/dm-2-8 D ffff8e7efea1acc0 0 1839 2 0x00000000
Aug 15 09:33:53 node16 kernel: Call Trace:
Aug 15 09:33:53 node16 kernel: [<ffffffff8e8d2ba0>] ? task_rq_unlock+0x20/0x20
Aug 15 09:33:53 node16 kernel: [<ffffffff8ef87169>] schedule+0x29/0x70
Aug 15 09:33:53 node16 kernel: [<ffffffffc118433c>] jbd2_journal_commit_transaction+0x23c/0x19c0 [jbd2]
Aug 15 09:33:53 node16 kernel: [<ffffffff8e8e111e>] ? account_entity_dequeue+0xae/0xd0
Aug 15 09:33:53 node16 kernel: [<ffffffff8e8e48bc>] ? dequeue_entity+0x11c/0x5c0
Aug 15 09:33:53 node16 kernel: [<ffffffff8e8e5ec1>] ? put_prev_entity+0x31/0x400
Aug 15 09:33:53 node16 kernel: [<ffffffff8e82b59e>] ? __switch_to+0xce/0x580
Aug 15 09:33:53 node16 kernel: [<ffffffff8e8c6d10>] ? wake_up_atomic_t+0x30/0x30
Aug 15 09:33:53 node16 kernel: [<ffffffff8ef86c8f>] ? __schedule+0x3af/0x860
Aug 15 09:33:53 node16 kernel: [<ffffffff8e8adf0e>] ? try_to_del_timer_sync+0x5e/0x90
Aug 15 09:33:53 node16 kernel: [<ffffffffc118af89>] kjournald2+0xc9/0x260 [jbd2]
Aug 15 09:33:53 node16 kernel: [<ffffffff8e8c6d10>] ? wake_up_atomic_t+0x30/0x30
Aug 15 09:33:53 node16 kernel: [<ffffffffc118aec0>] ? commit_timeout+0x10/0x10 [jbd2]
Aug 15 09:33:53 node16 kernel: [<ffffffff8e8c5c21>] kthread+0xd1/0xe0
Aug 15 09:33:53 node16 kernel: [<ffffffff8e8c5b50>] ? insert_kthread_work+0x40/0x40
Aug 15 09:33:53 node16 kernel: [<ffffffff8ef93ddd>] ret_from_fork_nospec_begin+0x7/0x21
Aug 15 09:33:53 node16 kernel: [<ffffffff8e8c5b50>] ? insert_kthread_work+0x40/0x40
Aug 15 09:33:53 node16 kernel: INFO: task containerd:225811 blocked for more than 120 seconds.
Aug 15 09:33:53 node16 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug 15 09:33:53 node16 kernel: containerd D ffff8e7efef1acc0 0 225811 1 0x00000080
Aug 15 09:33:53 node16 kernel: Call Trace:
Aug 15 09:33:53 node16 kernel: [<ffffffff8ef87169>] schedule+0x29/0x70
Aug 15 09:33:53 node16 kernel: [<ffffffffc1181085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Aug 15 09:33:53 node16 kernel: [<ffffffff8e8c6d10>] ? wake_up_atomic_t+0x30/0x30
Aug 15 09:33:53 node16 kernel: [<ffffffffc1181378>] add_transaction_credits+0x278/0x310 [jbd2]
Aug 15 09:33:53 node16 kernel: [<ffffffff8ea2830b>] ? __kmalloc+0x1eb/0x230
Aug 15 09:33:53 node16 kernel: [<ffffffffc11dd8c4>] ? ext4_htree_store_dirent+0x34/0x120 [ext4]
Aug 15 09:33:53 node16 kernel: [<ffffffffc1181601>] start_this_handle+0x1a1/0x430 [jbd2]
Aug 15 09:33:53 node16 kernel: [<ffffffff8ea287c2>] ? kmem_cache_alloc+0x1c2/0x1f0
Aug 15 09:33:53 node16 kernel: [<ffffffffc1181ab3>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Aug 15 09:33:53 node16 kernel: [<ffffffffc11eb0ba>] ? ext4_dirty_inode+0x2a/0x60 [ext4]
Aug 15 09:33:53 node16 kernel: [<ffffffffc1217759>] __ext4_journal_start_sb+0x69/0xe0 [ext4]
Aug 15 09:33:53 node16 kernel: [<ffffffffc11eb0ba>] ext4_dirty_inode+0x2a/0x60 [ext4]
Aug 15 09:33:53 node16 kernel: [<ffffffff8ea7f00d>] __mark_inode_dirty+0x15d/0x270
Aug 15 09:33:53 node16 kernel: [<ffffffff8ea6b8e9>] update_time+0x89/0xd0
Aug 15 09:33:53 node16 kernel: [<ffffffff8ea6bdfa>] touch_atime+0x10a/0x220
Aug 15 09:33:53 node16 kernel: [<ffffffff8ea63694>] iterate_dir+0xe4/0x130
Aug 15 09:33:53 node16 kernel: [<ffffffff8ea63c8c>] SyS_getdents64+0x9c/0x120
Aug 15 09:33:53 node16 kernel: [<ffffffff8ea63900>] ? fillonedir+0x110/0x110
Aug 15 09:33:53 node16 kernel: [<ffffffff8ef93f92>] system_call_fastpath+0x25/0x2a
Aug 15 09:33:53 node16 kernel: INFO: task containerd:2700571 blocked for more than 120 seconds.
Aug 15 09:33:53 node16 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug 15 09:33:53 node16 kernel: containerd D ffff8e3eff79acc0 0 2700571 1 0x00000080
Aug 15 09:33:53 node16 kernel: Call Trace:
Aug 15 09:33:53 node16 kernel: [<ffffffff8ef87169>] schedule+0x29/0x70
Aug 15 09:33:53 node16 kernel: [<ffffffffc1181085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Aug 15 09:33:53 node16 kernel: [<ffffffff8e8c6d10>] ? wake_up_atomic_t+0x30/0x30
Aug 15 09:33:53 node16 kernel: [<ffffffffc1181378>] add_transaction_credits+0x278/0x310 [jbd2]
Aug 15 09:33:53 node16 kernel: [<ffffffff8ef86c8f>] ? __schedule+0x3af/0x860
Aug 15 09:33:53 node16 kernel: [<ffffffffc1181601>] start_this_handle+0x1a1/0x430 [jbd2]
Aug 15 09:33:53 node16 kernel: [<ffffffff8ef87169>] ? schedule+0x29/0x70
Aug 15 09:33:53 node16 kernel: [<ffffffff8ef84c51>] ? schedule_timeout+0x221/0x2d0
Aug 15 09:33:53 node16 kernel: [<ffffffff8ea287c2>] ? kmem_cache_alloc+0x1c2/0x1f0
Aug 15 09:33:53 node16 kernel: [<ffffffffc1181ab3>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Aug 15 09:33:53 node16 kernel: [<ffffffffc11eb0ba>] ? ext4_dirty_inode+0x2a/0x60 [ext4]
Aug 15 09:33:53 node16 kernel: [<ffffffffc1217759>] __ext4_journal_start_sb+0x69/0xe0 [ext4]
Aug 15 09:33:53 node16 kernel: [<ffffffffc11eb0ba>] ext4_dirty_inode+0x2a/0x60 [ext4]
Aug 15 09:33:53 node16 kernel: [<ffffffff8ea7f00d>] __mark_inode_dirty+0x15d/0x270
Aug 15 09:33:53 node16 kernel: [<ffffffff8ea6b8e9>] update_time+0x89/0xd0
Aug 15 09:33:53 node16 kernel: [<ffffffff8eb8cfe4>] ? __radix_tree_lookup+0x84/0xf0
Aug 15 09:33:53 node16 kernel: [<ffffffff8ea6b9d0>] file_update_time+0xa0/0xf0
Aug 15 09:33:53 node16 kernel: [<ffffffff8e9c03d8>] __generic_file_aio_write+0x198/0x400
Aug 15 09:33:53 node16 kernel: [<ffffffff8e9c0699>] generic_file_aio_write+0x59/0xa0
Aug 15 09:33:53 node16 kernel: [<ffffffffc11de5c8>] ext4_file_write+0x348/0x600 [ext4]
Aug 15 09:33:53 node16 kernel: [<ffffffff8ea010bc>] ? page_add_file_rmap+0x8c/0xc0
Aug 15 09:33:53 node16 kernel: [<ffffffff8e9f339e>] ? do_numa_page+0x1be/0x250
Aug 15 09:33:53 node16 kernel: [<ffffffff8ea4d063>] do_sync_write+0x93/0xe0
Aug 15 09:33:53 node16 kernel: [<ffffffff8ea4db50>] vfs_write+0xc0/0x1f0
Aug 15 09:33:53 node16 kernel: [<ffffffff8ea4eaf2>] SyS_pwrite64+0x92/0xc0
Aug 15 09:33:53 node16 kernel: [<ffffffff8ef93f92>] system_call_fastpath+0x25/0x2a
Aug 15 09:33:53 node16 kernel: INFO: task dcgm-exporter:68381 blocked for more than 120 seconds.
Aug 15 09:33:53 node16 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug 15 09:33:53 node16 kernel: dcgm-exporter D ffff8e3eff81acc0 0 68381 57193 0x00000080
Aug 15 09:33:53 node16 kernel: Call Trace:
Aug 15 09:33:53 node16 kernel: [<ffffffff8ef87169>] schedule+0x29/0x70
Aug 15 09:33:53 node16 kernel: [<ffffffffc1181085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Aug 15 09:33:53 node16 kernel: [<ffffffff8e8c6d10>] ? wake_up_atomic_t+0x30/0x30
Aug 15 09:33:53 node16 kernel: [<ffffffffc1181378>] add_transaction_credits+0x278/0x310 [jbd2]
Aug 15 09:33:53 node16 kernel: [<ffffffffc1181601>] start_this_handle+0x1a1/0x430 [jbd2]
Aug 15 09:33:53 node16 kernel: [<ffffffff8e8e8a43>] ? load_balance+0x1a3/0xa10
Aug 15 09:33:53 node16 kernel: [<ffffffff8ea287c2>] ? kmem_cache_alloc+0x1c2/0x1f0
Aug 15 09:33:53 node16 kernel: [<ffffffffc1181ab3>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Aug 15 09:33:53 node16 kernel: [<ffffffffc11eb0ba>] ? ext4_dirty_inode+0x2a/0x60 [ext4]
Aug 15 09:33:53 node16 kernel: [<ffffffffc1217759>] __ext4_journal_start_sb+0x69/0xe0 [ext4]
Aug 15 09:33:53 node16 kernel: [<ffffffffc11eb0ba>] ext4_dirty_inode+0x2a/0x60 [ext4]
Aug 15 09:33:53 node16 kernel: [<ffffffff8ea7f00d>] __mark_inode_dirty+0x15d/0x270
Aug 15 09:33:53 node16 kernel: [<ffffffff8ea6b8e9>] update_time+0x89/0xd0
Aug 15 09:33:53 node16 kernel: [<ffffffff8ea6b9d0>] file_update_time+0xa0/0xf0
Aug 15 09:33:53 node16 kernel: [<ffffffff8e9c03d8>] __generic_file_aio_write+0x198/0x400
Aug 15 09:33:53 node16 kernel: [<ffffffff8e9c0699>] generic_file_aio_write+0x59/0xa0
Aug 15 09:33:53 node16 kernel: [<ffffffffc11de5c8>] ext4_file_write+0x348/0x600 [ext4]
Aug 15 09:33:53 node16 kernel: [<ffffffff8e9f339e>] ? do_numa_page+0x1be/0x250
Aug 15 09:33:53 node16 kernel: [<ffffffff8ea4d063>] do_sync_write+0x93/0xe0
Aug 15 09:33:53 node16 kernel: [<ffffffff8ea4db50>] vfs_write+0xc0/0x1f0
Aug 15 09:33:53 node16 kernel: [<ffffffff8ea4e92f>] SyS_write+0x7f/0xf0
Aug 15 09:33:53 node16 kernel: [<ffffffff8ef93f92>] system_call_fastpath+0x25/0x2a
Aug 15 09:33:53 node16 kernel: INFO: task gcs_server:21032 blocked for more than 120 seconds.

@ruidoBlanco 一会就补上老哥
@lrvy 一言难尽，反正就是能用红帽的包，但是人家不管我们出的问题

@Hormazed #8 生产系统不是非常敢用 elrepo 提供的内核，尽管我也想用新的
@Hormazed #9 我们的内核版本已经比这个新了，是 1160
@JackSlowFcck #11 周期太长了，机器托管到别的机房了，现在有的业务是单点，只能先再用上了
@Kumo31 #10 大概率我们要 3.10 一直用上去了，我们还有信创的机器是 4.19 的内核，也反馈过 bug ，厂商也没管

@hefish #5 只是怀疑，因为 reboot 了就好了
@liuchao719 #6 没人敢用新系统，没人背得起锅，大家都用老的

@iyiluo #1 当时看带外硬盘信息是正常的，而且 reboot 后恢复了，运行 1 天了也正常了
@barrysj #2 有，从监控数据看到当时有一块盘的 I/O 掉下去了，突然没有读/写了，进程也都 hung 了，iowait 当时没有
硬盘是 3 块 SAS 的 SSD 组的 RAID

@midknight #8 redis 不花钱的有么？我们要 redis 5 和 kafka 1.1 的

@GeekGao 是的，我会一些 py ，但是领导还是希望转 go 了，不行就 Java
@KaynW JAVA 系的也是吗
@eedwinhei 大佬，Redis 这个是 Redis 商业版吧，这个好像得花钱？我看了 bitnami 有 cluster 和 sentinel 的，但是用的人好像很少，而且 Redis 版本太高了

@ETiV 托管的机器，没有这么好使的功能呀

@Ericcccccccc 不是，随机的，大部分时候没有同时出现

定时踢掉重连

我们（国企）三个全中，版本都不带差的

@qq135449773
@EastLord
@zsc8917zsc
@defunct9
@FrankAdler
@Aliencn
@willx12123
@opengps
@xueling 我在想要不要鼓弄领导买一套阿里云/腾讯云的服务（纯测试用不跑实际业务），然后把监控告警也买了，然后对着补齐？

@Aliencn 国企，我们是纯私有化部署的，不能上云

@RangerWolf 的确是，现在最难受的就是告警配阈值，然后业务变来变去，阈值也改来改去，弄错了就背锅吧
@idontnowhat2say 就像 2 楼说的，异常检测，还有根因分析，我觉得都是比较有用的，现在业务多变，数据多变，而且系统越来越复杂，传统运维配告警阈值越来越不好使了

strace 一下卡主的进程，看看在干啥

生日快乐老哥

@Nitromethane 可以水平，Python 多进程和部分 Java 多线程