收到IO佔用高告警
系統信息：Linux version 2.6.32-696.18.7.1.el6.ucloud.x86_64 (root@59c188f3c79d) (gcc version 4.4.6 20120305
(Red Hat 4.4.6-5) (GCC) ) #1 SMP Fri Jan 5 16:48:58 CST 2018
1、到機器上看到io使用率忽高，同時iowait也高
$ iostat -d -x 1
Linux 2.6.32-696.18.7.1.el6.ucloud.x86_64 (bd-prod-acc-web0) 07/08/2019 _x86_64_ (2 CPU)
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await r_await w_await svctm %util
vda 0.00 1.17 0.00 0.88 0.06 16.40 18.58 0.01 16.88 0.90 16.92 4.06 0.36
vdb 0.00 0.86 0.00 1.03 0.00 15.14 14.68 0.01 11.48 1.79 11.48 4.56 0.47
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await r_await w_await svctm %util
vda 0.00 0.00 0.00 1.00 0.00 72.00 72.00 1.00 2387.00 0.00 2387.00 1000.00 100.00
vdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.06 0.00 0.00 0.00 0.00 100.00
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await r_await w_await svctm %util
vda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 100.00
vdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.00 0.00 0.00 0.00 0.00 100.00
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await r_await w_await svctm %util
vda 0.00 0.00 0.00 1.00 0.00 8.00 8.00 0.51 2258.00 0.00 2258.00 513.00 51.30
vdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.00 0.00 0.00 0.00 0.00 100.00
2、找到佔用高的進程
$ sudo iotop
Total DISK READ: 0.00 B/s | Total DISK WRITE: 51.21 K/s
TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
810 be/3 root 0.00 B/s 0.00 B/s 0.00 % 99.99 % [jbd2/vdb-8] #這裏看到是系統進程
2614 be/4 root 0.00 B/s 0.00 B/s 0.00 % 99.99 % sadc -F -L -S DISK -S DISK 1 1 -
412 be/3 root 0.00 B/s 3.94 K/s 0.00 % 99.99 % [jbd2/vda1-8]
3、jdb2進程佔用IO資源較高，jdb2進程是文件系統ext4的寫日誌進程，佔用高是內核一個bug
4、（1）升級內核
$ sudo yum update kernel
或
(2)通過修改文件系統的提交次數來降低
commit=60
或（3）重啓
Wait IO problem can be caused by several reasons, the basic road-map to find out is which process is "eating" your CPU first and then determine why. The main cause are those background processes with "D" status code which means "Uninterruptiable sleep". But the those processes with "D+" which means "Uninterruptible sleep foreground process" will generally not cause the serious problem as those background processes. In this example, the cause of Wait IO is the File System Journal, so the configuration of file system is the cause of the problem.
By Quentin Sherman Xue - CEO (http://www.chileoffshore.com/en/interesting-articles/126-linux-wait-io-problem)