硬盤性能指標測試

一.雲硬盤的性能衡量指標

雲硬盤的性能指標一般通過以下幾個指標進行衡量

• IOPS：每秒的讀寫次數，單位爲次（計數）。存儲設備的底層驅動類型決定了不同的IOPS
  總IOPS：每秒執行的I/O操作總次數
  隨機讀IOPS：每秒指定的隨機讀I/O操作的平均次數
  隨機寫IOPS 每秒指定的隨機寫I/O操作的平均次數
  順序讀IOPS 每秒指定的順序讀I/O操作的平均次數
  順序寫IOPS 每秒指定的順序寫I/O操作的平均次數
• 吞吐量：每秒的讀寫數據量，單位爲MB/S
  吞吐量市值單位時間內可以成功傳輸的數據數量。
  如果需要部署大量順序讀寫的應用，典型場景比如hadoop離線計算型業務，需要關注吞吐量

• 時延：IO操作的發送時間到接收確認所經過的時間，單位爲秒
  如果應用對時延比較敏感，比如數據庫（過高時延會導致應用性能下降或報錯），建議使用SSD存儲

  二.具體參數說明

[root@host-10-0-1-36 ~]# fio --bs=4k --ioengine=libaio --iodepth=1 --direct=1 --rw=read --time_based --runtime=600  --refill_buffers --norandommap --randrepeat=0 --group_reporting --name=fio-read --size=100G --filename=/dev/vdb 
fio-read: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.1
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=9784KiB/s,w=0KiB/s][r=2446,w=0 IOPS][eta 00m:00s]
fio-read: (groupid=0, jobs=1): err= 0: pid=22004: Wed Oct 10 21:35:42 2018
   read: IOPS=2593, BW=10.1MiB/s (10.6MB/s)(6078MiB/600001msec)
    slat (usec): min=4, max=1532, avg=11.98, stdev= 8.10
    clat (nsec): min=1021, max=66079k, avg=370367.39, stdev=395393.29
     lat (usec): min=44, max=66086, avg=382.88, stdev=399.21
    clat percentiles (usec):
     |  1.00th=[   42],  5.00th=[   44], 10.00th=[   45], 20.00th=[   46],
     | 30.00th=[   48], 40.00th=[   51], 50.00th=[  383], 60.00th=[  578],
     | 70.00th=[  644], 80.00th=[  701], 90.00th=[  783], 95.00th=[  865],
     | 99.00th=[  988], 99.50th=[ 1020], 99.90th=[ 1336], 99.95th=[ 2057],
     | 99.99th=[11338]
   bw (  KiB/s): min= 2000, max=68272, per=99.98%, avg=10370.55, stdev=8123.28, samples=1199
   iops        : min=  500, max=17068, avg=2592.62, stdev=2030.82, samples=1199
  lat (usec)   : 2=0.01%, 20=0.01%, 50=37.97%, 100=10.63%, 250=0.39%
  lat (usec)   : 500=6.09%, 750=31.54%, 1000=12.65%
  lat (msec)   : 2=0.67%, 4=0.03%, 10=0.01%, 20=0.01%, 50=0.01%
  lat (msec)   : 100=0.01%
  cpu          : usr=1.43%, sys=5.12%, ctx=1555984, majf=0, minf=35
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=1556003,0,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=10.1MiB/s (10.6MB/s), 10.1MiB/s-10.1MiB/s (10.6MB/s-10.6MB/s), io=6078MiB (6373MB), run=600001-600001msec

Disk stats (read/write):
  vdb: ios=1555778/0, merge=0/0, ticks=570518/0, in_queue=569945, util=95.04%

參數詳情	含義
io	執行了多少M的IO
bw	平均IO帶寬
iops	IOPS
runt	線程運行時間
slat	提交延遲
clat	完成延遲
lat	響應時間
cpu	cpu利用率
IO depths	io隊列
IO submit	單個IO提交要提交的IO數
IO complete
IO issued
IO latency

三.FIO測試工具參數說明

[root@host-10-0-1-36 ~]# fio -direct=1 -iodepth=64 -rw=read -ioengine=libaio -bs=4096k -size=100G -numjobs=1 -runtime=300 -group_reporting -filename=/dev/vdb -name=Write_PPS_Testing

參數	說明
-direct=1	表示測試時忽略I/O緩存，數據直寫
-iodepth=128	表示使用AIO時，同時發出I/O數的上限爲128
-rw=randwrite	表示測試時的讀寫策略爲隨機寫（random writes）。作其它測試時可以設置爲：randread（隨機讀random reads）read（順序讀sequential reads） write（順序寫sequential writes）randrw（混合隨機讀寫mixed random reads and writes）
-ioengine=libaio	表示測試方式爲libaio（Linux AIO，異步I/O）。應用程序使用I/O通常有兩種方式： 同步：同步的I/O一次只能發出一個I/O請求，等待內核完成才返回。這樣對於單個線程iodepth總是小於1，但是可以透過多個線程併發執行來解決。通常會用16−32根線程同時工作將iodepth塞滿。異步：異步的I/O通常使用libaio這樣的方式一次提交一批I/O請求，然後等待一批的完成，減少交互的次數，會更有效率。
-bs=4k	表示單次I/O的塊文件大小爲4 KB。未指定該參數時的默認大小也是4 KB，測試IOPS時，建議將bs設置爲一個比較小的值，如本示例中的4k。測試吞吐量時，建議將bs設置爲一個較大的值，如本示例中的1024k
-size=1G	表示測試文件大小爲1 GiB
-numjobs=1	表示測試線程數爲1
-runtime=1000	表示測試時間爲1000秒。如果未配置，則持續將前述-size指定大小的文件，以每次-bs值爲分塊大小寫完
-group_reporting	表示測試結果裏彙總每個進程的統計信息，而非以不同job彙總展示信息
-filename=iotest	指定測試文件的名稱，比如iotest。測試裸盤可以獲得真實的硬盤性能，但直接測試裸盤會破壞文件系統結構，請在測試前提前做好數據備份
-name=Rand_Write_Testing	表示測試任務名稱爲Rand_Write_Testing，可以隨意設定

四.測試結果

這裏我們以4K的數據塊測試隨機讀和隨機寫來測試最大的IOPS，4M的塊測試順序讀和順序寫來測試最大的吞吐量

• 測試4K塊的隨機寫

[root@host-10-0-1-36 ~]# fio --bs=4k --ioengine=libaio --iodepth=128 --direct=1 --rw=randwrite --time_based --runtime=300  --refill_buffers --norandommap --randrepeat=0 --group_reporting --name=fio-write --size=1G --filename=/dev/vdb 
fio-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128
fio-3.1
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=3996KiB/s][r=0,w=999 IOPS][eta 00m:00s]
fio-write: (groupid=0, jobs=1): err= 0: pid=22050: Wed Oct 10 22:29:32 2018
  write: IOPS=2819, BW=11.0MiB/s (11.5MB/s)(3309MiB/300484msec)
    slat (usec): min=2, max=2399, avg= 9.54, stdev=10.19
    clat (usec): min=1180, max=3604.0k, avg=45387.25, stdev=168013.09
     lat (usec): min=1201, max=3604.0k, avg=45397.35, stdev=168013.57
    clat percentiles (usec):
     |  1.00th=[   1713],  5.00th=[   2212], 10.00th=[   2835],
     | 20.00th=[   4015], 30.00th=[   5211], 40.00th=[   6849],
     | 50.00th=[   8979], 60.00th=[  11994], 70.00th=[  17695],
     | 80.00th=[  33162], 90.00th=[  61604], 95.00th=[ 137364],
     | 99.00th=[ 893387], 99.50th=[1266680], 99.90th=[2122318],
     | 99.95th=[2432697], 99.99th=[2969568]
   bw (  KiB/s): min=    8, max=49120, per=100.00%, avg=11603.11, stdev=10950.79, samples=584
   iops        : min=    2, max=12280, avg=2900.77, stdev=2737.69, samples=584
  lat (msec)   : 2=3.16%, 4=16.75%, 10=33.83%, 20=18.49%, 50=14.82%
  lat (msec)   : 100=6.56%, 250=2.86%, 500=1.49%, 750=0.70%, 1000=0.53%
  lat (msec)   : 2000=0.69%, >=2000=0.12%
  cpu          : usr=1.84%, sys=4.16%, ctx=323739, majf=0, minf=28
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwt: total=0,847133,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=128

Run status group 0 (all jobs):
  WRITE: bw=11.0MiB/s (11.5MB/s), 11.0MiB/s-11.0MiB/s (11.5MB/s-11.5MB/s), io=3309MiB (3470MB), run=300484-300484msec

Disk stats (read/write):
  vdb: ios=91/847074, merge=0/0, ticks=3/38321566, in_queue=38360706, util=100.00%

我們可以看到4K的數據塊，隨機寫的最大IOPS爲：12280

• 測試4K的隨機讀

[root@host-10-0-1-36 ~]# fio --bs=4k --ioengine=libaio --iodepth=128 --direct=1 --rw=randread --time_based --runtime=300  --refill_buffers --norandommap --randrepeat=0 --group_reporting --name=fio-write --size=1G --filename=/dev/vdb 
fio-write: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128
fio-3.1
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=87.1MiB/s,w=0KiB/s][r=22.3k,w=0 IOPS][eta 00m:00s]
fio-write: (groupid=0, jobs=1): err= 0: pid=22055: Wed Oct 10 22:51:54 2018
   read: IOPS=22.3k, BW=87.0MiB/s (91.2MB/s)(25.5GiB/300004msec)
    slat (usec): min=2, max=8626, avg= 7.91, stdev=11.85
    clat (usec): min=36, max=71810, avg=5735.17, stdev=1405.20
     lat (usec): min=45, max=71826, avg=5743.59, stdev=1405.30
    clat percentiles (usec):
     |  1.00th=[ 1958],  5.00th=[ 3556], 10.00th=[ 4424], 20.00th=[ 5145],
     | 30.00th=[ 5407], 40.00th=[ 5604], 50.00th=[ 5735], 60.00th=[ 5866],
     | 70.00th=[ 6063], 80.00th=[ 6259], 90.00th=[ 6783], 95.00th=[ 7504],
     | 99.00th=[10290], 99.50th=[12256], 99.90th=[16712], 99.95th=[18482],
     | 99.99th=[25035]
   bw (  KiB/s): min=74872, max=93240, per=100.00%, avg=89121.48, stdev=2687.55, samples=600
   iops        : min=18718, max=23310, avg=22280.35, stdev=671.89, samples=600
  lat (usec)   : 50=0.01%, 100=0.01%, 250=0.01%, 500=0.02%, 750=0.06%
  lat (usec)   : 1000=0.12%
  lat (msec)   : 2=0.86%, 4=6.02%, 10=91.77%, 20=1.11%, 50=0.03%
  lat (msec)   : 100=0.01%
  cpu          : usr=6.56%, sys=28.57%, ctx=3473136, majf=0, minf=160
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwt: total=6683408,0,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=128

Run status group 0 (all jobs):
   READ: bw=87.0MiB/s (91.2MB/s), 87.0MiB/s-87.0MiB/s (91.2MB/s-91.2MB/s), io=25.5GiB (27.4GB), run=300004-300004msec

Disk stats (read/write):
  vdb: ios=6680955/0, merge=0/0, ticks=37981396/0, in_queue=37983491, util=100.00%

我們可以看到4K的數據塊，隨機讀的最大IOPS爲：23310

• 測試4M的順序寫

[root@host-10-0-1-36 ~]# fio -direct=1 -iodepth=64 -rw=write -ioengine=libaio -bs=4096k -size=100G -numjobs=1 -runtime=300 -group_reporting -filename=/dev/vdb -name=Write_PPS_Testing
Write_PPS_Testing: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=64
fio-3.1
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=108MiB/s][r=0,w=27 IOPS][eta 00m:00s]
Write_PPS_Testing: (groupid=0, jobs=1): err= 0: pid=22098: Wed Oct 10 23:20:07 2018
  write: IOPS=28, BW=115MiB/s (120MB/s)(33.7GiB/300466msec)
    slat (usec): min=564, max=2554.1k, avg=34767.87, stdev=65067.32
    clat (msec): min=400, max=12179, avg=2188.04, stdev=1037.46
     lat (msec): min=473, max=12231, avg=2222.81, stdev=1047.45
    clat percentiles (msec):
     |  1.00th=[ 1435],  5.00th=[ 1586], 10.00th=[ 1653], 20.00th=[ 1754],
     | 30.00th=[ 1838], 40.00th=[ 1921], 50.00th=[ 2005], 60.00th=[ 2089],
     | 70.00th=[ 2198], 80.00th=[ 2333], 90.00th=[ 2534], 95.00th=[ 2802],
     | 99.00th=[ 8490], 99.50th=[ 9731], 99.90th=[11745], 99.95th=[12013],
     | 99.99th=[12147]
   bw (  KiB/s): min= 8192, max=196608, per=100.00%, avg=120954.04, stdev=31456.39, samples=580
   iops        : min=    2, max=   48, avg=29.53, stdev= 7.68, samples=580
  lat (msec)   : 500=0.02%, 750=0.03%, 1000=0.13%, 2000=49.25%, >=2000=50.56%
  cpu          : usr=1.04%, sys=2.16%, ctx=6387, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=0.4%, >=64=99.3%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwt: total=0,8627,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=115MiB/s (120MB/s), 115MiB/s-115MiB/s (120MB/s-120MB/s), io=33.7GiB (36.2GB), run=300466-300466msec

Disk stats (read/write):
  vdb: ios=91/77596, merge=0/0, ticks=6/37685137, in_queue=37717833, util=99.99%

我們可以看到4M的數據塊，順序寫的最大吞吐量爲：196M

• 測試4M的順序讀

[root@host-10-0-1-36 ~]# fio -direct=1 -iodepth=64 -rw=read -ioengine=libaio -bs=4096k -size=100G -numjobs=1 -runtime=300 -group_reporting -filename=/dev/vdb -name=Write_PPS_Testing
Write_PPS_Testing: (g=0): rw=read, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=64
fio-3.1
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=312MiB/s,w=0KiB/s][r=78,w=0 IOPS][eta 00m:00s]
Write_PPS_Testing: (groupid=0, jobs=1): err= 0: pid=22103: Wed Oct 10 23:26:31 2018
   read: IOPS=35, BW=142MiB/s (149MB/s)(41.6GiB/300201msec)
    slat (usec): min=469, max=95190, avg=28166.40, stdev=15966.53
    clat (msec): min=185, max=4070, avg=1772.98, stdev=551.21
     lat (msec): min=205, max=4107, avg=1801.15, stdev=558.25
    clat percentiles (msec):
     |  1.00th=[  518],  5.00th=[  634], 10.00th=[  751], 20.00th=[ 1536],
     | 30.00th=[ 1770], 40.00th=[ 1854], 50.00th=[ 1921], 60.00th=[ 1989],
     | 70.00th=[ 2039], 80.00th=[ 2140], 90.00th=[ 2299], 95.00th=[ 2433],
     | 99.00th=[ 2802], 99.50th=[ 2970], 99.90th=[ 3473], 99.95th=[ 3641],
     | 99.99th=[ 3775]
   bw (  KiB/s): min=106496, max=466944, per=99.97%, avg=145216.28, stdev=60136.24, samples=597
   iops        : min=   26, max=  114, avg=35.45, stdev=14.68, samples=597
  lat (msec)   : 250=0.10%, 500=0.66%, 750=9.21%, 1000=7.02%, 2000=46.51%
  lat (msec)   : >=2000=36.51%
  cpu          : usr=0.05%, sys=2.61%, ctx=10959, majf=0, minf=672
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=0.3%, >=64=99.4%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwt: total=10646,0,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=142MiB/s (149MB/s), 142MiB/s-142MiB/s (149MB/s-149MB/s), io=41.6GiB (44.7GB), run=300201-300201msec

Disk stats (read/write):
  vdb: ios=95740/0, merge=0/0, ticks=37360788/0, in_queue=37382226, util=100.00%

我們可以看到4M的數據塊，順序讀的最大吞吐量爲：466M