最近客戶那處理了一個case，case中客戶現場的mysql經常長時間hold在那裏。
如下是客戶現場的 INNODB MONITOR OUTPUT，我們來分析下：
首先是
InnoDB: Warning: a long semaphore wait:
--Thread 139979394471680 has waited at log0log.ic line 320 for 896.00 seconds the semaphore:
Mutex at 0x243d84d8 created file log0log.c line 771, lock var 0
waiters flag 0
wait has ended
InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info:
InnoDB: Pending preads 0, pwrites 0
這個warning顯示innodb工作線程139979394471680在準備寫日誌時有個長達896s的信號量等待。
lock var 0 表示該鎖的狀態爲空閒，1表示該鎖已被持有。
waiters flag 0 表示當前沒有正在等待該鎖的其他線程，若有1個那麼該值爲1。
wait has ended 表示 “mutex is already free for grabs but os has not yet scheduled thread”。
所以綜上可以初步認爲，運行mysql的機器負載太高了，內部進程和線程數太多了，導致系統頻繁地進行 context switch。
接着我們看下當前mysql的負載
-----------------
BACKGROUND THREAD
-----------------
srv_master_thread loops: 66488 1_second, 66092 sleeps, 6594 10_second, 6365 background, 6365 flush
srv_master_thread log flush and writes: 72129
1_second的數量和sleeps 的數量基本爲 1：1，且 10_second 的數量基本爲1_second的 1/10。
說明mysql的負載其實較小。
接着我們看下當前mysql統計的信號量信息
----------
SEMAPHORES
----------
OS WAIT ARRAY INFO: reservation count 91544822, signal count 67071774
。。。
Mutex spin waits 836769568, rounds 11105882188, OS waits 80039908
RW-shared spins 26704, rounds 383132, OS waits 11585
RW-excl spins 90035, rounds 556267, OS waits 13349
Spin rounds per wait: 13.27 mutex, 14.35 RW-shared, 6.18 RW-excl
可以看到 Mutex spin waits 836769568, rounds 11105882188, OS waits 80039908
解釋下這3個值：
Mutex spin waits 836769568 is the number of times a thread tried to get a mutex and it wasn’t available, so it waited in a spin-wait.
rounds 11105882188 is the number of times threads looped in the spin-wait cycle, checking the mutex.
OS waits 80039908 is the number of times the thread gave up spin-waiting and went to sleep instead.
這3個值都太高了，特別是 OS waits。 說明操作系統層面的鎖等待現象非常嚴重。
再接着看下I/O
--------
FILE I/O
--------
I/O thread 0 state: waiting for completed aio requests (insert buffer thread)
I/O thread 1 state: waiting for completed aio requests (log thread)
。。。
Pending normal aio reads: 0 [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] ,
ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0
Pending flushes (fsync) log: 1; buffer pool: 0
9612 OS file reads, 2628919 OS file writes, 277446 OS fsyncs
0.00 reads/s, 0 avg bytes/read, 0.17 writes/s, 0.04 fsyncs/s
fsyncs/s 的值非常的小，說明刷盤的頻率很低，也驗證了mysql負載較低的結論。
最後看下內存使用
----------------------
BUFFER POOL AND MEMORY
----------------------
Total memory allocated 21978152960; in additional pool allocated 0
Dictionary memory allocated 700351
Buffer pool size 1310719
Free buffers 1236849
Database pages 73175
Old database pages 26991
Modified db pages 23191
。。。
該mysql的 innodb buffer pool size 配置的是20G，與Buffer pool size 1310719相符。
Buffer pool size 表示的是block的個數，每個block大小爲16k。
Free buffers表示空閒的block數，可以看出佔了絕大部分。
所以說明數據庫當前操作的數據量其實很小。
綜上，我們可以確認，mysql本身負載不高，甚至較低，但運行mysql的機器的負載非常的高，導致mysql的工作線程無法被調度到。
參考：
http://www.mysqlperformanceblog.com/2011/09/02/understand-innodb-spin-waits-win-a-percona-live-ticket/
http://www.mysqlperformanceblog.com/2006/07/17/show-innodb-status-walk-through/
轉載請註明轉自高孝鑫的blog。
如下是客戶現場的 INNODB MONITOR OUTPUT，我們來分析下：
首先是
InnoDB: Warning: a long semaphore wait:
--Thread 139979394471680 has waited at log0log.ic line 320 for 896.00 seconds the semaphore:
Mutex at 0x243d84d8 created file log0log.c line 771, lock var 0
waiters flag 0
wait has ended
InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info:
InnoDB: Pending preads 0, pwrites 0
這個warning顯示innodb工作線程139979394471680在準備寫日誌時有個長達896s的信號量等待。
lock var 0 表示該鎖的狀態爲空閒，1表示該鎖已被持有。
waiters flag 0 表示當前沒有正在等待該鎖的其他線程，若有1個那麼該值爲1。
wait has ended 表示 “mutex is already free for grabs but os has not yet scheduled thread”。
所以綜上可以初步認爲，運行mysql的機器負載太高了，內部進程和線程數太多了，導致系統頻繁地進行 context switch。
接着我們看下當前mysql的負載
-----------------
BACKGROUND THREAD
-----------------
srv_master_thread loops: 66488 1_second, 66092 sleeps, 6594 10_second, 6365 background, 6365 flush
srv_master_thread log flush and writes: 72129
1_second的數量和sleeps 的數量基本爲 1：1，且 10_second 的數量基本爲1_second的 1/10。
說明mysql的負載其實較小。
接着我們看下當前mysql統計的信號量信息
----------
SEMAPHORES
----------
OS WAIT ARRAY INFO: reservation count 91544822, signal count 67071774
。。。
Mutex spin waits 836769568, rounds 11105882188, OS waits 80039908
RW-shared spins 26704, rounds 383132, OS waits 11585
RW-excl spins 90035, rounds 556267, OS waits 13349
Spin rounds per wait: 13.27 mutex, 14.35 RW-shared, 6.18 RW-excl
可以看到 Mutex spin waits 836769568, rounds 11105882188, OS waits 80039908
解釋下這3個值：
Mutex spin waits 836769568 is the number of times a thread tried to get a mutex and it wasn’t available, so it waited in a spin-wait.
rounds 11105882188 is the number of times threads looped in the spin-wait cycle, checking the mutex.
OS waits 80039908 is the number of times the thread gave up spin-waiting and went to sleep instead.
這3個值都太高了，特別是 OS waits。 說明操作系統層面的鎖等待現象非常嚴重。
再接着看下I/O
--------
FILE I/O
--------
I/O thread 0 state: waiting for completed aio requests (insert buffer thread)
I/O thread 1 state: waiting for completed aio requests (log thread)
。。。
Pending normal aio reads: 0 [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] ,
ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0
Pending flushes (fsync) log: 1; buffer pool: 0
9612 OS file reads, 2628919 OS file writes, 277446 OS fsyncs
0.00 reads/s, 0 avg bytes/read, 0.17 writes/s, 0.04 fsyncs/s
fsyncs/s 的值非常的小，說明刷盤的頻率很低，也驗證了mysql負載較低的結論。
最後看下內存使用
----------------------
BUFFER POOL AND MEMORY
----------------------
Total memory allocated 21978152960; in additional pool allocated 0
Dictionary memory allocated 700351
Buffer pool size 1310719
Free buffers 1236849
Database pages 73175
Old database pages 26991
Modified db pages 23191
。。。
該mysql的 innodb buffer pool size 配置的是20G，與Buffer pool size 1310719相符。
Buffer pool size 表示的是block的個數，每個block大小爲16k。
Free buffers表示空閒的block數，可以看出佔了絕大部分。
所以說明數據庫當前操作的數據量其實很小。
綜上，我們可以確認，mysql本身負載不高，甚至較低，但運行mysql的機器的負載非常的高，導致mysql的工作線程無法被調度到。
參考：
http://www.mysqlperformanceblog.com/2011/09/02/understand-innodb-spin-waits-win-a-percona-live-ticket/
http://www.mysqlperformanceblog.com/2006/07/17/show-innodb-status-walk-through/
轉載請註明轉自高孝鑫的blog。