性能分析系列

一、開發者的自測利器-Hprof命令

1、示例演示

例子程序：

<code class="hljs java has-numbering" style="display: block; padding: 0px; background-color: transparent; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-top-left-radius: 0px; border-top-right-radius: 0px; border-bottom-right-radius: 0px; border-bottom-left-radius: 0px; word-wrap: normal; background-position: initial initial; background-repeat: initial initial;"><span class="hljs-javadoc" style="color: rgb(136, 0, 0); box-sizing: border-box;">/**
 * PROJECT_NAME: test
 * DATE:         16/7/22
 * CREATE BY:    chao.cheng
 **/</span>
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span> <span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">HProfTest</span> {</span>
    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">void</span> <span class="hljs-title" style="box-sizing: border-box;">slowMethod</span>() {
        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">try</span> {
            Thread.sleep(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1000</span>);
        } <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">catch</span> (Exception e) {
            e.printStackTrace();
        }
    }

    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">void</span> <span class="hljs-title" style="box-sizing: border-box;">slowerMethod</span>() {
        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">try</span> {
            Thread.sleep(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10000</span>);
        } <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">catch</span> (Exception e) {
            e.printStackTrace();
        }
    }

    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">static</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">void</span> <span class="hljs-title" style="box-sizing: border-box;">main</span>(String[] args) {
        HProfTest test = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> HProfTest();
        test.slowerMethod();
        test.slowMethod();
    }
}</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li><li style="box-sizing: border-box; padding: 0px 5px;">27</li><li style="box-sizing: border-box; padding: 0px 5px;">28</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li><li style="box-sizing: border-box; padding: 0px 5px;">27</li><li style="box-sizing: border-box; padding: 0px 5px;">28</li></ul>

注：這是一段測試代碼通過sleep方法進行延時，在程序運行過程中很慢，我想知道到底是哪段程序影響的整體性能呢？

我在這個Java程序中，加了如下運行參數：

<code class="hljs haml has-numbering" style="display: block; padding: 0px; background-color: transparent; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-top-left-radius: 0px; border-top-right-radius: 0px; border-bottom-right-radius: 0px; border-bottom-left-radius: 0px; word-wrap: normal; background-position: initial initial; background-repeat: initial initial;">-<span class="ruby" style="box-sizing: border-box;"><span class="hljs-symbol" style="color: rgb(0, 102, 102); box-sizing: border-box;">agentlib:</span>hprof=cpu=times,interval=<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10</span>
</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">/* </span>
    times：java函數的執行時間
    hprof=cpu是針對cpu統計時間
    interval=10 採樣10次 
*/</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li></ul>

再次運行這段程序顯示如下圖：

 
這時候還發現在工程目錄裏面，多了一個文本文件java.hprof.txt，如下圖所示：

 
內容如下：

<code class="hljs avrasm has-numbering" style="display: block; padding: 0px; background-color: transparent; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-top-left-radius: 0px; border-top-right-radius: 0px; border-bottom-right-radius: 0px; border-bottom-left-radius: 0px; word-wrap: normal; background-position: initial initial; background-repeat: initial initial;">CPU TIME (ms) BEGIN (total = <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11542</span>) Fri Jul <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">22</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">34</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2016</span>
rank   self  accum   count trace method
   <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">86.65</span>% <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">86.65</span>%       <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">303422</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">com</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.test</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.HProfTest</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.slowerMethod</span>
   <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>  <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">8.66</span>% <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">95.31</span>%       <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">303423</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">com</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.test</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.HProfTest</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.slowMethod</span>
   <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3</span>  <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0.25</span>% <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">95.56</span>%      <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">36</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">300745</span> java<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.util</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.zip</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.ZipFile</span>.<init>
   <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">4</span>  <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0.20</span>% <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">95.76</span>%      <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">36</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">300434</span> java<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.lang</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.String</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.equals</span>
   <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">5</span>  <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0.13</span>% <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">95.89</span>%      <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">14</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">301138</span> java<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.net</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.URLStreamHandler</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.parseURL</span>
   <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">6</span>  <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0.11</span>% <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">96.01</span>%       <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">6</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">301339</span> java<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.net</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.URLClassLoader</span>$1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.run</span>
   <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">7</span>  <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0.10</span>% <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">96.10</span>%      <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">14</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">301124</span> java<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.lang</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.String</span>.<init>
   <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">8</span>  <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0.09</span>% <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">96.19</span>%    <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3407</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">300355</span> java<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.lang</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.String</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.charAt</span>
   <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">9</span>  <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0.08</span>% <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">96.27</span>%      <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">36</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">300443</span> java<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.io</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.UnixFileSystem</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.normalize</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li></ul>

注：通過上面內容可以看到，哪個類的方法執行時間長，耗費了cpu時間，一目瞭然，方便我們快速定位問題。

2、命令的具體講解 
hprof不是獨立的監控工具，它只是一個Java agent工具，它可以用在監控Java應用程序在運行時的CPU信息和堆內容，使用java -agentlib:hprof=help命令可以查看hprof的使用文檔。

通過上圖可以看到這個工具非常強大，可以統計的東西很多，上面的例子統計的是cpu時間，同樣我們還可以統計內存佔用的dump信息。 
如：-agentlib:hprof=heap,format=b,file=/test.hprof

這個hprof小工具，非常方便我們在用JUnit自測代碼的時候結合使用，既可以解決業務上的BUG，又能夠在一定程序上解決可發現的性能問題，非常實用。

二、性能排查工具-pidstat

1、示例演示 
例子程序：

<code class="hljs java has-numbering" style="display: block; padding: 0px; background-color: transparent; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-top-left-radius: 0px; border-top-right-radius: 0px; border-bottom-right-radius: 0px; border-bottom-left-radius: 0px; word-wrap: normal; background-position: initial initial; background-repeat: initial initial;"><span class="hljs-javadoc" style="color: rgb(136, 0, 0); box-sizing: border-box;">/**
 * PROJECT_NAME: test
 * DATE:         16/7/22
 * CREATE BY:    chao.cheng
 **/</span>
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span> <span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">PidstatTest</span> {</span>
    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">static</span> <span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span> <span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">PidstatTask</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">implements</span>  <span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">Runnable</span> {</span>
        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">void</span> <span class="hljs-title" style="box-sizing: border-box;">run</span>() {
            <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">while</span>(<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">true</span>) {
                <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">double</span> value = Math.random() * Math.random();
            }
        }
    }

    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">static</span> <span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span> <span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">LazyTask</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">implements</span> <span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">Runnable</span> {</span>
        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">void</span> <span class="hljs-title" style="box-sizing: border-box;">run</span>() {
            <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">try</span> {
                <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">while</span> (<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">true</span>) {
                    Thread.sleep(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1000</span>);
                }
            } <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">catch</span> (Exception e) {
                e.printStackTrace();
            }
        }
    }

    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">static</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">void</span> <span class="hljs-title" style="box-sizing: border-box;">main</span>(String[] args) {
        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> Thread(<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> PidstatTask()).start();
        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> Thread(<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> LazyTask()).start();
        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> Thread(<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> LazyTask()).start();
    }
}</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li><li style="box-sizing: border-box; padding: 0px 5px;">27</li><li style="box-sizing: border-box; padding: 0px 5px;">28</li><li style="box-sizing: border-box; padding: 0px 5px;">29</li><li style="box-sizing: border-box; padding: 0px 5px;">30</li><li style="box-sizing: border-box; padding: 0px 5px;">31</li><li style="box-sizing: border-box; padding: 0px 5px;">32</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li><li style="box-sizing: border-box; padding: 0px 5px;">27</li><li style="box-sizing: border-box; padding: 0px 5px;">28</li><li style="box-sizing: border-box; padding: 0px 5px;">29</li><li style="box-sizing: border-box; padding: 0px 5px;">30</li><li style="box-sizing: border-box; padding: 0px 5px;">31</li><li style="box-sizing: border-box; padding: 0px 5px;">32</li></ul>

注：這是一段測試用的java程序，將其運行起來。

在命令行輸入：

<code class="hljs haml has-numbering" style="display: block; padding: 0px; background-color: transparent; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-top-left-radius: 0px; border-top-right-radius: 0px; border-bottom-right-radius: 0px; border-bottom-left-radius: 0px; word-wrap: normal; background-position: initial initial; background-repeat: initial initial;">pidstat -p 843 1 3 -u -t
<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">/* </span>
-<span class="ruby" style="box-sizing: border-box;">u：代表對cpu使用率的監控
</span>參數1 3：表示每秒採樣一次，一共三次
-<span class="ruby" style="box-sizing: border-box;">t：將監控級別細化到線程 
</span>*/</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li></ul>

運行命令顯示如下圖所示： 
 
注：其實中TID就是線程ID，%usr表示用戶線程使用率，從圖中可以看到855這個線程佔用cpu非常的高。

再輸入如下命令：

<code class="hljs bash has-numbering" style="display: block; padding: 0px; background-color: transparent; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-top-left-radius: 0px; border-top-right-radius: 0px; border-bottom-right-radius: 0px; border-bottom-left-radius: 0px; word-wrap: normal; background-position: initial initial; background-repeat: initial initial;">jstack <span class="hljs-operator" style="box-sizing: border-box;">-l</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">843</span> > /tmp/testlog.txt</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

查看testlog.txt顯示如下部分內容：

 
注：我們關注的是日誌文件的NID這個字段，它對應的就是我們上面說的TID，NID是TID的16進製表示，將上面的十進制855轉換成十六進制爲357，在日誌中進行搜索看到如下內容：

<code class="hljs avrasm has-numbering" style="display: block; padding: 0px; background-color: transparent; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-top-left-radius: 0px; border-top-right-radius: 0px; border-bottom-right-radius: 0px; border-bottom-left-radius: 0px; word-wrap: normal; background-position: initial initial; background-repeat: initial initial;"><span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"Thread-0"</span> prio=<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10</span> tid=<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0x00007f7d90103800</span> nid=<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0x357</span> runnable [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0x00007f7d943d5000</span>]
   java<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.lang</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.Thread</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.State</span>: RUNNABLE
    at PidstatTest$PidstatTask<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.run</span>(PidstatTest<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.java</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">13</span>)
    at java<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.lang</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.Thread</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.run</span>(Thread<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.java</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">722</span>)

   Locked ownable synchronizers:
    - None</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li></ul>

以此可以推斷出有性能瓶頸的程序點。

2、pidstat具體命令詳解 
pidstat是一個功能非常強大的性能監測工具，他是Sysstat的組件之一，可以從http://sebastien.godard.pagesperso-orange.fr/download.html 進行下載，下載後可以通過./configure等命令進行安裝，這個命令的強大之處在於不僅可以監控進程的性能情況，也可以監控線程的性能情況。

pidstat監控cpu常用顯示字段內容如下：

<code class="hljs perl has-numbering" style="display: block; padding: 0px; background-color: transparent; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-top-left-radius: 0px; border-top-right-radius: 0px; border-bottom-right-radius: 0px; border-bottom-left-radius: 0px; word-wrap: normal; background-position: initial initial; background-repeat: initial initial;"><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>、PID - 被監控的任務的進程號
<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>、<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">%usr</span> - 當在用戶層執行（應用程序）時這個任務的cpu使用率，和 nice 優先級無關。注意這個字段計算的cpu時間不包括在虛擬處理器中花去的時間。
<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3</span>、<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">%system</span> - 這個任務在系統層使用時的cpu使用率。
<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">4</span>、<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">%guest</span> - 任務花費在虛擬機上的cpu使用率（運行在虛擬處理器）。
<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">5</span>、<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">%CPU</span> - 任務總的cpu使用率。在SMP環境（多處理器）中，如果在命令行中輸入-I參數的話，cpu使用率會除以你的cpu數量。
<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">6</span>、CPU - 正在運行這個任務的處理器編號。
<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">7</span>、Command - 這個任務的命令名稱。</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li></ul>

pidstat監控io常用的字段顯示內容如下：

<code class="hljs  has-numbering" style="display: block; padding: 0px; background-color: transparent; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-top-left-radius: 0px; border-top-right-radius: 0px; border-bottom-right-radius: 0px; border-bottom-left-radius: 0px; word-wrap: normal; background-position: initial initial; background-repeat: initial initial;">1、kB_rd/s - 任務從硬盤上的讀取速度（kb）
2、kB_wr/s - 任務向硬盤中的寫入速度（kb）
3、kB_ccwr/s - 任務寫入磁盤被取消的速率（kb）</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li></ul>

三、一個內存溢出案例分析

1、內存溢出現象 
系統共有8臺服務器，每次隨機只有一臺服務器報java.lang.OutOfMemoryError: GC overhead limit exceeded錯誤，然後接着就報內存溢出錯誤java.lang.OutOfMemoryError: Java heap space。

2、理論支撐 
我們先解釋一下什麼是GC overhead limit exceeded錯誤。

GC overhead limt exceed檢查是Hotspot VM 1.6定義的一個策略，通過統計GC時間來預測是否要OOM了，提前拋出異常，防止OOM發生。Sun 官方對此的定義是：“並行/併發回收器在GC回收時間過長時會拋出OutOfMemroyError。過長的定義是，超過98%的時間用來做GC並且回收了不到2%的堆內存。用來避免內存過小造成應用不能正常工作。

可以看到當堆中的對象無法被收回的時候，就提前遇警報出這樣的錯誤，此時內存並沒有溢出，這個特性在JDK中是默認添加的。

3、DUMP文件分析 
將dump文件導入VisualVM工具中，如下圖所示：

通過上圖可以看出類結構圖中，最佔用內存的是char[]，LinkedHashMap和String三項。但是這三項的實例數並沒有佔滿，看樣子不會內存溢出，怎麼才能具體分析呢？原因就在於GC overhead limt exceed，這個錯並不會在內存真正溢出纔會報，所以通過dump文件，我們只能自己去判斷分析，哪些項有可能會造成溢出，我們進入char[]項具體來看，會發現裏面有很多hessian的url字符被緩存，通過排除程序可以看到由於底層中間件程序爲了提高“性能”，將每次調用的url都緩存起來，不用每次都生成，但沒有相應緩存釋放操作，於是造成了大量字符對象長期持有從而報錯，在此就不截圖來具體看代碼，涉及一些公司信息。

4、問題解決方案 
* 可以添加JVM的啓動參數來去掉提前報警限制：-XX:-UseGCOverheadLimit，於其讓應用每次都提前報警，還不如讓暴風雨來的更猛些，直接內存溢出，因爲服務器是集羣，其中一臺掛掉不會影響線上正常交易，同時也方便我們通過日誌來排錯。

• 通過排查程序，檢查系統是否有使用大內存的代碼不釋放或死循環。