問題描述：
最近經常收到Hbase的FGC報警，但看gc日誌如下：
2019-12-24T14:33:12.332+0800: 9446402.403: [Full GC (System.gc()) 2019-12-24T14:33:12.332+0800: 9446402.404: [CMS: 9746276K->9749022K(44040192K), 1.8478848 secs] 9762182K->9749022K(45927680K), [Metaspace: 55550K->55550K(57344K)], 1.8536369 secs] [Times: user=1.86 sys=0.00, real=1.86 secs]
2019-12-24T14:33:27.973+0800: 9446418.045: [Full GC (System.gc()) 2019-12-24T14:33:27.974+0800: 9446418.045: [CMS: 9749022K->9741506K(44040192K), 2.9043246 secs] 10310705K->9741506K(45927680K), [Metaspace: 55550K->55550K(57344K)], 2.9099166 secs] [Times: user=2.92 sys=0.00, real=2.91 secs]
2019-12-24T14:33:42.051+0800: 9446432.122: [Full GC (System.gc()) 2019-12-24T14:33:42.051+0800: 9446432.122: [CMS: 9741506K->9734271K(44040192K), 1.7758858 secs] 10289122K->9734271K(45927680K), [Metaspace: 55550K->55550K(57344K)], 1.7813927 secs] [Times: user=1.78 sys=0.00, real=1.78 secs]
2019-12-24T14:34:01.748+0800: 9446451.820: [Full GC (System.gc()) 2019-12-24T14:34:01.749+0800: 9446451.820: [CMS: 9734271K->9887663K(44040192K), 3.0752197 secs] 10641723K->9887663K(45927680K), [Metaspace: 55550K->55550K(57344K)], 3.0813152 secs] [Times: user=3.08 sys=0.00, real=3.08 secs]
2019-12-24T14:34:05.479+0800: 9446455.550: [Full GC (System.gc()) 2019-12-24T14:34:05.479+0800: 9446455.550: [CMS: 9887663K->9868928K(44040192K), 1.6589594 secs] 10027519K->9868928K(45927680K), [Metaspace: 55550K->55550K(57344K)], 1.6645320 secs] [Times: user=1.67 sys=0.00, real=1.66 secs]
2019-12-24T14:34:07.893+0800: 9446457.964: [Full GC (System.gc()) 2019-12-24T14:34:07.893+0800: 9446457.965: [CMS: 9868928K->9869164K(44040192K), 1.8132781 secs] 10012277K->9869164K(45927680K), [Metaspace: 55550K->55550K(57344K)], 1.8188013 secs] [Times: user=1.82 sys=0.00, real=1.82 secs]8 secs]
hbase日誌如下：
2019-12-24 14:34:12,950 WARN [hconnection-0x239ab184-shared--pool4-t2855938] client.AsyncProcess: #79, table=auditor_default, attempt=4/4 failed=96ops, last exception: java.io.IOException: com.google.protobuf.ServiceException: java.lang.OutOfMemoryError: Direct buffer memory on hbase007,16020,1565365569654, tracking started Tue Dec 24 14:34:01 CST 2019; not retrying 96 - final failure
問題分析：
由於由於系統大量使用了NIO中的DirectByteBuffer（hbase regionserver設置了堆外內存DirectoryBuffer爲20G)，需要定期清理本地內存。
DirectByteBuffer通過內存映射，使java進程直接訪問與文件相關聯的虛擬地址空間，減少了文件拷貝帶來的開銷，提高了文件讀取效率。這一塊虛擬地址空間並不是分配在jvm堆上，而是分配在native堆上。yong gc不能回收這部分空間，只能通過Full gc順帶進行回收。
從上面的日誌可以看出每次Full gc，所花時間將近2秒，這2秒是停機回收，系統會有卡頓存在，用戶體驗不太好。
解決方法：
希望Full gc有更少的停機時間，可以啓用-XX:+ExplicitGCInvokesConcurrent或-XX:+ExplicitGCInvokesConcurrentAndUnloadsClasses選項。