最近線上的一個tomcat總是無緣無故崩潰，tomcat日誌裏又沒有報任何錯誤，於是調出JVM的崩潰日誌查看，一般崩潰日誌在啓動目錄下，比如tomcat的bin目錄下，但是如果你用自己寫的腳本啓動的tomcat，則這個日誌可能就在你放腳本的目錄下。
#
# A fatal error has been detected by the Java Runtime Environment:#
# SIGSEGV (0xb) at pc=0xfe572980, pid=2383, tid=108 // SIGSEGV (0xb)這種問題一般是調用JNI或者使用了JNI的第三方庫造成的，我們可以查'UNIX信號表"，可以查到這個是因爲非法訪問了內存或者是訪問了無效指針造成的。
#
# JRE version: Java(TM) SE Runtime Environment (7.0_80-b15) (build 1.7.0_80-b15)
# Java VM: Java HotSpot(TM) Server VM (24.80-b11 mixed mode solaris-sparc )
# Problematic frame:
# V [libjvm.so+0x572980] void Par_MarkFromRootsClosure::scan_oops_in_oop(HeapWord*)+0x8 // 這裏V表示一種frame type,這裏看到是垃圾回收時出現錯誤,這裏表示是執行libjvm造成的
#
# Core dump written. Default location: /glodon/deploy/core or core.2383 //表示崩潰的core文件的位置，實際上沒有找到，由於是用這個位置的腳本啓動的，所以core文件生在這裏
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
#
--------------- T H R E A D ---------------
Current thread (0x00501000): GCTaskThread [stack: 0x2e600000,0x2e680000] [id=108] // 這裏說明是垃圾回收線程執行時崩潰
siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR), si_addr=0x0000000c
Registers:
G1=0x0000000f G2=0x00000001 G3=0xffffffff G4=0x00000000
G5=0x00000010 G6=0x00000000 G7=0xfa0b5200 Y=0x00000000
O0=0x00000022 O1=0x00000379 O2=0x00000000 O3=0x00000022
O4=0x00000021 O5=0x00000000 O6=0x2e67f880 O7=0xfe572cac
L0=0x00003ffe L1=0x00003fff L2=0x004e5df0 L3=0x00000001
L4=0x004e5dac L5=0x0000037a L6=0x00000000 L7=0x00504358
I0=0x2e67fa84 I1=0x53a5a710 I2=0x00000005 I3=0x006b2d38
I4=0x006b2d38 I5=0x00000000 I6=0x2e67f928 I7=0xfe572968
PC=0xfe572980 nPC=0xfe572984
Top of Stack: (sp=0x2e67f880)
0x2e67f880: 00003ffe 00003fff 004e5df0 00000001
0x2e67f890: 004e5dac 0000037a 00000000 00504358
0x2e67f8a0: 2e67fa84 53a5a710 00000005 006b2d38
0x2e67f8b0: 006b2d38 00000000 2e67f928 fe572968
0x2e67f8c0: fee63d24 0013c800 ff2352ec 00000000
此處省略N行......................................
I3=0x006b2d38 is an unknown value
I4=0x006b2d38 is an unknown value
I5=0x00000000 is an unknown value
I6=0x2e67f928 is an unknown value
I7=0xfe572968: JVM_GetClassDeclaredMethods+0x2998ec in /usr/jdk/instances/jdk1.7.0/jre/lib/sparc/server/libjvm.so at 0xfe000000
Stack: [0x2e600000,0x2e680000], sp=0x2e67f880, free space=510k //當前棧還剩餘510K，夠用
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x572980] void Par_MarkFromRootsClosure::scan_oops_in_oop(HeapWord*)+0x8
V [libjvm.so+0x572970] bool Par_MarkFromRootsClosure::do_bit(unsigned)+0xb0
V [libjvm.so+0x3ac368] bool BitMap::iterate(BitMapClosure*,unsigned,unsigned)+0x84
V [libjvm.so+0x5663f8] void CMSConcMarkingTask::do_scan_and_mark(int,CompactibleFreeListSpace*)+0x2b0
V [libjvm.so+0x565e5c] void CMSConcMarkingTask::work(unsigned)+0xc4
V [libjvm.so+0xbb7d38] void YieldingFlexibleGangWorker::loop()+0xd0
V [libjvm.so+0xa04a00] java_start+0x338
--------------- P R O C E S S ---------------
Java Threads: ( => current thread )
0x02852c00 JavaThread "qtp28852019-947" [_thread_blocked, id=1053, stack(0x24a80000,0x24b00000)]
0x052a5000 JavaThread "qtp28852019-940" [_thread_blocked, id=1046, stack(0x17c80000,0x17d00000)]
0x04e23800 JavaThread "qtp28852019-931" [_thread_blocked, id=1037, stack(0x17e80000,0x17f00000)]
0x02035c00 JavaThread "http-bio-8080-exec-111" daemon [_thread_in_vm, id=1036, stack(0x18180000,0x18200000)]
0x04e16c00 JavaThread "http-bio-8080-exec-110" daemon [_thread_blocked, id=1035, stack(0x18580000,0x18600000)]
此處省略N行......................................
0x009cbc00 JavaThread "Finalizer" daemon [_thread_blocked, id=113, stack(0x2df80000,0x2e000000)]
0x009ca400 JavaThread "Reference Handler" daemon [_thread_blocked, id=112, stack(0x2e080000,0x2e100000)]
0x00028c00 JavaThread "main" [_thread_in_native, id=2, stack(0xfde80000,0xfdf00000)]
Other Threads:
0x009c7400 VMThread [stack: 0x2e180000,0x2e200000] [id=111]
0x009f3800 WatcherThread [stack: 0x2d980000,0x2da00000] [id=119]
=>0x00501000 (exited) GCTaskThread [stack: 0x2e600000,0x2e680000] [id=108]
VM state:not at safepoint (normal execution) // 不在安全點，說明這個時候JVM執行的代碼不在for循環結束或者方法結束的地方，也恰好說明調用的是JNI的東西，因爲JVM調用JNI的時候不會等待其執行到安全點，關於安全點的問題，這裏不討論，大家可以查其他資料。
VM Mutex/Monitor currently owned by a thread: None
Heap //崩潰瞬間還是回收回來了，因此基本排除是因爲JVM內存崩潰引起的問題。
par new generation total 235968K, used 21333K [0x36400000, 0x46400000, 0x46400000)
eden space 209792K, 8% used [0x36400000, 0x376654c0, 0x430e0000)
from space 26176K, 9% used [0x430e0000, 0x433501a0, 0x44a70000)
to space 26176K, 0% used [0x44a70000, 0x44a70000, 0x46400000)
concurrent mark-sweep generation total 474160K, used 436253K [0x46400000, 0x6330c000, 0xb6400000) // cms垃圾回收的老生代情況
concurrent-mark-sweep perm gen total 262144K, used 141342K [0xb6400000, 0xc6400000, 0xf6400000) // cms永久代情況
Card table byte_map: [0x35c00000,0x36202000] byte_map_base: 0x35a4e000
Polling page: 0xff3f6000
Code Cache [0xfbc00000, 0xfd4f8000, 0xfdc00000)
total_blobs=7341 nmethods=7026 adapters=247 free_code_cache=7377Kb largest_free_block=7400896
Compilation events (10 events):
Event: 14607.071 Thread 0x009ee400 8453 org.apache.coyote.http11.InternalInputBuffer::fill (178 bytes)
Event: 14607.077 Thread 0x009ee400 nmethod 8453 0xfc1ef488 code [0xfc1ef5a0, 0xfc1ef7c4]
Event: 14612.564 Thread 0x009ecc00 8454 % gboat2.base.core.web.MetadataSupportStrategy::getOperaList @ 146 (230 bytes)
Event: 14612.599 Thread 0x009ecc00 nmethod 8454% 0xfd4e05c8 code [0xfd4e07c0, 0xfd4e11d8]
Event: 14612.745 Thread 0x009ee400 8455 gboat2.base.core.web.MetadataSupportStrategy::getOperaList (230 bytes)
Event: 14612.797 Thread 0x009ee400 nmethod 8455 0xfd4ee208 code [0xfd4ee620, 0xfd4ef700]
Event: 14626.459 Thread 0x009ecc00 8456 com.sun.tools.javac.tree.TreeInfo::hasConstructors (34 bytes)
Event: 14626.462 Thread 0x009ecc00 nmethod 8456 0xfd0fec08 code [0xfd0fed00, 0xfd0fee10]
Event: 14643.412 Thread 0x009ee400 8457 com.itextpdf.text.pdf.PdfReader::readArray (71 bytes)
Event: 14643.424 Thread 0x009ee400 nmethod 8457 0xfd4dde88 code [0xfd4de000, 0xfd4de504]
GC Heap History (10 events): // 最後10次垃圾回收事件
Event: 14626.265 GC heap before // 這個是事件發生的時間戳，虛擬機（tomcat)啓動以來的秒數
{Heap before GC invocations=803 (full 3): //這裏的full 3表示從tomcat開始到現在一共進行了3次full類型的垃圾回收；“Heap before GC invocations=803”這個表示在當前垃圾回收之前，進行了多少次垃圾回收工作，這裏表示進行了803次的垃圾回收
par new generation total 235968K, used 215681K [0x36400000, 0x46400000, 0x46400000) //新生代使用約256M（不到256，因爲有一個survivor不能用）
eden space 209792K, 100% used [0x36400000, 0x430e0000, 0x430e0000)//約209M
from space 26176K, 22% used [0x44a70000, 0x450305d8, 0x46400000) //約26M，這個地址現在是from，下個階段將變成to
to space 26176K, 0% used [0x430e0000, 0x430e0000, 0x44a70000) //約26M
concurrent mark-sweep generation total 474160K, used 434490K [0x46400000, 0x6330c000, 0xb6400000) //老生代併發清理，總共約463M，用去424M，剩餘約39M
concurrent-mark-sweep perm gen total 262144K, used 141295K [0xb6400000, 0xc6400000, 0xf6400000) //永久代，總共約256M，用掉約141M，剩餘約115M
Event: 14626.282 GC heap after // 和before比較起來，垃圾回收消耗時間27毫秒
Heap after GC invocations=804 (full 3): // 回收
par new generation total 235968K, used 3183K [0x36400000, 0x46400000, 0x46400000)
eden space 209792K, 0% used [0x36400000, 0x36400000, 0x430e0000)
from space 26176K, 12% used [0x430e0000, 0x433fbd30, 0x44a70000)
to space 26176K, 0% used [0x44a70000, 0x44a70000, 0x46400000)
concurrent mark-sweep generation total 474160K, used 435308K [0x46400000, 0x6330c000, 0xb6400000) // 剩餘38M
concurrent-mark-sweep perm gen total 262144K, used 141295K [0xb6400000, 0xc6400000, 0xf6400000) // 不變
}
此處省略N行......................................
Event: 14692.050 GC heap before //相隔25秒
{Heap before GC invocations=807 (full 3): // 滿了
par new generation total 235968K, used 214960K [0x36400000, 0x46400000, 0x46400000)
eden space 209792K, 99% used [0x36400000, 0x430dfbe8, 0x430e0000)
from space 26176K, 19% used [0x44a70000, 0x44f7c738, 0x46400000)
to space 26176K, 0% used [0x430e0000, 0x430e0000, 0x44a70000)
concurrent mark-sweep generation total 474160K, used 435958K [0x46400000, 0x6330c000, 0xb6400000) // 不變
concurrent-mark-sweep perm gen total 262144K, used 141342K [0xb6400000, 0xc6400000, 0xf6400000) // 增加27K
Event: 14692.067 GC heap after
Heap after GC invocations=808 (full 3): // 回收
par new generation total 235968K, used 2496K [0x36400000, 0x46400000, 0x46400000)
eden space 209792K, 0% used [0x36400000, 0x36400000, 0x430e0000)
from space 26176K, 9% used [0x430e0000, 0x433501a0, 0x44a70000)
to space 26176K, 0% used [0x44a70000, 0x44a70000, 0x46400000)
concurrent mark-sweep generation total 474160K, used 436253K [0x46400000, 0x6330c000, 0xb6400000) // 增加295K，剩餘37M
concurrent-mark-sweep perm gen total 262144K, used 141342K [0xb6400000, 0xc6400000, 0xf6400000) // 不變120802K
}
Deoptimization events (10 events): // 非優化事件
Event: 14023.027 Thread 0x0382c400 Uncommon trap: reason=null_check action=make_not_entrant pc=0xfd3f6188
此處省略N行...................................... method=org.apache.struts2.jasper.compiler.TagLibraryInfoImpl.parseTLD(Lorg/apache/struts2/jasper/JspCompilationContext;Ljava/lang/String;Ljava/io/InputStream;Ljava/net/URL;)V @ 573
Internal exceptions (10 events): // 內部異常
Event: 14691.724 Thread 0x02035c00 Threw 0x41055a00 at /HUDSON/workspace/7u-2-build-solaris-sparc/jdk7u80/2329/hotspot/src/share/vm/prims/jvm.cpp:1319
Event: 14691.724 Thread 0x02035c00 Threw 0x410577a0 at /HUDSON/workspace/7u-2-build-solaris-sparc/jdk7u80/2329/hotspot/src/share/vm/prims/jvm.cpp:1319
此處省略N行......................................
Events (10 events):
Event: 14692.045 Executing VM operation: GenCollectForAllocation
Event: 14692.068 Executing VM operation: GenCollectForAllocation done
Event: 14692.068 Executing VM operation: CMS_Initial_Mark // 初始標記
Event: 14692.092 Executing VM operation: CMS_Initial_Mark done //
Event: 14692.093 Executing VM operation: RevokeBias
Event: 14692.096 Executing VM operation: RevokeBias done
Event: 14692.097 Executing VM operation: RevokeBias
Event: 14692.098 Executing VM operation: RevokeBias done
Event: 14692.099 Executing VM operation: RevokeBias
Event: 14692.101 Executing VM operation: RevokeBias done
Dynamic libraries:
0x00010000 /usr/jdk/instances/jdk1.7.0/jre/bin/java
0xff370000 /usr/lib/libthread.so.1
此處省略N行......................................
0x36220000 /usr/jdk/instances/jdk1.7.0/jre/lib/sparc/libjpeg.so
VM Arguments:
jvm_args: -Djava.util.logging.config.file=/GBP/apache-tomcat-7.0.57/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xms512m -Xmx2048m -Xmn256m -XX:NewSize=256m -XX:MaxNewSize=256m -XX:PermSize=256m -XX:MaxPermSize=1024m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+UnlockDiagnosticVMOptions -XX:+UnsyncloadClass -XX:ErrorFile=/var/log/java/java_error%p.log -Djava.endorsed.dirs=/GBP/apache-tomcat-7.0.57/endorsed -Dcatalina.base=/GBP/apache-tomcat-7.0.57 -Dcatalina.home=/GBP/apache-tomcat-7.0.57 -Djava.io.tmpdir=/GBP/apache-tomcat-7.0.57/temp
java_command: org.apache.catalina.startup.Bootstrap start
Launcher Type: SUN_STANDARD
Environment Variables:
JAVA_HOME=/usr/java
JRE_HOME=/usr/java/jre
CLASSPATH=:/usr/java/lib:/usr/java/bin:/usr/java/jre/lib
PATH=/usr/sbin:/usr/bin:/usr/java/bin:/usr/java/jre/bin:/GBP/apache-tomcat-7.0.57/bin:/usr/sfw/bin:/opt/csw/bin:/usr/ccs/bin:/usr/openwin/bin:/usr/dt/bin:/usr/platform/sun4v/sbin:/opt/sun/bin:/opt/SUNWexplo/bin:/opt/SUNWsneep/bin:/opt/CTEact/bin
LD_LIBRARY_PATH=/usr/lib:/usr/local/lib:/usr/openwin/lib
SHELL=/usr/bin/bash
Signal Handlers:
SIGSEGV: [libjvm.so+0xba1e58], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
SIGBUS: [libjvm.so+0xba1e58], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
SIGFPE: [libjvm.so+0x2257f4], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
SIGPIPE: [libjvm.so+0x2257f4], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
SIGXFSZ: [libjvm.so+0x2257f4], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
SIGILL: [libjvm.so+0x2257f4], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
SIGUSR1: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGUSR2: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGQUIT: [libjvm.so+0xa08414], sa_mask[0]=0xffbffeff, sa_flags=0x00000004
SIGHUP: SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGINT: SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGTERM: [libjvm.so+0xa08414], sa_mask[0]=0xffbffeff, sa_flags=0x00000004
SIG39: [libjvm.so+0xa0d058], sa_mask[0]=0x00000000, sa_flags=0x00000008
SIG40: [libjvm.so+0x2257f4], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
--------------- S Y S T E M ---------------
OS: Oracle Solaris 10 8/11 s10s_u10wos_17b SPARC
Copyright (c) 1983, 2011, Oracle and/or its affiliates. All rights reserved.
Assembled 23 August 2011
uname:SunOS 5.10 Generic_147440-01 sun4v
(T2 libthread)
rlimit: STACK 8192k, CORE infinity, NOFILE 65536, AS infinity
load average:1.18 1.11 1.05
CPU:total 256 v9, popc, vis1, vis2, vis3, blk_init, cbcond, sun4v, niagara_plus
Memory: 8k page, physical 267911168k(133769880k free)
vm_info: Java HotSpot(TM) Server VM (24.80-b11) for solaris-sparc JRE (1.7.0_80-b15), built on Apr 10 2015 18:48:54 by "java_re" with Sun Studio 12u1
time: Thu Mar 3 16:18:34 2016
elapsed time: 14704 seconds
上oracle官方查看並沒有人提出這個bug，因此判斷有可能是因爲java調用JNI庫造成的內存錯誤，盤了下項目中使用的技術，並且查看了tomcat打出的業務日誌，發現在調用jmagick庫的時候程序一直報錯，這是個上傳圖片打水印的組件，調用了動態庫造成的問題。剛好這個水印功能不需要，則關閉了這個功能，不再代用打水印的API，觀察了一週，系統再也沒有崩潰過。
當然了問題的排除過程，並沒有這麼順利，由於之前對JVM的崩潰日誌接觸不多，查了不少資料。這裏強調一句，網上的JVM崩潰場景不同，每個人遇到的問題也不一樣，注重思路就行。比如我這個文章能提醒大家調用JNI可能會出現JVM崩潰的問題，大家按照這個思路排查了也許能解決自己的問題，這種問題一般排除的時候需要好多天，但是真正動手結局也許一分鐘都用不到。
