本文目錄結構
線上Tomcat進程怎麼又不見了？
10月5日，作爲程序員的你正享受着難得國慶長假，突然項目經理來電，你驚悚的接起電話，電話那頭項目經理說：“那個’打羽毛球的碼農’啊，怎麼生產環境又訪問不了啦？”。你心中暗想：“窩草，怎麼服務又掛啦。 是我寫的程序有問題？還是哪位同仁又殺了我服務的進程？怎麼好端端的程序進程就沒有了呢？”。
上面這個橋段是不是很熟悉？是不是自己也經常遇到？ 遇到這樣的情況，你一般怎麼處理與分析呢：最簡單最粗暴的處理方式萬能的重啓服務；第二類是拿到應用日誌與Tomcat日誌初步分析後重啓，一般都能根據Error日誌找到原因做代碼修整；最可怕的情況是在日誌中沒有有效Error信息，是不是很懵逼？本文就根據Tomcat的啓停日誌關鍵信息，來分析Tomcat進行無故消失的3種情況。
Tomcat8.5正常啓動日誌[注意日誌解釋]
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=256M; support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support was removed in 8.0
19-Jul-2019 10:57:38.374 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version: Apache Tomcat/8.5.35
19-Jul-2019 10:57:38.376 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server built: Nov 3 2018 17:39:20 UTC
19-Jul-2019 10:57:38.376 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server number: 8.5.35.0
19-Jul-2019 10:57:38.376 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Name: Linux
19-Jul-2019 10:57:38.376 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Version: 3.10.0-957.1.3.el7.x86_64
19-Jul-2019 10:57:38.376 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Architecture: amd64
19-Jul-2019 10:57:38.376 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Java Home: /cloud/server/jdk8/jre
19-Jul-2019 10:57:38.376 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Version: 1.8.0_191-b12
19-Jul-2019 10:57:38.376 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor: Oracle Corporation
19-Jul-2019 10:57:38.377 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_BASE: /cloud/server/tomcat_crm
19-Jul-2019 10:57:38.377 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_HOME: /cloud/server/tomcat_crm
19-Jul-2019 10:57:38.377 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.config.file=/cloud/server/tomcat_crm/conf/logging.properties
19-Jul-2019 10:57:38.377 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
19-Jul-2019 10:57:38.377 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xms4096m
19-Jul-2019 10:57:38.377 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xmx4096m
19-Jul-2019 10:57:38.377 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:PermSize=256M
19-Jul-2019 10:57:38.377 INFO [main] ... /**其他JVM啓動參數日誌MaxPermSize=256m,-Dcatalina.base=/cloud/server/tomcat_crm*/
19-Jul-2019 10:57:38.545 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [...] /**localhost-startStop-1進程是Tomcat啓動和關閉的線程*/
...
19-Jul-2019 10:58:15.355 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [...] has finished in [36,809] ms /**應用部署完成日誌*/
19-Jul-2019 10:58:15.358 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8030"]
19-Jul-2019 10:58:15.378 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 36883 ms /**Tomcat服務啓動完成日誌*/
2019-07-19 10:58:07,330 [Thread-6] INFO /**業務進程日誌*/
2019-07-19 10:58:19,785 [EventThread] ERROR /**業務進程日誌*/
正常關閉Tomcat服務日誌[shutdown.sh]
正常命令關閉Tomcat服務，會有“A valid shutdown command was received via the shutdown port. Stopping the Server instance”日誌關鍵信息，如下：
19-Jul-2019 11:15:02.566 INFO [main] org.apache.catalina.core.StandardServer.await A valid shutdown command was received via the shutdown port. Stopping the Server instance. /**Tomcat收到正常停止服務的命令通知*/
19-Jul-2019 11:15:02.567 INFO [main] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["http-nio-8030"]
19-Jul-2019 11:15:02.573 INFO [main] org.apache.catalina.core.StandardService.stopInternal Stopping service [Catalina]
19-Jul-2019 11:15:02.629 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [chatbot] registered the JDBC driver [com.alibaba.druid.proxy.DruidDriver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered. /**localhost-startStop-2爲tomcat關閉服務線程*/
19-Jul-2019 11:15:02.630 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [chatbot] registered the JDBC driver [com.mysql.jdbc.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
19-Jul-2019 11:15:02.631 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [chatbot] appears to have started a thread named [Abandoned connection cleanup thread] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread: ...
19-Jul-2019 11:15:02.631 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [chatbot] appears to have started a thread named [Druid-ConnectionPool-Create-1269347162] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread: ...
19-Jul-2019 11:15:02.632 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [chatbot] appears to have started a thread named [Druid-ConnectionPool-Destroy-1269347162] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
19-Jul-2019 11:15:02.637 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [chatbot] appears to have started a thread named [MySQL Statement Cancellation Timer] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
19-Jul-2019 11:15:02.654 INFO [main] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler ["http-nio-8030"]
19-Jul-2019 11:15:02.681 INFO [main] org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler ["http-nio-8030"]
19-Jul-2019 11:15:05.387 INFO [Abandoned connection cleanup thread] org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading Illegal access: this web application instance has been stopped already. Could not load []. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
kill -9 進程號 關閉Tomcat
這種方式：什麼日誌都沒有! 什麼日誌都沒有! 什麼日誌都沒有!
tomcat的shutdownhook被觸發，執行了銷燬邏輯
Tomcat使用了關閉鉤子來完成退出過程的，在org,apache.catalina.startup.Catalina類中，可以找到這樣的代碼，Catalina類負責啓動管理其他組件的Server對象。一個名爲CatalinaShutdownHook的內部類繼承自Thread類，提供了run方法的實現，它調用server對象的stop方法，執行關閉操作，當出現如下情況的時候，Tomcat的shutdownhook會被執行:
- 情況1:應用代碼裏有地方用System.exit來退出jvm
- 情況2:系統發的信號(kill -9除外):一般來自啓動Tomcat終端shell命令進程,一旦終端進程結束(會話到期\ctrl+c\關閉窗口),都會因爲終端進程結束而觸發tomcat的shutdownhook,日誌示例如下:
19-Jul-2019 14:23:08.860 INFO [Thread-20] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["http-nio-8030"] /**shutdownhook被觸發的日誌與進程,相比於正常關閉Tomcat,進程號由main變成了Thread-N,更主要特徵是沒有'A valid shutdown command was received via the shutdown port. Stopping the Server instance'正常關閉信息*/
19-Jul-2019 14:23:08.881 INFO [Thread-20] org.apache.catalina.core.StandardService.stopInternal Stopping service [Catalina]
19-Jul-2019 14:23:08.950 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [chatbot] registered the JDBC driver [com.alibaba.druid.proxy.DruidDriver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered. /**localhost-startStop-2爲tomcat關閉服務線程*/
19-Jul-2019 14:23:08.950 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [chatbot] registered the JDBC driver [com.mysql.jdbc.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
19-Jul-2019 14:23:08.951 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [chatbot] appears to have started a thread named [Abandoned connection cleanup thread] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
19-Jul-2019 14:23:08.952 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [chatbot] appears to have started a thread named [Druid-ConnectionPool-Create-985155300] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
19-Jul-2019 14:23:08.952 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [chatbot] appears to have started a thread named [Druid-ConnectionPool-Destroy-985155300] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
19-Jul-2019 14:23:08.958 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [chatbot] appears to have started a thread named [MySQL Statement Cancellation Timer] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
19-Jul-2019 14:23:08.983 INFO [Thread-20] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler ["http-nio-8030"]
19-Jul-2019 14:23:09.042 INFO [Thread-20] org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler ["http-nio-8030"]
有了如上各種情況下Tomcat的日誌特徵，希望對你日後分析線上問題有幫助。