前言
ELKstack抓取分析應用日誌的前提是開發要將日誌的輸出格式改成json格式,否則產出的日誌很難進行分析。
主要流程: tomcat通過log4j輸出日誌--> logstash接收數據併發送到redis --> logstash從redis獲取數據並寫入es
tomcat怎樣輸出應用日誌?
tomcat7.0配置log4j日誌
詳細文檔http://tomcat.apache.org/tomcat-7.0-doc/logging.html
一、創建log4j.properties文件
在$CATALINA_BASE/lib文件夾下,新建log4j.properties文件,內容如下
log4j.rootLogger = INFO, CATALINA
# Define all the appenders log4j.appender.CATALINA = org.apache.log4j.DailyRollingFileAppender log4j.appender.CATALINA.File = ${catalina.base}/logs/catalina log4j.appender.CATALINA.Append = true log4j.appender.CATALINA.Encoding = UTF-8 # Roll-over the log once per day log4j.appender.CATALINA.DatePattern = '.'yyyy-MM-dd'.log' log4j.appender.CATALINA.layout = org.apache.log4j.PatternLayout log4j.appender.CATALINA.layout.ConversionPattern = %d [%t] %-5p %c- %m%n log4j.appender.LOCALHOST = org.apache.log4j.DailyRollingFileAppender log4j.appender.LOCALHOST.File = ${catalina.base}/logs/localhost log4j.appender.LOCALHOST.Append = true log4j.appender.LOCALHOST.Encoding = UTF-8 log4j.appender.LOCALHOST.DatePattern = '.'yyyy-MM-dd'.log' log4j.appender.LOCALHOST.layout = org.apache.log4j.PatternLayout log4j.appender.LOCALHOST.layout.ConversionPattern = %d [%t] %-5p %c- %m%n log4j.appender.MANAGER = org.apache.log4j.DailyRollingFileAppender log4j.appender.MANAGER.File = ${catalina.base}/logs/manager log4j.appender.MANAGER.Append = true log4j.appender.MANAGER.Encoding = UTF-8 log4j.appender.MANAGER.DatePattern = '.'yyyy-MM-dd'.log' log4j.appender.MANAGER.layout = org.apache.log4j.PatternLayout log4j.appender.MANAGER.layout.ConversionPattern = %d [%t] %-5p %c- %m%n log4j.appender.HOST-MANAGER = org.apache.log4j.DailyRollingFileAppender log4j.appender.HOST-MANAGER.File = ${catalina.base}/logs/host-manager log4j.appender.HOST-MANAGER.Append = true log4j.appender.HOST-MANAGER.Encoding = UTF-8 log4j.appender.HOST-MANAGER.DatePattern = '.'yyyy-MM-dd'.log' log4j.appender.HOST-MANAGER.layout = org.apache.log4j.PatternLayout log4j.appender.HOST-MANAGER.layout.ConversionPattern = %d [%t] %-5p %c- %m%n log4j.appender.CONSOLE = org.apache.log4j.ConsoleAppender log4j.appender.CONSOLE.Encoding = UTF-8 log4j.appender.CONSOLE.layout = org.apache.log4j.PatternLayout log4j.appender.CONSOLE.layout.ConversionPattern = %d [%t] %-5p %c- %m%n # Configure which loggers log to which appenders log4j.logger.org.apache.catalina.core.ContainerBase.[Catalina].[localhost] = INFO, LOCALHOST log4j.logger.org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/manager] =\ INFO, MANAGER log4j.logger.org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/host-manager] =\ INFO, HOST-MANAGER
二、下載jar包
(一)log4j.jar(1.2.x版本)
http://mvnrepository.com/artifact/log4j/log4j/1.2.17
(二)tomcat-juli.jar、tomcat-juli-adapters.jar
http://tomcat.apache.org/download-70.cgi
注意:下載時一定要下載"extras"的包,它與自帶的包有區別,官方說明:
This tomcat-juli.jar differs from the default one. It contains the full Apache Commons Logging implementation and thus is able to discover the presence of log4j and configure itself.
三、替換jar包
將log4j.jar、tomcat-juli-adapters.jar放入$CATALINA_BASE/lib目錄
將下載的tomcat-juli.jar替換$CATALINA_HOME/bin目錄下的tomcat-juli.jar
注意:兩個包放置的位置是不同的
四、刪除logging.properties
刪除tomcat默認的日誌文件$CATALINA_BASE/conf/logging.properties文件
五、啓動tomcat
啓動後在$CATALINA_BASE/logs目錄下就能看到log4j輸出的日誌
開發自定義應用日誌輸出
一般開發java程序,運行tomcat在應用上,日誌格式定義普遍放在工程下:WEB-INF/class 目錄下,一般都叫做log4j.properties
作者使用的logstash版本爲2.3 ,屬於最新版,如果小夥伴借鑑時出現異常,可以看下版本,對比官方文檔。
讓開發在log4j.properties中做以下更改就可:
#log4j.rootLogger = [ level ] , appenderName, appenderName, ... log4j.rootLogger = info,console,infoR,errorR,logstash #branch logger log4j.logger.org.springframework=WARN #log4j.logPath log4j.logPath=/opt/tomcat_xxxxxweb/log #console log4j.appender.console = org.apache.log4j.ConsoleAppender log4j.appender.console.layout = org.apache.log4j.PatternLayout log4j.appender.console.layout.ConversionPattern = [%p] %d{yyyy-MM-dd HH:mm:ss} [%l] %m%n #infoR log4j.appender.infoR = org.apache.log4j.DailyRollingFileAppender log4j.appender.infoR.File =${log4j.logPath}/XXXXXX.log log4j.appender.infoR.layout = org.apache.log4j.PatternLayout log4j.appender.infoR.layout.ConversionPattern=[%p] %d{yyyy-MM-dd HH:mm:ss} [%l] %m%n log4j.appender.infoR.datePattern='.'yyyyMMdd'.log' log4j.appender.infoR.Threshold = INFO #errorR log4j.appender.errorR = org.apache.log4j.DailyRollingFileAppender log4j.appender.errorR.File =${log4j.logPath}/XXXXXError.log log4j.appender.errorR.layout = org.apache.log4j.PatternLayout log4j.appender.errorR.layout.ConversionPattern=[%p] %d{yyyy-MM-dd HH:mm:ss} [%l] %m%n log4j.appender.errorR.datePattern='.'yyyyMMdd'.log' log4j.appender.errorR.Threshold = ERROR #logstash log4j.appender.logstash=org.apache.log4j.net.SocketAppender log4j.appender.logstash.Port=12201 log4j.appender.logstash.RemoteHost=sz-a-xxxxxlogstash01-logstash-xen.xxxxx.com log4j.appender.logstash.ReconnectionDelay=60000 log4j.appender.logstash.LocationInfo=true log4j.appender.logstash.Threshold = INFO
logstash如何接受log4j日誌並寫入redis?
此處將不再介紹logstash的安裝,建議使用yum安裝管理,使用起來比較方便。
可以將logstash配置文件放入/etc/logstash/conf.d 內
logstash配置文件展示:
input { tcp { port => "11011" type => "syslog" } log4j { type => "log4j-json-web" port=>12202 } log4j { type => "log4j-json-tomcat" port=>12201 } } filter { if [type] == "syslog" { grok { match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" } } } if [type] == "log4j-json-web" { json { source => "message" } } } output { if [type] == "syslog" { redis { host => "sz-a-xxxxxredis01-redis-xen.xxxxx.com" data_type => "list" key => "logstash:syslog-log" } } if [type] == "log4j-json-web" { redis { host => "sz-a-xxxxxredis01-redis-xen.xxxxx.com" data_type => "list" key => "logstash:xxxxxweb-web-log" } } if [type] == "log4j-json-tomcat" { redis { host => "sz-a-xxxxxredis01-redis-xen.xxxxx.com" data_type => "list" key => "logstash:xxxxxweb-tomcat-log" } } }
一個logstash進程可以有多個input輸入,然後爲每個輸入設置type屬性,此時需要注意!!日誌中不要出現type字段,不然是大坑!!!!
ouput可以通過if語句判斷input來源並輸出到不同的地方
redis使用list模式來存儲數據,這樣就起到緩存的作用,一入一出,不會撐爆redis