logback + nginx 實現最簡單的日誌框架

logback + nginx 實現最簡單的日誌框架

由來

 經理： 咱們現在業務量增加了很多，問題定位需要輸出大量的日誌！
 碼農：可以部署 elastic stock 分佈式日誌框架，它可以儲存PB級別的日誌 ……
 經理：咱們這東西很好但是有一定的學習成本，部署也相對麻煩！
 碼農：可以容器化部署，或者ansible，jenkins 一鍵式部署 ……
 經理: 嗯，但是我就是想要個hello world 能比較方便的查詢到系統日誌文件就好！
 碼農：嗯，那好的！

於是乎， logback + nginx 日誌框架誕生。（最終目標，logback + netty +web,實現日誌收集和實時推送，報警，儲存，分級，分類等）。

框架

首先準備Nginx

技術選型原因

很簡單，說到對靜態資源文件的解析，nginx無出其右，當然使用live-server之類的可以實現更多功能，例如：實時熱重載，日誌文件發生修改實時推送給頁面。但是文件監聽是有消耗的，而日誌文件體量很大，因此這裏首選nginx，還有個原因是，配置部署簡單，本身對nginx也有些粗淺的瞭解。

首先下載nginx（本文以windows爲例，因爲測試簡單）

下載地址：

http://nginx.org/en/download.html

安裝，不用安裝，不解釋。

配置nginx爲文件服務器

主要配置：

			allow all;
			autoindex on;
			autoindex_exact_size off;
			autoindex_localtime on;

1. autoindex on 表示顯示目錄
2. autoindex_exact_size 顯示文件大小顯示單位不一樣而已，可以自己實驗一下
3. autoindex_localtime 顯示文件時間

設置日誌文件目錄爲nginx資源目錄

root   D:/log/project/;

還有一點就是編碼格式，否則會出現日誌亂碼

charset utf-8;

我的配置文件如下：

然後啓動就可以了。
這個時候，nginx，已經完成了，但是如果你想要是通過瀏覽器查詢日誌，而不是下載日誌（我的需求是查看，而不是下載），並且你輸出的日誌拓展名爲 .log ,那麼你需要加上如下配置，將.log文件聲明爲文本文件。

最簡單的方式，添加一條mimeType的聲明：
conf/mime.types 文件中添加

text/plain                                       log;

配置截圖如下：

配置logback

logback 添加文件輸出的Appender

配置如下：

    <appender name="errorFILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <prudent>true</prudent>
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <fileNamePattern>${LogFilePath}/error-log/error-%d{yyyy-MM-dd}.txt</fileNamePattern>
            <maxHistory>72</maxHistory>
            <totalSizeCap>3GB</totalSizeCap>
        </rollingPolicy>
        <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
            <pattern>{"tags":"controller", "timestamp": "%date{yyyy-MM-dd'T'HH:mm:ss,SSSZ}", "level": "%level", "thread": "%thread", "location": "%logger{0}::%method", "message": "%message"}%n</pattern>
        </encoder>
    </appender>

簡單的說明一下：

1. TimeBasedRollingPolicy 配置什麼時候觸發，並且按照時間聲明文件名%d{yyyy-MM-dd} 爲聲明到天，%d{yyyy-MM-dd HH}聲明到小時，以此類推。
2. pattern 控制處處內容，這裏我希望輸出json，但是又懶得去自定義一個layout 所以直接使用了字符串的方式。
3. ${LogFilePath} 是定義的基礎日誌輸出位置變量 聲明方式爲 ：

<property name="LogFilePath" value="D:/log/project/schedule"/>

注：這裏的路徑一定爲nginx資源目錄，或者是子目錄，否則需要添加新的nginx配置才能監聽到這些日誌文件。

####### 定義logger 輸出到自定文件
這裏以錯誤日誌輸出爲例（上面也是錯誤日誌的appender聲明）

    <!-- 錯誤日誌輸出 -->
    <logger name="com.guodun" level="error">
        <appender-ref ref="errorFILE"/>
    </logger>

到這裏logback 已經準備完成了。下面的爲一些建議和自己的拙見

如果，你也像我一樣有點代碼潔癖，如果也需要很多的不同類型的日誌文件輸出，同時又不想讓logback的配置文件過於臃腫，那麼就添加一個appender的聲明配置文件吧。

引用語法：

<include resource="logback-file-appenders.xml"/>

聲明語法：

<included>
    <appender name="includedConsole" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>%d - %m%n</pattern>
        </encoder>
    </appender>
</included>

被引用的配置文件，必須以 included 標籤包裹

注：以上引用爲引用資源文件下的配置，也是比較常用的，還有些其他的引用，如下

引用指定文件

<include file="src/main/resources/includedConfig.xml" />

引用網絡配置:

<include url="http://some.host.com/includedConfig.xml"/>

運行結果如下：

打開日誌文件截圖：

附上源碼：

logback.xml

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <include resource="logback-file-appenders.xml"/>
    <!--自定義顏色配置-->
    <conversionRule conversionWord="customcolor" converterClass="com.guodun.serve.schedule.config.LogbackColorConfig"/>
    <property resource="application.properties" />

    <property name="jsonLog" value="{time:%d{yyyy-MM-dd HH:mm:ss.SSS},location: %logger{0}::%method, msg:%msg}%n"/>
   <!-- Logstash 分佈式日誌輸出-->
    <appender name="LOGSTASH"
              class="net.logstash.logback.appender.LogstashTcpSocketAppender">
        <destination>192.168.1.141:9601</destination>
        <!-- encoder必須配置,有多種可選 -->
        <encoder charset="UTF-8"
                 class="net.logstash.logback.encoder.LogstashEncoder" >
            <!-- "appname":"yang_test" 的作用是指定創建索引的名字時用，並且在生成的文檔中會多了這個字段  -->
            <customFields>{"appname":"${spring.application.name}","IP":"localhost","port":"${server.port}"}</customFields>
        </encoder>
    </appender>

    <!--    控制檯輸出-->
    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
            <!--格式化輸出：%d表示日期，%thread表示線程名，%-5level：級別從左顯示5個字符寬度%msg：日誌消息，%n是換行符-->
            <pattern>%d{yyyy-MM-dd HH:mm:ss.SSS}%customcolor([%thread]) %customcolor(%-5level) %logger{50} -  %customcolor(%msg)  %n</pattern>
        </encoder>
    </appender>

    <!--Cotroller 控制檯輸出-->
    <appender name="ControllerConsole" class="ch.qos.logback.core.ConsoleAppender">
        <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
            <!--格式化輸出：%d表示日期，%thread表示線程名，%-5level：級別從左顯示5個字符寬度%msg：日誌消息，%n是換行符-->
            <pattern>[ %d{HH:mm:ss.SSS} ] %customcolor(%-5level)| %logger{0} :: %method | -  %customcolor(%msg)  %n</pattern>
        </encoder>
    </appender>

    <!-- 控制檯輸出日誌級別 -->
    <root level="info">
         <appender-ref ref="STDOUT"/>
    </root>
    <logger name="com.guodun.serve.schedule" level="debug">
        <appender-ref ref="ControllerConsole"/>
    </logger>


    <logger name="com.guodun" level="info">
        <appender-ref ref="LOGSTASH"/>
    </logger>
    <!-- 錯誤日誌輸出 -->
    <logger name="com.guodun" level="error">
        <appender-ref ref="errorFILE"/>
    </logger>

    <!--策略日誌輸出-->
    <logger name="com.guodun.serve.schedule.strategy" level="debug">
        <appender-ref ref="strategyFILE"/>
    </logger>

   <!-- 控制層日誌輸出-->
    <logger name="com.guodun.serve.schedule.controller" level="debug">
        <appender-ref ref="ControlleFILE"/>
    </logger>


    <logger name="com.netflix" level="OFF"></logger>
    <logger name="o.a.c.core" level="OFF"></logger>
    <logger name="org.springframework.web" level="OFF"></logger>

</configuration>

logback-file-appenders.xml

<included>
    <property name="LogFilePath" value="D:/log/guodun/schedule"/>

    <!--  錯誤文件  -->
    <appender name="errorFILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <prudent>true</prudent>
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <fileNamePattern>${LogFilePath}/error-log/error-%d{yyyy-MM-dd}.txt</fileNamePattern>
            <maxHistory>72</maxHistory>
            <totalSizeCap>3GB</totalSizeCap>
        </rollingPolicy>
        <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
            <pattern>{"tags":"controller", "timestamp": "%date{yyyy-MM-dd'T'HH:mm:ss,SSSZ}", "level": "%level", "thread": "%thread", "location": "%logger{0}::%method", "message": "%message"}%n</pattern>
        </encoder>
    </appender>


    <!--   策略過程debug文件  -->
    <appender name="strategyFILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <prudent>true</prudent>
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <fileNamePattern>${LogFilePath}/strategy-log/log-%d{yyyy-MM-dd HH}.txt</fileNamePattern>
            <maxHistory>72</maxHistory>
            <totalSizeCap>3GB</totalSizeCap>
        </rollingPolicy>
        <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
            <pattern>{"tags":"controller", "timestamp": "%date{yyyy-MM-dd'T'HH:mm:ss,SSSZ}", "level": "%level", "thread": "%thread", "location": "%logger{0}::%method", "message": "%message"}%n</pattern>
        </encoder>
    </appender>


    <!-- 控制層 debug 日誌文件 -->
    <appender name="ControlleFILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <prudent>true</prudent>
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <fileNamePattern>${LogFilePath}/controller-log/ControllerLog-%d{yyyy-MM-dd HH}.txt</fileNamePattern>
            <maxHistory>72</maxHistory>
            <totalSizeCap>3GB</totalSizeCap>
        </rollingPolicy>
        <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
            <!--格式化輸出：%d表示日期，%thread表示線程名，%-5level：級別從左顯示5個字符寬度%msg：日誌消息，%n是換行符-->
            <pattern>{"tags":"controller", "timestamp": "%date{yyyy-MM-dd'T'HH:mm:ss,SSSZ}", "level": "%level", "thread": "%thread", "location": "%logger{0}::%method", "message": "%message"}%n</pattern>
        </encoder>
    </appender>


</included>

mime.types

types {
    text/html                                        html htm shtml;
    text/css                                         css;
    text/xml                                         xml;
    image/gif                                        gif;
    image/jpeg                                       jpeg jpg;
    application/javascript                           js;
    application/atom+xml                             atom;
    application/rss+xml                              rss;

    text/mathml                                      mml;
    text/plain                                       txt;
	text/plain                                       log;
    text/vnd.sun.j2me.app-descriptor                 jad;
    text/vnd.wap.wml                                 wml;
    text/x-component                                 htc;

    image/png                                        png;
    image/svg+xml                                    svg svgz;
    image/tiff                                       tif tiff;
    image/vnd.wap.wbmp                               wbmp;
    image/webp                                       webp;
    image/x-icon                                     ico;
    image/x-jng                                      jng;
    image/x-ms-bmp                                   bmp;

    font/woff                                        woff;
    font/woff2                                       woff2;

    application/java-archive                         jar war ear;
    application/json                                 json;
    application/mac-binhex40                         hqx;
    application/msword                               doc;
    application/pdf                                  pdf;
    application/postscript                           ps eps ai;
    application/rtf                                  rtf;
    application/vnd.apple.mpegurl                    m3u8;
    application/vnd.google-earth.kml+xml             kml;
    application/vnd.google-earth.kmz                 kmz;
    application/vnd.ms-excel                         xls;
    application/vnd.ms-fontobject                    eot;
    application/vnd.ms-powerpoint                    ppt;
    application/vnd.oasis.opendocument.graphics      odg;
    application/vnd.oasis.opendocument.presentation  odp;
    application/vnd.oasis.opendocument.spreadsheet   ods;
    application/vnd.oasis.opendocument.text          odt;
    application/vnd.openxmlformats-officedocument.presentationml.presentation
                                                     pptx;
    application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
                                                     xlsx;
    application/vnd.openxmlformats-officedocument.wordprocessingml.document
                                                     docx;
    application/vnd.wap.wmlc                         wmlc;
    application/x-7z-compressed                      7z;
    application/x-cocoa                              cco;
    application/x-java-archive-diff                  jardiff;
    application/x-java-jnlp-file                     jnlp;
    application/x-makeself                           run;
    application/x-perl                               pl pm;
    application/x-pilot                              prc pdb;
    application/x-rar-compressed                     rar;
    application/x-redhat-package-manager             rpm;
    application/x-sea                                sea;
    application/x-shockwave-flash                    swf;
    application/x-stuffit                            sit;
    application/x-tcl                                tcl tk;
    application/x-x509-ca-cert                       der pem crt;
    application/x-xpinstall                          xpi;
    application/xhtml+xml                            xhtml;
    application/xspf+xml                             xspf;
    application/zip                                  zip;

    application/octet-stream                         bin exe dll;
    application/octet-stream                         deb;
    application/octet-stream                         dmg;
    application/octet-stream                         iso img;
    application/octet-stream                         msi msp msm;

    audio/midi                                       mid midi kar;
    audio/mpeg                                       mp3;
    audio/ogg                                        ogg;
    audio/x-m4a                                      m4a;
    audio/x-realaudio                                ra;

    video/3gpp                                       3gpp 3gp;
    video/mp2t                                       ts;
    video/mp4                                        mp4;
    video/mpeg                                       mpeg mpg;
    video/quicktime                                  mov;
    video/webm                                       webm;
    video/x-flv                                      flv;
    video/x-m4v                                      m4v;
    video/x-mng                                      mng;
    video/x-ms-asf                                   asx asf;
    video/x-ms-wmv                                   wmv;
    video/x-msvideo                                  avi;
}

nginx.conf

#user  nobody;
worker_processes  1;

#error_log  logs/error.log;
#error_log  logs/error.log  notice;
#error_log  logs/error.log  info;

#pid        logs/nginx.pid;


events {
    worker_connections  1024;
}


http {
    include       mime.types;
    default_type  application/octet-stream;

    #log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
    #                  '$status $body_bytes_sent "$http_referer" '
    #                  '"$http_user_agent" "$http_x_forwarded_for"';

    #access_log  logs/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    #keepalive_timeout  0;
    keepalive_timeout  65;

    #gzip  on;

    server {
        listen       6780;
        server_name  localhost;

        charset utf-8;

        #access_log  logs/host.access.log  main;

        location / {
            root   D:/log/guodun/;
			allow all;
			autoindex on;
			autoindex_exact_size off;
			autoindex_localtime on;
            index  index.html index.htm;
        }

        #error_page  404              /404.html;

        # redirect server error pages to the static page /50x.html
        #
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }

        # proxy the PHP scripts to Apache listening on 127.0.0.1:80
        #
        #location ~ \.php$ {
        #    proxy_pass   http://127.0.0.1;
        #}

        # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
        #
        #location ~ \.php$ {
        #    root           html;
        #    fastcgi_pass   127.0.0.1:9000;
        #    fastcgi_index  index.php;
        #    fastcgi_param  SCRIPT_FILENAME  /scripts$fastcgi_script_name;
        #    include        fastcgi_params;
        #}

        # deny access to .htaccess files, if Apache's document root
        # concurs with nginx's one
        #
        #location ~ /\.ht {
        #    deny  all;
        #}
    }


    # another virtual host using mix of IP-, name-, and port-based configuration
    #
    #server {
    #    listen       8000;
    #    listen       somename:8080;
    #    server_name  somename  alias  another.alias;

    #    location / {
    #        root   html;
    #        index  index.html index.htm;
    #    }
    #}


    # HTTPS server
    #
    #server {
    #    listen       443 ssl;
    #    server_name  localhost;

    #    ssl_certificate      cert.pem;
    #    ssl_certificate_key  cert.key;

    #    ssl_session_cache    shared:SSL:1m;
    #    ssl_session_timeout  5m;

    #    ssl_ciphers  HIGH:!aNULL:!MD5;
    #    ssl_prefer_server_ciphers  on;

    #    location / {
    #        root   html;
    #        index  index.html index.htm;
    #    }
    #}

}

時間比較緊張，有些地方不完善，望請見諒。

如果感覺有用，點個贊再走唄。

----------------------------------------------------------這就是我的底線--------------------------------------------------------------