filebeat 獲取nginx日誌 發送給ElasticSearch

目的：通過filebeat獲取nginx日誌，發送給ElasticSearch，filebeat可以解析json格式日誌，所以設置nginx日誌爲json格式。

1、配置nginx配置文件

    

    log_format jsonTest '{"@timestamp":"$time_iso8601",'
                  '"host":"$server_addr",'
                  '"service":"nginxTest",'
                  '"trace":"$upstream_http_ctx_transaction_id",'
                  '"log":"log",'
                  '"clientip":"$remote_addr",'
                  '"remote_user":"$remote_user",'
                  '"request":"$request",'
                  '"http_user_agent":"$http_user_agent",'
                  '"size":$body_bytes_sent,'
                  '"responsetime":$request_time,'
                  '"upstreamtime":"$upstream_response_time",'
                  '"upstreamhost":"$upstream_addr",'
                  '"http_host":"$host",'
                  '"url":"$uri",'
                  '"domain":"$host",'
                  '"xff":"$http_x_forwarded_for",'
                  '"referer":"$http_referer",'
                  '"status":"$status"}';
     access_log /var/log/nginx/access.log jsonTest;

定義jsonTest的json格式，其中trace是頁面response headers的值 ctx-transaction-id，通過upstream_http_ctx_transaction_id可以獲取頭文件屬性。

將日誌輸出到/var/log/nginx/access.log

2、配置filebeat配置文件

filebeat.prospectors:
- type: log
  paths:
   - '/root/front/logs/*.log'
  json.message_key: log
  json.keys_under_root: true

output.elasticsearch:
  hosts: ["*.*.*.*:9200"]

將日誌傳輸給ElasticSearch