http://nosmoking.blog.51cto.com/3263888/1852115
一、安裝1、jdk 和 環境變量支持jdk-1.7以上，推薦jdk-1.8在環境變量配置：JAVA_HOME2、安裝有2種方式下載，推薦緩存rpm包到本地yum源1）直接使用rpmwget https://download.elastic.co/logstash/logstash/packages/centos/logstash-2.4.0.noarch.rpm2）使用yum源[root@vm49 ~]# rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch[root@vm49 ~]# vim /etc/yum.repos.d/logstash.repo[logstash-2.4]name=Logstash repository for 2.4.x packagesbaseurl=https://packages.elastic.co/logstash/2.4/centosgpgcheck=1gpgkey=https://packages.elastic.co/GPG-KEY-elasticsearchenabled=1[root@vm49 ~]# yum install logstash[root@vm49 ~]# whereis logstashlogstash: /etc/logstash /opt/logstash/bin/logstash /opt/logstash/bin/logstash.bat二、使用1、命令行測試1）輸出到 stdout[root@vm49 ~]# /opt/logstash/bin/logstash -e 'input { stdin { } } output { stdout {} }'hi, let us go（輸入）Settings: Default pipeline workers: 4Pipeline main started2016-09-12T02:42:59.110Z 0.0.0.0 hi, let us go（輸出）why not TRY IT OUT（輸入）2016-09-12T02:43:11.904Z 0.0.0.0 why not TRY IT OUT（輸出）（CTRL-D 退出）Pipeline main has been shutdownstopping pipeline {:id=>"main"}2）改變輸出到 stdout 的格式[root@vm49 ~]# /opt/logstash/bin/logstash -e 'input { stdin { } } output { stdout { codec => rubydebug } }'aaaaSettings: Default pipeline workers: 4Pipeline main started{ "message" => "aaaa", "@version" => "1", "@timestamp" => "2016-09-18T02:11:07.109Z", "host" => "0.0.0.0"}bbbb{ "message" => "bbbb", "@version" => "1", "@timestamp" => "2016-09-18T02:11:11.864Z", "host" => "0.0.0.0"}Pipeline main has been shutdownstopping pipeline {:id=>"main"}3）上述輸出發現一個小問題，得到的 host 爲 0.0.0.0，而不是期望中的當前 hostname檢查一下 hostname[root@vm49 ~]# hostnamevm49.localdomain[root@vm49 ~]# echo '10.50.200.49 vm49.localdomain' >>/etc/hosts再次測試：[root@vm49 ~]# /opt/logstash/bin/logstash -e 'input { stdin { } } output { stdout { codec => rubydebug } }'aaaaaSettings: Default pipeline workers: 4Pipeline main started{ "message" => "aaaaa", "@version" => "1", "@timestamp" => "2016-09-18T02:22:12.180Z", "host" => "vm49.localdomain"}Pipeline main has been shutdownstopping pipeline {:id=>"main"}符合預期。注1：還有一種場景（hostname使用本地名稱，而不是採取域名的命名方式）需要注意/etc/hosts 中包括下述內容------------------------127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4::1 localhost localhost.localdomain localhost6 localhost6.localdomain6127.0.0.1 local-vm5010.50.200.50 local-vm50------------------------期望的結果是："host" => "local-vm50"實際得到的結果是："host" => "localhost"此時，需要刪除：127.0.0.1 local-vm50然後才能得到期望的結果。注2：如果是在 logstash 服務啓動後纔去調整了 hostname 相關設置，則需要重啓 logstash 服務，否則依然是舊的結果。2、使用配置文件目的：從日誌文件中讀取數據，輸出到另一個文件中來查看。前提：已經配置了一個nginx服務，生成了對應的日誌文件。首先，因爲這裏要讀取nginx日誌，請將 logstash 加入 nginx 的用戶組。示例：[root@vm49 ~]# ls -l /var/log/nginx/access_*.log-rw-r----- 1 nginx adm 608083 Sep 13 18:18 /var/log/nginx/access_www.test.com_80.log-rw-r----- 1 nginx adm 597070 Sep 13 18:18 /var/log/nginx/access_www.work.com_80.lognginx屬於 adm 組，使用 logstash 讀取日誌，可能產生權限異常：failed to open /var/log/nginx/access.ecshop.test.d3dstore.com.log: Permission denied因此，要加入對應的組：[root@vm49 ~]# usermod -G adm logstash接着，我們嘗試這樣配置 logstash 來收集日誌：[root@vm49 ~]# cat /etc/logstash/conf.d/nginx.conf input { file { path => "/var/log/nginx/access_*.log" start_position => beginning ignore_older => 0 }}filter { grok { match => { "message" => "%{COMBINEDAPACHELOG}"} }}output { file { path => "/tmp/test.log" }}上面使用到以下插件：file：日誌數據的輸入和輸出grok：來匹配標準的apache日誌格式【細節延伸】顯然，在3個環節中，都有改進和調整的地方。input：使用 filebeatfilter：使用其他插件和規則output：使用ES，redis等具體請參考：https://www.elastic.co/guide/en/logstash/current/pipeline.html3、測試配置文件：[root@vm49 ~]# service logstash configtestConfiguration OK4、啓動服務：[root@vm49 ~]# service logstash start[root@vm49 ~]# chkconfig logstash on5、測試請求nginx服務，然後觀察輸出的內容：[root@vm49 ~]# cat /tmp/test.log符合預期。6、比較去掉 filter 這一節，我們來對比一下 /tmp/test.log 收集到的內容的差異【使用了 filter 的結果a】{"message":"10.50.200.219 - - [12/Sep/2016:13:00:03 +0800] \"GET / HTTP/1.1\" 200 13 \"-\" \"curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.14.0.0 zlib/1.2.3 libidn/1.18 libssh2/1.4.2\" \"-\" 0.000 \"-\" \"-\"","@version":"1","@timestamp":"2016-09-12T05:00:04.140Z","path":"/var/log/nginx/access_www.test.com_80.log","host":"vm49.localdomain","clientip":"10.50.200.219","ident":"-","auth":"-","timestamp":"12/Sep/2016:13:00:03 +0800","verb":"GET","request":"/","httpversion":"1.1","response":"200","bytes":"13","referrer":"\"-\"","agent":"\"curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.14.0.0 zlib/1.2.3 libidn/1.18 libssh2/1.4.2\""}【未使用 filter 的結果b】{"message":"10.50.200.219 - - [12/Sep/2016:13:07:49 +0800] \"GET / HTTP/1.1\" 200 13 \"-\" \"curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.14.0.0 zlib/1.2.3 libidn/1.18 libssh2/1.4.2\" \"-\" 0.000 \"-\" \"-\"","@version":"1","@timestamp":"2016-09-12T05:07:49.917Z","path":"/var/log/nginx/access_www.test.com_80.log","host":"vm49.localdomain"}a的內容中，多出來的地方，正是使用了 grok 分析和結構化數據---------------------------------------------------Information
Field Name----------- ----------IP Address clientipUser ID identUser Authentication
authtimestamp
timestampHTTP Verb
verbRequest body
requestHTTP Version
httpversionHTTP Status Code
responseBytes served
bytesReferrer URL
referrerUser agent
agent---------------------------------------------------注：也可以將 filter 調整到服務端，讓客戶端僅僅是收集日誌即可，這個取捨需要自行對比。7、改進Logstash 默認自帶了 apache 標準日誌的 grok 正則:如何使用自定義的日誌格式呢？例如，默認的 nginx 日誌是： log_format main '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"';改成自定義的日誌格式： log_format online '$remote_addr [$time_local] "$request" ' '"$http_content_type" "$request_body" "$http_referer" ' '$status $request_time $body_bytes_sent'; 對應的數據：【GET】# curl -H "Content-Type: text/html; charset=UTF-8" --referer 'www.abc.com/this_is_a_referer' http://www.test.com/a/b/c.html?key1=value1【結果】10.50.200.219 [12/Sep/2016:15:11:04 +0800] "GET /a/b/c.html?key1=value1 HTTP/1.1" "text/html; charset=UTF-8" "-" "www.abc.com/this_is_a_referer" 404 0.000 168【POST】# curl -H "Content-Type: application/xml" -d "{"name": "Mark Lee" }" "http://www.test.com/start"【結果】10.50.200.218 [12/Sep/2016:15:02:07 +0800] "POST /start HTTP/1.1" "application/xml" "-" "-" 404 0.000 168嘗試一下：[root@vm49 ~]# mkdir -p /etc/logstash/patterns.d[root@vm49 ~]# vim /etc/logstash/patterns.d/extra_patternsNGINXACCESS %{IPORHOST:clientip} \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" (?:%{QS:content_type}|-) (?:%{QS:request_body}|-) (?:"(?:%{URI:referrer}|-)"|%{QS:referrer}) %{NUMBER:response} %{BASE16FLOAT:request_time} (?:%{NUMBER:bytes}|-)調整配置爲：[root@vm49 ~]# cat /etc/logstash/conf.d/nginx.conf input { file { path => "/var/log/nginx/access_*.log" start_position => beginning ignore_older => 0 }}filter { grok { patterns_dir => ["/etc/logstash/patterns.d"] match => { "message" => "%{NGINXACCESS}" } }}output { file { path => "/tmp/test.log" }}[root@vm49 ~]# service logstash restart結果：{"message":"10.50.200.218 [12/Sep/2016:15:28:23 +0800] \"POST /start HTTP/1.1\" \"application/xml\" \"-\" \"-\" 404 0.000 168","@version":"1","@timestamp":"2016-09-12T07:28:24.007Z","path":"/var/log/nginx/access_www.test.com_80.log","host":"vm49.localdomain","clientip":"10.50.200.218","timestamp":"12/Sep/2016:15:28:23 +0800","verb":"POST","request":"/start","httpversion":"1.1","content_type":"\"application/xml\"","request_body":"\"-\"","response":"404","request_time":"0.000","bytes":"168"}{"message":"10.50.200.219 [12/Sep/2016:15:28:24 +0800] \"GET /a/b/c.html?key1=value1 HTTP/1.1\" \"text/html; charset=UTF-8\" \"-\" \"www.abc.com/this_is_a_referer\" 404 0.000 168","@version":"1","@timestamp":"2016-09-12T07:28:25.019Z","path":"/var/log/nginx/access_www.test.com_80.log","host":"vm49.localdomain","clientip":"10.50.200.219","timestamp":"12/Sep/2016:15:28:24 +0800","verb":"GET","request":"/a/b/c.html?key1=value1","httpversion":"1.1","content_type":"\"text/html; charset=UTF-8\"","request_body":"\"-\"","referrer":"\"www.abc.com/this_is_a_referer\"","response":"404","request_time":"0.000","bytes":"168"}符合預期。三、輸出到 redis+elasticsearch+kibana1、測試環境（已經部署了服務）【客戶端】10.50.200.49: logstash, nginx(www.test.com, www.work.com)【服務端】10.50.200.220: logstash, redis, elasticsearch, kibana【測試端】10.50.200.218, 10.50.200.219: curl 請求 nginx1）測試腳本--------------------------------------------------------[root@vm218 bin]# cat test_www_by_curl.sh#!/bin/bash##2016/9/20# crontab:# */2 * * * * /usr/local/bin/test_www_by_curl.sh >>/tmp/test_www_by_curl.log 2>&1 &echo "[S] at: `date`"tag="$(echo `hostname` |awk -F'-' '{print $NF}')"do_curl(){ www=$1 rnd=$2 #PUT curl -H "Content-Type: application/xml" -d "{"name": "Jack A"}" "http://${www}/from_${tag}" curl -H "Content-Type: application/json" -d "{"name": "Jack B"}" "http://${www}/from_${tag}" curl -H "Content-Type: application/png" -d "{"img_path": "images/$(date +%F)/${rnd}.png"}" "http://${www}/from_${tag}" #GET curl -H "Content-Type: text/html; charset=UTF-8" --referer "www.${tag}.com/this_is_referer" "http://${www}/" curl -H "Content-Type: text/html; charset=UTF-8" --referer "www.${tag}.com/this_is_referer" "http://${www}/a/b/c.html?key=${tag}" curl -H "Content-Type: text/html; charset=GBK" --referer "www.${tag}.com/this_is_referer" "http://${www}/action.do?event=login&user=${tag}&sid=${rnd}" #}for i in `seq 1 2000`; do do_curl www.work.com "`od /dev/urandom -w12 -tx4 -An |sed -e 's/ //g' |head -n 1`" do_curl www.test.com "`od /dev/urandom -w12 -tx4 -An |sed -e 's/ //g' |head -n 1`"done >/dev/null 2>&1echo "[D] at: `date`"--------------------------------------------------------2）hosts 寫入域名10.50.200.49 www.test.com10.50.200.49 www.work.com2、場景1：只有1個域名/模糊匹配N個域名目的：將匹配的 access 日誌收集起來集中展示。【客戶端】輸入：file輸出：redis[root@vm49 ~]# cat /etc/logstash/conf.d/nginx.conf input { file { type => "NginxAccess" path => "/var/log/nginx/access_*.log" start_position => beginning ignore_older => 0 }}filter { if[type] == "NginxAccess" { grok { patterns_dir => ["/etc/logstash/patterns.d"] match => { "message" => "%{NGINXACCESS}" } } date { match => [ "timestamp", "dd/MMM/YYYY:HH:mm:ss Z" ] remove_field => [ "timestamp" ] } }}output { if[type] == "NginxAccess" { redis { host => "10.50.200.220" data_type => "list" key => "logstash:nginxaccess" } }}[root@vm49 ~]# service logstash restart【服務端】輸入：redis輸出：elasticsearch[root@vm220 ~]# vim /etc/logstash/conf.d/redis.confinput { redis { host => '127.0.0.1' data_type => 'list' port => "6379" key => 'logstash:nginxaccess' type => 'redis-input' }}output { if[type] == "NginxAccess" { elasticsearch { hosts => "127.0.0.1:9200" index => "logstash-nginxaccess-%{+YYYY.MM.dd}" } }}[root@vm220 ~]# service logstash restart可以通過命令行去觀察 redis 的狀態：[root@vm220 ~]# redis-cli monitor指定kibana的索引名稱，例如：logstash-nginxaccess-*結果：符合預期。3、場景2：N個域名分開收集目的：將 www.test.com 和 www.work.com 的 access 日誌收集起來使用獨立的索引名來分開展示【客戶端】輸入：file輸出：redis[root@vm49 ~]# cat /etc/logstash/conf.d/nginx.conf input { file { type => "NginxAccess-www.test.com" path => "/var/log/nginx/access_www.test.com*.log" start_position => beginning ignore_older => 0 } file { type => "NginxAccess-www.work.com" path => "/var/log/nginx/access_www.work.com*.log" start_position => beginning ignore_older => 0 }}filter { if[type] =~ "NginxAccess-" { grok { patterns_dir => ["/etc/logstash/patterns.d"] match => { "message" => "%{NGINXACCESS}" } } date { match => [ "timestamp", "dd/MMM/YYYY:HH:mm:ss Z" ] remove_field => [ "timestamp" ] } }}output { if[type] =~ "NginxAccess-" { redis { host => "10.50.200.220" data_type => "list" key => "logstash:nginxaccess" } }}[root@vm49 ~]# service logstash restart【服務端】輸入：redis輸出：elasticsearch[root@vm220 ~]# cat /etc/logstash/conf.d/redis.conf input { redis { host => '127.0.0.1' data_type => 'list' port => "6379" key => 'logstash:nginxaccess' type => 'redis-input' }}output { if[type] == "NginxAccess-www.test.com" { elasticsearch { hosts => "127.0.0.1:9200" index => "logstash-nginxaccess-www.test.com-%{+YYYY.MM.dd}" } } if[type] == "NginxAccess-www.work.com" { elasticsearch { hosts => "127.0.0.1:9200" index => "logstash-nginxaccess-www.work.com-%{+YYYY.MM.dd}" } }}[root@vm220 ~]# service logstash restart當然了，要調整kibana的索引名稱，指定更具體的名稱，例如：logstash-nginxaccess-www.test.com-*logstash-nginxaccess-www.work.com-*結果：符合預期。四、小結FAQ1、數據流向------------------------------------------------------------------log_files -> logstash -> redis -> elasticsearch -> kibana------------------------------------------------------------------2、@timestamp 和 message 中記錄的時間字段（timestamp）不一致message 中記錄的是原始時間，而 @timestamp 中記錄的是事件採集到的時候記錄的時間，兩者可能差距有1s及以上。可以通過 date 插件來調整記錄到的時間，以日誌中記錄的原始時間爲準，可以用於讀取過去某段時間的日誌文件，這樣一來，收集日誌時記錄的時間不會錯亂。例如：filter { date { match => [ "timestamp", "dd/MMM/YYYY:HH:mm:ss Z" ] remove_field => [ "timestamp" ] }}將匹配 field "timestamp" 格式爲 "dd/MMM/YYYY:HH:mm:ss Z"實際匹配到：18/Sep/2016:16:43:15 +0800默認更新 @timestamp 爲：September 18th 2016, 16:43:15.000特別說明：1）timestamp 是我們在原始日誌中收集的時間字段，不一定是這個名稱（這意味是任意的自定義的字段名稱），因爲只是我們在前述操作中使用 grok 插件中在 patterns 中定義了 "%{NGINXACCESS}" 這個變量來解析原始字段，裏邊定義了這個 "timestamp" 字段。a、grok插件的定義 grok { patterns_dir => ["/etc/logstash/patterns.d"] match => { "message" => "%{NGINXACCESS}" } }b、grok插件指明的 patterns 的定義[root@vm49 ~]# cat /etc/logstash/patterns.d/extra_patternsNGINXACCESS %{IPORHOST:clientip} \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" (?:%{QS:content_type}|-) (?:%{QS:request_body}|-) (?:"(?:%{URI:referrer}|-)"|%{QS:referrer}) %{NUMBER:response} %{BASE16FLOAT:request_time} (?:%{NUMBER:bytes}|-)2）爲何要使用 remove_field 功能？a、重複數據我們已經有了一個時間字段 "@timestamp" 因此可以丟掉這個重複的 "timestamp" 字段。注：由此聯想一下，生產應用中，我們是不是應該好好利用下 "remove_field" 功能去篩選一下字段呢？主動的去丟棄一些字段，只保留特別重要的指標發送給 output 即可。b、導入到 elasticsearch 時，模版 mapping type 時異常以收集 nginx error log 爲例：error log 的時間字段格式是這樣的：2016/09/21 10:34:41對應的 date 插件是這樣寫的： date { match => [ "timestamp", "YYYY/MM/dd HH:mm:ss" ] remove_field => [ "timestamp" ] }如果不刪除字段 "timestamp"，當傳遞到 elasticsearch 時，模版將試圖解析這字段的格式，這個時候會遇到解析異常，從而影響到 index 的建立。在 elasticsearch 的日誌中將產生如下錯誤：MapperParsingException[failed to parse [timestamp]]; nested: IllegalArgumentException[Invalid format: "2016/09/21 10:34:41" is malformed at "/09/21 10:34:41"];3、如何使用 grok 插件來匹配 nginx error log1）示例日誌2016/09/21 10:34:41 [error] 11990#0: *6718852 no live upstreams while connecting to upstream, client: 10.50.200.219, server: c.test.com, request: "GET http://www.work.com/cgi-bin/common/attr?id=260714&r=0.1011513018987016 HTTP/1.1", upstream: "http://upstream_backend/cgi-bin/common/attr?id=260714&r=0.1011513018987016", host: "www.work.com"2016/09/21 11:26:41 [error] 1614#0: *25584123 open() "/data/www/www.test.com/404.html" failed (2: No such file or directory), client: 10.50.200.218, server: www.test.com, request: "GET /action.do?event=login&user=vm218&sid=15120ae826ca2290514bcd5b HTTP/1.1", host: "www.test.com", referrer: "www.vm218.com/this_is_referer"2）示例匹配規則NGINXERROR_DATESTAMP %{YEAR}/%{MONTHNUM}/%{MONTHDAY} %{TIME}NGINXERROR_PID (?:[0-9]+#[0-9]+\:)NGINXERROR_TID (?:\*[0-9]+)NGINXERROR %{NGINXERROR_DATESTAMP:timestamp} \[%{LOGLEVEL:loglevel}\] %{NGINXERROR_PID:pid} %{NGINXERROR_TID:tid} %{GREEDYDATA:errormsg}, client: %{IPORHOST:clientip}, server: %{HOSTNAME:server}, request: %{QS:request}(?:, upstream: %{QS:upstream})?, host: \"%{HOSTNAME:hostname}\"(?:, referrer: (?:"(?:%{URI:referrer}|-)"|%{QS:referrer}))?注：特別說明，“%{NGINXERROR_PID:pid} %{NGINXERROR_TID:tid}” 這一段只是爲了匹配“11990#0: *6718852”並打算丟棄掉，並非實際需要這2個字段。3）使用工具測試http://grokconstructor.appspot.com/do/match4）結果------------------------------------------------------------------2016/09/21 10:34:41 [error] 11990#0: *6718852 no live upstreams while connecting to upstream, client: 10.50.200.219, server: c.test.com, request: "GET http://www.work.com/cgi-bin/common/attr?id=260714&r=0.1011513018987016 HTTP/1.1", upstream: "http://upstream_backend/cgi-bin/common/attr?id=260714&r=0.1011513018987016", host: "www.work.com"【MATCHED】request "GET·http://www.work.com/cgi-bin/common/attr?id=260714&r=0.1011513018987016·HTTP/1.1"server
c.test.comhostname www.work.comtimestamp
2016/09/21·10:34:41clientip
10.50.200.219port pid
11990#0:upstream "http://upstream_backend/cgi-bin/common/attr?id=260714&r=0.1011513018987016"referrer tid
*6718852errormsg
no·live·upstreams·while·connecting·to·upstreamloglevel
error==================================================================2016/09/21 11:26:41 [error] 1614#0: *25584123 open() "/data/www/www.test.com/404.html" failed (2: No such file or directory), client: 10.50.200.218, server: www.test.com, request: "GET /action.do?event=login&user=vm218&sid=15120ae826ca2290514bcd5b HTTP/1.1", host: "www.test.com", referrer: "www.vm218.com/this_is_referer"【MATCHED】request "GET·/action.do?event=login&user=vm218&sid=15120ae826ca2290514bcd5b·HTTP/1.1"server
www.test.comhostname www.test.comtimestamp
2016/09/21·11:26:41clientip
10.50.200.218port pid
1614#0:upstream referrer "www.vm218.com/this_is_referer"tid
*25584123errormsg open()·"/data/www/www.test.com/404.html"·failed·(2:·No·such·file·or·directory)loglevel
error ------------------------------------------------------------------5）應用【服務端】a、增加上述匹配規則到配置文件[root@vm220 ~]# cat /etc/logstash/patterns.d/extra_patternsNGINXERROR_DATESTAMP %{YEAR}/%{MONTHNUM}/%{MONTHDAY} %{TIME}NGINXERROR_PID (?:[0-9]+#[0-9]+\:)NGINXERROR_TID (?:\*[0-9]+)NGINXERROR %{NGINXERROR_DATESTAMP:timestamp} \[%{LOGLEVEL:loglevel}\] %{NGINXERROR_PID:pid} %{NGINXERROR_TID:tid} %{GREEDYDATA:errormsg}, client: %{IPORHOST:clientip}, server: %{HOSTNAME:server}, request: %{QS:request}(?:, upstream: %{QS:upstream})?, host: \"%{HOSTNAME:hostname}\"(?:, referrer: (?:"(?:%{URI:referrer}|-)"|%{QS:referrer}))?b、調整對應的 logstash 配置[root@vm220 ~]# vim /etc/logstash/conf.d/filebeat.conf 在 filter 這一段增加： if[type] =~ "NginxError-" { grok { patterns_dir => ["/etc/logstash/patterns.d"] match => { "message" => "%{NGINXERROR}" } } date { match => [ "timestamp", "YYYY/MM/dd HH:mm:ss" ] remove_field => [ "timestamp" ] } }在 output 這一段增加： if[type] == "NginxError-www.test.com" { elasticsearch { hosts => ["10.50.200.218:9200", "10.50.200.219:9200", "10.50.200.220:9200"] manage_template => false index => "%{[@metadata][beat]}-nginxerror-www.test.com-%{+YYYY.MM.dd}" document_type => "%{[@metadata][type]}" } } if[type] == "NginxError-www.work.com" { elasticsearch { hosts => ["10.50.200.218:9200", "10.50.200.219:9200", "10.50.200.220:9200"] manage_template => false index => "%{[@metadata][beat]}-nginxerror-www.work.com-%{+YYYY.MM.dd}" document_type => "%{[@metadata][type]}" } }c、重啓 logstash 服務[root@vm220 ~]# service logstash restart【客戶端】d、調整對應的 filebeat 配置[root@vm49 ~]# vim /etc/filebeat/filebeat.yml（增加對應的 prospectors 來收集日誌，打上 type 標記，其他配置略） prospectors: - paths: - /var/log/nginx/error_www.test.com*.log input_type: log document_type: NginxError-www.test.com - paths: - /var/log/nginx/error_www.work.com*.log input_type: log document_type: NginxError-www.work.come、重啓 filebeat 服務[root@vm49 ~]# service filebeat restart4、怎麼調試問題直接改變 output 到 stdout 即可查看輸出是否符合需求。output { if[type] =~ "NginxAccess-" { stdout { codec => rubydebug } } if[type] =~ "NginxError-" { stdout { codec => rubydebug } }}ZYXW、參考1、官網https://www.elastic.co/guide/en/logstash/current/introduction.htmlhttps://www.elastic.co/guide/en/logstash/current/getting-started-with-logstash.htmlhttps://www.elastic.co/guide/en/logstash/current/installing-logstash.htmlhttps://www.elastic.co/guide/en/logstash/current/first-event.htmlhttps://www.elastic.co/guide/en/logstash/current/advanced-pipeline.htmlhttps://www.elastic.co/guide/en/logstash/current/pipeline.htmlhttps://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.htmlhttps://www.elastic.co/guide/en/logstash/current/plugins-filters-date.html2、ELK中文http://kibana.logstash.es/content/http://kibana.logstash.es/content/logstash/plugins/filter/grok.htmlhttp://kibana.logstash.es/content/beats/file.html3、用ELK搭建簡單的日誌收集分析系統http://blog.csdn.net/lzw_2006/article/details/512800584、logstash host field is 0.0.0.0http://www.v1en.com/2016/03/31/logstash-host-field-is-0-0-0-0/5、grokhttps://github.com/logstash-plugins/logstash-patterns-core/blob/master/patterns/grok-patternshttp://grokconstructor.appspot.com/do/match