Ubuntu下整合Filebeat和Logstash，讀取nginx的日誌

1.配置Logstash：在logstash目錄下新建配置文件並編輯。

命令：sudo vi test-pipeline.conf

input {
    beats {
        port => "5044"
    }
}
output {
    stdout { codec => rubydebug }
}

2.測試配置文件：

測試配置文件是否正確

# 啓動 --config.test_and_exit 

 

# 正式啓動 --config.reload.automatic 熱加載配置文件，修改配置文件後無需重新啓動。

 

3.配置filebeat：在filebeat目錄下新建配置文件並編輯。

命令：sudo vi test-nginx.yml

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/access.log
  tags: ["log"]
  fields:
    from: nginx
  fields_under_root: false
output.logstash:
  hosts: ["10.0.2.15:5044"]

4.啓動filebeat

命令：sudo ./filebeat -e -c test-nginx.yml

如下圖：證明與logstash成功建立連接。

 

5.訪問nginx

此時可以訪問nginx服務（之前已經通過service nginx start啓動nginx服務）。

 

出現如下信息，則證明在控制檯輸出了nginx的訪問日誌。

產生信息如下：

hong@hong-VirtualBox:/usr/local/work/logstash-6.5.4$ sudo bin/logstash -f test-pipeline.conf --config.reload.automatic
Sending Logstash logs to /usr/local/work/logstash-6.5.4/logs which is now configured via log4j2.properties
[2020-04-21T22:20:55,233][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2020-04-21T22:20:55,266][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.5.4"}
[2020-04-21T22:21:02,949][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2020-04-21T22:21:03,551][INFO ][logstash.inputs.beats    ] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[2020-04-21T22:21:03,585][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x598783ac run>"}
[2020-04-21T22:21:03,704][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2020-04-21T22:21:03,808][INFO ][org.logstash.beats.Server] Starting server on port: 5044
[2020-04-21T22:21:04,239][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
{
       "message" => "::1 - - [20/Apr/2020:20:02:16 +0800] \"GET / HTTP/1.1\" 200 396 \"-\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.113 Safari/537.36\"",
        "source" => "/var/log/nginx/access.log",
         "input" => {
        "type" => "log"
    },
        "offset" => 0,
    "prospector" => {
        "type" => "log"
    },
      "@version" => "1",
          "host" => {
        "name" => "hong-VirtualBox"
    },
    "@timestamp" => 2020-04-21T14:27:15.222Z,
          "beat" => {
         "version" => "6.5.4",
            "name" => "hong-VirtualBox",
        "hostname" => "hong-VirtualBox"
    },
          "tags" => [
        [0] "log",
        [1] "beats_input_codec_plain_applied"
    ],
        "fields" => {
        "from" => "nginx"
    }
}
{
       "message" => "::1 - - [20/Apr/2020:20:02:16 +0800] \"GET /favicon.ico HTTP/1.1\" 404 209 \"http://localhost/\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.113 Safari/537.36\"",
    "prospector" => {
        "type" => "log"
    },
        "source" => "/var/log/nginx/access.log",
        "offset" => 174,
         "input" => {
        "type" => "log"
    },
      "@version" => "1",
          "host" => {
        "name" => "hong-VirtualBox"
    },
    "@timestamp" => 2020-04-21T14:27:15.223Z,
          "beat" => {
         "version" => "6.5.4",
            "name" => "hong-VirtualBox",
        "hostname" => "hong-VirtualBox"
    },
          "tags" => [
        [0] "log",
        [1] "beats_input_codec_plain_applied"
    ],
        "fields" => {
        "from" => "nginx"
    }
}
{
       "message" => "::1 - - [21/Apr/2020:22:39:11 +0800] \"GET / HTTP/1.1\" 304 0 \"-\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.113 Safari/537.36\"",
        "source" => "/var/log/nginx/access.log",
         "input" => {
        "type" => "log"
    },
        "offset" => 375,
    "prospector" => {
        "type" => "log"
    },
      "@version" => "1",
          "host" => {
        "name" => "hong-VirtualBox"
    },
    "@timestamp" => 2020-04-21T14:39:15.254Z,
          "beat" => {
         "version" => "6.5.4",
            "name" => "hong-VirtualBox",
        "hostname" => "hong-VirtualBox"
    },
          "tags" => [
        [0] "log",
        [1] "beats_input_codec_plain_applied"
    ],
        "fields" => {
        "from" => "nginx"
    }
}
{
       "message" => "::1 - - [21/Apr/2020:22:39:12 +0800] \"GET /favicon.ico HTTP/1.1\" 404 209 \"http://localhost/\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.113 Safari/537.36\"",
    "prospector" => {
        "type" => "log"
    },
        "source" => "/var/log/nginx/access.log",
        "offset" => 547,
         "input" => {
        "type" => "log"
    },
      "@version" => "1",
          "host" => {
        "name" => "hong-VirtualBox"
    },
    "@timestamp" => 2020-04-21T14:39:15.255Z,
          "beat" => {
         "version" => "6.5.4",
            "name" => "hong-VirtualBox",
        "hostname" => "hong-VirtualBox"
    },
          "tags" => [
        [0] "log",
        [1] "beats_input_codec_plain_applied"
    ],
        "fields" => {
        "from" => "nginx"
    }
}