logstash信息採集

信息採集

採集之前完成elasticsearch的搭建。
elasticsearch的搭建

server1：

rpm -ivh logstash-2.3.3-1.noarch.rpm
cd /opt/logstash/bin/
[root@server1 bin]# /opt/logstash/bin/logstash -e 'input { stdin {} } output { stdout {} }'  #輸出終端
[root@server1 bin]# /opt/logstash/bin/logstash -e 'input { stdin {} } output { stdout { codec => rubydebug } }' # 更改輸出格式
[root@server1 bin]# /opt/logstash/bin/logstash -e 'input { stdin {} } output { elasticsearch { hosts => ["172.25.11.1"] index => "logstash-%{+YYYY.MM.dd}" }  stdout { codec => rubydebug } }'  # 輸出到es

編寫文件，採集信息（文件的名字是*.conf）

vim /etc/lagstash/es.conf
input {
        stdin {}
}
output {
        elasticsearch {
                hosts => ["172.25.11.1"]
                index => "logstash-%{+YYYY.MM.dd}"
        }
        stdout {
                codec => rubydebug
        }
        file {
                path => "/tmp/demo.file"
                codec => line { format => "custom format: %{message}" }
        }
}

/opt/logstash/bin/logstash -f /etc/logstash/es.conf

終端輸出：

elasticsearch輸出：


文件輸出：

將系統日誌文件輸出到elasticsearch中

vim /etc/lagstash/message.conf
input {
        file {
                path => "/var/log/messages"
                start_position => beginning
        }
}
output {
        elasticsearch {
                hosts => ["172.25.11.1"]
                index => "message-%{+YYYY.MM.dd}"
        }
}

/opt/logstash/bin/logstash -f /etc/logstash/es.conf

[root@server1 ~]# cat .sincedb_452905a167cf4509fd08acb964fdb20c  # elasticsearch 的讀取記錄，記錄文件讀取的內容地址
1044500 0 64768 32556

rsyslog 遠程控制

server1:
vim /etc/logstash/message.conf
input {
        syslog {
                port => 514
        }
}
output {
#       elasticsearch {
#               hosts => ["172.25.11.1"]
#               index => "message-%{+YYYY.MM.dd}"
#       }
        stdout {
                codec => rubydebug
        }
}

server1上沒有514端口

/opt/logstash/bin/logstash -f /etc/logstash/es.conf
netstat -antulp |grep :514

server2:
vim /etc/rsyslog.conf
*.*     @@172.25.11.1:514
/etc/init.d/rsyslog restart

多行日誌採集

input {
        file {
                path => "/var/log/elasticsearch/my-es.log"
                start_position => "beginning"
        }
}

filter {
        multiline {
#               type => "type"
                pattern => "^\["
                negate => true
                what => "previous"
        }
}

output {
        elasticsearch {
                hosts => ["172.25.11.1"]
                index => "message-%{+YYYY.MM.dd}"
        }
        stdout {
                codec => rubydebug
        }
}

多行成爲一行輸出

apache信息採集

vim /etc/logstash/conf.d/test.conf
input {
        file {
                path => ["/var/log/httpd/access_log","/var/log/httpd/error_log"]
                start_position => "beginning"
        }
}

filter {
        grok {
                match => { "message" => "%{COMBINEDAPACHELOG}" }
        }
}
output {
        elasticsearch {
                hosts => ["172.25.11.1"]
                index => "apache-%{+YYYY.MM.dd}"
        }
        stdout {
                codec => rubydebug
        }
}