logstash之filter-Geoip尋找日誌ip的經緯度和城市及Key-value拆分

geoip查詢

logstash可以將nginx的非格式化日誌進行格式化(參考https://blog.csdn.net/weixin_44062339/article/details/103221269)，那麼在nginx的日誌中有IP；往往會根據ip定位當前的地理位置，
然後在kibana上以高德地圖做展示；
Vim /conf/template/geoip.conf
啓動：bin/logstash -f /usr/local/elk/logstash-5.5.2/conf/template/geoip.conf
向控制檯輸入nginx日誌：

119.151.192.24 - - [10/May/2018:12:12:40 +0800] "GET /plugins/ml/ml.svg HTTP/1.1" 304 0 "http://hadoop01/app/kibana" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.139 Safari/537.36" "-"

截圖展示：

爲了更準確的定位ip的經緯度，可以下載GeoLite2-City.mmdb的ip-經緯度庫
下載地址：
http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.mmdb.gz

例子1

input{
        stdin{}
}

filter{
        grok{
                match=>{
                        "message" => "%{IPORHOST:remote_addr} - %{NGUSER:remote_addr} \[%{HTTPDATE:time_local}\] \"(?:%{WORD:request} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:status} %{NUMBER:body_bytes_sent} %{QS:http_referer} %{QS:http_user_agent} %{NOTSPACE:http_x_forwarded_for}"
                }
        }
        geoip{
                source => "remote_addr"
                database => "/export/servers/elk/logstash-5.5.2/GeoLite2-City.mmdb"
        }
}

output{
        stdout{
                codec=> rubydebug
        }
}

例子2

input{
        stdin{}
}

filter{
        grok{
                match=>{
                        "message" => "%{IPORHOST:remote_addr} - %{NGUSER:remote_addr} \[%{HTTPDATE:time_local}\] \"(?:%{WORD:request} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:status} %{NUMBER:body_bytes_sent} %{QS:http_referer} %{QS:http_user_agent} %{NOTSPACE:http_x_forwarded_for}"
                }
        }
        geoip{
                source => "remote_addr"
                database => "/export/servers/elk/logstash-5.5.2/GeoLite2-City.mmdb"
                target => "geoip"
                add_field => ["[geoip][coordinates]" , "&{[geoip][latitude]}"]
                add_field => ["[geoip][coordinates]" , "&{[geoip][longitude]}"]
                fields => ["country_name" , "region_name" , "city_name" , "latitude" , "longitude"]
                #remove_field => {[geoip][latitude] , []}   
     }
}

output{
        stdout{
                codec=> rubydebug
        }
}

Key-value拆分

在採集的日誌中，往往出現類似於這樣的URL：

https://www.baidu.com/s?wd=哈哈，這就是測試&a=1&b=2&c=3&d=4&e=5

類似這種url，字段的信息是按照&拼接而成的，所以需要把這些url進行拆分
Vim k_v_split.conf

input {
    stdin {
    }
}
filter {
         kv {
                prefix => "key_"
                source => "message"
                field_split => "&" 
                value_split => "="
            }
}
output {
    stdout{codec=>rubydebug}
}

啓動：bin/logstash -f /usr/local/elk/logstash-5.5.2/conf/template/k_v_split.conf
向控制檯輸入：

https://www.baidu.com/s?wd=哈哈，這就是測試&a=1&b=2&c=3&d=4&e=5

結果截圖：