kibana日誌收集

一、Kibana安裝
Kibana 是爲 Elasticsearch 設計的開源分析和可視化平臺。你可以使用 Kibana 來搜索,查看存儲在 Elasticsearch 索引中的數據並與之交互。你可以很容易實現高級的數據分析和可視化,以圖表的形式展現出來。

kiabana下載地址:https://artifacts.elastic.co/downloads/kibana/kibana-6.0.0-x86_64.rpm

[root@linux-node1 ~]# wget https://artifacts.elastic.co/downloads/kibana/kibana-6.0.0-x86_64.rpm
[root@linux-node1 ~]# yum install -y kibana-6.0.0-x86_64.rpm 
[root@linux-node1 ~]# vim /etc/kibana/kibana.yml 
[root@linux-node1 ~]# grep "^[a-Z]" /etc/kibana/kibana.yml 
server.port: 5601        #監聽端口
server.host: "192.168.56.11"      #監聽IP地址,建議內網ip
elasticsearch.url: "http://192.168.56.11:9200"       #elasticsearch連接kibana的URL,也可以填寫192.168.56.12,因爲它們是一個集羣
[root@linux-node1 ~]# systemctl enable kibana
Created symlink from /etc/systemd/system/multi-user.target.wants/kibana.service to /etc/systemd/system/kibana.service.
[root@linux-node1 ~]# systemctl start kibana
監聽端口爲:5601
[root@linux-node1 ~]# ss -tnl
State       Recv-Q Send-Q                                                 Local Address:Port                                                                Peer Address:Port              
LISTEN      0      128                                                                *:9100                                                                           *:*                  
LISTEN      0      128                                                                *:22                                                                             *:*                  
LISTEN      0      100                                                        127.0.0.1:25                                                                             *:*                  
LISTEN      0      128                                                    192.168.56.11:5601                                                                           *:*                  
LISTEN      0      128                                             ::ffff:192.168.56.11:9200                                                                          :::*                  
LISTEN      0      128                                             ::ffff:192.168.56.11:9300                                                                          :::*                  
LISTEN      0      128                                                               :::22                                                                            :::*                  
LISTEN      0      100                                                              ::1:25                                                                            :::*                  
LISTEN      0      80                                                                :::3306                                                                          :::*                  

瀏覽器訪問192.168.56.11:5601,如圖:
在這裏插入圖片描述
可以通過http://192.168.56.11:5601/status 來查看看是否正常,如果不正常,是無法進入到上圖界面
在這裏插入圖片描述

二、通過配置logstash文件收集message日誌
1、Kibana展示上一節的日誌
在Kibana上展示上一節收集的日誌信息,添加索引,如圖:
在這裏插入圖片描述
點擊“discover”查看收集的信息,如圖:
在這裏插入圖片描述

2、使用logstash配置文件收集messages日誌
編輯logstash的配置文件:

[root@linux-node1 ~]# vim /etc/logstash/conf.d/system.conf
input {
  file {
    path => "/var/log/messages"     #日誌路徑
    type => "systemlog"      #類型
    start_position => "beginning"    #logstash 從什麼位置開始讀取文件數據,默認是結束位置,也就是說 logstash 進程會以類似 tail -F 的形式運行。如果你是要導入原有數據,把這個設定改成"beginning",logstash 進程就從頭開始讀取,類似 less +F 的形式運行。
    stat_interval => "2"  #logstash 每隔多久檢查一次被監聽文件狀態(是否有更新) ,默認是 1 秒。
  }
}
 
output {
  elasticsearch {
    hosts => ["192.168.56.11:9200"]      #指定hosts
    index => "logstash-systemlog-%{+YYYY.MM.dd}"    #索引名稱
  }
 
}
[root@linux-node1 ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/system.conf -t     #檢測配置文件是否有語法錯誤
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
Configuration OK
[root@linux-node1 ~]# ll /var/log/messages 
-rw-------. 1 root root 791209 12月 27 11:43 /var/log/messages

#這裏可以看到該日誌文件是600權限,而elasticsearch是運行在elasticsearch用戶下,這樣elasticsearch是無法收集日誌的。所以這裏需要更改日誌的權限,否則會報權限拒絕的錯誤。在日誌中查看/var/log/logstash/logstash-plain.log 是否有錯誤

[root@linux-node1 ~]# chmod 644 /var/log/messages 
[root@linux-node1 ~]# systemctl restart logstash

在管理界面查看是否有相應的索引(logstash-systemlog-2017.12.27),如圖:
在這裏插入圖片描述
添加到Kibana中展示,創建索引:
在這裏插入圖片描述
查看日誌
在這裏插入圖片描述

三、使用一個配置文件收集多個日誌
修改logstash的配置文件,這裏增加收集數據庫mariadb的日誌:

[root@linux-node1 ~]# vim /etc/logstash/conf.d/system.conf 
input {
  file {
        path => "/var/log/messages"
        type => "systemlog"
        start_position => "beginning"
        stat_interval => "2"
  }
  file {
        path => "/var/log/mariadb/mariadb.log"
        type => "mariadblog"
        start_position => "beginning"
        stat_interval => "2"
  }
}
 
output {
  if [type] == "systemlog" {       #使用if來判斷類型,並輸出到elasticsearch和file,展示一個out可以作多樣輸出
  elasticsearch {
        hosts => ["192.168.56.11:9200"]
        index => "logstash-systemlog-%{+YYYY.MM.dd}"
  }
  file {
        path => "/tmp/logstash-systemlog-%{+YYYY.MM.dd}"
 
  }}
  if [type] == "mariadblog" {
  elasticsearch {
        hosts => ["192.168.56.11:9200"]
        index => "logstash-mariadblog-%{+YYYY.MM.dd}"
  }
  file {
        path => "/tmp/logstash-mariadblog-%{+YYYY.MM.dd}"
  }}
 
}

配置文件檢測語法是否正常:

[root@linux-node1 ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/system.conf -t
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
Configuration OK

重啓logstash:

[root@linux-node1 ~]# systemctl restart logstash
修改mariadb的日誌權限:
[root@linux-node1 ~]# ll /var/log/mariadb/ -d
drwxr-x--- 2 mysql mysql 24 12月  4 17:43 /var/log/mariadb/
[root@linux-node1 ~]# chmod 755 /var/log/mariadb/
[root@linux-node1 ~]# ll /var/log/mariadb/mariadb.log 
-rw-r----- 1 mysql mysql 114993 12月 27 14:23 /var/log/mariadb/mariadb.log
[root@linux-node1 ~]# chmod 644 /var/log/mariadb/mariadb.log 

通過head插件查看:
在這裏插入圖片描述
查看是否在/tmp下收集到了日誌數據

[root@linux-node1 ~]# ll /tmp/logstash-*
-rw-r--r-- 1 logstash logstash 288449 12月 27 14:27 /tmp/logstash-mariadblog-2017.12.27
-rw-r--r-- 1 logstash logstash  53385 12月 27 14:28 /tmp/logstash-systemlog-2017.12.27

Kibana創建索引:
在這裏插入圖片描述

發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章