做elastic stack集羣部署
主機1：192.168.31.200 kibana
主機2：192.168.31.201
主機3：192.168.31.203
首先去官網下載對應的包，這裏我們選擇rpm安裝
https://www.elastic.co/downloads
ElasticSearch 5的程序環境：
配置文件：
/etc/elasticsearch/elasticsearch.yml
/etc/elasticsearch/jvm.options
/etc/elasticsearch/log4j2.properties
Unit File：elasticsearch.service
程序文件：
/usr/share/elasticsearch/bin/elasticsearch
/usr/share/elasticsearch/bin/elasticsearch-keystore：
/usr/share/elasticsearch/bin/elasticsearch-plugin：管理插件程序
編輯配置文件
vim /etc/elasticsearch/elasticsearch.yml
cluster.name: myels
node.name: node1
path.data: /els/data
path.logs: /els/log
network.host: 192.168.31.200
discovery.zen.ping.unicast.hosts: ["node1", "node2","node3"]
discovery.zen.minimum_master_nodes: 2
vim vim /etc/elasticsearch/jvm.options
-Xms2g
-Xmx2g
#初始化分給它2g
然後創建對應的目錄
mkdir -pv /els/{data,log}
chown -R elasticsearch.elasticsearch /els
els的相關模塊說明
https://www.elastic.co/guide/en/elasticsearch/reference/current/modules.html
安裝完成後可以通過curl http://192.168.31.200:9200/_cat來查看狀態
curl http://192.168.31.200:9200/_cat/nodes?h=name,ip,port,uptime,jdk
node2 192.168.31.201 9300 26.5m 1.8.0_131
node3 192.168.31.203 9300 7.1m 1.8.0_131
node1 192.168.31.200 9300 34.5m 1.8.0_131
查看插件
/usr/share/elasticsearch/bin/elasticsearch-plugin list
5版本以後的插件可以作爲一個獨立的服務運行，這裏我們去github上下載然後安裝head。
https://github.com/mobz/elasticsearch-head.git
create a fork of elasticsearch-head on github
clone your fork to your machine
cd elasticsearch-head
npm install # downloads node dev dependencies
grunt dev # builds the distribution files, then watches the src directory for changes (if you have an warning like "Warning: Task “clean” failed. Use —force to continue.", well use —force ;) )
直接npm run start會佔據前臺，這裏我們可以使用nohup npm run start &運行於後臺
然後需要修改配置文件
http.cors.enabled: true
http.cors.allow-orign: "*"
自己上傳一個文檔測試，注意索引會自行創建
curl -XPUT 'node1:9200/students/major/1?pretty' -H 'Content-Type: application/json' -d '
{"name": "jerry", "age": 17, "course": "Pixie jianfa"}'
查看索引
curl 'node1:9200/_cat/indices'
curl -XGET 'node1:9200/students/_search?pretty'
curl -XGET 'node1:9200/_search/?q=course:shiba&pretty'
安裝kibana界面
rpm -ivh kibana-6.5.4-x86_64.rpm
vim /etc/kibana/kibana.yml
server.host: "192.168.31.200"
server.port: 5601
server.name: "node1"
elasticsearch.url: "http://node1:9200"
直接啓動即可
systemctl start kibana
我們再新添一臺nginx主機，然後在上面安裝filebeats和logstash軟件。
rpm -ivh filebeat-6.5.4-x86_64.rpm
vim /etc/filebeat/filebeat.yml
hosts: ["node1:9200", "node2:9200"]
可以看到filebeat已經將數據推送到elasticsesarch上了
那麼kibana上也可對數據進行處理了
但是filebeat處理數據的能力沒有logstash好，所以我們再增加一個logstash節點，這裏我們直接使用nginx主機
注意：logstash如果不使用logstash用戶可能會產生一些權
限衝突的問題。
logstash的各插件配置官方文檔https://www.elastic.co/guide/en/logstash/current/index.html
rpm -ivh logstash-6.5.4.rpm
vim /etc/logstash/conf.d/test.conf
input {
stdin{}
}
output {
stdout { codec => rubydebug }
}
檢測語法,去掉-t就可以直接運行了
/usr/share/logstash/bin/logstash --path.settings /etc/logstash/ -t -f /etc/logstash/conf.d/test.conf
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
然後我們就可以在終端直接輸入數據了
hello logstash
{
"@version" => "1",
"@timestamp" => 2019-01-15T12:35:51.470Z,
"host" => "node4.lvqing.com",
"message" => "hello logstash"
}
接下來我們配置logstash從beats讀取數據,當然輸出還是到屏幕上，稍後我們再進行配置輸出到elasticsearch上
input {
beats{
host => '0.0.0.0'
port => 5044
}
}
然後我們需要修改filebeats輸出的對象爲logstash
output.logstash:
hosts: ["192.168.31.204:5044"]
再啓動logstash
/usr/share/logstash/bin/logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/ceshi.conf
logstash就能收集到filebeat傳過來的日誌了,並且是已經幫我們切好片的
{
"prospector" => {
"type" => "log"
},
"input" => {
"type" => "log"
},
"host" => {
"os" => {
"version" => "7 (Core)",
"family" => "redhat",
"platform" => "centos",
"codename" => "Core"
},
"name" => "node4.lvqing.com",
"id" => "98b754e309454154b76d44862ecc843e",
"containerized" => true,
"architecture" => "x86_64"
},
"@timestamp" => 2019-01-15T13:51:36.416Z,
"beat" => {
"name" => "node4.lvqing.com",
"version" => "6.5.4",
"hostname" => "node4.lvqing.com"
},
"source" => "/var/log/nginx/access.log",
"tags" => [
[0] "beats_input_codec_plain_applied"
],
"@version" => "1",
"offset" => 2527,
"message" => "192.168.31.242 - - [15/Jan/2019:21:51:30 +0800] \"GET /dsa HTTP/1.1\" 404 3650 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36\" \"-\""
}
然後logstash相對於filebea強大的地方就是他的過濾器，這裏我們介紹使用grok插件,它已經事先定義好了正則，我們在使用的時候直接調用就可以了。
filter {
grok {
match => { "message" => ["%{IPORHOST:[nginx][access][remote_ip]} - %{DATA:[nginx][access][user_name]} \[%{HTTPDATE:[nginx
][access][time]}\] \"%{WORD:[nginx][access][method]} %{DATA:[nginx][access][url]} HTTP/%{NUMBER:[nginx][access][http_version]}\
" %{NUMBER:[nginx][access][response_code]} %{NUMBER:[nginx][access][body_sent][bytes]} \"%{DATA:[nginx][access][referrer]}\" \"
%{DATA:[nginx][access][agent]}\""] }
remove_field => "message"
}
date {
match => [ "[nginx][access][time]", "dd/MMM/YYYY:H:m:s Z" ]
remove_field => "[nginx][access][time]"
}
useragent {
source => "[nginx][access][agent]"
target => "[nginx][access][user_agent]"
remove_field => "[nginx][access][agent]"
}
geoip {
source => "[nginx][access][remote_ip]"
target => "geoip"
database => "/etc/logstash/GeoLite2-City.mmdb"
}
}
output {
elasticsearch {
hosts => ["node1:9200","node2:9200","node3:9200"]
index => "logstash-ngxaccesslog-%{+YYYY.MM.dd}"
}
}