006-ELK的二進制安裝

二進制安裝ELKstack

本次搭建屬於單點，在同一臺機器上進行安裝

基礎組件部署

 curl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
 wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo
 yum makecache
 yum install wget vim lsof net-tools lrzsz -y
 yum -y install ntp
 systemctl enable ntpd
 systemctl start ntpd
 ntpdate -u cn.pool.ntp.org
 hwclock --systohc
 timedatectl set-timezone Asia/Shanghai
 systemctl stop firewalld
 systemctl disable firewalld
 setenforce 0
 sed -i '/SELINUX/s/enforcing/disabled/' /etc/selinux/config
 echo '
* hard nofile 65536
* soft nofile 65536
* soft nproc  65536
* hard nproc  65536
'>>/etc/security/limit.conf
echo '
vm.max_map_count = 262144
net.core.somaxconn=65535
net.ipv4.ip_forward = 1
'>>/etc/sysctl.conf
sysctl -p
yum install -y java
mkdir /root/elk -p;cd /root/elk 
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.5.0.tar.gz
wget https://artifacts.elastic.co/downloads/kibana/kibana-5.5.0-linux-x86_64.tar.gz
wget https://artifacts.elastic.co/downloads/logstash/logstash-5.5.0.tar.gz

安裝Elasticsearch

mkdir /data/{es-data,es-logs} -p
tar xf elasticsearch-5.5.0.tar.gz -C /usr/local/
ln -s /usr/local/elasticsearch-5.5.0 /usr/local/elasticsearch
cat>>/usr/local/elasticsearch-5.5.0/config/elasticsearch.yml<<EOF
node.name: my-es
path.data: /data/es-data
path.logs: /data/es-logs
http.host: "192.168.83.60"
http.cors.enabled: true    
http.cors.allow-origin: "*"
EOF
useradd elk
chown -R elk /data/es-*
chown -R elk /usr/local/elasticsearch-5.5.0/

啓動es

/usr/local/elasticsearch/bin/elasticsearch -d -d 後臺運行 第一次不加-d參數，則可以看到啓動的報錯信息

[root@60 data]# lsof -i:9200
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
java    20586  elk  132u  IPv6  61210      0t0  TCP 60:wap-wsp (LISTEN)     
[root@60 data]# curl 192.168.83.60:9200
{
  "name" : "my-es",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "cYF13nSqQAi91gHnn9I_bg",
  "version" : {
    "number" : "5.5.0",
    "build_hash" : "260387d",
    "build_date" : "2017-06-30T23:16:05.735Z",
    "build_snapshot" : false,
    "lucene_version" : "6.6.0"
  },
  "tagline" : "You Know, for Search"
}

相關配置信息解說文章： https://blog.csdn.net/laoyang360/article/details/72850834

ES常用命令

curl -XDELETE 'http://host.IP.address:9200/logstash-*'  刪除索引(後面爲索引名稱)
curl -XGET 'host.IP.address:9200/_cat/health?v&pretty'  查看集羣狀態
curl -XGET 'host.IP.address:9200/_cat/indices?v&pretty'    查看索引

logstash安裝

tar xf logstash-5.5.0.tar.gz -C /usr/local/
ln -s /usr/local/logstash-5.5.0 /usr/local/logstash
一般情況下，我們可以不配置logstash直接就可以啓動，logstash下有一個叫logstash.yml的文件，裏面可以對logstash做一些簡單的優化
vim /usr/local/logstash/config/logstash.yml
config.reload.automatic: true    #開啓配置文件自動加載
config.reload.interval: 10    #定義配置文件重載時間週期

創建配置文件

mkdir /usr/local/logstash/conf/
vim /usr/local/logstash/conf/nginx.conf
input {
    file {
        path => " /usr/local/nginx/logs/access.log"
        start_position => "beginning"
        }
}
filter {
}
output {
        #stdout { codec=> rubydebug }
    elasticsearch {
        hosts => ["192.168.83.60:9200"]
        index => "static_nginx_access-%{+YYYY.MM.dd}"
 }
}

#這裏面實際只是定義了一個nginx log路徑
#stdout { codec=> rubydebug }  將內容打印出來，方便測試
#host 填寫es主機
#index 索引名稱

nginx中日誌配置

    log_format json '{"@timestamp":"$time_iso8601",'
                  '"host":"$server_addr",'
                  '"clientip":"$remote_addr",'
                  '"remote_user":"$remote_user",'
                  '"request":"$request",'
                  '"http_user_agent":"$http_user_agent",'
                  '"size":$body_bytes_sent,'
                  '"responsetime":$request_time,'
                  '"upstreamtime":"$upstream_response_time",'
                  '"upstreamhost":"$upstream_addr",'
                  '"http_host":"$host",'
                  '"requesturi":"$request_uri",'
                  '"url":"$uri",'
                  '"domain":"$host",'
                  '"xff":"$http_x_forwarded_for",'
                  '"referer":"$http_referer",'
                  '"status":"$status"}';
    access_log /usr/local/nginx/logs/access.log json;

測試logstash配置文件是否正常

儘量使用logstash chown -RHL elk.elk /usr/local/logstash su - elk

[elk@60 ~]$ /usr/local/logstash/bin/logstash -f /usr/local/logstash/conf/nginx.conf  -t
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.
Sending Logstash's logs to /usr/local/logstash/logs which is now configured via log4j2.properties
[2019-12-12T02:39:24,954][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/usr/local/logstash/data/queue"}
[2019-12-12T02:39:24,956][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/local/logstash/data/dead_letter_queue"}
Configuration OK
[2019-12-12T02:39:25,050][INFO ][logstash.runner          ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash

kibana安裝

cd /root/elk
tar xf kibana-5.5.0-linux-x86_64.tar.gz -C /usr/local/
cd /usr/local
ln -nfsv kibana-5.5.0-linux-x86_64 kibana
chown -RHL elk.elk kibana-5.5.0-linux-x86_64
vim /usr/local/kibana/config/kibana.yml
server.port: 5601
server.host: "192.168.83.60"
elasticsearch.url: "http://192.168.83.60:9200"

啓動kibana

su - elk
前臺啓動查看是否異常
/usr/local/kibana/bin/kibana
後臺啓動
nohup /usr/local/kibana/bin/kibana &
查看索引
[root[@60](https://my.oschina.net/bingo60) ~]# curl -XGET '192.168.83.60:9200/_cat/indices?v&pretty'
health status index                          uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   static_nginx_access-2019.12.11 7mbLeNcfSM-_Z-G3uW3_2Q   5   1          3            0     14.8kb         14.8kb
yellow open   .kibana                        cCBXRM7dREO4imbiIaUsww   1   1          1            0      3.2kb          3.2kb
其中static_nginx_access-2019.12.11爲索引名稱

kibana創建索引

Elasticsearch Head插件安裝

說明：界面化的集羣操作和管理工具，可以對集羣進行傻瓜式操作 顯示集羣的拓撲,並且能夠執行索引和節點級別操作 搜索接口能夠查詢集羣中原始json或表格格式的檢索數據 能夠快速訪問並顯示集羣的狀態 有一個輸入窗口,允許任意調用RESTful API。這個接口包含幾個選項,可以組合在一起以產生有趣的結果

官方文檔：https://github.com/mobz/elasticsearch-head

rpm -vih https://rpm.nodesource.com/pub_10.x/el/7/x86_64/nodejs-10.15.3-1nodesource.x86_64.rpm
npm install -g pm2
cd /usr/local
git clone https://github.com/mobz/elasticsearch-head.git
cd elasticsearch-head/
#需要注意的是執行以下命令需要在elasticsearch-head目錄下執行
npm install    
npm run start      
#後臺運行nohup npm run start &，或者可以針對npm的啓停做一個shell啓停腳本
vim /usr/local/elasticsearch/elasticsearch.yml
http.cors.enabled: true        # elasticsearch中啓用CORS
http.cors.allow-origin: "*"    #允許訪問的IP地址段，* 爲所有IP都可以訪問

修改啓動默認端口 vim /usr/local/elasticsearch-head/Gruntfile.js

修改連接的ip地址 vim /usr/local/elasticsearch-head/_site/app.js

修改完成後直接要npm run start &即可