ELK使用redis做中間鍵，減少整個elk的壓力

1、安裝redis，安裝部署過程這裏不介紹（這裏redis的ip爲192.168.0.197，端口爲6379）

2、配置logstash的收集數並導入redis的配置文件和從redis中獲取數據導入elasticsearch的兩個配置文件

1、配置導入redis數據的配置文件，並啓動logstash服務

input {
      file{
        path => "/var/log/messages"             #指定要收集的日誌文件
  type => "system"  #指定類型爲system，可以自定義，type值和output{ } 中的type對應即可
        start_position => "beginning"       #從開始處收集
        }
      file{
        path => "/home/otc/otc-web/logs/gxzx-otc-web.log"
        type => "otc"
        start_position => "beginning"
         }
       file{ 
        path => "/home/deploy/financial-management/logs/gxzx-fin-web.log"
        type => "financial"
        start_position => "beginning"
          }
       file{
        path => "/home/deploy/activity_service/logs/gxzx-act-web.log"
        type => "act"
        start_position => "beginning"
       }     
       file{
        path => "/home/deploy/mining/logs/gxzx-min-web.log"
        type => "mining"
        start_position => "beginning"
}     
       }
output {
        if [type] == "system" {
             redis {
                 host => "192.168.0.197"
                 password => '901Bcpct'
                 port => "6379"
                 db => "3"
                 data_type => "list"
                 key => 'logs_system'
             }
        }
        if [type] == "otc" {
             redis {
                 host => "192.168.0.197"
                 password => '901Bcpct'
                 port => "6379"
                 db => "3"
                 data_type => "list"
                 key => 'logs_otc'
             }
        }
        if [type] == "financial" {
             redis {
                 host => "192.168.0.197"
                 password => '901Bcpct'
                 port => "6379"
                 db => "3"
                 data_type => "list"
                 key => 'logs_financial'
             }
        }
        if [type] == "act" {
             redis {
                 host => "192.168.0.197"
                 password => '901Bcpct'
                 port => "6379"
                 db => "3"
                 data_type => "list"
                 key => 'logs_act'
             }
        }
        if [type] == "mining" {
             redis {
                 host => "192.168.0.197"
                 password => '901Bcpct'
                 port => "6379"
                 db => "3"
                 data_type => "list"
                 key => 'logs_mining'
             }
        }
}

啓動服務：

在源碼安裝的logstash的bin目錄下執行，後面還要加一路徑，默認是當前執行這的家目錄下
 ./logstash -f input_redis.conf &

2、配置從redis導出數據的配置文件

input {
        beats {
            port => 5045
        }
        if [type] == "system" {
             redis {
                 host => "192.168.0.197"
                 password => '901Bcpct'
                 port => "6379"
                 db => "3"
                 data_type => "list"
                 key => 'logs_system'
        }
        }
        if [type] == "otc" {
             redis {
                 host => "192.168.0.197"
                 password => '901Bcpct'
                 port => "6379"
                 db => "3"
                 data_type => "list"
                 key => 'logs_otc'
             }
        }
        if [type] == "financial" {
             redis {
                 host => "192.168.0.197"
                 password => '901Bcpct'
                 port => "6379"
                 db => "3"
                 data_type => "list"
                 key => 'logs_financial'
             }
        }
        if [type] == "act" {
             redis {
                 host => "192.168.0.197"
                 password => '901Bcpct'
                 port => "6379"
                 db => "3"
                 data_type => "list"
                 key => 'logs_act'
             }
        }
        if [type] == "mining" {
             redis {
                 host => "192.168.0.197"
                 password => '901Bcpct'
                 port => "6379"
                 db => "3"
                 data_type => "list"
                 key => 'logs_mining'
             }
        }
}


output {
        if [type] == "system" {           #如果type爲system，
           elasticsearch {                  #就輸出到Elasticsearch服務器
             hosts => ["192.168.0.117:9200"]               #Elasticsearch監聽地址及端口
             index => "system-%{+YYYY.MM.dd}"           #指定索引格式
                }
             }
        if [type] == "otc" {
  elasticsearch {
             hosts => ["192.168.0.117:9200"]
             index => "nginx_otc-%{+YYYY.MM.dd}"
               }
             } 
        if [type] == "financial" {
          elasticsearch {
             hosts => ["192.168.0.117:9200"]
             index => "nginx_financial-%{+YYYY.MM.dd}"
             }
          }  
        if [type] == "act" {
          elasticsearch {
             hosts => ["192.168.0.117:9200"]
             index => "act_log-%{+YYYY.MM.dd}"
             }
          }  
        if [type] == "mining" {
          elasticsearch {
             hosts => ["192.168.0.117:9200"]
             index => "mining_log-%{+YYYY.MM.dd}"
             }
          }  
        } 

同樣啓動服務

./logstash -f output_redis.conf --path.data=/home/elk/ &
此時我將路徑改成了另一個路徑了

此時我們就可以在redis上看到我們剛剛加的key和值

 此時我們的redis就加入到我們的elk當中