業務目標
藉助logstash工具 將mysql中的數據上傳到elasticsearch中
並藉助kibana工具可是化展示
a.下載解壓
wget https://artifacts.elastic.co/downloads/logstash/logstash-7.1.1.tar.gz
tar -zxvf logstash-7.1.1.tar.gz /usr/local/logstash
b.安裝插件
cd /usr/local/logstash/bin
./logstash-plugin install logstash-input-jdbc
./logstash-plugin install logstash-output-elasticsearch
c.獲取mysql驅動包
放在bin目錄下,與配置文件同級
d.創建啓動配置文件stdmysql.conf
藉助 config/pipelines.yml 可以同時啓動加載多個配置文件
vim stdmysql.conf
input {
jdbc {
jdbc_driver_library => "mysql-connector-java-5.1.7-bin.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
jdbc_connection_string => "jdbc:mysql://127.0.0.1:3306/luoforTest"
jdbc_user => "root"
jdbc_password => "092248"
schedule => "* * * * *"
statement => "SELECT * FROM UserDto WHERE update_time >= :sql_last_value"
use_column_value => true
tracking_column_type => "timestamp"
tracking_column => "update_time"
last_run_metadata_path => "syncpoint_table"
}
}
output {
elasticsearch {
hosts => ["127.0.0.1:9200"]
index => "userdto" #不支持大寫
document_id => "%{id}"
}
}
e.啓動 logstash(通過kibana可視化展示)
./logstash -f stdmysql.conf &
f.多配置同時啓動
vim config/pipelines.yml
- pipeline.id: elk
path.config: "../bin/stdin.conf"
- pipeline.id: mysql
path.config: "../bin/stdmysql.conf"
直接啓動無須指定配置
./logstash &