（一）prometheus + grafana + alertmanager 配置主機監控

安裝請看https://blog.51cto.com/liuqs/2027365 ，最好是對應的版本組件，否則可能會有差別。

（一） prometheus + grafana + alertmanager配置主機監控

1. 配置prometheus（登陸到prometheus服務器，prometheus grafana alertmanager在同一臺服務器上）

a. 打開vim /data/monitor/prometheus/conf/prometheus.yml文件。配置文件如下(所有的節點都是寫在json文件中)：

global:

# Server端抓取數據的時間間隔

scrape_interval: 1m

# 評估報警規則的時間間隔

evaluation_interval: 1m

# 數據抓取的超時時間

scrape_timeout: 20s

# 加全局標籤

#external_labels:

#monitor: "usa"

# 連接alertmanager

alerting:

alertmanagers:

- static_configs:

- targets: ["localhost:9093"]

# 告警規則

rule_files:

- /data/monitor/prometheus/conf/rule/*.yml

# A scrape configuration containing exactly one endpoint to scrape:

# Here it's Prometheus itself.

scrape_configs:

# 監控prometheus本機

- job_name: 'prometheus'

scrape_interval: 15s

static_configs:

- targets: ['10.8.9.2:9090']

# 監控指定主機

- job_name: 'node_resources'

scrape_interval: 1m

static_configs:

file_sd_configs:

- files:

- /data/monitor/prometheus/conf/node_conf/node_host_info.json

honor_labels: true

b. node主機json文件：

cat /data/monitor/prometheus/conf/node_conf/node_host_info.json

[

{

"labels": {

"desc": "ba_backend_10.8.9.35",

"group": "ba",

"host_ip": "10.8.9.35",

"hostname": "ba_backend"

"targets": [

"10.8.9.35:9100"

]

{

"labels": {

"desc": "ba3_10.8.32.67",

"group": "ba",

"host_ip": "10.8.32.67",

"hostname": "ba3"

"targets": [

"10.8.32.67:9100"

]

{

"labels": {

"desc": "ba1_10.8.46.117",

"group": "ba",

"host_ip": "10.8.46.117",

"hostname": "ba1"

"targets": [

"10.8.46.117:9100"

]

{

"labels": {

"desc": "ba2_10.8.80.126",

"group": "ba",

"host_ip": "10.8.80.126",

"hostname": "ba2"

"targets": [

"10.8.80.126:9100"

]

{

"labels": {

"desc": "openplatform_10.8.69.81",

"group": "openplatform",

"host_ip": "10.8.69.81",

"hostname": "openplatform"

"targets": [

"10.8.69.81:9100"

]

}

]

c. cd /data/monitor/prometheus, 然後 sh start.sh啓動prometheus，然後netstat -nltp |grep prometheus 查看9090端口是否已監聽

d. 在需要監控的服務器上（10.8.9.35，10.8.32.67，10.8.46.117，10.8.80.126，10.8.69.81）下載並安裝node_export（下載地址：https://pan.baidu.com/s/1gi-BM0rWWaGGKyWzUBFLPg），下載後解壓到/data/下，然後 cd /data/node_exporter , sh start.sh 啓動服務, netstat -nltp |grep node_exporter 查看9100是否已監聽

e. 登錄prometheus，在瀏覽器中打開 http://10.8.9.2:9090 ，先點菜單欄裏的Graph，然後在下面框中輸入 node_boot_time，最後點Execute查看下面是否有數據。

2. 配置grafana

a. /etc/init.d/grafana start 啓動grafana，然後 netstat -nltp |grep grafana 查看3000端口是否已監聽。

b. 在瀏覽器中打開 http://10.8.9.2:3000 登錄grafana，默認用戶名與密碼都是admin。

c. 首先點配置按鈕，然後在點Data Sources配置數據源。

d. 打開數據源頁面，然後點 Add data source

e. 打開選擇數據源頁面，選擇prometheus

f. 然後打開配置prometheus數據源頁面，在Name中輸入 Prometheus，並勾選Default, URL中填 http://localhost:9090（因爲prometheus和grafana在同一臺機上），最後點Save&Test 進行保存。

g. 下載主機監控模板到自己電腦 https://pan.baidu.com/s/19RLrebLh5lI3nla4jaq1QA（當然你也可以自己新建編輯，步驟是點+號，選擇圖表。）

h. 然後在grafana頁面中，點+，然後再點import

i. 然後在導入頁面，點Upload .json File,選擇剛下載下來的模板

j. 然後點change改變模板id，然後再點General, 然後點New Folder, 然後點Cancel，最後點Import，就會導入模板。

k. 然後就可以看到數據展示了。

l. 當然你也可對現有的模板進行修改，或者新增或者設置等。

3. alertmanager配置

a. 配置規則，cat /data/monitor/prometheus/conf/rule/host.yml

groups:

- name: host_alert

rules:

### 硬盤 ###

# 默認系統盤告警策略

- alert: 主機系統盤80%

expr: floor(100-((node_filesystem_avail{device!="rootfs", mountpoint="/"}*100)/(node_filesystem_size{device!="rootfs", mountpoint="/"}*0.95))) >= 80

for: 3m

labels:

severity: warning

annotations:

description: "[{{ $labels.desc }}],告警值爲:[{{ $value }}%],告警初始時長爲3分鐘."

# 默認120G內數據盤告警策略

- alert: 主機數據盤90%

expr: (floor(100-((node_filesystem_avail{device!="rootfs", mountpoint="/data"}*100)/(node_filesystem_size{device!="rootfs", mountpoint="/data"}*0.95))) >= 90) and (node_filesystem_size{device!="rootfs", mountpoint="/data"}/1024/1024/1024 <= 120)

for: 3m

labels:

severity: warning

annotations:

description: "[{{ $labels.desc }}],告警值爲:[{{ $value }}%],告警初始時長爲3分鐘."

# 默認120G以上數據盤告警策略

- alert: 主機數據盤不足20G

expr: (floor(node_filesystem_avail{device!="rootfs", mountpoint="/data"}/1024/1024/1024) <= 20) and (node_filesystem_size{device!="rootfs", mountpoint="/data"}/1024/1024/1024 > 120)

for: 3m

labels:

severity: warning

annotations:

description: "[{{ $labels.desc }}],告警值爲:[{{ $value }}G],告警初始時長爲3分鐘."

### CPU ###

# 默認CPU使用率告警策略

- alert: 主機CPU90%

expr: floor(100 - ( avg ( irate(node_cpu{mode='idle', hostname!~'consumer_service.*|backup_hk.*|bigdata.*master.*|3rdPart|htc_management|product_category_backend|sa_cluster_s.*'}[5m]) ) by (job, instance, hostname, desc) * 100 )) >= 90

for: 3m

labels:

severity: warning

annotations:

description: "[{{ $labels.desc }}],告警值爲:[{{ $value }}%],告警初始時長爲3分鐘."

# 持續時間較長的CPU使用率告警策略

- alert: 主機CPU90%

expr: floor(100 - ( avg ( irate(node_cpu{mode='idle', hostname=~'consumer_service.*|product_backend|sa_cluster_s.*'}[5m]) ) by (job, instance, hostname, desc) * 100 )) >= 90

for: 12m

labels:

severity: warning

annotations:

description: "[{{ $labels.desc }}],告警值爲:[{{ $value }}%],告警初始時長爲12分鐘."

# 持續時間較長的CPU使用率告警策略

- alert: 主機CPU90%

expr: floor(100 - ( avg ( irate(node_cpu{mode='idle', hostname=~'bigdata.*master.*|3rdPart|backup_hk.*'}[5m]) ) by (job, instance, hostname, desc) * 100 )) >= 90

for: 48m

labels:

severity: warning

annotations:

description: "[{{ $labels.desc }}],告警值爲:[{{ $value }}%],告警初始時長爲48分鐘."

### 內存 ###

# 默認內存使用率告警策略

- alert: 主機內存95%

expr: floor((node_memory_MemTotal - node_memory_MemFree - node_memory_Cached - node_memory_Buffers) / node_memory_MemTotal * 100) >= 95

for: 3m

labels:

severity: warning

annotations:

description: "[{{ $labels.desc }}],告警值爲:[{{ $value }}%],告警初始時長爲3分鐘."

### 負載 ###

# 默認負載過高告警策略

- alert: 主機負載過高

expr: floor(node_load1{hostname!~"sa_cluster_s.*|bigdata.*master.*"}) >= 20

for: 3m

labels:

severity: warning

annotations:

description: "[{{ $labels.desc }}],告警值爲:[{{ $value }}],告警初始時長爲3分鐘."

# 持續時間較長的負載過高告警策略

- alert: 主機負載過高

expr: floor(node_load1{hostname=~"sa_cluster_s.*|bigdata.*master.*"}) >= 20

for: 12m

labels:

severity: warning

annotations:

description: "[{{ $labels.desc }}],告警值爲:[{{ $value }}],告警初始時長爲12分鐘."

b. 重啓prometheus，cd /data/monitor/prometheus , sh reload.sh

c. 配置alertmanager, cat /data/prometheus/alertmanager/conf/alertmanager.yml

global:

resolve_timeout: 2m

smtp_auth_password: q5AYahvxi3WLDap3 #發送郵箱密碼

smtp_auth_username: [email protected] #發送郵箱

smtp_from: [email protected] #發送郵箱

smtp_require_tls: false

smtp_smarthost: smtp.163.com:465 #發送服務器

wechat_api_url: https://qyapi.weixin.qq.com/cgi-bin/ #微信接口鏈接

inhibit_rules:

- equal:

- instance

source_match:

alertname: "主機CPU90%"

target_match:

alertname: "主機負載過高"

- equal:

- instance

source_match:

alertname: "mysql運行進程數5分鐘增長數>150"

target_match:

alertname: "mysql慢查詢5分鐘100條"

- equal:

- instance

source_match:

severity: error

target_match:

severity: warning

- equal:

- instance

source_match:

severity: fatal

target_match:

severity: error

- equal:

- service_name

source_match:

severity: error

target_match:

severity: warning

receivers:

- email_configs: #定義test發送人模塊

- html: '{{ template "email.default.html" . }}' #調用的模板

send_resolved: true

to: [email protected] #將報警信息發給些郵箱，多人用|

name: test #發送人模板名

wechat_configs: #微信接收這些信息請看最下面的企業微信介紹

- agent_id: 1000002 #應用id

api_secret: hnyU1LTGnJUiBaCp47l3WVQLTEFF5RXyfNO751xlaHa #應用認證

corp_id: wwd397231fa801beaa #企業微信ID

send_resolved: true

to_user: LiuQingShan|liuqs #發送給企業微信通訊人的Id 多個人就用|分開

- email_configs: #定義默認的發送人

- html: '{{ template "email.default.html" . }}'

send_resolved: true

to: [email protected]

name: default_group

wechat_configs:

- agent_id: 1000002

api_secret: hnyU1LTGnJUiBaCp47l3WVQLTEFF5RXyfNO751xlaHa

corp_id: wwd397231fa801beaa

send_resolved: true

to_user: LiuQingShan

route: #定義資源報警規則

group_by:

- monitor

group_interval: 2m

group_wait: 30s

receiver: default_group

repeat_interval: 6h

routes:

- continue: true

match_re:

instance: 10.8.46.117:9100|10.8.80.126:9100|10.8.32.67:9100|10.8.9.35:9100|10.8.69.81:9100 #定義使用的資源

receiver: test #使用test發送人模板

templates:

- /data/monitor/alertmanager/template/*.tmpl #調用報警內容模板的路徑

d. 報警內容模板

cat/data/monitor/alertmanager/template/default.tmpl

{{ define "__alertmanager" }}AlertManager{{ end }}

{{ define "__alertmanagerURL" }}{{ .ExternalURL }}/#/alerts?receiver={{ .Receiver }}{{ end }}

{{ define "__subject" }}[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .GroupLabels.SortedPairs.Values | join " " }} {{ if gt (len .CommonLabels) (len .GroupLabels) }}({{ with .CommonLabels.Remove .GroupLabels.Names }}{{ .Values | join " " }}{{ end }}){{ end }}{{ end }}

{{ define "__text_alert_list" }}{{ range . }}Labels:

{{ range .Labels.SortedPairs }} - {{ .Name }} = {{ .Value }}

{{ end }}Annotations:

{{ range .Annotations.SortedPairs }} - {{ .Name }} = {{ .Value }}

{{ end }}Source: {{ .GeneratorURL }}

{{ define "slack.default.fallback" }}{{ template "slack.default.title" . }} | {{ template "slack.default.titlelink" . }}{{ end }}

Alerts Firing:

Alerts Resolved:

<!--

#{{ template "__subject" . }}

#{{ .CommonAnnotations.SortedPairs.Values | join " " }}

Alerts Firing:

#{{ template "__text_alert_list" .Alerts.Firing }}

Labels:

{{ range .Labels.SortedPairs }}{{ .Name }} = {{ .Value }}

Start_time:{{ .StartsAt }}

告警主題: {{ .Annotations.summary }}

告警詳情: {{ .Annotations.description }}

Alerts Resolved:

#{{ template "__text_alert_list" .Alerts.Resolved }}

Labels:

{{ range .Labels.SortedPairs }}{{ .Name }} = {{ .Value }}

Start_time:{{ .StartsAt }}

End_time:{{ .EndsAt }}

告警主題: {{ .Annotations.summary }}

告警詳情: {{ .Annotations.description }}

#AlertmanagerUrl:

#{{ template "__alertmanagerURL" . }}

#{{- end }}

-->

Alerts Firing:

Alerts Resolved:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<!--

Style and HTML derived from https://github.com/mailgun/transactional-email-templates

The MIT License (MIT)

Permission is hereby granted, free of charge, to any person obtaining a copy

of this software and associated documentation files (the "Software"), to deal

in the Software without restriction, including without limitation the rights

to use, copy, modify, merge, publish, distribute, sublicense, and/or sell

copies of the Software, and to permit persons to whom the Software is

furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all

copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR

IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,

FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE

AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER

LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,

OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE

SOFTWARE.

-->

<title style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">{{ template "__subject" . }}</title>

</head>

{{ .Alerts | len }} alert{{ if gt (len .Alerts) 1 }}s{{ end }} for {{ range .GroupLabels.SortedPairs }}

{{ .Name }}={{ .Value }}

</td>

</tr>

<a href="{{ template "__alertmanagerURL" . }}" style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; color: #FFF; text-decoration: none; line-height: 2em; font-weight: bold; text-align: center; cursor: pointer; display: inline-block; border-radius: 5px; text-transform: capitalize; background-color: #348eda; margin: 0; border-color: #348eda; border-style: solid; border-width: 10px 20px;">View in {{ template "__alertmanager" . }}</a>

</td>

</tr>

<strong style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">[{{ .Alerts.Firing | len }}] Firing</strong>

</td>

</tr>

<strong style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">Labels</strong><br style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />

{{ range .Labels.SortedPairs }}{{ .Name }} = {{ .Value }}<br style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />{{ end }}

<strong style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">Start_time:</strong><br style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />

{{ .StartsAt.Format "2006-01-02 15:04:05" }}<br style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />

{{ if gt (len .Annotations) 0 }}<strong style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">Annotations</strong><br style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />{{ end }}

{{ range .Annotations.SortedPairs }}{{ .Name }} = {{ .Value }}<br style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />{{ end }}

<a href="{{ .GeneratorURL }}" style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; color: #348eda; text-decoration: underline; margin: 0;">Source</a><br style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />

</td>

</tr>

</td>

</tr>

<strong style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">[{{ .Alerts.Resolved | len }}] Resolved</strong>

</td>

</tr>

{{ range .Labels.SortedPairs }}{{ .Name }} = {{ .Value }}<br style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />{{ end }}

{{ .StartsAt.Format "2006-01-02 15:04:05" }}<br style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />

{{ .EndsAt.Format "2006-01-02 15:04:05" }}<br style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />

{{ range .Annotations.SortedPairs }}{{ .Name }} = {{ .Value }}<br style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />{{ end }}

</td>

</tr>

</table>

</td>

</tr>

</table>

<td style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 12px; vertical-align: top; text-align: center; color: #999; margin: 0; padding: 0 0 20px;" align="center" valign="top"><a href="{{ .ExternalURL }}" style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 12px; color: #999; text-decoration: underline; margin: 0;">Sent by {{ template "__alertmanager" . }}</a></td>

</tr>

</table>

</div></div>

</td>

</tr>

</table>

</body>

</html>

Alerts Firing:

Alerts Resolved:

cat /data/monitor/alertmanager/template/wechat.tmpl

{{ if gt (len .Alerts.Firing) 0 -}}告警:

{{ range .Alerts.Firing }}類型：{{ .Labels.alertname }}

詳情: {{ .Annotations.description }}

開始: {{ .StartsAt.Format "2006-01-02 15:04:05" }}

======

{{ if gt (len .Alerts.Resolved) 0 -}}恢復:

{{ range .Alerts.Resolved }}類型：{{ .Labels.alertname }}

詳情: {{ .Annotations.description }}

開始: {{ .StartsAt.Format "2006-01-02 15:04:05" }}

結束: {{ .EndsAt.Format "2006-01-02 15:04:05" }}

======

e. cd /data/monitor/alertmanager 下， sh start.sh

f. 注意：配置微信接收告警

（1）需要首先註冊一個企業微信，然後點應用與小程序，然後點創建應用。

（2）然後在創建應用裏，傳logo及填對應信息和選擇接收人的範圍。

（3）然後點開這個新建的監控報警應用就可以看到對應的agent_id（AgentId）和api_secret（Secret）

（4）corp_id 在企業微信頁面中點菜單欄我的企業，然後最下方有個企業ID

（5）接收人to_user在企業微信頁面中點菜單欄通訊錄中，點每個聯繫人就可以看到帳號，這樣就可以用企業微信接收報警信息了。

（6）如果不想下載企業微信APP，直接用微信接收，需要在企業微信頁面點我的企業，然後點微工作臺，然後用微信掃一掃邀請關注後面的二維碼，點關注，如果是想其他人也用微信收報警信息，需要將此二維碼分享給相關人，或者在自己微信裏找到自己的企業號，推薦給朋友。

（一）prometheus + grafana + alertmanager 配置主機監控

（一）prometheus + grafana + alertmanager 配置主機監控

（三） prometheus + grafana + alertmanager 配置Redis監控

（二） prometheus + grafana + alertmanager 配置Mysql監控

（四） prometheus + grafana + alertmanager 配置Kafka監控

Confluence wiki上傳文件時報“不能上傳文件至Confluence中，服務器可能已失效”

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結