一、前言
從上週一直在調研找一款好用的kafka監控，我測試使用過的KafkaOffsetMonitor、Burrow、kafka-monitor、Kafka-Manager，他們各有優缺點，具體情況我這裏就不展開描述了，大家可以到它們的git上去查看， 並且它們基本上都是監控topic的寫入和讀取等等，沒有提供對於整體集羣的監控信息，比如集羣的分片、延時、內存使用情況等等，無意中發現了jmxtrans，jmxtrans它是一個通過jmx採集java應用的數據採集器，他的輸出可以是Graphite、StatsD、Ganglia、InfluxDb等等，剛好我們現有的監控是通過InfluxDb做數據存儲的，通過Grafana做展示，下面就給大家介紹一下jmxtrans+InfluxDb+Grafana監控kafka的整體解決方案，並且不需要任何額外的開發工作，完全使用原生的。
二、環境介紹
1、角色
a、10.10.10.10 InfluxDb b、10.10.10.100 Grafana c、10.10.30.69 jmxtrans d、kafka集羣 10.10.20.14 node1 10.10.20.15 node2 10.10.20.16 node3 10.10.20.17 node4
2、軟件版本
influxdb-1.2.4-1.x86_64 grafana-4.1.1-1484211277.x86_64 jmxtrans-266.rpm kafka_2.10-0.9.0.0.jar.asc
3、架構圖
三、配置規劃
1、jmxtrans我們可以分別在每臺kafka節點上部署，也可以部署到一臺機器上，我這裏是選擇了後者，因爲我的集羣小，這樣配置文件可以集中管理，如果集羣比較大，可以考慮分散部署。
2、關於jmxtrans的配置文件，分全局指標（每個kafka節點）和topic指標，全局指標每個節點一個配置文件，命名規則：base_10.10.20.14.json，topic指標是每個topic一個配置文件，命名規則：falcon_monitor_us_17.json
四、監控指標
1、全局指標
每秒輸入的流量
"obj" : "kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec"
"attr" : [ "Count" ]
"resultAlias":"BytesInPerSec"
"tags" : {"application" : "BytesInPerSec"}每秒輸入的流量
"obj" : "kafka.server:type=BrokerTopicMetrics,name=BytesOutPerSec"
"attr" : [ "Count" ]
"resultAlias":"BytesOutPerSec"
"tags" : {"application" : "BytesOutPerSec"}每秒輸入的流量
"obj" : "kafka.server:type=BrokerTopicMetrics,name=BytesRejectedPerSec"
"attr" : [ "Count" ]
"resultAlias":"BytesRejectedPerSec"
"tags" : {"application" : "BytesRejectedPerSec"}每秒的消息寫入總量
"obj" : "kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec"
"attr" : [ "Count" ]
"resultAlias":"MessagesInPerSec"
"tags" : {"application" : "MessagesInPerSec"}每秒FetchFollower的請求次數
"obj" : "kafka.network:type=RequestMetrics,name=RequestsPerSec,request=FetchFollower"
"attr" : [ "Count" ]
"resultAlias":"RequestsPerSec"
"tags" : {"request" : "FetchFollower"}每秒FetchConsumer的請求次數
"obj" : "kafka.network:type=RequestMetrics,name=RequestsPerSec,request=FetchConsumer"
"attr" : [ "Count" ]
"resultAlias":"RequestsPerSec"
"tags" : {"request" : "FetchConsumer"}每秒Produce的請求次數
"obj" : "kafka.network:type=RequestMetrics,name=RequestsPerSec,request=Produce"
"attr" : [ "Count" ]
"resultAlias":"RequestsPerSec"
"tags" : {"request" : "Produce"}內存使用的使用情況
"obj" : "java.lang:type=Memory"
"attr" : [ "HeapMemoryUsage", "NonHeapMemoryUsage" ]
"resultAlias":"MemoryUsage"
"tags" : {"application" : "MemoryUsage"}GC的耗時和次數
"obj" : "java.lang:type=GarbageCollector,name=*"
"attr" : [ "CollectionCount","CollectionTime" ]
"resultAlias":"GC"
"tags" : {"application" : "GC"}線程的使用情況
"obj" : "java.lang:type=Threading"
"attr" : [ "PeakThreadCount","ThreadCount" ]
"resultAlias":"Thread"
"tags" : {"application" : "Thread"}副本落後主分片的最大消息數量
"obj" : "kafka.server:type=ReplicaFetcherManager,name=MaxLag,clientId=Replica"
"attr" : [ "Value" ]
"resultAlias":"ReplicaFetcherManager"
"tags" : {"application" : "MaxLag"}該broker上的partition的數量
"obj" : "kafka.server:type=ReplicaManager,name=PartitionCount"
"attr" : [ "Value" ]
"resultAlias":"ReplicaManager"
"tags" : {"application" : "PartitionCount"}正在做複製的partition的數量
"obj" : "kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions"
"attr" : [ "Value" ]
"resultAlias":"ReplicaManager"
"tags" : {"application" : "UnderReplicatedPartitions"}Leader的replica的數量
"obj" : "kafka.server:type=ReplicaManager,name=LeaderCount"
"attr" : [ "Value" ]
"resultAlias":"ReplicaManager"
"tags" : {"application" : "LeaderCount"}一個請求FetchConsumer耗費的所有時間
"obj" : "kafka.network:type=RequestMetrics,name=TotalTimeMs,request=FetchConsumer"
"attr" : [ "Count","Max" ]
"resultAlias":"TotalTimeMs"
"tags" : {"application" : "FetchConsumer"}一個請求FetchFollower耗費的所有時間
"obj" : "kafka.network:type=RequestMetrics,name=TotalTimeMs,request=FetchFollower"
"attr" : [ "Count","Max" ]
"resultAlias":"TotalTimeMs"
"tags" : {"application" : "FetchFollower"}一個請求Produce耗費的所有時間
"obj" : "kafka.network:type=RequestMetrics,name=TotalTimeMs,request=Produce"
"attr" : [ "Count","Max" ]
"resultAlias":"TotalTimeMs"
"tags" : {"application" : "Produce"}2、topic的監控指標
falcon_monitor_us每秒的寫入流量
"kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=falcon_monitor_us"
"attr" : [ "Count" ]
"resultAlias":"falcon_monitor_us"
"tags" : {"application" : "BytesInPerSec"}falcon_monitor_us每秒的輸出流量
"kafka.server:type=BrokerTopicMetrics,name=BytesOutPerSec,topic=falcon_monitor_us"
"attr" : [ "Count" ]
"resultAlias":"falcon_monitor_us"
"tags" : {"application" : "BytesOutPerSec"}falcon_monitor_us每秒寫入消息的數量
"obj" : "kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec,topic=falcon_monitor_us"
"attr" : [ "Count" ]
"resultAlias":"falcon_monitor_us"
"tags" : {"application" : "MessagesInPerSec"}falcon_monitor_us在每個分區最後的Offset
"obj" : "kafka.log:type=Log,name=LogEndOffset,topic=falcon_monitor_us,partition=*"
"attr" : [ "Value" ]
"resultAlias":"falcon_monitor_us"
"tags" : {"application" : "LogEndOffset"}PS：
1、參數說明
"obj"對應jmx的ObjectName，就是我們要監控的指標
"attr"對應ObjectName的屬性，可以理解爲我們要監控的指標的值
"resultAlias"對應metric 的名稱，在InfluxDb裏面就是MEASUREMENTS名
"tags" 對應InfluxDb的tag功能，對與存儲在同一個MEASUREMENTS裏面的不同監控指標可以做區分，我們在用Grafana繪圖的時候會用到，建議對每個監控指標都打上tags
2、對於全局監控，每一個監控指標對應一個MEASUREMENTS，所有的kafka節點同一個監控指標數據寫同一個MEASUREMENTS ，對於topc監控的監控指標，同一個topic所有kafka節點寫到同一個MEASUREMENTS，並且以topic名稱命名
五、安裝
1、kafka
這裏不詳細介紹kafka集羣的安裝，主要說一下kafka的啓動方式，因爲我們需要通過jmx採集kafka的監控數據，所以在kafka的啓動時候需要啓動jmx端口，啓動方式如下：
cd /data/kafka/bin/ JMX_PORT=9999 nohup ./kafka-server-start.sh ../config/server.properties >/dev/null 2>&1 &
2、InfluxDb
yum -y install influxdb ##安裝 /etc/init.d/influxdb start ##啓動服務 [root@ip-10-10-10-10 jmxtrans]# influx Connected to http://localhost:8086 version 1.3.2 InfluxDB shell version: 1.3.2 > CREATE USER "root" WITH PASSWORD '123456' WITH ALL PRIVILEGES ##添加一個賬號 >
3、Grafana
yum -y install grafana ##安裝 /etc/init.d/grafana-server start ##啓動服務
4、jmxtrans
wget http://central.maven.org/maven2/org/jmxtrans/jmxtrans/266/jmxtrans-266.rpm rpm -ivh jmxtrans-266.rpm ##安裝 /etc/init.d/jmxtrans start ##啓動
六、配置
這裏主要介紹jmxtrans採集數據的配置文件撰寫和Grafana繪圖的配置注意事項，kafka和InfluxDb的配置這裏不做描述。
1、jmxtrans
a、jmxtrans默認讀取/var/lib/jmxtrans下的配置文件去採集數據的，所以我們把採集kafka監控數據的配置文件都在這個目錄下，下面是我的配置文件命名規範：
[root@ip-10-10-30-69 jmxtrans]# ll total 96 -rw-r--r-- 1 root root 1657 Aug 18 17:03 article-feedback-10min-json_14.json -rw-r--r-- 1 root root 1657 Aug 18 17:03 article-feedback-10min-json_15.json -rw-r--r-- 1 root root 1657 Aug 18 17:04 article-feedback-10min-json_16.json -rw-r--r-- 1 root root 1657 Aug 18 17:04 article-feedback-10min-json_17.json -rw-r--r-- 1 root root 8430 Aug 22 08:24 base_10.10.20.14.json -rw-r--r-- 1 root root 8431 Aug 22 08:24 base_10.10.20.15.json -rw-r--r-- 1 root root 8431 Aug 22 08:25 base_10.10.20.16.json -rw-r--r-- 1 root root 8431 Aug 22 08:25 base_10.10.20.17.json -rw-r--r-- 1 root root 2027 Aug 21 16:19 falcon_monitor_us_14.json -rw-r--r-- 1 root root 2027 Aug 21 16:20 falcon_monitor_us_15.json -rw-r--r-- 1 root root 2484 Aug 21 20:58 falcon_monitor_us_16.json -rw-r--r-- 1 root root 2027 Aug 21 16:20 falcon_monitor_us_17.json -rw-r--r-- 1 root root 2147 Aug 21 17:43 highgmp-articles-through-primary_14.json -rw-r--r-- 1 root root 2147 Aug 21 17:46 highgmp-articles-through-primary_15.json -rw-r--r-- 1 root root 2147 Aug 21 17:46 highgmp-articles-through-primary_16.json -rw-r--r-- 1 root root 2147 Aug 21 17:47 highgmp-articles-through-primary_17.json [root@ip-10-10-30-69 jmxtrans]# pwd /var/lib/jmxtrans
b、全局監控的配置文件，以10.10.20.14爲例：
[root@ip-10-10-30-69 jmxtrans]# cat base_10.10.20.14.json
{
"servers" : [ {
"port" : "9999",
"host" : "10.10.20.14",
"queries" : [ {
"obj" : "kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec",
"attr" : [ "Count","OneMinuteRate" ],
"resultAlias":"BytesInPerSec",
"outputWriters" : [ {
"@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
"url" : "http://10.10.10.10:8086/",
"username" : "root",
"password" : "root",
"database" : "jmxDB",
"tags" : {"application" : "BytesInPerSec"}
} ]
},
{
"obj" : "kafka.server:type=BrokerTopicMetrics,name=BytesOutPerSec",
"attr" : [ "Count","OneMinuteRate" ],
"resultAlias":"BytesOutPerSec",
"outputWriters" : [ {
"@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
"url" : "http://10.10.10.10:8086/",
"username" : "root",
"password" : "root",
"database" : "jmxDB",
"tags" : {"application" : "BytesOutPerSec"}
} ]
},
{
"obj" : "kafka.server:type=BrokerTopicMetrics,name=BytesRejectedPerSec",
"attr" : [ "Count","OneMinuteRate" ],
"resultAlias":"BytesRejectedPerSec",
"outputWriters" : [ {
"@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
"url" : "http://10.10.10.10:8086/",
"username" : "root",
"password" : "root",
"database" : "jmxDB",
"tags" : {"application" : "BytesRejectedPerSec"}
} ]
},
{
"obj" : "kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec",
"attr" : [ "Count","OneMinuteRate" ],
"resultAlias":"MessagesInPerSec",
"outputWriters" : [ {
"@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
"url" : "http://10.10.10.10:8086/",
"username" : "root",
"password" : "root",
"database" : "jmxDB",
"tags" : {"application" : "MessagesInPerSec"}
} ]
},
{
"obj" : "kafka.network:type=RequestMetrics,name=RequestsPerSec,request=FetchConsumer",
"attr" : [ "Count" ],
"resultAlias":"RequestsPerSec",
"outputWriters" : [ {
"@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
"url" : "http://10.10.10.10:8086/",
"username" : "root",
"password" : "root",
"database" : "jmxDB",
"tags" : {"request" : "FetchConsumer"}
} ]
},
{
"obj" : "kafka.network:type=RequestMetrics,name=RequestsPerSec,request=FetchFollower",
"attr" : [ "Count" ],
"resultAlias":"RequestsPerSec",
"outputWriters" : [ {
"@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
"url" : "http://10.10.10.10:8086/",
"username" : "root",
"password" : "root",
"database" : "jmxDB",
"tags" : {"request" : "FetchFollower"}
} ]
},
{
"obj" : "kafka.network:type=RequestMetrics,name=RequestsPerSec,request=Produce",
"attr" : [ "Count" ],
"resultAlias":"RequestsPerSec",
"outputWriters" : [ {
"@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
"url" : "http://10.10.10.10:8086/",
"username" : "root",
"password" : "root",
"database" : "jmxDB",
"tags" : {"request" : "Produce"}
} ]
},
{
"obj" : "java.lang:type=Memory",
"attr" : [ "HeapMemoryUsage", "NonHeapMemoryUsage" ],
"resultAlias":"MemoryUsage",
"outputWriters" : [ {
"@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
"url" : "http://10.10.10.10:8086/",
"username" : "root",
"password" : "root",
"database" : "jmxDB",
"tags" : {"application" : "MemoryUsage"}
} ]
},
{
"obj" : "java.lang:type=GarbageCollector,name=*",
"attr" : [ "CollectionCount","CollectionTime" ],
"resultAlias":"GC",
"outputWriters" : [ {
"@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
"url" : "http://10.10.10.10:8086/",
"username" : "root",
"password" : "root",
"database" : "jmxDB",
"tags" : {"application" : "GC"}
} ]
},
{
"obj" : "java.lang:type=Threading",
"attr" : [ "PeakThreadCount","ThreadCount" ],
"resultAlias":"Thread",
"outputWriters" : [ {
"@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
"url" : "http://10.10.10.10:8086/",
"username" : "root",
"password" : "root",
"database" : "jmxDB",
"tags" : {"application" : "Thread"}
} ]
},
{
"obj" : "kafka.server:type=ReplicaFetcherManager,name=MaxLag,clientId=Replica",
"attr" : [ "Value" ],
"resultAlias":"ReplicaFetcherManager",
"outputWriters" : [ {
"@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
"url" : "http://10.10.10.10:8086/",
"username" : "root",
"password" : "root",
"database" : "jmxDB",
"tags" : {"application" : "MaxLag"}
} ]
},
{
"obj" : "kafka.server:type=ReplicaManager,name=PartitionCount",
"attr" : [ "Value" ],
"resultAlias":"ReplicaManager",
"outputWriters" : [ {
"@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
"url" : "http://10.10.10.10:8086/",
"username" : "root",
"password" : "root",
"database" : "jmxDB",
"tags" : {"application" : "PartitionCount"}
} ]
},
{
"obj" : "kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions",
"attr" : [ "Value" ],
"resultAlias":"ReplicaManager",
"outputWriters" : [ {
"@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
"url" : "http://10.10.10.10:8086/",
"username" : "root",
"password" : "root",
"database" : "jmxDB",
"tags" : {"application" : "UnderReplicatedPartitions"}
} ]
},
{
"obj" : "kafka.server:type=ReplicaManager,name=LeaderCount",
"attr" : [ "Value" ],
"resultAlias":"ReplicaManager",
"outputWriters" : [ {
"@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
"url" : "http://10.10.10.10:8086/",
"username" : "root",
"password" : "root",
"database" : "jmxDB",
"tags" : {"application" : "LeaderCount"}
} ]
},
{
"obj" : "kafka.network:type=RequestMetrics,name=TotalTimeMs,request=FetchConsumer",
"attr" : [ "Count","Max" ],
"resultAlias":"TotalTimeMs",
"outputWriters" : [ {
"@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
"url" : "http://10.10.10.10:8086/",
"username" : "root",
"password" : "root",
"database" : "jmxDB",
"tags" : {"application" : "FetchConsumer"}
} ]
},
{
"obj" : "kafka.network:type=RequestMetrics,name=TotalTimeMs,request=FetchFollower",
"attr" : [ "Count","Max" ],
"resultAlias":"TotalTimeMs",
"outputWriters" : [ {
"@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
"url" : "http://10.10.10.10:8086/",
"username" : "root",
"password" : "root",
"database" : "jmxDB",
"tags" : {"application" : "FetchConsumer"}
} ]
},
{
"obj" : "kafka.network:type=RequestMetrics,name=TotalTimeMs,request=Produce",
"attr" : [ "Count","Max" ],
"resultAlias":"TotalTimeMs",
"outputWriters" : [ {
"@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
"url" : "http://10.10.10.10:8086/",
"username" : "root",
"password" : "root",
"database" : "jmxDB",
"tags" : {"application" : "Produce"}
} ]
},
{
"obj" : "kafka.server:type=ReplicaManager,name=IsrShrinksPerSec",
"attr" : [ "Count" ],
"resultAlias":"ReplicaManager",
"outputWriters" : [ {
"@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
"url" : "http://10.10.10.10:8086/",
"username" : "root",
"password" : "root",
"database" : "jmxDB",
"tags" : {"application" : "IsrShrinksPerSec"}
} ]
}
]
} ]
}c、topic監控的配置文件，以falcon_monitor_us的10.10.20.14節點爲例：
[root@ip-10-10-30-69 jmxtrans]# cat falcon_monitor_us_14.json
{
"servers" : [ {
"port" : "9999",
"host" : "10.10.20.14",
"queries" : [ {
"obj" : "kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=falcon_monitor_us",
"attr" : [ "Count" ],
"resultAlias":"falcon_monitor_us",
"outputWriters" : [ {
"@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
"url" : "http://10.10.10.10:8086/",
"username" : "root",
"password" : "root",
"database" : "jmxDB",
"tags" : {"application" : "BytesInPerSec"}
} ]
},
{
"obj" : "kafka.server:type=BrokerTopicMetrics,name=BytesOutPerSec,topic=falcon_monitor_us",
"attr" : [ "Count" ],
"resultAlias":"falcon_monitor_us",
"outputWriters" : [ {
"@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
"url" : "http://10.10.10.10:8086/",
"username" : "root",
"password" : "root",
"database" : "jmxDB",
"tags" : {"application" : "BytesOutPerSec"}
} ]
},
{
"obj" : "kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec,topic=falcon_monitor_us",
"attr" : [ "Count" ],
"resultAlias":"falcon_monitor_us",
"outputWriters" : [ {
"@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
"url" : "http://10.10.10.10:8086/",
"username" : "root",
"password" : "root",
"database" : "jmxDB",
"tags" : {"application" : "MessagesInPerSec"}
} ]
},
{
"obj" : "kafka.log:type=Log,name=LogEndOffset,topic=falcon_monitor_us,partition=*",
"attr" : [ "Value" ],
"resultAlias":"falcon_monitor_us",
"outputWriters" : [ {
"@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
"url" : "http://10.10.10.10:8086/",
"username" : "root",
"password" : "root",
"database" : "jmxDB",
"tags" : {"application" : "LogEndOffset"}
} ]
}
]
} ]
}2、Grafana配置
a、添加數據源
Url、Database、User、Password需要和jmxtrans採集數據配置文件裏面的寫一致，然後點擊Save&Test，提示成功就正常了
b、創建一個dashboard，然後在這裏配置每一個監控指標的圖
c、要點說明
1、對於監控指標爲Count的監控項，需要通過Grafana做計算得到我們想要的監控，比如BytesInPerSec這個指標，它的監控值是一個累計值，我們想要取到每秒的流量，肯定需要計算，(本次採集的值-上次採集的值)/60 ,jmxtrans是一分鐘採集一次數據，具體配置參考下面截圖：
因爲我們是一分鐘採集一次數據，所以group by 和derivative選1分鐘；因爲我們要每秒的流量，所以math這裏除以60
2、X軸的單位選擇，比如流量的單位、時間的單位、每秒消息的個數無單位等等，下面分佈舉一個例子介紹說明
設置流量的單位 ，點擊需要設置的圖，選擇"Edit"進入編輯頁面，切到Axes這個tab頁，Unit--》data（Metric）--》bytes
設置時間的單位 ，點擊需要設置的圖，選擇"Edit"進入編輯頁面，切到Axes這個tab頁，Unit--》time--》milliseconds(ms)
設置按原始值展示，無單位 ，點擊需要設置的圖，選擇"Edit"進入編輯頁面，切到Axes這個tab頁，Unit--》none--》none
七、收穫總結
1、關於jmx收集了kafka的那些指標，對應的值都是那些類型，對應這個問題走了很多彎路，各種谷歌百度拿到了有人整理過的，一個一個試，發現很多不能用，要不就是寫的是錯誤的，要不就是版本不同，寫法不一樣，最後看到了jconsole這個工具，他可以連接到本地或者遠程的jmx端口，能看到在收集的所有指標，在windows下裝好jdk，在bin目錄你可以找到這個工具。
2、關於consumer的延時，關官方介紹有一個type是 type=consumer-fetch-manager-metrics的指標，但是我這通過jconsole連進來死活沒有找到，昨天我已經用 pyhon寫了一個小腳本，通過crontab 一分鐘採集一次，入庫到InfluxDB，也可以通過Grafana做展示了。
a、數據採集的腳本
#!/usr/bin/env python
#-*-coding:utf8-*-
from influxdb import InfluxDBClient
from kafka.consumer import SimpleConsumer
from kazoo.client import KazooClient
from kafka.client import SimpleClient
import time
import datetime
#Zookeepers
zookeepers="10.10.20.14"
#Kafka
kafka="10.10.20.14:9092"
#consumer group
group_list = ["monitor","topic1"]
#InfluxDB client
client = InfluxDBClient('10.10.10.10', 8086, 'root', 'root', 'jmxDB', timeout=2, retries=3)
#time
##now = time.strftime("%Y-%m-%dT%H:%M:%SZ")
now = ((datetime.datetime.now()-datetime.timedelta(hours=8)).strftime("%Y-%m-%d %H:%M"))
if __name__ == '__main__':
try:
broker = SimpleClient(hosts=kafka)
zk = KazooClient(hosts=zookeepers, read_only=True)
zk.start()
for group in group_list:
print group
topics = zk.get_children("/consumers/%s/owners" %(group))
for topic in topics:
try:
consumer = SimpleConsumer(broker, group, str(topic))
lag = int(consumer.pending())
json_body = [
{
"measurement": "lag",
"tags": {
"Consumer_id": group,
"topic": topic,
"addr": kafka
},
"time": now,
"fields": {
"value": int(lag)
}
}
]
try:
client.write_points(json_body)
except Exception as e:
print e
except Exception as e:
print e
except Exception as e:
print e
finally:
zk.stop()b、Grafana的配置，下面這張圖是topic1的兩個Consumer的延時情況：
官網監控指標如下：
http://kafka.apache.org/documentation/#monitoring
八、grafana的報警（補充）
所有的指標都收集齊了，最後一步就是報警了，郵件報警配置很簡單，這裏就不多說了，好多公司現在辦公都用釘釘，當然我們也在用釘釘，grafana從4.3開始支持釘釘報警了，這裏介紹下具體的配置方法
1、創建一個釘釘羣，把需要接收報警的人拉到羣裏
2、創建一個釘釘機器人並獲取url，具體方法如下：
a、在PC版釘釘上，如下圖1、2、3操作
b、點擊“添加 ”
c、點擊複製
3、在grafana上配置釘釘報警
a、按下圖方法添加一條通知策略
b、Name自己隨便定義，勾選下面兩個勾，Url這個填寫上面在添加釘釘機器人複製的url，保存即可。
4、配置需要發報警的監控指標，編輯需要配置報警的監控圖，切換到Alert頁面，點擊"Create Alert"到下圖配置頁面，
Alert Config，Name 會根據圖的Title自動生成，可以修改，Evaluate every多久執行一次
Conditions，主要說一下query的第一個參數，如果以在一個圖上有多個監控指標，需要多那個指標報警，選擇對應的Metrics的行就可以了，後面兩個參數就是時間範圍了，這個報警的配置接收如下：
在5分鐘內，如果A這個Metrics的平均值大於100 就報警，或者C這個Metrics的最大值大於200就報警；如果沒有數據或者數據爲null也報警，如果執行錯誤或者超時也報警
切換到Notifications，點擊"+"號添加釘釘報警，並在Message添加報警內容，切記要執行“Ctrl +S”保存頁面，不然剛纔的配置白瞎了
ok，到此釘釘報警已經配置完成，點擊釘釘羣裏的鏈接可以查看報警監控指標的具體情況。
