希望大家有一天都能從借鑑中起飛
環境搭建解釋
主機信息規劃
- 192.168.198.131 - swarm01
- 192.168.198.132 - swarm02
- 192.168.198.133 - swarm03
安裝包下載地址[頁面訪問速度延遲，可以直接點擊備註中的地址直接下載]
- kafka: http://kafka.apache.org/downloads
- 能力強者請直接參考官方文檔
- http://kafka.apache.org/documentation/
安裝位置
- 192.168.198.131 /opt/kafka
- 192.168.198.132 /opt/kafka
- 192.168.198.133 /opt/kafka
配置修改
- 標識修改[建議沿用zk的標識，防止衝突或是重複]
131-主機
[root@swarm01 config]# pwd
/opt/kafka/config
[root@swarm01 config]# ls
connect-console-sink.properties connect-file-source.properties log4j.properties trogdor.conf
connect-console-source.properties connect-log4j.properties producer.properties zookeeper.properties
connect-distributed.properties connect-standalone.properties server.properties
connect-file-sink.properties consumer.properties tools-log4j.properties
[root@swarm01 config]# vi server.properties
---------------------------------------------------------------
broker.id=1
log.dirs=/opt/kafka/logs
zookeeper.connect=swarm01:2181,swarm02:2181,swarm03:2181
---------------------------------------------------------------
132-主機
---------------------------------------------------------------
broker.id=2
log.dirs=/opt/kafka/logs
zookeeper.connect=swarm01:2181,swarm02:2181,swarm03:2181
---------------------------------------------------------------
133-主機
---------------------------------------------------------------
broker.id=3
log.dirs=/opt/kafka/logs
zookeeper.connect=swarm01:2181,swarm02:2181,swarm03:2181
---------------------------------------------------------------
備註：上面這三點配置都需要更具自己的實際情況進行修改
環境配置
[root@swarm01 opt]# vi /etc/profile
# kafka home
export KAFKA_HOME=/opt/kafka
export PATH=$PATH:$KAFKA_HOME/bin
啓動
- 自檢環境配置
- zk環境查看是否正常
環境啓動需要依賴zk環境，因此需要首先啓動zk集羣環境
[root@swarm01 config]# jps
1856 QuorumPeerMain
2094 Jps
[root@swarm01 config]#
- kafka啓動
一定要啓動三個
[root@swarm01 bin]# ./kafka-server-start.sh -daemon ../config/server.properties
如果這裏你的環境之前沒有進行過任何操作，返回的沒有first這個數據
[root@swarm01 bin]# ./kafka-topics.sh --zookeeper 127.0.0.1:2181 --list
__consumer_offsets
first
[root@swarm01 bin]# pwd
/opt/kafka/bin
[root@swarm01 bin]#
- 使用jps查看啓動的進程
[root@swarm01 bin]# jps
1856 QuorumPeerMain
4321 Jps
4252 Kafka
[root@swarm01 bin]#
常用配置命令
- 查詢主題列表
[root@swarm01 bin]# ./kafka-topics.sh --zookeeper localhost:2181 --list
__consumer_offsets
first
[root@swarm01 bin]#
- 創建新的新主題
3各分區 3個副本
[root@swarm01 bin]# kafka-topics.sh --zookeeper localhost:2181 --create --topic my-topic --replication-factor 3 --partitions 3
Created topic my-topic.
[root@swarm01 bin]#
- 主題詳情
[root@swarm01 bin]# ./kafka-topics.sh --zookeeper localhost:2181 --describe
Topic:__consumer_offsets PartitionCount:50 ReplicationFactor:1 Configs:segment.bytes=104857600,cleanup.policy=compact,compression.type=producer
Topic: __consumer_offsets Partition: 0 Leader: -1 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 1 Leader: 1 Replicas: 1 Isr: 1
Topic: __consumer_offsets Partition: 2 Leader: 2 Replicas: 2 Isr: 2
Topic: __consumer_offsets Partition: 3 Leader: -1 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 4 Leader: 1 Replicas: 1 Isr: 1
Topic: __consumer_offsets Partition: 5 Leader: 2 Replicas: 2 Isr: 2
Topic: __consumer_offsets Partition: 6 Leader: -1 Replicas: 0 Isr: 0
......
描述具體某一個詳情
[root@swarm01 bin]# kafka-topics.sh --zookeeper localhost:2181 --describe --topic my-topic
Topic:my-topic PartitionCount:3 ReplicationFactor:3 Configs:
Topic: my-topic Partition: 0 Leader: 3 Replicas: 3,1,2 Isr: 1,2,3
Topic: my-topic Partition: 1 Leader: 1 Replicas: 1,2,3 Isr: 1,2,3
Topic: my-topic Partition: 2 Leader: 2 Replicas: 2,3,1 Isr: 2,1,3
[root@swarm01 bin]#
- 刪除主題
[root@swarm01 bin]# kafka-topics.sh --zookeeper localhost:2181 --delete --topic my-topic
Topic my-topic is marked for deletion.
請注意下面的提示，當設置這個屬性爲真的時候會徹底刪除這個主題
Note: This will have no impact if delete.topic.enable is not set to true.
[root@swarm01 bin]#
- 查看歷史消費者羣組
[root@swarm01 bin]# ./kafka-consumer-groups.sh --bootstrap-server swarm01:9092 --list
or
[root@swarm01 bin]# ./kafka-consumer-groups.sh --bootstrap-server swarm01:9092 --describe --group groupName
[root@swarm01 bin]#
- 查看新版消費者羣組
kafka-consumer-groups.sh --new-consumer --bootstrap-server 172.21.50.162:9092 --list
or
kafka-consumer-groups.sh --new-consumer --bootstrap-server 172.21.50.162:9092 --describe --group groupName
- 查看主題對應的消息數量
[root@swarm01 bin]# ./kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list swarm01:9092 --topic first --time -1
first:0:0
- 查看日誌文件內容
日誌所在位置
[root@swarm01 first-0]# pwd
/opt/kafka/logs/first-0
[root@swarm01 first-0]# ls
00000000000000000000.log 00000000000000000000.timeindex leader-epoch-checkpoint
[root@swarm01 first-0]#
kafka-run-class.sh kafka.tools.DumpLogSegments --files 00000000000000000000.log --print-data-log
- 生產者 or 消費者 動作
消費端動作
[root@swarm01 bin]# ls
connect-distributed.sh kafka-dump-log.sh kafka-topics.sh
connect-standalone.sh kafka-log-dirs.sh kafka-verifiable-consumer.sh
kafka-acls.sh kafka-mirror-maker.sh kafka-verifiable-producer.sh
kafka-broker-api-versions.sh kafka-preferred-replica-election.sh trogdor.sh
kafka-configs.sh kafka-producer-perf-test.sh windows
kafka-console-consumer.sh kafka-reassign-partitions.sh zookeeper-security-migration.sh
kafka-console-producer.sh kafka-replica-verification.sh zookeeper-server-start.sh
kafka-consumer-groups.sh kafka-run-class.sh zookeeper-server-stop.sh
kafka-consumer-perf-test.sh kafka-server-start.sh zookeeper-shell.sh
kafka-delegation-tokens.sh kafka-server-stop.sh
kafka-delete-records.sh kafka-streams-application-reset.sh
[root@swarm01 bin]# ./kafka-console-producer.sh --broker-list swarm01:9092 --topic first
>1
>2
>3
>4
>5
>
接收端動作
[root@swarm02 bin]# ./kafka-console-consumer.sh --bootstrap-server swarm01:9092 --topic first --from-beginning
1
2
3
4
5