一臺機器搭建Kafka集羣及Kafka的簡單操作（僞分佈式）

1.搭建Zookeeper集羣（僞分佈式）。參考如下鏈接
https://mp.csdn.net/postedit/102797469
2.將下載的kafkatar.gz文件解壓
tar zxvf kafka_2.11-2.2.1-kafka-4.1.0.tar.gz -C ~/app/
3.將afka_2.11-2.2.1-kafka-4.1.0/config/server.properties複製多份（一個server.properties就是一個broker）

[hadoop@hadoop002 config]$ cp server.properties server_01.properties
[hadoop@hadoop002 config]$ cp server.properties server_02.properties
[hadoop@hadoop002 config]$ cp server.properties server_03.properties

4.修改配置

[hadoop@hadoop002 config]$ vim server_01.properties
broker.id=0
host.name=hadoop002
log.dirs=/home/hadoop/tmp/kafka01/log/kafka-logs
port=9092
zookeeper.connect=hadoop002:2181,hadoop002:2182,hadoop002:2183/kafka

[hadoop@hadoop002 config]$ vim server_02.properties
broker.id=1
host.name=hadoop002
port=9093
log.dirs=/home/hadoop/tmp/kafka02/log/kafka-logs
zookeeper.connect=hadoop002:2181,hadoop002:2182,hadoop002:2183/kafka


[hadoop@hadoop002 config]$ vim server_03.properties
broker.id=2
host.name=hadoop002
port=9094
log.dirs=/home/hadoop/tmp/kafka03/log/kafka-logs
zookeeper.connect=hadoop002:2181,hadoop002:2182,hadoop002:2183/kafka

5.啓動

[hadoop@hadoop002 bin]$ cd  /home/hadoop/app/kafka_2.11-2.2.1-kafka-4.1.0/bin
[hadoop@hadoop002 bin]$ ./kafka-server-start.sh -daemon ../config/server_01.properties
[hadoop@hadoop002 bin]$ ./kafka-server-start.sh -daemon ../config/server_02.properties
[hadoop@hadoop002 bin]$ ./kafka-server-start.sh -daemon ../config/server_03.properties

6.創建topic

[hadoop@hadoop002 bin]$ ./kafka-topics.sh \
> --create \
> --zookeeper hadoop002:2181,hadoop002:2182,hadoop002:2183/kafka \
> --replication-factor 3 \
> --partitions 3 \
> --topic testkafka
Created topic testkafka.
[hadoop@hadoop002 bin]$ 

7.通過list命令查看創建的topic

[hadoop@hadoop002 bin]$ ./kafka-topics.sh \
> --list \
> --zookeeper hadoop002:2181,hadoop002:2182,hadoop002:2183/kafka
testkafka
[hadoop@hadoop002 bin]$ 

8.查看創建的Topic

[hadoop@hadoop002 bin]$ ./kafka-topics.sh \
> --describe \
> --zookeeper hadoop002:2181,hadoop002:2182,hadoop002:2183/kafka \
> --topic testkafka
Topic:testkafka	PartitionCount:3	ReplicationFactor:3	Configs:
	Topic: testkafka	Partition: 0	Leader: 0	Replicas: 0,1,2	Isr: 0,1,2
	Topic: testkafka	Partition: 1	Leader: 1	Replicas: 1,2,0	Isr: 1,2,0
	Topic: testkafka	Partition: 2	Leader: 2	Replicas: 2,0,1	Isr: 2,0,1
[hadoop@hadoop002 bin]$ 

第一行行列列出了了這個topic的總體情況，如topic名稱，分區數量量，副本數量量等。
第二行行開始，每一行行列列出了了一個分區的信息，如它是第幾個分區，這個分區的leader是哪個broker，副本位於哪些broker，有哪些副本處理理同步狀態。
Partition:分區
Leader:負責讀寫指定分區的節點
Replicas:複製該分區log的節點列表
Isr:“in-sync” replicas，當前活躍的副本列列表（是一個子集），並且可能成爲Leader我們可以通過Kafka自帶的bin/kafka-console-producer.sh和bin/kafka-console-consumer.sh腳
本，來驗證演示如果發佈消息、消費消息
9.刪除topic

[hadoop@hadoop002 bin]$ ./kafka-topics.sh \
> --delete \
> --zookeeper hadoop002:2181,hadoop002:2182,hadoop002:2183/kafka \
> --topic testkafka
Topic testkafka is marked for deletion.
Note: This will have no impact if delete.topic.enable is not set to true.
[hadoop@hadoop002 bin]$ 

修改分區
 

#使用—-alert原則上可以修改任何配置，以下列出了一些常用的修改選項：
[hadoop@hadoop002 bin]$ ./kafka-topics.sh \
> --create \
> --zookeeper hadoop002:2181,hadoop002:2182,hadoop002:2183/kafka \
> --replication-factor 1 \
> --partitions 1 \
> --topic test
Created topic test.
[hadoop@hadoop002 bin]$ ./kafka-topics.sh \
> --describe \
> --zookeeper hadoop002:2181,hadoop002:2182,hadoop002:2183/kafka \
> --topic test
Topic:test	PartitionCount:1	ReplicationFactor:1	Configs:
	Topic: test	Partition: 0	Leader: 0	Replicas: 0	Isr: 0
#改變分區數量爲3
[hadoop@hadoop002 bin]$ ./kafka-topics.sh \
> --alter \
> --zookeeper hadoop002:2181,hadoop002:2182,hadoop002:2183/kafka \
> --topic test --partitions 3
WARNING: If partitions are increased for a topic that has a key, the partition logic or ordering of the messages will be affected
Adding partitions succeeded!
[hadoop@hadoop002 bin]$ ./kafka-topics.sh \
> --describe \
> --zookeeper hadoop002:2181,hadoop002:2182,hadoop002:2183/kafka \
> --topic test
Topic:test	PartitionCount:3	ReplicationFactor:1	Configs:
	Topic: test	Partition: 0	Leader: 0	Replicas: 0	Isr: 0
	Topic: test	Partition: 1	Leader: 1	Replicas: 1	Isr: 1
	Topic: test	Partition: 2	Leader: 2	Replicas: 2	Isr: 2
[hadoop@hadoop002 bin]$ 

在log中查看kafka的分區

[hadoop@hadoop002 log]$ cd kafka-logs/
[hadoop@hadoop002 kafka-logs]$ ll
total 20
-rw-rw-r-- 1 hadoop hadoop   4 Oct 29 20:30 cleaner-offset-checkpoint
-rw-rw-r-- 1 hadoop hadoop   4 Oct 29 20:33 log-start-offset-checkpoint
-rw-rw-r-- 1 hadoop hadoop  54 Oct 29 20:09 meta.properties
-rw-rw-r-- 1 hadoop hadoop  13 Oct 29 20:33 recovery-point-offset-checkpoint
-rw-rw-r-- 1 hadoop hadoop  13 Oct 29 20:34 replication-offset-checkpoint
drwxrwxr-x 2 hadoop hadoop 137 Oct 29 20:33 test-0
[hadoop@hadoop002 kafka-logs]$