部署kafka之前需先部署zookeeper.
1 安裝ZOOKEEPER
1.1 安裝包
zookeeper-3.4.10.tar.gz
1.2 部署
下述操作爲在單機上安裝zookeeper僞集羣，也可直接安裝單機版。
實際安裝路徑視具體情況而定
解壓：
tar -zxf zookeeper-3.4.10.tar.gz創建文件夾
cd /具體安裝路徑
mkdir zookeeper-cluster
#3個子目錄分別對應三個zookeeper服務
mkdir zookeeper-cluster/server1
mkdir zookeeper-cluster/server2
mkdir zookeeper-cluster/server3
#建立三個目錄存放各自的數據文件
mkdir zookeeper-cluster/data
mkdir zookeeper-cluster/data/server1
mkdir zookeeper-cluster/data/server2
mkdir zookeeper-cluster/data/server3
#建立三個目錄存放各自的日誌文件
mkdir zookeeper-cluster/log
mkdir zookeeper-cluster/log/server1
mkdir zookeeper-cluster/log/server2
mkdir zookeeper-cluster/log/server3
#在每一個數據文件目錄中，新建一個myid文件，文件必須是唯一的服務標識，在後面的配置中會用到
echo '1' > zookeeper-cluster/data/server1/myid
echo '2' > zookeeper-cluster/data/server2/myid
echo '3' > zookeeper-cluster/data/server3/myid
• #將zookeeper複製三份
cp -rf zookeeper-3.4.10/* /具體路徑/zookeeper-cluster/server1
cp -rf zookeeper-3.4.10/* /具體路徑/zookeeper-cluster/server2
cp -rf zookeeper-3.4.10/* /具體路徑/zookeeper-cluster/server3
複製出zoo.cfg文件:
cp zookeeper-cluster/server1/zoo_sample.cfg zookeeper-cluster/server1/zoo.cfg
cp zookeeper-cluster/server2/zoo_sample.cfg zookeeper-cluster/server2/zoo.cfg
cp zookeeper-cluster/server3/zoo_sample.cfg zookeeper-cluster/server3/zoo.cfg
選擇行修改zoo.cfg
#將dataDir和dataLogDir設置爲各自獨立的目錄；然後保證clientPort不會和其它zookeeper衝突
#存儲路徑
dataDir= /具體路徑/zookeeper-cluster/data/server1
#方便查LOG
dataLogDir=/具體路徑/zookeeper-cluster/log/server1
#控制客戶的連接數，默認數爲60，太少
maxClientCnxns=300
#如果有多個ZOOKEEPER INSTANCE時
#server.X=IP:port1:port2
#X是在該zookeeper數據文件目錄中myid指定的服務ID.
#IP是當前zookeeper綁定的IP地址,因爲是演示，所以全都是localhost
#port1 是Quorum Port
#port2 是Leader Election Port
#由於3個zookeeper在同一臺機器上，需要使用不同的端口號避免衝突。
server.1=0.0.0.0:2888:3888
server.2=0.0.0.0:12888:13888
server.3=0.0.0.0:22888:23888
修改後的結果如下
/opt/zookeeper-cluster/server1/conf/zoo.cfg# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=/實際路徑/zookeeper-cluster/data/server1
dataLogDir=/實際路徑/zookeeper-cluster/log/server1
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
#
#http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1
server.1=0.0.0.0:2888:3888
server.2=0.0.0.0:12888:13888
server.3=0.0.0.0:22888:23888 /opt/zookeeper-cluster/server2/conf/zoo.cfg# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=/實際路徑/zookeeper-cluster/data/server2
dataLogDir=/實際路徑/zookeeper-cluster/log/server2
# the port at which the clients will connect
clientPort=12181
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
#
#http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1
server.1=0.0.0.0:2888:3888
server.2=0.0.0.0:12888:13888
server.3=0.0.0.0:22888:23888 /opt/zookeeper-cluster/server3/conf/zoo.cfg# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=/實際路徑/zookeeper-cluster/data/server3
dataLogDir=/實際路徑/zookeeper-cluster/log/server3
# the port at which the clients will connect
clientPort=22181
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
#
#http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1
server.1=0.0.0.0:2888:3888
server.2=0.0.0.0:12888:13888
server.3=0.0.0.0:22888:23888
1.3 啓動
然後分別啓動3個zookeeper服務
$ /實際路徑/zookeeper-cluster/server1/bin/zkServer.sh start
$ /實際路徑/zookeeper-cluster/server2/bin/zkServer.sh start
$ /實際路徑/zookeeper-cluster/server3/bin/zkServer.sh start 啓動完成後查看每個服務的狀態
$ /實際路徑/zookeeper-cluster/server1/bin/zkServer.sh status
$ /實際路徑/zookeeper-cluster/server2/bin/zkServer.sh status
$ /實際路徑/zookeeper-cluster/server3/bin/zkServer.sh status 2 安裝KAFKA
2.1安裝包：
kafka_2.11-0.10.2.0.tgz （版本號中2.11表示scala的版本，0.10.2.0才表示kafka版本）
2.2 部署
解壓
tar zxvf kafka_2.11-0.10.2.0.tgz修改配置文件
vi /實際路徑/kafka_2.11-0.10.2.0/config/server.properties#配置zookeeper連接,默認Kafka會使用ZooKeeper默認的/路徑，這樣有關Kafka的ZooKeeper配置就會散落在根路徑下面，如果你有其他的應用也在使用ZooKeeper集羣，查看ZooKeeper中數據可能會不直觀，所以強烈建議指定一個chroot路徑，直接在zookeeper.connect配置項中指定
zookeeper.connect=localhost:2181,localhost:12181,localhost:22181/kafka
#每個Kafka Broker應該配置一個唯一的ID
broker.id=0
#因爲是在同一臺機器上開多個Broker，所以使用不同的端口號區分
port=9092
#如果有多個網卡地址，也可以將不同的Broker綁定到不同的網卡
host.name=localhost
#因爲是在同一臺機器上開多個Broker,需要確保使用不同的日誌目錄來避免衝突
log.dirs=/tmp/kafka-logs-1
#設置topic可刪除，默認該項是被註釋的，需要放開
delete.topic.enable=true
#關閉自動創建topic，默認情況下Producer往一個不存在的Topic發送message時會自動創建這個Topic
auto.create.topics.enable=falsecp /實際路徑/kafka_2.11-0.10.2.0/config/server.properties /實際路徑/kafka_2.11-0.10.2.0/config/server-2.properties編輯server-2.properties
vi /實際路徑/kafka_2.11-0.10.2.0/config/server-2.properties
#配置zookeeper連接,默認Kafka會使用ZooKeeper默認的/路徑，這樣有關Kafka的ZooKeeper配置就會散落在根路徑下面，如果你有其他的應用也在使用ZooKeeper集羣，查看ZooKeeper中數據可能會不直觀，所以強烈建議指定一個chroot路徑，直接在zookeeper.connect配置項中指定
zookeeper.connect=localhost:2181,localhost:12181,localhost:22181/kafka
#每個Kafka Broker應該配置一個唯一的ID
broker.id=1
#因爲是在同一臺機器上開多個Broker，所以使用不同的端口號區分
port=19092
#如果有多個網卡地址，也可以將不同的Broker綁定到不同的網卡
host.name=localhost
#因爲是在同一臺機器上開多個Broker,需要確保使用不同的日誌目錄來避免衝突
log.dirs=/tmp/kafka-logs-2
#設置topic可刪除，默認該項是被註釋的，需要放開
delete.topic.enable=true
#關閉自動創建topic，默認情況下Producer往一個不存在的Topic發送message時會自動創建這個Topic
auto.create.topics.enable=false注：由於zookeeper.connect指定了chroot爲/kafka，所以在後面操作時連接zookeeper都需要加上/kafka，如查看topic屬性
bin/kafka-topics.sh --describe --zookeeper localhost:2181/kafka --topic test如果還需配置多個節點，只需新增server.properties配置文件即可。
2.3 啓動
kafka_2.11-0.10.2.0/bin/kafka-server-start.sh kafka_2.11-0.10.2.0/config/server.properties &
kafka_2.11-0.10.2.0/bin/kafka-server-start.sh kafka_2.11-0.10.2.0/config/server-2.properties &2.4 關閉
kafka_2.11-0.10.2.0/bin/kafka-server-stop.sh2.5 創建topic
#新創建一個topic, replication-factor表示該topic需要在不同的broker中保存幾份，這裏replication-factor設置爲2, 表示在兩個broker中保存
kafka_2.11-0.10.2.0/bin/kafka-topics.sh -topic test -create -partitions 1 -replication-factor 2 -zookeeper localhost:2181/kafka
2.6 查看指定topic屬性
kafka_2.11-0.10.2.0/bin/kafka-topics.sh -describe -zookeeper localhost:2181/kafka -topic test2.7 查看topic列表
#通過list命令查看創建的topic
kafka_2.11-0.10.2.0/bin/kafka-topics.sh -list -zookeeper localhost:2181/kafka
2.8 創建生產者
#新建連接
kafka_2.11-0.10.2.0/bin/kafka-console-producer.sh -broker-list localhost:9092 -topic test2.9 創建消費者
#重新打開一個ssh連接執行以下命令
kafka_2.11-0.10.2.0/bin/kafka-console-consumer.sh -zookeeper localhost:2181/kafka -topic test -from-beginning2.10 測試
在生成者連接中輸入內容.
test
在消費者連接中查看是否接收到消息
3 KAFKA常用命令
3.1修改topic
./bin/kafka-topics.sh -zookeeper localhost:2181/kafka -alter -topic test -partitions 23.2刪除topic
./bin/kafka-topics.sh -zookeeper localhost:2181/kafka -delete -topic test4 KAFKA相關配置參數
配置文件在config/server.properties
下面的一些配置可能是你需要進行修改的。
broker.id 整數，建議根據ip區分
log.dirs kafka存放消息文件的路徑， 默認/tmp/kafka-logs
port broker用於接收producer消息的端口
zookeeper.connnect zookeeper連接 格式爲 ip1:port,ip2:port,ip3:port
message.max.bytes 單條消息的最大長度
num.network.threads broker用於處理網絡請求的線程數 如不配置默認爲3，server.properties默認是2
num.io.threads broker用於執行網絡請求的IO線程數 如不配置默認爲8，server.properties默認是2可適當增大，
queued.max.requests 排隊等候IO線程執行的requests 默認爲500
host.name broker的hostname 默認null,建議寫主機的ip,不然消費端不配置hosts會有麻煩
num.partitions topic的默認分區數 默認1
log.retention.hours 消息被刪除前保存多少小時 默認1周168小時
auto.create.topics.enable 是否可以程序自動創建Topic 默認true,建議false
default.replication.factor 消息備份數目 默認1不做複製，建議修改
num.replica.fetchers 用於複製leader消息到follower的IO線程數 默認1