Kafka與flume
1）準備jar包
1、將Kafka主目錄lib下的如下jar拷貝至Flume的lib目錄下
kafka_2.10-0.8.2.1.jar、kafka-clients-0.8.2.1.jar、jopt-simple-3.2.jar、metrics-core-2.2.0.jar、scala-library-2.10.4.jar、zkclient-0.3.jar等
2、將如下jar拷貝至flume主目錄下,上述1是其依賴的jar包
下載flume、kafka插件包,flumeng-kafka-plugin.jar
2）配置flume.conf文件,如下
#agent section
producer.sources = s
producer.channels = c
producer.sinks = r
#source section
#producer.sources.s.type = seq
producer.sources.s.channels = c
producer.sources.s.type = exec
producer.sources.s.command = tail -fn 1 /letv/logs/test.log
# Each sink's type must be defined
producer.sinks.r.type = org.apache.flume.plugins.KafkaSink
producer.sinks.r.metadata.broker.list=10.148.13.10:9092,10.148.13.11:9092,10.148.13.12:9092,10.148.13.13:9092,10.148.13.14:9092,10.148.13.15:9092,10.148.13.16:9092,10.148.13.17:9092,10.148.13.18:9092,10.148.13.19:9092
#producer.sinks.r.partition.key=0
#producer.sinks.r.partitioner.class=org.apache.flume.plugins.SinglePartition
producer.sinks.r.serializer.class=kafka.serializer.StringEncoder
producer.sinks.r.request.required.acks=-1
producer.sinks.r.max.message.size=1000000
producer.sinks.r.producer.type=sync
producer.sinks.r.custom.encoding=UTF-8
producer.sinks.r.custom.topic.name=test-topic
#Specify the channel the sink should use
producer.sinks.r.channel = c
# Each channel's type is defined.
producer.channels.c.type = memory
producer.channels.c.capacity = 1000000
producer.channels.c.transactionCapacity = 1000000
測試：
- 啓動zookeeper服務，kafka依賴的組件
- 啓動kafka服務，同時創建topic名爲test-topic
./kafka-topics.sh --create --zookeeper master-active:2181 --replication-factor 1 --partitions 1 --topic test-topic
- 啓動Flume服務
./flume-ng agent -n agent -f ../conf/flume-kafka.conf -Dflume.root.logger=INFO,console
- 使用echo “hello world , kafka and flume !” >> /letv/logs/test.log
- 啓動kafka的consumer即可查看輸入文件流
./kafka-console-consumer.sh --zookeeper master-active:2181 --topic test-topic--from-beginning
