下載apache-flume-1.6.0-bin.tar.gz
hadoop1這個節點配置flume連接kafka即可
vim conf/flume-conf.properties文件，配置如下
a1.sources = r1
a1.sinks = k1
a1.channels = c1
# Describe/configure the source
a1.sources.r1.type=avro
a1.sources.r1.bind=hadoop1
a1.sources.r1.port=41414
# Describe the sink
a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink
a1.sinks.k1.topic = testflume
a1.sinks.k1.brokerList = 192.168.25.151:9092,192.168.25.152:9092,192.168.25.153:9092,192.168.25.154:9092
a1.sinks.k1.requiredAcks = 1
a1.sinks.k1.batchSize = 20
a1.sinks.k1.channel = c1
# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000000
a1.channels.c1.transactionCapacity = 10000
# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
注意這裏需要根據實際需求來配置sources，這裏是用avro的方式配置監聽master本機的41414端口注意這裏是啓動agent的配置 後續的flume client也需要用到這個配置，下面的sink配置a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink是固定的不需要修改，而a1.sinks.k1.topic = testflume是創建的話題名，需要根據自己需要來改，而a1.sinks.k1.brokerList = 192.168.57.4:9092,192.168.57.5:9092,192.168.57.6:9092是根據實際的kafka集羣情況配置的
啓動
zookeeper
kafka
單節點flume啓動
flume連接到kafka
bin/flume-ng agent -c ./conf/ -f conf/flume-conf.properties -Dflume.root.logger=DEBUG,console -n a1
注意這裏的a1指的是配置文件中的agent名字a1不是隨意寫的
這裏的flume對接kafka實際已經完成，接下來就是測試
將flume工程跑起來
在kafka集羣中隨意一個節點敲以下代碼
./bin/kafka-console-consumer.sh --zookeeper hadoop1:2181,hadoop2:2181,hadoop3:2181 --from-beginning --topic testflume
注意這裏的topic的名字要和配置文件中一致
執行上面的main方法作爲flume的client端來產生數據，可以在上面的consumer監聽裏面看到結果
接下來啓動storm連接到kafka
啓動storm集羣，hadoop1節點啓動nimbus，hadoop1啓動storm UI，hadoop1-hadoop4啓動supervisor
將項目storm中的logFilterTopology啓動起來
在stormUI上監控該DAG
在kafka上監控topic test
./bin/kafka-console-consumer.sh --zookeeper hadoop1:2181,hadoop2:2181,hadoop3:2181 --from-beginning --topic test