在測試flume的過程中,一直是將hbase和flume在同一臺機器上做測試,由於flume默認是插入本地的hbase中,所以一直沒有問題。
當真正部署到實際環境中是發現f需要再做設置才能流入遠程hbase數據庫中,flume端要做配置,hbase主機中的hosts文件也做修改
在此貼出配置文件
flume的conf文件
#Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1
# Describe/configure the source
a1.sources.r1.type = spooldir
a1.sources.r1.spoolDir = /home/scut/Downloads/testFlume
# Describe the sink
a1.sinks.k1.type = org.apache.flume.sink.hbase.AsyncHBaseSink
a1.sinks.k1.zookeeperQuorum = master,slave1,slave2
a1.sinks.k1.table = Router
a1.sinks.k1.columnFamily = log
a1.sinks.k1.serializer.payloadColumn=serviceTime,browerOS,clientTime,screenHeight,screenWidth,url,userAgent,mobileDevice,gwId,mac
a1.sinks.k1.serializer = org.apache.flume.sink.hbase.BaimiAsyncHbaseEventSerializer
# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
hbase master的 /etc/hosts文件
192.168.100.1 master hbase localhost ubuntu #將192.168.100.1映射爲master
192.168.100.2 slave1 #將192.168.100.2映射爲slave1
192.168.100.3 slave2 #將192.168.100.3映射爲slave2
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
主要的不同是刪除127.0.0.1/127.0.1.1個localhost和主機名的映射關係
第一行中,192.168.100.1映射爲 master,hbase, localhost, ubuntu。ubuntu是主機名,hbase是爲了讓flume能夠流入遠程hbase,具體原理是什麼,我也不清楚,只是需要建立這種映射關係。
hbase 的slave的/etc/hosts文件
<pre name="code" class="java">
192.168.100.1 master
192.168.100.2 slave1 localhost ubuntu
192.168.100.3 slave2
# The following lines are desirable for IPv6 capable hosts::1 ip6-localhost ip6-loopbackfe00::0 ip6-localnetff00::0 ip6-mcastprefixff02::1 ip6-allnodesff02::2 ip6-allroutershbase中的slave節點的/etc/hosts配置相似,刪除127.0.0.1/127.0.1.1和localhost於主機名之間的映射關係。
而是將localhost/ubuntu(主機名)直接映射爲本地IP地址