Flume 流入遠程 hbase

在測試flume的過程中,一直是將hbase和flume在同一臺機器上做測試,由於flume默認是插入本地的hbase中,所以一直沒有問題。

當真正部署到實際環境中是發現f需要再做設置才能流入遠程hbase數據庫中,flume端要做配置,hbase主機中的hosts文件也做修改

在此貼出配置文件

flume的conf文件

#Name the  components on this agent
a1.sources  = r1
a1.sinks =  k1
a1.channels  = c1

#  Describe/configure the source
a1.sources.r1.type  = spooldir
a1.sources.r1.spoolDir  = /home/scut/Downloads/testFlume

# Describe  the sink
a1.sinks.k1.type  = org.apache.flume.sink.hbase.AsyncHBaseSink
a1.sinks.k1.zookeeperQuorum = master,slave1,slave2
a1.sinks.k1.table = Router
a1.sinks.k1.columnFamily = log 
a1.sinks.k1.serializer.payloadColumn=serviceTime,browerOS,clientTime,screenHeight,screenWidth,url,userAgent,mobileDevice,gwId,mac
a1.sinks.k1.serializer = org.apache.flume.sink.hbase.BaimiAsyncHbaseEventSerializer

# Use a  channel which buffers events in memory
a1.channels.c1.type  = memory
a1.channels.c1.capacity  = 1000
a1.channels.c1.transactionCapacity  = 100

# Bind the  source and sink to the channel
a1.sources.r1.channels  = c1
a1.sinks.k1.channel  = c1

hbase master的 /etc/hosts文件

192.168.100.1	master hbase localhost ubuntu #將192.168.100.1映射爲master
192.168.100.2	slave1                        #將192.168.100.2映射爲slave1
192.168.100.3	slave2                        #將192.168.100.3映射爲slave2

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

主要的不同是刪除127.0.0.1/127.0.1.1個localhost和主機名的映射關係

第一行中,192.168.100.1映射爲 master,hbase, localhost, ubuntu。ubuntu是主機名,hbase是爲了讓flume能夠流入遠程hbase,具體原理是什麼,我也不清楚,只是需要建立這種映射關係。


hbase 的slave的/etc/hosts文件

<pre name="code" class="java">
192.168.100.1	master 
192.168.100.2	slave1 localhost ubuntu
192.168.100.3	slave2
# The following lines are desirable for IPv6 capable hosts::1 ip6-localhost ip6-loopbackfe00::0 ip6-localnetff00::0 ip6-mcastprefixff02::1 ip6-allnodesff02::2 ip6-allrouters


hbase中的slave節點的/etc/hosts配置相似,刪除127.0.0.1/127.0.1.1和localhost於主機名之間的映射關係。

而是將localhost/ubuntu(主機名)直接映射爲本地IP地址


發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章