processor示例

sink group允許組織多個sink到一個實體上,sink processors能夠提供在組內所有sink之間實現負載均衡的能力,而且在失敗的情況下能夠進行故障轉移從一個sink到另外一個sink
1、default sink processor 
接收單一的sink,不強制用戶爲sink創建processor
2、failover sink processor
其維護了一個優先級列表,保證沒一個有效的事件都會被處理。
故障轉移的工作原理是將連續失敗的sink分配到一個池中,在那裏被分配一個冷凍期。一旦sink成功發送一個event,sink將會被還原到live池中。
在這配置中,要設置sink group processor爲failover,需要爲所有的sink分配優先級,所有的優先級數字必須是唯一的。此外failover time的上限可以通過maxpenalty屬性進行設置
3、load balancing sink processor
負載均衡片處理器提供在多個sink之間負載均衡流量的能力。實現支持通過round_robin或者random參數來實現負載分發,默認情況下使用round_robin,但可以通過設置覆蓋這個默認值。還可以通過集成AbstractSinkProcessor類來實現用戶自己的選擇機制。
當被調用的時候,這個選擇器通過配置的選擇規則選擇下一個sink來調用。

13、case14_failover_sink.conf
#配置內容
#name the component on this agent
a1.sources = r1
a1.channels= c1 c2
a1.sinks = k1 k2

a1.sinkgroups = g1
a1.sinkgroups.g1.sinks = k1 k2
a1.sinkgroups.g1.processor.type = failover
a1.sinkgroups.g1.processor.priority.k1 = 5
a1.sinkgroups.g1.processor.priority.k2 = 10
a1.sinkgroups.g1.processor.maxpenalty = 10000

#describe/configure the source
a1.sources.r1.type = syslogtcp
a1.sources.r1.host= localhost
a1.sources.r1.port  = 5140
a1.sources.r1.selector.type = replicating
a1.sources.r1.channels = c1 c2

#decribe the sink
a1.sinks.k1.type = avro
a1.sinks.k1.hostname= 192.168.0.106
a1.sinks.k1.port = 4443
a1.sinks.k1.channel = c1

a1.sinks.k2.type = avro
a1.sinks.k2.hostname= 192.168.0.107
a1.sinks.k2.port = 4443
a1.sinks.k2.channel = c1

#use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

a1.channels.c2.type = memory
a1.channels.c2.capacity = 1000
a1.channels.c2.transactionCapacity = 100

case14_failover_s1.conf
#配置內容
#name the component on this agent
a1.sources = r1
a1.channels= c1
a1.sinks = k1

#describe/configure the source
a1.sources.r1.type = avro
a1.sources.r1.bind = 192.168.0.105
a1.sources.r1.port  = 4443
a1.sources.r1.channels = c1

#decribe the sink
a1.sinks.k1.type = logger
a1.sinks.k1.channel = c1

#use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100


case14_failover_s2.conf
#配置內容
#name the component on this agent
a1.sources = r1
a1.channels= c1
a1.sinks = k1

#describe/configure the source
a1.sources.r1.type = avro
a1.sources.r1.bind = 192.168.0.105
a1.sources.r1.port  = 4443
a1.sources.r1.channels = c1

#decribe the sink
a1.sinks.k1.type = logger
a1.sinks.k1.channel = c1

#use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

14、case15_load_sink.conf
#配置內容
#name the component on this agent
a1.sources = r1
a1.channels= c1
a1.sinks = k1 k2

a1.sinkgroups = g1
a1.sinkgroups.g1.sinks = k1 k2
a1.sinkgroups.g1.processor.type = load_balance
a1.sinkgroups.g1.processor.backoff = true
a1.sinkgroups.g1.processor.selector= round_robin

#describe/configure the source
a1.sources.r1.type = syslogtcp
a1.sources.r1.host= localhost
a1.sources.r1.port  = 5140
a1.sources.r1.channels = c1

#decribe the sink
a1.sinks.k1.type = avro
a1.sinks.k1.hostname= 192.168.0.106
a1.sinks.k1.port = 4443
a1.sinks.k1.channel = c1

a1.sinks.k2.type = avro
a1.sinks.k2.hostname= 192.168.0.107
a1.sinks.k2.port = 4443
a1.sinks.k2.channel = c1

#use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

case15_load_s1.conf
#配置內容
#name the component on this agent
a1.sources = r1
a1.channels= c1
a1.sinks = k1

#describe/configure the source
a1.sources.r1.type = avro
a1.sources.r1.bind = 192.168.0.105
a1.sources.r1.port  = 4443
a1.sources.r1.channels = c1

#decribe the sink
a1.sinks.k1.type = logger
a1.sinks.k1.channel = c1

#use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100


case14_load_s2.conf
#配置內容
#name the component on this agent
a1.sources = r1
a1.channels= c1
a1.sinks = k1

#describe/configure the source
a1.sources.r1.type = avro
a1.sources.r1.bind = 192.168.0.105
a1.sources.r1.port  = 4443
a1.sources.r1.channels = c1

#decribe the sink
a1.sinks.k1.type = logger
a1.sinks.k1.channel = c1

#use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章