Flume單機安裝及測試

一、安裝

1、官網下載

http://flume.apache.org/download.html

[root@master softWare]# cd flume/
[root@master flume]# ls
[root@master flume]# wget https://downloads.apache.org/flume/1.9.0/apache-flume-1.9.0-bin.tar.gz
--2020-04-02 09:12:02--  https://downloads.apache.org/flume/1.9.0/apache-flume-1.9.0-bin.tar.gz
Resolving downloads.apache.org (downloads.apache.org)... 88.99.95.219, 2a01:4f8:10a:201a::2
Connecting to downloads.apache.org (downloads.apache.org)|88.99.95.219|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 67938106 (65M) [application/x-gzip]
Saving to: ‘apache-flume-1.9.0-bin.tar.gz’

100%[=============================================================================================>] 67,938,106   320KB/s   in 4m 17s 

2020-04-02 09:16:31 (258 KB/s) - ‘apache-flume-1.9.0-bin.tar.gz’ saved [67938106/67938106]

[root@master flume]# ls
apache-flume-1.9.0-bin.tar.gz
[root@master flume]#

2、解壓安裝包並重命名

[root@master conf]# tar -zxvf apache-flume-1.9.0-bin.tar.gz
[root@master flume]# ls
apache-flume-1.9.0-bin  apache-flume-1.9.0-bin.tar.gz
[root@master flume]# mv apache-flume-1.9.0-bin flume-1.9.0
[root@master flume]# ls
apache-flume-1.9.0-bin.tar.gz  flume-1.9.0

3、在conf/flume-env.sh中配置jdk路徑

[root@master flume-1.9.0]# cd conf/
[root@master conf]# ls
flume-conf.properties.template  flume-env.ps1.template  flume-env.sh.template  log4j.properties
[root@master conf]# mv flume-env.sh.template flume-env.sh
[root@master conf]# vim flume-env.sh

二、測試

1、需求

flume監聽一個端口，並以日誌的形式打印出端口上面輸入的信息；

2、配置採集方案

在conf/下面新建一個文件netcat-logger.conf（注意命名格式爲:xxx.conf），並輸入下面信息

[root@master conf]# vim netcat-logger.conf

#從網絡端口接受數據，下沉到logger
#採集配置文件，netcat-logger.conf
 
#定義這個agent中各組件的名字
a1.sources=r1
a1.sinks=k1
a1.channels=c1
 
#描述和配置source組件：r1
# netcat類型用來監聽端口數據源
a1.sources.r1.type=netcat
a1.sources.r1.bind=localhost
a1.sources.r1.port=8888

#描述和配置sink組件：k1
a1.sinks.k1.type=logger

#描述和配置channel組件，此處使用是內存緩存的方式
a1.channels.c1.type=memory
a1.channels.c1.capacity=1000
a1.channels.c1.transactionCapacity=100

#描述和配置source channel sink之間的連接關係
a1.sources.r1.channels=c1
a1.sinks.k1.channel=c1

3、啓動agent去採集數據

[root@master bin]# ./flume-ng agent -c conf -f ../conf/netcat-logger.conf -n a1 -Dflume.root.logger=INFO,console

第二種啓動方式：

[root@master bin]# ./flume-ng agent --conf conf --conf-file ../conf/netcat-logger.conf --name a1 -Dflume.root.logger=INFO,console

命令說明：

-c conf 指定flume自身的配置文件所在目錄
-f ../conf/netcat-logger.conf 指定我們所描述的採集方案
-n a1 指定本次啓動agent的名字，與採集方案配置文件中一致

--conf conf： flume自身所需的conf文件路徑
--conf-file ../conf/netcat-logger.conf： 自定義的flume配置文件，也就是採集方案文件
--name a1： 自定義的flume配置文件中agent的name

4、向端口輸入數據

[root@master ~]# telnet localhost 8888
Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
gg
OK
yy
OK
gg
OK
jjjjj
OK

5、驗證flume輸出

20/04/02 09:29:24 INFO node.Application: Starting new configuration:{ sourceRunners:{r1=EventDrivenSourceRunner: { source:org.apache.flume.source.NetcatSource{name:r1,state:IDLE} }} sinkRunners:{k1=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@4552d7d1 counterGroup:{ name:null counters:{} } }} channels:{c1=org.apache.flume.channel.MemoryChannel{name: c1}} }
20/04/02 09:29:24 INFO node.Application: Starting Channel c1
20/04/02 09:29:24 INFO node.Application: Waiting for channel: c1 to start. Sleeping for 500 ms
20/04/02 09:29:24 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: CHANNEL, name: c1: Successfully registered new MBean.
20/04/02 09:29:24 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: c1 started
20/04/02 09:29:24 INFO node.Application: Starting Sink k1
20/04/02 09:29:24 INFO node.Application: Starting Source r1
20/04/02 09:29:24 INFO source.NetcatSource: Source starting
20/04/02 09:29:24 INFO source.NetcatSource: Created serverSocket:sun.nio.ch.ServerSocketChannelImpl[/127.0.0.1:8888]
20/04/02 09:31:22 INFO sink.LoggerSink: Event: { headers:{} body: 67 67 0D                                        gg. }
20/04/02 09:31:29 INFO sink.LoggerSink: Event: { headers:{} body: 79 79 0D                                        yy. }
20/04/02 09:31:30 INFO sink.LoggerSink: Event: { headers:{} body: 67 67 0D                                        gg. }
20/04/02 09:31:32 INFO sink.LoggerSink: Event: { headers:{} body: 6A 6A 6A 6A 6A 0D                               jjjjj. }

6、注意

一定要先啓動flume,再啓動telnet 192.168.230.21 8888,否則檢測失敗；

如果沒有telnet,要yum install telnet安裝；