在今年的5月22號，Flume-ng 1.5.0版本正式發佈，關於Flume-ng 1.5.0版本的新特性可以參見本博客的《Apache
Flume-ng 1.5.0正式發佈》進行了解。關於Apache flume-ng 1.4.0版本的編譯可以參見本博客《基於Hadoop-2.2.0編譯flume-ng 1.4.0及錯誤解決》。本文將講述如何用Maven編譯Apache
flume-ng 1.5.0源碼。
一、到官方網站下載相應版本的flume-ng源碼
2 |
#
tar -zxf apache-flume-1.5.0-src.tar.gz |
3 |
#
cd apache-flume-1.5.0-src |
二、用maven在相應的Hadoop上編譯Flume-ng 1.5.0源碼
Flume-ng 1.5.0默認的Hadoop版本是1.0.1，Hbase是0.92.1。我們可以在編譯的時候加上一些參數，指定Hadoop 2.2.0版本進行編譯：
三、Hadoop、Hbase對應的test不能下載
如果在編譯的時候出現了類似如下的情況，請將相應pom.xml文件中的Hadoop-test和Hbase-test依賴包去掉：
07 |
過往記憶博客，專注於hadoop、hive、spark、shark、flume的技術博客，大量的乾貨 |
08 |
過往記憶博客微信公共帳號：iteblog_hadoop |
11 |
<groupId>org.apache.hbase</groupId> |
12 |
<artifactId>hbase</artifactId> |
13 |
<version>${hbase.version}</version> |
14 |
<classifier>tests</classifier> |
19 |
<groupId>org.apache.hadoop</groupId> |
20 |
<artifactId>hadoop-test</artifactId> |
21 |
<version>${hadoop.version}</version> |
四、elasticsearch-0.90.1.jar read錯誤
如果你在編譯的過程中出現一下異常：
1 |
[ERROR]
Failed to execute goal org.apache.maven.plugins: |
2 |
maven-compiler-plugin:2.3.2:compile
(default-compile)
on |
3 |
project
flume-ng-elasticsearch-sink: Compilation failure |
4 |
[ERROR]
error: error reading |
5 |
/org/elasticsearch/elasticsearch/0.90.1/elasticsearch-0.90.1.jar; |
6 |
error
in opening zip file |
只需要到你maven本地倉庫將elasticsearch-0.90.1.jar文件刪除，然後重新下載即可：
1 |
#
rm elasticsearch-0.90.1.jar |
五、ua-parser-1.3.0下載失敗
編譯過程中，下載ua-parser-1.3.0.pom可能失敗，出現如下錯誤：
01 |
[ERROR]
Failed to execute goal on project flume-ng-morphline-solr-sink: |
02 |
Could
not resolve dependencies for
project |
03 |
org.apache.flume.flume-ng-sinks:flume-ng-morphline-solr-sink:jar:1.5.0: |
04 |
Failed
to collect dependencies for
[org.apache.flume:flume-ng-core:jar:1.5.0 |
05 |
(compile),
org.slf4j:slf4j-api:jar:1.6.1
(compile), |
06 |
org.kitesdk:kite-morphlines-all:pom:0.12.0
(compile?), |
07 |
org.slf4j:jcl-over-slf4j:jar:1.6.1
(provided), |
08 |
org.apache.solr:solr-test-framework:jar:4.3.0
(test), |
09 |
org.kitesdk:kite-morphlines-solr-core:jar:tests:0.12.0
(test), |
10 |
junit:junit:jar:4.10
(test)]: Failed to read artifact descriptor |
11 |
for
ua_parser:ua-parser:jar:1.3.0:
Could not transfer artifact |
12 |
ua_parser:ua-parser:pom:1.3.0
from/to maven-twttr (http: |
只需要加一個新的repository即可：
<repository>
<id>p2.jfrog.org</id>
<url>http://p2.jfrog.org/libs-releases</url>
</repository>
假如上面的repository仍然不能用，可以google “ua-parser-1.3.0.pom”獲取其他源。
六、編譯Flume-ng 1.5.0源碼
1 |
#
mvn install -Phadoop-2
-DskipTests -Dtar |
編譯需要一段時間，這個需要看你編譯時的網速。如果一切順利的話，你將會看到以下輸出，說明你的Flume-ng 1.5.0編譯成功：
01 |
[INFO]
------------------------------------------------------------------------ |
02 |
[INFO]
Reactor Summary: |
04 |
[INFO]
Apache Flume ...................................... SUCCESS [1.360s] |
05 |
[INFO]
Flume NG SDK ...................................... SUCCESS [2.903s] |
06 |
[INFO]
Flume NG Configuration ............................ SUCCESS [0.483s] |
07 |
[INFO]
Flume NG Core ..................................... SUCCESS [2.531s] |
08 |
[INFO]
Flume NG Sinks .................................... SUCCESS [0.110s] |
09 |
[INFO]
Flume NG HDFS Sink ................................ SUCCESS [1.668s] |
10 |
[INFO]
Flume NG IRC Sink ................................. SUCCESS [0.484s] |
11 |
[INFO]
Flume NG Channels ................................. SUCCESS [0.074s] |
12 |
[INFO]
Flume NG JDBC channel ............................. SUCCESS [0.579s] |
13 |
[INFO]
Flume NG file-based channel ....................... SUCCESS [0.712s] |
14 |
[INFO]
Flume NG Spillable Memory channel ................. SUCCESS [0.552s] |
15 |
[INFO]
Flume NG Node ..................................... SUCCESS [0.619s] |
16 |
[INFO]
Flume NG Embedded Agent ........................... SUCCESS [1.001s] |
17 |
[INFO]
Flume NG HBase Sink ............................... SUCCESS [2.373s] |
18 |
[INFO]
Flume NG ElasticSearch Sink ....................... SUCCESS [0.899s] |
19 |
[INFO]
Flume NG Morphline Solr Sink ...................... SUCCESS [2.483s] |
20 |
[INFO]
Flume NG Kite Dataset Sink ........................ SUCCESS [35.925s] |
21 |
[INFO]
Flume Sources ..................................... SUCCESS [0.054s] |
22 |
[INFO]
Flume Scribe Source ............................... SUCCESS [1.231s] |
23 |
[INFO]
Flume JMS Source .................................. SUCCESS [2.133s] |
24 |
[INFO]
Flume Twitter Source .............................. SUCCESS [1.204s] |
25 |
[INFO]
Flume legacy Sources .............................. SUCCESS [0.043s] |
26 |
[INFO]
Flume legacy Avro source .......................... SUCCESS [1.234s] |
27 |
[INFO]
Flume legacy Thrift Source ........................ SUCCESS [1.320s] |
28 |
[INFO]
Flume NG Clients .................................. SUCCESS [0.038s] |
29 |
[INFO]
Flume NG Log4j Appender ........................... SUCCESS [6.175s] |
30 |
[INFO]
Flume NG Tools .................................... SUCCESS [1.559s] |
31 |
[INFO]
Flume NG distribution ............................. SUCCESS [1:51.993s] |
32 |
[INFO]
Flume NG Integration Tests ........................ SUCCESS [1.517s] |
33 |
[INFO]
------------------------------------------------------------------------ |
35 |
[INFO]
------------------------------------------------------------------------ |
36 |
[INFO]
Total time: 3:03.921s |
37 |
[INFO]
Finished at: Mon Jun 16
23:37:51
CST 2014 |
38 |
[INFO]
Final Memory: 135M/745M |
39 |
[INFO]
------------------------------------------------------------------------ |
最後將會在flume-ng-dist/target目錄下產生以下的文件：
2 |
drwxr-xr-x 3
wyp wyp 4096
Jun 16
23:36
apache-flume-1.5.0-bin |
3 |
-rw-r--r-- 1
wyp wyp 27307850
Jun 16
23:36
apache-flume-1.5.0-bin.tar.gz |
4 |
drwxr-xr-x 3
wyp wyp 4096
Jun 16
23:36
apache-flume-1.5.0-src |
5 |
-rw-r--r-- 1
wyp wyp 2543174
Jun 16
23:37
apache-flume-1.5.0-src.tar.gz |
6 |
drwxr-xr-x 2
wyp wyp 4096
Jun 16
23:36
archive-tmp |
7 |
drwxr-xr-x 3
wyp wyp 4096
Jun 16
23:36
maven-shared-archive-resources |
8 |
-rw-r--r-- 1
wyp wyp 990
Jun 16
23:37
rat.txt |
apache-flume-1.5.0-bin這個文件夾裏面的東西可以直接拿來用。
