hadoop編譯

在YARN自帶的軟件包中有關於編譯的說明,具體參見:BUILDING.txt。
在RetHat中編譯YARN,Jdk爲1.6.0_23.

1.下載YARN源碼包
下載地址:http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.0.2-alpha/
(這裏下載的版本爲2.0.2),下載hadoop-2.0.2-alpha-src.tar.gz.
下載後解壓到/home/wyf目錄下。

2.安裝Maven
YARN使用Maven作爲編譯工具,以前沒用過,這次只好硬着頭皮試試了。首先下載Maven軟件包,下載地址爲:http://labs.mop.com/apache-mirror/maven/maven-3/3.0.4/binaries/apache-maven-3.0.4-bin.tar.gz
下載後解壓,設置MAVEN_HOME和PATH變量。設置好後,可以使用mvn-version檢查是否安裝成功。

3.安裝ProtoBuf
爲啥YARN編譯使用的工具我都沒用過,囧。。看來學習的太少了
同樣下載,解壓。下載地址:http://protobuf.googlecode.com/files/protobuf-2.4.1.tar.bz2
解壓: tarjxvf protobuf-2.4.1.tar.bz2
依次執行:./configure,  make,   sudo makeinstall,這三個命令安裝ProtoBuffer。
在執行./configure命令時,可以指定安裝目錄的前綴(默認安裝到/usr/local目錄下),指定前綴的方法爲(舉例):./configure--prefix=/home/wyf/protobuf,這樣的話,protobuf就會安裝到/home/wyf/protobuf目錄下,安裝後在該目錄下可以看到三個目錄:include、bin、lib。(之前我安裝好幾次都是隻有lib目錄,沒有其餘兩個,好像是解壓方法不對,最後使用上面所寫的解壓方法後安裝成功)。修改/etc/profile文件,加入exportLD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/wyf/protobuf/lib
exportPATH=$PATH:/home/wyf/protobuf/bin
最後使用protoc--version檢查ProtoBuffer是否安裝成功。

4.安裝autotool
依然沒用過。。。
命令安裝:sudo yum install autoconfautomake libtool

5.編譯
第一次編譯保證網絡連接,因爲要從網絡上下載很多支持的軟件包。
若解壓後的YARN源碼位置爲/home/wyf/hadoop-2.0.2-alpha-src,則cd到該目錄下,即可開始編譯。
編譯命令:mvn package -Pdist,native-DskipTests
接下來是漫長的等待。。。。直到出現所有Project均SUCCESS的界面,證明編譯完成且成功!
編譯成功應顯示的信息:
[INFO]------------------------------------------------------------------------
[INFO] ReactorSummary:
[INFO] 
[INFO] Apache Hadoop Main................................ SUCCESS [6.256s]
[INFO] Apache Hadoop ProjectPOM ......................... SUCCESS [5.964s]
[INFO] Apache HadoopAnnotations ......................... SUCCESS [4.507s]
[INFO] Apache Hadoop Assemblies.......................... SUCCESS [0.358s]
[INFO] Apache Hadoop ProjectDist POM .................... SUCCESS [4.044s]
[INFO] Apache Hadoop Auth................................ SUCCESS [4.816s]
[INFO] Apache Hadoop AuthExamples ....................... SUCCESS [4.517s]
[INFO] Apache Hadoop Common.............................. SUCCESS [3:37.475s]
[INFO] Apache Hadoop CommonProject ...................... SUCCESS [3.391s]
[INFO] Apache Hadoop HDFS................................ SUCCESS [3:25.417s]
[INFO] Apache Hadoop HttpFS.............................. SUCCESS [45.369s]
[INFO] Apache Hadoop HDFSBookKeeper Journal ............. SUCCESS [19.111s]
[INFO] Apache Hadoop HDFSProject ........................ SUCCESS [0.225s]
[INFO] hadoop-yarn....................................... SUCCESS[1.273s]
[INFO] hadoop-yarn-api................................... SUCCESS[1:49.559s]
[INFO] hadoop-yarn-common................................ SUCCESS [1:14.722s]
[INFO] hadoop-yarn-server................................ SUCCESS [3.524s]
[INFO]hadoop-yarn-server-common ......................... SUCCESS[20.320s]
[INFO]hadoop-yarn-server-nodemanager .................... SUCCESS[22.533s]
[INFO]hadoop-yarn-server-web-proxy ...................... SUCCESS[4.906s]
[INFO]hadoop-yarn-server-resourcemanager ................ SUCCESS[28.781s]
[INFO] hadoop-yarn-server-tests.......................... SUCCESS [2.583s]
[INFO] hadoop-yarn-client................................ SUCCESS [4.943s]
[INFO] hadoop-yarn-applications.......................... SUCCESS [0.149s]
[INFO]hadoop-yarn-applications-distributedshell ......... SUCCESS[3.314s]
[INFO] hadoop-mapreduce-client........................... SUCCESS [0.364s]
[INFO]hadoop-mapreduce-client-core ...................... SUCCESS[47.326s]
[INFO]hadoop-yarn-applications-unmanaged-am-launcher .... SUCCESS[11.138s]
[INFO] hadoop-yarn-site.................................. SUCCESS [0.267s]
[INFO] hadoop-yarn-project............................... SUCCESS [11.935s]
[INFO]hadoop-mapreduce-client-common .................... SUCCESS[47.074s]
[INFO]hadoop-mapreduce-client-shuffle ................... SUCCESS[11.811s]
[INFO]hadoop-mapreduce-client-app ....................... SUCCESS[14.339s]
[INFO]hadoop-mapreduce-client-hs ........................ SUCCESS[7.141s]
[INFO]hadoop-mapreduce-client-jobclient ................. SUCCESS[10.101s]
[INFO] Apache Hadoop MapReduceExamples .................. SUCCESS [7.732s]
[INFO] hadoop-mapreduce.................................. SUCCESS [8.388s]
[INFO] Apache Hadoop MapReduceStreaming ................. SUCCESS [5.928s]
[INFO] Apache HadoopDistributed Copy .................... SUCCESS[15.718s]
[INFO] Apache Hadoop Archives............................ SUCCESS [10.997s]
[INFO] Apache Hadoop Rumen............................... SUCCESS [9.091s]
[INFO] Apache Hadoop Gridmix............................. SUCCESS [5.791s]
[INFO] Apache Hadoop Data Join........................... SUCCESS [3.456s]
[INFO] Apache Hadoop Extras.............................. SUCCESS [4.658s]
[INFO] Apache Hadoop Pipes............................... SUCCESS [3.120s]
[INFO] Apache Hadoop Tools Dist.......................... SUCCESS [2.242s]
[INFO] Apache Hadoop Tools............................... SUCCESS [0.085s]
[INFO] Apache HadoopDistribution ........................ SUCCESS[42.718s]
[INFO] Apache Hadoop Client.............................. SUCCESS [17.661s]
[INFO] Apache HadoopMini-Cluster ........................ SUCCESS [0.736s]
[INFO]------------------------------------------------------------------------
[INFO] BUILDSUCCESS
[INFO]------------------------------------------------------------------------
[INFO] Total time:18:21.523s
[INFO] Finished at: Thu Jan 3115:28:01 CST 2013

6.錯誤處理
在編譯過程中出現了大致兩類錯誤,一是由於網絡原因,需要的軟件包沒有下載下來,導致編譯不能繼續,編譯失敗;二是CMake沒有安裝。
錯誤1:軟件包沒下載
在編譯的過程中會看到YARN自動download很多需要的軟件包,但有時會下載不成功。錯誤提示大致下面這個樣子:
[ERROR] Failed to execute goalorg.apache.maven.plugins:maven-dependency-plugin:2.1:build-classpath(build-classpath) on project hadoop-project: Executionbuild-classpath of goalorg.apache.maven.plugins:maven-dependency-plugin:2.1:build-classpathfailed: Plugin org.apache.maven.plugins:maven-dependency-plugin:2.1or one of its dependencies could not be resolved: Could nottransfer artifact org.apache.maven.doxia:doxia-core:jar:1.0-alpha-7from/to central (http://repo.maven.apache.org/maven2): GET requestof:org/apache/maven/doxia/doxia-core/1.0-alpha-7/doxia-core-1.0-alpha-7.jarfrom central failed: Connection reset -> [Help 1]
[ERROR] 
[ERROR] To see the full stacktrace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the-X switch to enable full debug logging.
[ERROR] 
[ERROR] For more informationabout the errors and possible solutions, please read the followingarticles:
[ERROR] [Help 1]http://cwiki.apache.org/confluence/display/MAVEN/PluginResolutionException
[ERROR] 
[ERROR] After correcting theproblems, you can resume the build with the command
[ERROR]   mvn-rf :hadoop-project
在編譯的過程中出現了好幾次ResolutionException(還有DependencyResolutionException),遇到這個問題,不用修改東西,直接運行命令繼續編譯就好。YARN會繼續從下載失敗的軟件包開始下載並繼續編譯。
錯誤2:CMake沒有安裝
錯誤:
[ERROR] Failed to execute goalorg.apache.maven.plugins:maven-antrun-plugin:1.6:run (make) onproject hadoop-common: An Ant BuildException has occured: Executefailed: java.io.IOException: Cannot run program "cmake" (in directory"/home/wyf/hadoop-2.0.2-alpha-src/hadoop-common-project/hadoop-common/target/native"):java.io.IOException: error=2, No such file or directory -> [Help1]
[ERROR] 
[ERROR] To see the full stacktrace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the-X switch to enable full debug logging.
[ERROR] 
[ERROR] For more informationabout the errors and possible solutions, please read the followingarticles:
[ERROR] [Help 1]http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
這是安裝CMake:
sudo yum installcmake
Complete!後繼續編譯就可以了。

7.驗證
驗證編譯成功的方法最簡單的爲使用自己編譯的jar文件,運行Hadoop系統。將編譯生成的jar文件(一般在target目錄下,由於YARN各個部分分的很細,jar包分佈在好多目錄下的target目錄中,需要耐心的找找),替換hadoop-2.0.2-alpha/share/hadoop目錄下各個部分對應的jar包,配置hadoop(這裏略),在master啓動hadoop。

8.疑問
如何只編譯YARN中的一部分代碼(如YARN模塊,或HDFS模塊,或MapReduce模塊),還沒有找到方法。
發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章