重新編譯hadoop 32bit-64bit

先安裝macports

參考:http://blog.csdn.net/bamuta/article/details/13506893
下載程序代碼
機器得連網,如果沒聯網找可以聯網的機器下載,但是編譯時還是要下載一些東西,所以,實在不行。最好找相同平臺(可以是虛擬機)能上網的機器做下面工作,弄好了再拷回來。
svn cohttp://svn.apache.org/repos/asf/hadoop/common/tags/release-2.2.0
都下載到這了:
[hadoop@hadoop01 hadoop]$ ls
BUILDING.txt  hadoop-common-project  hadoop-maven-plugins  hadoop-tools
dev-support  hadoop-dist hadoop-minicluster  hadoop-yarn-project
hadoop-assemblies  hadoop-hdfs-project  hadoop-project  pom.xml
hadoop-client  hadoop-mapreduce-project  hadoop-project-dist
安裝開發環境
1.必要的包
sudo port install libtool
sudo port install cmake
sudo port install ncurses
sudo port install openssl
sudo port install maven3  

安裝java7
偶是小白,今天被人科普了
JDK是可以進行編譯的,而jre只能用來運行,反正在java上要注意區分2者的區別,以後詳細學習java的時候再進一步瞭解吧。

官網下載mac版本的安裝文件
官網地址: http://www.oracle.com/technetwork/java/javase/downloads/index.html
選擇jdk7的Mac OS X x64版本,jdk6沒有Mac OS X版本的
下載對應的dmg文件安裝
安裝好在命令行輸入:java -version 現實下面的內容
java version "1.7.0_17" Java(TM) SE Runtime Environment (build 1.7.0_17-b02) Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode)
java就安裝完成了,也無需配置環境變量
刪除的方法也很簡單,在命令行中到 /Library/Java/JavaVirtualMachines 這個目錄下,刪除對應的jdk就可以了

修改.bash_profile,添加
export PATH=${PATH}:/Library/Java/JavaVirtualMachines/jdk1.7.0_45.jdk/Contents/Home
export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.7.0_45.jdk/Contents/Home  

gcc也很重要
gcc -v
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk/usr/include/c++/4.2.1
Apple LLVM version 5.0 (clang-500.2.79) (based on LLVM 3.3svn)
Target: x86_64-apple-darwin13.0.0
Thread model: posix

mvn3 -v
Apache Maven 3.0.5 (r01de14724cdef164cd33c7c8c2fe155faf9602da; 2013-02-19 21:51:28+0800)
Maven home: /opt/local/share/java/maven3
Java version: 1.7.0_45, vendor: Oracle Corporation
Java home: /Library/Java/JavaVirtualMachines/jdk1.7.0_45.jdk/Contents/Home/jre
Default locale: zh_CN, platform encoding: UTF-8
OS name: "mac os x", version: "10.9", arch: "x86_64", family: "mac"

qqqq

mvn3 package -Pdist,native -DskipTests -Dtar

報錯:
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 15.951s
[INFO] Finished at: Fri Dec 13 18:06:44 CST 2013
[INFO] Final Memory: 39M/525M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.5.1:testCompile (default-testCompile) on project hadoop-auth: Compilation failure: Compilation failure:
[ERROR] /Users/JuneMAC/hadoop/release-2.2.0/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java:[88,11] 錯誤: 無法訪問AbstractLifeCycle
[ERROR] 找不到org.mortbay.component.AbstractLifeCycle的類文件
[ERROR] /Users/JuneMAC/hadoop/release-2.2.0/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java:[96,29] 錯誤: 無法訪問LifeCycle
[ERROR] 找不到org.mortbay.component.LifeCycle的類文件
[ERROR] /Users/JuneMAC/hadoop/release-2.2.0/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java:[98,10] 錯誤: 找不到符號
[ERROR] 符號:  方法 start()
[ERROR] 位置: 類型爲Server的變量 server
[ERROR] /Users/JuneMAC/hadoop/release-2.2.0/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java:[104,12] 錯誤: 找不到符號
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1]http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]  mvn -rf :hadoop-auth

相關討論改錯誤的鏈接:
https://issues.apache.org/jira/browse/HADOOP-10110

解決方案:
vi hadoop-common-project/hadoop-auth/pom.xml


org.mortbay.jetty
jetty
test

這段之前加一段:

org.mortbay.jetty
jetty-util
test

mvn3 package -Pdist,native -DskipTests -Dtar
hadoop-auth終於成功了,然後報錯:
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:2.2.0:protoc (compile-protoc) on project hadoop-common: org.apache.maven.plugin.MojoExecutionException: 'protoc --version' did not return a version -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1]http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]  mvn -rf :hadoop-common
看到:http://blog.csdn.net/bamuta/article/details/13506893
要先裝protobuf,但是macports找不到這個安裝,貌似只能手動安裝
在https://code.google.com/p/protobuf/downloads/list下載protobuf-2.5.0.tar.gz
在/opt/local/var/macports/software/下建立protobuf文件夾
在這個文件夾下解壓縮protobuf-2.5.0.tar.gz
sudo tar -zxvfprotobuf-2.5.0.tar.gz
sudo sh configure
sudomake
sudo make check
sudo make install
protoc --version  //libprotoc 2.5.0
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib
bin的路徑:/usr/local/bin

再次 mvn3 package -Pdist,native -DskipTests -Dtar
新錯誤:
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1:33.033s
[INFO] Finished at: Fri Dec 13 22:49:22 CST 2013
[INFO] Final Memory: 45M/350M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.6:run (make) on project hadoop-common: An Ant BuildException has occured: exec returned: 2 -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1]http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]  mvn -rf :hadoop-common
根據http://blog.csdn.net/laozitianxia/article/details/8786922
安裝forrest
http://forrest.apache.org/mirrors.cgi
下載:apache-forrest-0.9-sources.tar.gzapache-forrest-0.9-dependencies.tar.gz
解壓apache-forrest-0.9-sources.tar.gz
然後將apache-forrest-0.9-dependencies.tar.gz解壓後,tools/下的文件整合進
/opt/local/var/macports/software/apache-forrest/apache-forrest-0.9/tools/文件夾下
修改forrest環境變量export FORREST_HOME=/opt/local/var/macports/software/apache-forrest/apache-forrest-0.9
export PATH=$PATH:$FORREST_HOME/bin
forrest -version
Apache Forrest.  Run 'forrest -projecthelp' to list options
Apache Ant version 1.7.1 compiled on September 26 2008  
再次 mvn3 package -Pdist,native -DskipTests -Dtar
新錯誤:
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1:56.366s
[INFO] Finished at: Sat Dec 14 00:59:51 CST 2013
[INFO] Final Memory: 58M/442M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.6:run (make) on project hadoop-common: An Ant BuildException has occured: exec returned: 2 -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1]http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]  mvn -rf :hadoop-common
根據http://blog.csdn.net/laozitianxia/article/details/8786922
安裝findbugs
在http://findbugs.sourceforge.net/downloads.html下載findbugs-2.0.3.tar.gz
解壓縮貌似直接能用,配置下環境變量
export FINDBUGS_HOME=/opt/local/var/macports/software/findbugs/findbugs-2.0.3
export PATH=$PATH:$FINDBUGS_HOME/bin
findbugs -version
2.0.3

安裝findbugs
在http://findbugs.sourceforge.net/downloads.html下載findbugs-2.0.3.tar.gz
解壓縮貌似直接能用,配置下環境變量
export FINDBUGS_HOME=/opt/local/var/macports/software/findbugs/findbugs-2.0.3
export PATH=$PATH:$FINDBUGS_HOME/bin
重啓終端
findbugs -version
2.0.3

安裝ANT
在http://ant.apache.org/bindownload.cgi下載apache-ant-1.9.2-bin.tar.gz
配置下環境變量
# Setting Path for Ant
export ANT_HOME=/opt/local/var/macports/software/apache-ant/apache-ant-1.9.2
export PATH=$PATH:$ANT_HOME/bin  
重啓終端
ant -version
Apache Ant(TM) version 1.9.2 compiled on July 8 2013
再次sudomvn3 package -Pdist,native -DskipTests -Dtar
新錯誤:
/bin/sh: /Users/JuneMAC/hadoop/release-2.2.0/hadoop-common-project/hadoop-common/${env.JAVA_HOME}/bin/javah: bad substitution
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 29.800s
[INFO] Finished at: Sat Dec 14 20:14:22 CST 2013
[INFO] Final Memory: 36M/96M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.codehaus.mojo:native-maven-plugin:1.0-alpha-7:javah (default) on project hadoop-common: Error running javah command: Error executing command line. Exit code:1 -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1]http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]  mvn -rf :hadoop-common
修改
hadoop-common-project/hadoop-common/pom.xml 文件中,將env.JAVA_HOME改爲java.home
再次sudomvn3 package -Pdist,native -DskipTests -Dtar
新錯誤:

[exec] /Users/JuneMAC/hadoop/release-2.2.0/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/security/JniBasedUnixGroupsNetgroupMapping.c:77:26: error: invalid operands to binary expression_r('void' and 'int')
[exec]  if(setnetgrent(cgroup) == 1) {
[exec]  ~~~~~~~~~~~~~~~~~~~ ^  ~
[exec] 1 error generated.
[exec] make[2]: *** [CMakeFiles/hadoop.dir/main/native/src/org/apache/hadoop/security/JniBasedUnixGroupsNetgroupMapping.c.o] Error 1
[exec] make[1]: *** [CMakeFiles/hadoop.dir/all] Error 2
[exec] make: *** [all] Error 2
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Hadoop Main ................................ SUCCESS [1.455s]
[INFO] Apache Hadoop Project POM ......................... SUCCESS [1.183s]
[INFO] Apache Hadoop Annotations ......................... SUCCESS [2.488s]
[INFO] Apache Hadoop Assemblies .......................... SUCCESS [0.400s]
[INFO] Apache Hadoop Project Dist POM .................... SUCCESS [2.396s]
[INFO] Apache Hadoop Maven Plugins ....................... SUCCESS [3.434s]
[INFO] Apache Hadoop Auth ................................ SUCCESS [2.611s]
[INFO] Apache Hadoop Auth Examples ....................... SUCCESS [2.349s]
[INFO] Apache Hadoop Common .............................. FAILURE [10.596s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 29.226s
[INFO] Finished at: Sat Dec 14 21:07:40 CST 2013
[INFO] Final Memory: 36M/94M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.6:run (make) on project hadoop-common: An Ant BuildException has occured: exec returned: 2 -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1]http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]  mvn -rf :hadoop-common
在hadoop羣裏求教之後採用如下方案:修改mvn3的配置文件:/opt/local/share/java/maven3/settings.xml
在…裏添加國內源:



nexus-osc
*
Nexusosc
http://maven.oschina.net/content/groups/public/



在...標籤中增加以下內容:

jdk-1.7

1.7



nexus
local private nexus
http://maven.oschina.net/content/groups/public/

true


false





nexus
local private nexus
http://maven.oschina.net/content/groups/public/

true


false






注意修改jdk version number

將剛纔的maven 配置文件拷貝到當前用戶的home目錄下:
settings.xml  copy 到  your_hadoop_usr_home/.m2/
cp settings.xml ~/.m2

最重要的一點,build your code是使用這個command line(Only for Mac OS):
mvn3 clean install -P-cbuild

編譯之前, 你在hadoop-2.2.0-src目錄(/Users/JuneMAC/hadoop/release-2.2.0)下執行
mvn3 clean install –DskipTests

上面的成功後,執行下面這個,生成安裝包
mvn3 clean install  package -Pdist -P-cbuild  -DskipTests  -Dtar

執行完成後,可以在/Users/JuneMAC/hadoop/release-2.2.0/hadoop-dist/target/
下找到
hadoop-2.2.0.tar.gz
將上面這個編譯好的源碼包解壓到:
/Users/JuneMAC/hadoop/
然後進行相關配置
解壓之後的源碼包和官網下載下來的源碼包相對比,沒有lib目錄
相關解釋:
“Here we use the additional options to stop compiling the native code. Here we use the additional options to stop compiling the native code.
this is the key reason why we need use -P-cbuild option”
上面這個是原因,好像不是很重要,這點先pass了,以後遇到問題再來補充說明吧






附:(官方說明)

Index: hadoop-common-project/hadoop-auth/pom.xml
===================================================================
--- hadoop-common-project/hadoop-auth/pom.xml	(revision 1543124)
+++ hadoop-common-project/hadoop-auth/pom.xml	(working copy)
@@ -54,6 +54,11 @@
     </dependency>
     <dependency>
       <groupId>org.mortbay.jetty</groupId>
+      <artifactId>jetty-util</artifactId>
+      <scope>test</scope>
+    </dependency>
+    <dependency>
+      <groupId>org.mortbay.jetty</groupId>
       <artifactId>jetty</artifactId>
       <scope>test</scope>
     </dependency>

詳細請查看:https://issues.apache.org/jira/browse/HADOOP-10110

發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章