[Hadoop培訓筆記]03-MapReduce集羣的安裝與部署

注：開源力量Hadoop Development網絡培訓，鏈接：http://new.osforce.cn/course/52  個人筆記，不具參考性。

說明：
1）實驗採用的是hadoop1.x，hadoop2.x的代碼做一個分析
2）學習hadoop的基礎知識；java基礎；Linux操作基礎

MapRedcue集羣環境搭建
MapReduce WordCount運行
Hadoop Eclipse插件

bin目錄下執行：./start-mapred.sh，啓動jobtracker和tasktracker
問題：啓動後使用jps命令，看不到上述tracker在運行。
原因：通過logs目錄裏的hadoop-michaelchen-tasktracker-mars.clustertech.com.log，查看後發現這麼一條：Can not start task tracker because java.lang.IllegalArgumentException: Does not contain a valid host:port authority: local
解決方法：沒有配置 mapred-site.xml，配置如下。再次啓動，使用jps可以看到jobtracker和tasktracker

<property> 
<name>mapred.job.tracker</name> 
<value>localhost:9101</value> 
</property>

bin目錄下創建 vim a.txt，裏面寫一些單詞作爲輸入文件。

  1 hello world
  2 hello java
  3 java c++

然後將a.txt放入一個創建的input目錄裏，讓hadoop example中的wordcount程序執行，從input目錄裏獲取數據，結果放到output目錄裏

./hadoop fs -mkdir /input
./hadoop fs -put a.txt /input
./hadoop jar ../hadoop-examples-1.2.1.jar wordcount /input /output

執行的過程比較有趣，所以也貼到這裏

13/12/04 12:23:07 INFO input.FileInputFormat: Total input paths to process : 1
13/12/04 12:23:07 INFO util.NativeCodeLoader: Loaded the native-hadoop library
13/12/04 12:23:07 WARN snappy.LoadSnappy: Snappy native library not loaded
13/12/04 12:23:08 INFO mapred.JobClient: Running job: job_201312041206_0001
13/12/04 12:23:09 INFO mapred.JobClient:  map 0% reduce 0%
13/12/04 12:23:22 INFO mapred.JobClient:  map 100% reduce 0%
13/12/04 12:23:34 INFO mapred.JobClient:  map 100% reduce 100%
13/12/04 12:23:38 INFO mapred.JobClient: Job complete: job_201312041206_0001
13/12/04 12:23:38 INFO mapred.JobClient: Counters: 29
13/12/04 12:23:38 INFO mapred.JobClient:   Job Counters
13/12/04 12:23:38 INFO mapred.JobClient:     Launched reduce tasks=1
13/12/04 12:23:38 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=14199
13/12/04 12:23:38 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
13/12/04 12:23:38 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
13/12/04 12:23:38 INFO mapred.JobClient:     Launched map tasks=1
13/12/04 12:23:38 INFO mapred.JobClient:     Data-local map tasks=1
13/12/04 12:23:38 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=12168
13/12/04 12:23:38 INFO mapred.JobClient:   File Output Format Counters
13/12/04 12:23:38 INFO mapred.JobClient:     Bytes Written=29
13/12/04 12:23:38 INFO mapred.JobClient:   FileSystemCounters
13/12/04 12:23:38 INFO mapred.JobClient:     FILE_BYTES_READ=51
13/12/04 12:23:38 INFO mapred.JobClient:     HDFS_BYTES_READ=134
13/12/04 12:23:38 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=117313
13/12/04 12:23:38 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=29
13/12/04 12:23:38 INFO mapred.JobClient:   File Input Format Counters
13/12/04 12:23:38 INFO mapred.JobClient:     Bytes Read=32
13/12/04 12:23:38 INFO mapred.JobClient:   Map-Reduce Framework
13/12/04 12:23:38 INFO mapred.JobClient:     Map output materialized bytes=51
13/12/04 12:23:38 INFO mapred.JobClient:     Map input records=3
13/12/04 12:23:38 INFO mapred.JobClient:     Reduce shuffle bytes=51
13/12/04 12:23:38 INFO mapred.JobClient:     Spilled Records=8
13/12/04 12:23:38 INFO mapred.JobClient:     Map output bytes=56
13/12/04 12:23:38 INFO mapred.JobClient:     Total committed heap usage (bytes)=181141504
13/12/04 12:23:38 INFO mapred.JobClient:     CPU time spent (ms)=5400
13/12/04 12:23:38 INFO mapred.JobClient:     Combine input records=6
13/12/04 12:23:38 INFO mapred.JobClient:     SPLIT_RAW_BYTES=102
13/12/04 12:23:38 INFO mapred.JobClient:     Reduce input records=4
13/12/04 12:23:38 INFO mapred.JobClient:     Reduce input groups=4
13/12/04 12:23:38 INFO mapred.JobClient:     Combine output records=4
13/12/04 12:23:38 INFO mapred.JobClient:     Physical memory (bytes) snapshot=175915008
13/12/04 12:23:38 INFO mapred.JobClient:     Reduce output records=4
13/12/04 12:23:38 INFO mapred.JobClient:     Virtual memory (bytes) snapshot=787206144
13/12/04 12:23:38 INFO mapred.JobClient:     Map output records=6

看看創建了什麼文件，使用命令 ./hadoop fs -lsr /

drwxr-xr-x   - michaelchen supergroup          0 2013-12-04 12:20 /input
-rw-r--r--   1 michaelchen supergroup         32 2013-12-04 12:20 /input/a.txt
drwxr-xr-x   - michaelchen supergroup          0 2013-12-04 12:23 /output
-rw-r--r--   1 michaelchen supergroup          0 2013-12-04 12:23 /output/_SUCCESS
drwxr-xr-x   - michaelchen supergroup          0 2013-12-04 12:23 /output/_logs
drwxr-xr-x   - michaelchen supergroup          0 2013-12-04 12:23 /output/_logs/history
-rw-r--r--   1 michaelchen supergroup      13815 2013-12-04 12:23 /output/_logs/history/job_201312041206_0001_1386130987951_michaelchen_word+count
-rw-r--r--   1 michaelchen supergroup      49533 2013-12-04 12:23 /output/_logs/history/job_201312041206_0001_conf.xml
-rw-r--r--   1 michaelchen supergroup         29 2013-12-04 12:23 /output/part-r-00000
drwxr-xr-x   - michaelchen supergroup          0 2013-12-04 10:26 /system
drwxr-xr-x   - michaelchen supergroup          0 2013-12-04 12:17 /tmp
drwxr-xr-x   - michaelchen supergroup          0 2013-12-04 12:17 /tmp/hadoop-michaelchen
drwxr-xr-x   - michaelchen supergroup          0 2013-12-04 12:23 /tmp/hadoop-michaelchen/mapred
drwxr-xr-x   - michaelchen supergroup          0 2013-12-04 12:23 /tmp/hadoop-michaelchen/mapred/staging
drwxr-xr-x   - michaelchen supergroup          0 2013-12-04 12:23 /tmp/hadoop-michaelchen/mapred/staging/michaelchen
drwx------   - michaelchen supergroup          0 2013-12-04 12:23 /tmp/hadoop-michaelchen/mapred/staging/michaelchen/.staging
drwx------   - michaelchen supergroup          0 2013-12-04 12:23 /tmp/hadoop-michaelchen/mapred/system
-rw-------   1 michaelchen supergroup          4 2013-12-04 12:17 /tmp/hadoop-michaelchen/mapred/system/jobtracker.info
drwxr-xr-x   - michaelchen supergroup          0 2013-12-04 10:33 /user
drwxr-xr-x   - michaelchen supergroup          0 2013-12-04 10:33 /user/michaelchen
drwxr-xr-x   - michaelchen supergroup          0 2013-12-04 10:33 /user/michaelchen/archiveDir
drwxr-xr-x   - michaelchen supergroup          0 2013-12-04 10:33 /user/michaelchen/archiveDir/pack.har
-rw-r--r--   1 michaelchen supergroup          0 2013-12-04 10:33 /user/michaelchen/archiveDir/pack.har/_SUCCESS
-rw-r--r--   5 michaelchen supergroup         72 2013-12-04 10:33 /user/michaelchen/archiveDir/pack.har/_index
-rw-r--r--   5 michaelchen supergroup         22 2013-12-04 10:33 /user/michaelchen/archiveDir/pack.har/_masterindex
-rw-r--r--   1 michaelchen supergroup      15147 2013-12-04 10:33 /user/michaelchen/archiveDir/pack.har/part-0
drwxr-xr-x   - michaelchen supergroup          0 2013-12-04 10:31 /xwchen
-rw-r--r--   1 michaelchen supergroup      15147 2013-12-04 10:31 /xwchen/hadoop

分析後的結果放在這裏，使用命令查看：./hadoop fs -cat /output/part-r-00000

c++     1
hello   2
java    2
world   1

Hadoop Eclipse插件

1）hadoop自0.20.x版本後不再提供現成的hadoop-eclipse插件，而是給出了源碼自行編譯
2）編譯採用的是ant+ivy
3）URL：http://wiki.apache.org/hadoop/EclipsePlugin

編譯的整個過程很是容易出問題，建議參考此鏈接：http://www.srccodes.com/p/article/30/build-hadoop-eclipse-plugin-jar-from-source-code-and-install-that-plugin-in-eclipse-ide

鏈接中注意版本號可能會有更改。

不過我編譯的hadoop plugin的插件還是在eclipse中無法使用，最後用的網上下載的。。。
無法使用是出現在eclipse菜單裏，show-others裏編輯mapreduce路徑，點new，彈不出窗口，淚奔啊～～

WordCount裏用的相關Jar包依賴如下：

輸入參數：hdfs://192.168.56.101:9100/input /output

注意output之前不能存在，否則報錯。

其他參考視頻。在1.2.1中編譯成功，運行成功。

試題中的一些筆記：

• hadoop對於小文件的處理方式：Archive、CombinedInputFormat、SequenceFile、MapFile
• Apache Mesos is a cluster manager that provides efficient resource isolation and sharing across distributed applications, or frameworks.
• Borg（來自Google），YARN（來自Apache，屬於Hadoop下面的一個分支，開源），Mesos（來自Twitter，開源），Torca（來自騰訊搜搜），Corona（來自Facebook，開源）一類系統被稱爲資源統一管理系統或者資源統一調度系統，它們是大數據時代的必然產物。
• SecondaryNamenode的作用是合併fsimage和editlogs
• Apache Tez由Hortonworks開發
• 關係型數據庫與HDFS之間相互數據遷移的工具是sqoop
• hadoop2.x採用maven構建源代碼