Hadoop步驟 openSUSE 42

1. ssh 無密碼登錄

cd ~/.ssh

ssh-keygen -t rsa  

cat id_rsa.pub >> authorized_keys

本地測試：ssh localhost （第一次需要輸入密碼，以後不用輸入密碼）

異機測試：將本機id_rsa.pub（公鑰）放到異機的authorized_keys即可

2.jdk 配置

vi  /etc/profile

export JAVA_HOME=/usr/java/jdk1.7.0_79
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export PATH=$JAVA_HOME/bin:$PATH

source /etc/profile (注意使用的用戶，source 使本用戶環境有效~~)

3.安裝Hadoop

1. sudo tar-zxf ~/下載/hadoop-2.6.0.tar.gz -C /usr/local# 解壓到/usr/local中
2. cd/usr/local/
3. sudo mv./hadoop-2.6.0/ ./hadoop# 將文件夾名改爲hadoop
4. sudo chown-R hadoop ./hadoop# 修改文件權限

安裝完成

測試：

1. cd/usr/local/hadoop
2. ./bin/hadoop version

1. 
   

4. 以下是Hadoop的配置（僞分佈式）

./etc/hadoop/core-site.xml 絕對路徑是/usr/local/hadoop/etc/hadoop/core-site.xml 不要搞錯嘞~~

修改配置文件 core-site.xml 

1. <configuration>
2. <property>
3. <name>hadoop.tmp.dir</name>
4. <value>file:/usr/local/hadoop/tmp</value>
5. <description>Abase for other temporary directories.</description>
6. </property>
7. <property>
8. <name>fs.defaultFS</name>
9. <value>hdfs://localhost:9000</value>
10. </property>
11. </configuration>

修改配置文件 hdfs-site.xml

1. <configuration>
2. <property>
3. <name>dfs.replication</name>
4. <value>1</value>
5. </property>
6. <property>
7. <name>dfs.namenode.name.dir</name>
8. <value>file:/usr/local/hadoop/tmp/dfs/name</value>
9. </property>
10. <property>
11. <name>dfs.datanode.data.dir</name>
12. <value>file:/usr/local/hadoop/tmp/dfs/data</value>
13. </property>
14. </configuration>

注意：如果沒有的文件夾，需要新建，比如/usr/local/hadoop/tmp 、 dfs  、name 、data

配置完成後，執行 NameNode 的格式化:

1. ./bin/hdfs namenode -format
2. 
   

   Q1: 

Hadoop格式化HDFS報警告 java.net.UnknownHostException: bogon: bogon

   解決方案：重啓虛擬機，bogon是DNS逆解析失敗導致的bogon，重啓後執行 namenode 格式化命令，如果還不能解決；

   則 vi /etc/hostname  改成slave02 ;vi /etc/hosts 添加127.0.0.1       slave02 重啓虛擬機~（Slave02 是自己取得名字）

關鍵信息如下：

17/01/05 14:33:41 INFO namenode.FSImage: Allocated new BlockPoolId: BP-627974405-127.0.0.1-1483598021204
17/01/05 14:33:41 INFO common.Storage: Storage directory /usr/local/hadoop/tmp/dfs/name has been successfully formatted.
17/01/05 14:33:41 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
17/01/05 14:33:41 INFO util.ExitUtil: Exiting with status 0
17/01/05 14:33:41 INFO namenode.NameNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at slave02/127.0.0.1
************************************************************/

成功了吧~~

格式化完成，下一步啓動Hadoop：

開啓 NameNode 和 DataNode 守護進程（啓動前要保證ssh服務是開着滴，命令service sshd start）

1. ./sbin/start-dfs.sh

啓動成功：

hadoop@slave02:/usr/local/hadoop> jps
2641 DataNode
2838 SecondaryNameNode
2536 NameNode
5853 Jps
hadoop@slave02:/usr/local/hadoop> 

果然成功~

5.以下是Hadoop配置（分佈式集羣）

一主兩從，即一個master兩個slave

1) 配置文件：/etc/hadoop/core-site.xml

<configuration>
        <property>
             <name>hadoop.tmp.dir</name>
             <value>file:/usr/local/hadoop/tmp</value>
             <description>Abase for other temporary directories.</description>
        </property>
        <property>
             <name>fs.defaultFS</name>
             <value>hdfs://master:9000</value>
        </property>
</configuration>

2) 配置文件：/etc/hadoop/mapred-site.xml     

<configuration>
<property>
<name>mapred.job.tracker</name>
<value>master:9001</value>
</property>

</configuration>

3) 配置文件：/etc/hadoop/hdfs-site.xml

<configuration>
<property>
             <name>dfs.replication</name>
             <value>1</value>
        </property>
        <property>
             <name>dfs.namenode.name.dir</name>
             <value>file:/usr/local/hadoop/tmp/dfs/name</value>
        </property>
        <property>
             <name>dfs.datanode.data.dir</name>
             <value>file:/usr/local/hadoop/tmp/dfs/data</value>
        </property>
</configuration>

4) 配置masters和slaves主從結點

etc/hadoop/ 目錄下沒有masters文件夾，不需要配置，只配置Slaves節點

$vim slaves：

輸入：

slave01

slave02

5) 配置/etc/hosts

需要root權限喲~

->#vi /etc/hosts

#127.0.0.1      localhost
192.168.126.137 master
192.168.126.138 slave01
192.168.126.136 slave02 

實際IP自己更改~

6) 複製虛擬機，複製2份，一共三個虛擬機，master slave01 slave02

更改三臺機器的主機名：vi /etc/hostname 分別爲master slave01 slave02

重新設置ssh 免密碼登錄，使master 能免密碼登錄master slave01 slave02（如果能登錄，就不用重配了~~）

驗證完成master能登錄master slave01 slave02

7）啓動hadoop，

hadoop@master:/usr/local/hadoop> ./sbin/start-dfs.sh 
Starting namenodes on [master]
master: starting namenode, logging to /usr/local/hadoop/logs/hadoop-hadoop-namenode-master.out
slave01: starting datanode, logging to /usr/local/hadoop/logs/hadoop-hadoop-datanode-slave01.out
slave02: starting datanode, logging to /usr/local/hadoop/logs/hadoop-hadoop-datanode-slave02.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /usr/local/hadoop/logs/hadoop-hadoop-secondarynamenode-master.out

jps一下看看：

hadoop@master:/usr/local/hadoop> jps
3419 Jps
3116 NameNode
3311 SecondaryNameNode

去slave01 jps 看看：

hadoop@slave01:~> jps
3278 Jps
3185 DataNode

去slave02 jps 看看：

hadoop@slave02:~> jps
2355 DataNode
2457 Jps

啓動成功了~~好消息。

打開瀏覽器，去master：50070看看吧~

啓動管理進程噻~

hadoop@master:/usr/local/hadoop> ./sbin/start-yarn.sh 
starting yarn daemons
starting resourcemanager, logging to /usr/local/hadoop/logs/yarn-hadoop-resourcemanager-master.out
slave01: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-hadoop-nodemanager-slave01.out
slave02: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-hadoop-nodemanager-slave02.out
hadoop@master:/usr/local/hadoop> jps
3116 NameNode
3785 ResourceManager
3311 SecondaryNameNode
3920 Jps