1.2.3 大數據2.x 集羣安裝系列-- YARN配置啓動

1、三文件

yarn-env.sh
yarn-site.xml
mapred-site.xml --先用template生成

2 etc/hadoop/mapred-site.xml

:

cd /usr/local/hadoop/hadoop-3.2.1/etc/hadoop/

3 日誌

<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
<property>
        <name>mapreduce.jobhistory.address</name>
        <value>bigdata-pro01.kfk.com:10020</value>
    </property>
<property>
        <name>mapreduce.jobhistory.webapp.address</name>
        <value>bigdata-pro01.kfk.com:19888</value>
    </property>
</configuration>

4 etc/hadoop/yarn-site.xml:

主機 、日誌聚集 、日誌聚集存放多久

<configuration>
<!-- Site specific YARN configuration properties -->
   <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
   <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>bigdata-pro01.kfk.com</value>
    </property>
   <property>
        <name>yarn.log-aggregation-enable</name>
        <value>true</value>
    </property>
   <property>
        <name>yarn.log-aggregation.retain-seconds</name>
        <value>10000</value>
    </property>
</configuration>

5 同步配置文件到集羣

scp -r ./* [email protected]:/opt/modules/hadoop-2.5.0/etc/hadoop/
scp -r ./* [email protected]:/opt/modules/hadoop-2.5.0/etc/hadoop/

6 實驗準備

[kfk@bigdata-pro01 opt]$ ls
datas  modules  softwares  tools
[kfk@bigdata-pro01 opt]$ cd datas
[kfk@bigdata-pro01 datas]$ ls
[kfk@bigdata-pro01 datas]$ touch wc.iput
[kfk@bigdata-pro01 datas]$ vim wc.iput 
 
hadoop hive
hive spark
hbase java
java spark

7 啓動實驗

[kfk@bigdata-pro01 bin]$ ./hdfs dfs -put /opt/datas/wc.iput  /user/kfk/data/
 
[kfk@bigdata-pro01 hadoop-2.5.0]$ sbin/yarn-daemon.sh start resourcemanager
starting resourcemanager, logging to //opt/modules/hadoop-2.5.0/logs/yarn-kfk-resourcemanager-bigdata-pro01.kfk.com.out
[kfk@bigdata-pro01 hadoop-2.5.0]$ sbin/yarn-daemon.sh start nodemanager    
starting nodemanager, logging to //opt/modules/hadoop-2.5.0/logs/yarn-kfk-nodemanager-bigdata-pro01.kfk.com.out
[kfk@bigdata-pro01 hadoop-2.5.0]$ jps
3184 ResourceManager
3426 NodeManager
3530 Jps
2525 DataNode
2447 NameNode

8 另兩臺查看

[kfk@bigdata-pro02 hadoop-2.5.0]$ sbin/yarn-daemon.sh start nodemanager
starting nodemanager, logging to /opt/modules/hadoop-2.5.0/logs/yarn-kfk-nodemanager-bigdata-pro02.kfk.com.out
[kfk@bigdata-pro02 hadoop-2.5.0]$ jps
2291 DataNode
2725 Jps
2619 NodeManager
 
[kfk@bigdata-pro03 hadoop-2.5.0]$ sbin/yarn-daemon.sh start nodemanager
starting nodemanager, logging to /opt/modules/hadoop-2.5.0/logs/yarn-kfk-nodemanager-bigdata-pro03.kfk.com.out
[kfk@bigdata-pro03 hadoop-2.5.0]$ jps
2610 NodeManager
2310 DataNode
2717 Jps

9 WINDOWS訪問產看節點

http://bigdata-pro01.kfk.com:8088/cluster

10 問題解決

只有兩臺起來了 還一臺報不健康 查看日誌發現 警告和錯誤

nodemanager. DirectoryCollection: Unable to create directory /modules/hadoop-2.5.0/logs/userlogs error mkdir of

1/1 log-dirs turned bad: /modules/hadoop-2.5.0/logs/userlogs

1、解決思路授權權限未果  
2、手工創建文件夾未果  
3、重新停服務 格式化 bin/hdfs namenode -format   
這裏需要注意的點是需要把存放dfs文件全部刪了 
默認在/tmp/目錄下的 hadoop yarn namenode datanode 等和hadoop相關的通通刪了 。
4、去其他從服務器也通通刪了 
5、啓動HDFS後 （主namenode datanode 從 datanode ）
6、再重試resourcemanager  nodemanager 命令 終於成功  目錄和文件都要重建重傳了

11 重新啓動

[kfk@bigdata-pro01 hadoop-2.5.0]$ sbin/hadoop-daemon.sh start namenode
[kfk@bigdata-pro01 hadoop-2.5.0]$ sbin/hadoop-daemon.sh start datanode
[kfk@bigdata-pro01 hadoop-2.5.0]$ sbin/yarn-daemon.sh start resourcemanager
[kfk@bigdata-pro01 hadoop-2.5.0]$ sbin/yarn-daemon.sh start nodemanager
[kfk@bigdata-pro01 hadoop-2.5.0]$ jps
7426 NameNode
7523 DataNode
7988 Jps
7879 NodeManager
7631 ResourceManager
[kfk@bigdata-pro01 hadoop-2.5.0]$ bin/hdfs dfs -mkdir -p /user/kfk/data/
[kfk@bigdata-pro01 hadoop-2.5.0]$ bin/hdfs dfs -put /opt/datas/wc.iput  /user/kfk/data/

11 啓動日誌

[kfk@bigdata-pro01 hadoop-2.5.0]$ sbin/mr-jobhistory-daemon.sh start historyserver
starting historyserver, logging to /opt/modules/hadoop-2.5.0/logs/mapred-kfk-historyserver-bigdata-pro01.kfk.com.out
[kfk@bigdata-pro01 hadoop-2.5.0]$ jps
7426 NameNode
7523 DataNode
8294 Jps
7879 NodeManager
8254 JobHistoryServer
7631 ResourceManager