環境需求
jdk 1.8
安裝資源
https://hadoop.apache.org/releases.html
步驟
1 環境準備
-
ssh免密登錄
ssh-keygen -t rsa # 一路回車 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys ssh localhost
-
ssh免密登錄失敗解決方法
chmod 700 ~/.ssh chmod 600 ~/.ssh/authorized_keys
-
如果還是失敗,用root用戶查看日誌,大部分是因爲文件或目錄權限問題
tail -20 /var/log/secure | grep "bad ownership"
2 安裝Hadoop
-
創建Hadoop程序文件夾
mkdir /DATA/hadoop
-
下載並解壓
wget https://downloads.apache.org/hadoop/common/hadoop-3.2.1/hadoop-3.2.1.tar.gz tar -zxvf hadoop-3.2.1.tar.gz
-
配置環境變量
vim ~/.bashrc
添加如下兩行export HADOOP_HOME=/DATA/hadoop/hadoop-3.2.1 export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH
使環境變量生效
source ~/.bashrc
-
Hadoop配置
vim $HADOOP_HOME/etc/hadoop/core-site.xml
<configuration> <property> <name>fs.defaultFS</name> <value>hdfs://localhost:9000</value> </property> </configuration>
vim $HADOOP_HOME/etc/hadoop/hdfs-site.xml
<configuration> <property> <name>dfs.replication</name> <value>1</value> </property> </configuration>
-
格式化HDFS(只需一次)
hdfs namenode -format
3 啓動Hadoop
-
啓動Hadoop
start-dfs.sh
-
查看NamNode:
http://localhost:9870 -
停止Hadoop
stop-dfs.sh
4 配置YARN
-
vim $HADOOP_HOME/etc/hadoop/mapred-site.xml
<configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> </configuration>
-
vim $HADOOP_HOME/etc/hadoop/yarn-site.xml
<configuration> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> </configuration>
5 啓動YARN
-
啓動YARN
start-yarn.sh
-
查看NamNode:
http://localhost:8088 -
停止Hadoop
stop-yarn.sh