Hadoop(3.2.0) 僞分佈式wordcount

實驗環境

• Hadoop 3.2.0
• 虛擬機 RHEL7.0

Hadoop(3.2.0) 僞分佈式wordcount

hdfs dfs -mkdir -p /user/yourUserName  #如報此文件夾不存在，需執行此句遞歸創建
hdfs dfs -mkdir input1
hdfs dfs -copyFromLocal ./input/* input1 
hdfs dfs -ls input1
hdfs dfs -rmr  output     #遞歸刪除(如果有)

bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.0.jar grep input1 output 'dfs[a-z.]+'

hdfs dfs -cat output/part*

免密登陸

ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
chmod  0600  ~/.ssh/authorized_keys

ssh權限問題

[admin@iz2ze22ghnnsejh781lqrqz ~]$  sudo chmod 600 ~/.ssh/id_rsa 
[admin@iz2ze22ghnnsejh781lqrqz ~]$ sudo chmod 600 ~/.ssh/id_rsa.pub
[admin@iz2ze22ghnnsejh781lqrqz ~]$ sudo chmod 755 ~/.ssh
[admin@iz2ze22ghnnsejh781lqrqz ~]$ 
[admin@iz2ze22ghnnsejh781lqrqz ~]$ start-dfs.sh
Starting namenodes on [localhost]
Starting datanodes
Starting secondary namenodes [iz2ze22ghnnsejh781lqrqz]
[admin@iz2ze22ghnnsejh781lqrqz ~]$ jps
11649 SecondaryNameNode
11764 Jps
11478 DataNode
[admin@iz2ze22ghnnsejh781lqrqz ~]$ 

input1文件夾

Found 9 items
-rw-r--r--   1 hadoop supergroup       8260 2019-09-16 15:53 input1/capacity-scheduler.xml
-rw-r--r--   1 hadoop supergroup       1110 2019-09-16 15:53 input1/core-site.xml
-rw-r--r--   1 hadoop supergroup      11392 2019-09-16 15:53 input1/hadoop-policy.xml
-rw-r--r--   1 hadoop supergroup       1293 2019-09-16 15:53 input1/hdfs-site.xml
-rw-r--r--   1 hadoop supergroup        620 2019-09-16 15:53 input1/httpfs-site.xml
-rw-r--r--   1 hadoop supergroup       3518 2019-09-16 15:53 input1/kms-acls.xml
-rw-r--r--   1 hadoop supergroup        682 2019-09-16 15:53 input1/kms-site.xml
-rw-r--r--   1 hadoop supergroup        758 2019-09-16 15:53 input1/mapred-site.xml
-rw-r--r--   1 hadoop supergroup        690 2019-09-16 15:53 input1/yarn-site.xml

詞頻統計結果

1	dfsadmin
1	dfs.replication
1	dfs.name.dir
1	dfs.data.dir

hdfs-site.xml

<configuration>  
<property>  
    <name>dfs.name.dir</name>  
        <value>/usr/hadoop/hdfs/name</value>  
	    <description>namenode上存儲hdfs名字空間元數據 </description>   
	    </property>  

	    <property>  
	        <name>dfs.data.dir</name>  
		    <value>/usr/hadoop/hdfs/data</value>  
		        <description>datanode上數據塊的物理存儲位置</description>  
			</property>  

			<!-- 設置hdfs副本數量 -->  
			<property>  
			 <name>dfs.replication</name>
			  <value>1</value>  
			</property>  
</configuration>

core-site.xml

  <configuration>  
  <!-- 指定HDFS namenode 的通信地址 -->  
<property>  
    <name>fs.defaultFS</name>  
     <value>hdfs://localhost:9000</value>  
	</property>  
	<!-- 指定hadoop運行時產生文件的存儲路徑 -->  
	<property>  
	    <name>hadoop.tmp.dir</name>  
	     <value>/usr/local/hadoop/hadoop-3.2.0/tmp</value>  
	</property>  
</configuration>