IDEA配置hadoop開發環境及操作HDFS演示

hadoop基礎環境搭建請看上篇(非高可用)

 

IDEA Maven配置

1. 下載maven

下載地址
從官網上，下載一個壓縮包，然後解壓到任意的文件夾
Maven的安裝必須需要jdk1.7+

2. 環境變量設置

M2_HOME改爲具體的路徑，其他的就直接複製就好 

M2_HOME=D:\maven\apache-maven-3.6.3-bin\apache-maven-3.6.3
M2=%M2_HOME%\bin
MAVEN_OPTS=-Xms256m -Xmx512m

Path目錄下添加以下

%M2%

打開cmd，輸入mvn --version，出現以下類似以下結果，則配置成功

Apache Maven 3.6.1 (d66c9c0b3152b2e69ee9bac180bb8fcc8e6af555; 2019-04-05T03:00:29+08:00)
Maven home: Q:\maven\apache-maven-3.6.1\bin\..
Java version: 1.8.0_131, vendor: Oracle Corporation, runtime: C:\Program Files\Java\jdk1.8.0_131\jre
Default locale: zh_CN, platform encoding: GBK
OS name: "windows 10", version: "10.0", arch: "amd64", family: "windows"

3. 配置鏡像和下載jar目錄

打開maven目錄下的conf/settings.xml
找到mirrors標籤，在其下面複製下面的代碼

<mirror>
      <id>alimaven</id>
      <name>aliyun maven</name>
      <url>http://maven.aliyun.com/nexus/content/groups/public/</url>
      <mirrorOf>central</mirrorOf>
    </mirror>
    
    <mirror>
      <id>repo2</id>
      <mirrorOf>central</mirrorOf>
      <name>Human Readable Name for this Mirror.</name>
      <url>http://repo2.maven.org/maven2/</url>
    </mirror>

    <mirror>
      <id>ui</id>
      <mirrorOf>central</mirrorOf>
      <name>Human Readable Name for this Mirror.</name>
      <url>http://uk.maven.org/maven2/</url>
</mirror>

找到localRepository標籤，在裏面填寫jar包下載的目錄，不寫的話默認就會下載到c盤

<localRepository>Q:\maven\maven_jar</localRepository>

4. IDEA設置Maven

修改默認設置，只對於新建的項目，設置纔會有效

爲當前項目添加Maven

在pom.xml文件裏添加hadoop的依賴包hadoop-common, hadoop-client, hadoop-hdfs

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
	<modelVersion>4.0.0</modelVersion>
	<groupId>com.bigdata.master</groupId>
	<artifactId>hadoop</artifactId>
	<version>0.0.1-SNAPSHOT</version>
 
	<dependencies>
		<!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-common -->
		<dependency>
			<groupId>org.apache.hadoop</groupId>
			<artifactId>hadoop-common</artifactId>
			<version>2.7.3</version>
		</dependency>
 
		<!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-hdfs -->
		<dependency>
			<groupId>org.apache.hadoop</groupId>
			<artifactId>hadoop-hdfs</artifactId>
			<version>2.7.3</version>
		</dependency>
 
		<!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-client -->
		<dependency>
			<groupId>org.apache.hadoop</groupId>
			<artifactId>hadoop-client</artifactId>
			<version>2.7.3</version>
		</dependency>
 
		<!-- https://mvnrepository.com/artifact/junit/junit -->
		<dependency>
			<groupId>junit</groupId>
			<artifactId>junit</artifactId>
			<version>4.12</version>
		</dependency>
	</dependencies>
</project>

在出現的選項中選擇maven,選擇右邊的maven選項，點擊install，idea就會自動下載一些jar

5、測試

代碼：

package hdfs.demo;
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.OutputStream;


import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.FSDataOutputStream;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IOUtils;

public class main {

    public static void main(String[] args) throws IOException{
        //查看
        catFile();
        //創建
        //mkdir();
    }
    public static void catFile() throws IOException {
        //獲取配置文件對象
        Configuration conf = new Configuration();
        //添加配置信息
        conf.set("fs.defaultFS","hdfs://master:9000");
        //獲取文件系統對象
        FileSystem fs = FileSystem.get(conf);
        //操作文件系統
        FSDataInputStream fis = fs.open(new Path("/tmp/hello"));
        //讀取文件輸出到控制檯
        IOUtils.copyBytes(fis, System.out, conf, true);
        //關閉文件系統
        if(fs != null) {
            fs.close();
        }
    }

    public static void mkdir() throws IOException {
        Configuration conf = new Configuration();
        conf.set("fs.defaultFS","hdfs://master:9000");
        FileSystem fs = FileSystem.get(conf);

        Path outputDir=new Path("/tmp/first");
        if(!fs.exists(outputDir)){//判斷如果不存在就創建
            fs.mkdirs(new Path("/tmp/first"));
            System.out.println("文件創建成功！");
        }else {
            System.out.println("文件路徑已經存在！");
        }
        fs.close();
    }
}

結果：