第一個MapReduce程序開發

第一個MapReduce程序開發

前言

上篇博文已經搭建完成了Hadoop的開發環境，後面我們就需要專注於MapReduce的開發了。本文介紹如何利用MapReduce進行單詞個數統計的代碼實現，完整介紹一個Job作業的開發流程。

一、Job作業體系結構

一次Job作業包括5個階段，其中只有Map階段和Reduce階段是需要我們去編寫邏輯代碼的，其它階段都是自動完成。

二、單詞統計（WordCount）例子分析

二、單詞統計（WordCount）程序開發

1、將數據上傳至HDFS

aa.log的數據如下

2、創建Maven項目

File->Project Structure->Modules添加hadoop安裝包下share/hadoop/common；share/hadoop/dfdf；share/hadoop/mapreduce；share/hadoop/yarn目錄下的Jar包

添加hadoop安裝包下share/hadoop/common/lib下的Jar包

3、編寫Job工作代碼

package com.sun.wordcount;

import org.apache.hadoop.conf.Configured;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;

import java.io.IOException;

//測試數據
/*
chenyn xiaohei xiaowang chenyn
zhaoliu wangwu zhangsan xiaoming xiaochen
chenyn chenyn xiaozhang xiaohei
xiaoliu xiaozi xiaosun xiaochen
*/

public class WordCountJob extends Configured implements Tool {

    //生成這個方法的快捷鍵（psvm）
    public static void main(String[] args) throws Exception {
        //執行Job作業的對象是誰
        ToolRunner.run(new WordCountJob(),args);
    }

    //查找待實現方法快捷鍵（Ctrl+i）
    //執行Job作業
    public int run(String[] strings) throws Exception {
        //創建Job作業
        Job job = Job.getInstance(getConf());
        job.setJarByClass(WordCountJob.class);
        //1、設置inputFormat
        job.setInputFormatClass(TextInputFormat.class);
        TextInputFormat.addInputPath(job,new Path("/wordcount/aa.log"));
        //2、設置map
        job.setMapperClass(WordCountMap.class);
        job.setMapOutputKeyClass(Text.class);
        job.setMapOutputValueClass(IntWritable.class);
        //3、設置shuffle 自動處理
        //4、設置reduce
        job.setReducerClass(WordCountReduce.class);
        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(IntWritable.class);
        //5、設置output format
        job.setOutputFormatClass(TextOutputFormat.class);
        TextOutputFormat.setOutputPath(job,new Path("/wordcount/result"));//必須保證output farmat輸出結果的目錄不存在(這個機制是爲了防止丟失你的數據)
        //6、提交Job作業
        //job.submit();//這種方式沒有返回狀態
        boolean status = job.waitForCompletion(true);//這種方式可以返回執行狀態
        System.out.println("word count status = " + status);//生成快捷鍵（soutv）
        return 0;
    }

    //map階段 (部分計算)
    // hadoop包裝了基本類型
    // int->intWritable Long->LongWritable
    // Double->DoubleWritable
    // Float->FloatWritable String->Text

    //泛型1:keyin inputFormat中的輸出key類型  泛型2:valuein inputFormat中的輸出value類型
    //泛型3:keyout map階段中的輸出key類型  泛型2:valueout map階段中的輸出value類型
    public static class WordCountMap extends Mapper<LongWritable, Text,Text,IntWritable>{
        //input format 輸出一次，調用一次map方法；
        // 參數key是本次input format輸出這行數據的行首偏移量
        // 參數value是當前input format輸出的這行值
        @Override //打開重寫方法的快捷鍵（Ctrl+o）
        protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
            //多讀取的行數據進行切分
            String[] words = value.toString().split(" ");
            for (String s : words) {
                context.write(new Text(s),new IntWritable(1));
            }
        }
    }

    //reduce階段(彙總計算)
    public static class WordCountReduce extends Reducer<Text, IntWritable,Text,IntWritable>
    {
        //所有map執行完，執行Reduce階段
        @Override
        protected void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException {
            int sum=0;
            for (IntWritable value : values) {
                sum+=value.get();
            }
            //輸出結果
            context.write(key,new IntWritable(sum));
        }
    }


}

4、Pacakge打包Job工作代碼

生成Jar包

5、將Pacakge打包的Jar包放到lunix系統中執行

yarn jar hadoop-wordcount-1.0-SNAPSHOT.jar com.sun.wordcount.WordCountJob

shell執行過程

[root@hadoop4 code]# yarn jar hadoop-wordcount-1.0-SNAPSHOT.jar com.sun.wordcount.WordCountJob
20/06/26 11:03:58 INFO client.RMProxy: Connecting to ResourceManager at hadoop4/192.168.23.134:8032
20/06/26 11:04:05 INFO input.FileInputFormat: Total input files to process : 1
20/06/26 11:04:06 INFO mapreduce.JobSubmitter: number of splits:1
20/06/26 11:04:06 INFO Configuration.deprecation: yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead, use yarn.system-metrics-publisher.enabled
20/06/26 11:04:07 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1593138102465_0001
20/06/26 11:04:08 INFO impl.YarnClientImpl: Submitted application application_1593138102465_0001
20/06/26 11:04:08 INFO mapreduce.Job: The url to track the job: http://hadoop4:8088/proxy/application_1593138102465_0001/
20/06/26 11:04:08 INFO mapreduce.Job: Running job: job_1593138102465_0001
20/06/26 11:04:42 INFO mapreduce.Job: Job job_1593138102465_0001 running in uber mode : false
20/06/26 11:04:42 INFO mapreduce.Job:  map 0% reduce 0%
20/06/26 11:05:08 INFO mapreduce.Job:  map 100% reduce 0%
20/06/26 11:05:40 INFO mapreduce.Job:  map 100% reduce 100%
20/06/26 11:05:42 INFO mapreduce.Job: Job job_1593138102465_0001 completed successfully
20/06/26 11:05:42 INFO mapreduce.Job: Counters: 49
	File System Counters
		FILE: Number of bytes read=245
		FILE: Number of bytes written=398523
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
		HDFS: Number of bytes read=241
		HDFS: Number of bytes written=123
		HDFS: Number of read operations=6
		HDFS: Number of large read operations=0
		HDFS: Number of write operations=2
	Job Counters 
		Launched map tasks=1
		Launched reduce tasks=1
		Data-local map tasks=1
		Total time spent by all maps in occupied slots (ms)=22254
		Total time spent by all reduces in occupied slots (ms)=21557
		Total time spent by all map tasks (ms)=22254
		Total time spent by all reduce tasks (ms)=21557
		Total vcore-milliseconds taken by all map tasks=22254
		Total vcore-milliseconds taken by all reduce tasks=21557
		Total megabyte-milliseconds taken by all map tasks=22788096
		Total megabyte-milliseconds taken by all reduce tasks=22074368
	Map-Reduce Framework
		Map input records=4
		Map output records=17
		Map output bytes=205
		Map output materialized bytes=245
		Input split bytes=101
		Combine input records=0
		Combine output records=0
		Reduce input groups=12
		Reduce shuffle bytes=245
		Reduce input records=17
		Reduce output records=12
		Spilled Records=34
		Shuffled Maps =1
		Failed Shuffles=0
		Merged Map outputs=1
		GC time elapsed (ms)=425
		CPU time spent (ms)=2330
		Physical memory (bytes) snapshot=314445824
		Virtual memory (bytes) snapshot=4174807040
		Total committed heap usage (bytes)=137498624
	Shuffle Errors
		BAD_ID=0
		CONNECTION=0
		IO_ERROR=0
		WRONG_LENGTH=0
		WRONG_MAP=0
		WRONG_REDUCE=0
	File Input Format Counters 
		Bytes Read=140
	File Output Format Counters 
		Bytes Written=123
word count status = true

part-r-00000文件就是計算結果

查看計算結果