大家好，我是不溫卜火，是一名計算機學院大數據專業大二的學生，暱稱來源於成語—不溫不火，本意是希望自己性情溫和。作爲一名互聯網行業的小白，博主寫博客一方面是爲了記錄自己的學習過程，另一方面是總結自己所犯的錯誤希望能夠幫助到很多和自己一樣處於起步階段的萌新。但由於水平有限，博客中難免會有一些錯誤出現，有紕漏之處懇請各位大佬不吝賜教！暫時只有csdn這一個平臺，博客主頁：https://buwenbuhuo.blog.csdn.net/

此篇爲大家帶來的是官方HBase-MapReduce與自定義。

1. 官方HBase-MapReduce

1．查看HBase的MapReduce任務的執行

[bigdata@hadoop002 hbase]$ bin/hbase mapredcp

上圖標記處爲所需jar包

2. 環境變量的導入

1. 執行環境變量的導入（臨時生效，在命令行執行下述操作）

$ export HBASE_HOME=/opt/module/hbase
$ export HADOOP_HOME=/opt/module/hadoop-2.7.2
$ export HADOOP_CLASSPATH=`${HBASE_HOME}/bin/hbase mapredcp`

// 也可以直接這樣
[bigdata@hadoop002 hbase]$ export HADOOP_CLASSPATH=`/opt/module/hbase/bin/hbase mapredcp`

配置完成後查看是否成功

2. 永久生效：在/etc/profile配置

export HBASE_HOME=/opt/module/hbase
export HADOOP_HOME=/opt/module/hadoop-2.7.2

在hadoop-env.sh中配置：（注意：在for循環之後配）

export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/opt/module/hbase/lib/*

3. 運行官方的MapReduce任務

– 案例一：統計Student表中有多少行數據

[bigdata@hadoop002 hbase]$ /opt/module/hadoop-2.7.2/bin/yarn jar lib/hbase-server-1.3.1.jar rowcounter student

– 案例二：使用MapReduce將HDFS導入到HBase

1.在本地創建一個tsv格式的文件：fruit.tsv

[bigdata@hadoop002 datas]$ vim fruit.tsv

1001	Apple	Red
1002	Pear	Yellow
1003	Pineapple	Yellow

2.創建HBase表

hbase(main):001:0> create 'fruit','info'

3.在HDFS中創建input_fruit文件夾並上傳fruit.tsv文件

[bigdata@hadoop002 datas]$ hadoop fs -mkdir /input_fruit/
[bigdata@hadoop002 datas]$ hadoop fs -put fruit.tsv /input_fruit/

4.執行MapReduce到HBase的fruit表中

[bigdata@hadoop002 hbase]$ /opt/module/hadoop-2.7.2/bin/yarn jar lib/hbase-server-1.3.1.jar importtsv \
-Dimporttsv.columns=HBASE_ROW_KEY,info:name,info:color fruit \
hdfs://hadoop002:9000/input_fruit

5.使用scan命令查看導入後的結果

hbase(main):001:0> scan ‘fruit’

經過測試證明是沒問題的

2. 自定義HBase-MapReduce1

目標：將fruit表中的一部分數據，通過MR遷入到fruit_mr表中。

1.構建ReadMapper類，用於讀取fruit表中的數據

package com.buwenbuhuo.hbase.mr;

import java.io.IOException;
import org.apache.hadoop.hbase.Cell;
import org.apache.hadoop.hbase.CellUtil;
import org.apache.hadoop.hbase.client.Put;
import org.apache.hadoop.hbase.client.Result;
import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
import org.apache.hadoop.hbase.mapreduce.TableMapper;
import org.apache.hadoop.hbase.util.Bytes;


/**
 * @author 卜溫不火
 * @create 2020-05-12 19:32
 * com.buwenbuhuo.hbase.mr - the name of the target package where the new class or interface will be created.
 * hbase0512 - the name of the current project.
 */
public class ReadMapper extends TableMapper<ImmutableBytesWritable,Put> {

    @Override
    protected void map(ImmutableBytesWritable key, Result value, Context context) throws IOException, InterruptedException {
        Put put = new Put(key.copyBytes());

        for (Cell cell : value.rawCells()) {
            if ("name".equals(Bytes.toString(CellUtil.cloneQualifier(cell)))){

                put.add(cell);

            }
        }
        context.write(key,put);
    }
}

2. 構建WriteReducer類，用於將讀取到的fruit表中的數據寫入到fruit_mr表中

package com.buwenbuhuo.hbase.mr;

import org.apache.hadoop.hbase.client.Put;
import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
import org.apache.hadoop.hbase.mapreduce.TableReducer;
import org.apache.hadoop.io.NullWritable;

import java.io.IOException;


/**
 * @author 卜溫不火
 * @create 2020-05-12 19:32
 * com.buwenbuhuo.hbase.mr - the name of the target package where the new class or interface will be created.
 * hbase0512 - the name of the current project.
 */
public class WriteReducer extends TableReducer<ImmutableBytesWritable,Put, NullWritable> {

    @Override
    protected void reduce(ImmutableBytesWritable key, Iterable<Put> values, Context context) throws IOException, InterruptedException {
        for (Put value : values) {
            context.write(NullWritable.get(),value);
        }
    }
}

3. 構建Driver用於組裝運行Job任務

package com.buwenbuhuo.hbase.mr;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.client.Put;
import org.apache.hadoop.hbase.client.Scan;
import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
import org.apache.hadoop.hbase.mapreduce.HRegionPartitioner;
import org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil;
import org.apache.hadoop.mapreduce.Job;

import java.io.IOException;

/**
 * @author 卜溫不火
 * @create 2020-05-12 19:32
 * com.buwenbuhuo.hbase.mr - the name of the target package where the new class or interface will be created.
 * hbase0512 - the name of the current project.
 */
public class Driver {

    public static void main(String[] args) throws IOException, ClassNotFoundException, InterruptedException {

        Configuration conf = HBaseConfiguration.create();
        conf.set("hbase.zookeeper.quorum", "hadoop002,hadoop003,hadoop004");
        conf.set("hbase.zookeeper.property.clientPort", "2181");

        Job job = Job.getInstance();

        job.setJarByClass(Driver.class);

        Scan scan = new Scan();

        TableMapReduceUtil.initTableMapperJob(
                "fruit",
                scan,
                ReadMapper.class,
                ImmutableBytesWritable.class,
                Put.class,
                job
        );

        job.setNumReduceTasks(100);

        TableMapReduceUtil.initTableReducerJob("fruit_mr",WriteReducer.class,job,HRegionPartitioner.class);

        job.waitForCompletion(true);

    }

}

4. 打包並上傳
5. 創建表

hbase(main):003:0> create 'fruit_mr','info'

6. 運行驗證

[bigdata@hadoop002 hbase]$ hadoop jar hbase-0512-1.0-SNAPSHOT.jar com.buwenbuhuo.hbase.mr.Driver

hbase(main):005:0> scan 'fruit_mr'

3. 自定義HBase-MapReduce2

目標：實現將HDFS中的數據寫入到HBase表中。

1. 構建ReadFruitFromHDFSMapper於讀取HDFS中的文件數據

package com.buwenbuhuo.hbase.mr2;

import org.apache.hadoop.hbase.client.Put;
import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
import org.apache.hadoop.hbase.util.Bytes;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;

import java.io.IOException;

/**
 * @author 卜溫不火
 * @create 2020-05-12 22:41
 * com.buwenbuhuo.hbase.mr2 - the name of the target package where the new class or interface will be created.
 * hbase0512 - the name of the current project.
 */
public class ReadMapper extends Mapper<LongWritable, Text, ImmutableBytesWritable, Put> {

    @Override
    protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {

        String[] split = value.toString().split("\t");

        if (split.length <= 3){
            return;
        }

        Put put = new Put(Bytes.toBytes(split[0]));

        put.addColumn(Bytes.toBytes("info"),Bytes.toBytes("name"),Bytes.toBytes(split[1]));
        put.addColumn(Bytes.toBytes("info"),Bytes.toBytes("color"),Bytes.toBytes(split[2]));

        context.write(new ImmutableBytesWritable(Bytes.toBytes(split[0])),put);

    }
}

2. 構建WriteFruitMRFromTxtReducer類

package com.buwenbuhuo.hbase.mr2;

import org.apache.hadoop.hbase.client.Put;
import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
import org.apache.hadoop.hbase.mapreduce.TableReducer;
import org.apache.hadoop.io.NullWritable;

import java.io.IOException;

/**
 * @author 卜溫不火
 * @create 2020-05-12 23:09
 * com.buwenbuhuo.hbase.mr2 - the name of the target package where the new class or interface will be created.
 * hbase0512 - the name of the current project.
 */
public class WriteReducer extends TableReducer<ImmutableBytesWritable, Put, NullWritable> {
    @Override
    protected void reduce(ImmutableBytesWritable key, Iterable<Put> values, Context context) throws IOException, InterruptedException {

        for (Put value : values){
            context.write(NullWritable.get(),value);
        }

    }
}

3.創建Driver

package com.buwenbuhuo.hbase.mr2;

import com.buwenbuhuo.hbase.mr.WriteReducer;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.client.Put;
import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
import org.apache.hadoop.hbase.mapreduce.HRegionPartitioner;
import org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;

import java.io.IOException;

/**
 * @author 卜溫不火
 * @create 2020-05-12 23:09
 * com.buwenbuhuo.hbase.mr2 - the name of the target package where the new class or interface will be created.
 * hbase0512 - the name of the current project.
 */
public class Driver {

    public static void main(String[] args) throws IOException, ClassNotFoundException, InterruptedException {
        Configuration conf = HBaseConfiguration.create();
        conf.set("hbase.zookeeper.quorum", "hadoop002,hadoop003,hadoop004");
        Job job = Job.getInstance(conf);

        job.setJarByClass(Driver.class);

        job.setMapperClass(ReadMapper.class);

        job.setMapOutputKeyClass(ImmutableBytesWritable.class);
        job.setMapOutputValueClass(Put.class);

        FileInputFormat.setInputPaths(job,new Path("/input_fruit"));

        job.setNumReduceTasks(10);

        TableMapReduceUtil.initTableReducerJob("fruit_mr", WriteReducer.class,job, HRegionPartitioner.class);

        job.waitForCompletion(true);
    }

}

4. 打包上傳
5.測試運行

[bigdata@hadoop002 hbase]$ hadoop jar hbase-0512-1.0-SNAPSHOT.jar com.buwenbuhuo.hbase.mr2.Driver

本次的分享就到這裏了,

好書不厭讀百回，熟讀課思子自知。而我想要成爲全場最靚的仔，就必須堅持通過學習來獲取更多知識，用知識改變命運，用博客見證成長，用行動證明我在努力。
如果我的博客對你有幫助、如果你喜歡我的博客內容，請“點贊” “評論”“收藏”一鍵三連哦！聽說點讚的人運氣不會太差，每一天都會元氣滿滿呦！如果實在要白嫖的話，那祝你開心每一天，歡迎常來我博客看看。
碼字不易，大家的支持就是我堅持下去的動力。點贊後不要忘了關注我哦！

HBase快速入門系列(7) | 官方HBase-MapReduce與自定義

目錄

1. 官方HBase-MapReduce

1．查看HBase的MapReduce任務的執行

2. 環境變量的導入

2. 自定義HBase-MapReduce1

3. 自定義HBase-MapReduce2

Win10 LTSC 2019 安裝後的一些步驟

推薦2款開源、美觀的WinForm UI控件庫

NET9 AspnetCore將整合OpenAPI的文檔生成功能而無需三方庫

在Linux下管理MySQL的大小寫敏感性

Spark快速入門系列(4) | Spark環境搭建—standalone(1) 集羣的搭建

Spark快速入門系列(3) | 簡單一文了解Spark核心概念

Spark快速入門系列(2) | Spark 運行模式之Local本地模式

Spark快速入門系列(1) | 深入淺出，一文讓你瞭解什麼是Spark

scala快速入門系列(1) | scala的簡單介紹

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結