hadoop-ha QJM 架構部署

公司之前老的hadoop集羣namenode有單點風險，最近學習此鏈接http://www.binospace.com/index.php/hdfs-ha-quorum-journal-manager/ 牛人上的hadoop高可用部署，受益非淺，自己搞了一個和自己集羣比較匹配的部署邏輯圖，供要用hadoop的兄弟們使用，

如下圖：

部署過程，有時間整理完了，給兄弟們奉上，供大家參考少走變路，哈哈！

一，安裝準備

操作系統 centos6.2

7臺虛擬機

192.168.10.138 yum-test.h.com #需要從 cloudera 取最新穩定的yum包到本地，

192.168.10.134 namenode.h.com

192.168.10.139 snamenode.h.com

192.168.10.135 datanode1.h.com

192.168.10.140 datanode2.h.com

192.168.10.141 datanode3.h.com

192.168.10.142 datanode4.h.com

以上對應的主機名和域名加到七臺主機的 /etc/hosts中，

二，安裝篇

master-namenode 上安裝如下包

yum install hadoop-yarn hadoop-mapreduce hadoop-hdfs-zkfc hadoop-hdfs-journalnode impala-lzo* hadoop-hdfs-namenode impala-state-store impala-catalog hive-metastore -y

注：最後安裝

standby-namenode 上安裝如下包

yum install hadoop-yarn hadoop-yarn-resourcemanager hadoop-hdfs-namenode hadoop-hdfs-zkfc hadoop-hdfs-journalnode hadoop-mapreduce hadoop-mapreduce-historyserver -y

datanode 集羣安裝（4臺）以下簡稱爲dn節點：

yum install zookeeper zookeeper-server hive-hbase hbase-master hbase hbase-regionserver impala impala-server impala-shell impala-lzo* hadoop-hdfs hadoop-hdfs-datanode hive hive-server2 hive-jdbc hadoop-yarn hadoop-yarn-nodemanager -y

三，服務配置篇：

nn 節點:

cd /etc/hadoop/conf/

vim core-site.xml

<name>fs.defaultFS</name>

<value>hdfs://namenode.h.com:8020/</value>

</property>

<name>fs.default.name</name>

<value>hdfs://namenode.h.com:8020/</value>

</property>

<name>ha.zookeeper.quorum</name>

<value>namenode.h.com,datanode01.h.com,datanode02.h.com,datanode03.h.com,datanode04.h.com</value>

</property>

<name>fs.trash.interval</name>

</property>

<name>io.file.buffer.size</name>

</property>

<name>io.compression.codecs</name>

<value>org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.BZip2Codec,com.hadoop.compression.lzo.LzopCodec,org.apache.hadoop.io.compress.SnappyCodec</value>

</property>

<name>io.compression.codec.lzo.class</name>

<value>com.hadoop.compression.lzo.LzoCodec</value>

</property>

</configuration>

cat hdfs-site.xml

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<name>dfs.namenode.name.dir</name>

</property>

<!-- hadoop-datanode-- >

<!--

<name>dfs.datanode.data.dir</name>

</property>

-->

<name>dfs.nameservices</name>

<value>wqkcluster</value>

</property>

<name>dfs.ha.namenodes.wqkcluster</name>

</property>

<name>dfs.namenode.rpc-address.wqkcluster.nn1</name>

<value>namenode.h.com:8020</value>

</property>

<name>dfs.namenode.rpc-address.wqkcluster.nn2</name>

<value>snamenode.h.com:8020</value>

</property>

<name>dfs.namenode.http-address.wqkcluster.nn1</name>

<value>namenode.h.com:50070</value>

</property>

<name>dfs.namenode.http-address.wqkcluster.nn2</name>

<value>snamenode.h.com:50070</value>

</property>

<name>dfs.namenode.shared.edits.dir</name>

<value>qjournal://namenode.h.com:8485;snamenode.h.com:8485;datanode01.h.com:8485/wqkcluster</value>

</property>

<name>dfs.journalnode.edits.dir</name>

</property>

<name>dfs.client.failover.proxy.provider.wqkcluster</name>

<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>

</property>

<name>dfs.ha.fencing.methods</name>

<value>sshfence(hdfs)</value>

</property>

<name>dfs.ha.fencing.ssh.private-key-files</name>

<value>/var/lib/hadoop-hdfs/.ssh/id_rsa</value>

</property>

<name>dfs.ha.automatic-failover.enabled</name>

</property>

<name>dfs.https.port</name>

</property>

<name>dfs.replication</name>

</property>

<name>dfs.block.size</name>

</property>

<name>dfs.datanode.max.xcievers</name>

</property>

<name>fs.permissions.umask-mode</name>

</property>

<name>dfs.permissions.superusergroup</name>

<value>hadoop</value>

</property>

<name>dfs.client.read.shortcircuit</name>

</property>

<name>dfs.domain.socket.path</name>

<value>/var/run/hadoop-hdfs/dn._PORT</value>

</property>

<name>dfs.client.file-block-storage-locations.timeout</name>

</property>

<name>dfs.datanode.hdfs-blocks-metadata.enabled</name>

</property>

<name>dfs.client.domain.socket.data.traffic</name>

<value>false</value>

</property>

</configuration>

cat yarn-site.xml

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<name>yarn.resourcemanager.resource-tracker.address</name>

<value>snamenode.h.com:8031</value>

</property>

<name>yarn.resourcemanager.address</name>

<value>snamenode.h.com:8032</value>

</property>

<name>yarn.resourcemanager.admin.address</name>

<value>snamenode.h.com:8033</value>

</property>

<name>yarn.resourcemanager.scheduler.address</name>

<value>snamenode.h.com:8030</value>

</property>

<name>yarn.resourcemanager.webapp.address</name>

<value>snamenode.h.com:8088</value>

</property>

<name>yarn.nodemanager.local-dirs</name>

<value>/data1/yarn/local,/data2/yarn/local,/data3/yarn/local,/data4/yarn/local</value>

</property>

<name>yarn.nodemanager.log-dirs</name>

</property>

<name>yarn.nodemanager.remote-app-log-dir</name>

</property>

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce.shuffle</value>

</property>

<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>

<value>org.apache.hadoop.mapred.ShuffleHandler</value>

</property>

<name>yarn.log-aggregation-enable</name>

</property>

<name>yarn.application.classpath</name>

<value>

$HADOOP_CONF_DIR,

$HADOOP_COMMON_HOME/*,

$HADOOP_COMMON_HOME/lib/*,

$HADOOP_HDFS_HOME/*,

$HADOOP_HDFS_HOME/lib/*,

$HADOOP_MAPRED_HOME/*,

$HADOOP_MAPRED_HOME/lib/*,

$YARN_HOME/*,

$YARN_HOME/lib/*</value>

</property>

<!--

<name>yarn.resourcemanager.scheduler.class</name>

<value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.FifoScheduler</value>

</property>

-->

<name>yarn.resourcemanager.max-completed-applications</name>

</property>

</configuration>

配置服務過程中，往其它節點分發配置的腳本：

cat /root/cmd.sh

#!/bin/sh

for ip in 134 139 135 140 141 142;do

echo "==============="$node"==============="

ssh 10.168.35.$ip $1

done

cat /root/syn.sh

#!/bin/sh

for ip in 127 120 121 122 123;do

scp -r $1 10.168.35.$ip:$2

done

journalnode部署在 namenode ,snamenode,datanode1三個節點上創建目錄：

namenode:

mkdir -p /data/dfs/jn ; chown -R hdfs:hdfs /data/dfs/jn

snamenode:

mkdir -p /data/dfs/jn ; chown -R hdfs:hdfs /data/dfs/jn

dn1

mkdir -p /data/dfs/jn ; chown -R hdfs:hdfs /data/dfs/jn

啓動三個journalnode

/root/cmd.sh "for x in `ls /etc/init.d/|grep hadoop-hdfs-journalnode` ; do service $x start ; done"

格式化集羣hdfs存儲（primary）：

namenode上創建目及給相關權限：

mkdir -p /data/dfs/nn ; chown hdfs.hdfs /data/dfs/nn -R

sudo -u hdfs hdfs namenode -format;/etc/init.d/hadoop-hdfs-namenode start

snamenode上操作（standby）

mkdir -p /data/dfs/nn ; chown hdfs.hdfs /data/dfs/nn -R

ssh snamenode 'sudo -u hdfs hdfs namenode -bootstrapStandby ; sudo service hadoop-hdfs-namenode start'

datanode上創建目錄及權限：

hdfs：

mkdir -p /data{1,2}/dfs ; chown hdfs.hdfs /data{1,2}/dfs -R

yarn:

mkdir -p /data{1,2}/yarn; chown yarn.yarn /data{1,2}/yarn -R

在namenode和snamenode上配置hdfs用戶間無密碼登陸

namenode:

#passwd hdfs

#su - hdfs

$ ssh-keygen

$ ssh-copy-id snamenode

snamenode:

#passwd hdfs

#su - hdfs

$ ssh-keygen

$ ssh-copy-id namenode

在兩個NameNode上安裝hadoop-hdfs-zkfc

yum install hadoop-hdfs-zkfc

hdfs zkfc -formatZK

service hadoop-hdfs-zkfc start

測試執行手動切換：

sudo -u hdfs hdfs haadmin -failover nn1 nn2

查看某Namenode的狀態：

sudo -u hdfs hdfs haadmin -getServiceState nn2

sudo -u hdfs hdfs haadmin -getServiceState nn1

配置啓動yarn

在 hdfs 上創建目錄：

sudo -u hdfs hadoop fs -mkdir -p /yarn/apps

sudo -u hdfs hadoop fs -chown yarn:mapred /yarn/apps

sudo -u hdfs hadoop fs -chmod -R 1777 /yarn/apps

sudo -u hdfs hadoop fs -mkdir /user

sudo -u hdfs hadoop fs -chmod 777 /user

sudo -u hdfs hadoop fs -mkdir -p /user/history

sudo -u hdfs hadoop fs -chmod -R 1777 /user/history

sudo -u hdfs hadoop fs -chown mapred:hadoop /user/history

snamenode 啓動yarn-mapred-historyserver

sh /root/cmd.sh ' for x in `ls /etc/init.d/|grep hadoop-mapreduce-historyserver` ; do service $x start ; done'

爲每個 MapReduce 用戶創建主目錄，比如說 hive 用戶或者當前用戶：

sudo -u hdfs hadoop fs -mkdir /user/$USER

sudo -u hdfs hadoop fs -chown $USER /user/$USER

每個節點啓動 YARN :

sh /root/cmd.sh ' for x in `ls /etc/init.d/|grep hadoop-yarn` ; do service $x start ; done'

檢查yarn是否啓動成功：

sh /root/cmd.sh ' for x in `ls /etc/init.d/|grep hadoop-yarn` ; do service $x status ; done'

測試yarn

sudo -u hdfs hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar randomwriter out

安裝hive（在namenode上進行）

sh /root/cmd.sh 'yum install hive hive-hbase hvie-server hive server2 hive-jdbc -y' 上面可能已安裝這些包，檢查一下，

下載mysql jar並設置軟連接：

ln -s /usr/share/java/mysql-connector-java-5.1.25-bin.jar /usr/lib/hive/lib/mysql-connector-java.jar

創建數據庫和用戶：

mysql -e "

CREATE DATABASE metastore;

USE metastore;

SOURCE /usr/lib/hive/scripts/metastore/upgrade/mysql/hive-schema-0.10.0.mysql.sql;

CREATE USER 'hiveuser'@'%' IDENTIFIED BY 'redhat';

CREATE USER 'hiveuser'@'localhost' IDENTIFIED BY 'redhat';

CREATE USER 'hiveuser'@'bj03-bi-pro-hdpnameNN' IDENTIFIED BY 'redhat';

REVOKE ALL PRIVILEGES, GRANT OPTION FROM 'hiveuser'@'%';

REVOKE ALL PRIVILEGES, GRANT OPTION FROM 'hiveuser'@'localhost';

REVOKE ALL PRIVILEGES, GRANT OPTION FROM 'hiveuser'@'bj03-bi-pro-hdpnameNN';

GRANT SELECT,INSERT,UPDATE,DELETE,LOCK TABLES,EXECUTE ON metastore.* TO 'hiveuser'@'%';

GRANT SELECT,INSERT,UPDATE,DELETE,LOCK TABLES,EXECUTE ON metastore.* TO 'hiveuser'@'localhost';

GRANT SELECT,INSERT,UPDATE,DELETE,LOCK TABLES,EXECUTE ON metastore.* TO 'hiveuser'@'bj03-bi-pro-hdpnameNN';

FLUSH PRIVILEGES;

修改hive配置文件：

cat /etc/hive/conf/hive-site.xml

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<name>javax.jdo.rootion.ConnectionURL</name>

<value>jdbc:mysql://namenode.h.com:3306/metastore?useUnicode=true&characterEncoding=UTF-8</value>

</property>

<name>javax.jdo.rootion.ConnectionDriverName</name>

<value>com.mysql.jdbc.Driver</value>

</property>

<name>javax.jdo.rootion.ConnectionUserName</name>

<value>hiveuser</value>

</property>

<name>javax.jdo.rootion.ConnectionPassword</name>

<value>redhat</value>

</property>

<name>datanucleus.autoCreateSchema</name>

<value>false</value>

</property>

<name>mapreduce.framework.name</name>

</property>

<name>hive.metastore.local</name>

<value>false</value>

</property>

<name>hive.files.umask.value</name>

</property>

<name>hive.metastore.uris</name>

<value>thrift://namenode.h.com:9083</value>

</property>

<name>yarn.resourcemanager.resource-tracker.address</name>

<value>namenode.h.com:8031</value>

</property>

<name>hive.metastore.warehouse.dir</name>

<value>/user/hive/warehouse</value>

</property>

<name>hive.metastore.cache.pinobjtypes</name>

<value>Table,Database,Type,FieldSchema,Order</value>

</property>

</configuration>

創建目錄並設置權限：

sudo -u hdfs hadoop fs -mkdir /user/hive

sudo -u hdfs hadoop fs -chown hive /user/hive

sudo -u hdfs hadoop fs -mkdir /user/hive/warehouse

sudo -u hdfs hadoop fs -chmod 1777 /user/hive/warehouse

sudo -u hdfs hadoop fs -chown hive /user/hive/warehouse

啓動metastore:

service hive-metastore start

安裝zk：（安裝namenode和4個dn節點上）

sh /root/cmd.sh 'yum install zookeeper* -y'

修改zoo.cfg，添加下面代碼：

server.1=namenode.h.com:2888:3888

server.2=datanode01.h.com:2888:3888

server.3=datanode02.h.com:2888:3888

server.4=datanode03.h.com:2888:3888

server.5=datanode04.h.com:2888:3888

將配置文件同步到其他節點：

sh /root/syn.sh /etc/zookeeper/conf /etc/zookeeper/

在每個節點上初始化並啓動 zookeeper，注意 n 的值需要和 zoo.cfg 中的編號一致。

sh /root/cmd.sh 'mkdir -p /data/zookeeper; chown -R zookeeper:zookeeper /data/zookeeper ; rm -rf /data/zookeeper/*'

ssh 192.168.10.134 'service zookeeper-server init --myid=1'

ssh 192.168.10.135 'service zookeeper-server init --myid=2'

ssh 192.168.10.140 'service zookeeper-server init --myid=3'

ssh 192.168.10.141'service zookeeper-server init --myid=4'

ssh 192.168.10.142 'service zookeeper-server init --myid=5'

檢查是否初始化成功：

sh /root/cmd.sh 'cat /data/zookeeper/myid'

啓動zk:

sh /root/cmd.sh 'service zookeeper-server start'

通過下面命令測試是否啓動成功：

zookeeper-client -server namenode.h.com:2181

安裝hbase（部署在4個dn節點上）

設置時鐘同步：

sh /root/cmd.sh 'yum install ntpdate -y; ntpdate pool.ntp.org;

sh /root/cmd.sh ' ntpdate pool.ntp.org'

設置crontab：

sh /root/cmd.sh ‘echo "* 3 * * * ntpdate pool.ntp.org" > /var/spool/cron/root’

在4個數據節點上安裝hbase：

注：上面yum 已完成安裝

在 hdfs 中創建 /hbase 目錄

sudo -u hdfs hadoop fs -mkdir /hbase;sudo -u hdfs hadoop fs -chown hbase:hbase /hbase

修改hbase配置文件，

配置 cat /etc/hbase/conf/hbase-site.xml

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<name>hbase.rootdir</name>

<value>hdfs://wqkcluster/hbase</value>

</property>

<name>dfs.support.append</name>

</property>

<name>hbase.cluster.distributed</name>

</property>

<name>hbase.hregion.max.filesize</name>

</property>

<name>hbase.hregion.memstore.flush.size</name>

</property>

<name>hbase.security.authentication</name>

<value>simple</value>

</property>

<name>zookeeper.session.timeout</name>

</property>

<name>hbase.zookeeper.quorum</name>

<value>datanode01.h.com,datanode02.h.com,datanode03.h.com,datanode04.h.com,namenode.h.com</value>

</property>

<name>hbase.zookeeper.property.clientPort</name>

</property>

<name>hbase.hregion.memstore.mslab.enabled</name>

</property>

<name>hbase.regions.slop</name>

</property>

<name>hbase.regionserver.handler.count</name>

</property>

<name>hbase.regionserver.lease.period</name>

</property>

<name>hbase.client.pause</name>

</property>

<name>hbase.ipc.client.tcpnodelay</name>

</property>

<name>ipc.ping.interval</name>

</property>

<name>hbase.client.retries.number</name>

</property>

<name>hbase.rpc.timeout</name>

</property>

<name>hbase.zookeeper.property.maxClientCnxns</name>

</property>

</configuration>

同步到其它四個dn節點：

sh /root/syn.sh /etc/hbase/conf /etc/hbase/

創建本地目錄：

sh /root/cmd.sh 'mkdir /data/hbase ; chown -R hbase:hbase /data/hbase/'

啓動HBase:

sh /root/cmd.sh ' for x in `ls /etc/init.d/|grep hbase` ; do service $x start ; done'

檢查是否啓動成功：

sh /root/cmd.sh ' for x in `ls /etc/init.d/|grep hbase` ; do service $x status ; done'

安裝impala（安裝在namenode和4個dn節點上）

在namenode節點安裝impala-state-store impala-catalog

安裝過程參考上面

在4個dn節點上安裝impala impala-server impala-shell impala-udf-devel：

安裝過程參考上面

拷貝mysql jdbc jar到impala目錄，並分發到四個dn節點上

sh /root/syn.sh /usr/lib/hive/lib/mysql-connector-java.jar /usr/lib/impala/lib/

在每個節點上創建/var/run/hadoop-hdfs:

sh /root/cmd.sh 'mkdir -p /var/run/hadoop-hdfs'

將hive和hdfs配置文件拷貝到impala conf,並分發到4個dn節點上。

cp /etc/hive/conf/hive-site.xml /etc/impala/conf/

cp /etc/hadoop/conf/hdfs-site.xml /etc/impala/conf/

cp /etc/hadoop/conf/core-site.xml /etc/impala/conf/

sh /root/syn.sh /etc/impala/conf /etc/impala/

修改 /etc/default/impala，然後將其同步到impala節點上：

IMPALA_CATALOG_SERVICE_HOST=bj03-bi-pro-hdpnameNN

IMPALA_STATE_STORE_HOST=bj03-bi-pro-hdpnameNN

IMPALA_STATE_STORE_PORT=24000

IMPALA_BACKEND_PORT=22000

IMPALA_LOG_DIR=/var/log/impala

sh /root/syn.sh /etc/default/impala /etc/default/

啓動 impala:

sh /root/cmd.sh ' for x in `ls /etc/init.d/|grep impala` ; do service $x start ; done'

檢查是否啓動成功：

sh /root/cmd.sh ' for x in `ls /etc/init.d/|grep impala` ; do service $x status ; done'

四，測試篇：

hdfs 服務狀態測試

sudo -u hdfs hadoop dfsadmin -report

hdfs 文件上傳，下載

su - hdfs hadoop dfs -put test.txt /tmp/

mapreduce 任務測試

bin/Hadoop jar \

share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar \

wordcount \

-files hdfs:///tmp/text.txt \

/test/input \

/test/output

測試yarn

sudo -u hdfs hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar randomwriter out

namenode 自動切換測試

執行手動切換：

sudo -u hdfs hdfs haadmin -failover nn1 nn2

[root@snamenode ~]# sudo -u hdfs hdfs haadmin -getServiceState nn1

active

[root@snamenode ~]# sudo -u hdfs hdfs haadmin -getServiceState nn2

standby

hadoop-ha QJM 架構部署

公司剛入職了一名 Java 中級開發，短短 4 行代碼居然湊齊了 3 個 bug！我哭了~~

公衆號5月C#/.NET熱文一覽

git 下載大陸鏡像地址

mfs vs fastdfs

Spark：一個高效的分佈式計算系統

hadoop-ha QJM 架構部署

我的友情鏈接

flume-ng+Kafka+Storm+HDFS 實時系統搭建

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結