大數據平臺 CDH6.1.0 安裝配置
標籤(空格分隔): 大數據平臺構建
- 一: 關於CDH6.1.0 的介紹
- 二: 系統環境初始化
- 三: 安裝CDH6.1.0所需環境
- 四: 配置CDH6.1.0的cloudera-manager安裝
一:關於CDH6.1.0 的介紹
Cloudera在前天12月19日,對外宣佈正式發佈Cloudera Enterprise 6.1包含大數據生態版本如下:
1.組件版本:Kafka 2.0, Spark 2.4, HBase 2.1.1, Accumulo 1.9.2, SOLR 7.4
2.CDH免費版限制100個節點內
3.HDFS的糾刪碼完全支持,包括Hive,Navigator(auditing),BDR,MR,Spark
4.對象存儲支持:On-prem (Ceph), Cloud (ADLS gen 2)
5.Impala的多個distinct語句支持
6.Sqoop的HiveServer2與導入Parquet文件支持
7.OpenJDK支持
##二:系統環境初始化
2.1: 系統環境介紹
系統:CentOS7.5X64
cat /etc/hosts
192.168.20.111 NN01.yl.com NN01
192.168.20.112 SN01.yl.com SN01
192.168.20.113 DN01.yl.com DN01
192.168.20.114 DN02.yl.com DN02
2.2:無密鑰登錄配置
做root用戶無密鑰認證
ssh-keygen ---一直敲回車到最後
cat id_rsa.pub >> authorized_keys
將所有的公鑰導入authorized_keys 分發到 所有的 機器的.ssh/ 下面
然後測試
2.3:關閉 firewalld,iptables 與 selinux
所有節點都要配置:
yum install -y firewall* iptable*
(1) 關閉firewalld
systemctl stop firewalld.service
systemctl disable firewalld.service
firewall-cmd --state
(2) 關閉iptables
iptables -F
systemctl stop iptables.service
service iptables save
systemctl disable iptables.service
(3) 禁用 selinux
sed -i '/SELINUX/s/enforcing/disabled/' /etc/selinux/config
setenforce 0
getenforce 0
sestatus
重啓機器
reboot
2.4:所有機器更改CentOS7.5x64的文件句柄數
vim /etc/security/limits.conf
---
* soft nofile 65535
* hard nofile 1029345
* soft nproc unlimited
* hard nproc unlimited
* soft memlock unlimited
* hard memlock unlimited
在其它 節點上面執行命令:
rsync -arvzP [email protected]:/etc/security/limits.conf /etc/security/
同步limits.conf 文件
2.5 時間同步:
所有機器 yum install chrony*
1. NN01.yl.com:
vim /etc/chrony.conf 增加
----
註釋掉相關默認的server
server ntp1.aliyun.com iburst
allow 192.168.20.111/16
service chronyd start
chkconfig chronyd on
2. SN01.yl.com---DN01.yl.com 同步NN01.yl.com
vim /etc/chrony.conf
----
註釋掉默認的server
server 192.168.20.111 iburst
----
service chronyd stop
service chronyd start
chkconfig chronyd on
2.6: cdh6.1.0 安裝所需文件下載
cdh6.1.0 離線包:
CDH-6.1.0-1.cdh6.1.0.p0.770702-el7.parcel
CDH-6.1.0-1.cdh6.1.0.p0.770702-el7.parcel.sha256
manifest.json
下載: https://archive.cloudera.com/cdh6/6.1.0/parcels/
cdh6 的CM 包:
cloudera-manager-agent-6.1.0-769885.el7.x86_64.rpm
cloudera-manager-daemons-6.1.0-769885.el7.x86_64.rpm
cloudera-manager-server-6.1.0-769885.el7.x86_64.rpm
cloudera-manager-server-db-2-6.1.0-769885.el7.x86_64.rpm
oracle-j2sdk1.8-1.8.0+update141-1.x86_64.rpm
allkeys.asc
下載地址:
https://archive.cloudera.com/cm6/6.1.0/redhat7/yum/RPMS/x86_64/
jdbc 驅動:
mysql-connector-java-8.0.15-1.el7.noarch.rpm
下載:mysql官網下載
2.7:安裝jdk
所用大數據節點都要配置:jdk
rpm -ivh oracle-j2sdk1.8-1.8.0+update141-1.x86_64.rpm
vim /etc/profile
---
export JAVA_HOME=/usr/java/jdk1.8.0_141-cloudera
export CLASSPATH=.:$JAVA_HOME/jre/lib:$JAVA_HOME/lib:$JAVA_HOME/lib/tools.jar
PATH=$PATH:$HOME/bin:$JAVA_HOME/bin
---
source /etc/profile
java -version
2.8 安裝mariadb-server
yum install -y mariadb-server
service mariadb start
chkconfig mariadb on
mysql_secure_installation
mysql -uroot -pQingchun123.com
2.9配置 httpd 分發服務器
NN01.yl.com:
yum install -y httpd
vim /etc/httpd/conf/httpd.conf
----
修改/etc/httpd/conf/httpd.conf配置文件,在<IfModule mime_module>中修改以下內容
AddType application/x-gzip .gz .tgz .parcel
----
service httpd start
chkconfig httpd start
做CM6.1 的 分發源
yum install -y createrepo
cd /root/cm6.1/
createrepo .
mv /root/cm6.1/ /var/www/html/
mv /root/cdh6.1/ /var/www/html/
vim /etc/yum.repos.d/cm.repo
------
[cmrepo]
name = cm_repo
baseurl =http://192.168.20.111/cm6.1
enable = true
gpgcheck = false
-----
yum makecache
所有主機執行命令同步:
rsync -avrzP [email protected]:/etc/yum.repos.d/cm.repo /etc/yum.repos.d/
yum makecache
2.10 配置mysql-jdbc
rpm -ivh mysql-connector-java-8.0.15-1.el7.noarch.rpm
cd /usr/share/java/
cp -p mysql-connector-java-8.0.15.jar mysql-connector-java.jar
三: 安裝CDH6.1所需環境
3.1:安裝所需依賴包
所有大數據節點都要安裝:
yum -y install chkconfig python bind-utils psmisc libxslt zlib sqlite cyrus-sasl-plain cyrus-sasl-gssapi fuse fuse-libs redhat-lsb postgresql* portmap mod_ssl openssl-devel python-psycopg2 MySQL-python
3.2 安裝cloudera-server
yum install -y cloudera-manager-server
3.3 在 mysql 當中創建 CM 需要的註冊庫
# mysql -uroot -pQingchun123.com
mysql> create database cmf character set utf8;
mysql> grant all privileges on cmf.* to 'cmf'@'%' identified by 'cmf_1234' with grant option;
mysql> grant all privileges on cmf.* to 'cmf'@'NN01.yl.com' identified by 'cmf_1234' with grant option;
mysql> flush privileges;
3.4 注入CM-server庫 與 CM-agent端的server
/opt/cloudera/cm/schema/scm_prepare_database.sh mysql cmf cmf cmf_1234
四: 配置CDH6.1的cloudera-manager安裝
4.1 安裝啓動CM
service cloudera-scm-server start
cd /var/log/cloudera-scm-server/
tail -f cloudera-scm-server.log
選擇更多parcel
所有節點執行:
echo vm.swappiness = 10 >> /etc/sysctl.conf
echo 10 > /proc/sys/vm/swappiness
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
將下面 兩條命令
mysql 授權的配置
數據庫授權:
mysql -uroot -pQingchun123.com
---
創建hive 所需要的庫並授權:
mysql >create database cdh_hive default character set latin1;
字符必須爲拉丁(latin1),如果uft8可能會報錯Specified key was too long; max key length is 767 bytes
mysql >grant all privileges on cdh_hive.* to 'hive'@'%' identified by 'Hive_123' with grant option;
mysql >grant all privileges on cdh_hive.* to 'hive'@'NN01.yl.com' identified by 'Hive_123' with grant option;
mysql >flush privileges;
創建oozie 所需要的庫
mysql >create database cdh_oozie default character set utf8;
mysql >grant all privileges on cdh_oozie.* to 'oozie'@'%' identified by 'Oozie_123' with grant option;
mysql >grant all privileges on cdh_oozie.* to 'oozie'@'NN01.yl.com' identified by 'Oozie_123' with grant option;
mysql >flush privileges;
創建hue所需要的庫:
mysql >create database cdh_hue default character set utf8;
mysql >grant all privileges on cdh_hue.* to 'hue'@'%' identified by 'Hue_123' with grant option;
mysql >grant all privileges on cdh_hue.* to 'hue'@'NN01.yl.com' identified by 'Hue_123' with grant option;
mysql >flush privileges;
創建CM 的 監控端Activity Monitor 的庫
Activity Monitor:
mysql >create database cdh_am default character set utf8;
mysql >grant all privileges on cdh_am.* to 'am'@'%' identified by 'Am_123' with grant option;
mysql >grant all privileges on cdh_am.* to 'am'@'NN01.yl.com' identified by 'Am_123' with grant option;
mysql >flush privileges;
創建RM 的 監控端 Reports Manager 的庫
Reports Manager
mysql >create database cdh_rm default character set utf8;
mysql >grant all privileges on cdh_rm.* to 'rm'@'%' identified by 'Rm_123' with grant option;
mysql >grant all privileges on cdh_rm.* to 'rm'@'NN01.yl.com' identified by 'Rm_123' with grant option;
mysql >flush privileges;