如何在Redhat7.7中安裝CDP DC7.0.3
標籤(空格分隔): 大數據平臺構建
- 一:關於CDP DC 7.0.3 概述
- 二: 系統環境初始化
- 三: 構建CDP DC 7.0.3
一: 關於CDP DC 7.0.3 概述
1.1 CDP dc 7.0.3
CDP DC7.0.3是Cloudera與Hortonworks合併後,第一個融合CDH和HDP所有組件的on-premise版本,CDP Data Center主要由Cloudera Runtime構成,Cloudera Runtime由超過35個開源項目組成,當然CDP Data Center還包括其它功能如管理功能Cloudera Manager,Key Management,專業支持等,如下圖所示:
1.2 Cloudera Runtime的主要組件版本如下圖所示:
二: 系統環境初始化
2.1 系統環境介紹
OS: rhel7.7x64
cat /etc/hosts
---
192.168.100.21 rhel01.flyfish
192.168.100.22 rhel02.flyfish
192.168.100.23 rhel03.flyfish
192.168.100.24 rhel04.flyfish
192.168.100.25 rhel05.flyfish
192.168.100.26 rhel06.flyfish
---
本次採用 前四臺 機器
2.2 無密鑰登錄配置
做root用戶無密鑰認證
ssh-keygen ---一直敲回車到最後
cat id_rsa.pub >> authorized_keys
將所有的公鑰導入authorized_keys 分發到 所有的 機器的.ssh/ 下面
然後測試
2.3關閉 firewalld,iptables 與 selinux
所有節點都要配置:
yum install -y firewall* iptable*
(1) 關閉firewalld
systemctl stop firewalld.service
systemctl disable firewalld.service
firewall-cmd --state
(2) 關閉iptables
iptables -F
systemctl stop iptables.service
service iptables save
systemctl disable iptables.service
(3) 禁用 selinux
sed -i '/SELINUX/s/enforcing/disabled/' /etc/selinux/config
setenforce 0
getenforce 0
sestatus
重啓機器
reboot
2.4 所有機器更改RHEL7.7X64的文件句柄數
vim /etc/security/limits.conf
---
* soft nofile 65535
* hard nofile 1029345
* soft nproc unlimited
* hard nproc unlimited
* soft memlock unlimited
* hard memlock unlimited
---
ansible cdh7 -m copy -a "src=/etc/security/limits.conf dest=/etc/security/"
2.5 時間同步:
所有機器 yum install chrony*
1. rhel01.flyfish:
vim /etc/chrony.conf 增加
----
註釋掉相關默認的server
server ntp1.aliyun.com iburst
allow 192.168.100.0/24
service chronyd start
chkconfig chronyd on
2. rhel02.flyfish----rhel04.flyfish 同步rhel01.flyfish
vim /etc/chrony.conf
----
註釋掉默認的server
server 192.168.100.21 iburst
----
service chronyd stop
service chronyd start
chkconfig chronyd on
2.7 cdp dc 7.0.3 安裝所需文件下載
CM 的rpm包:
---
cloudera-manager-agent-7.0.3-1635136.el7.x86_64.rpm
cloudera-manager-daemons-7.0.3-1635136.el7.x86_64.rpm
cloudera-manager-server-7.0.3-1635136.el7.x86_64.rpm
cloudera-manager-server-db-2-7.0.3-1635136.el7.x86_64.rpm
enterprise-debuginfo-7.0.3-1635136.el7.x86_64.rpm
openjdk8-8.0+232_9-cloudera.x86_64.rpm
allkeys.asc
下載地址:
http://archive.cloudera.com/cm7/7.0.3/redhat7/yum/RPMS/x86_64/
http://archive.cloudera.com/cm7/7.0.3/allkeys.asc
----
cdh7.0.3 的parcel:
--
CDH-7.0.3-1.cdh7.0.3.p0.1635019-el7.parcel
CDH-7.0.3-1.cdh7.0.3.p0.1635019-el7.parcel.sha256
manifest.json
下載地址:
http://archive.cloudera.com/cdh7/7.0.3.0/parcels/
---
2.8 設置swap
將所有的swap爲1,並即時生效
ansible cdh7 -m shell -a "echo '1' > /proc/sys/vm/swappiness"
ansible cdh7 -m shell -a "sysctl -a | grep vm.swappiness"
vim /etc/sysctl.conf
----
最後加上:
vm.swappiness = 1
----
sysctl -p
2.8 設置最大透明數
ansible cdh7 -m shell -a "echo 'never' > /sys/kernel/mm/transparent_hugepage/defrag"
ansible cdh7 -m shell -a "echo 'never' > /sys/kernel/mm/transparent_hugepage/enabled"
ansible cdh7 -m shell -a "cat /sys/kernel/mm/transparent_hugepage/enabled"
ansible cdh7 -m shell -a "cat /sys/kernel/mm/transparent_hugepage/defrag"
修改所有節點的/etc/rc.d/rc.local文件的權限以實現開機執行
ansible cdh7 -m shell -a "chmod +x /etc/rc.d/rc.local"
vim /etc/rc.d/rc.local
---
最後加上:
if test -f /sys/kernel/mm/transparent_hugepage/enabled;
then
echo never > /sys/kernel/mm/transparent_hugepage/enabled
fi
if test -f /sys/kernel/mm/transparent_hugepage/defrag;
then
echo never > /sys/kernel/mm/transparent_hugepage/defrag
fi
---
ansible cdh7 -m copy -a "src=/etc/rc.d/rc.local dest=/etc/rc.d/"
ansible cdh7 -m shell -a "tail -f /etc/rc.d/rc.local"
2.9 配置httpd 服務器
ansible rhel01.flyfish -m yum -a "state=present name=httpd"
ansible rhel01.flyfish -m service -a "name=httpd state=started"
ansible rhel01.flyfish -m shell -a "ps -ef |grep httpd"
修改 /etc/httpd/conf/httpd.conf 在<IfModule mime_module>中修改以下內容
vim /etc/httpd/conf/httpd.conf
---
AddType application/x-compress .Z
AddType application/x-gzip .gz .tgz .parcel
---
service httpd restart
2.10 安裝maridb 數據庫
yum install mariadb*
service mariadb start
chkonfig mariadb on
mysql_secure_installation
2.11 創建CM,Hive等需要的表
vim cm.sql
----
create database metastore default character set utf8;
CREATE USER 'hive'@'%' IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON metastore. * TO 'hive'@'%';
FLUSH PRIVILEGES;
create database cm default character set utf8;
CREATE USER 'cm'@'%' IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON cm. * TO 'cm'@'%';
FLUSH PRIVILEGES;
create database rm default character set utf8;
CREATE USER 'rm'@'%' IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON rm. * TO 'rm'@'%';
FLUSH PRIVILEGES;
create database hue default character set utf8;
CREATE USER 'hue'@'%' IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON hue. * TO 'hue'@'%';
FLUSH PRIVILEGES;
create database oozie default character set utf8;
CREATE USER 'oozie'@'%' IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON oozie. * TO 'oozie'@'%';
FLUSH PRIVILEGES;
create database ranger default character set utf8;
CREATE USER 'rangeradmin'@'%' IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON ranger. * TO 'rangeradmin'@'%';
FLUSH PRIVILEGES;
----
mysql -uroot -pflyfish225
source cm.sql
2.12 安裝JDBC驅動
mkdir -p /usr/share/java/
tar -zxvf mysql-connector-java-5.1.44.tar.gz
mv mysql-connector-java-5.1.44-bin.jar /usr/share/java
cd /usr/share/java/
mv mysql-connector-java-5.1.44-bin.jar mysql-connector-java.jar
chmod 777 mysql-connector-java.jar
三: 構建CDP DC 7.0.3
3.1 安裝 jdk
所有節點 卸載系統自帶jdk
rpm -qa |grep jdk
rpm -e java-1.8.0-openjdk-1.8.0.222.b03-1.el7.x86_64 copy-jdk-configs-3.3-10.el7_5.noarch java-1.8.0-openjdk-headless-1.8.0.222.b03-1.el7.x86_64 --nodeps
yum -y install openjdk8-8.0+232_9-cloudera
vim /etc/profile
---
到最後加上jdk 的環境變量
export JAVA_HOME=/usr/java/jdk1.8.0_232-cloudera
export CLASSPATH=.:$JAVA_HOME/jre/lib:$JAVA_HOME/lib:$JAVA_HOME/lib/tools.jar
PATH=$PATH:$HOME/bin:$JAVA_HOME/bin
----
source /etc/profile
java -version
3.2 配置cdh7.0.3 的源
mv cdh7.0.3 cdh7
cd cdh7
createrepo .
cd ..
mv cdh7 /var/www/html/
製作Cloudera Manager的repo源
cd /etc/yum.reos.d/
vim cdh.repo
---
[cmrepo]
name = cm_repo
baseurl = http://192.168.100.21/cdh7/
enable = true
gpgcheck = false
---
ansible cdh7 -m copy -a "src=/etc/repos.d/cdh7.repo dest=/etc/repos.d/"
yum repolist
3.3 安裝Cloudera Manager Server
yum -y install cloudera-manager-server
初始化數據庫
/opt/cloudera/cm/schema/scm_prepare_database.sh mysql cm cm password
啓動Cloudera Manager Server
systemctl start cloudera-scm-server
systemctl status cloudera-scm-server
cd /var/log/cloudera-scm-server
tail -100f cloudera-scm-server.log
netstat -nultp |grep 7180
3.4 通過http://192.168.100.21:7180/cmf/login訪問CM
用戶名:admin
密碼: admin
登錄到CM,選擇60天試用,點擊繼續
集羣安裝:
集羣名字
識別集羣的主機
Parcel Repository & Network Settings
安裝 cloudera-scm-agent
分配parcel包
點擊Inspect Network Performance和Inspect Hosts,檢查主機
定義服務:
點擊“繼續”,進入下一步,測試數據庫連接
首先輸入CM的用戶名和密碼admin/admin
以下4個密碼,都需要包含至少一個字母與一個數字,爲了安裝方便都輸入admin@123
安裝服務初始化