Cloudera-manager(CDH6.3.0)大數據平臺搭建一指禪指南(impala,kudu,hdfs,hive,kafka,yarn,spark,hbase,hue)
CHD6，大量hadoop生態的重大更新升級，果斷把現有系統升級到CHD6上。
準備：
192.168.88.31 master--》機器名修改爲:manager 還是重複的master把
192.168.88.32 utility
192.168.88.33 gateway
192.168.88.34 worker1
每臺虛擬機8g內存（不想給自己找麻煩，就搞8G內存，否則會卡主各種莫名其妙的問題的）,建議最低4臺測試。3臺虛擬機也是可以安裝環境的。
[root@manager ~]# vi /etc/sysconfig/network
HOSTNAME=master
# [root@manager ~]# vi /etc/hosts
127.0.0.1 localhost manager localhost4 localhost4.localdomain4 （需要刪除）
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 （需要刪除）
//永久性的修改主機名稱，重啓後能保持修改後的。
[root@manager ~]# hostnamectl set-hostname master
[root@manager ~]# cat /etc/hostname
[root@manager ~]# reboot
[root@manager ~]# hostname
manager
設置靜態IP、DNS、網關
[hadoop@utility ~]$ sudo vi /etc/sysconfig/network-scripts/ifcfg-ens33
#BOOTPROTO=dhcp
BOOTPROTO=static #設置網卡獲得ip地址的方式，可能的選項爲static，dhcp或bootp，分別對應靜態指定的 ip地址，通過dhcp協議獲得的ip地址，通過bootp協議獲得的ip地址
IPADDR=192.168.88.31 #如果設置網卡獲得 ip地址的方式爲靜態指定，此字段就指定了網卡對應的ip地址
NETMASK=255.255.255.0 #網卡對應的網絡掩碼
NETWORK=192.168.88.0 #網卡對應的網絡地址
GATEWAY=192.168.88.2 #有時候叫做default route address
DNS1=192.168.88.2
[hadoop@utility ~]$ sudo service network restart
[hadoop@utility ~]$ sudo ifconfig -a
[hadoop@utility ~]$ sudo nslookup www.baidu.com
[hadoop@utility ~]$ sudo nmcli con mod ens33 ipv4.dns "192.168.88.2"
[hadoop@utility ~]$ sudo nmcli con up ens33
[hadoop@utility ~]$ cat /etc/resolv.conf
給每臺機器的hadoop用戶添加sudo免密碼的功能
[root@localhost ~]# vi /etc/sudoers
添加如下的聲明:
hadoop ALL=(ALL) NOPASSWD: ALL
[hadoop@manager ~]$ sudo yum install -y vim wget ntp net-tools
給每臺機器都增加hosts文件的配置,修改完成後，相互之間可以ping通
[hadoop@manager ~]$ sudo vi /etc/hosts
192.168.88.31 master
192.168.88.32 utility
192.168.88.33 gateway
192.168.88.34 worker1
在這裏有個坑需要注意的，以前初始的hosts文件中是有
127.0.0.1 localhost manager localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
的，這個是需要刪除的，否則後面安裝報錯。
[hadoop@manager ~]$ ping master
[hadoop@manager ~]$ ping www.baidu.com
每臺主機關閉防火牆（折行寫的命令最後不能有空格，否則報錯）
[hadoop@manager ~]$
sudo systemctl stop firewalld \
&& sudo systemctl disable firewalld \
&& sudo systemctl status firewalld
每臺機器關閉selinux
[hadoop@manager ~]$ sudo setenforce 0
[hadoop@manager ~]$ sudo vim /etc/selinux/config
SELINUX=permissive
[hadoop@manager ~]$ getenforce
Permissive
每臺機器配置NTP服務
[hadoop@manager ~]$ sudo vim /etc/ntp.conf
#將時鐘服務器更改爲如下4個
server 0.cn.pool.ntp.org
server 1.cn.pool.ntp.org
server 2.cn.pool.ntp.org
server 3.cn.pool.ntp.org
[hadoop@manager ~]$
sudo systemctl start ntpd \
&& sudo systemctl enable ntpd \
&& sudo ntpdate -u 0.cn.pool.ntp.org \
&& sudo hwclock --systohc \
&& sudo date
在tools主機上安裝（生產環境，減輕master的負擔，不過在我本機開發，都是用的master，所以這裏我用master進行安裝）
上傳下載的文件到
[hadoop@manager cloudera]$ pwd
/home/hadoop/Downloads/cloudera
cloudera/
├── cloudera-manager
│ ├── allkeys.asc
│ ├── cloudera-manager-agent-6.3.0-1281944.el7.x86_64.rpm
│ ├── cloudera-manager-daemons-6.3.0-1281944.el7.x86_64.rpm
│ ├── cloudera-manager-server-6.3.0-1281944.el7.x86_64.rpm
│ ├── cloudera-manager-server-db-2-6.3.0-1281944.el7.x86_64.rpm
│ ├── enterprise-debuginfo-6.3.0-1281944.el7.x86_64.rpm
│ └── oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm
├── mysql
│ └── mysql-5.7.27-1.el7.x86_64.rpm-bundle.tar
├── mysql-jdbc
│ └── mysql-connector-java-5.1.47.tar.gz
└── parcel
├── CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel
└── manifest.json
安裝httpd服務，以及爲本地創建yum安裝的庫做準備
[hadoop@manager cloudera]$
sudo yum -y install httpd createrepo \
&& sudo systemctl start httpd \
&& sudo systemctl enable httpd \
&& sudo cd /home/hadoop/Downloads/cloudera/cloudera-manager/ \
&& sudo createrepo . \
&& sudo mv /home/hadoop/Downloads/cloudera/cloudera-manager/ /var/www/html/
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
* base: mirrors.njupt.edu.cn
* extras: mirrors.njupt.edu.cn
* updates: mirrors.njupt.edu.cn
Package createrepo-0.9.9-28.el7.noarch already installed and latest version
Resolving Dependencies
--> Running transaction check
---> Package httpd.x86_64 0:2.4.6-90.el7.centos will be installed
--> Processing Dependency: httpd-tools = 2.4.6-90.el7.centos for package: httpd-2.4.6-90.el7.centos.x86_64
--> Running transaction check
---> Package httpd-tools.x86_64 0:2.4.6-90.el7.centos will be installed
--> Finished Dependency Resolution
Dependencies Resolved
===================================================================================================================================================================================================================
Package Arch Version Repository Size
===================================================================================================================================================================================================================
Installing:
httpd x86_64 2.4.6-90.el7.centos base 2.7 M
Installing for dependencies:
httpd-tools x86_64 2.4.6-90.el7.centos base 91 k
Transaction Summary
===================================================================================================================================================================================================================
Install 1 Package (+1 Dependent package)
Total download size: 2.8 M
Installed size: 9.5 M
Downloading packages:
(1/2): httpd-2.4.6-90.el7.centos.x86_64.rpm | 2.7 MB 00:00:01
httpd-tools-2.4.6-90.el7.cento FAILED
http://mirrors.cqu.edu.cn/CentOS/7.7.1908/os/x86_64/Packages/httpd-tools-2.4.6-90.el7.centos.x86_64.rpm: [Errno 14] curl#6 - "Could not resolve host: mirrors.cqu.edu.cn; Unknown error"B/s | 2.2 MB 00:00:00 ETA
Trying other mirror.
(2/2): httpd-tools-2.4.6-90.el7.centos.x86_64.rpm | 91 kB 00:00:00
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total 271 kB/s | 2.8 MB 00:00:10
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : httpd-tools-2.4.6-90.el7.centos.x86_64 1/2
Installing : httpd-2.4.6-90.el7.centos.x86_64 2/2
Verifying : httpd-tools-2.4.6-90.el7.centos.x86_64 1/2
Verifying : httpd-2.4.6-90.el7.centos.x86_64 2/2
Installed:
httpd.x86_64 0:2.4.6-90.el7.centos
Dependency Installed:
httpd-tools.x86_64 0:2.4.6-90.el7.centos
Complete!
Created symlink from /etc/systemd/system/multi-user.target.wants/httpd.service to /usr/lib/systemd/system/httpd.service.
Spawning worker 0 with 2 pkgs
Spawning worker 1 with 2 pkgs
Spawning worker 2 with 1 pkgs
Spawning worker 3 with 1 pkgs
Workers Finished
Saving Primary metadata
Saving file lists metadata
Saving other metadata
Generating sqlite DBs
Sqlite DBs complete
[hadoop@manager ~]$
[root@manager cloudera-manager]# pwd
/var/www/html/cloudera-manager
[root@manager cloudera-manager]# createrepo .
Spawning worker 0 with 2 pkgs
Spawning worker 1 with 2 pkgs
Spawning worker 2 with 1 pkgs
Spawning worker 3 with 1 pkgs
Workers Finished
Saving Primary metadata
Saving file lists metadata
Saving other metadata
Generating sqlite DBs
Sqlite DBs complete
[root@manager cloudera-manager]# ls -al
total 1378008
drwxr-xr-x. 3 hadoop hadoop 4096 Oct 31 00:17 .
drwxr-xr-x. 3 root root 30 Oct 30 23:03 ..
-rw-r--r--. 1 hadoop hadoop 14041 Oct 30 20:25 allkeys.asc
-rw-r--r--. 1 hadoop hadoop 10479136 Oct 30 20:25 cloudera-manager-agent-6.3.0-1281944.el7.x86_64.rpm
-rw-r--r--. 1 hadoop hadoop 1201341068 Oct 30 20:28 cloudera-manager-daemons-6.3.0-1281944.el7.x86_64.rpm
-rw-r--r--. 1 hadoop hadoop 11464 Oct 30 20:25 cloudera-manager-server-6.3.0-1281944.el7.x86_64.rpm
-rw-r--r--. 1 hadoop hadoop 10996 Oct 30 20:25 cloudera-manager-server-db-2-6.3.0-1281944.el7.x86_64.rpm
-rw-r--r--. 1 hadoop hadoop 14209884 Oct 30 20:25 enterprise-debuginfo-6.3.0-1281944.el7.x86_64.rpm
-rw-r--r--. 1 hadoop hadoop 184988341 Oct 30 20:26 oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm
drwxr-xr-x. 2 root root 4096 Oct 31 00:17 repodata -----》這個目錄要檢查一下，否則將來的本地yum是沒法通過的
[root@manager cloudera-manager]#
完成後，我們使用firefox打開--》http://192.168.88.31/cloudera-manager/ ，之後我們要能看到一個網頁的文件列表
Index of /cloudera-manager
[ICO] Name Last modified Size Description
[PARENTDIR] Parent Directory -
[TXT] allkeys.asc 2019-10-30 20:25 14K
[ ] cloudera-manager-age..> 2019-10-30 20:25 10M
[ ] cloudera-manager-dae..> 2019-10-30 20:28 1.1G
[ ] cloudera-manager-ser..> 2019-10-30 20:25 11K
[ ] cloudera-manager-ser..> 2019-10-30 20:25 11K
[ ] enterprise-debuginfo..> 2019-10-30 20:25 14M
[ ] oracle-j2sdk1.8-1.8...> 2019-10-30 20:26 176M
[DIR] repodata/ ----》這個是需要檢查的，如果沒有，將來是會報錯的，必須檢查
安裝JDK
[hadoop@manager cloudera-manager]$ java -version
openjdk version "1.8.0_222-ea"
OpenJDK Runtime Environment (build 1.8.0_222-ea-b03)
OpenJDK 64-Bit Server VM (build 25.222-b03, mixed mode)
[hadoop@manager cloudera-manager]$ rpm -qa | grep java
java-1.7.0-openjdk-headless-1.7.0.221-2.6.18.1.el7.x86_64
javamail-1.4.6-8.el7.noarch
tzdata-java-2019b-1.el7.noarch
java-1.8.0-openjdk-1.8.0.222.b03-1.el7.x86_64
java-1.8.0-openjdk-headless-1.8.0.222.b03-1.el7.x86_64
python-javapackages-3.4.1-11.el7.noarch
javapackages-tools-3.4.1-11.el7.noarch
java-1.7.0-openjdk-1.7.0.221-2.6.18.1.el7.x86_64
javassist-3.16.1-10.el7.noarch
原來安裝的統統都刪除掉，但是noarch文件可以不用刪除
[hadoop@manager cloudera-manager]$
sudo rpm -e --nodeps java-1.7.0-openjdk-headless-1.7.0.221-2.6.18.1.el7.x86_64 \
&& sudo rpm -e --nodeps java-1.8.0-openjdk-1.8.0.222.b03-1.el7.x86_64 \
&& sudo rpm -e --nodeps java-1.8.0-openjdk-headless-1.8.0.222.b03-1.el7.x86_64 \
&& sudo rpm -e --nodeps java-1.7.0-openjdk-1.7.0.221-2.6.18.1.el7.x86_64
[hadoop@manager cloudera-manager]$ sudo rpm -ivh /var/www/html/cloudera-manager/oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm
warning: /var/www/html/cloudera-manager/oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID b0b19c9f: NOKEY
Preparing... ################################# [100%]
Updating / installing...
1:oracle-j2sdk1.8-1.8.0+update181-1################################# [100%]
[hadoop@manager jdk1.8.0_181-cloudera]$ pwd
/usr/java/jdk1.8.0_181-cloudera
[hadoop@manager jdk1.8.0_181-cloudera]$ sudo vi /etc/profile
# set java environment
export JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
export PATH=${JAVA_HOME}/bin:${PATH}
[hadoop@manager jdk1.8.0_181-cloudera]$ source /etc/profile
[hadoop@manager jdk1.8.0_181-cloudera]$ java -version
java version "1.8.0_181"
Java(TM) SE Runtime Environment (build 1.8.0_181-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)
安裝mysql
[hadoop@manager ~]$ sudo yum -y remove mariadb-libs
Loaded plugins: fastestmirror, langpacks
Resolving Dependencies
--> Running transaction check
---> Package mariadb-libs.x86_64 1:5.5.64-1.el7 will be erased
--> Processing Dependency: libmysqlclient.so.18()(64bit) for package: perl-DBD-MySQL-4.023-6.el7.x86_64
--> Processing Dependency: libmysqlclient.so.18()(64bit) for package: 1:net-snmp-agent-libs-5.7.2-43.el7.x86_64
--> Processing Dependency: libmysqlclient.so.18()(64bit) for package: 2:postfix-2.10.1-7.el7.x86_64
--> Processing Dependency: libmysqlclient.so.18()(64bit) for package: 1:net-snmp-5.7.2-43.el7.x86_64
--> Processing Dependency: libmysqlclient.so.18()(64bit) for package: 1:qt-mysql-4.8.7-3.el7_6.x86_64
--> Processing Dependency: libmysqlclient.so.18(libmysqlclient_18)(64bit) for package: perl-DBD-MySQL-4.023-6.el7.x86_64
--> Processing Dependency: libmysqlclient.so.18(libmysqlclient_18)(64bit) for package: 1:net-snmp-agent-libs-5.7.2-43.el7.x86_64
--> Processing Dependency: libmysqlclient.so.18(libmysqlclient_18)(64bit) for package: 2:postfix-2.10.1-7.el7.x86_64
--> Processing Dependency: libmysqlclient.so.18(libmysqlclient_18)(64bit) for package: 1:qt-mysql-4.8.7-3.el7_6.x86_64
--> Processing Dependency: mariadb-libs(x86-64) = 1:5.5.64-1.el7 for package: 1:mariadb-server-5.5.64-1.el7.x86_64
--> Processing Dependency: mariadb-libs(x86-64) = 1:5.5.64-1.el7 for package: 1:mariadb-5.5.64-1.el7.x86_64
--> Processing Dependency: mysql-libs for package: 1:net-snmp-5.7.2-43.el7.x86_64
--> Running transaction check
---> Package mariadb.x86_64 1:5.5.64-1.el7 will be erased
---> Package mariadb-server.x86_64 1:5.5.64-1.el7 will be erased
--> Processing Dependency: mariadb-server for package: akonadi-mysql-1.9.2-4.el7.x86_64
---> Package net-snmp.x86_64 1:5.7.2-43.el7 will be erased
---> Package net-snmp-agent-libs.x86_64 1:5.7.2-43.el7 will be erased
--> Processing Dependency: libnetsnmpagent.so.31()(64bit) for package: keepalived-1.3.5-16.el7.x86_64
--> Processing Dependency: libnetsnmpmibs.so.31()(64bit) for package: keepalived-1.3.5-16.el7.x86_64
---> Package perl-DBD-MySQL.x86_64 0:4.023-6.el7 will be erased
---> Package postfix.x86_64 2:2.10.1-7.el7 will be erased
---> Package qt-mysql.x86_64 1:4.8.7-3.el7_6 will be erased
--> Running transaction check
---> Package akonadi-mysql.x86_64 0:1.9.2-4.el7 will be erased
---> Package keepalived.x86_64 0:1.3.5-16.el7 will be erased
--> Finished Dependency Resolution
Dependencies Resolved
===================================================================================================================================================================================================================
Package Arch Version Repository Size
===================================================================================================================================================================================================================
Removing:
mariadb-libs x86_64 1:5.5.64-1.el7 @anaconda 4.4 M
Removing for dependencies:
akonadi-mysql x86_64 1.9.2-4.el7 @anaconda 9.6 k
keepalived x86_64 1.3.5-16.el7 @anaconda 1.0 M
mariadb x86_64 1:5.5.64-1.el7 @anaconda 49 M
mariadb-server x86_64 1:5.5.64-1.el7 @anaconda 58 M
net-snmp x86_64 1:5.7.2-43.el7 @anaconda 882 k
net-snmp-agent-libs x86_64 1:5.7.2-43.el7 @anaconda 2.1 M
perl-DBD-MySQL x86_64 4.023-6.el7 @anaconda 323 k
postfix x86_64 2:2.10.1-7.el7 @anaconda 12 M
qt-mysql x86_64 1:4.8.7-3.el7_6 @anaconda 74 k
Transaction Summary
===================================================================================================================================================================================================================
Remove 1 Package (+9 Dependent packages)
Installed size: 128 M
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Warning: RPMDB altered outside of yum.
** Found 1 pre-existing rpmdb problem(s), 'yum check' output follows:
icedtea-web-1.7.1-1.el7.x86_64 has missing requires of java-1.8.0-openjdk
Erasing : akonadi-mysql-1.9.2-4.el7.x86_64 1/10
Erasing : 1:net-snmp-5.7.2-43.el7.x86_64 2/10
Erasing : 1:mariadb-server-5.5.64-1.el7.x86_64 3/10
Erasing : 1:mariadb-5.5.64-1.el7.x86_64 4/10
Erasing : perl-DBD-MySQL-4.023-6.el7.x86_64 5/10
Erasing : 1:qt-mysql-4.8.7-3.el7_6.x86_64 6/10
Erasing : 2:postfix-2.10.1-7.el7.x86_64 7/10
Erasing : keepalived-1.3.5-16.el7.x86_64 8/10
Erasing : 1:net-snmp-agent-libs-5.7.2-43.el7.x86_64 9/10
Erasing : 1:mariadb-libs-5.5.64-1.el7.x86_64 10/10
Verifying : 1:mariadb-libs-5.5.64-1.el7.x86_64 1/10
Verifying : 1:mariadb-5.5.64-1.el7.x86_64 2/10
Verifying : 1:qt-mysql-4.8.7-3.el7_6.x86_64 3/10
Verifying : keepalived-1.3.5-16.el7.x86_64 4/10
Verifying : akonadi-mysql-1.9.2-4.el7.x86_64 5/10
Verifying : 2:postfix-2.10.1-7.el7.x86_64 6/10
Verifying : perl-DBD-MySQL-4.023-6.el7.x86_64 7/10
Verifying : 1:net-snmp-agent-libs-5.7.2-43.el7.x86_64 8/10
Verifying : 1:mariadb-server-5.5.64-1.el7.x86_64 9/10
Verifying : 1:net-snmp-5.7.2-43.el7.x86_64 10/10
Removed:
mariadb-libs.x86_64 1:5.5.64-1.el7
Dependency Removed:
akonadi-mysql.x86_64 0:1.9.2-4.el7 keepalived.x86_64 0:1.3.5-16.el7 mariadb.x86_64 1:5.5.64-1.el7 mariadb-server.x86_64 1:5.5.64-1.el7 net-snmp.x86_64 1:5.7.2-43.el7
net-snmp-agent-libs.x86_64 1:5.7.2-43.el7 perl-DBD-MySQL.x86_64 0:4.023-6.el7 postfix.x86_64 2:2.10.1-7.el7 qt-mysql.x86_64 1:4.8.7-3.el7_6
Complete!
[hadoop@manager ~]$ cd /home/hadoop/Downloads/cloudera/mysql/;tar -xvf mysql-5.7.27-1.el7.x86_64.rpm-bundle.tar
mysql-community-libs-5.7.27-1.el7.x86_64.rpm
mysql-community-embedded-devel-5.7.27-1.el7.x86_64.rpm
mysql-community-libs-compat-5.7.27-1.el7.x86_64.rpm
mysql-community-devel-5.7.27-1.el7.x86_64.rpm
mysql-community-embedded-compat-5.7.27-1.el7.x86_64.rpm
mysql-community-common-5.7.27-1.el7.x86_64.rpm
mysql-community-client-5.7.27-1.el7.x86_64.rpm
mysql-community-server-5.7.27-1.el7.x86_64.rpm
mysql-community-test-5.7.27-1.el7.x86_64.rpm
mysql-community-embedded-5.7.27-1.el7.x86_64.rpm
[hadoop@manager mysql]$ pwd
/home/hadoop/Downloads/cloudera/mysql
[hadoop@manager ~]$
sudo yum install -y libaio \
&& sudo rpm -ivh mysql-community-common-5.7.27-1.el7.x86_64.rpm \
&& sudo rpm -ivh mysql-community-libs-5.7.27-1.el7.x86_64.rpm \
&& sudo rpm -ivh mysql-community-client-5.7.27-1.el7.x86_64.rpm \
&& sudo rpm -ivh mysql-community-server-5.7.27-1.el7.x86_64.rpm \
&& sudo rpm -ivh mysql-community-libs-compat-5.7.27-1.el7.x86_64.rpm
[root@manager ~]# echo character-set-server=utf8 >> /etc/my.cnf
這裏的數據庫密碼【123456Aa.】,比較重要，後面都會用到這個密碼的
數據庫名稱也是比較重要的，後面也會要求填寫這些數據庫名稱的（也就是在安裝CDH之前，就要想好要安裝哪些hadoop組件，一個組件，一個mysql庫）
[hadoop@manager mysql]$
(
cat <<EOF
set password for root@localhost = password('123456Aa.');
grant all privileges on *.* to 'root'@'%' identified by '123456Aa.';
flush privileges;
CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON scm.* TO 'scm'@'%' IDENTIFIED BY '123456Aa.';
GRANT ALL ON amon.* TO 'amon'@'%' IDENTIFIED BY '123456Aa.';
GRANT ALL ON rman.* TO 'rman'@'%' IDENTIFIED BY '123456Aa.';
GRANT ALL ON hue.* TO 'hue'@'%' IDENTIFIED BY '123456Aa.';
GRANT ALL ON metastore.* TO 'hive'@'%' IDENTIFIED BY '123456Aa.';
GRANT ALL ON sentry.* TO 'sentry'@'%' IDENTIFIED BY '123456Aa.';
GRANT ALL ON nav.* TO 'nav'@'%' IDENTIFIED BY '123456Aa.';
GRANT ALL ON navms.* TO 'navms'@'%' IDENTIFIED BY '123456Aa.';
GRANT ALL ON oozie.* TO 'oozie'@'%' IDENTIFIED BY '123456Aa.';
SHOW DATABASES;
EOF
) >> /home/hadoop/Downloads/cloudera/c.sql
[root@manager ~]# systemctl enable mysqld && systemctl start mysqld && grep password /var/log/mysqld.log | sed 's/.*\(............\)$/\1/'
dqfwl;rFO4>o --》首次登錄的密碼
mysql> ALTER USER 'root'@'localhost' IDENTIFIED BY '123xxxxxx';
Query OK, 0 rows affected (0.01 sec)
mysql> exit;
mysql> source /home/hadoop/Downloads/cloudera/c.sql
Query OK, 0 rows affected, 1 warning (0.00 sec)
Query OK, 0 rows affected, 1 warning (0.00 sec)
Query OK, 0 rows affected (0.00 sec)
Query OK, 1 row affected (0.00 sec)
Query OK, 1 row affected (0.01 sec)
Query OK, 1 row affected (0.00 sec)
Query OK, 1 row affected (0.00 sec)
Query OK, 1 row affected (0.00 sec)
Query OK, 1 row affected (0.00 sec)
Query OK, 1 row affected (0.00 sec)
Query OK, 1 row affected (0.00 sec)
Query OK, 1 row affected (0.00 sec)
Query OK, 0 rows affected, 1 warning (0.00 sec)
Query OK, 0 rows affected, 1 warning (0.00 sec)
Query OK, 0 rows affected, 1 warning (0.00 sec)
Query OK, 0 rows affected, 1 warning (0.00 sec)
Query OK, 0 rows affected, 1 warning (0.00 sec)
Query OK, 0 rows affected, 1 warning (0.00 sec)
Query OK, 0 rows affected, 1 warning (0.00 sec)
Query OK, 0 rows affected, 1 warning (0.00 sec)
Query OK, 0 rows affected, 1 warning (0.00 sec)
+--------------------+
| Database |
+--------------------+
| information_schema |
| amon |
| hue |
| metastore |
| mysql |
| nav |
| navms |
| oozie |
| performance_schema |
| rman |
| scm |
| sentry |
| sys |
+--------------------+
13 rows in set (0.00 sec)
mysql>
準備jdbc的驅動程序
[hadoop@manager mysql-jdbc]$
sudo mkdir -p /usr/share/java/ \
&& cd /home/hadoop/Downloads/cloudera/mysql-jdbc/;tar -zxvf mysql-connector-java-5.1.47.tar.gz \
&& sudo cp /home/hadoop/Downloads/cloudera/mysql-jdbc/mysql-connector-java-5.1.47/mysql-connector-java-5.1.47-bin.jar /usr/share/java/mysql-connector-java.jar
所有機器更換國內的阿里yum鏡像（這裏有坑，不是master機器做就可以了，是所有機器都要做）
[root@manager ~]# mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.bk
[root@manager ~]# wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
[root@manager ~]#
(
cat <<EOF
[cloudera-manager]
name=Cloudera Manager 6.3.0
baseurl=http://192.168.88.31/cloudera-manager/
gpgcheck=0
enabled=1
EOF
) >> /etc/yum.repos.d/cloudera-manager.repo \
&& yum clean all \
&& yum makecache
[root@manager ~]# vi /etc/yum.repos.d/cloudera-manager.repo
[root@manager ~]# cat /etc/yum.repos.d/cloudera-manager.repo
安裝Cloudera Manager
[hadoop@manager ~]$ sudo yum install -y cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
* base: mirrors.aliyun.com
* extras: mirrors.aliyun.com
* updates: mirrors.aliyun.com
Resolving Dependencies
--> Running transaction check
---> Package cloudera-manager-agent.x86_64 0:6.3.0-1281944.el7 will be installed
--> Processing Dependency: python-psycopg2 for package: cloudera-manager-agent-6.3.0-1281944.el7.x86_64
--> Processing Dependency: openssl-devel for package: cloudera-manager-agent-6.3.0-1281944.el7.x86_64
--> Processing Dependency: mod_ssl for package: cloudera-manager-agent-6.3.0-1281944.el7.x86_64
--> Processing Dependency: MySQL-python for package: cloudera-manager-agent-6.3.0-1281944.el7.x86_64
--> Processing Dependency: /lib/lsb/init-functions for package: cloudera-manager-agent-6.3.0-1281944.el7.x86_64
--> Processing Dependency: libpq.so.5()(64bit) for package: cloudera-manager-agent-6.3.0-1281944.el7.x86_64
---> Package cloudera-manager-daemons.x86_64 0:6.3.0-1281944.el7 will be installed
---> Package cloudera-manager-server.x86_64 0:6.3.0-1281944.el7 will be installed
--> Running transaction check
---> Package MySQL-python.x86_64 0:1.2.5-1.el7 will be installed
---> Package mod_ssl.x86_64 1:2.4.6-90.el7.centos will be installed
---> Package openssl-devel.x86_64 1:1.0.2k-19.el7 will be installed
--> Processing Dependency: zlib-devel(x86-64) for package: 1:openssl-devel-1.0.2k-19.el7.x86_64
--> Processing Dependency: krb5-devel(x86-64) for package: 1:openssl-devel-1.0.2k-19.el7.x86_64
---> Package postgresql-libs.x86_64 0:9.2.24-1.el7_5 will be installed
---> Package python-psycopg2.x86_64 0:2.5.1-4.el7 will be installed
---> Package redhat-lsb-core.x86_64 0:4.1-27.el7.centos.1 will be installed
--> Processing Dependency: redhat-lsb-submod-security(x86-64) = 4.1-27.el7.centos.1 for package: redhat-lsb-core-4.1-27.el7.centos.1.x86_64
--> Processing Dependency: spax for package: redhat-lsb-core-4.1-27.el7.centos.1.x86_64
--> Processing Dependency: /usr/sbin/sendmail for package: redhat-lsb-core-4.1-27.el7.centos.1.x86_64
--> Running transaction check
---> Package krb5-devel.x86_64 0:1.15.1-37.el7_7.2 will be installed
--> Processing Dependency: libkadm5(x86-64) = 1.15.1-37.el7_7.2 for package: krb5-devel-1.15.1-37.el7_7.2.x86_64
--> Processing Dependency: krb5-libs(x86-64) = 1.15.1-37.el7_7.2 for package: krb5-devel-1.15.1-37.el7_7.2.x86_64
--> Processing Dependency: libverto-devel for package: krb5-devel-1.15.1-37.el7_7.2.x86_64
--> Processing Dependency: libselinux-devel for package: krb5-devel-1.15.1-37.el7_7.2.x86_64
--> Processing Dependency: libcom_err-devel for package: krb5-devel-1.15.1-37.el7_7.2.x86_64
--> Processing Dependency: keyutils-libs-devel for package: krb5-devel-1.15.1-37.el7_7.2.x86_64
---> Package postfix.x86_64 2:2.10.1-7.el7 will be installed
---> Package redhat-lsb-submod-security.x86_64 0:4.1-27.el7.centos.1 will be installed
---> Package spax.x86_64 0:1.5.2-13.el7 will be installed
---> Package zlib-devel.x86_64 0:1.2.7-18.el7 will be installed
--> Running transaction check
---> Package keyutils-libs-devel.x86_64 0:1.5.8-3.el7 will be installed
---> Package krb5-libs.x86_64 0:1.15.1-37.el7_6 will be updated
---> Package krb5-libs.x86_64 0:1.15.1-37.el7_7.2 will be an update
---> Package libcom_err-devel.x86_64 0:1.42.9-16.el7 will be installed
---> Package libkadm5.x86_64 0:1.15.1-37.el7_7.2 will be installed
---> Package libselinux-devel.x86_64 0:2.5-14.1.el7 will be installed
--> Processing Dependency: libsepol-devel(x86-64) >= 2.5-10 for package: libselinux-devel-2.5-14.1.el7.x86_64
--> Processing Dependency: pkgconfig(libsepol) for package: libselinux-devel-2.5-14.1.el7.x86_64
--> Processing Dependency: pkgconfig(libpcre) for package: libselinux-devel-2.5-14.1.el7.x86_64
---> Package libverto-devel.x86_64 0:0.2.5-4.el7 will be installed
--> Running transaction check
---> Package libsepol-devel.x86_64 0:2.5-10.el7 will be installed
---> Package pcre-devel.x86_64 0:8.32-17.el7 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
===================================================================================================================================================================================================================
Package Arch Version Repository Size
===================================================================================================================================================================================================================
Installing:
cloudera-manager-agent x86_64 6.3.0-1281944.el7 cloudera-manager 10 M
cloudera-manager-daemons x86_64 6.3.0-1281944.el7 cloudera-manager 1.1 G
cloudera-manager-server x86_64 6.3.0-1281944.el7 cloudera-manager 11 k
Installing for dependencies:
MySQL-python x86_64 1.2.5-1.el7 base 90 k
keyutils-libs-devel x86_64 1.5.8-3.el7 base 37 k
krb5-devel x86_64 1.15.1-37.el7_7.2 updates 272 k
libcom_err-devel x86_64 1.42.9-16.el7 base 32 k
libkadm5 x86_64 1.15.1-37.el7_7.2 updates 178 k
libselinux-devel x86_64 2.5-14.1.el7 base 187 k
libsepol-devel x86_64 2.5-10.el7 base 77 k
libverto-devel x86_64 0.2.5-4.el7 base 12 k
mod_ssl x86_64 1:2.4.6-90.el7.centos base 112 k
openssl-devel x86_64 1:1.0.2k-19.el7 base 1.5 M
pcre-devel x86_64 8.32-17.el7 base 480 k
postfix x86_64 2:2.10.1-7.el7 base 2.4 M
postgresql-libs x86_64 9.2.24-1.el7_5 base 234 k
python-psycopg2 x86_64 2.5.1-4.el7 base 132 k
redhat-lsb-core x86_64 4.1-27.el7.centos.1 base 38 k
redhat-lsb-submod-security x86_64 4.1-27.el7.centos.1 base 15 k
spax x86_64 1.5.2-13.el7 base 260 k
zlib-devel x86_64 1.2.7-18.el7 base 50 k
Updating for dependencies:
krb5-libs x86_64 1.15.1-37.el7_7.2 updates 805 k
Transaction Summary
===================================================================================================================================================================================================================
Install 3 Packages (+18 Dependent packages)
Upgrade ( 1 Dependent package)
Total download size: 1.1 G
Downloading packages:
No Presto metadata available for updates
(1/22): cloudera-manager-agent-6.3.0-1281944.el7.x86_64.rpm | 10 MB 00:00:00
(2/22): cloudera-manager-server-6.3.0-1281944.el7.x86_64.rpm | 11 kB 00:00:00
(3/22): MySQL-python-1.2.5-1.el7.x86_64.rpm | 90 kB 00:00:00
(4/22): keyutils-libs-devel-1.5.8-3.el7.x86_64.rpm | 37 kB 00:00:00
(5/22): libcom_err-devel-1.42.9-16.el7.x86_64.rpm | 32 kB 00:00:00
(6/22): krb5-devel-1.15.1-37.el7_7.2.x86_64.rpm | 272 kB 00:00:00
(7/22): krb5-libs-1.15.1-37.el7_7.2.x86_64.rpm | 805 kB 00:00:00
(8/22): libkadm5-1.15.1-37.el7_7.2.x86_64.rpm | 178 kB 00:00:00
(9/22): libsepol-devel-2.5-10.el7.x86_64.rpm | 77 kB 00:00:00
(10/22): libselinux-devel-2.5-14.1.el7.x86_64.rpm | 187 kB 00:00:00
(11/22): libverto-devel-0.2.5-4.el7.x86_64.rpm | 12 kB 00:00:00
(12/22): mod_ssl-2.4.6-90.el7.centos.x86_64.rpm | 112 kB 00:00:00
(13/22): pcre-devel-8.32-17.el7.x86_64.rpm | 480 kB 00:00:00
(14/22): openssl-devel-1.0.2k-19.el7.x86_64.rpm | 1.5 MB 00:00:00
(15/22): postgresql-libs-9.2.24-1.el7_5.x86_64.rpm | 234 kB 00:00:00
(16/22): python-psycopg2-2.5.1-4.el7.x86_64.rpm | 132 kB 00:00:00
(17/22): postfix-2.10.1-7.el7.x86_64.rpm | 2.4 MB 00:00:00
(18/22): redhat-lsb-core-4.1-27.el7.centos.1.x86_64.rpm | 38 kB 00:00:00
(19/22): redhat-lsb-submod-security-4.1-27.el7.centos.1.x86_64.rpm | 15 kB 00:00:00
(20/22): zlib-devel-1.2.7-18.el7.x86_64.rpm | 50 kB 00:00:00
(21/22): spax-1.5.2-13.el7.x86_64.rpm | 260 kB 00:00:00
(22/22): cloudera-manager-daemons-6.3.0-1281944.el7.x86_64.rpm | 1.1 GB 00:00:31
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total 37 MB/s | 1.1 GB 00:00:31
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Warning: RPMDB altered outside of yum.
** Found 1 pre-existing rpmdb problem(s), 'yum check' output follows:
icedtea-web-1.7.1-1.el7.x86_64 has missing requires of java-1.8.0-openjdk
Updating : krb5-libs-1.15.1-37.el7_7.2.x86_64 1/23
Installing : postgresql-libs-9.2.24-1.el7_5.x86_64 2/23
Installing : cloudera-manager-daemons-6.3.0-1281944.el7.x86_64 3/23
Installing : python-psycopg2-2.5.1-4.el7.x86_64 4/23
Installing : libkadm5-1.15.1-37.el7_7.2.x86_64 5/23
Installing : spax-1.5.2-13.el7.x86_64 6/23
Installing : libsepol-devel-2.5-10.el7.x86_64 7/23
Installing : 1:mod_ssl-2.4.6-90.el7.centos.x86_64 8/23
Installing : 2:postfix-2.10.1-7.el7.x86_64 9/23
Installing : zlib-devel-1.2.7-18.el7.x86_64 10/23
Installing : libverto-devel-0.2.5-4.el7.x86_64 11/23
Installing : pcre-devel-8.32-17.el7.x86_64 12/23
Installing : libselinux-devel-2.5-14.1.el7.x86_64 13/23
Installing : MySQL-python-1.2.5-1.el7.x86_64 14/23
Installing : libcom_err-devel-1.42.9-16.el7.x86_64 15/23
Installing : keyutils-libs-devel-1.5.8-3.el7.x86_64 16/23
Installing : krb5-devel-1.15.1-37.el7_7.2.x86_64 17/23
Installing : 1:openssl-devel-1.0.2k-19.el7.x86_64 18/23
Installing : redhat-lsb-submod-security-4.1-27.el7.centos.1.x86_64 19/23
Installing : redhat-lsb-core-4.1-27.el7.centos.1.x86_64 20/23
Installing : cloudera-manager-agent-6.3.0-1281944.el7.x86_64 21/23
Created symlink from /etc/systemd/system/multi-user.target.wants/cloudera-scm-agent.service to /usr/lib/systemd/system/cloudera-scm-agent.service.
Created symlink from /etc/systemd/system/multi-user.target.wants/supervisord.service to /usr/lib/systemd/system/supervisord.service.
Installing : cloudera-manager-server-6.3.0-1281944.el7.x86_64 22/23
Created symlink from /etc/systemd/system/multi-user.target.wants/cloudera-scm-server.service to /usr/lib/systemd/system/cloudera-scm-server.service.
Cleanup : krb5-libs-1.15.1-37.el7_6.x86_64 23/23
Verifying : redhat-lsb-submod-security-4.1-27.el7.centos.1.x86_64 1/23
Verifying : keyutils-libs-devel-1.5.8-3.el7.x86_64 2/23
Verifying : krb5-devel-1.15.1-37.el7_7.2.x86_64 3/23
Verifying : libcom_err-devel-1.42.9-16.el7.x86_64 4/23
Verifying : MySQL-python-1.2.5-1.el7.x86_64 5/23
Verifying : pcre-devel-8.32-17.el7.x86_64 6/23
Verifying : libverto-devel-0.2.5-4.el7.x86_64 7/23
Verifying : zlib-devel-1.2.7-18.el7.x86_64 8/23
Verifying : 2:postfix-2.10.1-7.el7.x86_64 9/23
Verifying : cloudera-manager-daemons-6.3.0-1281944.el7.x86_64 10/23
Verifying : cloudera-manager-server-6.3.0-1281944.el7.x86_64 11/23
Verifying : cloudera-manager-agent-6.3.0-1281944.el7.x86_64 12/23
Verifying : krb5-libs-1.15.1-37.el7_7.2.x86_64 13/23
Verifying : python-psycopg2-2.5.1-4.el7.x86_64 14/23
Verifying : 1:mod_ssl-2.4.6-90.el7.centos.x86_64 15/23
Verifying : 1:openssl-devel-1.0.2k-19.el7.x86_64 16/23
Verifying : libsepol-devel-2.5-10.el7.x86_64 17/23
Verifying : spax-1.5.2-13.el7.x86_64 18/23
Verifying : libkadm5-1.15.1-37.el7_7.2.x86_64 19/23
Verifying : libselinux-devel-2.5-14.1.el7.x86_64 20/23
Verifying : redhat-lsb-core-4.1-27.el7.centos.1.x86_64 21/23
Verifying : postgresql-libs-9.2.24-1.el7_5.x86_64 22/23
Verifying : krb5-libs-1.15.1-37.el7_6.x86_64 23/23
Installed:
cloudera-manager-agent.x86_64 0:6.3.0-1281944.el7 cloudera-manager-daemons.x86_64 0:6.3.0-1281944.el7 cloudera-manager-server.x86_64 0:6.3.0-1281944.el7
Dependency Installed:
MySQL-python.x86_64 0:1.2.5-1.el7 keyutils-libs-devel.x86_64 0:1.5.8-3.el7 krb5-devel.x86_64 0:1.15.1-37.el7_7.2 libcom_err-devel.x86_64 0:1.42.9-16.el7
libkadm5.x86_64 0:1.15.1-37.el7_7.2 libselinux-devel.x86_64 0:2.5-14.1.el7 libsepol-devel.x86_64 0:2.5-10.el7 libverto-devel.x86_64 0:0.2.5-4.el7
mod_ssl.x86_64 1:2.4.6-90.el7.centos openssl-devel.x86_64 1:1.0.2k-19.el7 pcre-devel.x86_64 0:8.32-17.el7 postfix.x86_64 2:2.10.1-7.el7
postgresql-libs.x86_64 0:9.2.24-1.el7_5 python-psycopg2.x86_64 0:2.5.1-4.el7 redhat-lsb-core.x86_64 0:4.1-27.el7.centos.1 redhat-lsb-submod-security.x86_64 0:4.1-27.el7.centos.1
spax.x86_64 0:1.5.2-13.el7 zlib-devel.x86_64 0:1.2.7-18.el7
Dependency Updated:
krb5-libs.x86_64 0:1.15.1-37.el7_7.2
Complete!
[hadoop@manager ~]$
配置本地Parcel存儲庫
[root@manager ~]# mv /home/hadoop/Downloads/cloudera/parcel/* /opt/cloudera/parcel-repo
[root@manager ~]# cd /opt/cloudera/parcel-repo
[root@manager parcel-repo]# pwd
/opt/cloudera/parcel-repo
[root@manager ~]# sha1sum CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel | awk '{ print $1 }' > CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.sha
[root@manager ~]# chown -R cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo/*
初始化scm庫
[root@manager parcel-repo]# /opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm 123456Aa.
JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera
Verifying that we can write to /etc/cloudera-scm-server
Creating SCM configuration file in /etc/cloudera-scm-server
Executing: /usr/java/jdk1.8.0_181-cloudera/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:/opt/cloudera/cm/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.
Thu Oct 31 00:33:50 CST 2019 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
[ main] DbCommandExecutor INFO Successfully connected to database.
All done, your SCM database is configured correctly!
啓動CM
[root@manager ~]#
systemctl stop cloudera-scm-server \
&& systemctl start cloudera-scm-server \
&& sleep 2 \
&& tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log | grep "INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server"
1分鐘等待後，如果出現：
2019-10-31 00:39:00,113 INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server.
則代表CM啓動成功了
訪問---》【http://192.168.88.31:7180】
賬號密碼爲admin/admin
在整個啓動過程中，可以新開一個窗口，然後tail -f看看日誌有啥錯誤
[root@manager ~]# tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log
如果在中間出現問題，要先
[root@manager ~]# systemctl stop cloudera-scm-server
再運行上面的啓動命令。數據庫密碼【123456Aa.】
初始化CM
如果在中間報異常了，那麼基本上就需要做回滾和清理動作了（還不如重新來個新的操作系統來得簡單）
【一】、Cluster Basics
給個人看得懂的集羣名字即可
【二】、Accept JDK License
接受，不接受走不下去
【三】、Enter Login Credentials
root用戶可以安裝（sudo用戶？？？？）安裝 Cloudera 包需要有主機的 root 訪問權限。此安裝程序將通過 SSH 連接到您的主機，然後直接以 root 用戶身份登錄，或者以另一個具有變爲 root 用戶的無密碼 sudo/pbrun 權限的用戶身份登錄。
【四】、Specify Hosts
192.168.88.[31-34]，可以通過這樣的表達式來搜索多臺機器
【五】、Select Repository
填入本地的yum庫---》http://192.168.88.31/cloudera-manager/
CDH-6.3.0-1.cdh6.3.0.p0.1279813，這裏我們需要明確我們自己安裝的版本，因爲系統會把比較新的版本都顯示出來，如果選擇錯了，就是從網絡下載了，那個時間就超長了
【六】、Install Agents
耗時10、20分鐘的樣子，由於頁面是js寫的，因此不要刷新瀏覽器。如果出錯了，根據錯誤修正接節點的問題，然後retry。如果我們在這裏放棄了，基本上這個操作系統也就廢掉了（刪除清理，估計也是非常麻煩的）
【七】、Install Parcels
耗時幾個小時，distubuted哪裏會刷新傳遞分發的速度 xxM/s。這個階段磁盤佔用率非常高。頁面上是有簡單的進度條顯示整體進度的。沒有到100%，continue的按鈕是點擊不了的。
【八】、Inspect Cluster
點擊Inspect Network Performance按鈕
點擊Inspect Hosts按鈕，可以參考這裏：https://blog.csdn.net/qq_39680564/article/details/100013275，相關的報錯彙總
主機健康檢查、I understand the risks, let me continue with cluster setup.，點選這個走下去。
各個組件的驗證
【選擇安裝的服務】
impala、hdfs、hbase等組件
【自行分配角色至主機】
這裏官方文檔中，有按照規模建議指定機器的分配方式
【測試數據庫鏈接】
這裏的數據庫實例名，用戶名、密碼都是上面的腳本創建的，這裏是直接使用已經創建好的庫，不能亂填的，要按照上面的創建mysql庫的腳本來進行填寫。
常見報錯1：JDBC driver cannot be found. Unable to find the JDBC database jar on host
[hadoop@manager mysql-jdbc]$
sudo mkdir -p /usr/share/java/ \
&& cd /home/hadoop/Downloads/cloudera/mysql-jdbc/;tar -zxvf mysql-connector-java-5.1.47.tar.gz \
&& sudo cp /home/hadoop/Downloads/cloudera/mysql-jdbc/mysql-connector-java-5.1.47/mysql-connector-java-5.1.47-bin.jar /usr/share/java/mysql-connector-java.jar
常見報錯2:Cloudera 建議將 /proc/sys/vm/swappiness 設置爲最大值 10
臨時生效：
sysctl vm.swappiness=10 && cat /proc/sys/vm/swappiness
echo 'vm.swappiness=10'>> /etc/sysctl.conf
永久生效：
echo 'vm.swappiness=10'>> /etc/sysctl.conf
常見報錯3:已啓用透明大頁面壓縮，可能會導致重大性能問題。
臨時生效：
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo 'echo never > /sys/kernel/mm/transparent_hugepage/defrag' >> /etc/rc.local
echo 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' >> /etc/rc.local
永久生效：
echo 'echo never > /sys/kernel/mm/transparent_hugepage/defrag' >> /etc/rc.local
echo 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' >> /etc/rc.local
【配置實例的安裝位置】
kudu的配置參數
Kudu Master WAL Directory:/kudumaster/wal
Kudu Master Data Directories:/kudumaster/data
Kudu Tablet Server WAL Directory:/kudutablet/wal
Kudu Tablet Server Data Directories:/kudutablet/data
HDFS 塊大小 128
接受的 DataNode 失敗的卷:0
DataNode 數據目錄 /dfs/dn
NameNode 數據目錄 /dfs/nn
HDFS 檢查點目錄 /dfs/snn
Host Monitor 存儲目錄 /var/lib/cloudera-host-monitor
Service Monitor 存儲目錄 /var/lib/cloudera-service-monitor
NodeManager 本地目錄 /yarn/nm
數據目錄 /var/lib/zookeeper
事務日誌目錄 /var/lib/zookeeper
【安裝服務】
各種錯誤，可以參照：
https://www.cnblogs.com/felixzh/p/9082344.html
https://blog.csdn.net/sardtass/article/details/93723610
來進行問題解決。解決後，不停重試安裝即可。
【安裝完成成功提示】
關閉所有服務：
在頁面Cluster集羣的cluster1後面有個下拉三角，點擊出下拉列表後，點擊stop，這個stop也就是把集羣中的所有服務，全部定製
在Cloudera Management Service後面有個下拉三角，點擊出下拉列表後，點擊stop，這個stop也就是把集羣中的所有服務，全部定製
僅僅在主機上運行
[root@master cloudera]# service cloudera-scm-server stop
Redirecting to /bin/systemctl stop cloudera-scm-server.service
每臺機器都要運行這個指令
[root@master cloudera]# service cloudera-scm-agent stop
Redirecting to /bin/systemctl stop cloudera-scm-agent.service
[root@manager ~]#
service cloudera-scm-server stop \
&& service cloudera-scm-agent stop
重新啓動整個集羣:
service cloudera-scm-server start
所有都需要運行service cloudera-scm-agent start
先需要啓動CM
[root@manager ~]#
systemctl stop cloudera-scm-server \
&& systemctl start cloudera-scm-server \
&& sleep 2 \
&& tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log | grep "INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server"
[root@manager ~]#
service cloudera-scm-server restart \
&& service cloudera-scm-agent restart
在Cloudera Management Service後面有個下拉三角，點擊出下拉列表後，點擊stop，這個start也就是把集羣中的所有服務，全部定製
在頁面Cluster集羣的cluster1後面有個下拉三角，點擊出下拉列表後，點擊stop，這個start也就是把集羣中的所有服務，全部定製
service cloudera-scm-server status
service cloudera-scm-agent status
[root@manager ~]#
service cloudera-scm-server status \
&& service cloudera-scm-agent status