CDH 6.3.2 安裝（一）

1、安裝vim

yum -y install vim

2、主機名配置

hostnamectl set-hostname cdh1

vim /etc/sysconfig/network

NETWORKING=yes
HOSTNAME=cdh1

cat /etc/sysconfig/network

3、網絡配置

vim /etc/hosts

yum -y install lrzsz

cat /etc/hosts

cp /etc/hosts /etc/hosts.template

在本地編輯一個hosts文件，上傳至 /etc/ 目錄下

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.232.131 cdh1
192.168.232.132 cdh2
192.168.232.133 cdh3
192.168.232.134 cdh4
192.168.232.135 cdh5
192.168.232.136 cdh6

在Windows下的host文件追加

# cdh company cluster
# 127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
# ::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.232.131 cdh1
192.168.232.132 cdh2
192.168.232.133 cdh3
192.168.232.134 cdh4
192.168.232.135 cdh5
192.168.232.136 cdh6

4、禁用ipv6網絡協議

CentOS7 mini版本沒有安裝 ipconfig模塊，需要用戶手動安裝

yum -y install net-tools

安裝pstree模塊

yum -y install psmisc

樹形圖顯示父子進程：

pstree -aup

禁用IPV6地址

echo -e "
# 禁用整個系統所有接口的IPv6
net.ipv6.conf.all.disable_ipv6 = 1
# 禁用某一個指定接口的IPv6（eth0，lo）
net.ipv6.conf.lo.disable_ipv6 = 1
net.ipv6.conf.eth0.disable_ipv6 = 1
" >> /etc/sysctl.conf

重啓sysctl：

sudo sysctl -p

再次查看網絡協議

ifconfig

查看防火牆狀態

systemctl status firewalld.service

安裝iptables-services模塊

yum -y install iptables-services

關閉防火牆

service iptables stop

查看狀態

service iptables status

關閉防火牆，禁止開機自啓

systemctl stop firewalld.service

systemctl disable firewalld.service

關閉SELINUX

setenforce 0

vim /etc/selinux/config

把 SELINUX=enforcing 改成 SELINUX=disable

重啓服務器

reboot

下載 oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm

mkdir /usr/java

cd /usr/java/

將下載的 oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm 上傳到 /usr/java目錄下，並解壓

rpm -ivh oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm

刪除文件 oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm

rm -rf oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm

創建軟鏈接

ln -s /usr/java/jdk1.8.0_181-cloudera /usr/java/jdk1.8

查看軟鏈接

配置環境變量 /etc/profile

cp /etc/profile /etc/profile.template

echo '
export JAVA_HOME=/usr/java/jdk1.8
export MYSQL_HOME=/usr/local/mysql
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export PATH=${JAVA_HOME}/bin:${PATH}:${CLASSPATH}:${MYSQL_HOME}/bin
' >> /etc/profile

tail -n 10 /etc/profile

source /etc/profile

java -version

which java

安裝 NTP 模塊

yum -y install ntp

設置開機自啓

chkconfig ntpd on

查看ntp是否開機自啓

systemctl is-enabled ntpd

設置 NTP 服務

ntpdate -u ntp.aliyun.com

半個小時同步一次（非必須）

crontab -e

0,30 * * * * ntpdate -u ntp.aliyun.com

或者將命令添加到 /etc/crontab文件中（非必須）

echo '0,30 * * * * root ntpdate -u ntp.aliyun.com' >> /etc/crontab
cat /etc/crontab

查看時間是否準確

date

安裝MySQL數據庫

CDH官方安裝文檔：https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/cm_ig_mysql.html#cmig_topic_5_5

cd /usr/local

使用SFTP工具（大文件傳輸速度更快更穩定），上傳MySQL包

cd /usr/local/

先解壓 tar 文件

tar -xvf mysql-5.7.29-linux-glibc2.12-x86_64.tar

再解壓 tar.gz 文件

tar -zxvf mysql-5.7.29-linux-glibc2.12-x86_64.tar.gz

刪除 tar文件和 gz文件

rm -rf /usr/local/*.tar /usr/local/*.gz

重命名 mysql-5.7.29-linux-glibc2.12-x86_64 爲 mysql

mv mysql-5.7.29-linux-glibc2.12-x86_64/ mysql

檢查是否有老版本

如果沒有任何提示信息則表示Linux上沒有安裝mysql，如果有會顯示mysql的版本

rpm -qa | grep mysql

創建mysql用戶組和用戶並修改權限

groupadd mysql
useradd -r -g mysql mysql

創建數據目錄並賦權

mkdir -p /data/mysql
chown mysql:mysql -R /data/mysql

查看mysql配置信息

cat /etc/my.cnf

配置文件前，先關閉MySQL服務

sudo systemctl stop mysqld

配置 /etc/my.cnf 文件

[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
transaction-isolation = READ-COMMITTED
# Disabling symbolic-links is recommended to prevent assorted security risks;
# to do so, uncomment this line:
symbolic-links = 0

key_buffer_size = 32M
max_allowed_packet = 32M
thread_stack = 256K
thread_cache_size = 64
query_cache_limit = 8M
query_cache_size = 64M
query_cache_type = 1

max_connections = 1000
#expire_logs_days = 10
#max_binlog_size = 100M

#log_bin should be on a disk with enough free space.
#Replace '/var/lib/mysql/mysql_binary_log' with an appropriate path for your
#system and chown the specified folder to the mysql user.
log_bin=/var/lib/mysql/mysql_binary_log

#In later versions of MySQL, if you enable the binary log and do not set
#a server_id, MySQL will not start. The server_id must be unique within
#the replicating group.
server_id=1

binlog_format = mixed

read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M

# InnoDB settings
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit  = 2
innodb_log_buffer_size = 64M
innodb_buffer_pool_size = 4G
innodb_thread_concurrency = 8
innodb_flush_method = O_DIRECT
innodb_log_file_size = 512M

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

sql_mode=STRICT_ALL_TABLES

初始化數據庫，進入mysql的bin目錄

mv mysql-5.7.29-linux-glibc2.12-x86_64 mysql

cd /usr/local/mysql/bin/

初始化 MySQL

./mysqld --defaults-file=/etc/my.cnf --basedir=/usr/local/mysql --datadir=/data/mysql/
--socket=/data/mysql/mysql.sock log-error=/data/mysql/mysqld.log --pid-file=/data/mysql/mysql.pid --user=root  --initialize

查看臨時密碼

cat /data/mysql/mysql.err

啓動MySQL

service mysql start

ps -ef|grep mysql

修改MySQL密碼

cd /usr/local/mysql/bin/

./mysql -uroot -p

set password=PASSWORD('root');

將 /etc/profile 文件下載到本地，方便修改

sz /etc/profile

配置 MySQL環境變量

vim /etc/profile

將之前配置的Java環境變量註釋掉，重新配置Java和MySQL環境變量

# export JAVA_HOME=/usr/java/jdk1.8
# export CLASSPATH=.:$JAVA_HOMdE/lib/dt.jar:$JAVA_HOME/lib/tools.jar
# export PATH=$JAVA_HOME/bin:$PATH
JAVA_HOME=/usr/java/jdk1.8
CLASSPATH=.:$JAVA_HOMdE/lib/dt.jar:$JAVA_HOME/lib/tools.jar
MYSQL_HOME=/usr/local/mysql

PATH=$PATH:$JAVA_HOME/bin:$CLASSPATH:$MYSQL_HOME/bin

export JAVA_HOME PATH CLASSPATH MYSQL_HOME

設置PATH的語法都爲：

export PATH=$PATH:<PATH 1>:<PATH 2>:<PATH 3>:------:<PATH N>

重新加載配置文件

source /etc/profile

重新驗證環境變量

java -verion
mysql -uroot -proot

設置mysql開機自啓

首先查看開機自啓的程序

chkconfig --list

將mysql服務添加到開機自啓選項

把mysql的啓動腳本放到開機初始化目錄

cp /usr/local/mysql/support-files/mysql.server /etc/init.d/mysqld

對文件賦予執行權限

chmod +x /etc/init.d/mysqld 或 chmod 777 /etc/init.d/mysqld

增加mysqld服務

chkconfig --add mysqld

查詢mysqld服務情況

chkconfig --list mysqld

開啓運行級別語法（不需要執行）

chkconfig --level 345 mysqld on

重啓服務器驗證：reboot

給 root 用戶賦權

grant all privileges on *.* to root@'%' identified by 'root';

添加遠程訪問權限

uer mysql;
update user set host='%' where user = 'root';

發現報錯：ERROR 1062 (23000): Duplicate entry '%-root' for key 'PRIMARY'

查看 mysql.user 表的信息，發現 root 用戶已經有了遠程訪問的權限

select user,host from user;

刷新權限，重啓MySQL

flush privileges;

service mysql restart

創建大數據集羣所需要的數據庫

##DROP DATABASE scm;
##DROP USER scm;

CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;


GRANT ALL ON scm.* TO 'scm'@'%' IDENTIFIED BY 'scm';
GRANT ALL ON amon.* TO 'amon'@'%' IDENTIFIED BY 'amon';
GRANT ALL ON rman.* TO 'rman'@'%' IDENTIFIED BY 'rman';
GRANT ALL ON hue.* TO 'hue'@'%' IDENTIFIED BY 'hue';
GRANT ALL ON metastore.* TO 'hive'@'%' IDENTIFIED BY 'hive';
GRANT ALL ON sentry.* TO 'sentry'@'%' IDENTIFIED BY 'sentry';
GRANT ALL ON nav.* TO 'nav'@'%' IDENTIFIED BY 'nav';
GRANT ALL ON navms.* TO 'navms'@'%' IDENTIFIED BY 'navms';
GRANT ALL ON oozie.* TO 'oozie'@'%' IDENTIFIED BY 'oozie';

GRANT ALL ON *.* TO 'root'@'%' IDENTIFIED BY 'root';

flush privileges;

安裝 CDH 前先把需要的依賴包下載好

yum -y install httpd
yum -y install chkconfig
yum -y install python
yum -y install bind-utils
yum -y install psmisc
yum -y install libxslt
yum -y install zlib
yum -y install sqlite
yum -y install cyrus-sasl-plain
yum -y install cyrus-sasl-gssapi
yum -y install fuse
yum -y install portmap
yum -y install fuse-libs
yum -y install redhat-lsb

yum -y install lsb
yum -y install openssl-devel
yum -y install mod_ssl
yum -y install python-psycopg2
yum -y install MySQL-python
yum -y install postgresql-libs.x86_64

yum -y install gcc
yum -y install gcc-c++
yum -y install libxslt-devel python-devel

yum  -y install chkconfig python bind-utils psmisc libxslt zlib sqlite cyrus-sasl-plain cyrus-sasl-gssapi fuse fuse-libs redhat-lsb

yum -y install openssl-devel perl bind-utils libxslt cyrus-sasl-plain cyrus-sasl-gssapi portmap fuse-libs /lib/lsb/init-functions httpd mod_ssl python-psycopg2 MySQL-python

安裝HTTP服務

CM的管理方式是通過Web請求訪問管理的，需要在主節點上安裝HTTP服務，並且啓動開機自啓服務

yum -y install httpd
chkconfig httpd on
service httpd start

查看是否開機自啓

systemctl is-enabled httpd.service
systemctl is-active httpd

如果爲disable，則設置開機自啓

systemctl enable httpd.service

這裏還有一個官網推薦的Linux配置，直接粘上去就行了

echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /etc/rc.local 

sysctl vm.swappiness=10
echo 'vm.swappiness=10'>> /etc/sysctl.conf
cat /proc/sys/vm/swappiness

yum -y install krb5-workstation krb5-libs

至此，CDH 安裝前的環境變量已經配好了

接下來直接複製虛擬機安裝Cloudera Manager Server&Agent

CDH 6.3.2 安裝（一）：https://blog.csdn.net/qq262593421/article/details/105048233

CDH 6.3.2 安裝（二）：https://blog.csdn.net/qq262593421/article/details/105173259

CDH 6.3.2 安裝（一）

如何在低代碼平臺中引用 JavaScript ？

探究職業發展的關鍵：能力模型解讀

高效率使用windows

如何使用 JavaScript 獲取當前頁面幀率 FPS

工程款拖欠，農民工怎麼了？就得一直忍着委屈求全嗎？

HarmonyOS 實現下拉刷新，上拉加載更多

語音信號處理中的“窗函數”

智能決策新時代：可視化大屏是否能夠超越傳統白板？

解密Prompt系列28. LLM Agent之金融領域摸索：FinMem & FinAgent

分享幾個.NET開源的AI和LLM相關項目框架

spark上傳文件和追加文件到hdfs

spark遍歷hdfs目錄下所有文件

大數據集羣之hbase2.1.0高可用安裝配置

java讀取文件獲取文件名多線程移動文件

Linux在windows上共享目錄

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結