Centos7.3安裝Ambari2.4.2+HDP2.5.3搭建Hadoop集羣

前言

本文是以前輩文檔做基礎,配上了自己(藍色字體)安裝過程的一些理解和解決辦法

原文檔連接:https://www.yp14.cn/2016/12/09/Centos7-2%E5%AE%89%E8%A3%85Ambari2-4-2-HDP2-5-3%E6%90%AD%E5%BB%BAHadoop%E9%9B%86%E7%BE%A4/

官方搭建文檔:https://docs.hortonworks.com/HDPDocuments/Ambari-2.4.2.0/bk_ambari-installation/content/ch_Getting_Ready.html
離線搭建yum源,包下載列表:https://docs.hortonworks.com/HDPDocuments/Ambari-2.4.2.0/bk_ambari-installation/content/hdp_25_repositories.html

適合系統:RedHat 7 CentOS 7 Oracle Linux 7

本機環境:CentOS Linux release 7.3.1611 (Core

一、 環境準備


1、主機
172.19.80.1 SY-001.hadoop(這檯安裝ambari,作爲管理其它節點的主節點)
172.19.80.2 SY-002.hadoop
172.19.80.3 ambari-mysql

2、安裝基礎環境 (在主機SY-001.hadoop,SY-002.hadoop,SY-003.hadoop部署)
(1)安裝基本包
$ yum install openssh wget vim openssh-clients openssl gcc openssh-server mysql-connector-odbc -y
$ systemctl enable sshd && systemctl start sshd
(2)安裝mysql jdbc 驅動$ yum install mysql-connector-java -y
(3)安裝JDK版本要求:Oracle JDK 1.8 64-bit (minimum JDK 1.8.0_77) (default)
Oracle JDK 1.7 64-bit (minimum JDK 1.7_67)
OpenJDK 8 64-bit (not supported on SLES)
OpenJDK 7 64-bit (not supported on SLES)
$ mkdir /usr/local/src/Ambari/
$ wget http://download.oracle.com/otn-pub/java/jdk/8u112-b15/jdk-8u112-linux-x64.tar.gz?AuthParam=1480914946_a78f7ba2e70d2f0abed0b0af73c74214
(若下載連接無效,可在百度網盤下載,鏈接:http://pan.baidu.com/s/1c1Hv8o4 密碼:bbgx)
$ mkdir -p /opt/java/ && tar xf jdk-8u112-linux-x64.tar.gz -C /opt/java
$ vim /etc/profile
添加下面內容:
export JAVA_HOME=/opt/java/jdk1.8.0_112
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
PATH=$PATH:$HOME/bin:$JAVA_HOME/bin
$ source /etc/profile
(4)配置SSH免密碼登錄主節點裏root用戶登錄執行如下步驟
$ ssh-keygen
$ cd ~/.ssh/

$ cat id_rsa.pub >> authorized_keys

$ ssh root@172.19.80.2 cat ~/.ssh/id_rsa.pub>> authorized_keys

$ ssh root@172.19.80.3 cat ~/.ssh/id_rsa.pub>> authorized_keys
$ scp authorized_keys 172.19.80.2:/~/.ssh (把這個文件複製給其它兩臺)
$ scp authorized_keys 172.19.80.3:/~/.ssh
$ chmod 600 ~/.ssh/authorized_key
(5)開啓NTP服務

$ yum -y install ntp

$ systemctl is-enabled ntpd$ systemctl enable ntpd

$ systemctl start ntpd

(6)檢查DNS和NSCDambari在安裝時需要配置全域名,所以需要檢查DNS。爲了減輕DNS的負擔, 建議在節點裏用 Name Service Caching Daemon (NSCD)

$ cat /etc/hosts 172.19.80.1

SY-001.hadoop 172.19.80.2

SY-002.hadoop 172.19.80.3

ambari-mysql 172.19.80.3

(7)每臺節點裏配置FQDN,如下以主節點爲例

$ cat /etc/sysconfig/network

NETWORKING=yes

HOSTNAME=SY-001.hadoop

(8)關閉防火牆和SELinux

$ systemctl disable firewalld

$ systemctl stop firewalld臨時關閉,不用重啓機器:

$ setenforce 0

修改配置文件讓機器重啓也生效:

$ cat /etc/sysconfig/selinux

SELINUX=disabled

二、搭建數據庫(在主機ambari-mysql上部署)

Ambari安裝會將安裝等信息寫入數據庫,建議使用自己安裝的Mysql數據庫,也可以不安裝而使用默認數據庫PostgreSQL

版本要求:
- PostgreSQL 8
- PostgreSQL 9.1.13+,9.3, 9.4***
- MariaDB 10*
- MySQL 5.6
- Oracle 11gr2
- Oracle 12c**

按你係統來選擇源:http://downloads.mariadb.org/mariadb/repositories/

Here is your custom MariaDB YUM repository entry for CentOS. Copy and paste it into a file under /etc/yum.repos.d/ (we suggest naming the file MariaDB.repo or something similar).

# MariaDB 10.1 CentOS repository list - created 2016-12-09 06:03 UTC
# http://downloads.mariadb.org/mariadb/repositories/
[mariadb]
name = MariaDB
baseurl = http://yum.mariadb.org/10.1/centos7-amd64
gpgkey=https://yum.mariadb.org/RPM-GPG-KEY-MariaDB
gpgcheck=1
After the file is in place, install MariaDB with:

$ sudo yum install MariaDB-server MariaDB-client
If you haven't already accepted the MariaDB GPG key, you will be prompted to do so. See "Installing MariaDB with yum" for detailed information.
-------------------------------數據庫登陸解決方法----------------------------------------
備註:如果部署完數據庫,root登陸時報如下錯誤,可以這樣解決

[root@Centos7T ~]#mysql -uroot -p

Enter password:

ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)

解決方法:#systemctl stop mariadb#systemctl set-environmentMYSQLD_OPTS="--skip-grant-tables"(跳過密碼登陸)#systemctl start mariadb && systemctl status mariadb(啓動,查看啓動狀態)

#mysql -u root

Welcome to the MariaDBmonitor.  Commands end with ; or \g.

Your MariaDB connection id is 14

Server version: 10.2.8-MariaDB MariaDB Server

 

Copyright (c) 2000, 2017, Oracle, MariaDB Corporation Ab and others.

 

Type 'help;' or '\h' for help. Type '\c' to clear the currentinput statement.

 

MariaDB [(none)]>

MariaDB [(none)]> USE mysql;

Reading table information for completion of table and column names

You can turn off this feature to get a quicker startup with -A

 

Database changed

MariaDB [mysql]> UPDATE user SET password=PASSWORD('NewPassword') WHERE User='root' AND Host = 'localhost';

Query OK, 1 row affected (0.01 sec)

Rows matched: 1  Changed: 1  Warnings: 0

 

MariaDB [mysql]> FLUSHPRIVILEGES;

Query OK, 0 rows affected (0.00 sec)

 

MariaDB [mysql]> exit;

Bye

還沒解決完,要取消跳過密碼登陸,否則等於沒改一樣

systemctl stop mariadb

#systemctl unset-environment MYSQLD_OPTS(取消跳過密碼登陸)

#systemctl start mariadb && systemctl status mariadb

至此就可以用mysql -u root -p,輸入剛設置NewPassword(密碼隨自己改)的密碼登陸了

----------------------------------數據庫登陸解決方法-----------------------------------------

安裝完成後創建ambari數據庫及用戶,登錄root用戶執行下面語句:

create database ambari character set utf8 ;

CREATE USER 'ambari'@'%'IDENTIFIED BY 'Ambari-123';

GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'%';

FLUSH PRIVILEGES;

如果要安裝Hive,再創建Hive數據庫和用戶 再執行下面的語句:

create database hive character set utf8 ;

CREATE USER 'hive'@'%'IDENTIFIED BY 'Hive-123';

GRANT ALL PRIVILEGES ON *.* TO 'hive'@'%';

FLUSH PRIVILEGES;

如果要安裝Oozie,再創建Oozie數據庫和用戶 再執行下面的語句:

create database oozie character set utf8 ;

CREATE USER 'oozie'@'%'IDENTIFIED BY 'Oozie-123';

GRANT ALL PRIVILEGES ON *.* TO 'oozie'@'%';

FLUSH PRIVILEGES;

三、製作本地源 (在SY-001.hadoop部署)

下載 Ambari HDP HDP-UTILS 源包(文件很大,需要很長下載時間,請耐心等待。。。)
$ cd /usr/local/src/Ambari/
$ wget http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.4.2.0/ambari-2.4.2.0-centos7.tar.gz
$ wget http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.5.3.0/HDP-2.5.3.0-centos7-rpm.tar.gz
$ wget http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.21/repos/centos7/HDP-UTILS-1.1.0.21-centos7.tar.gz

安裝本地源製作相關工具
$ yum install yum-utils createrepo -y

安裝httpd服務
$ yum install httpd -y
$ systemctl enable httpd && systemctl start httpd
$ cd /var/www/html/ && mkdir ambari
$ cd /usr/local/src/Ambari/
$ tar xf ambari-2.4.2.0-centos7.tar.gz -C /var/www/html/ambari/
$ tar xf HDP-2.5.3.0-centos7-rpm.tar.gz -C /var/www/html/ambari/
$ tar xf HDP-UTILS-1.1.0.21-centos7.tar.gz -C /var/www/html/ambari/

驗證httd網站是否可用,可以使用links 命令或者瀏覽器直接訪問下面的地址:
$ links http://172.19.80.1/ambari/
結果是看到剛纔解壓的目錄

配置ambari、HDP、HDP-UTILS的本地源
首先下載上面資源列表中的相應repo文件,修改其中的URL爲本地的地址,相關配置如下:
$ cd /etc/yum.repos.d/
$ cat ambari.repo
#VERSION_NUMBER=2.4.2.0-136
[Updates-ambari-2.4.2.0]
name=ambari-2.4.2.0 - Updates
baseurl=http://172.19.80.1/ambari/AMBARI-2.4.2.0/centos7/2.4.2.0-136
gpgcheck=1
gpgkey=http://172.19.80.1/ambari/AMBARI-2.4.2.0/centos7/2.4.2.0-136/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

$ cat hdp.repo
#VERSION_NUMBER=2.5.3.0
[HDP-2.5.3.0]
name=HDP Version - HDP-2.5.3.0
baseurl=http://172.19.80.1/ambari/HDP/centos7/2.x/updates/2.5.3.0
gpgcheck=1
gpgkey=http://172.19.80.1/ambari/HDP/centos7/2.x/updates/2.5.3.0/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

[HDP-UTILS-1.1.0.21]
name=HDP Utils Version - HDP-UTILS-1.1.0.21
baseurl=http://172.19.80.1/ambari/HDP-UTILS-1.1.0.21/repos/centos7
gpgcheck=1
gpgkey=http://172.19.80.1/ambari/HDP/centos7/2.x/updates/2.5.0.0/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

備註:紅色標記的路徑需要根據自己的實際解壓路徑來,否則makecache不能成功
$ yum clean all
$ yum makecache
$ yum repolist

四、安裝Ambari (在SY-001.hadoop部署)

安裝Ambari
$ yum install ambari-server -y

配置Ambari
$ ambari-server setup

下面是配置執行流程,按照提示操作
(1) 提示是否自定義設置。輸入:y
Customize user account for ambari-server daemon [y/n] (n)? y

(2)ambari-server 賬號。輸入:ambari
Enter user account for ambari-server daemon (root):ambari
Adjusting ambari-server permissions and ownership...

(3)設置JDK。輸入:3
Checking JDK...
Do you want to change Oracle JDK [y/n] (n)? y
[] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[] Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7
[] Custom JDK
==============================================================================
Enter choice (1): 3

(4)如果上面選擇3自定義JDK,則需要設置JAVA_HOME。輸入:/opt/java/jdk1.8.0_112
WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts.
WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts.
Path to JAVA_HOME: /opt/java/jdk1.8.0_112
Validating JDK on Ambari Server...done.
Completing setup...

(5)數據庫配置。選擇:y
Configuring database...
Enter advanced database configuration [y/n] (n)? y

(6)選擇數據庫類型。輸入:3
Configuring database...
==============================================================================
Choose one of the following options:
[1] - PostgreSQL (Embedded)
[2] - Oracle
[3] - MySQL
[4] - PostgreSQL
[5] - Microsoft SQL Server (Tech Preview)
[6] - SQL Anywhere
==============================================================================
Enter choice (3): 3

(7)設置數據庫的具體配置信息,根據實際情況輸入,如果和括號內相同,則可以直接回車。
Hostname (localhost):(是ambari-mysql安裝了數據庫那臺的主機名)
Port (3306):
Database name (ambari):
Username (ambari):
Enter Database Password (Ambari-123):

(8)將Ambari數據庫腳本導入到數據庫
如果使用自己定義的數據庫,必須在啓動Ambari服務之前導入Ambari的sql腳本

用Ambari用戶(上面設置的用戶)登錄mysql
mysql -u ambari -p
use ambari
source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
備註,如果此命令不能執行,就直接將sy-001.hadoop主機的/var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql,
這個目錄下的文件發送給
ambari-mysql主機沒這個目錄,就mkdir -p 建立同樣的目錄
(9)啓動Amabri
$ ambari-server start
(10)成功啓動後在瀏覽器輸入Ambari地址:http://主機名:8080

五、使用Ambari界面安裝Hadoop、Hbase、Log Search服務

1、出現登錄界面,默認管理員賬戶登錄, 賬戶:admin 密碼:admin


2、登錄成功後出現下面的界面,至此Ambari的安裝成功

到這裏,ambari就裝完了,剩下的配置可以去看原文檔(文章開頭有地址),也需要根據業務需求再裝,就不詳細說了

發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章