openshift部署

Copy of 聯城科技paas平臺離線安裝文檔

一.環境說明

Ip列表	Host	備註
192.168.12.76	master1.qingyuanos.com	master/node 角色
192.168.12.77	master2.qingyuanos.com	master/node 角色
192.168.12.78	master3.qingyuanos.com	master/node 角色
192.168.12.5	node1.qingyuanos.com	node角色
192.168.12.6	node2.qingyuanos.com	node角色
192.168.12.7	node3.qingyuanos.com	node角色

二.基礎環境的配置

1.配置主機名及hosts

hostnamectl set-hostname master1.qingyuanos.com
hostnamectl set-hostname master2.qingyuanos.com
hostnamectl set-hostname master3.qingyuanos.com
hostnamectl set-hostname node1.qingyuanos.com
hostnamectl set-hostname node2.qingyuanos.com
hostnamectl set-hostname node3.qingyuanos.com

hosts 文件配置如下 注：所有主機都修改hosts文件

[root@master3 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.12.78  master1.qingyuanos.com  master.qingyuanos.com 
192.168.12.77  master2.qingyuanos.com
192.168.12.76  master3.qingyuanos.com
192.168.12.5   node1.qingyuanos.com
192.168.12.6   node1.qingyuanos.com
192.168.12.7   node1.qingyuanos.com

2.配置面密鑰登陸（在master3上配置）

[root@master3 ~]# ssh-keygen    #一路回車
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): 
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
36:90:90:0f:24:86:29:9d:aa:dd:24:1c:f3:c9:44:46 [email protected]
The key's randomart image is:
+--[ RSA 2048]----+
| +o=E.           |
|+.=o+. .         |
|.o * +o          |
|. o = ..         |
|.. +    S        |
|. . .  . .       |
|                 |
|                 |
|                 |
[root@master3 ~]# ssh-copy-id 192.168.12.76 # yes ，然後輸入密碼
[root@master3 ~]# ssh-copy-id 192.168.12.77
[root@master3 ~]# ssh-copy-id 192.168.12.78
[root@master3 ~]# ssh-copy-id 192.168.12.5
[root@master3 ~]# ssh-copy-id 192.168.12.6
[root@master3 ~]# ssh-copy-id 192.168.12.7

3.安裝httpd 做本地yum源（以下爲搭建paas平臺的所有包）

進入http_rpm 目錄

[root@master3 http_rpm]# rpm -ivh *.rpm

修改http端口（修改42行，改爲81端口）

[root@master3 oc]# grep 81 -n  /etc/httpd/conf/httpd.conf 
42:Listen 81
[root@master3 oc]# httpd

將本地源的tar包openshift-reop-3.9.tar.gz 拷貝到節點，並解壓到http根目錄下

[root@master3 oc]# tar zxf openshift-v3.6.tar.gz  -C /var/www/html/

備份原repo文件並指定本地源

[root@master3 yum.repos.d]# mkdir /etc/yum.repos.d/repo.bak
[root@master3 yum.repos.d]# mv /etc/yum.repos.d/*.repo /etc/yum.repos.d/repo.bak/
[root@master3 yum.repos.d]# cat openshift.repo 
[centos-openshift-origin]
name=CentOS OpenShift Origin
baseurl=http://192.168.12.78:81/openshift-v3.6
enabled=1
gpgcheck=0
gpgkey=file:///etc/pki/rpm-gpg/openshift-ansible-CentOS-SIG-PaaS

此時需要爲防火牆添加一個81端口，防止跑腳本的時候防火牆啓動以後無法訪問本地源

[root@master3 yum.repos.d]# iptables -A INPUT -p tcp -m state --state NEW -m tcp --dport 81 -j ACCEPT

4.安裝基礎依賴包

所有節點執行
執行前先檢查有無NetworkManager,有的情況下

yum install wget git net-tools bind-utils iptables-services bridge-utils bash-completion -y

否則

yum install wget git net-tools bind-utils iptables-services bridge-utils bash-completion  NetworkManager  -y

啓動服務（所有節點執行）

#systemctl enable  NetworkManager; systemctl start  NetworkManager

三.安裝oc

1.安裝docker  （所有節點執行）

[root@master3 yum.repos.d]# yum -y install  docker

2.load  docekr鏡像（所有節點執行）

[root@node1 ~]# tar zxf origin-images-3.6.tar.gz 
[root@node1 ~]# cd origin-images-3.6
[root@node1 origin-images-3.6]# ls
origin-deployer.tar        origin-docker-registry.tar  origin-pod.tar
origin-docker-builder.tar  origin-haproxy-router.tar   origin-sti-builder.tar
[root@node1 origin-images-3.6]#  for i in `ls ` ;do docker load -i $i ;done
[root@node3 ~]# docker load -i kubernetes.tar.gz

3.安裝配置ansible（在做無密鑰登陸的節點上安裝）

[root@master3 oc]# yum -y install ansible
[root@master3 oc]# tar zxf openshift-ansible-3.6wanda.tar.gz 
[root@master3 oc]# cd openshift-ansible-openshift-ansible-3.6.173.0.63-1/

修改 ./roles/openshift_repos/templates/CentOS-OpenShift-Origin36.repo.j2
配置文件
將內容替換爲以下，注意修改IP地址爲http節點的IP地址

[root@master3 openshift-ansible-openshift-ansible-3.6.173.0.63-1]# cat  ./roles/openshift_repos/templates/CentOS-OpenShift-Origin36.repo.j2
[centos-openshift-origin]
name=CentOS OpenShift Origin
baseurl=http://192.168.12.78:81/openshift-v3.6
enabled=1
gpgcheck=0

4.配置hosts文件

# Create an OSEv3 group that contains the masters and nodes groups
[OSEv3:children]
masters
nodes
etcd
 
# Set variables common for all OSEv3 hosts
[OSEv3:vars]
# SSH user, this user should allow ssh based auth without requiring a password
ansible_ssh_user=root
 
# If ansible_ssh_user is not root, ansible_become must be set to true
#ansible_become=true
 
deployment_type=origin

 
# uncomment the following to enable htpasswd authentication; defaults to DenyAllPasswordIdentityProvider
 
 
openshift_master_cluster_method = native
openshift_rolling_restart_mode = services
os_sdn_network_plugin_name=redhat/openshift-ovs-multitenant
openshift_master_portal_net=172.30.0.0/16
openshift_node_proxy_mode=iptables
osm_cluster_network_cidr=10.128.0.0/14
osm_host_subnet_length=9
openshift_disable_check=memory_availability,disk_availability,docker_storage,docker_storage_driver,docker_image_availability,package_version,package_availability,package_update

#openshift_router_selector='region=infra'
#openshift_registry_selector='region=infra'
openshift_use_openshift_sdn=true

openshift_master_default_subdomain=qingyuanos.com
openshift_master_cluster_method=native
openshift_master_cluster_hostname=master.qingyuanos.com
openshift_master_cluster_public_hostname=master.qingyuanos.com
openshift_clock_enabled=true
openshift_public_ip=192.168.12.76
#openshift_master_ca_certificate={'certfile': '/root/openshift-ansible/custom_ca/ca.crt', 'keyfile': '/root/openshift-ansible/custom_ca/ca.key'}

# host group for masters
[masters]
master1.qingyuanos.com ansible_host=192.168.12.76
master2.qingyuanos.com ansible_host=192.168.12.77
master3.qingyuanos.com ansible_host=192.168.12.78

[etcd]
master1.qingyuanos.com
master2.qingyuanos.com
master3.qingyuanos.com
# Specify load balancer host
# host group for nodes, includes region info
[nodes]
master1.qingyuanos.com  ansible_host=192.168.12.76  openshift_node_labels="{'region': 'infra', 'zone': 'default'}" openshift_schedulable=true
master2.qingyuanos.com  ansible_host=192.168.12.77  openshift_node_labels="{'region': 'infra', 'zone': 'default'}" openshift_schedulable=true
master3.qingyuanos.com  ansible_host=192.168.12.78  openshift_node_labels="{'region': 'infra', 'zone': 'default'}" openshift_schedulable=true
node1.qingyuanos.com ansible_host=192.168.12.5 openshift_node_labels="{'region': 'primary', 'zone': 'default'}"
node2.qingyuanos.com ansible_host=192.168.12.6  openshift_node_labels="{'region': 'primary', 'zone': 'east'}"

四.配置dns

如果route 和鏡像倉庫起啓動的情況下部署，配置dnsmasq服務在所有節點

[root@master origin-images-3.9]# cat /etc/dnsmasq.d/origin-dns.conf 
no-resolv
domain-needed
no-negcache
max-cache-ttl=1
enable-dbus
dns-forward-max=5000
cache-size=5000
bind-dynamic
except-interface=lo

listen-address=172.28.90.84 #修改自定義域名/本地ip地址
address=/.qyos.com/192.168.1.121  #修改自定義域名/本地外網ip地址
address=/docker-registry.default.svc/172.30.136.106 #私有倉庫的IP地址 通過oc get svc 進行查看
# End of config

啓動dns並設置爲開機自啓

[root@master3 ~]# systemctl enable   dnsmasq 
[root@master3 ~]# systemctl  start   dnsmasq

五. 安裝Hawkular

1.安裝相關依賴包

yum install httpd-tools java-1.8.0-openjdk-headless java-1.8.0-openjdk-headless -y

2.load 鏡像（所有節點都執行一下）

[root@master2 ~]# ls
anaconda-ks.cfg  hawkular.3.6.tar.gz
[root@master2 ~]# tar zxf hawkular.3.6.tar.gz 
[root@master2 ~]# cd hawkular/
[root@master2 hawkular]# ls
origin-metrics-cassandra.tar  origin-metrics-deployer.tar  origin-metrics-hawkular-metrics.tar.gz  origin-metrics-heapster.tar
[root@master2 hawkular]# for i in `ls`;do docker load -i $i;done

3.ansible 安裝hawkular 注意修改域名（openshift_metrics_hawkular_hostname)這一項

ansible-playbook -i hosts playbooks/byo/openshift-cluster/openshift-metrics.yml \
-e openshift_metrics_install_metrics=True \
-e openshift_metrics_hawkular_hostname=hawkular-metrics-openshift-infra.qingyuanos.com \  
-e openshift_metrics_image_version=v3.6.0    \
-e openshift_metrics_image_prefix=openshift/origin-    \
-e openshift_metrics_resolution=60s    \
-e openshift_metrics_duration=1 \

4.如果想重新安裝，先運行下列命令清除上次安裝的。

oc delete all --selector="metrics-infra" -n openshift-infra
oc delete sa --selector="metrics-infra" -n openshift-infra
oc delete templates --selector="metrics-infra" -n openshift-infra
oc delete secrets --selector="metrics-infra" -n openshift-infra
oc delete pvc --selector="metrics-infra" -n openshift-infra

六. 倉庫添加持久化存儲

1.搭建nfs

[root@master3 ~]# yum -y install nfs-utils

2.修改配置文件

[root@master3 ~]# sed -n 5p  /etc/idmapd.conf
[root@master3 ~]# cat /etc/exports
/opt/ *(rw,sync,no_subtree_check,no_root_squash,fsid=8)

3.創建相關文件夾

mkdir /opt/gogs
mkdir /opt/influxdb
mkdir /opt/logging
mkdir /opt/mongo
mkdir /opt/monito
mkdir /opt/mysql
mkdir /opt/prometheus
mkdir /opt/registry

4.添加防火牆規則

iptables -I INPUT -p tcp -m tcp --dport 2049 -j ACCEPT
iptables -I INPUT -p udp -m udp --dport 2049 -j ACCEPT
iptables -I INPUT -p tcp -m tcp --dport 53 -j ACCEPT
iptables -I INPUT -p udp -m udp --dport 53 -j ACCEPT

當前環境直接生效，爲避免重啓之後被刷新，這裏需要

[root@master3 ~]# service iptables save

5.啓動服務，並添加到開機自啓中

[root@master3 ~]#  service nfs start
[root@master3 ~]#  systemctl  enable nfs

6.給私有倉庫創建pv，以及pvc 進行持久化

首先切換項目到default

[root@master3 ~]#  oc project default
[root@master3 oc]# cat pv.yaml     #修改IP地址和路徑以及存儲大小
apiVersion: v1
kind: PersistentVolume
metadata:
  name: registry
spec:
  capacity:
    storage: 2000Gi
  accessModes:
    - ReadWriteOnce
  nfs:
    # FIXME: use the right IP
    server: 192.168.12.78
    path: /opt/registry
[root@master3 oc]# cat pvc.yaml 
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: registry-pvc
  labels:
    app: registry
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 2000Gi
  volumeName: registry
[root@master3 oc]# oc create -f pv.yaml
[root@master3 oc]# oc create -f pvc.yaml

修改掛載pv

oc volume deploymentconfigs/docker-registry \
 --add --name=registry-storage -t pvc --claim-name=registry-pvc --overwrite

查看是否掛載成功

[root@master3 oc]# oc get pv
NAME               CAPACITY   ACCESSMODES   RECLAIMPOLICY   STATUS    CLAIM                    STORAGECLASS   REASON    AGE
registry           2000Gi     RWO           Retain          Bound     default/registry-pvc                              14h

七.安裝cicd

1.把鏡像傳到所有節點

[root@master3 cicd]# ll /oc/cicd/images/
total 1414384
-rw-------. 1 qy qy 151784448 May 18  2017 gogs.tar
-rw-------. 1 qy qy 679963136 May 18  2017 jenkins1.tar
-rw-------. 1 qy qy 616568320 May 18  2017 slave.tar
[root@master3 cicd]# cd /oc/cicd/
[root@master3 cicd]# scp -r images/ master1.qingyuanos.com:/root

2.在所有節點load鏡像

[root@master3 images]# for i in `ls`;do docker load -i $i ;done

3.修改pv.yaml文件中的nfs地址以及掛載路徑

[root@master3 cicd]# cat pv.yaml | grep 192 -n -A 1
15:    server: 192.168.12.78
16-    path: /opt/gogs

4.修改nfs掛載目錄的權限

[root@master3 opt]# chmod -R 777 gogs/ registry/

5.修改腳本後執行腳本（把$3修改爲$2）

[root@master3 cicd]# cat start-cicd.sh | grep "\$2" -n
7:export HUB=`oc get svc -n default|grep docker-registry|awk '{print $2}'`:5000
[root@master3 cicd]# bash -x start-cicd.sh

6.gog初始化
a.首先查看gogs的域名

[root@master3 cicd]# oc  get route
NAME      HOST/PORT                     PATH      SERVICES   PORT      TERMINATION   WILDCARD
gogs      gogs-cicd.qingyuanos.com                gogs       3000                    None
jenkins   jenkins-cicd.qingyuanos.com             jenkins    <all>                   None

b.登陸gogs進行初始化


c.登陸錄http://gogs-cicd.qingyuanos.com/ 賬號gogs，密碼password
     點擊+號，創建新的倉庫
     只需要填寫倉庫名稱： 
    openshift-tasks
d.登陸到安裝cicd的節點（注意get remote 路徑）

[root@master3 cicd]# cd /oc/cicd/openshift-tasks/
[root@master3 cicd]# rm -rf .git   ##注意git前面一個點
[root@master3 cicd]# git init
[root@master3 cicd]# git config --global user.email [email protected]
[root@master3 cicd]# git add .
[root@master3 cicd]# git commit -a -m "init"
[root@master3 cicd]# git remote add origin http://gogs-cicd.qingyuanos.com/gogs/openshift-tasks.git
[root@master3 cicd]# git push -u origin master (賬號：gogs 密碼password)

八.部署普羅米修斯

1.創建一個project，用來部署prometheus.建議名字定爲“monitor”

[root@master3 prometheus]#  oc adm new-project monitor --node-selector=''

2.解壓文件並執行腳本，需要一個參數——在步驟1裏創建的project的name：

[root@master3 oc]# tar zxf prometheus.tar.gz 
[root@master3 oc]# cd prometheus/
[root@master3 prometheus]# ./sa-scc.sh monitor

3.執行腳本，需要兩個參數——鏡像倉庫地址、project name

[root@master3 prometheus]# REGISTRY=`oc get svc -n default| grep docker-registry| awk  '{print $2":5000"}'`
[root@master3 prometheus]# ./tar-to-image.sh $REGISTRY monitor 
[root@master3 prometheus]# bash -x prom.sh  $REGISTRY monitor

九.安裝ux

1.添加權限

oadm policy add-scc-to-user anyuid system:serviceaccount:qybe:default
oadm policy add-cluster-role-to-user cluster-admin admin
oadm policy add-role-to-user cluster-admin admin 
oadm policy add-scc-to-group anyuid system:authenticated -n qybe
oadm policy add-scc-to-user anyuid -z qybe

2.執行以下命令

cd images
for i in `ls *.tar`; do docker load -i $i ;done
oc login -u system:admin
oadm policy add-cluster-role-to-user cluster-admin admin  
oadm policy add-role-to-user cluster-admin admin          
oc new-project qybe --display-name="QingYuan OS"
export HUB=`oc get svc -n default|grep docker-registry|awk '{print $2}'`:5000   
docker tag qybe/auth-server $HUB/qybe/auth-server       
docker tag qybe/paas-service $HUB/qybe/paas-service
docker tag qybe/ng2-qyweb $HUB/qybe/ng2-qyweb
docker tag 172.30.187.6:5000/dcsp/api-gateway  $HUB/qybe/api-gateway
docker tag qybe/eureka-service $HUB/qybe/eureka
docker tag qybe/config-server $HUB/qybe/config-server
docker tag qybe/caas-service:latest $HUB/qybe/alert
docker tag 172.30.117.248:5000/qybe/mysql $HUB/qybe/mysql
docker tag mongo $HUB/qybe/mongo
oc login -u admin -p cow
export TOKEN=`oc whoami -t`
docker login -u admin -p $TOKEN -e [email protected] $HUB

docker push $HUB/qybe/auth-server   
docker push $HUB/qybe/paas-service
docker push $HUB/qybe/ng2-qyweb
docker push $HUB/qybe/api-gateway
docker push $HUB/qybe/eureka
docker push $HUB/qybe/config-server
docker push $HUB/qybe/mysql
docker push $HUB/qybe/mongo
docker push $HUB/qybe/alert

3.將ux文件夾的qybe全部文件以及隱藏文件拷到nfs的qybe目錄

[root@master3 ocux]# cp -r qybe/ /opt/

4.修改yaml文件

vi api-gateway/api-gateway.yaml
- name: openshift_url
value: https://10.0.1.10:8443   #修改成三臺master 其中之一的，也可以事域名
- name: prometheus_route
value: http://prometheus-kube-system.eu.qingyuanos.com $修改成prometheus 的route 通過oc get route -n monitor 進行查看
  vi config-server/config-server-pv.yaml
nfs:
# FIXME: use the right IP
server: 10.0.1.11 # nfs的IP地址
path: /opt/qybe #根據實際情況進行修改
   3.  vi  mongo/mongo.yaml
  nfs:
# FIXME: use the right IP
server: 10.0.1.11  # nfs的IP地址
path: /opt/mongo #根據實際情況進行修改
4.  vi mysql/mysql-pv.yaml
nfs:
# FIXME: use the right IP
server: 10.0.1.11 # nfs的IP地址
path: /mnt/mysql #根據實際情況進行修改
vi ng2-qyweb/ng2-qyweb.yaml

- name: QY_OAUTH2_ADDR
value: http://api-gateway.eu.qingyuanos.com/uaa # 注意域名ansible 中hosts 文件指定什麼域名這裏就改爲什麼 
- name: QY_OS_WS_ADDR
value: wss://10.0.1.10:8443 #修改成三臺master 其中之一的，也可以事域名
- name: QY_NODE_ADDR
value: http://ng2-qybe.eu.qingyuanos.com # 注意域名ansible 中hosts 文件指定什麼域名這裏就改爲什麼 
- name: QY_WS_GW_ADDR
value: ws://ng2-ws-qybe.eu.qingyuanos.com # 注意域名ansible 中hosts 文件指定什麼域名這裏就改爲什麼 
paas-service/paas-service.yaml
value:jdbc:mysql://mysql-svc:3306/qyweb_paas?useUnicode=true&characterEncoding=utf8&useSSL=false
需創建Mysql庫，需修改mysql svc地址。

5.使用oc命令創建pod （注：一定按照以下順序 創建時候 running一個後 再去創建下一個）

[root@master3 ~]# oc project qybe
[root@master3 ~]# cd /oc/ocux/
[root@master3 ocux]# oc create -f mongo/
[root@master3 ocux]# oc create -f mysql/
插入mysql權限 
[root@master3 ocux]# oc exec -it `oc get po | grep mysql | awk  '{print $1}'` bash
root@mysql-1-b9wzv:/# mysql -u root -p1q2w3e4r
use qy_oauth2; 
insert into user_role values(5,1,1);
create database qyweb_paas;
exit
exit
[root@master3 ocux]# oc create -f config-server/
[root@master3 ocux]# oc create -f auth-server/
[root@master3 ocux]# oc create -f eureka/
[root@master3 ocux]# oc create -f api-gateway/
[root@master3 ocux]# oc create -f paas-service/
[root@master3 ocux]# oc create -f alert/ 
[root@master3 ocux]# oc create -f ng2-qyweb

6.打一些 label

oc label user admin org.test-org1=admin
oc describe user admin
oc label namespace cicd organization=test-org1
oc label namespace default organization=test-org1
oc label namespace dev organization=test-org1
oc label namespace kube-syste morganization=test-org1
oc label namespace logging organization=test-org1
oc label namespace management-infra organization=test-org1
oc label namespace openshift organization=test-org1
oc label namespace openshift-infra organization=test-org1
oc label namespace qyosiaas organization=test-org1
oc label namespace stage organization=test-org1
oc label namespace monitor organization=test-org1