概覽
packstack安裝openstack
準備
控制節點與計算節點同時運行。
關閉防火牆
1. systemctl stop firewalld
2. systemctl disable firewalld
關閉SELinux
1. sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/config
2. setenforce 0
修改yum源
1. yum -y install wget
2. wget http://mirrors.aliyun.com/repo/Centos-7.repo
3. yum update
下載安裝包
控制節點與計算節點同時運行。
1. yum install -y centos-release-openstack-rocky
2. yum install -y openstack-packstack
控制節點生成並修改answer文件
控制節點運行下述命令生成answer file。
1. packstack --gen-answer-file answer.txt
修改生成的answer.txt，根據實際控制節點與計算節點的IP，修改下述信息。
1. #關閉非必須組件減少安裝時間
2. CONFIG_CINDER_INSTALL=n
3. CONFIG_MANILA_INSTALL=n
4. CONFIG_SWIFT_INSTALL=n
5. CONFIG_AODH_INSTALL=n
6. CONFIG_CONTROLLER_HOST=<控制、網絡節點IP>
7. CONFIG_COMPUTE_HOSTS=<計算節點IP>
8. CONFIG_NETWORK_HOSTS=<控制、網絡節點IP>
9. #啓用LB
10.CONFIG_LBAAS_INSTALL=y
11.CONFIG_NEUTRON_METERING_AGENT_INSTALL=n
12.CONFIG_NEUTRON_ML2_TYPE_DRIVERS=vxlan,flat
13.CONFIG_NEUTRON_ML2_TENANT_NETWORK_TYPES=vxlan
14.#將默認的OVN backend切換爲ovs
15.CONFIG_NEUTRON_ML2_MECHANISM_DRIVERS=openvswitch
16.CONFIG_NEUTRON_L2_AGENT=openvswitch
控制節點運行安裝
1. packstack --answer-file answer.txt
準備openstack中K8S所需項目、用戶及網絡信息
創建K8S所需資源可以在dashboard中進行，也可以直接通過命令行操作。
創建K8S專用的項目，用戶信息
1. openstack project create k8s
2. openstack user create --password 99cloud k8s-user
3. openstack role add --project k8s --user k8s-user admin
在K8S項目下創建pod 網絡子網及service 網絡子網
1. openstack network create pod_network
2. openstack network create service_network
3. openstack subnet create --ip-version 4 --subnet-range 10.1.0.0/16 --network pod_network pod_subnet
4. openstack subnet create --ip-version 4 --subnet-range 10.2.0.0/16 --network service_network service_subnet
在K8S項目路由器連接pod網絡子網與service網絡子網
1. openstack router create k8s-router
2. openstack router add subnet k8s-router pod_subnet
3. openstack router add subnet k8s-router service_subnet
在K8S項目創建pod安全組
1. openstack security group create service_pod_access_sg
2. openstack security group rule create --remote-ip 10.1.0.0/16 --ethertype IPv4 --protocol tcp service_pod_access_sg
3. openstack security group rule create --remote-ip 10.2.0.0/16 --ethertype IPv4 --protocol tcp service_pod_access_sg
注意，這裏的創建的project id，user name，password，pod subnet id，service subnet id，security group id均需要記錄，用於後期配置kuryr controller。
kubeadm安裝Kubernetes
準備
master節點
以下操作在master節點和work 節點均執行。
關閉虛擬內存
1. swapoff -a
2. sed -i 's/.*swap.*/#&/' /etc/fstab
配置轉發參數
1. cat <<EOF > /etc/sysctl.d/k8s.conf
2. net.bridge.bridge-nf-call-ip6tables = 1
3. net.bridge.bridge-nf-call-iptables = 1
4. EOF
5. sysctl --system
配置kubernetes阿里源
1. cat <<EOF > /etc/yum.repos.d/kubernetes.repo
2. [kubernetes]
3. name=Kubernetes
4. baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
5. enabled=1
6. gpgcheck=1
7. repo_gpgcheck=1
8. gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
9. EOF
設置docker源
1. wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -P /etc/yum.repos.d/
安裝docker
1. yum install docker-ce-18.06.1.ce -y
設置鏡像倉庫加速
1. sudo mkdir -p /etc/docker
2. sudo tee /etc/docker/daemon.json <<-'EOF'
3. {
4. "registry-mirrors": ["https://hdi5v8p1.mirror.aliyuncs.com"]
5. }
6. EOF
啓動docker
1. systemctl daemon-reload
2. systemctl enable docker
3. systemctl start docker
安裝kubernetes相關組件
1. yum install kubelet-1.12.2 kubeadm-1.12.2 kubectl-1.12.2 kubernetes-cni-0.6.0 -y
2. systemctl enable kubelet && systemctl start kubelet
開啓IPVS
加載ipvs內核，使node節點kube-proxy支持ipvs代理規則。
1. modprobe ip_vs_rr
2. modprobe ip_vs_wrr
3. modprobe ip_vs_sh
4.
5. cat <<EOF >> /etc/rc.local
6. modprobe ip_vs_rr
7. modprobe ip_vs_wrr
8. modprobe ip_vs_sh
9. EOF
下載鏡像
1. cat <<EOF >> master.sh
2. #!/bin/bash
3. kube_version=:v1.12.2
4. kube_images=(kube-proxy kube-scheduler kube-controller-manager kube-apiserver)
5. addon_images=(etcd-amd64:3.2.24 coredns:1.2.2 pause-amd64:3.1)
6.
7. for imageName in \${kube_images[@]} ; do
8. docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/\$imageName-amd64\$kube_version
9. docker image tag registry.cn-hangzhou.aliyuncs.com/google_containers/\$imageName-amd64\$kube_version k8s.gcr.io/\$imageName\$kube_version
10. docker image rm registry.cn-hangzhou.aliyuncs.com/google_containers/\$imageName-amd64\$kube_version
11.done
12.
13.for imageName in \${addon_images[@]} ; do
14. docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/\$imageName
15. docker image tag registry.cn-hangzhou.aliyuncs.com/google_containers/\$imageName k8s.gcr.io/\$imageName
16. docker image rm registry.cn-hangzhou.aliyuncs.com/google_containers/\$imageName
17.done
18.
19.docker tag k8s.gcr.io/etcd-amd64:3.2.24 k8s.gcr.io/etcd:3.2.24
20.docker image rm k8s.gcr.io/etcd-amd64:3.2.24
21.docker tag k8s.gcr.io/pause-amd64:3.1 k8s.gcr.io/pause:3.1
22.docker image rm k8s.gcr.io/pause-amd64:3.1
23.EOF
24.
25.chmod u+x master.sh
26../master.sh
kubeadm init安裝master節點
注意這裏的pod network cidr與service cidr必須和前文中創建的openstack中pod subnet cidr及service subnet cidr保持一致。
1.kubeadm init --kubernetes-version=v1.12.2 --pod-network-cidr=10.1.0.0/16 --service-cidr=10.2.0.0/16 在成功運行該命令後會輸出類似如下圖所示的結果：
① 表示kubernetes master節點安裝成功。
② 根據命令執行同樣操作。
③ 記錄該指令，用於後期添加kubernetes node節點。
配置flannel
這裏配置flannel網絡插件，僅用於對接kuryr前簡單驗證kubernetes功能是否正常。
下載配置文件
1. curl -LO https://raw.githubusercontent.com/coreos/flannel/bc79dd1505b0c8681ece4de4c0d86c5cd2643275/Documentation/kube-flannel.yml
修改pod network cidr
1. sed -i "s/10.244.0.0/10.1.0.0/g" kube-flannel.yml
啓動flannel
1. kubectl apply -f kube-flannel.yml
設置API代理
1. kubectl proxy --port=8080 --accept-hosts='.*' --address='0.0.0.0'
node節點
下載鏡像
1. cat <<EOF >> node01.sh
2. #!/bin/bash
3. kube_version=:v1.12.2
4. coredns_version=1.2.2
5. pause_version=3.1
6.
7. docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy-amd64\$kube_version
8. docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy-amd64\$kube_version k8s.gcr.io/kube-proxy\$kube_version
9. docker image rm registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy-amd64\$kube_version
10.
11.docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause-amd64:\$pause_version
12.docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/pause-amd64:\$pause_version k8s.gcr.io/pause:\$pause_version
13.docker image rm registry.cn-hangzhou.aliyuncs.com/google_containers/pause-amd64:\$pause_version
14.
15.docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:\$coredns_version
16.docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:\$coredns_version k8s.gcr.io/coredns:\$coredns_version
17.docker image rm registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:\$coredns_version
18.EOF
19.
20.chmod u+x node01.sh
21../node01.sh
添加node節點
使用在master節點安裝成功後保存的hubedem join命令，添加node節點。
1. kubeadm join 192.168.1.12:6443 --token lgbnlx.ehciqy1p1rpu6g6g --discovery-token-ca-cert-hash sha256:0acd9f6c7afcab4f4a0ebc1a1dd064f32ef09113829a30ce51dea9822d2f4afd
驗證
在master節點運行下述命令可以看到node節點被正確加入
1. [root@kuryra-1 ~]# kubectl get nodes
2. NAME STATUS ROLES AGE VERSION
3. kuryra-1 Ready master 21h v1.12.2
4. kuryra-2 Ready <none> 21h v1.12.2
安裝配置kuryr
安裝kuryr-k8s-controller
安裝
1. mkdir kuryr-k8s-controller
2. cd kuryr-k8s-controller
3. virtualenv env
4. git clone https://git.openstack.org/openstack/kuryr-kubernetes -b stable/rocky
5. . env/bin/activate
6. pip install -e kuryr-kubernetes
配置
1. cd kuryr-kubernetes
2. ./tools/generate_config_file_samples.sh
3. mkdir /etc/kuryr
4. cp etc/kuryr.conf.sample /etc/kuryr/kuryr.conf
這裏需要修改配置文件kuryr.conf。具體配置項需根據上文中在openstack內創建的用於K8S環境的信息填寫。
1. [DEFAULT]
2. use_stderr = true
3. bindir = /usr/local/libexec/kuryr
4.
5. [kubernetes]
6. api_root = http://192.168.1.11:8080
7.
8. [neutron]
9. # 需根據上文章節中創建的項目、用戶信息填寫
10.auth_url = http://192.168.1.11:5000/v3
11.username = k8s-user
12.user_domain_name = Default
13.password = 99cloud
14.project_name = ks
15.project_domain_name = Default
16.auth_type = password
17.
18.[neutron_defaults]
19.ovs_bridge = br-int
20.# 下面的網絡資源ID需根據上文章節中創建的資源ID填寫
21.pod_security_groups = a19813e3-f5bc-41d9-9a1f-54133facb6da
22.pod_subnet = 53f5b742-482b-40b6-b5d0-bf041e98270c
23.project = aced9738cfd44562a22235b1cb6f7993
24.service_subnet = d69dae26-d750-42f0-b844-5eb78a6bb873
運行
1. kuryr-k8s-controller --config-file /etc/kuryr/kuryr.conf -d
安裝kuryr-cni
安裝
1. mkdir kuryr-k8s-cni
2. cd kuryr-k8s-cni
3. virtualenv env
4. . env/bin/activate
5. git clone https://git.openstack.org/openstack/kuryr-kubernetes -b stable/rocky
6. pip install -e kuryr-kubernetes
配置
1. cd kuryr-kubernetes
2. ./tools/generate_config_file_samples.sh
3. mkdir /etc/kuryr
4. cp etc/kuryr.conf.sample /etc/kuryr/kuryr.conf
此處需要修改配置文件kuryr.conf。
1. [DEFAULT]
2. use_stderr = true
3. bindir = /usr/local/libexec/kuryr
4. lock_path=/var/lib/kuryr/tmp
5.
6. [kubernetes]
7. # 填k8s master的IP
8. api_root = http://192.168.1.11:8080
修改cni配置
執行下述命令。
1. mkdir -p /opt/cni/bin
2. ln -s $(which kuryr-cni) /opt/cni/bin/
由於前文安裝k8s時已經安裝過flannel，這裏/opt/cni/bin目錄已經存在。新增/etc/cni/net.d/10-kuryr.conf文件，按如下信息修改配置文件，同時刪除同目錄下flannel的配置文件。
1. {
2. "cniVersion": "0.3.1",
3. "name": "kuryr",
4. "type": "kuryr-cni",
5. "kuryr_conf": "/etc/kuryr/kuryr.conf",
6. "debug": true
7. }
安裝相應依賴包
1. sudo pip install 'oslo.privsep>=1.20.0' 'os-vif>=1.5.0'
運行
1. kuryr-daemon --config-file /etc/kuryr/kuryr.conf -d