準備
工具
https://github.com/easzlab/kubeasz
kubeasz 使用 ansible 快速部署非容器化 高可用 k8s 集羣
kubeasz 主要有 1.x 和 2.x 兩個版本，版本比較參考：https://github.com/easzlab/kubeasz/blob/master/docs/mixes/branch.md
這裏使用 2.x 版本
環境
| 主機 | 內網ip | 外網ip | 系統 |
|---|---|---|---|
| k8s-1 | 10.0.0.18 | 61.184.241.187 | ubuntu 18.04 |
| k8s-2 | 10.0.0.19 | ubuntu 18.04 | |
| k8s-3 | 10.0.0.20 | ubuntu 18.04 |
架構
k8s集羣內部訪問 apiserver，可以通過節點自身安裝的 haproxy 實現負載，高可用，而外部訪問 apiserver，是通過另外的 haproxy + keepalived 實現的，這個外部的負載均衡是可選的，因爲該負載隻影響能不能高可用訪問 k8s 管理界面，並不影響 k8s 內部集羣的高可用，所以可以選擇不安裝
參考：https://github.com/easzlab/kubeasz/issues/585#issuecomment-502948966
master與node節點共用時，無法在master上部署haproxy+keepalived實現外部訪問apiserver高可用，詳情：https://github.com/easzlab/kubeasz/issues/585#issuecomment-575954720
規劃
| 部署節點 | k8s-1 | ||
|---|---|---|---|
| etcd 節點 | k8s-1 | k8s-2 | k8s-3 |
| master 節點 | k8s-1 | k8s-2 | |
| node 節點 | k8s-1 | k8s-2 | k8s-3 |
部署
默認全部使用 root 用戶操作
配置 DNS
# 所有節點
vim /etc/hosts
10.0.0.18 k8s-1
10.0.0.19 k8s-2
10.0.0.20 k8s-3
修改 apt 源
# 所有節點
cp /etc/apt/sources.list /etc/apt/sources.list.bakcup
cat > /etc/apt/sources.list <<EOF
deb http://mirrors.aliyun.com/ubuntu/ xenial main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ xenial-security main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ xenial-updates main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ xenial-proposed main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ xenial-backports main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ xenial main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ xenial-security main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ xenial-updates main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ xenial-proposed main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ xenial-backports main restricted universe multiverse
EOF
apt update
配置ssh免密登陸
# 在部署節點 k8s-1 上
ssh-keygen
ssh-copy-id k8s-1
ssh-copy-id k8s-2
ssh-copy-id k8s-3
內核升級
ubuntu 18.04 使用內核 4.15, 達到要求, 不需要更新, 其他系統內核更新參考:
https://github.com/easzlab/kubeasz/blob/master/docs/guide/kernel_upgrade.md
安裝依賴
# 在所有節點上
apt install -y python2.7
下載
cd /opt
# 下載工具腳本easzup，舉例使用kubeasz版本2.2.0
export release=2.2.0
curl -C- -fLO --retry 3 https://github.com/easzlab/kubeasz/releases/download/${release}/easzup
chmod +x ./easzup
# 使用工具腳本下載
./easzup -D
上述腳本運行成功後，所有文件（kubeasz代碼、二進制、離線鏡像）均已整理好放入目錄/etc/ansible
/etc/ansible包含kubeasz版本爲${release}的發佈代碼/etc/ansible/bin包含k8s/etcd/docker/cni等二進制文件/etc/ansible/down包含集羣安裝時需要的離線容器鏡像/etc/ansible/down/packages包含集羣安裝時需要的系統基礎軟件
安裝集羣
# 容器化運行 kubeasz
cd /opt
./easzup -S
# 安裝集羣
# 創建集羣上下文
docker exec -it kubeasz easzctl checkout myk8s
# 修改hosts文件
cd /etc/ansible && cp example/hosts.multi-node hosts
# 'etcd' cluster should have odd member(s) (1,3,5,...)
# variable 'NODE_NAME' is the distinct name of a member in 'etcd' cluster
[etcd]
10.0.0.18 NODE_NAME=etcd1
10.0.0.19 NODE_NAME=etcd2
10.0.0.20 NODE_NAME=etcd3
# master node(s)
[kube-master]
10.0.0.18
10.0.0.19
# work node(s)
[kube-node]
10.0.0.18
10.0.0.19
10.0.0.20
# [optional] harbor server, a private docker registry
# 'NEW_INSTALL': 'yes' to install a harbor server; 'no' to integrate with existed one
# 'SELF_SIGNED_CERT': 'no' you need put files of certificates named harbor.pem and harbor-key.pem in directory 'down'
[harbor]
#192.168.1.8 HARBOR_DOMAIN="harbor.yourdomain.com" NEW_INSTALL=no SELF_SIGNED_CERT=yes
# [optional] loadbalance for accessing k8s from outside
[ex-lb]
#192.168.1.6 LB_ROLE=backup EX_APISERVER_VIP=192.168.1.250 EX_APISERVER_PORT=8443
#192.168.1.7 LB_ROLE=master EX_APISERVER_VIP=192.168.1.250 EX_APISERVER_PORT=8443
# [optional] ntp server for the cluster
[chrony]
10.0.0.18
[all:vars]
# --------- Main Variables ---------------
# Cluster container-runtime supported: docker, containerd
CONTAINER_RUNTIME="docker"
# Network plugins supported: calico, flannel, kube-router, cilium, kube-ovn
CLUSTER_NETWORK="flannel"
# Service proxy mode of kube-proxy: 'iptables' or 'ipvs'
PROXY_MODE="ipvs"
# K8S Service CIDR, not overlap with node(host) networking
SERVICE_CIDR="10.68.0.0/16"
# Cluster CIDR (Pod CIDR), not overlap with node(host) networking
CLUSTER_CIDR="172.20.0.0/16"
# NodePort Range
NODE_PORT_RANGE="20000-40000"
# Cluster DNS Domain
CLUSTER_DNS_DOMAIN="cluster.local."
# -------- Additional Variables (don't change the default value right now) ---
# Binaries Directory
bin_dir="/opt/kube/bin"
# CA and other components cert/key Directory
ca_dir="/etc/kubernetes/ssl"
# Deploy Directory (kubeasz workspace)
base_dir="/etc/ansible"
# 配置整個 k8s 集羣
docker exec -it kubeasz easzctl setup
驗證
# 如果提示kubectl: command not found，退出重新ssh登錄一下，環境變量生效即可
$ kubectl version # 驗證集羣版本
$ kubectl get componentstatus # 驗證 scheduler/controller-manager/etcd等組件狀態
$ kubectl get node # 驗證節點就緒 (Ready) 狀態
$ kubectl get pod --all-namespaces # 驗證集羣pod狀態，默認已安裝網絡插件、coredns、metrics-server等
$ kubectl get svc --all-namespaces # 驗證集羣服務狀態
清理
# 清理集羣
docker exec -it kubeasz easzctl destroy
# 進一步清理, 清理部署節點上的 docker 以及下載的所有文件
# 清理運行的容器 easzup -C
# 清理容器鏡像 docker system prune -a
# 停止docker服務 systemctl stop docker
# 刪除下載文件 rm -rf /etc/ansible /etc/docker /opt/kube
# 刪除docker文件
$ umount /var/run/docker/netns/default
$ umount /var/lib/docker/overlay
$ rm -rf /var/lib/docker /var/run/docker
# 重啓節點
