準備
工具
https://github.com/kubernetes-sigs/kubespray
kubespray 使用 ansible 快速部署容器化 高可用 k8s 集羣
環境
| 主機 | 內網ip | 外網ip | 系統 |
|---|---|---|---|
| k8s-1 | 10.0.0.18 | 61.xxx.xxx.187 | ubuntu 18.04 |
| k8s-2 | 10.0.0.19 | ubuntu 18.04 | |
| k8s-3 | 10.0.0.20 | ubuntu 18.04 |
規劃
| 部署節點 | k8s-1 | ||
|---|---|---|---|
| etcd 節點 | k8s-1 | k8s-2 | k8s-3 |
| master 節點 | k8s-1 | k8s-2 | |
| node 節點 | k8s-1 | k8s-2 | k8s-3 |
部署
默認全部使用 root 用戶操作
配置 DNS
# 所有節點
vim /etc/hosts
10.0.0.18 k8s-1
10.0.0.19 k8s-2
10.0.0.20 k8s-3
修改 apt 源
# 所有節點
cp /etc/apt/sources.list /etc/apt/sources.list.bakcup
cat > /etc/apt/sources.list <<EOF
deb http://mirrors.aliyun.com/ubuntu/ bionic main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ bionic-security main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ bionic-updates main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ bionic-proposed main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ bionic-backports main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-security main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-updates main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-proposed main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-backports main restricted universe multiverse
EOF
apt update
配置ssh免密登陸
# 在部署節點 k8s-1 上
ssh-keygen
ssh-copy-id k8s-1
ssh-copy-id k8s-2
ssh-copy-id k8s-3
內核升級
ubuntu 18.04 使用內核 4.15, 達到要求, 不需要更新, 其他系統內核更新參考:
https://github.com/easzlab/kubeasz/blob/master/docs/guide/kernel_upgrade.md
安裝依賴
# 在所有節點上
apt install -y python2.7
# 在部署節點 k8s-1 上
apt install -y python3-pip
pip3 install pip --upgrade -i https://mirrors.aliyun.com/pypi/simple/
修改 pip 源
mkdir ~/.pip
cat > ~/.pip/pip.conf << EOF
[global]
trusted-host=mirrors.aliyun.com
index-url=https://mirrors.aliyun.com/pypi/simple/
EOF
修改 docker 源
# 所有節點上
# 阿里雲鏡像加速, 可以使用自己的加速器地址
mkdir /etc/docker
cat > /etc/docker/daemon.json << EOF
{
"registry-mirrors": ["https://jzngeu7d.mirror.aliyuncs.com"]
}
EOF
systemctl restart docker
kubespray
cd /opt
git clone https://github.com/kubernetes-sigs/kubespray
# 安裝依賴
cd kubespray
pip3 install -U setuptools cryptography
pip3 install -r requirements.txt
# 拷貝配置文件
cp -rfp inventory/sample inventory/mycluster
# 使用腳本配置 ansible inventory 文件
declare -a IPS=(10.0.0.18 10.0.0.19 10.0.0.20)
CONFIG_FILE=inventory/mycluster/hosts.yaml python3 contrib/inventory_builder/inventory.py ${IPS[@]}
# DEBUG: Adding group all
# DEBUG: Adding group kube-master
# DEBUG: Adding group kube-node
# DEBUG: Adding group etcd
# DEBUG: Adding group k8s-cluster
# DEBUG: Adding group calico-rr
# DEBUG: adding host node1 to group all
# DEBUG: adding host node2 to group all
# DEBUG: adding host node3 to group all
# DEBUG: adding host node1 to group etcd
# DEBUG: adding host node2 to group etcd
# DEBUG: adding host node3 to group etcd
# DEBUG: adding host node1 to group kube-master
# DEBUG: adding host node2 to group kube-master
# DEBUG: adding host node1 to group kube-node
# DEBUG: adding host node2 to group kube-node
# DEBUG: adding host node3 to group kube-node
# 修改劇本默認配置
# 主要是 group_vars/all/all.yml 和 group_vars/k8s-cluster/k8s-cluster.yml
# 優先級 k8s-cluster.yml > all.yml > roles/xxx/defalut/main.yml
# 所以想要覆蓋 role 裏面的默認配置, 優先看 k8s-cluster.yml 裏面是否有同名配置, 如果有就同時修改 k8s-cluster.yml 和 all.yml, 沒有就在 all.yml 裏面添加
# 或者直接使用 ansible-playbook -e @foo.yml 的方式, 因爲 -e 指定的變量具有最高優先級
# kubespray 常用變量參考: https://kubespray.io/#/docs/vars?id=common-vars-that-are-used-in-kubespray
vim inventory/mycluster/group_vars/all/all.yml
# 加載內核模塊，否則 ceph, gfs 等無法掛載客戶端
kubelet_load_modules: true
gcr_image_repo: "gcr.azk8s.cn"
kube_image_repo: "gcr.azk8s.cn/google-containers"
quay_image_repo: "quay.mirrors.ustc.edu.cn"
docker_ubuntu_repo_base_url: "http://mirrors.aliyun.com/docker-ce/linux/ubuntu"
docker_ubuntu_repo_gpgkey: 'http://mirrors.aliyun.com/docker-ce/linux/ubuntu/gpg'
vim inventory/mycluster/group_vars/k8s-cluster/k8s-cluster.yml
kube_image_repo: "gcr.azk8s.cn/google-containers"
ansible-playbook -i inventory/mycluster/hosts.yaml --become --become-user=root cluster.yml
# 劇本可使用的 tag 參考: https://kubespray.io/#/docs/ansible?id=ansible-tags
# 如果有些github資源下載過慢, 如 https://github.com/containernetworking/plugins/releases/download/v0.8.3/cni-plugins-linux-amd64-v0.8.3.tgz 下載過慢
# 可以配置在 all.yml 中 download_force_cache: true
# 並手動下載好之後放到各節點的 /tmp/kubespray_cache/ 目錄, 沒有該目錄自行創建
管理
增加節點
- 修改
hosts.yml文件 ansible-playbook -i inventory/mycluster/hosts.yml scale.yml -b
刪除節點
ansible-playbook -i inventory/mycluster/hosts.yml remove-node.yml -b --extra-vars "node=nodename,nodename2"- 如果待刪除的節點無法通過
ssh連接，需要添加--extra-vars reset_nodes=no
組件升級
參考：https://kubespray.io/#/docs/upgrades
HA
https://kubespray.io/#/docs/ha-mode
生產環境上推薦使用外部自建負載均衡，內部負載通過 node 節點上的 nginx 或是 haproxy 實現，與 kubeasz 2.x 架構思想一樣
大型部署
https://kubespray.io/#/docs/large-deployments
備註
- 雲上環境儘量使用
flannel網絡插件 - 如果使用
calico或kube-router網絡插件, 並且在openstack上, 需要對集羣所有主機的端口設置可用地址對，允許10.233.0.0/18和10.233.64.0/18，詳情：https://kubespray.io/#/docs/openstack
參考
https://kubespray.io/#/