k8s實踐(十四)：Pod驅逐遷移和Node節點維護

環境說明：

主機名	操作系統版本	ip	docker version	kubelet version	配置	備註
master	Centos 7.6.1810	172.27.9.131	Docker 18.09.6	V1.14.2	2C2G	master主機
node01	Centos 7.6.1810	172.27.9.135	Docker 18.09.6	V1.14.2	2C2G	node節點
node02	Centos 7.6.1810	172.27.9.136	Docker 18.09.6	V1.14.2	2C2G	node節點

k8s集羣部署詳見：Centos7.6部署k8s(v1.14.2)集羣

k8s學習資料詳見：基本概念、kubectl命令和資料分享

emptyDir詳見：存儲卷和數據持久化(Volumes and Persistent Storage)

一、背景

當node節點進行如打補丁、操作系統升級等操作時，需停機維護，這就涉及pod驅逐遷移，本文將詳細介紹node節點維護的整個過程。

二、pdb簡介

pdb爲poddisruptionbudgets縮寫，意爲主動驅逐保護；

沒有pdb。當進行節點維護時，如果某個服務的多個pod在該節點上，則節點的停機可能會造成服務中斷或者服務降級。舉個例子，某服務有5個pod，最低3個pod能保證服務質量，否則會造成響應慢等影響，此時該服務的4個pod在node01上，如果對node01進行停機維護，此時只有1個pod能正常對外服務，在node01的4個pod遷移過程中，就會影響該服務正常響應；

pdb能保證應用在節點維護時不低於一定數量的pod運行，從而保持服務質量；

三、準備工作

1.新建pod

[root@master ~]# more nginx-master.yml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx-master
spec:
  replicas: 10 
  template:
    metadata:
      labels:
        app: nginx
    spec:
      restartPolicy: Always
      containers:
      - name: nginx
        image: nginx:latest
[root@master ~]# kubectl apply -f nginx-master.yml 
deployment.extensions/nginx-master created
[root@master ~]# kubectl get po -o wide
NAME                           READY   STATUS    RESTARTS   AGE   IP             NODE     NOMINATED NODE   READINESS GATES
nginx-master-9d4cf4f77-47vfj   1/1     Running   0          28s   10.244.0.129   master   <none>           <none>
nginx-master-9d4cf4f77-69jn6   1/1     Running   0          28s   10.244.2.206   node02   <none>           <none>
nginx-master-9d4cf4f77-6drhg   1/1     Running   0          28s   10.244.1.218   node01   <none>           <none>
nginx-master-9d4cf4f77-b7zfd   1/1     Running   0          28s   10.244.1.219   node01   <none>           <none>
nginx-master-9d4cf4f77-fxsjd   1/1     Running   0          28s   10.244.2.204   node02   <none>           <none>
nginx-master-9d4cf4f77-ktnvk   1/1     Running   0          28s   10.244.0.128   master   <none>           <none>
nginx-master-9d4cf4f77-mzrx7   1/1     Running   0          28s   10.244.1.217   node01   <none>           <none>
nginx-master-9d4cf4f77-pcznk   1/1     Running   0          28s   10.244.2.203   node02   <none>           <none>
nginx-master-9d4cf4f77-px98b   1/1     Running   0          28s   10.244.2.205   node02   <none>           <none>
nginx-master-9d4cf4f77-wtcwt   1/1     Running   0          28s   10.244.1.220   node01   <none>           <none>

新建pod，鏡像爲最新版的nginx，deployment爲nginx-master，數量爲10。可以看到10個pod分佈在node01、node02和master 3臺不同主機上。

2.新建pdb

[root@master ~]# more pdb-nginx.yaml 
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
  name: pdb-nginx
spec:
  minAvailable: 9
  selector:
    matchLabels:
      app: nginx
[root@master ~]# kubectl apply -f pdb-nginx.yaml 
poddisruptionbudget.policy/pdb-nginx created
[root@master ~]# kubectl get pdb
NAME        MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
pdb-nginx   9               N/A               1                     8s

新建pdb pdb-nginx，Label Selector和deployment一樣都爲app: nginx，minAvailable: 9意爲存活的nginx pod至少爲9個。

四、節點維護

本文以節點node02維護爲例介紹。

1.設置節點不可調度

[root@master ~]# kubectl cordon node02
node/node02 cordoned
[root@master ~]# kubectl get node
NAME     STATUS                     ROLES    AGE    VERSION
master   Ready                      master   184d   v1.14.2
node01   Ready                      <none>   183d   v1.14.2
node02   Ready,SchedulingDisabled   <none>   182d   v1.14.2
[root@master ~]# kubectl get po -o wide
NAME                           READY   STATUS    RESTARTS   AGE   IP             NODE     NOMINATED NODE   READINESS GATES
nginx-master-9d4cf4f77-47vfj   1/1     Running   0          30m   10.244.0.129   master   <none>           <none>
nginx-master-9d4cf4f77-69jn6   1/1     Running   0          30m   10.244.2.206   node02   <none>           <none>
nginx-master-9d4cf4f77-6drhg   1/1     Running   0          30m   10.244.1.218   node01   <none>           <none>
nginx-master-9d4cf4f77-b7zfd   1/1     Running   0          30m   10.244.1.219   node01   <none>           <none>
nginx-master-9d4cf4f77-fxsjd   1/1     Running   0          30m   10.244.2.204   node02   <none>           <none>
nginx-master-9d4cf4f77-ktnvk   1/1     Running   0          30m   10.244.0.128   master   <none>           <none>
nginx-master-9d4cf4f77-mzrx7   1/1     Running   0          30m   10.244.1.217   node01   <none>           <none>
nginx-master-9d4cf4f77-pcznk   1/1     Running   0          30m   10.244.2.203   node02   <none>           <none>
nginx-master-9d4cf4f77-px98b   1/1     Running   0          30m   10.244.2.205   node02   <none>           <none>
nginx-master-9d4cf4f77-wtcwt   1/1     Running   0          30m   10.244.1.220   node01   <none>           <none>

設置node02不可調度，查看各節點狀態，發現node02爲SchedulingDisabled，此時master不會將新的pod調度到該節點上，但是node02上pod還是正常運行。

2.驅逐節點上的pod

[root@master ~]# kubectl drain node02 --delete-local-data --ignore-daemonsets --force 
node/node02 already cordoned

參數說明：

--delete-local-data 即使pod使用了emptyDir也刪除

--ignore-daemonsets 忽略deamonset控制器的pod，如果不忽略，deamonset控制器控制的pod被刪除後可能馬上又在此節點上啓動起來,會成爲死循環；

--force 不加force參數只會刪除該NODE上由ReplicationController, ReplicaSet, DaemonSet,StatefulSet or Job創建的Pod，加了後還會刪除'裸奔的pod'(沒有綁定到任何replication controller)

可以看到同一時刻只有一個pod進行遷移，對外提供服務的pod始終有9個。

遷移pod nginx-master-9d4cf4f77-pcznk到node01

遷移pod nginx-master-9d4cf4f77-px98b到master，此時前一個pod nginx-master-9d4cf4f77-pcznk已經遷移完成。

遷移pod nginx-master-9d4cf4f77-69jn6到master

遷移pod nginx-master-9d4cf4f77-fxsjd到master

這個也再次驗證了同一時刻只有一個pod遷移，nginx服務始終有9個pod對外提供服務。

3.維護結束

[root@master ~]# kubectl uncordon node02
node/node02 uncordoned
[root@master ~]# kubectl get nodes      
NAME     STATUS   ROLES    AGE    VERSION
master   Ready    master   184d   v1.14.2
node01   Ready    <none>   183d   v1.14.2
node02   Ready    <none>   183d   v1.14.2

維護結束，重新將node02節點置爲可調度狀態。

五、pod回遷

pod回遷貌似還沒什麼好的辦法，這裏採用delete然後重建的方式回遷。

[root@master ~]# kubectl get po -o wide
NAME                           READY   STATUS    RESTARTS   AGE   IP             NODE     NOMINATED NODE   READINESS GATES
nginx-master-9d4cf4f77-2vnvk   1/1     Running   0          33m   10.244.1.222   node01   <none>           <none>
nginx-master-9d4cf4f77-47vfj   1/1     Running   0          73m   10.244.0.129   master   <none>           <none>
nginx-master-9d4cf4f77-6drhg   1/1     Running   0          73m   10.244.1.218   node01   <none>           <none>
nginx-master-9d4cf4f77-7n7pt   1/1     Running   0          32m   10.244.0.131   master   <none>           <none>
nginx-master-9d4cf4f77-b7zfd   1/1     Running   0          73m   10.244.1.219   node01   <none>           <none>
nginx-master-9d4cf4f77-ktnvk   1/1     Running   0          73m   10.244.0.128   master   <none>           <none>
nginx-master-9d4cf4f77-mzrx7   1/1     Running   0          73m   10.244.1.217   node01   <none>           <none>
nginx-master-9d4cf4f77-pdkst   1/1     Running   0          32m   10.244.0.130   master   <none>           <none>
nginx-master-9d4cf4f77-pskmp   1/1     Running   0          32m   10.244.0.132   master   <none>           <none>
nginx-master-9d4cf4f77-wtcwt   1/1     Running   0          73m   10.244.1.220   node01   <none>           <none>
[root@master ~]# kubectl delete po nginx-master-9d4cf4f77-47vfj
pod "nginx-master-9d4cf4f77-47vfj" deleted
[root@master ~]# kubectl delete po nginx-master-9d4cf4f77-2vnvk
pod "nginx-master-9d4cf4f77-2vnvk" deleted
[root@master ~]# kubectl get po -o wide
NAME                           READY   STATUS    RESTARTS   AGE   IP             NODE     NOMINATED NODE   READINESS GATES
nginx-master-9d4cf4f77-6drhg   1/1     Running   0          76m   10.244.1.218   node01   <none>           <none>
nginx-master-9d4cf4f77-7n7pt   1/1     Running   0          35m   10.244.0.131   master   <none>           <none>
nginx-master-9d4cf4f77-b7zfd   1/1     Running   0          76m   10.244.1.219   node01   <none>           <none>
nginx-master-9d4cf4f77-f92hp   1/1     Running   0          44s   10.244.2.207   node02   <none>           <none>
nginx-master-9d4cf4f77-ktnvk   1/1     Running   0          76m   10.244.0.128   master   <none>           <none>
nginx-master-9d4cf4f77-mzrx7   1/1     Running   0          76m   10.244.1.217   node01   <none>           <none>
nginx-master-9d4cf4f77-pdkst   1/1     Running   0          35m   10.244.0.130   master   <none>           <none>
nginx-master-9d4cf4f77-pskmp   1/1     Running   0          35m   10.244.0.132   master   <none>           <none>
nginx-master-9d4cf4f77-tdghn   1/1     Running   0          15s   10.244.2.208   node02   <none>           <none>
nginx-master-9d4cf4f77-wtcwt   1/1     Running   0          76m   10.244.1.220   node01   <none>           <none>

在業務低峯delete pod nginx-master-9d4cf4f77-47vfj和nginx-master-9d4cf4f77-2vnvk，由於node02上的pod之前都被驅逐，此時資源使用率最低，所以pod重建時會調度值該節點，完成pod回遷。

六、節點刪除

1.刪除節點

實際運維過程中可能會刪除某個node節點，本文還是以node02爲例，介紹如果刪除節點。

[root@master ~]# kubectl cordon node02
[root@master ~]# kubectl drain node02 --delete-local-data --ignore-daemonsets --force 
[root@master ~]# kubectl delete node node02

[root@node02 ~]# kubeadm reset

2.節點重新加入

master節點上運行

[root@master ~]# kubeadm token create --print-join-command
kubeadm join 172.27.9.131:6443 --token kpz40z.tuxb4t4m1q37vwl1     --discovery-token-ca-cert-hash sha256:5f656ae26b5e7d4641a979cbfdffeb7845cc5962bbfcd1d5435f00a25c02ea50

node02重新加入集羣

[root@node02 ~]# kubeadm join 172.27.9.131:6443 --token svrip0.lajrfl4jgal0ul6i     --discovery-token-ca-cert-hash sha256:5f656ae26b5e7d4641a979cbfdffeb7845cc5962bbfcd1d5435f00a25c02ea50

查看node

本文所有腳本和配置文件已上傳：Pode Eviction and Node Manage

k8s實踐(十四)：Pod驅逐遷移和Node節點維護

一、背景

二、pdb簡介

三、準備工作

1.新建pod

2.新建pdb

四、節點維護

1.設置節點不可調度

2.驅逐節點上的pod

3.維護結束

五、pod回遷

六、節點刪除

1.刪除節點

2.節點重新加入

Python 潮流週刊#50：我最喜歡的 Python 3.13 新特性！

查詢數據庫內歷史同期和總計腳本

k8s實踐(十五)：Centos7.6部署k8s v1.16.4高可用集羣(主備模式)

Centos7.6搭建nginx 1.16.1並使用upstream_check_module模塊

k8s實踐(十六)：lvs+keepalived部署k8s v1.16.4高可用集羣

k8s實踐(十四)：Pod驅逐遷移和Node節點維護

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結