阿里雲Kubernetes CSI實踐 - TopologyAware雲盤卷

Topology Aware

對於雲盤這種存儲卷類型，只能在相同zone的ecs上進行掛載，在多可用區的集羣環境中，就需要對pv、pvc和pod進行一致性zone調度；

1. 傳統使用方式：

通過PV、PVC所在zoneid信息，把pod調度到相應zone；

即在PV中定義相應的zoneid信息，當一個pod使用這個pv的時候，scheduler會根據pv的zoneid把pod調度到相應可用區；

具體配置：

apiVersion: v1
kind: PersistentVolume
metadata:
  labels:
    failure-domain.beta.kubernetes.io/region: cn-beijing
    failure-domain.beta.kubernetes.io/zone: cn-beijing-e
  name: d-2ze6wbcvihjeaw5bae0c
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 20Gi
  csi:
    driver: diskplugin.csi.alibabacloud.com
    volumeHandle: d-bp1fqy1enb2rtymx5g5y
  persistentVolumeReclaimPolicy: Delete
  storageClassName: alicloud-disk-ssd

通過在PV中添加下面label來完成調度：

  labels:
    failure-domain.beta.kubernetes.io/region: cn-beijing
    failure-domain.beta.kubernetes.io/zone: cn-beijing-e

2. 通過Topology Aware方式

上述方式需要先確定雲盤的可用區，然後與pod進行關聯調度。

而Topology Aware方式是雲盤根據pod所在的zone進行創建。

使用下面storageclass，WaitForFirstConsumer配置會使pv controller在pod啓動的時候去創建pv，並通過pod所在的zone信息來確定pv所在的zone。

kind: StorageClass
apiVersion: storage.k8s.io/v1beta1
metadata:
  name: alicloud-disk-aware
provisioner: diskplugin.csi.alibabacloud.com
volumeBindingMode: WaitForFirstConsumer
parameters:
  type: cloud_ssd

環境準備

集羣創建、依賴配置、CSI插件部署等請參考：CSI部署詳解

創建StorageClass

WaitForFirstConsumer：表示只有pod消費這個pvc的時候才驅動Provisioner進行pv創建；

kind: StorageClass
apiVersion: storage.k8s.io/v1beta1
metadata:
  name: alicloud-disk-aware
provisioner: diskplugin.csi.alibabacloud.com
volumeBindingMode: WaitForFirstConsumer
parameters:
  type: cloud_ssd

創建動態PVC PV

通過下面模板創建動態卷PV:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: disk-pvc
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 25Gi
  storageClassName: alicloud-disk-aware

# kubectl get pvc
NAME       STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS          AGE
disk-pvc   Pending                                      alicloud-disk-aware   23m

創建應用

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-disk
  labels:
    app: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        ports:
        - containerPort: 80
        volumeMounts:
          - name: disk-pvc
            mountPath: "/data"
      volumes:
        - name: disk-pvc
          persistentVolumeClaim:
            claimName: disk-pvc

驗證掛載、高可用

查看pvc已經綁定完成
# kubectl get pvc
NAME       STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS          AGE
disk-pvc   Bound    pvc-bf7732a6-a7cf-11e9-8dec-00163e0a6ecc   25Gi       RWO            alicloud-disk-aware   18s
# kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM              STORAGECLASS          REASON   AGE
pvc-bf7732a6-a7cf-11e9-8dec-00163e0a6ecc   25Gi       RWO            Delete           Bound    default/disk-pvc   alicloud-disk-aware            7s

查看pvc詳情，volume.kubernetes.io/selected-node即爲pv的調度信息：
# kubectl describe pvc disk-pvc
Name:          disk-pvc
Namespace:     default
StorageClass:  alicloud-disk-aware
Status:        Bound
Volume:        pvc-bf7732a6-a7cf-11e9-8dec-00163e0a6ecc
Labels:        <none>
Annotations:   pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: diskplugin.csi.alibabacloud.com
               volume.kubernetes.io/selected-node: cn-hangzhou.192.168.6.149
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      25Gi
Access Modes:  RWO
VolumeMode:    Filesystem

查看pod，驗證雲盤掛載成功，創建測試文件；
# kubectl get pod
NAME                          READY   STATUS    RESTARTS   AGE
nginx-disk-6d5659d745-2ts26   1/1     Running   0          2m1s
# kubectl exec nginx-disk-6d5659d745-2ts26 ls /data
lost+found
# kubectl exec nginx-disk-6d5659d745-2ts26 touch /data/test
# kubectl exec nginx-disk-6d5659d745-2ts26 ls /data
lost+found
test

刪除Pod，查看重建Pod是否數據穩定；
# kubectl delete pod nginx-disk-6d5659d745-2ts26
pod "nginx-disk-6d5659d745-2ts26" deleted

# kubectl get pod
NAME                          READY   STATUS    RESTARTS   AGE
nginx-disk-6d5659d745-m5z9p   1/1     Running   0          11s
# kubectl exec nginx-disk-6d5659d745-m5z9p ls /data
lost+found
test