我們之前使用EBS作爲k8s的動態pvc的pv，但是EBS不支持ReadWriteMany類型的聲明。
所以嘗試使用efs作爲存儲底層來聲明pvc。
動態pvc聲明方案
創建efs文件存儲系統
使用命令
# 創建EFS Security group
VPC_ID=$(aws eks describe-cluster --name ${CLUSTER_NAME} --region ${AWS_REGION} --query "cluster.resourcesVpcConfig.vpcId" --output text)
VPC_CIDR=$(aws ec2 describe-vpcs --vpc-ids ${VPC_ID} --query "Vpcs[].CidrBlock" --region ${AWS_REGION} --output text)
aws ec2 create-security-group --description ${CLUSTER_NAME}-efs-eks-sg --group-name efs-sg --vpc-id ${VPC_ID}
SGGroupID=上一步的結果訪問
aws ec2 authorize-security-group-ingress --group-id ${SGGroupID} --protocol tcp --port 2049 --cidr ${VPC_CIDR}
# 創建EFS file system 和 mount-target, 請根據你的環境替換 FileSystemId， SubnetID， SGGroupID
aws efs create-file-system --creation-token eks-efs --region ${AWS_REGION}
aws efs create-mount-target --file-system-id FileSystemId --subnet-id SubnetID --security-group SGGroupID
也可以通過界面操作，步驟參考
安裝efs驅動csi項目
這裏需要藉助 Amazon EFS CSI Driver項目，csi項目github地址
git clone https://github.com/kubernetes-sigs/aws-efs-csi-driver
cd aws-efs-csi-driver-master/deploy/kubernetes/overlays/stable
cat kustomization.yaml
當前版本內容如下:
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
bases:
- ../../base
images:
- name: amazon/aws-efs-csi-driver
newName: 602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/aws-efs-csi-driver
newTag: v0.2.0
- name: quay.io/k8scsi/livenessprobe
newName: 602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/csi-liveness-probe
newTag: v1.1.0
- name: quay.io/k8scsi/csi-node-driver-registrar
newName: 602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/csi-node-driver-registrar
newTag: v1.1.0
需要修改如下:
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
bases:
- ../../base
images:
- name: amazon/aws-efs-csi-driver
newTag: v0.3.0
newName: amazon/aws-efs-csi-driver
- name: quay.io/k8scsi/livenessprobe
newName: 602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/csi-liveness-probe
newTag: v1.1.0
- name: quay.io/k8scsi/csi-node-driver-registrar
newName: 602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/csi-node-driver-registrar
newTag: v1.1.0
創建
kubectl create -k .
kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
alb-ingress-controller-649b854d75-m8c75 1/1 Running 0 2d18h
aws-node-ct6rz 1/1 Running 0 4d18h
aws-node-sfjtn 1/1 Running 0 3d21h
aws-node-xzfx9 1/1 Running 0 4d18h
coredns-6565755d58-pd5nm 1/1 Running 0 4d18h
coredns-6565755d58-v9nl7 1/1 Running 0 4d18h
ebs-csi-controller-6dcc4dc6f4-6k4s5 4/4 Running 0 2d17h
ebs-csi-controller-6dcc4dc6f4-vtklz 4/4 Running 0 2d17h
ebs-csi-node-2zmct 3/3 Running 0 2d17h
ebs-csi-node-plljf 3/3 Running 0 2d17h
ebs-csi-node-s9lbz 3/3 Running 0 2d17h
efs-csi-node-5jtlc 3/3 Running 0 10h
efs-csi-node-lqdz9 3/3 Running 0 10h
efs-csi-node-snqmh 3/3 Running 0 10h
kube-proxy-g4mcw 1/1 Running 0 4d18h
kube-proxy-mb88w 1/1 Running 0 4d18h
kube-proxy-tpx4x 1/1 Running 0 3d21h
kubernetes-dashboard-5f7b999d65-dcc6h 1/1 Running 0 2d23h
metrics-server-7fcf9cc98b-rntrh 1/1 Running 0 44h
kubectl exec -ti efs-csi-node-5jtlc -n kube-system -- mount.efs --version
# Make sure the version is > 1.19
可能遇到的問題—Failed to resolve “fs-XXXXX.efs.cn-northwest-1.amazonaws.com” - check that your file system ID is correct
我們在創建efs的界面中其實可以看到 中國區的DNS是帶有cn後綴的，比如:amazonaws.com.cn
eks-workshop-greater-china的文檔中說v0.3.0不支持中國區，需要用v0.2.0。
https://github.com/kubernetes-sigs/aws-efs-csi-driver/issues/138 v0.2.0 image contains old version of efs-utils, efs-utils added China region support from v1.19 The v.0.3.0 does work, you can also build your image to use v.0.2.0 CSI
v0.2.0版本的鏡像中包含了老版本的efs-utils從v1.19增加對中國區的支持,但是v0.3.0是不起作用的，需要使用v0.2.0版本的鏡像。
這個問題其實v0.3.0已經修復了，而且v0.2.0反而不支持中國區，目前下載的master的kustomization.yaml默認使用的就是v0.2.0，是會遇到問題的，無法轉換出 帶有cn後綴的 DNS路徑，所以需要使用v0.3.0。
如上文使用v0.3.0的鏡像即可
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
bases:
- ../../base
images:
- name: amazon/aws-efs-csi-driver
newTag: v0.3.0
newName: amazon/aws-efs-csi-driver
- name: quay.io/k8scsi/livenessprobe
newName: 602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/csi-liveness-probe
newTag: v1.1.0
- name: quay.io/k8scsi/csi-node-driver-registrar
newName: 602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/csi-node-driver-registrar
newTag: v1.1.0
詳情參考鏈接:
support EFS in china regions
解決方案
as v0.3.0 is released: https://github.com/kubernetes-sigs/aws-efs-csi-driver/releases/tag/v0.3.0
Please use amazon/aws-efs-csi-driver:v0.3.0
The endpoints in China region are different from others
部署樣例測試
## Deploy app use the EFS
cd examples/kubernetes/multiple_pods/
aws efs describe-file-systems --query "FileSystems[*].[FileSystemId,Name]" --region ${AWS_REGION} --output text
# 修改 the specs/pv.yaml file and replace the volumeHandle with FILE_SYSTEM_ID
# 例子：
#csi:
# driver: efs.csi.aws.com
# volumeHandle: fs-9c21a999
# 部署 the efs-sc storage class, efs-claim pv claim, efs-pv, and app1 and app2 sample applications.
kubectl apply -f specs/
kubectl describe storageclass efs-sc
kubectl get pv
kubectl describe pv efs-pv
kubectl get pods --watch
kubectl get events
# 驗證
kubectl exec -ti app1 -- tail /data/out1.txt
kubectl exec -ti app2 -- tail /data/out1.txt
# 清理
kubectl delete -f specs/
靜態pvc聲明方案
pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv0001
spec:
capacity:
storage: 1Gi
accessModes:
- ReadWriteOnce
nfs:
path: /tmp
server: 172.17.0.2
persistentVolumeReclaimPolicy: Retain #Here is policy
claimRef: #Here is claim reference
name: claim1
namespace: default
pvc.yaml
apiVersion: "v1"
kind: "PersistentVolumeClaim"
metadata:
name: "claim1"
spec:
accessModes:
- "ReadWriteOnce"
resources:
requests:
storage: "1Gi"
volumeName: "pv0001"
參考資料
How do I use Amazon EFS with Amazon EKS