擴展kubernetes兩個最常用最需要掌握的東西:自定義資源CRD 和 adminsion webhook, 本文教你如何十分鐘掌握CRD開發.
kubernetes允許用戶自定義自己的資源對象,就如同deployment statefulset一樣,這個應用非常廣泛,比如prometheus opterator就自定義Prometheus對象,再加上一個自定義的controller監聽到kubectl create Prometheus時就去創建Pod組成一個pormetheus集羣。rook等等同理。
我需要用kubernetes調度虛擬機,所以這裏自定義一個 VirtualMachine 類型
<!--more-->
kubebuilder
kubebuilder能幫我們節省大量工作,讓開發CRD和adminsion webhook變得異常簡單。
安裝
通過源碼安裝:
git clone https://github.com/kubernetes-sigs/kubebuilder
cd kubebuilder
make build
cp bin/kubebuilder $GOPATH/bin
或者下載二進制:
os=$(go env GOOS)
arch=$(go env GOARCH)
# download kubebuilder and extract it to tmp
curl -sL https://go.kubebuilder.io/dl/2.0.0-beta.0/${os}/${arch} | tar -xz -C /tmp/
# move to a long-term location and put it on your path
# (you'll need to set the KUBEBUILDER_ASSETS env var if you put it somewhere else)
sudo mv /tmp/kubebuilder_2.0.0-beta.0_${os}_${arch} /usr/local/kubebuilder
export PATH=$PATH:/usr/local/kubebuilder/bin
還需要裝下kustomize 這可是個渲染yaml的神器,讓helm顫抖。
go install sigs.k8s.io/kustomize/v3/cmd/kustomize
使用
注意你得先有個kubernetes集羣,一步安裝走你
創建CRD
kubebuilder init --domain sealyun.com --license apache2 --owner "fanux"
kubebuilder create api --group infra --version v1 --kind VirtulMachine
安裝CRD並啓動controller
make install # 安裝CRD
make run # 啓動controller
然後我們就可以看到創建的CRD了
# kubectl get crd
NAME AGE
virtulmachines.infra.sealyun.com 52m
來創建一個虛擬機:
# kubectl apply -f config/samples/
# kubectl get virtulmachines.infra.sealyun.com
NAME AGE
virtulmachine-sample 49m
看一眼yaml文件:
# cat config/samples/infra_v1_virtulmachine.yaml
apiVersion: infra.sealyun.com/v1
kind: VirtulMachine
metadata:
name: virtulmachine-sample
spec:
# Add fields here
foo: bar
這裏僅僅是把yaml存到etcd裏了,我們controller監聽到創建事件時啥事也沒幹。
把controller部署到集羣中
make docker-build docker-push IMG=fanux/infra-controller
make deploy
我是連的遠端的kubenetes, make docker-build時test過不去,沒有etcd的bin文件,所以先把test關了。
修改Makefile:
# docker-build: test
docker-build:
Dockerfile裏的gcr.io/distroless/static:latest
這個鏡像你也可能拉不下來,隨意改改就行,我改成了golang:1.12.7
也有可能構建時有些代碼拉不下來,啓用一下go mod vendor 把依賴打包進去
go mod vendor
如果你本地有些代碼拉不下來,可以用proxy:
export GOPROXY=https://goproxy.io
再改下Dockerfile, 註釋掉download:
修改後:
# Build the manager binary
FROM golang:1.12.7 as builder
WORKDIR /go/src/github.com/fanux/sealvm
# Copy the Go Modules manifests
COPY . .
# Build
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o manager main.go
# Use distroless as minimal base image to package the manager binary
# Refer to https://github.com/GoogleContainerTools/distroless for more details
# FROM gcr.io/distroless/static:latest
FROM golang:1.12.7
WORKDIR /
COPY --from=builder /go/src/github.com/fanux/sealvm/manager .
ENTRYPOINT ["/manager"]
make deploy
時報錯: Error: json: cannot unmarshal string into Go struct field Kustomization.patches of type types.Patch
把 config/default/kustomization.yaml
中的 patches:
改成 patchesStrategicMerge:
即可
kustomize build config/default
這個命令就渲染出了controller的yaml文件,可以體驗下
看 你的controller已經跑起來了:
kubectl get deploy -n sealvm-system
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
sealvm-controller-manager 1 1 1 0 3m
kubectl get svc -n sealvm-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
sealvm-controller-manager-metrics-service ClusterIP 10.98.71.199 <none> 8443/TCP 4m
開發
增加對象數據參數
看下config/samples下面的yaml文件:
apiVersion: infra.sealyun.com/v1
kind: VirtulMachine
metadata:
name: virtulmachine-sample
spec:
# Add fields here
foo: bar
這裏參數裏有foo:bar
, 那我們來加個虛擬CPU,內存信息:
直接api/v1/virtulmachine_types.go
即可
// VirtulMachineSpec defines the desired state of VirtulMachine
// 在這裏加信息
type VirtulMachineSpec struct {
// INSERT ADDITIONAL SPEC FIELDS - desired state of cluster
// Important: Run "make" to regenerate code after modifying this file
CPU string `json:"cpu"` // 這是我增加的
Memory string `json:"memory"`
}
// VirtulMachineStatus defines the observed state of VirtulMachine
// 在這裏加狀態信息,比如虛擬機是啓動狀態,停止狀態啥的
type VirtulMachineStatus struct {
// INSERT ADDITIONAL STATUS FIELD - define observed state of cluster
// Important: Run "make" to regenerate code after modifying this file
}
然後make一下:
make && make install && make run
這時再去渲染一下controller的yaml就會發現CRD中已經帶上CPU和內存信息了:
kustomize build config/default
properties:
cpu:
description: 'INSERT ADDITIONAL SPEC FIELDS - desired state of cluster
Important: Run "make" to regenerate code after modifying this file'
type: string
memory:
type: string
修改一下yaml:
apiVersion: infra.sealyun.com/v1
kind: VirtulMachine
metadata:
name: virtulmachine-sample
spec:
cpu: "1"
memory: "2G"
# kubectl apply -f config/samples
virtulmachine.infra.sealyun.com "virtulmachine-sample" configured
# kubectl get virtulmachines.infra.sealyun.com virtulmachine-sample -o yaml
apiVersion: infra.sealyun.com/v1
kind: VirtulMachine
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"infra.sealyun.com/v1","kind":"VirtulMachine","metadata":{"annotations":{},"name":"virtulmachine-sample","namespace":"default"},"spec":{"cpu":"1","memory":"2G"}}
creationTimestamp: 2019-07-26T08:47:34Z
generation: 2
name: virtulmachine-sample
namespace: default
resourceVersion: "14811698"
selfLink: /apis/infra.sealyun.com/v1/namespaces/default/virtulmachines/virtulmachine-sample
uid: 030e2b9a-af82-11e9-b63e-5254bc16e436
spec: # 新的CRD已生效
cpu: "1"
memory: 2G
Status 同理,就不再贅述了,比如我把status里加一個Create, 表示controller要去創建虛擬機了(主要一些控制層面的邏輯),創建完了把狀態改成Running
Reconcile 唯一需要實現的接口
controller把輪訓與事件監聽都封裝在這一個接口裏了.你不需要關心怎麼事件監聽的.
獲取虛擬機信息
func (r *VirtulMachineReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) {
ctx = context.Background()
_ = r.Log.WithValues("virtulmachine", req.NamespacedName)
vm := &v1.VirtulMachine{}
if err := r.Get(ctx, req.NamespacedName, vm); err != nil { # 獲取VM信息
log.Error(err, "unable to fetch vm")
} else {
fmt.Println(vm.Spec.CPU, vm.Spec.Memory) # 打印CPU內存信息
}
return ctrl.Result{}, nil
}
make && make install && make run
這個時候去創建一個虛擬機kubectl apply -f config/samples
,日誌裏就會輸出CPU內存了. List接口同理,我就不贅述了
r.List(ctx, &vms, client.InNamespace(req.Namespace), client.MatchingField(vmkey, req.Name))
更新狀態
在status結構體中加入狀態字段:
type VirtulMachineStatus struct {
Status string `json:"status"`
}
controller裏去更新狀態:
vm.Status.Status = "Running"
if err := r.Status().Update(ctx, vm); err != nil {
log.Error(err, "unable to update vm status")
}
如果出現:the server could not find the requested resource
這個錯誤,那麼在CRD結構體上需要加個註釋 // +kubebuilder:subresource:status
:
// +kubebuilder:subresource:status
// +kubebuilder:object:root=true
type VirtulMachine struct {
metav1.TypeMeta `json:",inline"`
metav1.ObjectMeta `json:"metadata,omitempty"`
Spec VirtulMachineSpec `json:"spec,omitempty"`
Status VirtulMachineStatus `json:"status,omitempty"`
}
這樣就好了
編譯啓動後再去apply發現狀態已經變成running:
# kubectl get virtulmachines.infra.sealyun.com virtulmachine-sample -o yaml
...
status:
status: Running
刪除
time.Sleep(time.Second * 10)
if err := r.Delete(ctx, vm); err != nil {
log.Error(err, "unable to delete vm ", "vm", vm)
}
10s之後我們將GET不到
刪除回收器 Finalizers
如果不使用Finalizers,kubectl delete 時直接就刪了etcd數據,controller再想去拿CRD時已經拿不到了:
ERRO[0029] VirtulMachine.infra.sealyun.com "virtulmachine-sample" not foundunable to fetch vm source="virtulmachine_controller.go:48"
所以在創建時我們需要給CRD加上Finalizer:
vm.ObjectMeta.Finalizers = append(vm.ObjectMeta.Finalizers, "virtulmachine.infra.sealyun.com")
然後刪除時就只會給CRD打上一個刪除時間戳,供我們做後續處理, 處理完了我們刪除掉Finalizers:
如果 DeleteionTimestamp不存在
如果沒有Finalizers
加上Finalizers,並更新CRD
要不然,說明是要被刪除的
如果存在Finalizers,刪除Finalizers,並更新CRD
看個完整的代碼示例:
if cronJob.ObjectMeta.DeletionTimestamp.IsZero() {
if !containsString(cronJob.ObjectMeta.Finalizers, myFinalizerName) {
cronJob.ObjectMeta.Finalizers = append(cronJob.ObjectMeta.Finalizers, myFinalizerName)
if err := r.Update(context.Background(), cronJob); err != nil {
return ctrl.Result{}, err
}
}
} else {
if containsString(cronJob.ObjectMeta.Finalizers, myFinalizerName) {
if err := r.deleteExternalResources(cronJob); err != nil {
return ctrl.Result{}, err
}
cronJob.ObjectMeta.Finalizers = removeString(cronJob.ObjectMeta.Finalizers, myFinalizerName)
if err := r.Update(context.Background(), cronJob); err != nil {
return ctrl.Result{}, err
}
}
}
失敗重試
假設我們A依賴B而B又後創建,那麼在處理A CRD時直接返回失敗即可,這樣很快會重試
webhook
kuberentes有三種webhook,admission webhook, authorization webhook and CRD conversion webhook.
這裏比如我們要給CRD設置一些默認值,又或者是用戶創建時少填了一些參數,那麼我們得禁止創建等等這些事。
使用webhook也非常的簡單,只需給定義的結構體實現 Defaulter
和 Validator
接口即可.
其它接口
Reconcile結構體聚合了Client接口,所以client的所有方法都是可以直接調用,大部分是對CRD object的相關操作
type Client interface {
Reader
Writer
StatusClient
}
// Reader knows how to read and list Kubernetes objects.
type Reader interface {
// Get retrieves an obj for the given object key from the Kubernetes Cluster.
// obj must be a struct pointer so that obj can be updated with the response
// returned by the Server.
Get(ctx context.Context, key ObjectKey, obj runtime.Object) error
// List retrieves list of objects for a given namespace and list options. On a
// successful call, Items field in the list will be populated with the
// result returned from the server.
List(ctx context.Context, list runtime.Object, opts ...ListOptionFunc) error
}
// Writer knows how to create, delete, and update Kubernetes objects.
type Writer interface {
// Create saves the object obj in the Kubernetes cluster.
Create(ctx context.Context, obj runtime.Object, opts ...CreateOptionFunc) error
// Delete deletes the given obj from Kubernetes cluster.
Delete(ctx context.Context, obj runtime.Object, opts ...DeleteOptionFunc) error
// Update updates the given obj in the Kubernetes cluster. obj must be a
// struct pointer so that obj can be updated with the content returned by the Server.
Update(ctx context.Context, obj runtime.Object, opts ...UpdateOptionFunc) error
// Patch patches the given obj in the Kubernetes cluster. obj must be a
// struct pointer so that obj can be updated with the content returned by the Server.
Patch(ctx context.Context, obj runtime.Object, patch Patch, opts ...PatchOptionFunc) error
}
// StatusClient knows how to create a client which can update status subresource
// for kubernetes objects.
type StatusClient interface {
Status() StatusWriter
}
掃碼關注sealyun
探討可加QQ羣:98488045