部署kube-state-metrics， kube-state-metrics用來獲取k8s集羣所有資源的狀態：
準備鏡像：
[root@hdss7-200 ~]# docker pull quay.io/coreos/kube-state-metrics:v1.5.0 v1.5.0: Pulling from coreos/kube-state-metrics cd784148e348: Pull complete f622528a393e: Pull complete Digest: sha256:b7a3143bd1eb7130759c9259073b9f239d0eeda09f5210f1cd31f1a530599ea1 Status: Downloaded newer image for quay.io/coreos/kube-state-metrics:v1.5.0 quay.io/coreos/kube-state-metrics:v1.5.0 [root@hdss7-200 ~]# docker images|grep kube-state-metrics quay.io/coreos/kube-state-metrics v1.5.0 91599517197a 15 months ago 31.8MB [root@hdss7-200 ~]# docker tag 91599517197a harbor.od.com/public/kube-state-metrics:v1.5.0 [root@hdss7-200 ~]# docker push harbor.od.com/public/kube-state-metrics:v1.5.0 The push refers to repository [harbor.od.com/public/kube-state-metrics] 5b3c36501a0a: Pushed 7bff100f35cb: Pushed v1.5.0: digest: sha256:16e9a1d63e80c19859fc1e2727ab7819f89aeae5f8ab5c3380860c2f88fe0a58 size: 739
準備資源配置清單：
[root@hdss7-200 kube-state-metrics]# cat rbac.yaml apiVersion: v1 kind: ServiceAccount metadata: labels: addonmanager.kubernetes.io/mode: Reconcile kubernetes.io/cluster-service: "true" name: kube-state-metrics namespace: kube-system --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: labels: addonmanager.kubernetes.io/mode: Reconcile kubernetes.io/cluster-service: "true" name: kube-state-metrics rules: - apiGroups: - "" resources: - configmaps - secrets - nodes - pods - services - resourcequotas - replicationcontrollers - limitranges - persistentvolumeclaims - persistentvolumes - namespaces - endpoints verbs: - list - watch - apiGroups: - policy resources: - poddisruptionbudgets verbs: - list - watch - apiGroups: - extensions resources: - daemonsets - deployments - replicasets verbs: - list - watch - apiGroups: - apps resources: - statefulsets verbs: - list - watch - apiGroups: - batch resources: - cronjobs - jobs verbs: - list - watch - apiGroups: - autoscaling resources: - horizontalpodautoscalers verbs: - list - watch --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: labels: addonmanager.kubernetes.io/mode: Reconcile kubernetes.io/cluster-service: "true" name: kube-state-metrics roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: kube-state-metrics subjects: - kind: ServiceAccount name: kube-state-metrics namespace: kube-system [root@hdss7-200 kube-state-metrics]# cat dp.yaml apiVersion: extensions/v1beta1 kind: Deployment metadata: annotations: deployment.kubernetes.io/revision: "2" labels: grafanak8sapp: "true" app: kube-state-metrics name: kube-state-metrics namespace: kube-system spec: selector: matchLabels: grafanak8sapp: "true" app: kube-state-metrics strategy: rollingUpdate: maxSurge: 25% maxUnavailable: 25% type: RollingUpdate template: metadata: labels: grafanak8sapp: "true" app: kube-state-metrics spec: containers: - name: kube-state-metrics image: harbor.od.com/public/kube-state-metrics:v1.5.0 imagePullPolicy: IfNotPresent ports: - containerPort: 8080 name: http-metrics protocol: TCP readinessProbe: failureThreshold: 3 httpGet: path: /healthz port: 8080 scheme: HTTP initialDelaySeconds: 5 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 5 serviceAccountName: kube-state-metrics
應用資源配置清單：
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/kube-state-metrics/rbac.yaml serviceaccount/kube-state-metrics created clusterrole.rbac.authorization.k8s.io/kube-state-metrics created clusterrolebinding.rbac.authorization.k8s.io/kube-state-metrics created [root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/kube-state-metrics/dp.yaml deployment.extensions/kube-state-metrics created [root@hdss7-21 ~]# kubectl get pod -n kube-system NAME READY STATUS RESTARTS AGE coredns-6b6c4f9648-rrgfx 1/1 Running 0 49m kube-state-metrics-8669f776c6-gb6nd 0/1 Running 0 15s kubernetes-dashboard-76dcdb4677-c847s 1/1 Running 0 38m traefik-ingress-h2jpc 1/1 Running 0 2d traefik-ingress-k5hgk 1/1 Running 0 2d [root@hdss7-21 ~]# curl 172.7.21.9:8080/healthz ok
部署node-exporter，node-exporter是幫我們監控宿主機的資源：
準備鏡像：
[root@hdss7-200 ~]# docker pull prom/node-exporter:v0.15.0 v0.15.0: Pulling from prom/node-exporter Image docker.io/prom/node-exporter:v0.15.0 uses outdated schema1 manifest format. Please upgrade to a schema2 image for better future compatibility. More information at https://docs.docker.com/registry/spec/deprecated-schema-v1/ aa3e9481fcae: Pull complete a3ed95caeb02: Pull complete afc308b02dc6: Pull complete 4cafbffc9d4f: Pull complete Digest: sha256:a59d1f22610da43490532d5398b3911c90bfa915951d3b3e5c12d3c0bf8771c3 Status: Downloaded newer image for prom/node-exporter:v0.15.0 docker.io/prom/node-exporter:v0.15.0 [root@hdss7-200 ~]# docker images|grep node-exporter:v0.15.0 [root@hdss7-200 ~]# docker images|grep node-exporter prom/node-exporter v0.15.0 12d51ffa2b22 2 years ago 22.8MB [root@hdss7-200 ~]# docker tag 12d51ffa2b22 harbor.od.com/public/node-exporter:v0.15.0 [root@hdss7-200 ~]# docker push harbor.od.com/public/node-exporter:v0.15.0 The push refers to repository [harbor.od.com/public/node-exporter] 5f70bf18a086: Mounted from public/pause 1c7f6350717e: Pushed a349adf62fe1: Pushed c7300f623e77: Pushed v0.15.0: digest: sha256:57d9b335b593e4d0da1477d7c5c05f23d9c3dc6023b3e733deb627076d4596ed size: 1979
準備資源配置清單：
[root@hdss7-200 node-exporter]# cat ds.yaml kind: DaemonSet apiVersion: extensions/v1beta1 metadata: name: node-exporter namespace: kube-system labels: daemon: "node-exporter" grafanak8sapp: "true" spec: selector: matchLabels: daemon: "node-exporter" grafanak8sapp: "true" template: metadata: name: node-exporter labels: daemon: "node-exporter" grafanak8sapp: "true" spec: volumes: - name: proc hostPath: path: /proc type: "" - name: sys hostPath: path: /sys type: "" containers: - name: node-exporter image: harbor.od.com/public/node-exporter:v0.15.0 imagePullPolicy: IfNotPresent args: - --path.procfs=/host_proc - --path.sysfs=/host_sys ports: - name: node-exporter hostPort: 9100 containerPort: 9100 protocol: TCP volumeMounts: - name: sys readOnly: true mountPath: /host_sys - name: proc readOnly: true mountPath: /host_proc hostNetwork: true [root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/node-exporter/ds.yaml daemonset.extensions/node-exporter created
因爲node-exporter部署的方式是daemonset，在每個運算節點都會運行一個POD
[root@hdss7-21 ~]# kubectl get pod -n kube-system -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES coredns-6b6c4f9648-rrgfx 1/1 Running 0 63m 172.7.22.7 hdss7-22.host.com <none> <none> kube-state-metrics-8669f776c6-gb6nd 1/1 Running 0 14m 172.7.21.9 hdss7-21.host.com <none> <none> kubernetes-dashboard-76dcdb4677-c847s 1/1 Running 0 52m 172.7.22.10 hdss7-22.host.com <none> <none> node-exporter-gn2hb 1/1 Running 0 3m 10.4.7.22 hdss7-22.host.com <none> <none> node-exporter-nzww6 1/1 Running 0 3m 10.4.7.21 hdss7-21.host.com <none> <none> traefik-ingress-h2jpc 1/1 Running 0 2d1h 172.7.21.3 hdss7-21.host.com <none> <none> traefik-ingress-k5hgk 1/1 Running 0 2d 172.7.22.2 hdss7-22.host.com <none> <none>
部署cadvisor，cadvisor是幫我們去向kubelet要每個容器所消耗的資源：
準備鏡像：
[root@hdss7-200 node-exporter]# docker pull google/cadvisor:v0.28.3 v0.28.3: Pulling from google/cadvisor ab7e51e37a18: Pull complete a2dc2f1bce51: Pull complete 3b017de60d4f: Pull complete Digest: sha256:9e347affc725efd3bfe95aa69362cf833aa810f84e6cb9eed1cb65c35216632a Status: Downloaded newer image for google/cadvisor:v0.28.3 docker.io/google/cadvisor:v0.28.3 [root@hdss7-200 node-exporter]# [root@hdss7-200 node-exporter]# [root@hdss7-200 node-exporter]# docker images|grep google/cadvisor google/cadvisor v0.28.3 75f88e3ec333 2 years ago 62.2MB [root@hdss7-200 node-exporter]# docker tag 75f88e3ec333 harbor.od.com/public/cadvisor:v0.28.3 [root@hdss7-200 node-exporter]# docker push harbor.od.com/public/cadvisor:v0.28.3 The push refers to repository [harbor.od.com/public/cadvisor] f60e27acaccf: Pushed f04a25da66bf: Pushed 52a5560f4ca0: Pushed v0.28.3: digest: sha256:34d9d683086d7f3b9bbdab0d1df4518b230448896fa823f7a6cf75f66d64ebe1 size: 951
修改所有運算節點軟連接：
mount -o remount,rw /sys/fs/cgroup/ ln -s /sys/fs/cgroup/cpu,cpuacct /sys/fs/cgroup/cpuacct,cpu
應用資源配置清單：
[root@hdss7-22 ~]# kubectl apply -f http://k8s-yaml.od.com/cadvisor/ds.yaml daemonset.apps/cadvisor created [root@hdss7-22 ~]# kubectl -n kube-system get pod NAME READY STATUS RESTARTS AGE cadvisor-cdtzg 1/1 Running 0 12s cadvisor-pbmb7 1/1 Running 0 12s coredns-6b6c4f9648-rrgfx 1/1 Running 0 82m kube-state-metrics-8669f776c6-gb6nd 1/1 Running 0 33m kubernetes-dashboard-76dcdb4677-c847s 1/1 Running 0 71m node-exporter-gn2hb 1/1 Running 0 22m node-exporter-nzww6 1/1 Running 0 22m traefik-ingress-h2jpc 1/1 Running 0 2d1h traefik-ingress-k5hgk 1/1 Running 0 2d1h
部署blackbox-exporter：
[root@hdss7-200 cadvisor]# docker pull prom/blackbox-exporter:v0.15.1 v0.15.1: Pulling from prom/blackbox-exporter 8e674ad76dce: Pull complete e77d2419d1c2: Pull complete 969c24328c68: Pull complete d9df4d63dd8a: Pull complete Digest: sha256:0ccbb0bb08bbc00f1c765572545e9372a4e4e4dc9bafffb1a962024f61d6d996 Status: Downloaded newer image for prom/blackbox-exporter:v0.15.1 docker.io/prom/blackbox-exporter:v0.15.1 [root@hdss7-200 cadvisor]# docker images|grep black prom/blackbox-exporter v0.15.1 81b70b6158be 6 months ago 19.7MB [root@hdss7-200 cadvisor]# docker tag 81b70b6158be harbor.od.com/public/blackbox-exporter:v0.15.1 [root@hdss7-200 cadvisor]# docker push harbor.od.com/public/blackbox-exporter:v0.15.1 The push refers to repository [harbor.od.com/public/blackbox-exporter] 2e93bab0c159: Pushed 4f2b5ab68d7f: Pushed 3163e6173fcc: Pushed 6194458b07fc: Pushed v0.15.1: digest: sha256:f7c335cc7898c6023346a0d5fba8566aca4703b69d63be8dc5367476c77cf2c4 size: 1155
準備資源配置清單：
[root@hdss7-200 blackbox-exporter]# cat cm.yaml apiVersion: v1 kind: ConfigMap metadata: labels: app: blackbox-exporter name: blackbox-exporter namespace: kube-system data: blackbox.yml: |- modules: http_2xx: prober: http timeout: 2s http: valid_http_versions: ["HTTP/1.1", "HTTP/2"] valid_status_codes: [200,301,302] method: GET preferred_ip_protocol: "ip4" tcp_connect: prober: tcp timeout: 2s [root@hdss7-200 blackbox-exporter]# cat dp.yaml kind: Deployment apiVersion: extensions/v1beta1 metadata: name: blackbox-exporter namespace: kube-system labels: app: blackbox-exporter annotations: deployment.kubernetes.io/revision: 1 spec: replicas: 1 selector: matchLabels: app: blackbox-exporter template: metadata: labels: app: blackbox-exporter spec: volumes: - name: config configMap: name: blackbox-exporter defaultMode: 420 containers: - name: blackbox-exporter image: harbor.od.com/public/blackbox-exporter:v0.15.1 imagePullPolicy: IfNotPresent args: - --config.file=/etc/blackbox_exporter/blackbox.yml - --log.level=info - --web.listen-address=:9115 ports: - name: blackbox-port containerPort: 9115 protocol: TCP resources: limits: cpu: 200m memory: 256Mi requests: cpu: 100m memory: 50Mi volumeMounts: - name: config mountPath: /etc/blackbox_exporter readinessProbe: tcpSocket: port: 9115 initialDelaySeconds: 5 timeoutSeconds: 5 periodSeconds: 10 successThreshold: 1 failureThreshold: 3 [root@hdss7-200 blackbox-exporter]# cat svc.yaml kind: Service apiVersion: v1 metadata: name: blackbox-exporter namespace: kube-system spec: selector: app: blackbox-exporter ports: - name: blackbox-port protocol: TCP port: 9115 [root@hdss7-200 blackbox-exporter]# cat ingress.yaml apiVersion: extensions/v1beta1 kind: Ingress metadata: name: blackbox-exporter namespace: kube-system spec: rules: - host: blackbox.od.com http: paths: - path: / backend: serviceName: blackbox-exporter servicePort: blackbox-port
應用資源配置清單：
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/blackbox-exporter/cm.yaml configmap/blackbox-exporter created [root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/blackbox-exporter/dp.yaml deployment.extensions/blackbox-exporter created [root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/blackbox-exporter/svc.yaml service/blackbox-exporter created [root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/blackbox-exporter/ingress.yaml ingress.extensions/blackbox-exporter created [root@hdss7-21 ~]# kubectl get pod -n kube-system -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES blackbox-exporter-659fc46b55-p8wxp 0/1 Running 0 19s 172.7.22.11 hdss7-22.host.com <none> <none> cadvisor-cdtzg 1/1 Running 0 15m 10.4.7.21 hdss7-21.host.com <none> <none> cadvisor-pbmb7 1/1 Running 0 15m 10.4.7.22 hdss7-22.host.com <none> <none> coredns-6b6c4f9648-rrgfx 1/1 Running 0 97m 172.7.22.7 hdss7-22.host.com <none> <none>
