Kebernetes 學習總結(11) K8s 資源監控

kubernetes新一代的監控模型由：核心指標流水線和第三方非核心監控流水線組成。核心指標流水線由kubelet、metric-server 以及由API-server提供的API組成；負責CPU累積使用率、內存實時使用率、POD資源佔用率、Container磁盤佔用率等。而第三方非核心監控流水線負責從OS收集各種指標數據並提供給終端用戶、存儲系統、以及HPA等。
監控系統收集兩種指標：資源指標與自定義指標。

Metrics-server 是資源指標API 。它提供核心指標，包括CPU累積使用率、內存實時使用率、Pod 的資源佔用率及容器的磁盤佔用率。這些指標由kubelet、metrics-server以及由API server提供的。
Prometheus是自定義指標的提供者。它收集的數據還需要經過kube-state-metrics轉換處理，再由 k8s-prometheus-adapter 輸出爲metrics-api 才能被 kubernetes cluster 所讀取。用於從系統收集各種指標數據，並經過處理提供給終端用戶、存儲系統以及HPA，這些數據包括核心指標和許多非核心指標。
資源指標API 負責收集各種資源指標，但它需要擴展APIServer 。可以利用 aggregator 將 metrics-server 與 APIServer進行聚合，達到擴展功能的效果。這樣就可以利用擴展的 API Server 功能(即資源指標API)進行收集各種資源指標(1.8+支持)。kubectl top 、HPA等功能組件必須依賴資源指標API （早期版本它們依賴heapster）。
HPA 根據CPU、Memory、IO、net connections等指標進行擴展或收縮(早期的heapster只能提供CPU、Memory指標)
一、metrics-server
是託管在kubernetes cluster上的一個Pod ，再由 kube-aggregator 將它和原API Server 進行聚合，達到擴展API 的效果。它是現在 kubectl top 、HPA的前提依賴。
部署metrics-server 如下：
參考：https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/metrics-server

[root@k8s-master-dev metric-v0.3]# cat metrics-server.yaml
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: metrics-server
  namespace: kube-system

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: system:metrics-server
rules:
- apiGroups:
  - ""
  resources:
  - pods
  - nodes
  - nodes/stats
  - namespaces
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - "extensions"
  resources:
  - deployments
  verbs:
  - get
  - list
  - watch

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: system:metrics-server
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:metrics-server
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system

---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: metrics-server-auth-reader
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: extension-apiserver-authentication-reader
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: metrics-server:system:auth-delegator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system

---
apiVersion: apiregistration.k8s.io/v1beta1
kind: APIService
metadata:
  name: v1beta1.metrics.k8s.io
spec:
  service:
    name: metrics-server
    namespace: kube-system
  group: metrics.k8s.io
  version: v1beta1
  insecureSkipTLSVerify: true
  groupPriorityMinimum: 100
  versionPriority: 100

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: metrics-server
  namespace: kube-system
  labels:
    k8s-app: metrics-server
spec:
  selector:
    matchLabels:
      k8s-app: metrics-server
  template:
    metadata:
      name: metrics-server
      labels:
        k8s-app: metrics-server
    spec:
      serviceAccountName: metrics-server
      volumes:
      # mount in tmp so we can safely use from-scratch images and/or read-only containers
      - name: tmp-dir
        emptyDir: {}
      containers:
      - name: metrics-server
        image: k8s.gcr.io/metrics-server-amd64:v0.3.0
        imagePullPolicy: IfNotPresent
        command:
        - /metrics-server
        - --kubelet-insecure-tls
        - --kubelet-preferred-address-types=InternalIP
        volumeMounts:
        - name: tmp-dir
          mountPath: /tmp

---
apiVersion: v1
kind: Service
metadata:
  name: metrics-server
  namespace: kube-system
  labels:
    kubernetes.io/name: "Metrics-server"
spec:
  selector:
    k8s-app: metrics-server
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443
[root@k8s-master-dev metric-v0.3]# kubectl apply -f metrics-server.yaml 
[root@k8s-master-dev metric-v0.3]# cd
[root@k8s-master-dev ~]# kubectl api-versions
admissionregistration.k8s.io/v1beta1
apiextensions.k8s.io/v1beta1
apiregistration.k8s.io/v1
apiregistration.k8s.io/v1beta1
apps/v1
apps/v1beta1
apps/v1beta2
authentication.k8s.io/v1
authentication.k8s.io/v1beta1
authorization.k8s.io/v1
authorization.k8s.io/v1beta1
autoscaling/v1
autoscaling/v2beta1
batch/v1
batch/v1beta1
certificates.k8s.io/v1beta1
custom.metrics.k8s.io/v1beta1
events.k8s.io/v1beta1
extensions/v1beta1
*metrics.k8s.io/v1beta1*
networking.k8s.io/v1
policy/v1beta1
rbac.authorization.k8s.io/v1
rbac.authorization.k8s.io/v1beta1
scheduling.k8s.io/v1beta1
storage.k8s.io/v1
storage.k8s.io/v1beta1
v1
[root@k8s-master-dev ~]#
[root@k8s-master-dev ~]# kubectl top nodes
NAME             CPU(cores)   CPU%      MEMORY(bytes)   MEMORY%
k8s-master-dev   299m         3%        1884Mi          11%
k8s-node1-dev    125m         1%        4181Mi          26%
k8s-node2-dev    66m          3%        2736Mi          17%
k8s-node3-dev    145m         1%        2686Mi          34%
[root@k8s-master-dev metric-v0.3]# kubectl top pods
NAME      CPU(cores)   MEMORY(bytes)
mongo-0   12m          275Mi
mongo-1   11m          251Mi
mongo-2   8m           271Mi
[root@k8s-master-dev metric-v0.3]#

當metrics-server部署完畢後，如上所示可以查看到 metrics相關的API，並且可以使用kubectl top 命令查看node或pod的資源佔用情況。
二、Prometheus
架構圖如下：

Prometheus 通過node_exporter獲取各Nodes的信息。 node_exporter它只負責節點級別的信息彙總，如果需要採集其它指標數據，就需要部署專用的exporter 。
Prometheus 通過 metrics-url 地址到各Pods獲取數據。
prometheus 提供了一個Restful 風格的PromQL接口，可以讓用戶輸入查詢表達式。但K8s的 API Server 無法查詢其值，因爲它們默認的數據格式不統一。數據需要kube-state-metrics組件將其處理、轉換，然後由k8s-prometheus-adapter組件讀取並聚合到API上，最後 kubernetes cluster 的API server 才能識別。
所以各節點需要部署node_exporter 組件，然後Prometheus從各節點的node_exporter上獲取infomation，然後就可以通過 PromQL 查詢各種數據。這些數據的格式再由kube-state-metrics組件進行轉換，然後再由kube-prometheus-adapter組件將轉換後的數據輸出爲Custom metrics API ，並聚合到API上，以便用戶使用。
示意圖如下

部署Prometheus, 如下：
參考：https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/prometheus
1) 定義名稱空間

[root@k8s-master-dev prometheus]# cd k8s-prom/
[root@k8s-master-dev k8s-prom]#
[root@k8s-master-dev k8s-prom]# ls
k8s-prometheus-adapter  namespace.yaml  podinfo     README.md
kube-state-metrics      node_exporter   prometheus
[root@k8s-master-dev k8s-prom]# cat namespace.yaml
---
apiVersion: v1
kind: Namespace
metadata:
  name: prom
[root@k8s-master-dev k8s-prom]# kubectl apply -f namespace.yaml
namespace/prom created

2) 部署node_exporter

[root@k8s-master-dev k8s-prom]# cd node_exporter/
[root@k8s-master-dev node_exporter]# ls
node-exporter-ds.yaml  node-exporter-svc.yaml
[root@k8s-master-dev node_exporter]# vim node-exporter-ds.yaml
[root@k8s-master-dev node_exporter]# kubectl apply -f ./
daemonset.apps/prometheus-node-exporter created
service/prometheus-node-exporter created
[root@k8s-master-dev node_exporter]# kubectl get pods -n prom
NAME                             READY     STATUS    RESTARTS   AGE
prometheus-node-exporter-7729r   1/1       Running   0          17s
prometheus-node-exporter-hhc7f   1/1       Running   0          17s
prometheus-node-exporter-jxjcq   1/1       Running   0          17s
prometheus-node-exporter-pswbb   1/1       Running   0          17s
[root@k8s-master-dev node_exporter]# cd ..

3) 部署prometheus

[root@k8s-master-dev k8s-prom]# cd prometheus/
[root@k8s-master-dev prometheus]# ls
prometheus-cfg.yaml  prometheus-deploy.yaml  prometheus-rbac.yaml  prometheus-svc.yaml
[root@k8s-master-dev prometheus]# kubectl apply -f ./
configmap/prometheus-config created
deployment.apps/prometheus-server created
clusterrole.rbac.authorization.k8s.io/prometheus created
serviceaccount/prometheus created
clusterrolebinding.rbac.authorization.k8s.io/prometheus created
service/prometheus created
[root@k8s-master-dev prometheus]# kubectl get all -n prom
NAME                                     READY     STATUS    RESTARTS   AGE
pod/prometheus-node-exporter-7729r       1/1       Running   0          1m
pod/prometheus-node-exporter-hhc7f       1/1       Running   0          1m
pod/prometheus-node-exporter-jxjcq       1/1       Running   0          1m
pod/prometheus-node-exporter-pswbb       1/1       Running   0          1m
pod/prometheus-server-65f5d59585-5fj6n   1/1       Running   0          33s

NAME                               TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)          AGE
service/prometheus                 NodePort    10.98.96.66   <none>        9090:30090/TCP   34s
service/prometheus-node-exporter   ClusterIP   None          <none>        9100/TCP         1m

NAME                                      DESIRED   CURRENT   READY     UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/prometheus-node-exporter   4         4         4         4            4           <none>          1m

NAME                                DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/prometheus-server   1         1         1            1           34s

NAME                                           DESIRED   CURRENT   READY     AGE
replicaset.apps/prometheus-server-65f5d59585   1         1         1         34s
[root@k8s-master-dev prometheus]#

然後就可以以PromQL的方式查詢數據了，如下所示：

4) 部署kube-state-metrics

[root@k8s-master-dev prometheus]# cd ..
[root@k8s-master-dev k8s-prom]# cd kube-state-metrics/
[root@k8s-master-dev kube-state-metrics]# ls
kube-state-metrics-deploy.yaml  kube-state-metrics-rbac.yaml  kube-state-metrics-svc.yaml
[root@k8s-master-dev kube-state-metrics]# kubectl apply -f ./
deployment.apps/kube-state-metrics created
serviceaccount/kube-state-metrics created
clusterrole.rbac.authorization.k8s.io/kube-state-metrics created
clusterrolebinding.rbac.authorization.k8s.io/kube-state-metrics created
service/kube-state-metrics created
[root@k8s-master-dev kube-state-metrics]#
[root@k8s-master-dev kube-state-metrics]# kubectl get all -n prom
NAME                                      READY     STATUS             RESTARTS   AGE
pod/kube-state-metrics-58dffdf67d-j4jdv   0/1       Running   0          34s
pod/prometheus-node-exporter-7729r        1/1       Running            0          3m
pod/prometheus-node-exporter-hhc7f        1/1       Running            0          3m
pod/prometheus-node-exporter-jxjcq        1/1       Running            0          3m
pod/prometheus-node-exporter-pswbb        1/1       Running            0          3m
pod/prometheus-server-65f5d59585-5fj6n    1/1       Running            0          2m

NAME                               TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
service/kube-state-metrics         ClusterIP   10.108.165.171   <none>        8080/TCP         35s
service/prometheus                 NodePort    10.98.96.66      <none>        9090:30090/TCP   2m
service/prometheus-node-exporter   ClusterIP   None             <none>        9100/TCP         3m

NAME                                      DESIRED   CURRENT   READY     UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/prometheus-node-exporter   4         4         4         4            4           <none>          3m

NAME                                 DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/kube-state-metrics   1         1         1            0           35s
deployment.apps/prometheus-server    1         1         1            1           2m

NAME                                            DESIRED   CURRENT   READY     AGE
replicaset.apps/kube-state-metrics-58dffdf67d   1         1         0         35s
replicaset.apps/prometheus-server-65f5d59585    1         1         1         2m
[root@k8s-master-dev kube-state-metrics]# cd ..

5) 部署prometheus-adapter
參考：https://github.com/DirectXMan12/k8s-prometheus-adapter/tree/master/deploy

[root@k8s-master-dev k8s-prom]# cd k8s-prometheus-adapter/
[root@k8s-master-dev k8s-prometheus-adapter]# ls
custom-metrics-apiserver-auth-delegator-cluster-role-binding.yaml   custom-metrics-apiserver-service.yaml
custom-metrics-apiserver-auth-reader-role-binding.yaml              custom-metrics-apiservice.yaml
custom-metrics-apiserver-deployment.yaml                            custom-metrics-cluster-role.yaml
custom-metrics-apiserver-deployment.yaml.bak                        custom-metrics-config-map.yaml
custom-metrics-apiserver-resource-reader-cluster-role-binding.yaml  custom-metrics-resource-reader-cluster-role.yaml
custom-metrics-apiserver-service-account.yaml                       hpa-custom-metrics-cluster-role-binding.yaml
[root@k8s-master-dev k8s-prometheus-adapter]# grep secretName custom-metrics-apiserver-deployment.yaml
          secretName: cm-adapter-serving-certs

[root@k8s-master-dev k8s-prometheus-adapter]# cd /etc/kubernetes/pki/
[root@k8s-master-dev pki]# (umask 077; openssl genrsa -out serving.key 2048)
Generating RSA private key, 2048 bit long modulus
.....................+++
..........+++
e is 65537 (0x10001)
[root@k8s-master-dev pki]#
[root@k8s-master-dev pki]# openssl req -new -key serving.key -out serving.csr -subj "/CN=serving"
[root@k8s-master-dev pki]# openssl x509 -req -in serving.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out serving.crt -days 3650
Signature ok
subject=/CN=serving
Getting CA Private Key
[root@k8s-master-dev pki]# kubectl create secret generic cm-adapter-serving-certs --from-file=serving.crt=./serving.crt --from-file=serving.key=./serving.key -n prom
secret/cm-adapter-serving-certs created
[root@k8s-master-dev pki]#  kubectl get secret -n prom
NAME                             TYPE                                  DATA      AGE
cm-adapter-serving-certs         Opaque                                2         9s
default-token-w4f44              kubernetes.io/service-account-token   3         8m
kube-state-metrics-token-dfcmf   kubernetes.io/service-account-token   3         4m
prometheus-token-4lb78           kubernetes.io/service-account-token   3         6m
[root@k8s-master-dev pki]#

[root@k8s-master-dev pki]# cd -
/root/manifests/prometheus/k8s-prom/k8s-prometheus-adapter
[root@k8s-master-dev k8s-prometheus-adapter]# ls custom-metrics-config-map.yaml
custom-metrics-config-map.yaml
[root@k8s-master-dev k8s-prometheus-adapter]# cat custom-metrics-config-map.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: adapter-config
  namespace: prom
data:
  config.yaml: |
    rules:
    - seriesQuery: '{__name__=~"^container_.*",container_name!="POD",namespace!="",pod_name!=""}'
      seriesFilters: []
      resources:
        overrides:
          namespace:
            resource: namespace
          pod_name:
            resource: pod
      name:
        matches: ^container_(.*)_seconds_total$
        as: ""
      metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>,container_name!="POD"}[5m]))
        by (<<.GroupBy>>)
    - seriesQuery: '{__name__=~"^container_.*",container_name!="POD",namespace!="",pod_name!=""}'
      seriesFilters:
      - isNot: ^container_.*_seconds_total$
      resources:
        overrides:
          namespace:
            resource: namespace
          pod_name:
            resource: pod
      name:
        matches: ^container_(.*)_total$
        as: ""
      metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>,container_name!="POD"}[5m]))
        by (<<.GroupBy>>)
    - seriesQuery: '{__name__=~"^container_.*",container_name!="POD",namespace!="",pod_name!=""}'
      seriesFilters:
      - isNot: ^container_.*_total$
      resources:
        overrides:
          namespace:
            resource: namespace
          pod_name:
            resource: pod
      name:
        matches: ^container_(.*)$
        as: ""
      metricsQuery: sum(<<.Series>>{<<.LabelMatchers>>,container_name!="POD"}) by (<<.GroupBy>>)
    - seriesQuery: '{namespace!="",__name__!~"^container_.*"}'
      seriesFilters:
      - isNot: .*_total$
      resources:
        template: <<.Resource>>
      name:
        matches: ""
        as: ""
      metricsQuery: sum(<<.Series>>{<<.LabelMatchers>>}) by (<<.GroupBy>>)
    - seriesQuery: '{namespace!="",__name__!~"^container_.*"}'
      seriesFilters:
      - isNot: .*_seconds_total
      resources:
        template: <<.Resource>>
      name:
        matches: ^(.*)_total$
        as: ""
      metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>}[5m])) by (<<.GroupBy>>)
    - seriesQuery: '{namespace!="",__name__!~"^container_.*"}'
      seriesFilters: []
      resources:
        template: <<.Resource>>
      name:
        matches: ^(.*)_seconds_total$
        as: ""
      metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>}[5m])) by (<<.GroupBy>>)
[root@k8s-master-dev k8s-prometheus-adapter]#  grep namespace custom-metrics-apiserver-deployment.yaml
  namespace: prom
[root@k8s-master-dev k8s-prometheus-adapter]# kubectl apply -f ./
clusterrolebinding.rbac.authorization.k8s.io/custom-metrics:system:auth-delegator created
rolebinding.rbac.authorization.k8s.io/custom-metrics-auth-reader created
deployment.apps/custom-metrics-apiserver created
clusterrolebinding.rbac.authorization.k8s.io/custom-metrics-resource-reader created
serviceaccount/custom-metrics-apiserver created
service/custom-metrics-apiserver created
apiservice.apiregistration.k8s.io/v1beta1.custom.metrics.k8s.io created
clusterrole.rbac.authorization.k8s.io/custom-metrics-server-resources created
configmap/adapter-config created
clusterrole.rbac.authorization.k8s.io/custom-metrics-resource-reader created
clusterrolebinding.rbac.authorization.k8s.io/hpa-controller-custom-metrics created
[root@k8s-master-dev k8s-prometheus-adapter]# kubectl get cm -n prom
NAME                DATA      AGE
adapter-config      1         21s
prometheus-config   1         21m
[root@k8s-master-dev k8s-prometheus-adapter]# kubectl get all -n prom
NAME                                           READY     STATUS             RESTARTS   AGE
pod/custom-metrics-apiserver-65f545496-2hfvb   1/1       Running            0          40s
pod/kube-state-metrics-58dffdf67d-j4jdv        0/1       Running   0          20m
pod/prometheus-node-exporter-7729r             1/1       Running            0          23m
pod/prometheus-node-exporter-hhc7f             1/1       Running            0          23m
pod/prometheus-node-exporter-jxjcq             1/1       Running            0          23m
pod/prometheus-node-exporter-pswbb             1/1       Running            0          23m
pod/prometheus-server-65f5d59585-5fj6n         1/1       Running            0          22m

NAME                               TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
service/custom-metrics-apiserver   ClusterIP   10.100.7.28      <none>        443/TCP          41s
service/kube-state-metrics         ClusterIP   10.108.165.171   <none>        8080/TCP         20m
service/prometheus                 NodePort    10.98.96.66      <none>        9090:30090/TCP   22m
service/prometheus-node-exporter   ClusterIP   None             <none>        9100/TCP         23m

NAME                                      DESIRED   CURRENT   READY     UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/prometheus-node-exporter   4         4         4         4            4           <none>          23m

NAME                                       DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/custom-metrics-apiserver   1         1         1            1           42s
deployment.apps/kube-state-metrics         1         1         1            0           20m
deployment.apps/prometheus-server          1         1         1            1           22m

NAME                                                 DESIRED   CURRENT   READY     AGE
replicaset.apps/custom-metrics-apiserver-65f545496   1         1         1         42s
replicaset.apps/kube-state-metrics-58dffdf67d        1         1         0         20m
replicaset.apps/prometheus-server-65f5d59585         1         1         1         22m
[root@k8s-master-dev k8s-prometheus-adapter]#
[root@k8s-master-dev k8s-prometheus-adapter]# kubectl api-versions | grep custom
custom.metrics.k8s.io/v1beta1
[root@k8s-master-dev k8s-prometheus-adapter]#

三、Grafana
grafana 是一個可視化面板，有着非常漂亮的圖表和佈局展示，功能齊全的度量儀表盤和圖形編輯器，支持 Graphite、zabbix、InfluxDB、Prometheus、OpenTSDB、Elasticsearch 等作爲數據源，比 Prometheus 自帶的圖表展示功能強大太多，更加靈活，有豐富的插件，功能更加強大。(使用promQL語句查詢出了一些數據，並且在 Prometheus 的 Dashboard 中進行了展示，但是明顯可以感覺到 Prometheus 的圖表功能相對較弱，所以一般會使用第三方的工具展示這些數據，例Grafana)
部署Grafana
參考：https://github.com/kubernetes/heapster/tree/master/deploy/kube-config/influxdb

[root@k8s-master-dev prometheus]# ls
grafana  k8s-prom
[root@k8s-master-dev prometheus]# cd grafana/
[root@k8s-master-dev grafana]# head -11 grafana.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: monitoring-grafana
  namespace: prom
spec:
  replicas: 1
  selector:
    matchLabels:
      task: monitoring
      k8s-app: grafana
[root@k8s-master-dev grafana]# tail -2 grafana.yaml
    k8s-app: grafana
  type: NodePort
[root@k8s-master-dev grafana]# kubectl apply -f grafana.yaml
deployment.apps/monitoring-grafana created
service/monitoring-grafana created
[root@k8s-master-dev grafana]#  kubectl get pods -n prom
NAME                                       READY     STATUS              RESTARTS   AGE
custom-metrics-apiserver-65f545496-2hfvb   1/1       Running             0          13m
kube-state-metrics-58dffdf67d-j4jdv        1/1       Running             0          32m
monitoring-grafana-ffb4d59bd-w9lg9         0/1       Running   0          8s
prometheus-node-exporter-7729r             1/1       Running             0          35m
prometheus-node-exporter-hhc7f             1/1       Running             0          35m
prometheus-node-exporter-jxjcq             1/1       Running             0          35m
prometheus-node-exporter-pswbb             1/1       Running             0          35m
prometheus-server-65f5d59585-5fj6n         1/1       Running             0          34m
[root@k8s-master-dev grafana]# kubectl get svc -n prom
NAME                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
custom-metrics-apiserver   ClusterIP   10.100.7.28      <none>        443/TCP          13m
kube-state-metrics         ClusterIP   10.108.165.171   <none>        8080/TCP         32m
monitoring-grafana         NodePort    10.100.131.108   <none>        80:42690/TCP     22s
prometheus                 NodePort    10.98.96.66      <none>        9090:30090/TCP   34m
prometheus-node-exporter   ClusterIP   None             <none>        9100/TCP         35m
[root@k8s-master-dev grafana]#

Grafana的使用，默認用戶名密碼都是admin，登錄後首先添加數據源

指定Prometheus 的數據源從哪個PromQL URL獲取：

然後導入Dashboard (Dashboard可以在https://grafana.com/dashboards下載)

Grafana暫不熟悉，回頭再續...

Kebernetes 學習總結(11) K8s 資源監控

Android啓動過程-萬字長文(Android14)

optional install error: Error: Unsupported URL Type: npm:vue-loader@^16.1.0

這種嵌套字典類型的數據，我想把它讀取到df裏，如何操作？

【SQL進階】CASE語句的使用

微調真的能讓LLM學到新東西嗎:引入新知識可能讓模型產生更多的幻覺

iNeuOS工業互聯網操作系統，增加電力IEC104協議

微服務實踐k8s&dapr開發部署實驗（3）訂閱發佈

chromedriver版本

kbgressdb之數據結構V0.2

Kubernetes 學習總結(12) scheduler

Kebernetes 學習總結(11) K8s 網絡模型-canal策略控制(3)

Kebernetes 學習總結(11) K8s 網絡模型分析(1)

Kebernetes 學習總結(11) K8s 網絡模型分析(2)

Kebernetes 學習總結(11) K8s 資源監控

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結