本文分享自華爲雲社區《根據Nginx Ingress指標對指定後端進行HPA》,作者: 可以交個朋友。
背景
生產場景下,Nginx Ingress的流量會通過域名和path路徑最終轉發至不同的應用,而有時候cpu和內存並不是nginx的性能瓶頸,此時可以基於nginx_ingress_controller_requests
指標,爲其對應的應用配置HPA,以實現基於不同域名和path的請求量彈性指定後端工作負載
簡介
環境準備
- nginx-ingress已部署
- 雲原生監控插件kube-prometheus-stack已安裝(server模式),插件默認監控nginx-ingress,開源環境請自行配置監控。
- 已配置kubectl命令或使用cloudshell
注意:由於HPA規則中scaleTargetRef
和describedObject
兩個字段都無法指定命名空間,所以指標來源、HPA和彈性目標需在同一命名空間,而nginx-ingress和業務工作負載一般處在不同命名空間;本次方案採用external類型的HPA,可以忽略指標來源的命名空間
操作步驟
創建演示需要的彈性目標工作負載,service以及ingress
apiVersion: apps/v1 kind: Deployment metadata: name: test-hpa labels: app: test-app spec: replicas: 1 selector: matchLabels: app: test-app template: metadata: labels: app: test-app spec: containers: - image: skto/sample-app:v2 name: metrics-provider ports: - name: http containerPort: 8080 --- apiVersion: v1 kind: Service metadata: name: test-app namespace: default labels: app: test-app spec: ports: - port: 8080 name: http protocol: TCP targetPort: 8080 selector: app: test-app type: ClusterIP --- apiVersion: apps/v1 kind: Deployment metadata: name: sample-app labels: app: sample-app spec: replicas: 1 selector: matchLabels: app: sample-app template: metadata: labels: app: sample-app spec: containers: - image: skto/sample-app:v2 name: metrics-provider ports: - name: http containerPort: 8080 --- apiVersion: v1 kind: Service metadata: name: sample-app namespace: default labels: app: sample-app spec: ports: - port: 80 name: http protocol: TCP targetPort: 8080 selector: app: sample-app type: ClusterIP --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: test-ingress namespace: default spec: ingressClassName: nginx rules: - host: test.example.com http: paths: - backend: service: name: sample-app port: number: 80 path: / pathType: ImplementationSpecific - backend: service: name: test-app port: number: 8080 path: /home pathType: ImplementationSpecific
分別查詢test.example.com/
和test.example.com/home
的nginx_ingress_controller_requests指標,指標正常
創建external類型的apiservices資源;創建後apiservices的狀態爲false是正常現象,添加externalRules後狀態變爲true
apiVersion: apiregistration.k8s.io/v1 kind: APIService metadata: name: v1beta1.external.metrics.k8s.io spec: group: external.metrics.k8s.io groupPriorityMinimum: 100 insecureSkipTLSVerify: true service: #指定prometheus-adapter對應的service,華爲CCE插件中adapter名稱爲custom-metrics-apiserver name: custom-metrics-apiserver namespace: monitoring port: 443 version: v1beta1 versionPriority: 100
將externalRules規則添加到adapter的configmap中,修改後需要重啓prometheus-adapter服務
kubectl -n monitoring edit configmap user-adapter-config
externalRules: - metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>) name: as: ${1}_per_second matches: ^(.*) resources: namespaced: false #忽略指標來源的命名空間,該配置不適用rules規則 seriesQuery: nginx_ingress_controller_requests
seriesQuery:原始指標;可以直接寫指標名稱,也可以使用{labelKey=labelValue}的方式篩選出原始指標
metricsQuery:用PromQL對指標進行篩選匯聚;.Series
表示原始指標,.LabelMatchers
表示對指標進行標籤篩選,hpa中可以配置具體的篩選規則
name:將指標重命名
resources:hpa查詢指標時通過api的方式調用,調用路徑爲:
而resources的作用就是將指標中命名空間標籤的值替換路徑中的${namespace},而我們本次方案需要忽略指標來源命名空間。
custom-metrics-apiserver服務重啓後需要等待1分鐘左右,執行命令查看指標是否正常
kubectl get --raw /apis/external.metrics.k8s.io/v1beta1/namespaces/*/nginx_ingress_controller_requests_per_second
創建HPA規則
apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: sample-hpa spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: sample-app minReplicas: 1 maxReplicas: 10 metrics: - type: External external: metric: name: nginx_ingress_controller_requests_per_second selector: matchLabels: #可以通過該字段對指標進行過濾,這裏標籤篩選條件會加入到externalRules的<<.LabelMatchers>>中。 exported_service: sample-app #篩選後端服務爲sample-app的請求 host: test.example.com #篩選訪問域名爲test.example.com的請求 target: type: AverageValue #External指標類型下只支持Value和AverageValue類型的目標值。 averageValue: 30 --- apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: test-hpa spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: test-app minReplicas: 1 maxReplicas: 10 metrics: - type: External external: metric: name: nginx_ingress_controller_requests_per_second selector: matchLabels: exported_service: test-app host: test.example.com target: type: AverageValue averageValue: 30
彈性演示
使用命令壓測sample-app對應的訪問域名和路徑,正常觸發彈性;請自行配置域名與ELB地址的映射
ab -c 50 -n 5000 http://test.example.com/
用同樣的方式壓測test-app,正常觸發彈性
ab -c 50 -n 5000 http://test.example.com/home