K8s-Ingress高可用IP透傳方案

簡介：

k8s暴露服務的方式有以下幾種：

Proxy + clusterIP
NodePort
LoadBalancer
Ingress

Proxy + ClusterIP

自定義代理 + 集羣內存IP

有一些場景下，你得使用 Kubernetes 的 proxy 模式來訪問你的服務：

• 由於某些原因，你需要調試你的服務，或者需要直接通過筆記本電腦去訪問它們。

• 容許內部通信，展示內部儀表盤等。

這種方式要求我們運行 kubectl 作爲一個未認證的用戶，因此我們不能用這種方式把服務暴露到 internet 或者在生產環境使用。

NodePort （當前採用方式）：

NodePort 服務是引導外部流量到你的服務的最原始方式。NodePort，正如這個名字所示，在所有節點（虛擬機）上開放一個特定端口，任何發送到該端口的流量都被轉發到對應服務。

這種方式的優點是：簡單快速；

缺點是：

1 通過這種方式暴露服務，相當於在給 k8s 環境打孔，需要所有工作節點都開放對外訪問的端口，隨着應用的增加，集羣的端口會被大量消耗且很難管理，更麻煩的問題是無法制作邊緣服務器，所有的服務器都是邊緣節點。

2 由於通過 nodePort 方式暴露服務，請求是通過集羣內部第二跳做到請求轉發的：

第二跳會通過 SNAT的方式替換請求頭，導致集羣內部應用無法獲取用戶真實 IP，開發網關無法根據IP進行流量控制。

3 LoadBalancer方式

LoadBalancer 服務是暴露服務到 internet 的標準方式。在 GKE 上，這種方式會啓動一個 Network Load Balancer[2]，它將給你一個單獨的 IP 地址，轉發所有流量到你的服務。

這個方式的最大缺點是每一個用 LoadBalancer 暴露的服務都會有它自己的 IP 地址，每個用到的 LoadBalancer 都需要付費，這將是非常昂貴的。

4 Ingress

Ingress 可能是暴露服務的最強大方式，但同時也是最複雜的。Ingress 控制器有各種類型，包括 Google Cloud Load Balancer， Nginx，Contour，Istio，等等。它還有各種插件，比如 cert-manager[5]，它可以爲你的服務自動提供 SSL 證書。

優點：功能強大、免費；

缺點：搭建複雜

當前我們的需求：

系統面向第三方系統需要控制訪問的真實 IP，需要做到 IP 透傳
保證高可用
藍綠部署需要做到流量切換，需要使用到Ingress的部分

高可用Ingress 架構如下：

安裝步驟：
1、申請VIP虛擬IP，並將虛擬IP映射到DNS上，如果你沒有添加公司內網 DNS的權限，又需要在本地進行測試，可以在本地計算機上，更改 host 文件，將虛擬IP和域名的映射添加上
2、在三臺工作節點分別安裝 keepAlive

#安裝keepalive
yum install -y keepalived

配置：/etc/keepalived/keepalived.conf
備份：
cd /etc/keepalived
cp keepalived.conf keepalived_BK.conf


第一臺機器執行：
cat <<EOF > /etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
   notification_email {
     xxxxxx
   }
   notification_email_from xxxxxxx
   smtp_server 10.xx.xx.xx
   smtp_connect_timeout 30
   router_id LVS_DEVEL
   vrrp_skip_check_adv_addr
   #vrrp_strict
   vrrp_garp_interval 0
   vrrp_gna_interval 0
}

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 52
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        10.xx.xx.xx
    }
}
EOF

第二臺機器執行：
cat <<EOF > /etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
   notification_email {
     xxxxxxx
   }
   notification_email_from xxxxxxx
   smtp_server 10.xx.xx.xx
   smtp_connect_timeout 30
   router_id LVS_DEVEL
   vrrp_skip_check_adv_addr
   #vrrp_strict
   vrrp_garp_interval 0
   vrrp_gna_interval 0
}

vrrp_instance VI_1 {
    state SLAVE
    interface eth0
    virtual_router_id 52
    priority 90
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        10.xx.xx.xx
    }
}
EOF


#啓動keepalive
systemctl start keepalived && systemctl enable keepalived

#驗證：
#1、 ping 虛擬IP
#2、 systemctl status keepalived
#3、 ip a 查看虛擬IP是否綁定到物理網卡
#4、 systemctl stop/start keepalived 停止100優先級的節點，查看虛擬IP是否有漂移到其他物理機的物理網卡上，重啓後查看是否飄回

3、給工作節點加上label標籤
kubectl label node node1 labelName=ingress-controller
kubectl label node node2 labelName=ingress-controller

4、在三臺工作節點上安裝nginx-ingress-controller： mandotory.yaml 修改點如下：

（1） Depolyment 修改爲 Damonset 守護

（2） pod 選擇器修改到打上標籤的node

（3）鏡像地址修改

apiVersion: v1
kind: Namespace
metadata:
  name: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---

kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-configuration
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---
kind: ConfigMap
apiVersion: v1
metadata:
  name: tcp-services
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---
kind: ConfigMap
apiVersion: v1
metadata:
  name: udp-services
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nginx-ingress-serviceaccount
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: nginx-ingress-clusterrole
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - endpoints
      - nodes
      - pods
      - secrets
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - events
    verbs:
      - create
      - patch
  - apiGroups:
      - "extensions"
      - "networking.k8s.io"
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - "extensions"
      - "networking.k8s.io"
    resources:
      - ingresses/status
    verbs:
      - update

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  name: nginx-ingress-role
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - pods
      - secrets
      - namespaces
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - configmaps
    resourceNames:
      # Defaults to "<election-id>-<ingress-class>"
      # Here: "<ingress-controller-leader>-<nginx>"
      # This has to be adapted if you change either parameter
      # when launching the nginx-ingress-controller.
      - "ingress-controller-leader-nginx"
    verbs:
      - get
      - update
  - apiGroups:
      - ""
    resources:
      - configmaps
    verbs:
      - create
  - apiGroups:
      - ""
    resources:
      - endpoints
    verbs:
      - get

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: nginx-ingress-role-nisa-binding
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: nginx-ingress-role
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: ingress-nginx

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: nginx-ingress-clusterrole-nisa-binding
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: nginx-ingress-clusterrole
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: ingress-nginx

---

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nginx-ingress-controller
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: ingress-nginx
      app.kubernetes.io/part-of: ingress-nginx
  template:
    metadata:
      labels:
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
      annotations:
        prometheus.io/port: "10254"
        prometheus.io/scrape: "true"
    spec:
      # wait up to five minutes for the drain of connections
      terminationGracePeriodSeconds: 300
      serviceAccountName: nginx-ingress-serviceaccount
      nodeSelector:
        kubernetes.io/os: linux
        labelName: ingress-controller
      containers:
        - name: nginx-ingress-controller
          image: siriuszg/nginx-ingress-controller:0.26.1
          args:
            - /nginx-ingress-controller
            - --configmap=$(POD_NAMESPACE)/nginx-configuration
            - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
            - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
            - --publish-service=$(POD_NAMESPACE)/ingress-nginx
            - --annotations-prefix=nginx.ingress.kubernetes.io
          securityContext:
            allowPrivilegeEscalation: true
            capabilities:
              drop:
                - ALL
              add:
                - NET_BIND_SERVICE
            # www-data -> 33
            runAsUser: 33
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          ports:
            - name: http
              containerPort: 80
            - name: https
              containerPort: 443
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 10
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 10
          lifecycle:
            preStop:
              exec:
                command:
                  - /wait-shutdown

---

5、創建可以透傳真實IP的service：注意 externalTrafficPolicy 爲 local

apiVersion: v1
kind: Service
metadata:
  name: ingress-nginx
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  type: NodePort
  ports:
    - name: http
      port: 80
      targetPort: 80
      nodePort: 30030
      protocol: TCP
    - name: https
      port: 443
      targetPort: 443
      nodePort: 30031
      protocol: TCP
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
  externalTrafficPolicy: Local

6、創建對應服務的service，注意類型由NodePort轉換爲ClusterIP

apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: test-kube-dev
  name: test-deploy
  labels:
    app: test-deploy
spec:
  replicas: 1
  template:
    metadata:
      name: test
      labels:
        app: test-dev
    spec:
      containers:
        - name: test
          image: xxxxxxxxxxxx
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8080
      restartPolicy: Always
      imagePullSecrets:
        - name: harbor-secret-name
  selector:
    matchLabels:
      app: test-dev
---
apiVersion: v1
kind: Service
metadata:
  namespace: test-kube-dev
  name: test-service
spec:
  selector:
    app: test-dev
  ports:
    - name: tomcatport
      port: 8080
      targetPort: 8080
  type: ClusterIP
  sessionAffinity: ClientIP

7、創建對應service的Ingress

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  namespace: test-kube-dev
  name: test-dev-ing
#  annotations:
#    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - host: ingress.xxxx.com.cn
    http:
      paths:
      # 老版本服務
      - path: /test
        backend:
          serviceName: test-service
          servicePort: 8080

8、部署應用之後，驗證是否可以正常訪問，以及IP透傳是否生效：
訪問地址爲：虛擬IP域名:ingress的Service端口/path

K8s-Ingress高可用IP透傳方案

釘釘打卡速度慢

Nginx R31 doc 官方文檔-01-nginx 如何安裝

Python 潮流週刊#51：用 Python 繪製美觀的圖表

Qt/C++音視頻開發74-合併標籤圖形/生成yolo運算結果圖形/文字和圖形合併成一個/水印濾鏡

挑戰程序設計競賽 2.2章習題 POJ - 3617 Best Cow Line 貪心

字節面試：MySQL什麼時候鎖表？如何防止鎖表？

.NET8連接SQL SERVER 2008 R2 報：證書鏈是由不受信任的頒發機構頒發的

golang開發環境搭建(win10)

python計算機視覺學習筆記——PIL庫的用法

Golang初學：獲取程序內存使用情況，std runtime

k8s環境物理機切換DNS後不能立即生效問題解決

kubernetes之network policy

解決 k8s 限制內存後還出現 OOM 錯誤的問題：資源整合

K8s節點重置、加入集羣

封裝一個帶有tcpdump工具的鏡像，在容器中使用tcp抓包

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結