k8s集羣外訪問集羣內部服務的幾種方式

前言

       目前k8s+docker算是運維必修了，docker能讓我們的環境一鍵遷移，k8s能自動編排,還能保證服務高可用，兩者結合自然是無敵，那我們要不要使用k8s呢？這要根據自身情況，任何技術的使用，都需要根據具體情況來定，就像你要殺雞，卻非要選牛刀，牛刀是很鋒利，但還不夠你折騰的

       本文重點介紹k8s集羣外如何訪問集羣內的服務，主要方式有：hostPort或hostNetwork、NodePort、Ingress

一、hostPort或hostNetwork

 hostPort和hostNetwork 放在首位是因爲大家很容易忽略它們，它們也可讓集羣外訪問集羣內應用，用法：

• hostNetwork

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx
  name: nginx-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec: 
      nodeSelector:      # node節點選擇器
        role: master     # node節點標籤(Label)
      hostNetwork: true  # 使用node節點網絡
      containers:
      - image: nginx
        imagePullPolicy: IfNotPresent
        name: nginx
        ports:
        - containerPort: 8080

重點在和containers平級的hostNetwork: true,表示pod使用宿主機網絡，配合nodeSelector，把pod實例化在固定節點，如上，我給mater節點加上標籤role: master,通過nodeSelector,nginx就會實例化在master節點，這樣就可以通過master節點的ip和8080端口訪問這個nginx了

• hostPort

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx
  name: nginx-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec: 
      nodeSelector:      # node節點選擇器
        role: master     # node節點標籤(Label)
      containers:
      - image: nginx
        imagePullPolicy: IfNotPresent
        name: nginx
        ports:
        - containerPort: 8080
          hostPort: 80                #重點

和hostNetwork相比多了映射能力，可以把容器端口映射爲node節點不同端口，hostPort,當然也需要nodeSelector來固定節點，不然每次創建，節點不同，ip也會改變

訪問方式：nodeSelector所選節點ip:hostPort, 如上：role=Master標籤節點Ip:80

二、NodePort

NodePort是最常見的提供集羣外訪問的方式之一，該方式使用Service提供集羣外訪問

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx
  name: nginx-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec: 
      containers:
      - image: nginx
        imagePullPolicy: IfNotPresent
        name: nginx
        ports:
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-pod-service
  labels:
    app: nginx                  #自身標籤
spec:
  type: NodePort                # 類型ExternalName, ClusterIP, NodePort, and LoadBalancer
  ports:
  - port: 8080                 # service在k8s集羣內服務端口
    targetPort: 8080           # 關聯pod對外開放端口
    nodePort: 30088            # 集羣外訪問端口，端口範圍【30000-32767】
  selector:
    app: nginx                 # pod標籤

訪問方式：集羣內任意節點ip加nodePort所配端口號，如上：集羣內任一節點ip:30088，即可訪問nginx

三、Ingress

Ingress基本生產必備，可以簡單理解爲部署了一個nginx服務，該服務使用hostNetwork或hostPort方式提供集羣外訪問，再根據配置的路由規則，路由的集羣內部各個service

• Ingress安裝

Ingress有各種實現版本，這裏安裝的爲ingress-nginx,其yaml文件如下：ingress.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---

kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-configuration
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---
kind: ConfigMap
apiVersion: v1
metadata:
  name: tcp-services
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---
kind: ConfigMap
apiVersion: v1
metadata:
  name: udp-services
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nginx-ingress-serviceaccount
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: nginx-ingress-clusterrole
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - endpoints
      - nodes
      - pods
      - secrets
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - events
    verbs:
      - create
      - patch
  - apiGroups:
      - "extensions"
      - "networking.k8s.io"
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - "extensions"
      - "networking.k8s.io"
    resources:
      - ingresses/status
    verbs:
      - update

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  name: nginx-ingress-role
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - pods
      - secrets
      - namespaces
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - configmaps
    resourceNames:
      # Defaults to "<election-id>-<ingress-class>"
      # Here: "<ingress-controller-leader>-<nginx>"
      # This has to be adapted if you change either parameter
      # when launching the nginx-ingress-controller.
      - "ingress-controller-leader-nginx"
    verbs:
      - get
      - update
  - apiGroups:
      - ""
    resources:
      - configmaps
    verbs:
      - create
  - apiGroups:
      - ""
    resources:
      - endpoints
    verbs:
      - get

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: nginx-ingress-role-nisa-binding
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: nginx-ingress-role
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: ingress-nginx

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: nginx-ingress-clusterrole-nisa-binding
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: nginx-ingress-clusterrole
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: ingress-nginx

---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-ingress-controller
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: ingress-nginx
      app.kubernetes.io/part-of: ingress-nginx
  template:
    metadata:
      labels:
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
      annotations:
        prometheus.io/port: "10254"
        prometheus.io/scrape: "true"
    spec:
      # wait up to five minutes for the drain of connections
      terminationGracePeriodSeconds: 300
      serviceAccountName: nginx-ingress-serviceaccount
      nodeSelector:                  #注意修改自己nodeSelector
        role: master
      hostNetwork: true
      containers:
        - name: nginx-ingress-controller
          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.30.0
          args:
            - /nginx-ingress-controller
            - --configmap=$(POD_NAMESPACE)/nginx-configuration
            - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
            - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
            - --publish-service=$(POD_NAMESPACE)/ingress-nginx
            - --annotations-prefix=nginx.ingress.kubernetes.io
          securityContext:
            allowPrivilegeEscalation: true
            capabilities:
              drop:
                - ALL
              add:
                - NET_BIND_SERVICE
            # www-data -> 101
            runAsUser: 101
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          ports:
            - name: http
              containerPort: 80
              protocol: TCP
            - name: https
              containerPort: 443
              protocol: TCP
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 10
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 10
          lifecycle:
            preStop:
              exec:
                command:
                  - /wait-shutdown

---

apiVersion: v1
kind: LimitRange
metadata:
  name: ingress-nginx
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  limits:
  - min:
      memory: 90Mi
      cpu: 100m
    type: Container

kubectl apply -f ingress.yaml

該yaml會創建ingress-nginx命名空間，待對應pod進入running狀態，ingress服務就安裝完成了

• 配置ingress路由規則

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx
  name: nginx-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec: 
      containers:
      - image: nginx
        imagePullPolicy: IfNotPresent
        name: nginx
        ports:
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-pod-service
  labels:
    app: nginx                  #自身標籤
spec:
  type: NodePort                # 類型ExternalName, ClusterIP, NodePort, and LoadBalancer
  ports:
  - port: 8080                 # service在k8s集羣內服務端口
    targetPort: 8080           # 關聯pod對外開放端口
  selector:
    app: nginx                 # pod標籤
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx-ingress
  labels:
    app.kubernetes.io/version: v1
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - host: nginx.test.com
    http:
      paths:
      - path: /
        backend:
          serviceName: nginx-pod-service     # 上面nginx Service配置的name
          servicePort: 8080                  #  Service配置的port

集羣外配置好host

ingress服務部署節點的ip   nginx.test.com  

通過域名進行訪問，且僅能通過域名

總結各方式利弊

hostPort和hostNetwork直接使用節點網絡，部署時節點需固定，訪問ip也固定(也可以用host)，端口爲正常端口

nodeport方式部署時不要求固定節點，可通過集羣內任一ip進行訪問，就是端口爲30000以上，很多時候由於公司安全策略導致不能訪問

ingress需要額外安裝ingress模塊，配置路由規則，且僅能通過所配置域名訪問，配置好域名後，可以直接對外提供服務，和傳統的nginx作用類似

可根據各自情況靈活使用