kubernetes系列教程(十六)基於nginx ingress實現服務暴露

文章目錄

• 寫在前面
• 1. Ingress簡介
• 2. Nginx Ingress
     • 2.1 Nginx ingress介紹
     • 2.2 Nginx ingress安裝
• 3. Ingress資源定義
     • 3.1 Ingress定義
     • 3.2 Ingress動態配置
     • 3.3 Ingress路徑轉發
     • 3.4 Ingress虛擬主機
     • 3.5 Ingress TLS加密
• 4. Nginx Ingress高級功能
     • 4.1 定製化參數
     • 4.2 虛擬主機和路由
• 寫在最後
• 參考文獻

寫在前面

本章介紹kubernetes系列教程的ingress概念，在kubernetes中對外暴露服務的方式有兩種：service（NodePort或者外部LoadBalancer）和ingress，其中service是提供四層的負載均衡，通過iptables DNAT或lvs nat模式實現後端Pod的代理請求。如需實現http，域名，URI，證書等請求方式，service是無法實現的，需要藉助於ingress來來實現，本文將來介紹ingress相關的內容。

1. Ingress簡介

An API object that manages external access to the services in a cluster, typically HTTP.
Ingress can provide load balancing, SSL termination and name-based virtual hosting.

引用官方關於ingress的介紹我們可以得知，ingress是一種通過http協議暴露kubernetes內部服務的api對象，即充當Edge Router邊界路由器的角色對外基於七層的負載均衡調度機制，能夠提供以下幾個功能：

• 負載均衡，將請求自動負載均衡到後端的Pod上；
• SSL加密，客戶端到Ingress Controller爲https加密，到後端Pod爲明文的http；
• 基於名稱的虛擬主機，提供基於域名或URI更靈活的路由方式

實現Ingress包含的組件有：

• Ingress，客戶端，負責定義ingress配置，將請求轉發給Ingress Controller；
• Ingress Controller，Ingress控制器，實現七層轉發的Edge Router，通過調用k8s的api動態感知集羣中Pod的變化而動態更新配置文件並重載， Controller需要部署在k8s集羣中以實現和集羣中的pod通信，通常以DaemonSets或Deployments的形式部署，並對外暴露80和443端口，對於DaemonSets來說，一般是以hostNetwork或者hostPort的形式暴露，Deployments則以NodePort的方式暴露，控制器的多個節點則藉助外部負載均衡ExternalLB以實現統一接入；
• Ingress配置規則，Controller控制器通過service服務發現機制動態實現後端Pod路由轉發規則的實現；
• Service，kuberntes中四層的負載均衡調度機制，Ingress藉助service的服務發現機制實現集羣中Pod資源的動態感知；
• Pod，後端實際負責響應請求容器，由控制器如Deployment創建，通過標籤Labels和service關聯，服務發現。

簡而言之，ingress控制器藉助service的服務發現機制實現配置的動態更新以實現Pod的負載均衡機制實現，由於涉及到Ingress Controller的動態更新，目前社區Ingress Controller大體包含兩種類型的控制器：

• 傳統的七層負載均衡如Nginx，HAproxy，開發了適應微服務應用的插件，具有成熟，高性能等優點；
• 新型微服務負載均衡如Traefik，Envoy，Istio，專門適用於微服務+容器化應用場景，具有動態更新特點；

類型	常見類型	優點	缺點
傳統負載均衡	nginx，haproxy	成熟，穩定，高性能	動態更新需reload配置文件
微服務負載均衡	Traefik，Envoy，Istio	天生爲微服務而生，動態更新	性能還有待提升

2. Nginx Ingress

2.1 Nginx ingress介紹

By default, pods of Kubernetes services are not accessible from the external network, but only by other pods within the Kubernetes cluster. Kubernetes has a built‑in configuration for HTTP load balancing, called Ingress, that defines rules for external connectivity to Kubernetes services. Users who need to provide external access to their Kubernetes services create an Ingress resource that defines rules, including the URI path, backing service name, and other information. The Ingress controller can then automatically program a frontend load balancer to enable Ingress configuration. The NGINX Ingress Controller for Kubernetes is what enables Kubernetes to configure NGINX and NGINX Plus for load balancing Kubernetes services.

Nginx Ingress Controller是實現ingress的具體實現，包含有兩個版本：Ngnix OSS和Nginx Plus版，後者是商業化增強版，支持更多的功能，詳情參考官方文檔介紹https://www.nginx.com/products/nginx/kubernetes-ingress-controller#compare-versions

2.2 Nginx ingress安裝

首先需要安裝Nginx Ingress Controller控制器，控制器安裝方式包含兩種：DaemonSets和Deployments。

• DaemonSets通過hostPort的方式暴露80和443端口，可通過Node的調度由專門的節點實現部署
• Deployments則通過NodePort的方式實現控制器端口的暴露，藉助外部負載均衡實現高可用負載均衡

除此之外，還需要部署Namespace，ServiceAccount，RBAC，Secrets，Custom Resource Definitions等資源，如下開始部署。

2.2.1 基礎依賴環境準備

1、github中下載源碼包,安裝部署文件在kubernetes-ingress/deployments/目錄下

[root@node-1 ~]# git clone https://github.com/nginxinc/kubernetes-ingress.git
[root@node-1 ~]# tree kubernetes-ingress/deployments/
kubernetes-ingress/deployments/
├── common
│   ├── custom-resource-definitions.yaml  自定義資源
│   ├── default-server-secret.yaml        Secrets
│   ├── nginx-config.yaml
│   └── ns-and-sa.yaml                    Namspace+ServiceAccount
├── daemon-set
│   ├── nginx-ingress.yaml                DaemonSets控制器
│   └── nginx-plus-ingress.yaml
├── deployment
│   ├── nginx-ingress.yaml                Deployments控制器
│   └── nginx-plus-ingress.yaml
├── helm-chart                            Helm安裝包
│   ├── chart-icon.png
│   ├── Chart.yaml
│   ├── README.md
│   ├── templates
│   │   ├── controller-configmap.yaml
│   │   ├── controller-custom-resources.yaml
│   │   ├── controller-daemonset.yaml
│   │   ├── controller-deployment.yaml
│   │   ├── controller-leader-election-configmap.yaml
│   │   ├── controller-secret.yaml
│   │   ├── controller-serviceaccount.yaml
│   │   ├── controller-service.yaml
│   │   ├── controller-wildcard-secret.yaml
│   │   ├── _helpers.tpl
│   │   ├── NOTES.txt
│   │   └── rbac.yaml
│   ├── values-icp.yaml
│   ├── values-plus.yaml
│   └── values.yaml
├── rbac                                RBAC認證授權
│   └── rbac.yaml
├── README.md
└── service                            Service定義
    ├── loadbalancer-aws-elb.yaml
    ├── loadbalancer.yaml              DaemonSets暴露服務方式
    └── nodeport.yaml                  Deployments暴露服務方式

2、創建Namespace和ServiceAccount, kubectl apply -f common/default-server-secret.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: nginx-ingress 
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nginx-ingress 
  namespace: nginx-ingress

3、創建Secrets自簽名證書，kubectl apply -f common/default-server-secret.yaml

apiVersion: v1
kind: Secret
metadata:
  name: default-server-secret
  namespace: nginx-ingress
type: Opaque
data:
  tls.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN2akNDQWFZQ0NRREFPRjl0THNhWFhEQU5CZ2txaGtpRzl3MEJBUXNGQURBaE1SOHdIUVlEVlFRRERCWk8KUjBsT1dFbHVaM0psYzNORGIyNTBjbTlzYkdWeU1CNFhEVEU0TURreE1qRTRNRE16TlZvWERUSXpNRGt4TVRFNApNRE16TlZvd0lURWZNQjBHQTFVRUF3d1dUa2RKVGxoSmJtZHlaWE56UTI5dWRISnZiR3hsY2pDQ0FTSXdEUVlKCktvWklodmNOQVFFQkJRQURnZ0VQQURDQ0FRb0NnZ0VCQUwvN2hIUEtFWGRMdjNyaUM3QlBrMTNpWkt5eTlyQ08KR2xZUXYyK2EzUDF0azIrS3YwVGF5aGRCbDRrcnNUcTZzZm8vWUk1Y2Vhbkw4WGM3U1pyQkVRYm9EN2REbWs1Qgo4eDZLS2xHWU5IWlg0Rm5UZ0VPaStlM2ptTFFxRlBSY1kzVnNPazFFeUZBL0JnWlJVbkNHZUtGeERSN0tQdGhyCmtqSXVuektURXUyaDU4Tlp0S21ScUJHdDEwcTNRYzhZT3ExM2FnbmovUWRjc0ZYYTJnMjB1K1lYZDdoZ3krZksKWk4vVUkxQUQ0YzZyM1lma1ZWUmVHd1lxQVp1WXN2V0RKbW1GNWRwdEMzN011cDBPRUxVTExSakZJOTZXNXIwSAo1TmdPc25NWFJNV1hYVlpiNWRxT3R0SmRtS3FhZ25TZ1JQQVpQN2MwQjFQU2FqYzZjNGZRVXpNQ0F3RUFBVEFOCkJna3Foa2lHOXcwQkFRc0ZBQU9DQVFFQWpLb2tRdGRPcEsrTzhibWVPc3lySmdJSXJycVFVY2ZOUitjb0hZVUoKdGhrYnhITFMzR3VBTWI5dm15VExPY2xxeC9aYzJPblEwMEJCLzlTb0swcitFZ1U2UlVrRWtWcitTTFA3NTdUWgozZWI4dmdPdEduMS9ienM3bzNBaS9kclkrcUI5Q2k1S3lPc3FHTG1US2xFaUtOYkcyR1ZyTWxjS0ZYQU80YTY3Cklnc1hzYktNbTQwV1U3cG9mcGltU1ZmaXFSdkV5YmN3N0NYODF6cFErUyt1eHRYK2VBZ3V0NHh3VlI5d2IyVXYKelhuZk9HbWhWNThDd1dIQnNKa0kxNXhaa2VUWXdSN0diaEFMSkZUUkk3dkhvQXprTWIzbjAxQjQyWjNrN3RXNQpJUDFmTlpIOFUvOWxiUHNoT21FRFZkdjF5ZytVRVJxbStGSis2R0oxeFJGcGZnPT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
  tls.key: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcEFJQkFBS0NBUUVBdi91RWM4b1JkMHUvZXVJTHNFK1RYZUprckxMMnNJNGFWaEMvYjVyYy9XMlRiNHEvClJOcktGMEdYaVN1eE9ycXgrajlnamx4NXFjdnhkenRKbXNFUkJ1Z1B0ME9hVGtIekhvb3FVWmcwZGxmZ1dkT0EKUTZMNTdlT1l0Q29VOUZ4amRXdzZUVVRJVUQ4R0JsRlNjSVo0b1hFTkhzbysyR3VTTWk2Zk1wTVM3YUhudzFtMApxWkdvRWEzWFNyZEJ6eGc2clhkcUNlUDlCMXl3VmRyYURiUzc1aGQzdUdETDU4cGszOVFqVUFQaHpxdmRoK1JWClZGNGJCaW9CbTVpeTlZTW1hWVhsMm0wTGZzeTZuUTRRdFFzdEdNVWozcGJtdlFmazJBNnljeGRFeFpkZFZsdmwKMm82MjBsMllxcHFDZEtCRThCay90elFIVTlKcU56cHpoOUJUTXdJREFRQUJBb0lCQVFDZklHbXowOHhRVmorNwpLZnZJUXQwQ0YzR2MxNld6eDhVNml4MHg4Mm15d1kxUUNlL3BzWE9LZlRxT1h1SENyUlp5TnUvZ2IvUUQ4bUFOCmxOMjRZTWl0TWRJODg5TEZoTkp3QU5OODJDeTczckM5bzVvUDlkazAvYzRIbjAzSkVYNzZ5QjgzQm9rR1FvYksKMjhMNk0rdHUzUmFqNjd6Vmc2d2szaEhrU0pXSzBwV1YrSjdrUkRWYmhDYUZhNk5nMUZNRWxhTlozVDhhUUtyQgpDUDNDeEFTdjYxWTk5TEI4KzNXWVFIK3NYaTVGM01pYVNBZ1BkQUk3WEh1dXFET1lvMU5PL0JoSGt1aVg2QnRtCnorNTZud2pZMy8yUytSRmNBc3JMTnIwMDJZZi9oY0IraVlDNzVWYmcydVd6WTY3TWdOTGQ5VW9RU3BDRkYrVm4KM0cyUnhybnhBb0dCQU40U3M0ZVlPU2huMVpQQjdhTUZsY0k2RHR2S2ErTGZTTXFyY2pOZjJlSEpZNnhubmxKdgpGenpGL2RiVWVTbWxSekR0WkdlcXZXaHFISy9iTjIyeWJhOU1WMDlRQ0JFTk5jNmtWajJTVHpUWkJVbEx4QzYrCk93Z0wyZHhKendWelU0VC84ajdHalRUN05BZVpFS2FvRHFyRG5BYWkyaW5oZU1JVWZHRXFGKzJyQW9HQkFOMVAKK0tZL0lsS3RWRzRKSklQNzBjUis3RmpyeXJpY05iWCtQVzUvOXFHaWxnY2grZ3l4b25BWlBpd2NpeDN3QVpGdwpaZC96ZFB2aTBkWEppc1BSZjRMazg5b2pCUmpiRmRmc2l5UmJYbyt3TFU4NUhRU2NGMnN5aUFPaTVBRHdVU0FkCm45YWFweUNweEFkREtERHdObit3ZFhtaTZ0OHRpSFRkK3RoVDhkaVpBb0dCQUt6Wis1bG9OOTBtYlF4VVh5YUwKMjFSUm9tMGJjcndsTmVCaWNFSmlzaEhYa2xpSVVxZ3hSZklNM2hhUVRUcklKZENFaHFsV01aV0xPb2I2NTNyZgo3aFlMSXM1ZUtka3o0aFRVdnpldm9TMHVXcm9CV2xOVHlGanIrSWhKZnZUc0hpOGdsU3FkbXgySkJhZUFVWUNXCndNdlQ4NmNLclNyNkQrZG8wS05FZzFsL0FvR0FlMkFVdHVFbFNqLzBmRzgrV3hHc1RFV1JqclRNUzRSUjhRWXQKeXdjdFA4aDZxTGxKUTRCWGxQU05rMXZLTmtOUkxIb2pZT2pCQTViYjhibXNVU1BlV09NNENoaFJ4QnlHbmR2eAphYkJDRkFwY0IvbEg4d1R0alVZYlN5T294ZGt5OEp0ek90ajJhS0FiZHd6NlArWDZDODhjZmxYVFo5MWpYL3RMCjF3TmRKS2tDZ1lCbyt0UzB5TzJ2SWFmK2UwSkN5TGhzVDQ5cTN3Zis2QWVqWGx2WDJ1VnRYejN5QTZnbXo5aCsKcDNlK2JMRUxwb3B0WFhNdUFRR0xhUkcrYlNNcjR5dERYbE5ZSndUeThXczNKY3dlSTdqZVp2b0ZpbmNvVlVIMwphdmxoTUVCRGYxSjltSDB5cDBwWUNaS2ROdHNvZEZtQktzVEtQMjJhTmtsVVhCS3gyZzR6cFE9PQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=

4、創建ConfigMap自定義配置文件，kubectl apply -f common/nginx-config.yaml

kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-config
  namespace: nginx-ingress
data:

5、爲虛擬雲主機和虛擬雲主機路由定義自定義資源，支持自定義虛擬主機和虛擬路由，kubectl apply -f common/custom-resource-definitions.yaml

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: virtualservers.k8s.nginx.org
spec:
  group: k8s.nginx.org
  versions:
  - name: v1
    served: true
    storage: true
  scope: Namespaced
  names:
    plural: virtualservers
    singular: virtualserver
    kind: VirtualServer
    shortNames:
    - vs
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: virtualserverroutes.k8s.nginx.org
spec:
  group: k8s.nginx.org
  versions:
  - name: v1
    served: true
    storage: true
  scope: Namespaced
  names:
    plural: virtualserverroutes
    singular: virtualserverroute
    kind: VirtualServerRoute
    shortNames:
    - vsr

6、配置RBAC認證授權，實現ingress控制器訪問集羣中的其他資源，kubectl apply -f rbac/rbac.yaml

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: nginx-ingress
rules:
- apiGroups:
  - ""
  resources:
  - services
  - endpoints
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - secrets
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - configmaps
  verbs:
  - get
  - list
  - watch
  - update
  - create
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - events
  verbs:
  - create
  - patch
- apiGroups:
  - extensions
  resources:
  - ingresses
  verbs:
  - list
  - watch
  - get
- apiGroups:
  - "extensions"
  resources:
  - ingresses/status
  verbs:
  - update
- apiGroups:
  - k8s.nginx.org
  resources:
  - virtualservers
  - virtualserverroutes
  verbs:
  - list
  - watch
  - get
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: nginx-ingress
subjects:
- kind: ServiceAccount
  name: nginx-ingress
  namespace: nginx-ingress
roleRef:
  kind: ClusterRole
  name: nginx-ingress
  apiGroup: rbac.authorization.k8s.io

2.2.2 部署Ingress控制器

1、 部署控制器，控制器可以DaemonSets和Deployment的形式部署，如下是DaemonSets的配置文件

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nginx-ingress
  namespace: nginx-ingress
spec:
  selector:
    matchLabels:
      app: nginx-ingress
  template:
    metadata:
      labels:
        app: nginx-ingress
     #annotations:
       #prometheus.io/scrape: "true"
       #prometheus.io/port: "9113"
    spec:
      serviceAccountName: nginx-ingress
      containers:
      - image: nginx/nginx-ingress:edge
        imagePullPolicy: Always
        name: nginx-ingress
        ports:
        - name: http
          containerPort: 80
          hostPort: 80            #通過hostPort的方式暴露端口
        - name: https
          containerPort: 443
          hostPort: 443
       #- name: prometheus
         #containerPort: 9113
        securityContext:
          allowPrivilegeEscalation: true
          runAsUser: 101 #nginx
          capabilities:
            drop:
            - ALL
            add:
            - NET_BIND_SERVICE
        env:
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        args:
          - -nginx-configmaps=$(POD_NAMESPACE)/nginx-config
          - -default-server-tls-secret=$(POD_NAMESPACE)/default-server-secret
         #- -v=3 # Enables extensive logging. Useful for troubleshooting.
         #- -report-ingress-status
         #- -external-service=nginx-ingress
         #- -enable-leader-election
         #- -enable-prometheus-metrics

Deployments的配置文件

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-ingress
  namespace: nginx-ingress
spec:
  replicas: 1                  #副本的個數
  selector:
    matchLabels:
      app: nginx-ingress
  template:
    metadata:
      labels:
        app: nginx-ingress
     #annotations:
       #prometheus.io/scrape: "true"
       #prometheus.io/port: "9113"
    spec:
      serviceAccountName: nginx-ingress
      containers:
      - image: nginx/nginx-ingress:edge
        imagePullPolicy: Always
        name: nginx-ingress
        ports:                #內部暴露的服務端口，需要通過NodePort的方式暴露給外部
        - name: http
          containerPort: 80
        - name: https
          containerPort: 443
       #- name: prometheus
         #containerPort: 9113
        securityContext:
          allowPrivilegeEscalation: true
          runAsUser: 101 #nginx
          capabilities:
            drop:
            - ALL
            add:
            - NET_BIND_SERVICE
        env:
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        args:
          - -nginx-configmaps=$(POD_NAMESPACE)/nginx-config
          - -default-server-tls-secret=$(POD_NAMESPACE)/default-server-secret
         #- -v=3 # Enables extensive logging. Useful for troubleshooting.
         #- -report-ingress-status
         #- -external-service=nginx-ingress
         #- -enable-leader-election
         #- -enable-prometheus-metrics

2、我們以DaemonSets的方式部署，DaemonSet部署集羣中各個節點都是對等，如果有外部LoadBalancer則通過外部負載均衡路由至Ingress中

[root@node-1 deployments]# kubectl apply -f daemon-set/nginx-ingress.yaml 
daemonset.apps/nginx-ingress created
[root@node-1 deployments]# kubectl get daemonsets -n nginx-ingress
NAME            DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
nginx-ingress   3         3         3       3            3           <none>          15s

[root@node-1 ~]# kubectl get pods -n nginx-ingress -o wide 
NAME                  READY   STATUS    RESTARTS   AGE     IP             NODE     NOMINATED NODE   READINESS GATES
nginx-ingress-7mpfc   1/1     Running   0          2m44s   10.244.0.50    node-1   <none>           <none>
nginx-ingress-l2rtj   1/1     Running   0          2m44s   10.244.1.144   node-2   <none>           <none>
nginx-ingress-tgf6r   1/1     Running   0          2m44s   10.244.2.160   node-3   <none>           <none>

3、校驗Nginx Ingress安裝情況，此時三個節點均是對等，即訪問任意一個節點均能實現相同的效果，統一入口則通過外部負載均衡，如果在雲環境下執行kubectl apply -f service/loadbalancer.yaml創建外部負載均衡實現入口調度，自建的可以通過lvs或nginx等負載均衡實現接入，本文不再贅述，讀者可以自行研究。

備註說明：如果以Deployments的方式部署，則需要執行service/nodeport.yaml創建NodePort類型的Service，實現的效果和DaemonSets類似。

3. Ingress資源定義

上面的章節已安裝了一個Nginx Ingress Controller控制器，有了Ingress控制器後，我們就可以定義Ingress資源來實現七層負載轉發了，大體上Ingress支持三種使用方式：1. 基於虛擬主機轉發，2. 基於虛擬機主機URI轉發，3. 支持TLS加密轉發。

3.1 Ingress定義

1、環境準備，先創建一個nginx的Deployment應用，包含2個副本

[root@node-1 ~]# kubectl run ingress-demo --image=nginx:1.7.9 --port=80 --replicas=2
[root@node-1 ~]# kubectl get deployments
NAME           READY   UP-TO-DATE   AVAILABLE   AGE
ingress-demo   2/2     2            2           116s

2、以service方式暴露服務端口

[root@node-1 ~]# kubectl expose deployment ingress-demo --port=80 --protocol=TCP --target-port=80
service/ingress-demo exposed
[root@node-1 ~]# kubectl get services 
NAME           TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
ingress-demo   ClusterIP   10.109.33.91   <none>        80/TCP    2m15s

3、上述兩個步驟已創建了一個service，如下我們定義一個ingress對象將起轉發至ingress-demo這個service，通過ingress.class指定控制器的類型爲nginx

[root@node-1 nginx-ingress]# cat nginx-ingress-demo.yaml 
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx-ingress-demo
  labels:
    ingres-controller: nginx
  annotations:
    kubernets.io/ingress.class: nginx
spec:
  rules:
  - host: www.happylau.cn
    http:
      paths:
      - path: /
        backend:
          serviceName: ingress-demo
          servicePort: 80

4、創建ingress對象

[root@node-1 nginx-ingress]# kubectl apply -f nginx-ingress-demo.yaml 
ingress.extensions/nginx-ingress-demo created

查看ingress資源列表
[root@node-1 nginx-ingress]# kubectl get ingresses
NAME                 HOSTS                ADDRESS   PORTS   AGE
nginx-ingress-demo   www.happylau.cn             80      4m4s

5、查看ingress詳情，可以在Rules規則中看到後端Pod的列表，自動發現和關聯相關Pod

[root@node-1 ~]# kubectl describe ingresses nginx-ingress-demo 
Name:             nginx-ingress-demo
Namespace:        default
Address:          
Default backend:  default-http-backend:80 (<none>)
Rules:
  Host                Path  Backends
  ----                ----  --------
  www.happylau.cn  
                      /   ingress-demo:80 (10.244.1.146:80,10.244.2.162:80)
Annotations:
  kubectl.kubernetes.io/last-applied-configuration:  {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernets.io/ingress.class":"nginx"},"labels":{"ingres-controller":"nginx"},"name":"nginx-ingress-demo","namespace":"default"},"spec":{"rules":[{"host":"www.happylaulab.cn","http":{"paths":[{"backend":{"serviceName":"ingress-demo","servicePort":80},"path":"/"}]}}]}}

  kubernets.io/ingress.class:  nginx
Events:
  Type    Reason          Age   From                      Message
  ----    ------          ----  ----                      -------
  Normal  AddedOrUpdated  9m7s  nginx-ingress-controller  Configuration for default/nginx-ingress-demo was added or updated
  Normal  AddedOrUpdated  9m7s  nginx-ingress-controller  Configuration for default/nginx-ingress-demo was added or updated
  Normal  AddedOrUpdated  9m7s  nginx-ingress-controller  Configuration for default/nginx-ingress-demo was added or updated

6、測試驗證，ingress規則的配置信息已注入到Ingress Controller中，環境中Ingress Controller是以DaemonSets的方式部署在集羣中，如果有外部的負載均衡，則將www.happylau.cn域名的地址解析爲負載均衡VIP。由於測試環境沒有搭建負載均衡，將hosts解析執行node-1，node-2或者node-3任意一個IP都能實現相同的功能。

上述測試解析正常，當然也可以解析爲node-1和node-2的IP，如下：

[root@node-1 ~]# curl -I http://www.happylau.cn --resolve www.happylau.cn:80:10.254.100.101
HTTP/1.1 200 OK
Server: nginx/1.17.6
Date: Tue, 24 Dec 2019 10:32:22 GMT
Content-Type: text/html
Content-Length: 612
Connection: keep-alive
Last-Modified: Tue, 23 Dec 2014 16:25:09 GMT
ETag: "54999765-264"
Accept-Ranges: bytes

[root@node-1 ~]# curl -I http://www.happylau.cn --resolve www.happylau.cn:80:10.254.100.102
HTTP/1.1 200 OK
Server: nginx/1.17.6
Date: Tue, 24 Dec 2019 10:32:24 GMT
Content-Type: text/html
Content-Length: 612
Connection: keep-alive
Last-Modified: Tue, 23 Dec 2014 16:25:09 GMT
ETag: "54999765-264"
Accept-Ranges: bytes

3.2 Ingress動態配置

上面的章節介紹了ingress資源對象的申明配置，在這個章節中我們探究一下Nginx Ingress Controller的實現機制和動態配置更新機制，以方便了解Ingress控制器的工作機制。

1、 查看Nginx Controller控制器的配置文件，在nginx-ingress pod中存儲着ingress的配置文件

[root@node-1 ~]# kubectl get pods -n nginx-ingress 
NAME                  READY   STATUS    RESTARTS   AGE
nginx-ingress-7mpfc   1/1     Running   0          6h15m
nginx-ingress-l2rtj   1/1     Running   0          6h15m
nginx-ingress-tgf6r   1/1     Running   0          6h15m

#查看配置文件，每個ingress生成一個配置文件，文件名爲：命名空間-ingres名稱.conf
[root@node-1 ~]# kubectl exec -it nginx-ingress-7mpfc -n nginx-ingress -- ls -l /etc/nginx/conf.d
total 4
-rw-r--r-- 1 nginx nginx 1005 Dec 24 10:06 default-nginx-ingress-demo.conf

#查看配置文件
[root@node-1 ~]# kubectl exec -it nginx-ingress-7mpfc -n nginx-ingress -- cat /etc/nginx/conf.d/default-nginx-ingress-demo.conf
# configuration for default/nginx-ingress-demo

#upstream的配置，會用least_conn算法，通過service服務發現機制動態識別到後端的Pod
upstream default-nginx-ingress-demo-www.happylau.cn-ingress-demo-80 {
    zone default-nginx-ingress-demo-www.happylau.cn-ingress-demo-80 256k;
    random two least_conn;
    server 10.244.1.146:80 max_fails=1 fail_timeout=10s max_conns=0;
    server 10.244.2.162:80 max_fails=1 fail_timeout=10s max_conns=0;
}

server {
    listen 80;
    server_tokens on;
    server_name www.happylau.cn;
    location / {
        proxy_http_version 1.1;
        proxy_connect_timeout 60s;
        proxy_read_timeout 60s;
        proxy_send_timeout 60s;
        client_max_body_size 1m;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Port $server_port;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_buffering on;
        proxy_pass http://default-nginx-ingress-demo-www.happylau.cn-ingress-demo-80;   #調用upstream實現代理
    }
}

通過上述查看配置文件可得知，Nginx Ingress Controller實際是根據ingress規則生成對應的nginx配置文件，以實現代理轉發的功能，加入Deployments的副本數變更後nginx的配置文件會發生什麼改變呢？

2、更新控制器的副本數，由2個Pod副本擴容至3個

[root@node-1 ~]# kubectl scale --replicas=3 deployment ingress-demo 
deployment.extensions/ingress-demo scaled
[root@node-1 ~]# kubectl get deployments
NAME           READY   UP-TO-DATE   AVAILABLE   AGE
ingress-demo   3/3     3            3           123m

3、再次查看nginx的配置文件，ingress藉助於service的服務發現機制，將加入的Pod自動加入到nginx upstream中

4、查看nginx pod的日誌（kubectl logs nginx-ingress-7mpfc -n nginx-ingress），有reload優雅重啓的記錄，即通過更新配置文件+reload實現配置動態更新。

通過上述的配置可知，ingress調用kubernetes api去感知kubernetes集羣中的變化情況，Pod的增加或減少這些變化，然後動態更新nginx ingress controller的配置文件，並重新載入配置。當集羣規模越大時，會頻繁涉及到配置文件的變動和重載，因此nginx這方面會存在先天的劣勢，專門爲微服務負載均衡應運而生，如Traefik，Envoy，Istio，這些負載均衡工具能夠提供大規模，頻繁動態更新的場景，但性能相比Nginx，HAproxy還存在一定的劣勢。往後的章節中，我們再對其他的Ingress控制器做介紹。

3.3 Ingress路徑轉發

Ingress支持URI格式的轉發方式，同時支持URL重寫，如下以兩個service爲例演示，service-1安裝nginx，service-2安裝httpd，分別用http://demo.happylau.cn/news和http://demo.happylau.cn/sports轉發到兩個不同的service

1、環境準備，創建兩個應用並實現service暴露，創建deployments時指定--explose創建service

[root@node-1 ~]# kubectl run service-1 --image=nginx:1.7.9 --port=80 --replicas=1 --expose=true 
service/service-1 created
deployment.apps/service-1 created

[root@node-1 ~]# kubectl run service-2 --image=httpd --port=80 --replicas=1 --expose=true 
service/service-2 created
deployment.apps/service-2 created

查看deployment狀態
[root@node-1 ~]# kubectl get deployments 
NAME           READY   UP-TO-DATE   AVAILABLE   AGE
ingress-demo   4/4     4            4           4h36m
service-1      1/1     1            1           65s
service-2      1/1     1            1           52s

查看service狀態，服務已經正常
[root@node-1 ~]# kubectl get services 
NAME           TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
ingress-demo   ClusterIP   10.109.33.91     <none>        80/TCP    4h36m
kubernetes     ClusterIP   10.96.0.1        <none>        443/TCP   101d
service-1      ClusterIP   10.106.245.71    <none>        80/TCP    68s
service-2      ClusterIP   10.104.204.158   <none>        80/TCP    55s

2、創建ingress對象，通過一個域名將請求轉發至後端兩個service

[root@node-1 nginx-ingress]# cat nginx-ingress-uri-demo.yaml 
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx-ingress-uri-demo
  labels:
    ingres-controller: nginx
  annotations:
    kubernets.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - host: demo.happylau.cn
    http:
      paths:
      - path: /news
        backend:
          serviceName: service-1 
          servicePort: 80
      - path: /sports
        backend:
          serviceName: service-2
          servicePort: 80

3、創建ingress規則，查看詳情

[root@node-1 nginx-ingress]# kubectl apply -f nginx-ingress-uri-demo.yaml 
ingress.extensions/nginx-ingress-uri-demo created

#查看詳情
[root@node-1 nginx-ingress]# kubectl get ingresses.
NAME                     HOSTS              ADDRESS   PORTS   AGE
nginx-ingress-demo       www.happylau.cn              80      4h35m
nginx-ingress-uri-demo   demo.happylau.cn             80      4s
[root@node-1 nginx-ingress]# kubectl describe ingresses nginx-ingress-uri-demo 
Name:             nginx-ingress-uri-demo
Namespace:        default
Address:          
Default backend:  default-http-backend:80 (<none>)
Rules:              #對應的轉發url規則
  Host              Path  Backends
  ----              ----  --------
  demo.happylau.cn  
                    /news     service-1:80 (10.244.2.163:80)
                    /sports   service-2:80 (10.244.1.148:80)
Annotations:
  kubectl.kubernetes.io/last-applied-configuration:  {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernets.io/ingress.class":"nginx","nginx.ingress.kubernetes.io/rewrite-target":"/"},"labels":{"ingres-controller":"nginx"},"name":"nginx-ingress-uri-demo","namespace":"default"},"spec":{"rules":[{"host":"demo.happylau.cn","http":{"paths":[{"backend":{"serviceName":"service-1","servicePort":80},"path":"/news"},{"backend":{"serviceName":"service-2","servicePort":80},"path":"/sports"}]}}]}}

  kubernets.io/ingress.class:                  nginx
  nginx.ingress.kubernetes.io/rewrite-target:  /
Events:
  Type    Reason          Age   From                      Message
  ----    ------          ----  ----                      -------
  Normal  AddedOrUpdated  11s   nginx-ingress-controller  Configuration for default/nginx-ingress-uri-demo was added or updated
  Normal  AddedOrUpdated  11s   nginx-ingress-controller  Configuration for default/nginx-ingress-uri-demo was added or updated
  Normal  AddedOrUpdated  11s   nginx-ingress-controller  Configuration for default/nginx-ingress-uri-demo was added or updated

4、準備測試，站點中創建對應的路徑

[root@node-1 ~]# kubectl exec -it service-1-7b66bf758f-xj9jh /bin/bash
root@service-1-7b66bf758f-xj9jh:/# echo "service-1 website page" >/usr/share/nginx/html/news

[root@node-1 ~]# kubectl exec -it service-2-7c7444684d-w9cv9 /bin/bash
root@service-2-7c7444684d-w9cv9:/usr/local/apache2# echo "service-2 website page" >/usr/local/apache2/htdocs/sports

5、測試驗證

[root@node-1 ~]# curl http://demo.happylau.cn/news --resolve demo.happylau.cn:80:10.254.100.101
service-1 website page
[root@node-1 ~]# curl http://demo.happylau.cn/sports --resolve demo.happylau.cn:80:10.254.100.101
service-2 website page

6、通過上述的驗證測試可以得知，ingress支持URI的路由方式轉發，其對應在ingress中的配置文件內容是怎樣的呢，我們看下ingress controller生成對應的nginx配置文件內容，實際是通過ingress的location來實現，將不同的localtion轉發至不同的upstream以實現service的關聯，配置文件如下：

[root@node-1 ~]# kubectl exec -it nginx-ingress-7mpfc -n nginx-ingress /bin/bash
nginx@nginx-ingress-7mpfc:/$ cat /etc/nginx/conf.d/default-nginx-ingress-uri-demo.conf |grep -v "^$"
# configuration for default/nginx-ingress-uri-demo
#定義兩個upstream和後端的service關聯
upstream default-nginx-ingress-uri-demo-demo.happylau.cn-service-1-80 {
    zone default-nginx-ingress-uri-demo-demo.happylau.cn-service-1-80 256k;
    random two least_conn;
    server 10.244.2.163:80 max_fails=1 fail_timeout=10s max_conns=0;
}

upstream default-nginx-ingress-uri-demo-demo.happylau.cn-service-2-80 {
    zone default-nginx-ingress-uri-demo-demo.happylau.cn-service-2-80 256k;
    random two least_conn;
    server 10.244.1.148:80 max_fails=1 fail_timeout=10s max_conns=0;    
}

server {
    listen 80;
    server_tokens on;
    server_name demo.happylau.cn;

  #定義location實現代理，通過proxy_pass和後端的service關聯
    location /news {
        proxy_http_version 1.1;
        proxy_connect_timeout 60s;
        proxy_read_timeout 60s;
        proxy_send_timeout 60s;
        client_max_body_size 1m;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Port $server_port;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_buffering on;
        proxy_pass http://default-nginx-ingress-uri-demo-demo.happylau.cn-service-1-80;
    }

    location /sports {
        proxy_http_version 1.1;
        proxy_connect_timeout 60s;
        proxy_read_timeout 60s;
        proxy_send_timeout 60s;
        client_max_body_size 1m;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Port $server_port;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_buffering on;
        proxy_pass http://default-nginx-ingress-uri-demo-demo.happylau.cn-service-2-80; 
    }   
}

3.4 Ingress虛擬主機

ingress支持基於名稱的虛擬主機，實現單個IP多個域名轉發的需求，通過請求頭部攜帶主機名方式區分開，將上個章節的ingress刪除，使用service-1和service-2兩個service來做演示。

1、創建ingress規則，通過主機名實現轉發規則

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx-ingress-virtualname-demo
  labels:
    ingres-controller: nginx
  annotations:
    kubernets.io/ingress.class: nginx
spec:
  rules:
  - host: news.happylau.cn
    http:
      paths:
      - path: /
        backend:
          serviceName: service-1 
          servicePort: 80
  - host: sports.happylau.cn
    http:
      paths:
      - path: /
        backend:
          serviceName: service-2 
          servicePort: 80

2、生成ingress規則並查看詳情，一個ingress對應兩個HOSTS

[root@node-1 nginx-ingress]# kubectl apply -f nginx-ingress-virtualname.yaml 
ingress.extensions/nginx-ingress-virtualname-demo created

#查看列表
[root@node-1 nginx-ingress]# kubectl get ingresses nginx-ingress-virtualname-demo 
NAME                             HOSTS                                 ADDRESS   PORTS   AGE
nginx-ingress-virtualname-demo   news.happylau.cn,sports.happylau.cn             80      12s

#查看詳情
[root@node-1 nginx-ingress]# kubectl describe ingresses nginx-ingress-virtualname-demo
Name:             nginx-ingress-virtualname-demo
Namespace:        default
Address:          
Default backend:  default-http-backend:80 (<none>)
Rules:
  Host                Path  Backends
  ----                ----  --------
  news.happylau.cn    
                      /   service-1:80 (10.244.2.163:80)
  sports.happylau.cn  
                      /   service-2:80 (10.244.1.148:80)
Annotations:
  kubectl.kubernetes.io/last-applied-configuration:  {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernets.io/ingress.class":"nginx"},"labels":{"ingres-controller":"nginx"},"name":"nginx-ingress-virtualname-demo","namespace":"default"},"spec":{"rules":[{"host":"news.happylau.cn","http":{"paths":[{"backend":{"serviceName":"service-1","servicePort":80},"path":"/"}]}},{"host":"sports.happylau.cn","http":{"paths":[{"backend":{"serviceName":"service-2","servicePort":80},"path":"/"}]}}]}}

  kubernets.io/ingress.class:  nginx
Events:
  Type    Reason          Age   From                      Message
  ----    ------          ----  ----                      -------
  Normal  AddedOrUpdated  28s   nginx-ingress-controller  Configuration for default/nginx-ingress-virtualname-demo was added or updated
  Normal  AddedOrUpdated  28s   nginx-ingress-controller  Configuration for default/nginx-ingress-virtualname-demo was added or updated
  Normal  AddedOrUpdated  28s   nginx-ingress-controller  Configuration for default/nginx-ingress-virtualname-demo was added or updated

3、準備測試數據並測試

[root@node-1 ~]# kubectl exec -it service-1-7b66bf758f-xj9jh /bin/bash
root@service-1-7b66bf758f-xj9jh:/# echo "news demo" >/usr/share/nginx/html/index.html

[root@node-1 ~]# kubectl exec -it service-2-7c7444684d-w9cv9 /bin/bash  
root@service-2-7c7444684d-w9cv9:/usr/local/apache2# echo "sports demo"  >/usr/local/apache2/htdocs/index.html

測試：

[root@node-1 ~]# curl http://news.happylau.cn --resolve news.happylau.cn:80:10.254.100.102
news demo
[root@node-1 ~]# curl http://sports.happylau.cn --resolve sports.happylau.cn:80:10.254.100.102
sports demo

4、查看nginx的配置文件內容，通過在server中定義不同的server_name以區分，代理到不同的upstream以實現service的代理。

# configuration for default/nginx-ingress-virtualname-demo
upstream default-nginx-ingress-virtualname-demo-news.happylau.cn-service-1-80 {
    zone default-nginx-ingress-virtualname-demo-news.happylau.cn-service-1-80 256k;
    random two least_conn;
    server 10.244.2.163:80 max_fails=1 fail_timeout=10s max_conns=0;
}

upstream default-nginx-ingress-virtualname-demo-sports.happylau.cn-service-2-80 {
    zone default-nginx-ingress-virtualname-demo-sports.happylau.cn-service-2-80 256k;
    random two least_conn;
    server 10.244.1.148:80 max_fails=1 fail_timeout=10s max_conns=0;
}

server {
    listen 80;
    server_tokens on;
    server_name news.happylau.cn;
    location / {
        proxy_http_version 1.1;
        proxy_connect_timeout 60s;
        proxy_read_timeout 60s;
        proxy_send_timeout 60s;
        client_max_body_size 1m;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Port $server_port;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_buffering on;

        proxy_pass http://default-nginx-ingress-virtualname-demo-news.happylau.cn-service-1-80;

  }
}
server {
    listen 80;  
    server_tokens on;
    server_name sports.happylau.cn;

    location / {
        proxy_http_version 1.1;
        proxy_connect_timeout 60s;
        proxy_read_timeout 60s;
        proxy_send_timeout 60s;
        client_max_body_size 1m;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Port $server_port;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_buffering on;

        proxy_pass http://default-nginx-ingress-virtualname-demo-sports.happylau.cn-service-2-80;   

    }   
}

3.5 Ingress TLS加密

四層的負載均衡無法支持https請求，當前大部分業務都要求以https方式接入，Ingress能支持https的方式接入，通過Secrets存儲證書+私鑰，實現https接入，同時還能支持http跳轉功能。對於用戶的請求流量來說，客戶端到ingress controller是https流量，ingress controller到後端service則是http，提高用戶訪問性能，如下介紹ingress TLS功能實現步驟。

1、生成自簽名證書和私鑰

[root@node-1 ~]# openssl req -x509 -newkey rsa:2048 -nodes -days 365 -keyout tls.key -out tls.crt
Generating a 2048 bit RSA private key
....................................................+++
........................................+++
writing new private key to 'tls.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:CN        #國家
State or Province Name (full name) []:GD    #省份
Locality Name (eg, city) [Default City]:ShenZhen  #城市
Organization Name (eg, company) [Default Company Ltd]:Tencent    #公司 
Organizational Unit Name (eg, section) []:HappyLau  #組織
Common Name (eg, your name or your server's hostname) []:www.happylau.cn  #域名
Email Address []:[email protected]       #郵箱地址

#tls.crt爲證書，tls.key爲私鑰
[root@node-1 ~]# ls tls.* -l
-rw-r--r-- 1 root root 1428 12月 26 13:21 tls.crt
-rw-r--r-- 1 root root 1708 12月 26 13:21 tls.key

2、配置Secrets，將證書和私鑰配置到Secrets中

[root@node-1 ~]# kubectl create secret tls happylau-sslkey --cert=tls.crt --key=tls.key 
secret/happylau-sslkey created

查看Secrets詳情,證書和私要包含在data中，文件名爲兩個不同的key：tls.crt和tls.key
[root@node-1 ~]# kubectl describe secrets happylau-sslkey 
Name:         happylau-sslkey
Namespace:    default
Labels:       <none>
Annotations:  <none>

Type:  kubernetes.io/tls

Data
====
tls.crt:  1428 bytes
tls.key:  1708 bytes

3、配置ingress調用Secrets實現SSL證書加密

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx-ingress-tls-demo
  labels:
    ingres-controller: nginx
  annotations:
    kubernets.io/ingress.class: nginx
spec:
  tls:
  - hosts:
    - news.happylau.cn
    - sports.happylau.cn
    secretName: happylau-sslkey
  rules:
  - host: news.happylau.cn
    http:
      paths:
      - path: /
        backend:
          serviceName: service-1 
          servicePort: 80
  - host: sports.happylau.cn
    http:
      paths:
      - path: /
        backend:
          serviceName: service-2 
          servicePort: 80

4、創建ingress並查看ingress詳情

[root@node-1 nginx-ingress]# kubectl describe ingresses nginx-ingress-tls-demo 
Name:             nginx-ingress-tls-demo
Namespace:        default
Address:          
Default backend:  default-http-backend:80 (<none>)
TLS:
  happylau-sslkey terminates news.happylau.cn,sports.happylau.cn
Rules:
  Host                Path  Backends
  ----                ----  --------
  news.happylau.cn    
                      /   service-1:80 (10.244.2.163:80)
  sports.happylau.cn  
                      /   service-2:80 (10.244.1.148:80)
Annotations:
  kubectl.kubernetes.io/last-applied-configuration:  {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernets.io/ingress.class":"nginx"},"labels":{"ingres-controller":"nginx"},"name":"nginx-ingress-tls-demo","namespace":"default"},"spec":{"rules":[{"host":"news.happylau.cn","http":{"paths":[{"backend":{"serviceName":"service-1","servicePort":80},"path":"/"}]}},{"host":"sports.happylau.cn","http":{"paths":[{"backend":{"serviceName":"service-2","servicePort":80},"path":"/"}]}}],"tls":[{"hosts":["news.happylau.cn","sports.happylau.cn"],"secretName":"happylau-sslkey"}]}}

  kubernets.io/ingress.class:  nginx
Events:
  Type    Reason          Age   From                      Message
  ----    ------          ----  ----                      -------
  Normal  AddedOrUpdated  22s   nginx-ingress-controller  Configuration for default/nginx-ingress-tls-demo was added or updated
  Normal  AddedOrUpdated  22s   nginx-ingress-controller  Configuration for default/nginx-ingress-tls-demo was added or updated
  Normal  AddedOrUpdated  22s   nginx-ingress-controller  Configuration for default/nginx-ingress-tls-demo was added or updated

5、 將news.happylau.cn和sports.happylau.cn寫入到hosts文件中，並通過https://news.happylau.cn 的方式訪問，瀏覽器訪問內容提示證書如下，信任證書即可訪問到站點內容。

查看證書詳情，正是我們製作的自簽名證書，生產實際使用時，推薦使用CA機構頒發簽名證書。

6、接下來查看一下tls配置https的nginx配置文件內容，可以看到在server塊啓用了https並配置證書，同時配置了http跳轉，因此直接訪問http也能夠實現自動跳轉到https功能。

# configuration for default/nginx-ingress-tls-demo
upstream default-nginx-ingress-tls-demo-news.happylau.cn-service-1-80 {
    zone default-nginx-ingress-tls-demo-news.happylau.cn-service-1-80 256k;
    random two least_conn;
    server 10.244.2.163:80 max_fails=1 fail_timeout=10s max_conns=0;    
}

upstream default-nginx-ingress-tls-demo-sports.happylau.cn-service-2-80 {
    zone default-nginx-ingress-tls-demo-sports.happylau.cn-service-2-80 256k;
    random two least_conn;
    server 10.244.1.148:80 max_fails=1 fail_timeout=10s max_conns=0;    
}

server {
    listen 80;

    listen 443 ssl;     #https監聽端口，證書和key，實現和Secrets關聯
    ssl_certificate /etc/nginx/secrets/default-happylau-sslkey;
    ssl_certificate_key /etc/nginx/secrets/default-happylau-sslkey;

    server_tokens on;
    server_name news.happylau.cn;

  #http跳轉功能，即訪問http會自動跳轉至https
    if ($scheme = http) {
        return 301 https://$host:443$request_uri;
    }

    location / {
        proxy_http_version 1.1;
        proxy_connect_timeout 60s;
        proxy_read_timeout 60s;
        proxy_send_timeout 60s;
        client_max_body_size 1m;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Port $server_port;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_buffering on;

        proxy_pass http://default-nginx-ingress-tls-demo-news.happylau.cn-service-1-80; 
  } 
}

server {
    listen 80;
    listen 443 ssl;
    ssl_certificate /etc/nginx/secrets/default-happylau-sslkey;
    ssl_certificate_key /etc/nginx/secrets/default-happylau-sslkey;

    server_tokens on;
    server_name sports.happylau.cn;

    if ($scheme = http) {
        return 301 https://$host:443$request_uri;
    }

    location / {
        proxy_http_version 1.1;
        proxy_connect_timeout 60s;
        proxy_read_timeout 60s;
        proxy_send_timeout 60s;
        client_max_body_size 1m;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Port $server_port;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_buffering on;

        proxy_pass http://default-nginx-ingress-tls-demo-sports.happylau.cn-service-2-80;   
    }   
}

4. Nginx Ingress高級功能

4.1 定製化參數

ingress controller提供了基礎反向代理的功能，如果需要定製化nginx的特性或參數，需要通過ConfigMap和Annotations來實現，兩者實現的方式有所不同，ConfigMap用於指定整個ingress集羣資源的基本參數，修改後會被所有的ingress對象所繼承；Annotations則被某個具體的ingress對象所使用，修改只會影響某個具體的ingress資源，衝突時其優先級高於ConfigMap。

4.1.1 ConfigMap自定義參數

安裝nginx ingress controller時默認會包含一個空的ConfigMap，可以通過ConfigMap來自定義nginx controller的默認參數，如下以修改一些參數爲例：

1、 定義ConfigMap參數

kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-config
  namespace: nginx-ingress
data:
  proxy-connect-timeout: "10s"
  proxy-read-timeout: "10s"
  proxy-send-timeout: "10"
  client-max-body-size: "3m"

2、 應用配置並查看ConfigMap配置

[root@node-1 ~]# kubectl get configmaps -n nginx-ingress nginx-config -o yaml
apiVersion: v1
data:
  client-max-body-size: 3m
  proxy-connect-timeout: 10s
  proxy-read-timeout: 10s
  proxy-send-timeout: 10s
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"client-max-body-size":"3m","proxy-connect-timeout":"10s","proxy-read-timeout":"10s","proxy-send-timeout":"10"},"kind":"ConfigMap","metadata":{"annotations":{},"name":"nginx-config","namespace":"nginx-ingress"}}
  creationTimestamp: "2019-12-24T04:39:23Z"
  name: nginx-config
  namespace: nginx-ingress
  resourceVersion: "13845543"
  selfLink: /api/v1/namespaces/nginx-ingress/configmaps/nginx-config
  uid: 9313ae47-a0f0-463e-a25a-1658f1ca0d57

3 、此時，ConfigMap定義的配置參數會被集羣中所有的Ingress資源繼承（除了annotations定義之外）

有很多參數可以定義，詳情配置可參考方文檔說明：https://github.com/nginxinc/kubernetes-ingress/blob/master/docs/configmap-and-annotations.md#Summary-of-ConfigMap-and-Annotations

4.1.2 Annotations自定義參數

ConfigMap定義的是全局的配置參數，修改後所有的配置都會受影響，如果想針對某個具體的ingress資源自定義參數，則可以通過Annotations來實現，下面開始以實際的例子演示Annotations的使用。

1、修改ingress資源，添加annotations的定義,通過nginx.org組修改了一些參數，如proxy-connect-timeout，調度算法爲round_robin（默認爲least _conn）

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx-ingress-demo
  labels:
    ingres-controller: nginx
  annotations:
    kubernets.io/ingress.class: nginx
    nginx.org/proxy-connect-timeout: "30s"
    nginx.org/proxy-send-timeout: "20s"
    nginx.org/proxy-read-timeout: "20s"
    nginx.org/client-max-body-size: "2m"
    nginx.org/fail-timeout: "5s"
    nginx.org/lb-method: "round_robin" 
spec:
  rules:
  - host: www.happylau.cn
    http:
      paths:
      - path: /
        backend:
          serviceName: ingress-demo
          servicePort: 80

2、 重新應用ingress對象並查看參數配置情況

由上面的演示可得知，Annotations的優先級高於ConfigMapMap，Annotations修改參數只會影響到某一個具體的ingress資源，其定義的方法和ConfigMap相相近似，但又有差別，部分ConfigMap的參數Annotations無法支持，反過來Annotations定義的參數ConfigMap也不一定支持，下圖列舉一下常規支持參數情況：

ConfigMap和Annotations詳細支持說明：鏈接說明

4.2 虛擬主機和路由

安裝nginx ingress時我們安裝了一個customresourcedefinitions自定義資源，其能夠提供除了默認ingress功能之外的一些高級特性如

• 虛擬主機VirtualServer
• 虛擬路由VirtualServerRoute
• 健康檢查Healthcheck
• 流量切割Split
• 會話保持SessionCookie
• 重定向Redirect

這些功能大部分依賴於Nginx Plus高級版本的支持，社區版本僅支持部分，對於企業級開發而言，豐富更多的功能可以購買企業級Nginx Plus版本。如下以通過VirtualServer和VirtualServerRoute定義upstream配置爲例演示功能使用。

1、定義VirtualServer資源,其配置和ingress資源對象類似，能支持的功能會更豐富一點

apiVersion: k8s.nginx.org/v1
kind: VirtualServer
metadata:
  name: cafe
spec:
  host: cafe.example.com
  tls:
    secret: cafe-secret
  upstreams:
  - name: tea
    service: tea-svc
    port: 80
    name: tea
    service: ingress-demo 
    subselector:
    version: canary
    lb-method: round_robin
    fail-timeout: 10s
    max-fails: 1
    max-conns: 32
    keepalive: 32
    connect-timeout: 30s
    read-timeout: 30s
    send-timeout: 30s
    next-upstream: "error timeout non_idempotent"
    next-upstream-timeout: 5s
    next-upstream-tries: 10
    client-max-body-size: 2m
    tls:
      enable: true
  routes:
  - path: /tea
    action:
      pass: tea

2、 應用資源並查看VirtualServer資源列表

[root@node-1 ~]# kubectl apply -f vs.yaml 
virtualserver.k8s.nginx.org/cafe unchanged
[root@node-1 ~]# kubectl get virtualserver
NAME                 AGE
cafe                 2m52s

3、檢查ingress控制器的配置文件情況,生成的配置和upstream定義一致

nginx@nginx-ingress-7mpfc:/etc/nginx/conf.d$ cat vs_default_cafe.conf 
upstream vs_default_cafe_tea {
    zone vs_default_cafe_tea 256k;
    server 10.244.0.51:80 max_fails=1 fail_timeout=10s max_conns=32;
    server 10.244.1.146:80 max_fails=1 fail_timeout=10s max_conns=32;
    server 10.244.1.147:80 max_fails=1 fail_timeout=10s max_conns=32;
    server 10.244.2.162:80 max_fails=1 fail_timeout=10s max_conns=32;
    keepalive 32;
}

server {
    listen 80;
    server_name cafe.example.com;
    listen 443 ssl;
    ssl_certificate /etc/nginx/secrets/default;
    ssl_certificate_key /etc/nginx/secrets/default;
    ssl_ciphers NULL;
    server_tokens "on";

    location /tea {
        proxy_connect_timeout 30s;
        proxy_read_timeout 30s;
        proxy_send_timeout 30s;
        client_max_body_size 2m;
        proxy_max_temp_file_size 1024m;
        proxy_buffering on;
        proxy_http_version 1.1;
        set $default_connection_header "";
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $vs_connection_header;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Port $server_port;
        proxy_set_header X-Forwarded-Proto $scheme;

        proxy_pass https://vs_default_cafe_tea;
        proxy_next_upstream error timeout non_idempotent;
        proxy_next_upstream_timeout 5s;
        proxy_next_upstream_tries 10;   
    }   
}

寫在最後

本文詳細介紹了基於nginx的ingress實現，通過實際的案例演示ingress的安裝部署，基於虛擬主機的配置，基於TLS加密實現https，高級章節中介紹了負載均衡參數定製，自定義資源虛擬主機和虛擬路由的實現，通過該章節相信能加深對ingress服務暴露機制的理解。實現ingress controller的方式有多種，下一個章節我們將介紹基於HAproxy和TKE Ingress控制器的實現。

參考文獻

Ingress配置：https://kubernetes.io/docs/concepts/services-networking/ingress/

Ingress控制器：https://kubernetes.io/docs/concepts/services-networking/ingress-controllers/

Nginx ingress安裝文檔：https://github.com/nginxinc/kubernetes-ingress/blob/master/docs/installation.md

Nginx ingress文檔說明：https://github.com/nginxinc/kubernetes-ingress/tree/master/docs

虛擬主機和路由：https://github.com/nginxinc/kubernetes-ingress/blob/master/docs/virtualserver-and-virtualserverroute.md

---

當你的才華撐不起你的野心時，你就應該靜下心來學習

返回kubernetes系列教程目錄