kubernetes容器集羣管理（9）- Ingress部署

一、Ingress邏輯

一個ingress可以配置用於提供外部可訪問的服務url、負載均衡流量、SSL終端和提供虛擬主機名配置。ingress controller負責實現（通常使用負載均衡器(loadbalancer)）入口（ingress）。但是它也可以配置你的邊緣路由器或額外的前端來幫助處理流量。

ingress不暴露任何端口或協議。將HTTP和HTTPS之外的服務公開到因特網通常使用類型是NodePort或loadbalance的service。

Ingress工具在每個節點上創建一個負載均衡用來代理所有pod，當客戶發起請求時，會直接請求ingress contraller  ，再根據關聯的server配置，由其轉發至具體pod。

二、Ingress搭建

2.1、部署github包及目錄規劃

[root@manage01 ~]# mkdir /opt/kubernetes/ingress
[root@manage01 ~]# cd /opt/kubernetes/ingress
[root@manage01 ~]# https://github.com/kubernetes/ingress-nginx/tree/nginx-0.18.0/deploy
[root@manage01 ~]# http://zhoudaxiaa.gitee.io/downgit/#/home  #github下載工具

[root@manage01 ingress-install]# ls
configmap.yaml        deployment.yaml  namespace.yaml  tcp-services-configmap.yaml
default-backend.yaml  mandatory.yaml   rbac.yaml       udp-services-configmap.yaml

2.2、創建一個命名空間，放置ingress相關配置。

[root@manage01 ingress-install]# cat namespace.yaml 
---

apiVersion: v1
kind: Namespace
metadata:
  name: ingress-nginx

[root@manage01 ingress-install]# kubectl create -f namespace.yaml

2.3、默認域名配置

該配置用來創建default-backend的depolyment和service
如果外界訪問的域名不存在的話，則會默認轉發到defalut-http-backend這個service，會直接返回404

[root@manage01 ingress-install]# cat default-backend.yaml 
---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: default-http-backend
  labels:
    app: default-http-backend
  namespace: ingress-nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: default-http-backend
  template:
    metadata:
      labels:
        app: default-http-backend
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - name: default-http-backend
        # Any image is permissible as long as:
        # 1. It serves a 404 page at /
        # 2. It serves 200 on a /healthz endpoint
        image: registry.cn-hangzhou.aliyuncs.com/google_containers/defaultbackend:1.4
#        image: gcr.io/google_containers/defaultbackend:1.4
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 30
          timeoutSeconds: 5
        ports:
        - containerPort: 8080
        resources:
          limits:
            cpu: 10m
            memory: 20Mi
          requests:
            cpu: 10m
            memory: 20Mi
---

apiVersion: v1
kind: Service
metadata:
  name: default-http-backend
  namespace: ingress-nginx
  labels:
    app: default-http-backend
spec:
  ports:
  - port: 80
    targetPort: 8080
  selector:
    app: default-http-backend

[root@manage01 ingress-install]# kubectl create -f default-backend.yaml

2.4、configmap 存放tcp udp 虛擬主機的配置

[root@manage01 ingress-install]# cat tcp-services-configmap.yaml 
---

kind: ConfigMap
apiVersion: v1
metadata:
  name: tcp-services
  namespace: ingress-nginx
[root@manage01 ingress-install]# cat udp-services-configmap.yaml 
---

kind: ConfigMap
apiVersion: v1
metadata:
  name: udp-services
  namespace: ingress-nginx

[root@manage01 ingress-install]# kubectl create -f tcp-services-configmap.yaml ^C
[root@manage01 ingress-install]# kubectl create -f udp-services-configmap.yaml

2.5、創建rbac授權

這個yaml文件主要是角色的創建和綁定，負責Ingress的RBAC授權的控制，其創建了Ingress用到的ServiceAccount、ClusterRole、Role、RoleBinding、ClusterRoleBinding

[root@manage01 ingress-install]# cat rbac.yaml 
---

apiVersion: v1
kind: ServiceAccount
metadata:
  name: nginx-ingress-serviceaccount
  namespace: ingress-nginx

---

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: nginx-ingress-clusterrole
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - endpoints
      - nodes
      - pods
      - secrets
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - "extensions"
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
        - events
    verbs:
        - create
        - patch
  - apiGroups:
      - "extensions"
    resources:
      - ingresses/status
    verbs:
      - update

---

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  name: nginx-ingress-role
  namespace: ingress-nginx
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - pods
      - secrets
      - namespaces
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - configmaps
    resourceNames:
      # Defaults to "<election-id>-<ingress-class>"
      # Here: "<ingress-controller-leader>-<nginx>"
      # This has to be adapted if you change either parameter
      # when launching the nginx-ingress-controller.
      - "ingress-controller-leader-nginx"
    verbs:
      - get
      - update
  - apiGroups:
      - ""
    resources:
      - configmaps
    verbs:
      - create
  - apiGroups:
      - ""
    resources:
      - endpoints
    verbs:
      - get

---

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: nginx-ingress-role-nisa-binding
  namespace: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: nginx-ingress-role
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: ingress-nginx

---

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: nginx-ingress-clusterrole-nisa-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: nginx-ingress-clusterrole
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: ingress-nginx

2.6、創建deployment

mv with-rbac.yaml deployment.yaml && vim deployment.yaml
這個文件創建nginx-ingress-controller這個deployment，副本數選擇兩個，一個node一個。Ingress-controller的作用是將新加入的Ingress進行轉化爲Nginx的配置。

Ingress Contronler 通過與 Kubernetes API 交互，能夠動態的獲取cluster中Ingress rules的變化，生成一段 Nginx 配置，再寫到 Nginx-ingress-control的 Pod 裏，reload pod 使規則生效。從而實現註冊的service及其對應域名/IP/Port的動態添加和解析。

[root@manage01 ingress-install]# cat deployment.yaml 
---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-ingress-controller
  namespace: ingress-nginx 
spec:
  replicas: 2
  selector:
    matchLabels:
      app: ingress-nginx
  template:
    metadata:
      labels:
        app: ingress-nginx
      annotations:
        prometheus.io/port: '10254'
        prometheus.io/scrape: 'true'
    spec:
      serviceAccountName: nginx-ingress-serviceaccount
##########增加hostNetwork：true這一行，這是直接定義Pod網絡的方式。定義後，Ingress-controller的IP就與宿主機上一樣，並且端口也是宿主機上的端口。這樣就可以通過宿主機直接訪問到Ingress-controller，然後Ingress-controller則會轉發我們的請求到響應後端。
##########
      hostNetwork: true
      containers:
        - name: nginx-ingress-controller
##########使用國內鏡像##########
          image: registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller:0.18.0
#          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.18.0
          args:
            - /nginx-ingress-controller
            - --default-backend-service=$(POD_NAMESPACE)/default-http-backend
            - --configmap=$(POD_NAMESPACE)/nginx-configuration
            - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
            - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
            - --publish-service=$(POD_NAMESPACE)/ingress-nginx
            - --annotations-prefix=nginx.ingress.kubernetes.io
##########部分安全選項配置，如果不熟建議註釋，否則啓動報錯##########
#          securityContext:
#            capabilities:
#                drop:
#                - ALL
#                add:
#                - NET_BIND_SERVICE
            # www-data -> 33
#            runAsUser: 33
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          ports:
          - name: http
            containerPort: 80
          - name: https
            containerPort: 443
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1

[root@manage01 ingress-install]# kubectl create -f deployment.yaml

2.7、節點部署完成

[root@manage01 ingress]# kubectl get pods -n ingress-nginx -o wide
NAME                                        READY   STATUS    RESTARTS   AGE   IP                NODE              NOMINATED NODE   READINESS GATES
default-http-backend-7765847556-lrs4w       1/1     Running   3          82m   172.18.20.7       192.168.192.130   <none>           <none>
nginx-ingress-controller-75d499dd56-scw28   1/1     Running   0          82m   192.168.192.129   192.168.192.129   <none>           <none>
nginx-ingress-controller-75d499dd56-vvpbd   1/1     Running   1          88m   192.168.192.130   192.168.192.130   <none>           <none>

三、Ingress-http配置

思路：首先創建deployment，和它所擁有的pod；然後創建service，和deployment關聯；
最後創建Ingress，和service關聯。

3.1、創建nginx和httpd的deployment和pod

[root@manage01 ingress]# kubectl run --image=nginx nginx
deployment.apps/nginx created

[root@manage01 ingress]# kubectl run --image=httpd httpd
deployment.apps/httpd created

[root@manage01 ingress]# kubectl get pods
NAME                    READY   STATUS    RESTARTS   AGE
httpd-7db5849b8-bxpcg   1/1     Running   0          2m51s
nginx-dbddb74b8-wtr7v   1/1     Running   0          3m2s

3.2、創建service

[root@manage01 ingress]# kubectl expose deployment nginx --port=80 --target-port=80
service/nginx exposed
[root@manage01 ingress]# kubectl expose deployment httpd --port=80 --target-port=80
service/httpd exposed

[root@manage01 ingress]# kubectl get svc -o wide
NAME         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)               AGE     SELECTOR
httpd        ClusterIP   10.10.10.221   <none>        80/TCP                3h18m   run=httpd
kubernetes   ClusterIP   10.10.10.1     <none>        443/TCP               11d     <none>
my-service   ClusterIP   10.10.10.242   <none>        10080/TCP,10443/TCP   3d3h    app=nginx
nginx        ClusterIP   10.10.10.109   <none>        80/TCP                80m     run=nginx

3.3、修改httpd和nginx容器首頁信息方便測試比對

[root@manage01 ingress]# kubectl exec -it httpd-7db5849b8-bxpcg bash
root@httpd-7db5849b8-bxpcg:/usr/local/apache2# cd htdocs/
root@httpd-7db5849b8-bxpcg:/usr/local/apache2/htdocs# echo "hello httpd!" > index.html   root@httpd-7db5849b8-bxpcg:/usr/local/apache2/htdocs# exit

[root@manage01 ingress]# kubectl exec -it nginx-dbddb74b8-wtr7v bash
root@nginx-dbddb74b8-wtr7v:/# cd /usr/share/nginx/html/
root@nginx-dbddb74b8-wtr7v:/usr/share/nginx/html# echo "hello nginx!" > index.html
root@nginx-dbddb74b8-wtr7v:/usr/share/nginx/html# exit

3.4、在node節點curl測試一下能否訪問

[root@manage01 ingress]#curl 10.10.10.221
hello httpd!
[root@manage01 ingress]#curl 10.10.10.109
hello nginx!

3.5、創建Ingress匹配serviceName(http訪問)

[root@manage01 ingress-xieyi]# cat ingress-nginx-http.yaml 
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: httpd-test
spec:
  rules:
  - host: haha.httpd.com
    http:
      paths:
      - backend:
          serviceName: httpd
          servicePort: 80
  - host: haha.nginx.com
    http:
      paths:
      - backend:
          serviceName: nginx
          servicePort: 80

[root@manage01 ingress-xieyi]# kubectl create -f  ingress-nginx-http.yaml

[root@manage01 ingress-xieyi]# kubectl get ingress
NAME         HOSTS                           ADDRESS   PORTS     AGE
httpd-test   haha.httpd.com,haha.nginx.com             80        82m

3.6、修改電腦host文件，把域名和IP對應，然後瀏覽器訪問

#C:\Windows\System32\drivers\etc\hosts
192.168.192.129 haha.httpd.com
192.168.192.130 haha.nginx.com

四、Ingress-https配置

4.1、需要創建證書授權

cfssl print-defaults csr > ca-csr.json #證書頒發機構
vim ca-csr.json
[root@k8s-master-101 https]# cat ca-csr.json 
{
    "CN": "amusitelangpao",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing"
        }
    ]
}


cfssl print-defaults config >ca-config.json
vim ca-config.json

[root@k8s-master-101 https]# cat ca-config.json 
{
    "signing": {
        "default": {
            "expiry": "168h"
        },
        "profiles": {
            "www": {
                "expiry": "8760h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth"
                ]
            },
            "client": {
                "expiry": "8760h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "client auth"
                ]
            }
        }
    }
}

cfssl gencert --initca ca-csr.json | cfssljson -bare ca –

cfssl print-defaults csr >server-csr.json
vim server-csr.json

[root@k8s-master-101 https]# cat server-csr.json 
{
    "CN": "www.amusitelangpao.com",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing"
        }
    ]
}

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem --config=ca-config.json --profile=www server-csr.json | cfssljson -bare server

kubectl create secret tls wangxiaoyu-https --key server-key.pem --cert server.pem

[root@manage01 ingress-xieyi]#  kubectl get secret
NAME                   TYPE                                  DATA   AGE
amusitelangpao-https   kubernetes.io/tls                     2      87m
default-token-zdl48    kubernetes.io/service-account-token   3      11d

4.2、創建Ingress匹配serviceName(https訪問)

[root@manage01 ingress-xieyi]# cat ingress-nginx-https.yaml 
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: https-test
spec:
  tls:
  - hosts:
    - www.amusitelangpao.com
    secretName: amusitelangpao-https
  rules:
  - host: www.amusitelangpao.com
    http:
      paths:
      - backend:
          serviceName: nginx
          servicePort: 80

[root@manage01 ingress-xieyi]# kubectl create -f ingress-nginx-https.yaml

4.3、修改hosts文件，瀏覽器訪問

#C:\Windows\System32\drivers\etc\hosts
192.168.192.129 www.amusitelangpao.com
192.168.192.130 www.amusitelangpao.com