關於作者 劉海平(HappyLau )雲計算高級顧問 目前在騰訊雲從事公有云相關工作,曾就職於酷狗,EasyStack,擁有多年公有云+私有云計算架構設計,運維,交付相關經驗,參與了酷狗,南方電網,國泰君安等大型私有云平臺建設,精通Linux,Kubernetes,OpenStack,Ceph等開源技術,具有豐富RHCA/OpenStack/Linux授課經驗。
寫在前面
上一篇文章中介紹了基於Nginx實現Ingress Controller的實現,介紹了Nginx Ingress Controller安裝、相關功能,TLS,高級特性等介紹,本章開始介紹基於騰訊雲TKE實現ingress服務暴露。
1. TKE ingress
1.1 TKE ingress架構
TKE是Tencent Kubernetes Engine即騰訊雲基於kubernetes提供的公有云上容器雲服務,TKE提供了兩種暴露服務的方式:service和ingress。
- 內網CLB,四層負載均衡,提供VPC內訪問,通過node節點的NodePort轉發至service;
- 外網CLB,四層負載均衡,提供公網訪問,需要node節點具有訪問公網的能力;
- ingress, 七層負載均衡,提供http和https接入,提供ingress控制器的功能,藉助NodePort轉發
要使用TKE的ingress功能,需要了解一下相關的組件內容:
- l7-lb-controller ingress客戶端,安裝在kube-system,用於解析ingress配置並更新CLB的規則
- CLB 七層負載均衡,提供ingress controller的功能,根據ingress規則創建http/https監聽器,配置轉發規則,以NodePort端口綁定後端RS
- Service 用於ingress服務發現,通過NodePort方式接入CLB
- 證書 用於提供https接入,配置在CLB負載均衡上,提供CA簽名證書,通過Secrets封裝給CLB使用
由於nginx ingress controller是直接以Pod的形勢部署在kubernetes集羣中,藉助於service的服務發現可直接實現和pod通訊,而TKE中ingress controller未直接部署在k8s集羣中,網絡的接入需藉助於service的NodePort實現接入,其數據流如下圖:
1.2 ingress虛擬主機
環境說明: 創建兩個Deployment並以NodePort方式暴露服務,www1.happylau.cn對應tke-app-1服務,同理www2.happylau.cn對應tke-app-2服務,如下演示操作過程:
1、創建兩個Deployments
[root@VM_10_2_centos ingress]# kubectl create deployment tke-app-1 --image=nginx:1.7.9
[root@VM_10_2_centos ingress]# kubectl create deployment tke-app-2 --image=nginx:1.7.9
2、 將兩個Deployment以NodePort的方式暴露服務
[root@VM_10_2_centos ~]# kubectl expose deployment tke-app-1 --port=80 --type=NodePort
[root@VM_10_2_centos ~]# kubectl expose deployment tke-app-2 --port=80 --type=NodePort
查看服務列表
[root@VM_10_2_centos ~]# kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 172.16.255.1 <none> 443/TCP 83d
tke-app-1 NodePort 172.16.255.91 <none> 80:30597/TCP 2s
tke-app-2 NodePort 172.16.255.236 <none> 80:31674/TCP 73s
3、定義ingress規則,定義兩個host將不同主機轉發至backend不同的service
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: tke-ingress-demo
annotations:
kubernetes.io/ingress.class: qcloud
spec:
rules:
- host: www1.happylau.cn
http:
paths:
- path: /
backend:
serviceName: tke-app-1
servicePort: 80
- host: www2.happylau.cn
http:
paths:
- path: /
backend:
serviceName: tke-app-2
servicePort: 80
4、 應用ingress規則,並查看ingress詳情,可以看到ingress創建了一個公網CLB實例
#應用ingress規則
[root@VM_10_2_centos ingress]# kubectl apply -f tke-ingress-demo.yaml
ingress.extensions/tke-ingress-demo created
#查看ingress列表
[root@VM_10_2_centos ingress]# kubectl get ingresses
NAME HOSTS ADDRESS PORTS AGE
tke-ingress-demo www1.happylau.cn,www2.happylau.cn 140.143.84.xxx 80 67s
#查看 ingress詳情
[root@VM_10_2_centos ingress]# kubectl describe ingresses tke-ingress-demo
Name: tke-ingress-demo
Namespace: default
Address: 140.143.84.xxx
Default backend: default-http-backend:80 (<none>)
Rules:
Host Path Backends
---- ---- --------
www1.happylau.cn
/ tke-app-1:80 (172.16.1.15:80)
www2.happylau.cn
/ tke-app-2:80 (172.16.2.17:80)
Annotations:
kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.class":"qcloud"},"name":"tke-ingress-demo","namespace":"default"},"spec":{"rules":[{"host":"www1.happylau.cn","http":{"paths":[{"backend":{"serviceName":"tke-app-1","servicePort":80},"path":"/"}]}},{"host":"www2.happylau.cn","http":{"paths":[{"backend":{"serviceName":"tke-app-2","servicePort":80},"path":"/"}]}}]}}
kubernetes.io/ingress.class: qcloud
kubernetes.io/ingress.qcloud-loadbalance-id: lb-a0xwhcx3
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal EnsuringIngress 69s (x3 over 89s) loadbalancer-controller Ensuring ingress
Normal CREATE 69s (x2 over 70s) loadbalancer-controller create loadbalancer succ
Normal EnsuredIngress 68s (x3 over 70s) loadbalancer-controller Ensured ingress
5、測試驗證,將IP和域名寫入到hosts文件中,訪問域名測試驗證,如下通過curl解析的方式測試驗證
6、ingress會創建一個CLB,並在CLB中創建監聽器、設置轉發規則、綁定後端RS,下圖是CLB上自動生成的規則
通過上面演示可知:
- 自動創建CLB實例
- CLB實例上配置監聽器
- 配置轉發規則
- 綁定Node節點
- 綁定端口爲service創建的NodePort
1.3 ingress證書加密
TKE支持將在CLB中加載證書實現https加密傳輸,證書是經過第三方認證的CA簽名過的證書,需要先購買好證書,通過Secrets對象在kubernetes集羣中定義,如下演示https的實現。
1、 通過Secrets創建證書,先獲取到證書的id,如果沒有則先創建證書,證書管理,本文以證書id TKPmsWb3 爲例,通過stringData能實現base64自動加密
apiVersion: v1
kind: Secret
metadata:
name: ingress-ssl-key
stringData:
qcloud_cert_id: TKPmsWb3
type: Opaque
#生成Secrets對象
[root@VM_10_2_centos ingress]# kubectl apply -f ingress-secret.yaml
secret/ingress-ssl-key created
[root@VM_10_2_centos ingress]# kubectl get secrets ingress-ssl-key
NAME TYPE DATA AGE
ingress-ssl-key Opaque 1 7s
#查看secrets詳情,可得知VEtQbXNXYjM= 已自動通過base64加密
[root@VM_10_2_centos ingress]# kubectl get secrets ingress-ssl-key -o yaml
apiVersion: v1
data:
qcloud_cert_id: VEtQbXNXYjM=
kind: Secret
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","kind":"Secret","metadata":{"annotations":{},"name":"ingress-ssl-key","namespace":"default"},"stringData":{"qcloud_cert_id":"TKPmsWb3"},"type":"Opaque"}
creationTimestamp: "2020-01-03T11:53:33Z"
name: ingress-ssl-key
namespace: default
resourceVersion: "7083702418"
selfLink: /api/v1/namespaces/default/secrets/ingress-ssl-key
uid: aaea4a86-2e1f-11ea-a618-ae9224ffad1a
type: Opaque
#可以通過base64查看解密後的內容,和配置文件中定義的id一致
[root@VM_10_2_centos ingress]# echo VEtQbXNXYjM= | base64 -d
TKPmsWb3
2、準備環境,創建一個nginx的Deployment
[root@VM_10_2_centos ~]# kubectl create deployment tke-ingress-ssl-demo --image=nginx:1.7.9
deployment.apps/tke-ingress-ssl-demo created
[root@VM_10_2_centos ~]# kubectl get deployments
NAME READY UP-TO-DATE AVAILABLE AGE
tke-ingress-ssl-demo 1/1 1 1 6s
3、將Deployment暴露以NodePort類型暴露service
[root@VM_10_2_centos ~]# kubectl expose deployment tke-ingress-ssl-demo --port=80 --type=NodePort
service/tke-ingress-ssl-demo exposed
[root@VM_10_2_centos ~]# kubectl get service tke-ingress-ssl-demo -o yaml
apiVersion: v1
kind: Service
metadata:
creationTimestamp: "2020-01-03T12:00:05Z"
labels:
app: tke-ingress-ssl-demo
name: tke-ingress-ssl-demo
namespace: default
resourceVersion: "7083890283"
selfLink: /api/v1/namespaces/default/services/tke-ingress-ssl-demo
uid: 94659f42-2e20-11ea-a618-ae9224ffad1a
spec:
clusterIP: 172.16.255.64
externalTrafficPolicy: Cluster
ports:
- nodePort: 30324
port: 80
protocol: TCP
targetPort: 80
selector:
app: tke-ingress-ssl-demo
sessionAffinity: None
type: NodePort #類型爲NodePort
status:
loadBalancer: {}
4、定義ingress規則,加載證書實現https轉發
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: tke-ingress-ssl
annotations:
kubernetes.io/ingress.class: qcloud
qcloud_cert_id: TKPmsWb3
spec:
rules:
- host: www.happylauliu.cn
http:
paths:
- path: /
backend:
serviceName: tke-ingress-ssl-demo
servicePort: 80
tls:
- hosts:
- www.happylauliu.cn
secretName: ingress-ssl-key
5、應用ingress規則,並查看詳情,此時已正常創建CLB並配置規則
[root@VM_10_2_centos ingress]# kubectl apply -f ingress-demo.yaml
ingress.extensions/tke-ingress-ssl created
#查看ingress詳情
[root@VM_10_2_centos ingress]# kubectl describe ingresses tke-ingress-ssl
Name: tke-ingress-ssl
Namespace: default
Address: 140.143.83.xxx #CLB的外網IP
Default backend: default-http-backend:80 (<none>)
TLS:
ingress-ssl-key terminates www.happylauliu.cn
Rules:
Host Path Backends
---- ---- --------
www.happylauliu.cn
/ tke-ingress-ssl-demo:80 (172.16.0.25:80)
Annotations:
qcloud_cert_id: TKPmsWb3
kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.class":"qcloud","qcloud_cert_id":"TKPmsWb3"},"name":"tke-ingress-ssl","namespace":"default"},"spec":{"rules":[{"host":"www.happylauliu.cn","http":{"paths":[{"backend":{"serviceName":"tke-ingress-ssl-demo","servicePort":80},"path":"/"}]}}],"tls":[{"hosts":["www.happylauliu.cn"],"secretName":"ingress-ssl-key"}]}}
kubernetes.io/ingress.class: qcloud
kubernetes.io/ingress.qcloud-loadbalance-id: lb-2kcrtwbn #CLB的實例id
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal EnsuringIngress 51s (x3 over 73s) loadbalancer-controller Ensuring ingress
Normal CREATE 51s (x2 over 52s) loadbalancer-controller create loadbalancer succ
Normal EnsuredIngress 49s (x3 over 52s) loadbalancer-controller Ensured ingress
6、測試驗證,hosts文件中解析www.happylauliu.cn到CLB的VIP,或者DNS解析,打開瀏覽器訪問站點,由於是經過CA認證簽名的證書,因此沒有提示告警信息,查看證書的詳情信息
7、查看CLB的配置可得知,CLB上配置了443的監聽端口,並關聯了證書,採用單向認證方式
通過CLB的配置規則可知,CLB配置了監聽443的監聽器,80端口並未設置規則,因此此時無法訪問http,如何實現在TKE使用ingress實現http和https共存呢,可以通過定義kubernetes.io/ingress.http-rules和
kubernetes.io/ingress.https-rules實現
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: tke-ingress-ssl
annotations:
kubernetes.io/ingress.class: qcloud
kubernetes.io/ingress.rule-mix: "true" #開啓混合規則配置,kubernetes.io/ingress.http-rules配置規則
kubernetes.io/ingress.http-rules: '[{"host":"www.happylauliu.cn","path":"/","backend":{"serviceName":"tke-ingress-ssl-demo","servicePort":"80"}}]'
qcloud_cert_id: TKPmsWb3
spec:
rules:
- host: www.happylauliu.cn
http:
paths:
- path: /
backend:
serviceName: tke-ingress-ssl-demo
servicePort: 80
tls:
- hosts:
- www.happylauliu.cn
secretName: ingress-ssl-key
設置ingress.http-rules和ingress.https-rules註解之後,會在監聽器中創建http和https的轉發規則,並綁定RS,此時訪問http和https均能實現站點訪問,CLB對應的規則內容如下圖:
通過測試訪問http://www.happylauliu.cn/和https://www.happylauliu.cn/均能正常訪問,如果要實現訪問http自動跳轉到https,則可以在控制檯開啓自動跳轉的功能,如下圖:
開啓重定向功能後再次訪問http站點後此時會自動跳轉到https,如下圖所示location已經跳轉至https://www.happylauliu.cn/
寫在最後
通過上述的演示在騰訊雲公有云環境下ingress controller的實現,騰訊雲TKE通過使用CLB實現和kubernetes ingress集成,藉助於service的NodePort實現轉發,通過公有云專用的CLB能夠最大程度保障ingress接入性能。同時,ingress能夠使用騰訊雲上的證書實現https加密功能。
參考文獻
Ingress配置:https://kubernetes.io/docs/concepts/services-networking/ingress/
Ingress控制器:https://kubernetes.io/docs/concepts/services-networking/ingress-controllers/
ingress基本配置:https://cloud.tencent.com/document/product/457/31711
ingress證書:https://cloud.tencent.com/document/product/457/40538
CLB配置http自動跳轉:https://cloud.tencent.com/document/product/214/8839
當你的才華撐不起你的野心時,你就應該靜下心來學習