Kubernetes(K8S)（六）——service（ClusterIP、NodePort、無頭服務、LoadBalancer、ExternalName等）

1.Service介紹

Service可以看作是一組提供相同服務的Pod對外的訪問接口。藉助Service，應用可以方便地實現服務發現和負載均衡。 service默認只支持4層負載均衡能力，沒有7層功能。（可以通過Ingress實現）

service的類型：
• ClusterIP：默認值，k8s系統給service自動分配的虛擬IP，只能在集羣內部訪問。
• NodePort：將Service通過指定的Node上的端口暴露給外部，訪問任意一個 NodeIP:nodePort都將路由到ClusterIP。
• LoadBalancer：在 NodePort 的基礎上，藉助 cloud provider 創建一個外部的負 載均衡器，並將請求轉發到 :NodePort，此模式只能在雲服務器上使用。
• ExternalName：將服務通過 DNS CNAME 記錄方式轉發到指定的域名（通過 spec.externlName 設定）。

Service 是由 kube-proxy 組件，加上 iptables 來共同實現的.
• kube-proxy 通過 iptables 處理 Service 的過程，需要在宿主機上設置相當多的 iptables 規則，如果宿主機有大量的Pod，不斷刷新iptables規則，會消耗大量的 CPU資源。
• IPVS模式的service，可以使K8s集羣支持更多量級的Pod。

---

2.開啓kube-proxy的ipvs模式

開啓kube-proxy的ipvs模式：
yum install -y ipvsadm 所有節點安裝
kubectl edit cm kube-proxy -n kube-system 修改IPVS模式 mode: “ipvs”
kubectl get pod -n kube-system |grep kube-proxy | awk '{system("kubectl delete pod "$1" -n kube-system")}'更新kube-proxy pod

IPVS模式下，kube-proxy會在service創建後，在宿主機上添加一個虛擬網卡： kube-ipvs0，並分配service IP。

kube-proxy通過linux的IPVS模塊，以rr輪詢方式調度service中的Pod。

---

3.創建service

3.1 ClusterIP

Kubernetes 提供了一個 DNS 插件 Service

[kubeadm@server1 mainfest]$ vim service.yml 
[kubeadm@server1 mainfest]$ cat service.yml 
kind: Service
apiVersion: v1
metadata:
  name: myservice
spec:
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  selector:
    app: myapp
[kubeadm@server1 mainfest]$ kubectl apply -f service.yml 
service/myservice created

[kubeadm@server1 mainfest]$ vim pod2.yml 
[kubeadm@server1 mainfest]$ cat pod2.yml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: deployment-example
spec:
  replicas: 2
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
      - name: myapp
        image: myapp:v2
[kubeadm@server1 mainfest]$ kubectl apply -f pod2.yml 
deployment.apps/deployment-example created
[kubeadm@server1 mainfest]$ kubectl describe svc myservice 
Name:              myservice
Namespace:         default
Labels:            <none>
Annotations:       Selector:  app=myapp
Type:              ClusterIP
IP:                10.107.6.65
Port:              <unset>  80/TCP
TargetPort:        80/TCP
Endpoints:         10.244.1.47:80,10.244.2.70:80
Session Affinity:  None
Events:            <none>
[kubeadm@server1 mainfest]$ kubectl run demo --image=busyboxplus -it --restart=Never
If you don't see a command prompt, try pressing enter.
[ root@demo:/ ]$ curl 10.107.6.65
Hello MyApp | Version: v2 | <a href="hostname.html">Pod Name</a>
[ root@demo:/ ]$ curl 10.107.6.65
Hello MyApp | Version: v2 | <a href="hostname.html">Pod Name</a>
[ root@demo:/ ]$ curl 10.107.6.65/hostname.html
deployment-example-67764dd8bd-pvkc6
[ root@demo:/ ]$ curl 10.107.6.65/hostname.html
deployment-example-67764dd8bd-p5qnr
[ root@demo:/ ]$ curl 10.107.6.65/hostname.html
deployment-example-67764dd8bd-pvkc6
[ root@demo:/ ]$ curl 10.107.6.65/hostname.html
deployment-example-67764dd8bd-p5qnr
[ root@demo:/ ]$ curl myservice/hostname.html
deployment-example-67764dd8bd-pvkc6
[ root@demo:/ ]$ curl myservice/hostname.html
deployment-example-67764dd8bd-p5qnr
[ root@demo:/ ]$ curl myservice/hostname.html
deployment-example-67764dd8bd-pvkc6
[ root@demo:/ ]$ nslookup myservice
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local
Name:      myservice
Address 1: 10.107.6.65 myservice.default.svc.cluster.local
[ root@demo:/ ]$ [kubeadm@server1 mainfest]$ 
[kubeadm@server1 mainfest]$  kubectl get services kube-dns --namespace=kube-system
NAME       TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
kube-dns   ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   7d3h

---

3.2 NodePort（可外部訪問）

[kubeadm@server1 mainfest]$ cat pod2.yml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: deployment-example
spec:
  replicas: 2
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
      - name: myapp
        image: myapp:v2
[kubeadm@server1 mainfest]$ kubectl apply -f pod2.yml 
deployment.apps/deployment-example unchanged
[kubeadm@server1 mainfest]$ kubectl get pod
NAME                                  READY   STATUS      RESTARTS   AGE
demo                                  0/1     Completed   0          7h16m
deployment-example-67764dd8bd-p5qnr   1/1     Running     0          7h17m
deployment-example-67764dd8bd-pvkc6   1/1     Running     0          7h17m

[kubeadm@server1 mainfest]$ vim service.yml 
[kubeadm@server1 mainfest]$ cat service.yml 
kind: Service
apiVersion: v1
metadata:
  name: myservice
spec:
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  selector:
    app: myapp
[kubeadm@server1 mainfest]$ kubectl apply -f service.yml 
service/myservice created
[kubeadm@server1 mainfest]$ kubectl get svc
NAME         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1      <none>        443/TCP   7d10h
myservice    ClusterIP   10.100.83.68   <none>        80/TCP    5s
[kubeadm@server1 mainfest]$ kubectl edit svc myservice 
service/myservice edited
[kubeadm@server1 mainfest]$ kubectl describe svc myservice 
Name:                     myservice
Namespace:                default
Labels:                   <none>
Annotations:              Selector:  app=myapp
Type:                     NodePort
IP:                       10.100.83.68
Port:                     <unset>  80/TCP
TargetPort:               80/TCP
NodePort:                 <unset>  31059/TCP
Endpoints:                10.244.1.47:80,10.244.2.70:80
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>
[kubeadm@server1 mainfest]$ kubectl get svc
NAME         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
kubernetes   ClusterIP   10.96.0.1      <none>        443/TCP        7d10h
myservice    NodePort    10.100.83.68   <none>        80:31059/TCP   5m30s

[kubeadm@server1 mainfest]$ kubectl get svc -n kube-system 
NAME       TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
kube-dns   ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   7d10h
[kubeadm@server1 mainfest]$ dig myservice.default.svc.cluster.local @10.96.0.10
; <<>> DiG 9.9.4-RedHat-9.9.4-72.el7 <<>> myservice.default.svc.cluster.local @10.96.0.10
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 541
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;myservice.default.svc.cluster.local. IN A
;; ANSWER SECTION:
myservice.default.svc.cluster.local. 30 IN A 10.100.83.68
;; Query time: 1 msec
;; SERVER: 10.96.0.10#53(10.96.0.10)
;; WHEN: Fri Jun 26 14:31:56 CST 2020
;; MSG SIZE  rcvd: 115
[kubeadm@server1 mainfest]$ kubectl run demo --image=busyboxplus -it --restart=Never
If you don't see a command prompt, try pressing enter.
[ root@demo:/ ]$ cat /etc/resolv.conf 
nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5
[ root@demo:/ ]$ ping myservice
PING myservice (10.100.83.68): 56 data bytes
64 bytes from 10.100.83.68: seq=0 ttl=64 time=0.034 ms
^C
--- myservice ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.034/0.034/0.034 ms

---

3.3 無頭服務

Headless Service “無頭服務” 。 Headless Service不需要分配一個VIP，而是直接以DNS記錄的方式解析出被代理 Pod的IP地址。
域名格式：$(servicename).$(namespace).svc.cluster.local
yum install -y bind-utils.x86_64安裝解析工具

[kubeadm@server1 mainfest]$ vim service.yml 
[kubeadm@server1 mainfest]$ cat service.yml 
kind: Service
apiVersion: v1
metadata:
  name: myservice
spec:
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  selector:
    app: myapp
  clusterIP: None
[kubeadm@server1 mainfest]$ kubectl apply -f service.yml 
service/myservice created
[kubeadm@server1 mainfest]$ kubectl get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   7d11h
myservice    ClusterIP   None         <none>        80/TCP    10s
[kubeadm@server1 mainfest]$ kubectl describe svc myservice 
Name:              myservice
Namespace:         default
Labels:            <none>
Annotations:       Selector:  app=myapp
Type:              ClusterIP
IP:                None
Port:              <unset>  80/TCP
TargetPort:        80/TCP
Endpoints:         10.244.1.47:80,10.244.1.48:80,10.244.2.70:80
Session Affinity:  None
Events:            <none>

[kubeadm@server1 mainfest]$ kubectl -n kube-system describe svc kube-dns 
Name:              kube-dns
Namespace:         kube-system
Labels:            k8s-app=kube-dns
                   kubernetes.io/cluster-service=true
                   kubernetes.io/name=KubeDNS
Annotations:       prometheus.io/port: 9153
                   prometheus.io/scrape: true
Selector:          k8s-app=kube-dns
Type:              ClusterIP
IP:                10.96.0.10
Port:              dns  53/UDP
TargetPort:        53/UDP
Endpoints:         10.244.0.10:53,10.244.0.11:53
Port:              dns-tcp  53/TCP
TargetPort:        53/TCP
Endpoints:         10.244.0.10:53,10.244.0.11:53
Port:              metrics  9153/TCP
TargetPort:        9153/TCP
Endpoints:         10.244.0.10:9153,10.244.0.11:9153
Session Affinity:  None
Events:            <none>

[kubeadm@server1 mainfest]$ dig myservice.default.svc.cluster.local @10.96.0.10
; <<>> DiG 9.9.4-RedHat-9.9.4-72.el7 <<>> myservice.default.svc.cluster.local @10.96.0.10
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 19011
;; flags: qr aa rd; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;myservice.default.svc.cluster.local. IN A
;; ANSWER SECTION:
myservice.default.svc.cluster.local. 30 IN A 10.244.2.70
myservice.default.svc.cluster.local. 30 IN A 10.244.1.48
myservice.default.svc.cluster.local. 30 IN A 10.244.1.47
;; Query time: 0 msec
;; SERVER: 10.96.0.10#53(10.96.0.10)
;; WHEN: Fri Jun 26 15:01:23 CST 2020
;; MSG SIZE  rcvd: 217

[kubeadm@server1 mainfest]$ dig myservice.default.svc.cluster.local @10.244.0.11
; <<>> DiG 9.9.4-RedHat-9.9.4-72.el7 <<>> myservice.default.svc.cluster.local @10.244.0.11
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 16525
;; flags: qr aa rd; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;myservice.default.svc.cluster.local. IN A
;; ANSWER SECTION:
myservice.default.svc.cluster.local. 30 IN A 10.244.1.48
myservice.default.svc.cluster.local. 30 IN A 10.244.2.70
myservice.default.svc.cluster.local. 30 IN A 10.244.1.47
;; Query time: 0 msec
;; SERVER: 10.244.0.11#53(10.244.0.11)
;; WHEN: Fri Jun 26 15:02:57 CST 2020
;; MSG SIZE  rcvd: 217

Pod滾動更新後，依然可以解析：

[kubeadm@server1 mainfest]$ kubectl delete pod --all
pod "deployment-example-67764dd8bd-p5qnr" deleted
pod "deployment-example-67764dd8bd-pvkc6" deleted
pod "deployment-example-67764dd8bd-smr7c" deleted
[kubeadm@server1 mainfest]$ kubectl get pod -o wide
NAME                                  READY   STATUS    RESTARTS   AGE   IP            NODE      NOMINATED NODE   READINESS GATES
deployment-example-67764dd8bd-6pwdw   1/1     Running   0          7s    10.244.2.74   server3   <none>           <none>
deployment-example-67764dd8bd-jl7nl   1/1     Running   0          7s    10.244.1.49   server2   <none>           <none>
deployment-example-67764dd8bd-zvd28   1/1     Running   0          7s    10.244.2.73   server3   <none>           <none>
[kubeadm@server1 mainfest]$ dig myservice.default.svc.cluster.local @10.244.0.11
; <<>> DiG 9.9.4-RedHat-9.9.4-72.el7 <<>> myservice.default.svc.cluster.local @10.244.0.11
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 48989
;; flags: qr aa rd; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;myservice.default.svc.cluster.local. IN A
;; ANSWER SECTION:
myservice.default.svc.cluster.local. 30 IN A 10.244.1.49
myservice.default.svc.cluster.local. 30 IN A 10.244.2.73
myservice.default.svc.cluster.local. 30 IN A 10.244.2.74
;; Query time: 0 msec
;; SERVER: 10.244.0.11#53(10.244.0.11)
;; WHEN: Fri Jun 26 15:14:37 CST 2020
;; MSG SIZE  rcvd: 217

---

3.4 LoadBalancer（可外部訪問）

從外部訪問 Service 的第二種方式，適用於公有云上的 Kubernetes 服務。這時候，你可以指定一個 LoadBalancer 類型的 Service。
在service提交後，Kubernetes就會調用 CloudProvider 在公有云上創建一個負載均衡服務，並且把被代理的 Pod 的 IP地址配置給負載均衡服務做後端。

[kubeadm@server1 mainfest]$ vim service.yml 
[kubeadm@server1 mainfest]$ cat service.yml 
kind: Service
apiVersion: v1
metadata:
  name: myservice
spec:
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  selector:
    app: myapp
  type: LoadBalancer
[kubeadm@server1 mainfest]$ kubectl apply -f service.yml 
service/myservice created
[kubeadm@server1 mainfest]$ kubectl get svc 
NAME         TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
kubernetes   ClusterIP      10.96.0.1        <none>        443/TCP        7d11h
myservice    LoadBalancer   10.104.173.144   <pending>     80:32646/TCP   5s

---

3.5 ExternalName（可外部訪問）

從外部訪問的第三種方式叫做ExternalName。適用於集羣內部容器訪問外部資源

[kubeadm@server1 mainfest]$ vim service.yml 
[kubeadm@server1 mainfest]$ cat service.yml 
kind: Service
apiVersion: v1
metadata:
  name: myservice
spec:
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  selector:
    app: myapp
  type:  ExternalName  
  externalName: www.baidu.com
[kubeadm@server1 mainfest]$ kubectl apply -f service.yml 
service/myservice created
[kubeadm@server1 mainfest]$ kubectl get svc 
NAME         TYPE           CLUSTER-IP   EXTERNAL-IP     PORT(S)   AGE
kubernetes   ClusterIP      10.96.0.1    <none>          443/TCP   7d11h
myservice    ExternalName   <none>       www.baidu.com   80/TCP    4s
[kubeadm@server1 mainfest]$ dig myservice.default.svc.cluster.local @10.244.0.11
; <<>> DiG 9.9.4-RedHat-9.9.4-72.el7 <<>> myservice.default.svc.cluster.local @10.244.0.11
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 44555
;; flags: qr aa rd; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;myservice.default.svc.cluster.local. IN A
;; ANSWER SECTION:
myservice.default.svc.cluster.local. 30 IN CNAME www.baidu.com.
www.baidu.com.  30 IN CNAME www.a.shifen.com.
www.a.shifen.com. 30 IN A 61.135.169.121
www.a.shifen.com. 30 IN A 61.135.169.125
;; Query time: 75 msec
;; SERVER: 10.244.0.11#53(10.244.0.11)
;; WHEN: Fri Jun 26 15:28:43 CST 2020
;; MSG SIZE  rcvd: 233

---

3.6 service直接分配一個公有IP（外網地址）

[kubeadm@server1 mainfest]$ vim service.yml 
[kubeadm@server1 mainfest]$ cat service.yml 
kind: Service
apiVersion: v1
metadata:
  name: myservice
spec:
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  selector:
    app: myapp
  externalIPs: 
  - 172.25.1.100
[kubeadm@server1 mainfest]$ kubectl apply -f service.yml 
service/myservice created
[kubeadm@server1 mainfest]$ kubectl get svc 
NAME         TYPE        CLUSTER-IP       EXTERNAL-IP    PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1        <none>         443/TCP   7d11h
myservice    ClusterIP   10.105.122.218   172.25.1.100   80/TCP    4s
[kubeadm@server1 mainfest]$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:bb:3e:1d brd ff:ff:ff:ff:ff:ff
    inet 192.168.43.11/24 brd 192.168.43.255 scope global ens33
       valid_lft forever preferred_lft forever
    inet 172.25.1.1/24 brd 172.25.1.255 scope global ens33
       valid_lft forever preferred_lft forever
    inet 192.168.3.201/24 brd 192.168.3.255 scope global ens33
       valid_lft forever preferred_lft forever
    inet6 2408:84fb:1:1209:20c:29ff:febb:3e1d/64 scope global mngtmpaddr dynamic 
       valid_lft 3317sec preferred_lft 3317sec
    inet6 fe80::20c:29ff:febb:3e1d/64 scope link 
       valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:da:68:ae:81 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
4: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default 
    link/ether d2:fd:d6:3b:73:d3 brd ff:ff:ff:ff:ff:ff
    inet 10.244.0.0/32 scope global flannel.1
       valid_lft forever preferred_lft forever
    inet6 fe80::d0fd:d6ff:fe3b:73d3/64 scope link 
       valid_lft forever preferred_lft forever
5: cni0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
    link/ether 4a:a1:c4:bb:75:78 brd ff:ff:ff:ff:ff:ff
    inet 10.244.0.1/24 scope global cni0
       valid_lft forever preferred_lft forever
    inet6 fe80::48a1:c4ff:febb:7578/64 scope link 
       valid_lft forever preferred_lft forever
6: veth3fc18a62@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default 
    link/ether a2:7d:16:36:60:6a brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::a07d:16ff:fe36:606a/64 scope link 
       valid_lft forever preferred_lft forever
7: veth99a251bf@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default 
    link/ether 86:20:80:1b:26:22 brd ff:ff:ff:ff:ff:ff link-netnsid 1
    inet6 fe80::8420:80ff:fe1b:2622/64 scope link 
       valid_lft forever preferred_lft forever
8: dummy0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether d2:d9:2f:45:a0:05 brd ff:ff:ff:ff:ff:ff
9: kube-ipvs0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN group default 
    link/ether 9a:26:f3:9b:57:38 brd ff:ff:ff:ff:ff:ff
    inet 10.96.0.1/32 brd 10.96.0.1 scope global kube-ipvs0
       valid_lft forever preferred_lft forever
    inet 10.96.0.10/32 brd 10.96.0.10 scope global kube-ipvs0
       valid_lft forever preferred_lft forever
    inet 10.105.122.218/32 brd 10.105.122.218 scope global kube-ipvs0
       valid_lft forever preferred_lft forever
    inet 172.25.1.100/32 brd 172.25.1.100 scope global kube-ipvs0
       valid_lft forever preferred_lft forever
[kubeadm@server1 mainfest]$ logout
[root@server1 ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  172.25.1.100:80 rr
  -> 10.244.1.49:80               Masq    1      0          0         
  -> 10.244.2.73:80               Masq    1      0          0         
  -> 10.244.2.74:80               Masq    1      0          0         
TCP  10.96.0.1:443 rr
  -> 192.168.43.11:6443           Masq    1      0          0         
TCP  10.96.0.10:53 rr
  -> 10.244.0.10:53               Masq    1      0          0         
  -> 10.244.0.11:53               Masq    1      0          0         
TCP  10.96.0.10:9153 rr
  -> 10.244.0.10:9153             Masq    1      0          0         
  -> 10.244.0.11:9153             Masq    1      0          0         
TCP  10.105.122.218:80 rr
  -> 10.244.1.49:80               Masq    1      0          0         
  -> 10.244.2.73:80               Masq    1      0          0         
  -> 10.244.2.74:80               Masq    1      0          0         
UDP  10.96.0.10:53 rr
  -> 10.244.0.10:53               Masq    1      0          0         
  -> 10.244.0.11:53               Masq    1      0          0   

---