過程簡介：
1. master,nodes: 安裝kubelet、kubeadm，docker-ce,
systemctl disable firewalld ,systemctl disable iptables
2. Master： kubeadm init
3. nodes： kubeadm join （節點加入集羣）
https://github.com/kubernetes/kubeadm/blob/master/docs/design/design_v1.10.md
下面是中文文檔：
kubeadm是Kubernetes官方提供的用於快速安裝Kubernetes集羣的工具，伴隨Kubernetes每個版本的發佈都會同步更新，kubeadm會對集羣配置方面的一些實踐做調整，通過實驗kubeadm可以學習到Kubernetes官方在集羣配置上一些新的最佳實踐。
最近發佈的Kubernetes 1.13中，kubeadm的主要特性已經GA了，但還不包含高可用，不過說明kubeadm可在生產環境中使用的距離越來越近了。
| Area | Maturity Level |
|---|---|
| Command line UX | GA |
| Implementation | GA |
| Config file API | beta |
| CoreDNS | GA |
| kubeadm alpha subcommands | alpha |
| High availability | alpha |
| DynamicKubeletConfig | alpha |
| Self-hosting | alpha |
當然我們線上穩定運行的Kubernetes集羣是使用ansible以二進制形式的部署的高可用集羣，這裏體驗Kubernetes 1.13中的kubeadm是爲了跟隨官方對集羣初始化和配置方面的最佳實踐，進一步完善我們的ansible部署腳本。
1.準備
1.1系統配置
在安裝之前，需要先做如下準備。兩臺CentOS 7.4主機如下：
cat /etc/hosts
192.168.61.11 node1
192.168.61.12 node2
如果各個主機啓用了防火牆，需要開放Kubernetes各個組件所需要的端口，可以查看Installing kubeadm中的”Check required ports”一節。 這裏簡單起見在各節點禁用防火牆：
systemctl stop firewalld
systemctl disable firewalld
禁用SELINUX：
setenforce 0
vi /etc/selinux/config
SELINUX=disabled
創建/etc/sysctl.d/k8s.conf文件，添加如下內容：
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
執行命令使修改生效。
modprobe br_netfilter
sysctl -p /etc/sysctl.d/k8s.conf
1.2kube-proxy開啓ipvs的前置條件
由於ipvs已經加入到了內核的主幹，所以爲kube-proxy開啓ipvs的前提需要加載以下的內核模塊：
ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_sh
nf_conntrack_ipv4
在所有的Kubernetes節點node1和node2上執行以下腳本:
cat > /etc/sysconfig/modules/ipvs.modules <<EOF
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4
EOF
chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack_ipv4
上面腳本創建了的/etc/sysconfig/modules/ipvs.modules文件，保證在節點重啓後能自動加載所需模塊。 使用lsmod | grep -e ip_vs -e nf_conntrack_ipv4命令查看是否已經正確加載所需的內核模塊。
接下來還需要確保各個節點上已經安裝了ipset軟件包yum install ipset。 爲了便於查看ipvs的代理規則，最好安裝一下管理工具ipvsadm yum install ipvsadm。
如果以上前提條件如果不滿足，則即使kube-proxy的配置開啓了ipvs模式，也會退回到iptables模式。
1.3安裝Docker
Kubernetes從1.6開始使用CRI(Container Runtime Interface)容器運行時接口。默認的容器運行時仍然是Docker，使用的是kubelet中內置dockershim CRI實現。
安裝docker的yum源:
yum install -y yum-utils device-mapper-persistent-data lvm2
yum-config-manager \
--add-repo \
https://download.docker.com/linux/centos/docker-ce.repo
查看最新的Docker版本：
yum list docker-ce.x86_64 --showduplicates |sort -r
docker-ce.x86_64 3:18.09.0-3.el7 docker-ce-stable
docker-ce.x86_64 18.06.1.ce-3.el7 docker-ce-stable
docker-ce.x86_64 18.06.0.ce-3.el7 docker-ce-stable
docker-ce.x86_64 18.03.1.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 18.03.0.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 17.12.1.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 17.12.0.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 17.09.1.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 17.09.0.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 17.06.2.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 17.06.1.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 17.06.0.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 17.03.3.ce-1.el7 docker-ce-stable
docker-ce.x86_64 17.03.2.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 17.03.1.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 17.03.0.ce-1.el7.centos docker-ce-stable
Kubernetes 1.12已經針對Docker的1.11.1, 1.12.1, 1.13.1, 17.03, 17.06, 17.09, 18.06等版本做了驗證，需要注意Kubernetes 1.12最低支持的Docker版本是1.11.1。Kubernetes 1.13對Docker的版本依賴方面沒有變化。 我們這裏在各節點安裝docker的18.06.1版本。
yum makecache fast
yum install -y --setopt=obsoletes=0 \
docker-ce-18.06.1.ce-3.el7
systemctl start docker
systemctl enable docker
確認一下iptables filter表中FOWARD鏈的默認策略(pllicy)爲ACCEPT。
iptables -nvL
Chain INPUT (policy ACCEPT 263 packets, 19209 bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 DOCKER-USER all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 DOCKER-ISOLATION-STAGE-1 all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all -- * docker0 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
0 0 DOCKER all -- * docker0 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all -- docker0 !docker0 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all -- docker0 docker0 0.0.0.0/0 0.0.0.0/0
Docker從1.13版本開始調整了默認的防火牆規則，禁用了iptables filter表中FOWARD鏈，這樣會引起Kubernetes集羣中跨Node的Pod無法通信。但這裏通過安裝docker 1806，發現默認策略又改回了ACCEPT，這個不知道是從哪個版本改回的，因爲我們線上版本使用的1706還是需要手動調整這個策略的。
2.使用kubeadm部署Kubernetes
2.1 安裝kubeadm和kubelet
下面在各節點安裝kubeadm和kubelet：
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg
https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF
測試地址https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64是否可用，如果不可用需要科學上網。
curl https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
yum makecache fast
yum install -y kubelet kubeadm kubectl
...
Installed:
kubeadm.x86_64 0:1.13.0-0 kubectl.x86_64 0:1.13.0-0 kubelet.x86_64 0:1.13.0-0
Dependency Installed:
cri-tools.x86_64 0:1.12.0-0 kubernetes-cni.x86_64 0:0.6.0-0 socat.x86_64 0:1.7.3.2-2.el7
- 從安裝結果可以看出還安裝了cri-tools, kubernetes-cni, socat三個依賴：
- 官方從Kubernetes 1.9開始就將cni依賴升級到了0.6.0版本，在當前1.12中仍然是這個版本
- socat是kubelet的依賴
- cri-tools是CRI(Container Runtime Interface)容器運行時接口的命令行工具
運行kubelet –help可以看到原來kubelet的絕大多數命令行flag參數都被DEPRECATED了，如：
......
--address 0.0.0.0 The IP address for the Kubelet to serve on (set to 0.0.0.0 for all IPv4 interfaces and `::` for all IPv6 interfaces) (default 0.0.0.0) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
......
而官方推薦我們使用–config指定配置文件，並在配置文件中指定原來這些flag所配置的內容。具體內容可以查看這裏Set Kubelet parameters via a config file。這也是Kubernetes爲了支持動態Kubelet配置（Dynamic Kubelet Configuration）才這麼做的，參考Reconfigure a Node’s Kubelet in a Live Cluster。
kubelet的配置文件必須是json或yaml格式，具體可查看這裏。
Kubernetes 1.8開始要求關閉系統的Swap，如果不關閉，默認配置下kubelet將無法啓動。
關閉系統的Swap方法如下:
swapoff -a修改 /etc/fstab 文件，註釋掉 SWAP 的自動掛載，使用free -m確認swap已經關閉。 swappiness參數調整，修改/etc/sysctl.d/k8s.conf添加下面一行：
vm.swappiness=0執行sysctl -p /etc/sysctl.d/k8s.conf使修改生效。
因爲這裏本次用於測試兩臺主機上還運行其他服務，關閉swap可能會對其他服務產生影響，所以這裏修改kubelet的配置去掉這個限制。 之前的Kubernetes版本我們都是通過kubelet的啓動參數–fail-swap-on=false去掉這個限制的。前面已經分析了Kubernetes不再推薦使用啓動參數，而推薦使用配置文件。 所以這裏我們改成配置文件配置的形式。
查看/etc/systemd/system/kubelet.service.d/10-kubeadm.conf，看到了下面的內容：
# Note: This dropin only works with kubeadm and kubelet v1.11+
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
# This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
# the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/sysconfig/kubelet
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS
上面顯示kubeadm部署的kubelet的配置文件–config=/var/lib/kubelet/config.yaml，實際去查看/var/lib/kubelet和這個config.yaml的配置文件都沒有被創建。 可以猜想肯定是運行kubeadm初始化集羣時會自動生成這個配置文件，而如果我們不關閉Swap的話，第一次初始化集羣肯定會失敗的。
所以還是老老實實的回到使用kubelet的啓動參數–fail-swap-on=false去掉必須關閉Swap的限制。 修改/etc/sysconfig/kubelet，加入：
KUBELET_EXTRA_ARGS=--fail-swap-on=false
2.2 使用kubeadm init初始化集羣
在各節點開機啓動kubelet服務：
systemctl enable kubelet.service
接下來使用kubeadm初始化集羣，選擇node1作爲Master Node，在node1上執行下面的命令：
kubeadm init \
--kubernetes-version=v1.13.0 \
--pod-network-cidr=10.244.0.0/16 \
--apiserver-advertise-address=192.168.61.11
因爲我們選擇flannel作爲Pod網絡插件，所以上面的命令指定–pod-network-cidr=10.244.0.0/16。
執行時報了下面的錯誤：
[init] using Kubernetes version: v1.13.0
[preflight] running pre-flight checks
[preflight] Some fatal errors occurred:
[ERROR Swap]: running with swap on is not supported. Please disable swap
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
有一個錯誤信息是running with swap on is not supported. Please disable swap。因爲我們決定配置failSwapOn: false，所以重新添加–ignore-preflight-errors=Swap參數忽略這個錯誤，重新運行。
kubeadm init \
--kubernetes-version=v1.13.0 \
--pod-network-cidr=10.244.0.0/16 \
--apiserver-advertise-address=192.168.61.11 \
--ignore-preflight-errors=Swap
[init] Using Kubernetes version: v1.13.0
[preflight] Running pre-flight checks
[WARNING Swap]: running with swap on is not supported. Please disable swap
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [node1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.61.11]
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [node1 localhost] and IPs [192.168.61.11 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [node1 localhost] and IPs [192.168.61.11 127.0.0.1 ::1]
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 19.506551 seconds
[uploadconfig] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.13" in namespace kube-system with the configuration for the kubelets in the cluster
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "node1" as an annotation
[mark-control-plane] Marking the node node1 as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node node1 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: 702gz5.49zhotgsiyqimwqw
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstraptoken] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstraptoken] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstraptoken] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstraptoken] creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes master has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
You can now join any number of machines by running the following on each node
as root:
kubeadm join 192.168.61.11:6443 --token 702gz5.49zhotgsiyqimwqw --discovery-token-ca-cert-hash sha256:2bc50229343849e8021d2aa19d9d314539b40ec7a311b5bb6ca1d3cd10957c2f
上面記錄了完成的初始化輸出的內容，根據輸出的內容基本上可以看出手動初始化安裝一個Kubernetes集羣所需要的關鍵步驟。
其中有以下關鍵內容：
- [kubelet-start] 生成kubelet的配置文件”/var/lib/kubelet/config.yaml”
- [certificates]生成相關的各種證書
- [kubeconfig]生成相關的kubeconfig文件
- [bootstraptoken]生成token記錄下來，後邊使用kubeadm join往集羣中添加節點時會用到
- 下面的命令是配置常規用戶如何使用kubectl訪問集羣：
mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config- 最後給出了將節點加入集羣的命令kubeadm join 192.168.61.11:6443 –token 702gz5.49zhotgsiyqimwqw –discovery-token-ca-cert-hash sha256:2bc50229343849e8021d2aa19d9d314539b40ec7a311b5bb6ca1d3cd10957c2f
查看一下集羣狀態：
kubectl get cs
NAME STATUS MESSAGE ERROR
controller-manager Healthy ok
scheduler Healthy ok
etcd-0 Healthy {"health": "true"}
確認個組件都處於healthy狀態。
集羣初始化如果遇到問題，可以使用下面的命令進行清理：
kubeadm reset
ifconfig cni0 down
ip link delete cni0
ifconfig flannel.1 down
ip link delete flannel.1
rm -rf /var/lib/cni/
2.3 安裝Pod Network
接下來安裝flannel network add-on：
mkdir -p ~/k8s/
cd ~/k8s
wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
kubectl apply -f kube-flannel.yml
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.extensions/kube-flannel-ds-amd64 created
daemonset.extensions/kube-flannel-ds-arm64 created
daemonset.extensions/kube-flannel-ds-arm created
daemonset.extensions/kube-flannel-ds-ppc64le created
daemonset.extensions/kube-flannel-ds-s390x created
這裏注意kube-flannel.yml這個文件裏的flannel的鏡像是0.10.0，quay.io/coreos/flannel:v0.10.0-amd64
如果Node有多個網卡的話，參考flannel issues 39701，目前需要在kube-flannel.yml中使用–iface參數指定集羣主機內網網卡的名稱，否則可能會出現dns無法解析。需要將kube-flannel.yml下載到本地，flanneld啓動參數加上–iface=<iface-name>
......
containers:
- name: kube-flannel
image: quay.io/coreos/flannel:v0.10.0-amd64
command:
- /opt/bin/flanneld
args:
- --ip-masq
- --kube-subnet-mgr
- --iface=eth1
......
使用kubectl get pod –all-namespaces -o wide確保所有的Pod都處於Running狀態。
kubectl get pod --all-namespaces -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE
kube-system coredns-576cbf47c7-njt7l 1/1 Running 0 12m 10.244.0.3 node1 <none>
kube-system coredns-576cbf47c7-vg2gd 1/1 Running 0 12m 10.244.0.2 node1 <none>
kube-system etcd-node1 1/1 Running 0 12m 192.168.61.11 node1 <none>
kube-system kube-apiserver-node1 1/1 Running 0 12m 192.168.61.11 node1 <none>
kube-system kube-controller-manager-node1 1/1 Running 0 12m 192.168.61.11 node1 <none>
kube-system kube-flannel-ds-amd64-bxtqh 1/1 Running 0 2m 192.168.61.11 node1 <none>
kube-system kube-proxy-fb542 1/1 Running 0 12m 192.168.61.11 node1 <none>
kube-system kube-scheduler-node1 1/1 Running 0 12m 192.168.61.11 node1 <none>
2.4 master node參與工作負載
使用kubeadm初始化的集羣，出於安全考慮Pod不會被調度到Master Node上，也就是說Master Node不參與工作負載。這是因爲當前的master節點node1被打上了node-role.kubernetes.io/master:NoSchedule的污點：
kubectl describe node node1 | grep Taint
Taints: node-role.kubernetes.io/master:NoSchedule
因爲這裏搭建的是測試環境，去掉這個污點使node1參與工作負載：
kubectl taint nodes node1 node-role.kubernetes.io/master-
node "node1" untainted
2.5 測試DNS
kubectl run curl --image=radial/busyboxplus:curl -it
kubectl run --generator=deployment/apps.v1beta1 is DEPRECATED and will be removed in a future version. Use kubectl create instead.
If you don't see a command prompt, try pressing enter.
[ root@curl-5cc7b478b6-r997p:/ ]$
進入後執行nslookup kubernetes.default確認解析正常:
nslookup kubernetes.default
Server: 10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local
Name: kubernetes.default
Address 1: 10.96.0.1 kubernetes.default.svc.cluster.local
2.6 向Kubernetes集羣中添加Node節點
下面我們將node2這個主機添加到Kubernetes集羣中，因爲我們同樣在node2上的kubelet的啓動參數中去掉了必須關閉swap的限制，所以同樣需要–ignore-preflight-errors=Swap這個參數。 在node2上執行:
kubeadm join 192.168.61.11:6443 --token 702gz5.49zhotgsiyqimwqw --discovery-token-ca-cert-hash sha256:2bc50229343849e8021d2aa19d9d314539b40ec7a311b5bb6ca1d3cd10957c2f \
--ignore-preflight-errors=Swap
[preflight] Running pre-flight checks
[WARNING Swap]: running with swap on is not supported. Please disable swap
[discovery] Trying to connect to API Server "192.168.61.11:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://192.168.61.11:6443"
[discovery] Requesting info from "https://192.168.61.11:6443" again to validate TLS against the pinned public key
[discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "192.168.61.11:6443"
[discovery] Successfully established connection with API Server "192.168.61.11:6443"
[join] Reading configuration from the cluster...
[join] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet] Downloading configuration for the kubelet from the "kubelet-config-1.13" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Activating the kubelet service
[tlsbootstrap] Waiting for the kubelet to perform the TLS Bootstrap...
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "node2" as an annotation
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the master to see this node join the cluster.
node2加入集羣很是順利，下面在master節點上執行命令查看集羣中的節點：
kubectl get nodes
NAME STATUS ROLES AGE VERSION
node1 Ready master 16m v1.13.0
node2 Ready <none> 4m5s v1.13.0
如何從集羣中移除Node
如果需要從集羣中移除node2這個Node執行下面的命令：
在master節點上執行：
kubectl drain node2 --delete-local-data --force --ignore-daemonsets
kubectl delete node node2
在node2上執行：
kubeadm reset
ifconfig cni0 down
ip link delete cni0
ifconfig flannel.1 down
ip link delete flannel.1
rm -rf /var/lib/cni/
在node1上執行：
kubectl delete node node2
2.7 kube-proxy開啓ipvs
修改ConfigMap的kube-system/kube-proxy中的config.conf，mode: “ipvs”：
kubectl edit cm kube-proxy -n kube-system
之後重啓各個節點上的kube-proxy pod：
kubectl get pod -n kube-system | grep kube-proxy | awk '{system("kubectl delete pod "$1" -n kube-system")}'
kubectl get pod -n kube-system | grep kube-proxy
kube-proxy-pf55q 1/1 Running 0 9s
kube-proxy-qjnnc 1/1 Running 0 14s
kubectl logs kube-proxy-pf55q -n kube-system
I1208 06:12:23.516444 1 server_others.go:189] Using ipvs Proxier.
W1208 06:12:23.516738 1 proxier.go:365] IPVS scheduler not specified, use rr by default
I1208 06:12:23.516840 1 server_others.go:216] Tearing down inactive rules.
I1208 06:12:23.575222 1 server.go:464] Version: v1.13.0
I1208 06:12:23.585142 1 conntrack.go:52] Setting nf_conntrack_max to 131072
I1208 06:12:23.586203 1 config.go:202] Starting service config controller
I1208 06:12:23.586243 1 controller_utils.go:1027] Waiting for caches to sync for service config controller
I1208 06:12:23.586269 1 config.go:102] Starting endpoints config controller
I1208 06:12:23.586275 1 controller_utils.go:1027] Waiting for caches to sync for endpoints config controller
I1208 06:12:23.686959 1 controller_utils.go:1034] Caches are synced for endpoints config controller
I1208 06:12:23.687056 1 controller_utils.go:1034] Caches are synced for service config controller
日誌中打印出了Using ipvs Proxier，說明ipvs模式已經開啓。
3.Kubernetes常用組件部署
越來越多的公司和團隊開始使用Helm這個Kubernetes的包管理器，我們也將使用Helm安裝Kubernetes的常用組件。
3.1 Helm的安裝
Helm由客戶端命helm令行工具和服務端tiller組成，Helm的安裝十分簡單。 下載helm命令行工具到master節點node1的/usr/local/bin下，這裏下載的2.12.0版本：
wget https://storage.googleapis.com/kubernetes-helm/helm-v2.12.0-linux-amd64.tar.gz
tar -zxvf helm-v2.12.0-linux-amd64.tar.gz
cd linux-amd64/
cp helm /usr/local/bin/
爲了安裝服務端tiller，還需要在這臺機器上配置好kubectl工具和kubeconfig文件，確保kubectl工具可以在這臺機器上訪問apiserver且正常使用。 這裏的node1節點以及配置好了kubectl。
因爲Kubernetes APIServer開啓了RBAC訪問控制，所以需要創建tiller使用的service account: tiller並分配合適的角色給它。 詳細內容可以查看helm文檔中的Role-based Access Control。 這裏簡單起見直接分配cluster-admin這個集羣內置的ClusterRole給它。創建rbac-config.yaml文件：
apiVersion: v1
kind: ServiceAccount
metadata:
name: tiller
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: tiller
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: tiller
namespace: kube-system
kubectl create -f rbac-config.yaml
serviceaccount/tiller created
clusterrolebinding.rbac.authorization.k8s.io/tiller created
接下來使用helm部署tiller:
helm init --service-account tiller --skip-refresh
Creating /root/.helm
Creating /root/.helm/repository
Creating /root/.helm/repository/cache
Creating /root/.helm/repository/local
Creating /root/.helm/plugins
Creating /root/.helm/starters
Creating /root/.helm/cache/archive
Creating /root/.helm/repository/repositories.yaml
Adding stable repo with URL: https://kubernetes-charts.storage.googleapis.com
Adding local repo with URL: http://127.0.0.1:8879/charts
$HELM_HOME has been configured at /root/.helm.
Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.
Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.
To prevent this, run `helm init` with the --tiller-tls-verify flag.
For more information on securing your installation see: https://docs.helm.sh/using_helm/#securing-your-helm-installation
Happy Helming!
tiller默認被部署在k8s集羣中的kube-system這個namespace下：
kubectl get pod -n kube-system -l app=helm
NAME READY STATUS RESTARTS AGE
tiller-deploy-c4fd4cd68-dwkhv 1/1 Running 0 83s
helm version
Client: &version.Version{SemVer:"v2.12.0", GitCommit:"d325d2a9c179b33af1a024cdb5a4472b6288016a", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.12.0", GitCommit:"d325d2a9c179b33af1a024cdb5a4472b6288016a", GitTreeState:"clean"}
注意由於某些原因需要網絡可以訪問gcr.io和kubernetes-charts.storage.googleapis.com，如果無法訪問可以通過helm init –service-account tiller –tiller-image <your-docker-registry>/tiller:v2.11.0 –skip-refresh使用私有鏡像倉庫中的tiller鏡像
3.2 使用Helm部署Nginx Ingress
爲了便於將集羣中的服務暴露到集羣外部，從集羣外部訪問，接下來使用Helm將Nginx Ingress部署到Kubernetes上。 Nginx Ingress Controller被部署在Kubernetes的邊緣節點上，關於Kubernetes邊緣節點的高可用相關的內容可以查看我前面整理的Bare metal環境下Kubernetes Ingress邊緣節點的高可用(基於IPVS)。
我們將node1(192.168.61.11)和node2(192.168.61.12)同時做爲邊緣節點，打上Label：
kubectl label node node1 node-role.kubernetes.io/edge=
node/node1 labeled
kubectl label node node2 node-role.kubernetes.io/edge=
node/node2 labeled
kubectl get node
NAME STATUS ROLES AGE VERSION
node1 Ready edge,master 24m v1.13.0
node2 Ready edge 11m v1.13.0
stable/nginx-ingress chart的值文件ingress-nginx.yaml：
controller:
replicaCount: 2
service:
externalIPs:
- 192.168.61.10
nodeSelector:
node-role.kubernetes.io/edge: ''
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values:
- nginx-ingress
- key: component
operator: In
values:
- controller
topologyKey: kubernetes.io/hostname
tolerations:
- key: node-role.kubernetes.io/master
operator: Exists
effect: NoSchedule
defaultBackend:
nodeSelector:
node-role.kubernetes.io/edge: ''
tolerations:
- key: node-role.kubernetes.io/master
operator: Exists
effect: NoSchedule
nginx ingress controller的副本數replicaCount爲2，將被調度到node1和node2這兩個邊緣節點上。externalIPs指定的192.168.61.10爲VIP，將綁定到kube-proxy kube-ipvs0網卡上。
helm repo update
helm install stable/nginx-ingress \
-n nginx-ingress \
--namespace ingress-nginx \
-f ingress-nginx.yaml
kubectl get pod -n ingress-nginx -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-ingress-controller-85f8597fc6-g2kcx 1/1 Running 0 5m2s 10.244.1.3 node2 <none> <none>
nginx-ingress-controller-85f8597fc6-g7pp5 1/1 Running 0 5m2s 10.244.0.5 node1 <none> <none>
nginx-ingress-default-backend-6dc6c46dcc-7plm8 1/1 Running 0 5m2s 10.244.1.4 node2 <none> <none>
如果訪問http://192.168.61.10返回default backend，則部署完成。
實際測試的結果是無法訪問，於是懷疑kube-proxy出了問題，查看kube-proxy的日誌，不停的刷下面的log：
I1208 07:59:28.902970 1 graceful_termination.go:160] Trying to delete rs: 10.104.110.193:80/TCP/10.244.1.5:80
I1208 07:59:28.903037 1 graceful_termination.go:170] Deleting rs: 10.104.110.193:80/TCP/10.244.1.5:80
I1208 07:59:28.903072 1 graceful_termination.go:160] Trying to delete rs: 10.104.110.193:80/TCP/10.244.0.6:80
I1208 07:59:28.903105 1 graceful_termination.go:170] Deleting rs: 10.104.110.193:80/TCP/10.244.0.6:80
I1208 07:59:28.903713 1 graceful_termination.go:160] Trying to delete rs: 192.168.61.10:80/TCP/10.244.1.5:80
I1208 07:59:28.903764 1 graceful_termination.go:170] Deleting rs: 192.168.61.10:80/TCP/10.244.1.5:80
I1208 07:59:28.903798 1 graceful_termination.go:160] Trying to delete rs: 192.168.61.10:80/TCP/10.244.0.6:80
I1208 07:59:28.903824 1 graceful_termination.go:170] Deleting rs: 192.168.61.10:80/TCP/10.244.0.6:80
I1208 07:59:28.904654 1 graceful_termination.go:160] Trying to delete rs: 10.0.2.15:31698/TCP/10.244.0.6:80
I1208 07:59:28.904837 1 graceful_termination.go:170] Deleting rs: 10.0.2.15:31698/TCP/10.244.0.6:80
在Kubernetes的Github上找到了這個ISSUE https://github.com/kubernetes/kubernetes/issues/71071，大致是最近更新的IPVS proxier mode now support connection based graceful termination.引入了bug，導致Kubernetes的1.11.5、1.12.1~1.12.3、1.13.0都有這個問題，即kube-proxy在ipvs模式下不可用。而官方稱在1.11.5、1.12.3、1.13.0中修復了12月4日k8s的特權升級漏洞(CVE-2018-1002105)，如果針對這個漏洞做k8s升級的同學，需要小心，確認是否開啓了ipvs，避免由升級引起k8s網絡問題。由於我們線上的版本是1.11並且已經啓用了ipvs，所以這裏我們只能先把線上master node升級到了1.11.5，而kube-proxy還在使用1.11.4的版本。
https://github.com/kubernetes/kubernetes/issues/71071中已經描述有相關PR解決這個問題，後續只能跟蹤一下1.11.5、1.12.3、1.13.0之後的小版本了。
[root@ns166 system]# kubeadm init --kubernetes-version=v1.13.3 --pod-network-cidr=10.244.0.0/16 --service-cidr=10.96.0.0/12 --ignore-preflight-errors=Swap
[init] Using Kubernetes version: v1.13.3
[preflight] Running pre-flight checks
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 18.09.0. Latest validated version: 18.06
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [ns166 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 10.1.50.166]
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [ns166 localhost] and IPs [10.1.50.166 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [ns166 localhost] and IPs [10.1.50.166 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 21.506378 seconds
[uploadconfig] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.13" in namespace kube-system with the configuration for the kubelets in the cluster
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "ns166" as an annotation
[mark-control-plane] Marking the node ns166 as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node ns166 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: tfw9ca.54r34mgz5dxy6uts
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstraptoken] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstraptoken] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstraptoken] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstraptoken] creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes master has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
You can now join any number of machines by running the following on each node
as root:
kubeadm join 10.1.50.166:6443 --token tfw9ca.54r34mgz5dxy6uts --discovery-token-ca-cert-hash sha256:420b74d911abdf757575619ffbf384253c61c779aa6d243cd61022a9ca81e3bb