kubernetes 1.15.3部署windows結點

注意：windows結點需要使用windows server 2019 1809以上的英文版，最新的中文版也許可以，但是老一點的中文版裝不上KB4489899補丁包

master端

1. 安裝所需包
執行如下命令

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

echo "nameserver 114.114.114.114" > /etc/resolv.conf
wget http://dl.fedoraproject.org/pub/epel/7/x86_64/Packages/e/epel-release-7-11.noarch.rpm
rpm -ivh epel-release-7-11.noarch.rpm
cd /etc/yum.repos.d/
wget http://mirrors.163.com/.help/CentOS7-Base-163.repo
yum clean all
yum install -y conntrack-tools
yum install -y kubelet kubeadm kubectl ipvsadm  kubernetes-cni

2. 準備環境配置

###############
#配置環境
# 臨時禁用selinux
# 永久關閉 修改/etc/sysconfig/selinux文件設置
sed -i 's/SELINUX=permissive/SELINUX=disabled/' /etc/sysconfig/selinux
setenforce 0
 
# 臨時關閉swap
# 永久關閉 註釋/etc/fstab文件裏swap相關的行
swapoff -a
 
# 開啓forward
# Docker從1.13版本開始調整了默認的防火牆規則
# 禁用了iptables filter表中FOWARD鏈
# 這樣會引起Kubernetes集羣中跨Node的Pod無法通信
 
iptables -P FORWARD ACCEPT
iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X
 
# 配置轉發相關參數，否則可能會出錯
cat <<EOF >  /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
vm.swappiness=0
EOF
sysctl --system 

# 加載ipvs相關內核模塊
# 如果重新開機，需要重新加載
yum install ipset ipvsadm -y
cat > /etc/sysconfig/modules/ipvs.modules <<EOF
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4
EOF
chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack_ipv4

lsmod | grep ip_vs

   從https://download.docker.com/linux/centos/7/x86_64/stable/Packages/下載rpm包安裝docker

3. 安裝kubernetes

docker pull mirrorgooglecontainers/kube-proxy:v1.15.3
docker tag mirrorgooglecontainers/kube-proxy:v1.15.3 k8s.gcr.io/kube-proxy:v1.15.3

docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.3.1
docker tag  registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.3.1   k8s.gcr.io/coredns:1.3.1

docker pull mirrorgooglecontainers/kube-controller-manager:v1.15.3
docker tag mirrorgooglecontainers/kube-controller-manager:v1.15.3  k8s.gcr.io/kube-controller-manager:v1.15.3

docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.15.3
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.15.3  k8s.gcr.io/kube-apiserver:v1.15.3

docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.15.3
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.15.3 k8s.gcr.io/kube-scheduler:v1.15.3

docker pull mirrorgooglecontainers/pause:3.1
docker tag  mirrorgooglecontainers/pause:3.1 k8s.gcr.io/pause:3.1

docker pull mirrorgooglecontainers/etcd:3.3.10
docker tag mirrorgooglecontainers/etcd:3.3.10 k8s.gcr.io/etcd:3.3.10

kubeadm init --pod-network-cidr=10.244.0.0/16 --service-cidr=10.96.0.0/12
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
mkdir -p kube/yaml && cd kube/yaml
kubectl get ds/kube-proxy -o go-template='{{.spec.updateStrategy.type}}{{"\n"}}' --namespace=kube-system
wget https://raw.githubusercontent.com/Microsoft/SDN/master/Kubernetes/flannel/l2bridge/manifests/node-selector-patch.yml
kubectl patch ds/kube-proxy --patch "$(cat node-selector-patch.yml)" -n=kube-system

4. 使用flanne的vxlan方式組網（關鍵步驟）

sudo sysctl net.bridge.bridge-nf-call-iptables=1
wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

使用vi命令修改該文件，
找到net-conf.json項，加入
        "VNI" : 4096,
        "Port": 4789
使其看來如下：

  net-conf.json: |
    {
      "Network": "10.244.0.0/16",
      "Backend": {
        "Type": "vxlan",
        "VNI" : 4096,
        "Port": 4789
      }
    }

找到cni-conf.json項，修改Name爲vxlan0（其默認名爲cbr0）
使其看來如下：

cni-conf.json: |
    {
      "name": "vxlan0",
      "plugins": [
        {
          "type": "flannel",
          "delegate": {
            "hairpinMode": true,
            "isDefaultGateway": true
          }
        },
        {
          "type": "portmap",
          "capabilities": {
            "portMappings": true
          }
        }
      ]
    }

針對很多系統的docker鏡像
修改保存退出後，執行

kubectl apply -f kube-flannel.yml
kubectl patch ds/kube-flannel-ds-amd64 --patch "$(cat node-selector-patch.yml)" -n=kube-system

5. 加入其他結點：
其他需要加入的結點執行，通常命令會在kubeadm執行完成時給出提示
kubeadm join <Master_IP>:6443 --token <some_token> --discovery-token-ca-cert-hash <some_hash>
例如：
kubeadm join 192.168.110.100:6443 --token jsfh39.4855pr1w1234231 \
    --discovery-token-ca-cert-hash sha256:c55eb95acfc056ccaaf62e71ee4033a47b49eb9b103f1233214567 

systemctl enable docker.service

systemctl enable kubelet.service

windows端

1. 安裝docker
Install-Module -Name DockerMsftProvider -Repository PSGallery -Force
Install-Package -Name docker -ProviderName DockerMsftProvider
#如果下載失敗，就手動下載：
https://dockermsft.blob.core.windows.net/dockercontainer/docker-19-03-1.zip
之後，將其拷貝到C:\Users\Administrator\AppData\Local\Temp\2\DockerMsftProvider\目錄下，執行如下操作

 Get-FileHash -Path C:\Users\Administrator\AppData\Local\Temp\2\DockerMsftProvider\Docker-19-03-1.zip -Algorithm SHA256
 Install-Package -Name docker -ProviderName DockerMsftProvider -Verbose
最後重啓系統，執行：

Restart-Computer -Force
之後，重啓docker的命令爲
net stop docker
net start docker
啓動docker服務（自啓動）
Restart-Service docker

2. 啓動hyper-v
Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Hyper-V -All

3. 安裝補丁
https://www.catalog.update.microsoft.com/Search.aspx?q=KB4489899
下載server 2019對應的補丁包

命令行執行
sconfig
菜單選擇6，在Search for for (A)ll updates or (R)ecommended updates only?選擇A，安裝所以更新包

4. 下載鏡像
docker pull mcr.microsoft.com/k8s/core/pause:1.0.0
docker pull mcr.microsoft.com/k8s/core/pause:1.2.0
docker pull mcr.microsoft.com/windows/nanoserver:1809
docker tag mcr.microsoft.com/windows/nanoserver:1809 microsoft/nanoserver:latest

5. 部署程序
mkdir c:\k
$env:Path += ";C:\k"
[Environment]::SetEnvironmentVariable("Path", $env:Path + ";C:\k", [EnvironmentVariableTarget]::Machine)

下載windows程序
進入https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.15.md，例如
https://dl.k8s.io/v1.15.3/kubernetes-node-windows-amd64.tar.gz，解壓後拷貝到c:\k下

將linux的master結點上的/etc/kubernetes/admin.conf文件拷貝爲config到k目錄下
$env:KUBECONFIG="C:\k\config"
[Environment]::SetEnvironmentVariable("KUBECONFIG", "C:\k\config", [EnvironmentVariableTarget]::User)

下載啓動腳本
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
wget https://raw.githubusercontent.com/Microsoft/SDN/master/Kubernetes/flannel/start.ps1 -o c:\k\start.ps1

6. 進入控制面板，關閉防火牆

7. 啓動程序

docker run microsoft/nanoserver:latest

docker run -d mcr.microsoft.com/k8s/core/pause:1.0.0  #這一步很重要，如果漏掉這一步，可能會導致vxlan的網絡創建失敗，可以通過Get-NetAdapter | Format-Table Name,InterfaceDescription,ifIndex,Status,LinkSpeed,MediaConnectionState來檢查
cd c:\k
.\start.ps1 -ManagementIP <Windows Node IP> -NetworkMode <network mode>  -ClusterCIDR <Cluster CIDR> -ServiceCIDR <Service CIDR> -KubeDnsServiceIP <Kube-dns Service IP> -LogDir <Log directory> -KubeletFeatureGates "WinOverlay=true"
<Windows Node IP> 爲windows本機的ip
<network mode>爲網絡模式，這裏使用了vxlan方式，填入overlay
<Cluster CIDR>輸入10.244.0.0/16其值是在kubeadm初始化的過程中定義的
<Service CIDR>輸入10.96.0.0/12
<Kube-dns Service IP>輸入10.96.0.10
<Log directory>輸入日誌路徑c:\var\k8s

例如
.\start.ps1 -ManagementIP 192.168.110.159 -NetworkMode overlay  -ClusterCIDR 10.244.0.0/16 -ServiceCIDR 10.96.0.0/12 -KubeDnsServiceIP 10.96.0.10 -LogDir c:\var\k8s\ -KubeletFeatureGates "WinOverlay=true"
第一次執行時，耗時較多，如果中間卡頓，可以找到具體位置，手動下載文件，並拷貝到相應路徑下