kubeadm 快速安裝 Kubernetes 集羣

前言：官方提供Kubernetes部署3種方式

minikube
Minikube是一個工具，可以在本地快速運行一個單點的Kubernetes，嘗試Kubernetes或日常開發的用戶使用。不能用於生產環境。
官方文檔：https://kubernetes.io/docs/setup/minikube/
kubeadm
kubeadm可幫助你快速部署一套kubernetes集羣。kubeadm設計目的爲新用戶開始嘗試kubernetes提供一種簡單的方法。目前是Beta版。
官方文檔：https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm/
https://kubernetes.io/docs/setup/independent/install-kubeadm/
二進制包
從官方下載發行版的二進制包，手動部署每個組件，組成Kubernetes集羣。目前企業生產環境中主要使用該方式。
下載地址：https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.11.md#v1113

1. 安裝要求
   操作系統
   Ubuntu 16.04+、Debian 9、CentOS 7、RHEL 7、Fedora 25/26 (best-effort)、其他
   內存2GB + ，2核CPU +
   集羣節點之間可以通信
   每個節點唯一主機名，MAC地址和product_uuid
   檢查MAC地址：使用ip link或者ifconfig -a
   檢查product_uuid：cat /sys/class/dmi/id/product_uuid
   禁止swap分區。這樣才能使kubelet正常工作
2. 準備環境
   以下內容（master和node都需要執行）
   關閉防火牆：
   systemctl stop firewalld
   systemctl disable firewalld

關閉selinux：
sed -i 's/enforcing/disabled/' /etc/selinux/config
setenforce 0

關閉swap：
swapoff -a # 臨時
vim /etc/fstab # 永久

添加主機名與IP對應關係：
cat /etc/hosts
192.168.0.246 k8s-master
192.168.0.247 k8s-node1
192.168.0.248 k8s-node2

同步時間：
yum install ntpdate -y
ntpdate ntp.api.bz

3. 安裝Docker
   Docker與Kubernetes關係圖：

設置阿里雲docker（master和node都需要執行）
yum remove docker docker-common docker-selinux docker-engine –y
yum install -y yum-utils device-mapper-persistent-data lvm2
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
yum list docker-ce --showduplicates | sort –r
yum makecache fast
yum install docker-ce-18.06.1.ce -y
或者可以設置官方docker
yum install -y yum-utils device-mapper-persistent-data lvm2
yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
yum install docker-ce-18.06.1.ce -y #目前kubeadm最大支持docker-ce-17.03，所以要指定該版本安裝
systemctl enable docker && systemctl start docker
如果提示container-selinux依賴問題，先安裝ce-18.06匹配版本：
yum localinstall https://download.docker.com/linux/centos/7/x86_64/stable/Packages/docker-ce-selinux-17.03.3.ce-1.el7.noarch.rpm
最新版本是用一下內容：
wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo
yum -y install docker-ce-18.06.1.ce-3.el7
systemctl enable docker && systemctl start docker
docker --version

4. 安裝kubeadm，kubelet和kubectl
   kubeadm： 引導集羣的命令
   kubelet：集羣中運行任務的代理程序
   kubectl：命令行管理工具
   4.1 添加阿里雲YUM軟件源
   cat << EOF > /etc/yum.repos.d/kubernetes.repo
   [kubernetes]
   name=Kubernetes
   baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
   enabled=1
   gpgcheck=1
   repo_gpgcheck=1
   gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
   EOF
   4.2 安裝kubeadm，kubelet和kubectl
   yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes
   systemctl enable kubelet && systemctl start kubelet
   指定版本操作如下：
   yum install -y kubelet-1.13.3 kubeadm-1.13.3 kubectl-1.13.3
   systemctl enable kubelet.service
   注意：使用Docker時，kubeadm會自動檢查kubelet的cgroup驅動程序，並/var/lib/kubelet/kubeadm-flags.env在運行時將其設置在文件中。如果使用的其他CRI，則必須在/etc/default/kubelet中cgroup-driver值修改爲cgroupfs：
   cat /var/lib/kubelet/kubeadm-flags.env
   KUBELET_KUBEADM_ARGS=--cgroup-driver=cgroupfs --cni-bin-dir=/opt/cni/bin --cni-conf-dir=/etc/cni/net.d --network-plugin=cni
   systemctl daemon-reload
   systemctl restart kubelet
5. 使用kubeadm創建單個Master集羣
   5.1 默認下載鏡像地址在國外無法訪問，先從準備好所需鏡像
   保存到腳本之間運行：
   K8S_VERSION=v1.13.3
   ETCD_VERSION=3.2.18
   DASHBOARD_VERSION=v1.8.3
   FLANNEL_VERSION=v0.10.0-amd64
   DNS_VERSION=1.1.3
   PAUSE_VERSION=3.1

   基本組件

   docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver-amd64:$K8S_VERSION
   docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager-amd64:$K8S_VERSION
   docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler-amd64:$K8S_VERSION
   docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy-amd64:$K8S_VERSION
   docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/etcd-amd64:$ETCD_VERSION
   docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:$PAUSE_VERSION
   docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:$DNS_VERSION

   網絡組件

   docker pull quay.io/coreos/flannel:$FLANNEL_VERSION

   修改tag

   docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver-amd64:$K8S_VERSION k8s.gcr.io/kube-apiserver-amd64:$K8S_VERSION
   docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager-amd64:$K8S_VERSION k8s.gcr.io/kube-controller-manager-amd64:$K8S_VERSION
   docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler-amd64:$K8S_VERSION k8s.gcr.io/kube-scheduler-amd64:$K8S_VERSION
   docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy-amd64:$K8S_VERSION k8s.gcr.io/kube-proxy-amd64:$K8S_VERSION
   docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/etcd-amd64:$ETCD_VERSION k8s.gcr.io/etcd-amd64:$ETCD_VERSION
   docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/pause:$PAUSE_VERSION k8s.gcr.io/pause:$PAUSE_VERSION
   docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:$DNS_VERSION k8s.gcr.io/coredns:$DNS_VERSION
   5.2 初始化Master
   kubeadm init \
   --apiserver-advertise-address=192.168.0.246 \
   --image-repository registry.aliyuncs.com/google_containers \
   --kubernetes-version v1.13.3 \
   --service-cidr=10.1.0.0/16 \
   --pod-network-cidr=10.244.0.0/16
   ...

Your Kubernetes master has initialized successfully!

To start using your cluster, you need to run (as a regular user):

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the addon options listed at:
http://kubernetes.io/docs/admin/addons/

You can now join any number of machines by running the following on each node
as root:

kubeadm join --token <token> <master-ip>:<master-port> --discovery-token-ca-cert-hash sha256:<hash>
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
5.3 安裝Pod網絡 - 插件
wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
kubectl apply -f kube-flannel.yml
或使用一下方法
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/v0.10.0/Documentation/kube-flannel.yml
5.4 加入工作節點
在Node節點切換到root賬號執行：
kubeadm join 192.168.0.11:6443 --token 6hk68y.0rdz1wdjyh85ntkr --discovery-token-ca-cert-hash sha256:d1d3f59ae37fbd632707cbeb9b095d0d0b19af535078091993c4bc4d9d2a7782
格式：kubeadm join --token <token> <master-ip>:<master-port> --discovery-token-ca-cert-hash sha256:<hash>
5.5.測試kubernetes集羣
kubectl expose deployment nginx --port=80 --type=NodePort
kubectl get pod,svc
查看nginx分配到那個節點等信息
Kubectl get pod,svc –o wide
kubernetes dashboard
先將yaml文件下載下來，修改裏面鏡像地址和Service NodePort類型。
wget https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/recommended/kubernetes-dashboard.yaml
通過一下地址確認dashboard版本列表
https://github.com/kubernetes/dashboard
修改鏡像地址：(Dashboard Deployment 的位置)
將 image: k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.1
修改爲：
registry.cn-hangzhou.aliyuncs.com/google_containers/kubernetes-dashboard-amd64:v1.10.1
或者直接使用：或者直接：docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kubernetes-dashboard-amd64:v1.10.1
修改Service：（注意因爲yaml文件不支持使用tab鍵）
kind: Service
apiVersion: v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kube-system
spec:
type: NodePort
ports:
port: 443
targetPort: 8443
nodePort: 30001
selector:
k8s-app: kubernetes-dashboard
應用kubernetes-dashboard.yaml
kubectl apply -f kubernetes-dashboard.yaml
登陸方法1：創建一個管理員角色：
apiVersion: v1
kind: ServiceAccount
metadata:
name: dashboard-admin
namespace: kube-system

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: dashboard-admin
subjects:
kind: ServiceAccount
name: dashboard-admin
namespace: kube-system
roleRef:
kind: ClusterRole
name: cluster-admin
apiGroup: rbac.authorization.k8s.io
kubectl apply -f k8s-admin.yaml
使用上述創建賬號的token登錄Kubernetes Dashboard：
kubectl get secret -n kube-system
kubectl describe secret dashboard-admin-token-bwdjj -n kube-system
...
token: eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJkYXNoYm9hcmQtYWRtaW4tdG9rZW4tYndkamoiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGFzaGJvYXJkLWFkbWluIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiNzIyOTRmNTUtYjc1OC0xMWU4LThkY2UtMDAwYzI5ZGUyNWVhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Omt1YmUtc3lzdGVtOmRhc2hib2FyZC1hZG1pbiJ9.0hQU5Di_P1OX1DcnW2AYzjDAED66EOrqhF5iupv39wvB8wE-aLRSQyp0twX2M8u1KMZ67n6LxbH17VwEQkMDRVXs7ZlUCyAAD6kHDz3k-f7PAzH5vcuyO4veQ9ooVjk3DKjrP4zXQChHllBB1wmD_oyLjoWxK3Z5MBTlVGzSixVwuQNpFPbuS6Z7iLGwUOgjI0cGZ9Tt6cXzcK81KfAEpDIP_CtFV_Jw4s98EgBex9mZh6vq1dcxr03qfuK--udd_8GWZctu_p_P15hZZLoKEm5GCbs6JGvKL2aao_DEHfLp3XYEnApojI91vU4qAqdkvMZ2qWQNGYv4KNi2yPOOJQ

登錄方法2：創建sa並綁定默認的cluster-admin管理員集羣角色
kubectl create serviceaccount dashboard-admin -n kube-system

kubectl create clusterrolebinding dashboard-admin --clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-admin
使用上述創建賬號的token登錄Kubernetes Dashboard：
kubectl get secret -n kube-system
kubectl describe secret dashboard-admin-token-bwdjj -n kube-system

7解決Google瀏覽器不能打開kubernetes dashboard方法
mkdir key && cd key
#生成證書
openssl genrsa -out dashboard.key 2048
openssl req -new -out dashboard.csr -key dashboard.key -subj '/CN=192.168.246.200'
openssl x509 -req -in dashboard.csr -signkey dashboard.key -out dashboard.crt
#刪除原有的證書secret
kubectl delete secret kubernetes-dashboard-certs -n kube-system
#創建新的證書secret
kubectl create secret generic kubernetes-dashboard-certs --from-file=dashboard.key --from-file=dashboard.crt -n kube-system
#查看pod
kubectl get pod -n kube-system
#重啓pod
kubectl delete pod <pod name> -n kube-system （pad name 爲kubernetes-dashboard的名稱）