CentOS7下搭建Kubernetes集羣

1、測試環境

操作系統：CentOS Linux release 7.4.1708

• 操作機 192.168.1.200
• kube-master 192.168.1.210
• kube-minion-1 192.168.1.211
• kube-minion-2 192.168.1.212
• kube-minion-3 192.168.1.213

2、準備工作

2.1 所有服務器設置靜態IP

在 /etc/sysconfig/network-scripts 路徑下找到 ifcfg- ， 代表具體網卡，本文修改的網卡是 ifcfg-enp0s3

ONBOOT=yes #開機啓動
BOOTPROTO=static #靜態IP
IPADDR=192.168.1.200 #本機地址
NETMASK=255.255.255.0 #子網掩碼
GATEWAY=192.168.1.1 #默認網關

2.2 所有服務器設置DNS

配置文件/etc/sysconfig/network

# Created by anaconda
DNS1=192.168.1.1
DNS2=8.8.8.8

2.3 所有服務器修改hostname

hostnamectl --static set-hostname [主機名]

注：修改成與/etc/hosts對應的相同名稱！

2.4 操作機向所有服務器添加公鑰實現免密碼登錄

通過ssh-keygen -t rsa和ssh-copy-id命令，不再贅述

2.5 在操作機安裝Ansible

yum install ansible

2.6 配置Ansible

編輯/etc/ansible/hosts,在末尾添加服務器信息，添加組：kube、master、nodes:

[kube]
192.168.1.[210:213]

[master]
192.168.1.210

[nodes]
192.168.1.[211:213]

2.7 測試Ansible

查詢kube組內所有服務器啓動運行時間：

ansible kube -a 'uptime'

2.8 所有服務器安裝EPEL擴展源

ansible kube -m shell -a 'yum -y install epel-release'

2.9 所有服務器修改/etc/hosts

ansible kube -m shell -a 'echo -e "192.168.1.210    kube-master\n192.168.1.211    kube-minion-1\n192.168.1.212    kube-minion-2\n192.168.1.213    kube-minion-3" >> /etc/hosts'

注：如果不加參數-m shell，會默認使用command模塊導致添加失敗

2.10 查看所有主機的/etc/hosts

ansible kube -a 'cat /etc/hosts'

2.11 所有服務器安裝Docker

ansible kube -m shell -a 'yum -y install docker'
或
ansible kube -m yum -a 'name=docker state=present'

2.12 查看所有服務器是否成功安裝Docker

ansible kube -m yum -a 'name=docker state=present'

2.13 所有服務器設置Docker開機啓動並啓動服務

ansible kube -m service -a 'name=docker state=restarted enabled=yes'

2.14 檢查所有服務器Docker服務是否正常啓動

ansible kube -m shell -a 'systemctl status docker'

2.15 所有服務器安裝時間同步工具NTP

ansible kube -m yum -a 'name=ntp state=present'

注：NTP配置文件在/etc/ntp.conf，可以對NTP服務器進行設置，NTP服務器可訪問http://www.pool.ntp.org/zh/查看，pool.ntp.org是一個高可用時間服務器虛擬集羣項目，網站建議使用下列默認域名，每個域名會每小時隨機一組NTP服務器，進行時間同步時它會隨機返回離你較近的NTP服務器。

server 0.pool.ntp.org
server 1.pool.ntp.org
server 2.pool.ntp.org
server 3.pool.ntp.org

2.16 所有服務器啓動NTP服務並設開機啓動

ansible kube -m service -a 'name=ntpd state=restarted enabled=yes'

2.17 查看所有服務器NTP服務是否正常開啓

ansible kube -m shell -a 'systemctl status ntpd'

2.18 查看所有服務器獲得到的NTP服務器列表

ansible kube -a 'ntpq -p'

注：NTP服務器列表可能需要等幾分鐘後才能獲得並完成對時，每個服務器獲得的NTP服務器不一樣是正常的。

3、安裝配置Kubernetes

以下步驟參考Kubernetes官網教程：
https://kubernetes.io/docs/getting-started-guides/centos/centos_manual_config/

3.1 所有服務器配置YUM庫源

ansible kube -m shell -a 'echo "[virt7-docker-common-release]
name=virt7-docker-common-release
baseurl=http://cbs.centos.org/repos/virt7-docker-common-release/x86_64/os/
gpgcheck=0" > /etc/yum.repos.d/virt7-docker-common-release.repo'

3.2 所有服務器安裝Kubernetes,etcd,flannel

ansible kube -m shell -a 'yum -y install --enablerepo=virt7-docker-common-release kubernetes etcd flannel'

3.3 所有服務器修改/etc/kubernetes/config配置文件

默認內容：

###
# kubernetes system config
#
# The following values are used to configure various aspects of all
# kubernetes services, including
#
#   kube-apiserver.service
#   kube-controller-manager.service
#   kube-scheduler.service
#   kubelet.service
#   kube-proxy.service
# logging to stderr means we get it in the systemd journal
KUBE_LOGTOSTDERR="--logtostderr=true"

# journal message level, 0 is debug
KUBE_LOG_LEVEL="--v=0"

# Should this cluster be allowed to run privileged docker containers
KUBE_ALLOW_PRIV="--allow-privileged=false"

# How the controller-manager, scheduler, and proxy find the apiserver
KUBE_MASTER="--master=http://127.0.0.1:8080"

需要把KUBE_MASTER改成:

KUBE_MASTER="--master=http://kube-master:8080"

操作機執行修改命令：

ansible kube -m shell -a 'echo  "###
# kubernetes system config
#
# The following values are used to configure various aspects of all
# kubernetes services, including
#
#   kube-apiserver.service
#   kube-controller-manager.service
#   kube-scheduler.service
#   kubelet.service
#   kube-proxy.service
# logging to stderr means we get it in the systemd journal
KUBE_LOGTOSTDERR=\"--logtostderr=true\"

# journal message level, 0 is debug
KUBE_LOG_LEVEL=\"--v=0\"

# Should this cluster be allowed to run privileged docker containers
KUBE_ALLOW_PRIV=\"--allow-privileged=false\"

# How the controller-manager, scheduler, and proxy find the apiserver
KUBE_MASTER=\"--master=http://kube-master:8080\"" > /etc/kubernetes/config'

3.4 所有服務器關掉SELinux和防火牆，並重啓

ansible kube -m shell -a 'setenforce 0;
systemctl disable firewalld;
systemctl stop firewalld;
reboot' 

3.5 kube-master修改etcd配置文件

配置文件在/etc/etcd/etcd.conf，注意確認配置文件中的以下參數與下文一致，主要是兩個localhost改成0.0.0.0

# [member]
ETCD_NAME=default
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379"

#[cluster]
ETCD_ADVERTISE_CLIENT_URLS="http://0.0.0.0:2379"

3.6 kube-master修改apiserver配置文件

打開/etc/kubernetes/apiserver，用以下內容覆蓋：

# The address on the local server to listen to.
KUBE_API_ADDRESS="--address=0.0.0.0"

# The port on the local server to listen on.
KUBE_API_PORT="--port=8080"

# Port kubelets listen on
KUBELET_PORT="--kubelet-port=10250"

# Comma separated list of nodes in the etcd cluster
KUBE_ETCD_SERVERS="--etcd-servers=http://kube-master:2379"

# Address range to use for services
KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=10.254.0.0/16"

# default admission control policies
KUBE_ADMISSION_CONTROL="--admission-control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ResourceQuota"

# Add your own!
KUBE_API_ARGS=""

注：KUBE_ADMISSION_CONTROL裏去掉了ServiceAccount

3.7 kube-master上啓動ETCD

systemctl start etcd
etcdctl mkdir /kube-centos/network
etcdctl mk /kube-centos/network/config "{ \"Network\": \"172.30.0.0/16\", \"SubnetLen\": 24, \"Backend\": { \"Type\": \"vxlan\" } }"

3.8 所有服務器上修改flannel配置

配置文件/etc/sysconfig/flanneld，修改成以下內容：

# Flanneld configuration options

# etcd url location.  Point this to the server where etcd runs
FLANNEL_ETCD_ENDPOINTS="http://kube-master:2379"

# etcd config key.  This is the configuration key that flannel queries
# For address range assignment
FLANNEL_ETCD_PREFIX="/kube-centos/network"

# Any additional options that you want to pass
#FLANNEL_OPTIONS=""

批量修改指令：

ansible kube -m shell -a 'echo "# Flanneld configuration options

# etcd url location.  Point this to the server where etcd runs
FLANNEL_ETCD_ENDPOINTS=\"http://kube-master:2379\"

# etcd config key.  This is the configuration key that flannel queries
# For address range assignment
FLANNEL_ETCD_PREFIX=\"/kube-centos/network\"

# Any additional options that you want to pass
#FLANNEL_OPTIONS=\"\"" > /etc/sysconfig/flanneld'

3.9 kube-master上啓動服務

for SERVICES in etcd kube-apiserver kube-controller-manager kube-scheduler flanneld; do
    systemctl restart $SERVICES
    systemctl enable $SERVICES
    systemctl status $SERVICES
done

3.10 所有nodes服務器配置kubelet

配置文件/etc/kubernetes/kubelet，改爲以下內容：

# The address for the info server to serve on
KUBELET_ADDRESS="--address=0.0.0.0"

# The port for the info server to serve on
KUBELET_PORT="--port=10250"

# You may leave this blank to use the actual hostname
# Check the node number!
KUBELET_HOSTNAME="--hostname-override=kube-minion-n"

# Location of the api-server
KUBELET_API_SERVER="--api-servers=http://kube-master:8080"

KUBELET_POD_INFRA_CONTAINER="--pod-infra-container-image=registry.access.redhat.com/rhel7/pod-infrastructure:latest"

# Add your own!
KUBELET_ARGS=""

注：這裏註釋掉了KUBELET_HOSTNAME，是爲了使用服務器主機名當kubelet名
批量操作指令：

ansible nodes -m shell -a 'echo "# The address for the info server to serve on
KUBELET_ADDRESS=\"--address=0.0.0.0\"

# The port for the info server to serve on
KUBELET_PORT=\"--port=10250\"

# You may leave this blank to use the actual hostname
# Check the node number!
KUBELET_HOSTNAME=\"kube-minion-n\"

# Location of the api-server
KUBELET_API_SERVER=\"--api-servers=http://kube-master:8080\"

# Add your own!
KUBELET_ARGS=\"\"" >/etc/kubernetes/kubelet'

注：KUBELET_HOSTNAME要改成和/etc/hosts裏的一致

3.11 所有nodes服務器啓動服務

ansible nodes -m shell -a 'for SERVICES in kube-proxy kubelet flanneld docker; do
    systemctl restart $SERVICES
    systemctl enable $SERVICES
    systemctl status $SERVICES
done'

3.12 kube-master上啓動Kuberneters集羣

kubectl config set-cluster default-cluster --server=http://kube-master:8080
kubectl config set-context default-context --cluster=default-cluster --user=default-admin
kubectl config use-context default-context
kubectl get nodes

3.13 大功告成！~

目前爲止Kubernetes就搭建完了，拍拍自己肩膀說乾的不錯小夥~ :P

4、搭建Dashboard

4.1 獲得Dashboard的Docker鏡像

需要用到下列兩個Docker鏡像:

• gcr.io/google_containers/kubernetes-dashboard-amd64:v1.5.1
• registry.access.redhat.com/rhel7/pod-infrastructure:latest

由於國內被屏蔽無法直接下載到，所以要用可以訪問的主機下載後添加到所有主機裏。也可以通過docker tag命令添加到Docker私有庫後使用。

操作指令:

# 導出鏡像
docker save gcr.io/google_containers/kubernetes-dashboard-amd64:v1.5.1 > dashboard.tar

# 導入鏡像
docker load < dashboard.tar

4.2 編輯Dashboard的YAML

編輯kubernetes-dashboard.yaml,內容如下：

kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  labels:
    app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kubernetes-dashboard
  template:
    metadata:
      labels:
        app: kubernetes-dashboard
      # Comment the following annotation if Dashboard must not be deployed on master
      annotations:
        scheduler.alpha.kubernetes.io/tolerations: |
          [
            {
              "key": "dedicated",
              "operator": "Equal",
              "value": "master",
              "effect": "NoSchedule"
            }
          ]
    spec:
      containers:
      - name: kubernetes-dashboard
        image: gcr.io/google_containers/kubernetes-dashboard-amd64:v1.5.1
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 9090
          protocol: TCP
        args:
          # Uncomment the following line to manually specify Kubernetes API server Host
          # If not specified, Dashboard will attempt to auto discover the API server and connect
          # to it. Uncomment only if the default does not work.
          - --apiserver-host=http://192.168.1.210:8080    #注意這裏是master的api的地址，要寫master的IP，寫域名會報錯提示訪問不到
        livenessProbe:
          httpGet:
            path: /
            port: 9090
          initialDelaySeconds: 30
          timeoutSeconds: 30
---
kind: Service
apiVersion: v1
metadata:
  labels:
    app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kube-system
spec:
  type: NodePort
  ports:
  - port: 80
    targetPort: 9090
  selector:
    app: kubernetes-dashboard

4.2 部署Dashboard

在kube-master上運行：

#開啓Dashboard
kubectl create -f kubernetes-dashboard.yaml

#查看pod運行狀態
kubectl get pods --all-namespaces
#返回結果示例：
#NAMESPACE     NAME                                    READY     STATUS    #RESTARTS   AGE
#kube-system   kubernetes-dashboard-3345393181-6vq94   1/1       Running   0          44m
#kube-system   zl-redis-1545002913-89r4m               1/1       Running   0          38m
#kube-system   zl-redis-1545002913-cbgv5               1/1       Running   0          38m

#查看單個pod的描述
kubectl describe pod/[pod名字] --namespace=[命名空間]
#例：kubectl describe pod/zl-redis-1545002913-cbgv5 --namespace=kube-system

#查看pod日誌
kubectl logs -f [pod名字] --namespace=[命名空間]
#例：kubectl logs -f zl-redis-1545002913-cbgv5 --namespace=kube-system

4.4 瀏覽器訪問

訪問kube-master網址：http://192.168.1.210:8080/ui