CentOS7.3下二進制安裝Kubernetes 1.16.0高可用集羣

服務器規劃：

至少三臺服務器組建高可用集羣，配置2C4G以上

master，node節點均已安裝docker，版本18以上

VIP 172.30.2.60

172.30.0.109 k8s-master1 nginx keepalived

172.30.0.89 k8s-master2 nginx keepalived

172.30.0.81 k8s-node1

二進制Kubernetes安裝路徑：/opt/kubernetes/{ssl,cfg,bin,logs}分別存放密鑰，配置，可執行，日誌文件

二進制etcd安裝路徑：/opt/etcd/{ssl,cfg,bin}分別存放密鑰，配置，可執行文件

1、系統初始化

1.關閉防火牆：

# systemctl stop firewalld

# systemctl disable firewalld

2.關閉selinux：

# setenforce 0 # 臨時

# sed -i 's/enforcing/disabled/' /etc/selinux/config # 永久

3.關閉swap：

# swapoff -a # 臨時

# vim /etc/fstab # 永久

4.同步系統時間：

# ntpdate time.windows.com

5.添加hosts：

# vim /etc/hosts

172.30.0.109 k8s-master1

172.30.0.81 k8s-master2

172.30.0.89 k8s-node1

6.修改主機名：

hostnamectl set-hostname k8s-master1

2、Etcd集羣

1.安裝cfssl工具

# wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
# wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
# wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
# chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64
# mv cfssl_linux-amd64 /usr/local/bin/cfssl
# mv cfssljson_linux-amd64 /usr/local/bin/cfssljson

# mv cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo

2.生成etcd證書

①生成server.csr.json 修改請求文件中hosts字段包含所有etcd節點IP：

# vi server-csr.json

{

"CN": "etcd",

"hosts": [

"etcd01節點IP",

"etcd02節點IP",

"etcd03節點IP"

"key": {

"algo": "rsa",

"size": 2048

"names": [

{

"C": "CN",

"L": "BeiJing",

"ST": "BeiJing"

}

]

}

②生成自定義CA，初始化配置文件

# vim ca-config.json

{

"signing": {

"default": {

"expiry": "87600h"

"profiles": {

"www": {

"expiry": "87600h",

"usages": [

"signing",

"key encipherment",

"server auth",

"client auth"

]

}

# vim ca-csr.json

{

"CN": "etcd CA",

"key": {

"algo": "rsa",

"size": 2048

"names": [

{

"C": "CN",

"L": "Beijing",

"ST": "Beijing"

}

]

}

etcd密鑰初始化文件準備好後，即可生成密鑰

# cfssl gencert -initca ca-csr.json | cfssljson -bare ca -

# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server

生成的密鑰放置在指定的etcd/ssl目錄下

etcd二進制可執行文件etcd，etcdctl放置在etcd/bin下

etcd配置文件放置在etcd/cfg下，配置文件如下

# cat /opt/etcd/cfg/etcd

#[Member]

ETCD_NAME="etcd02"

ETCD_DATA_DIR="/var/lib/etcd/default.etcd"

ETCD_LISTEN_PEER_URLS="https://172.30.0.81:2380"

ETCD_LISTEN_CLIENT_URLS="https://172.30.0.81:2379"

#[Clustering]

ETCD_INITIAL_ADVERTISE_PEER_URLS="https://172.30.0.81:2380"

ETCD_ADVERTISE_CLIENT_URLS="https://172.30.0.81:2379"

ETCD_INITIAL_CLUSTER="etcd01=https://172.30.2.10:2380,etcd02=https://172.30.0.81:2380,etcd03=https://172.30.0.89:2380"

ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

ETCD_INITIAL_CLUSTER_STATE="new"

將etcd.service放置在/usr/lib/systemd/system/下

# cat /usr/lib/systemd/system/etcd.service

[Unit]

Description=Etcd Server

After=network.target

After=network-online.target

Wants=network-online.target

[Service]

Type=notify

EnvironmentFile=-/opt/etcd/cfg/etcd

ExecStart=/opt/etcd/bin/etcd \

--name=${ETCD_NAME} \

--data-dir=${ETCD_DATA_DIR} \

--listen-peer-urls=${ETCD_LISTEN_PEER_URLS} \

--listen-client-urls=${ETCD_LISTEN_CLIENT_URLS},http://127.0.0.1:2379 \

--advertise-client-urls=${ETCD_ADVERTISE_CLIENT_URLS} \

--initial-advertise-peer-urls=${ETCD_INITIAL_ADVERTISE_PEER_URLS} \

--initial-cluster=${ETCD_INITIAL_CLUSTER} \

--initial-cluster-token=${ETCD_INITIAL_CLUSTER} \

--initial-cluster-state=new \

--cert-file=/opt/etcd/ssl/server.pem \

--key-file=/opt/etcd/ssl/server-key.pem \

--peer-cert-file=/opt/etcd/ssl/server.pem \

--peer-key-file=/opt/etcd/ssl/server-key.pem \

--trusted-ca-file=/opt/etcd/ssl/ca.pem \

--peer-trusted-ca-file=/opt/etcd/ssl/ca.pem

Restart=on-failure

LimitNOFILE=65536

[Install]

WantedBy=multi-user.target

分別拷貝到Etcd三個節點：

# scp –r /opt/etcd root@etcd節點IP:/opt

# scp /usr/lib/systemd/system/etcd.service root@etcd節點IP:/usr/lib/systemd/system

同步修改etcd節點配置文件的配置名稱，使其與主機相對應，如

啓動etcd節點，etcd安裝完成

# systemctl start etcd && systemctl enable etcd

檢查etcd節點健康狀態

# /opt/etcd/bin/etcdctl --ca-file=/opt/etcd/ssl/ca.pem --cert-file=/opt/etcd/ssl/server.pem --key-file=/opt/etcd/ssl/server-key.pem -- endpoints="https://172.30.2.10:2379,https://172.30.0.81:2379,https://172.30.0.89:2379" cluster-health

member 37f20611ff3d9209 is healthy: got healthy result from https://172.30.2.10:2379

member b10f0bac3883a232 is healthy: got healthy result from https://172.30.0.81:2379

member b46624837acedac9 is healthy: got healthy result from https://172.30.0.89:2379

cluster is healthy

3、部署K8S集羣——master節點

①重新生成自定義CA，與etcd進行區分，生成apiserver，kube-proxy，kubectl-admin證書

# vim ca-config.json

{

"signing": {

"default": {

"expiry": "87600h"

"profiles": {

"kubernetes": {

"expiry": "87600h",

"usages": [

"signing",

"key encipherment",

"server auth",

"client auth"

]

}

# vim ca-csr.json

{

"CN": "kubernetes",

"key": {

"algo": "rsa",

"size": 2048

"names": [

{

"C": "CN",

"L": "Beijing",

"ST": "Beijing",

"O": "k8s",

"OU": "System"

}

]

}

②生成kube-proxy密鑰初始化文件

cat kube-proxy-csr.json

{

"CN": "system:kube-proxy",

"hosts": [],

"key": {

"algo": "rsa",

"size": 2048

"names": [

{

"C": "CN",

"L": "BeiJing",

"ST": "BeiJing",

"O": "k8s",

"OU": "System"

}

]

}

③生成apiserver密鑰初始化文件，注意需要寫入所有master節點IP，以及需要訪問apiserver的IP地址，包括nginx，nginx vip，否則需要重新生成密鑰

# vim server-csr.json

{

"CN": "kubernetes",

"hosts": [

"10.0.0.1",

"127.0.0.1",

"kubernetes",

"kubernetes.default",

"kubernetes.default.svc",

"kubernetes.default.svc.cluster",

"kubernetes.default.svc.cluster.local",

"172.30.2.60",

"172.30.0.109",

"172.30.0.81",

"172.30.2.10",

"172.30.0.89"

"key": {

"algo": "rsa",

"size": 2048

"names": [

{

"C": "CN",

"L": "BeiJing",

"ST": "BeiJing",

"O": "k8s",

"OU": "System"

}

]

}

④admin密鑰，供kubectl遠程客戶端運行

# vim admin-csr.json

{

"CN": "admin",

"hosts": [],

"key": {

"algo": "rsa",

"size": 2048

"names": [

{

"C": "CN",

"L": "BeiJing",

"ST": "BeiJing",

"O": "system:masters",

"OU": "System"

}

]

}

⑤生成密鑰

# cfssl gencert -initca ca-csr.json | cfssljson -bare ca -

# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server

# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy

# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin

同步*.pem密鑰至所有節點的/opt/kubernetes/ssl（也可以針對性同步密鑰），kubelet密鑰是後面部署kubelet自動生成的

生成admin的kubeconfig文件，用於客戶端訪問k8s集羣

# kubectl config set-cluster kubernetes --certificate-authority=/opt/kubernetes/ssl/ca.pem --embed-certs=true --server=https://172.30.0.109:6443 --kubeconfig=/root/.kube/kubectl.kubeconfig

# kubectl config set-credentials kube-admin --client-certificate=/opt/kubernetes/ssl/admin.pem --client-key=/opt/kubernetes/ssl/admin-key.pem --embed-certs=true --kubeconfig=/root/.kube/kubectl.kubeconfig

# kubectl config set-context kube-admin@kubernetes --cluster=kubernetes --user=kube-admin --kubeconfig=/root/.kube/kubectl.kubeconfig

# kubectl config use-context kube-admin@kubernetes --kubeconfig=/root/.kube/kubectl.kubeconfig

# mv /root/.kube/{kubectl.kubeconfig,config}

⑥下載master二進制包放置在/opt/kubernetes/bin下

⑦配置配置文件，service啓動文件

kube-apiserve 注意：部署另一臺master時，需要更換IP地址

# vim /opt/kubernetes/cfg/kube-apiserver.conf

KUBE_APISERVER_OPTS="--logtostderr=false \

--v=2 \

--log-dir=/opt/kubernetes/logs \

--etcd-servers=https://172.30.2.10:2379,https://172.30.0.81:2379,https://172.30.0.89:2379 \

--bind-address=172.30.0.109 \

--secure-port=6443 \

--advertise-address=172.30.0.109 \

--allow-privileged=true \

--service-cluster-ip-range=10.0.0.0/24 \

--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \

--authorization-mode=RBAC,Node \

--enable-bootstrap-token-auth=true \

--token-auth-file=/opt/kubernetes/cfg/token.csv \

--service-node-port-range=30000-32767 \

--kubelet-client-certificate=/opt/kubernetes/ssl/server.pem \

--kubelet-client-key=/opt/kubernetes/ssl/server-key.pem \

--tls-cert-file=/opt/kubernetes/ssl/server.pem \

--tls-private-key-file=/opt/kubernetes/ssl/server-key.pem \

--client-ca-file=/opt/kubernetes/ssl/ca.pem \

--service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \

--etcd-cafile=/opt/etcd/ssl/ca.pem \

--etcd-certfile=/opt/etcd/ssl/server.pem \

--etcd-keyfile=/opt/etcd/ssl/server-key.pem \

--audit-log-maxage=30 \

--audit-log-maxbackup=3 \

--audit-log-maxsize=100 \

--audit-log-path=/opt/kubernetes/logs/k8s-audit.log"

# cat /usr/lib/systemd/system/kube-apiserver.service

[Unit]

Description=Kubernetes API Server

Documentation=https://github.com/kubernetes/kubernetes

[Service]

EnvironmentFile=/opt/kubernetes/cfg/kube-apiserver.conf

ExecStart=/opt/kubernetes/bin/kube-apiserver $KUBE_APISERVER_OPTS

Restart=on-failure

[Install]

WantedBy=multi-user.target

kube-controller-manager

# vim /opt/kubernetes/cfg/kube-controller-manager.conf

KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=false \

--v=2 \

--log-dir=/opt/kubernetes/logs \

--leader-elect=true \

--master=127.0.0.1:8080 \

--address=127.0.0.1 \

--allocate-node-cidrs=true \

--cluster-cidr=10.244.0.0/16 \

--service-cluster-ip-range=10.0.0.0/24 \

--cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \

--cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem \

--root-ca-file=/opt/kubernetes/ssl/ca.pem \

--service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem \

--experimental-cluster-signing-duration=87600h0m0s"

# vim /usr/lib/systemd/system/kube-controller-manager.service

[Unit]

Description=Kubernetes Controller Manager

Documentation=https://github.com/kubernetes/kubernetes

[Service]

EnvironmentFile=/opt/kubernetes/cfg/kube-controller-manager.conf

ExecStart=/opt/kubernetes/bin/kube-controller-manager $KUBE_CONTROLLER_MANAGER_OPTS

Restart=on-failure

[Install]

WantedBy=multi-user.target

kube-scheduler

# vim /opt/kubernetes/cfg/kube-scheduler.conf

KUBE_SCHEDULER_OPTS="--logtostderr=false \

--v=2 \

--log-dir=/opt/kubernetes/logs \

--leader-elect \

--master=127.0.0.1:8080 \

--address=127.0.0.1"

# vim /usr/lib/systemd/system/kube-scheduler.service

[Unit]

Description=Kubernetes Scheduler

Documentation=https://github.com/kubernetes/kubernetes

[Service]

EnvironmentFile=/opt/kubernetes/cfg/kube-scheduler.conf

ExecStart=/opt/kubernetes/bin/kube-scheduler $KUBE_SCHEDULER_OPTS

Restart=on-failure

[Install]

WantedBy=multi-user.target

生成token文件，供node節點與apiserver進行通信，注意token值，master需要與node保持一致

# cat /opt/kubernetes/cfg/token.csv

c47ffb939f5ca36231d9e3121a252940,kubelet-bootstrap,10001,"system:node-bootstrapper"

格式：token,用戶,uid,用戶組

token也可自行生成替換：

# head -c 16 /dev/urandom | od -An -t x | tr -d ' '

授權kubelet-bootstrap，使node節點kubelet能夠正常訪問到apiserver

# kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap

啓動master節點，觀察日誌/opt/kubernetes/logs

# systemctl start kube-controller-manager

# systemctl start kube-scheduler

# systemctl enable kube-apiserver

# systemctl enable kube-controller-manager

# systemctl enable kube-scheduler

原樣部署第二個master節點172.30.0.81，不同之處僅在於apiserver配置文件當中的IP地址

4、部署K8S集羣——node節點

①node節點密鑰準備

已提前同步至node節點的/opt/kubernetes/ssl下

②配置node節點kubelet，kube-proxy配置文件

kubelet 注意：不同節點的kubelet需要修改hostname配置

# vim kubelet.conf

KUBELET_OPTS="--logtostderr=false \

--v=2 \

--log-dir=/opt/kubernetes/logs \

--hostname-override=k8s-node1 \

--network-plugin=cni \

--kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \

--bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \

--config=/opt/kubernetes/cfg/kubelet-config.yml \

--cert-dir=/opt/kubernetes/ssl \

--pod-infra-container-image=lizhenliang/pause-amd64:3.0"

bootstrap.kubeconfig 這個文件是kubelet與apiserver的通信認證文件，內部token需要與master節點的token文件保持一致

#vim bootstrap.kubeconfig

apiVersion: v1

clusters:

- cluster:

certificate-authority: /opt/kubernetes/ssl/ca.pem

server: https://172.30.0.109:6443

contexts:

- context:

cluster: kubernetes

user: kubelet-bootstrap

current-context: default

kind: Config

preferences: {}

users:

- name: kubelet-bootstrap

user:

token: c47ffb939f5ca36231d9e3121a252940

# vim kubelet-config.yml

kind: KubeletConfiguration

apiVersion: kubelet.config.k8s.io/v1beta1

address: 0.0.0.0

port: 10250

readOnlyPort: 10255

cgroupDriver: cgroupfs

clusterDNS:

- 10.0.0.2

clusterDomain: cluster.local

failSwapOn: false

authentication:

anonymous:

enabled: false

webhook:

cacheTTL: 2m0s

enabled: true

x509:

clientCAFile: /opt/kubernetes/ssl/ca.pem

authorization:

mode: Webhook

webhook:

cacheAuthorizedTTL: 5m0s

cacheUnauthorizedTTL: 30s

evictionHard:

imagefs.available: 15%

memory.available: 100Mi

nodefs.available: 10%

nodefs.inodesFree: 5%

maxOpenFiles: 1000000

maxPods: 110

# vim /usr/lib/systemd/system/kubelet.service

[Unit]

Description=Kubernetes Kubelet

After=docker.service

Before=docker.service

[Service]

EnvironmentFile=/opt/kubernetes/cfg/kubelet.conf

ExecStart=/opt/kubernetes/bin/kubelet $KUBELET_OPTS

Restart=on-failure

LimitNOFILE=65536

[Install]

WantedBy=multi-user.target

kube-proxy

# vim kube-proxy.conf

KUBE_PROXY_OPTS="--logtostderr=false \

--v=2 \

--log-dir=/opt/kubernetes/logs \

--config=/opt/kubernetes/cfg/kube-proxy-config.yml"

# vim kube-proxy-config.yml

kind: KubeProxyConfiguration

apiVersion: kubeproxy.config.k8s.io/v1alpha1

address: 0.0.0.0

metricsBindAddress: 0.0.0.0:10249

clientConnection:

kubeconfig: /opt/kubernetes/cfg/kube-proxy.kubeconfig

hostnameOverride: k8s-node1

clusterCIDR: 10.0.0.0/24

mode: ipvs

ipvs:

scheduler: "rr"

iptables:

masqueradeAll: true

kube-proxy.kubeconfig kube-proxy通信認證文件

# vim kube-proxy.kubeconfig

apiVersion: v1

clusters:

- cluster:

certificate-authority: /opt/kubernetes/ssl/ca.pem

server: https://172.30.0.109:6443

contexts:

- context:

cluster: kubernetes

user: kube-proxy

current-context: default

kind: Config

preferences: {}

users:

- name: kube-proxy

user:

client-certificate: /opt/kubernetes/ssl/kube-proxy.pem

client-key: /opt/kubernetes/ssl/kube-proxy-key.pem

# vim /usr/lib/systemd/system/kube-proxy.service

[Unit]

Description=Kubernetes Proxy

After=network.target

[Service]

EnvironmentFile=/opt/kubernetes/cfg/kube-proxy.conf

ExecStart=/opt/kubernetes/bin/kube-proxy $KUBE_PROXY_OPTS

Restart=on-failure

LimitNOFILE=65536

[Install]

WantedBy=multi-user.target

/opt/kubernetes/cfg配置如下kubelet.kubeconfig是啓動後自動生成的

/opt/kubernetes/bin

③啓動node節點

# systemctl start kubelet

# systemctl start kube-proxy

# systemctl enable kubelet

# systemctl enable kube-proxy

④允許給node頒發證書

# kubectl get csr

# kubectl certificate approve node-csr-MYUxbmf_nmPQjmH3LkbZRL2uTO-_FCzDQUoUfTy7YjI

# kubectl get node

5、部署CNI網絡

二進制包下載地址：https://github.com/containernetworking/plugins/releases

# mkdir /opt/cni/bin /etc/cni/net.d

# tar zxvf cni-plugins-linux-amd64-v0.8.2.tgz –C /opt/cni/bin

確保kubelet啓用CNI：

# cat /opt/kubernetes/cfg/kubelet.conf

--network-plugin=cni

在Master執行：

# kubectl apply -f kube-flannel.yaml

# kubectl get pods -n kube-system

NAME READY STATUS RESTARTS AGE

kube-flannel-ds-amd64-5xmhh 1/1 Running 6 171m

kube-flannel-ds-amd64-ps5fx 1/1 Running 0 150m

6、授權apiserver訪問kubelet

爲提供安全性，kubelet禁止匿名訪問，必須授權纔可以。

# vim apiserver-to-kubelet-rbac.yaml

apiVersion: rbac.authorization.k8s.io/v1

kind: ClusterRole

metadata:

annotations:

rbac.authorization.kubernetes.io/autoupdate: "true"

labels:

kubernetes.io/bootstrapping: rbac-defaults

rules:

- apiGroups:

- ""

resources:

- nodes/proxy

- nodes/stats

- nodes/log

- nodes/spec

- nodes/metrics

- pods/log

verbs:

- "*"

---

apiVersion: rbac.authorization.k8s.io/v1

kind: ClusterRoleBinding

metadata:

namespace: ""

roleRef:

apiGroup: rbac.authorization.k8s.io

kind: ClusterRole

subjects:

- apiGroup: rbac.authorization.k8s.io

kind: User

# kubectl apply -f apiserver-to-kubelet-rbac.yaml

這樣就能通過kubectl操作pod，查看日誌

7、部署coredns

# kubectl apply –f coredns.yaml

8、K8S高可用配置

kube-controller-manager以及kube-scheduler高可用已經在部署集羣的時候在配置中體現

選舉進行負載均衡，所以我們只需要關注apiserver高可用即可

kube-apiserver高可用

①首先配置好兩臺master節點

②在兩臺master節點上各自部署nginx，keepalived

keepalived監聽nginx健康狀態

nginx配置爲四層監聽兩臺master的6443端口，四層轉發

nginx配置如下

# rpm -vih http://nginx.org/packages/rhel/7/x86_64/RPMS/nginx-1.16.0-1.el7.ngx.x86_64.rpm

# vim /etc/nginx/nginx.conf

……

stream {

log_format main '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent';

access_log /var/log/nginx/k8s-access.log main;

upstream k8s-apiserver {

server 172.30.0.109:6443;

server 172.30.0.81:6443;

}

server {

listen 6443;

proxy_pass k8s-apiserver;

}

……

啓動nginx

# systemctl start nginx

# systemctl enable nginx

keepliaved vip配置爲172.30.2.60

③配置node節點訪問master apiserver經由172.30.2.60進行四層轉發

修改node節點配置，將單點的master的節點，更改爲VIP 172.30.2.60即可

批量修改：

# sed -i 's#172.30.0.109#172.30.2.60#g' *

CentOS7.3下二進制安裝Kubernetes 1.16.0高可用集羣

認知提升的方法

螞蟻面試：Springcloud核心組件的底層原理，你知道多少？

Go隨筆——函數的用法

Go語言學習歷程——Go語言基礎語法入門

Kubernetes1.16基於Prometheus自定義指標彈性伸縮

基於Kubernetes 1.16.0 建立企業Jenkins CI/CD發佈平臺

CentOS7.3下二進制安裝Kubernetes 1.16.0高可用集羣

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結