Kubernetes(k8s)1.14 離線版集羣 - 部署高可用的Etcd（3.3.17版）集羣並搭建Flannel網絡

聲明：
如果您有更好的技術與作者分享，或者商業合作；
請訪問作者個人網站 http://www.esqabc.com/view/message.html 留言給作者。
如果該案例觸犯您的專利，請在這裏：http://www.esqabc.com/view/message.html 留言給作者說明原由
作者一經查實，馬上刪除。

.
.

1、Etcd集羣各節點的名稱和ip如下：

前提提條件、服務器，請查看這個地址：https://blog.csdn.net/esqabc/article/details/102726771

內網IP	名稱
172.26.16.249	k8s-01
172.26.16.250	k8s-02
172.26.16.251	k8s-03

2、下載etcd

a、官方下載地址：https://github.com/etcd-io/etcd/releases/tag/v3.3.17/etcd-v3.3.17-linux-amd64.tar.gz
下載後上傳到：/opt/k8s/work

[root@k8s-01 ~]# cd /opt/k8s/work
[root@k8s-01 ~]# tar -xvf etcd-v3.3.17-linux-amd64.tar.gz

b、分發二進制文件到集羣節點

[root@k8s-01 ~]# cd /opt/k8s/work
[root@k8s-01 ~]# source /opt/k8s/bin/environment.sh

for node_ip in ${ETCD_IPS[@]}
  do
    echo ">>> ${node_ip}"
    scp etcd-v3.3.17-linux-amd64/etcd* root@${node_ip}:/opt/k8s/bin
    ssh root@${node_ip} "chmod +x /opt/k8s/bin/*"
  done

3、創建etcd證書和私鑰

a、創建證書json

[root@k8s-01 ~]# cd /opt/k8s/work
[root@k8s-01 ~]# cat > etcd-csr.json <<EOF
添加下面內容：

{
  "CN": "etcd",
  "hosts": [
    "127.0.0.1",
    "172.26.16.249",
    "172.26.16.250",
    "172.26.16.251"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "4Paradigm"
    }
  ]
}
EOF

說明一下：

hosts：字段指定授權使用該證書的etcd節點IP或域名列表，需要將etcd集羣的3個節點都添加其中

b、生成證書和私鑰

[root@k8s-01 ~]# cd /opt/k8s/work

cfssl gencert -ca=/opt/k8s/work/ca.pem \
    -ca-key=/opt/k8s/work/ca-key.pem \
    -config=/opt/k8s/work/ca-config.json \
    -profile=kubernetes etcd-csr.json | cfssljson -bare etcd

[root@k8s-01 work]# ls etcd*pem

c、分發證書和私鑰到etcd各個節點

[root@k8s-01 ~]# cd /opt/k8s/work
[root@k8s-01 ~]# source /opt/k8s/bin/environment.sh

for node_ip in ${ETCD_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "mkdir -p /etc/etcd/cert"
    scp etcd*.pem root@${node_ip}:/etc/etcd/cert/
  done

d、創建etcd的啓動文件

[root@k8s-01 ~]# cd /opt/k8s/work
[root@k8s-01 ~]# cat > etcd.service.template <<EOF

[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos
[Service]
Type=notify
WorkingDirectory=${ETCD_DATA_DIR}
ExecStart=/opt/k8s/bin/etcd \\
  --data-dir=${ETCD_DATA_DIR} \\
  --wal-dir=${ETCD_WAL_DIR} \\
  --name=##NODE_NAME## \\
  --cert-file=/etc/etcd/cert/etcd.pem \\
  --key-file=/etc/etcd/cert/etcd-key.pem \\
  --trusted-ca-file=/etc/kubernetes/cert/ca.pem \\
  --peer-cert-file=/etc/etcd/cert/etcd.pem \\
  --peer-key-file=/etc/etcd/cert/etcd-key.pem \\
  --peer-trusted-ca-file=/etc/kubernetes/cert/ca.pem \\
  --peer-client-cert-auth \\
  --client-cert-auth \\
  --listen-peer-urls=https://##NODE_IP##:2380 \\
  --initial-advertise-peer-urls=https://##NODE_IP##:2380 \\
  --listen-client-urls=https://##NODE_IP##:2379,http://127.0.0.1:2379 \\
  --advertise-client-urls=https://##NODE_IP##:2379 \\
  --initial-cluster-token=etcd-cluster-0 \\
  --initial-cluster=${ETCD_NODES} \\
  --initial-cluster-state=new \\
  --auto-compaction-mode=periodic \\
  --auto-compaction-retention=1 \\
  --max-request-bytes=33554432 \\
  --quota-backend-bytes=6442450944 \\
  --heartbeat-interval=250 \\
  --election-timeout=2000
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF

說明一下：

WorkDirectory、–data-dir 指定etcd工作目錄和數據存儲爲${ETCD_DATA_DIR},需要在啓動前創建這個目錄
–wal-dir 指定wal目錄，爲了提高性能，一般使用SSD和–data-dir不同的盤
–name
指定節點名稱，當–initial-cluster-state值爲new時，–name的參數值必須位於–initial-cluster列表中
–cert-file、–key-file ETCD server與client通信時使用的證書和私鑰
–trusted-ca-file 簽名client證書的CA證書，用於驗證client證書
–peer-cert-file、–peer-key-file ETCD與peer通信使用的證書和私鑰
–peer-trusted-ca-file 簽名peer證書的CA證書，用於驗證peer證書

e、分發啓動文件到各個節點

(1）、分發會將配置文件中的#替換成ip

[root@k8s-01 ~]# cd /opt/k8s/work

for (( i=0; i < 3; i++ ))
  do
    sed -e "s/##NODE_NAME##/${MASTER_NAMES[i]}/" -e "s/##NODE_IP##/${ETCD_IPS[i]}/" etcd.service.template > etcd-${ETCD_IPS[i]}.service 
  done

[root@k8s-01 ~]# ls *.service

(2）、分發會將配置文件中的#替換成ip

[root@k8s-01 ~]# cd /opt/k8s/work
[root@k8s-01 ~]# source /opt/k8s/bin/environment.sh

for node_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${node_ip}"
    scp etcd-${node_ip}.service root@${node_ip}:/etc/systemd/system/etcd.service
  done

4、啓動etcd服務

a、重命名etcd啓動文件並啓動etcd服務

[root@k8s-01 ~]# cd /opt/k8s/work
[root@k8s-01 ~]# source /opt/k8s/bin/environment.sh

for node_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "mkdir -p ${ETCD_DATA_DIR} ${ETCD_WAL_DIR}"
    ssh root@${node_ip} "systemctl daemon-reload && systemctl enable etcd && systemctl restart etcd " &
  done

b、檢查etcd啓動結果

for node_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "systemctl status etcd|grep Active"
  done

如果啓動正常，如下圖：

如果etcd集羣狀態不是active (running)，請使用下面命令查看etcd日誌

[root@k8s-01 ~]# journalctl -fu etcd

c、驗證ETCD集羣狀態，在任意etcd節點執行

[root@k8s-01 ~]# cd /opt/k8s/work

for node_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ETCDCTL_API=3 /opt/k8s/bin/etcdctl \
    --endpoints=https://${node_ip}:2379 \
    --cacert=/etc/kubernetes/cert/ca.pem \
    --cert=/etc/etcd/cert/etcd.pem \
    --key=/etc/etcd/cert/etcd-key.pem endpoint health
  done

正常狀態，如下圖：

d、查看當前etcd集羣leader

[root@k8s-01 ~]# cd /opt/k8s/work

ETCDCTL_API=3 /opt/k8s/bin/etcdctl \
  -w table --cacert=/etc/kubernetes/cert/ca.pem \
  --cert=/etc/etcd/cert/etcd.pem \
  --key=/etc/etcd/cert/etcd-key.pem \
  --endpoints=${ETCD_ENDPOINTS} endpoint status

正常，如下圖：

.

5、部署Flannel網絡

a、下載分發flanneld二進制文件

[root@k8s-01 ~]# cd /opt/k8s/work
[root@k8s-01 work]# mkdir flannel
[root@k8s-01 work]# wget http://down.i4t.com/k8s1.14/flannel-v0.11.0-linux-amd64.tar.gz
[root@k8s-01 work]# tar -xzvf flannel-v0.11.0-linux-amd64.tar.gz -C flannel

b、分發二進制文件到所有集羣的節點

[root@k8s-01 ~]# cd /opt/k8s/work

for node_ip in ${NODE_IPS[@]}
  do
    echo ">>> ${node_ip}"
    scp flannel/{flanneld,mk-docker-opts.sh} root@${node_ip}:/opt/k8s/bin/
    ssh root@${node_ip} "chmod +x /opt/k8s/bin/*"
  done

c、創建Flannel證書和私鑰

[root@k8s-01 ~]# cd /opt/k8s/work
[root@k8s-01 ~]# cat > flanneld-csr.json <<EOF
添加下面內容：

{
  "CN": "flanneld",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "4Paradigm"
    }
  ]
}
EOF

d、生成證書和私鑰

[root@k8s-01 ~]# cd /opt/k8s/work

cfssl gencert -ca=/opt/k8s/work/ca.pem \
  -ca-key=/opt/k8s/work/ca-key.pem \
  -config=/opt/k8s/work/ca-config.json \
  -profile=kubernetes flanneld-csr.json | cfssljson -bare flanneld

[root@k8s-01 ~]# ls flanneld*pem

e、將生成的證書和私鑰分發到所有節點

[root@k8s-01 ~]# cd /opt/k8s/work

for node_ip in ${NODE_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "mkdir -p /etc/flanneld/cert"
    scp flanneld*.pem root@${node_ip}:/etc/flanneld/cert
  done

f、向etcd寫入Pod網段信息

[root@k8s-01 ~]# cd /opt/k8s/work

etcdctl \
  --endpoints=${ETCD_ENDPOINTS} \
  --ca-file=/opt/k8s/work/ca.pem \
  --cert-file=/opt/k8s/work/flanneld.pem \
  --key-file=/opt/k8s/work/flanneld-key.pem \
  mk ${FLANNEL_ETCD_PREFIX}/config '{"Network":"'${CLUSTER_CIDR}'", "SubnetLen": 21, "Backend": {"Type": "vxlan"}}'

g、創建flanneld的啓動文件

[root@k8s-01 ~]# cd /opt/k8s/work
[root@k8s-01 work]# cat > flanneld.service << EOF

[Unit]
Description=Flanneld overlay address etcd agent
After=network.target
After=network-online.target
Wants=network-online.target
After=etcd.service
Before=docker.service
[Service]
Type=notify
ExecStart=/opt/k8s/bin/flanneld \\
  -etcd-cafile=/etc/kubernetes/cert/ca.pem \\
  -etcd-certfile=/etc/flanneld/cert/flanneld.pem \\
  -etcd-keyfile=/etc/flanneld/cert/flanneld-key.pem \\
  -etcd-endpoints=${ETCD_ENDPOINTS} \\
  -etcd-prefix=${FLANNEL_ETCD_PREFIX} \\
  -iface=${IFACE} \\
  -ip-masq
ExecStartPost=/opt/k8s/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker
Restart=always
RestartSec=5
StartLimitInterval=0
[Install]
WantedBy=multi-user.target
RequiredBy=docker.service
EOF

h、分發啓動文件到所有節點

[root@k8s-01 ~]# cd /opt/k8s/work

for node_ip in ${NODE_IPS[@]}
  do
    echo ">>> ${node_ip}"
    scp flanneld.service root@${node_ip}:/etc/systemd/system/
  done

i、啓動flanneld服務

[root@k8s-01 ~]# cd /opt/k8s/work

for node_ip in ${NODE_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "systemctl daemon-reload && systemctl enable flanneld && systemctl restart flanneld"
  done

j、檢查啓動結果

[root@k8s-01 ~]# cd /opt/k8s/work

for node_ip in ${NODE_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "systemctl status flanneld|grep Active"
  done

正常結果：

k、檢查分配給flanneld的Pod網段信息

[root@k8s-01 ~]# cd /opt/k8s/work

etcdctl \
  --endpoints=${ETCD_ENDPOINTS} \
  --ca-file=/etc/kubernetes/cert/ca.pem \
  --cert-file=/etc/flanneld/cert/flanneld.pem \
  --key-file=/etc/flanneld/cert/flanneld-key.pem \
  get ${FLANNEL_ETCD_PREFIX}/config

l、查看已分配的Pod子網網段列表

[root@k8s-01 ~]# cd /opt/k8s/work

etcdctl \
  --endpoints=${ETCD_ENDPOINTS} \
  --ca-file=/etc/kubernetes/cert/ca.pem \
  --cert-file=/etc/flanneld/cert/flanneld.pem \
  --key-file=/etc/flanneld/cert/flanneld-key.pem \
  ls ${FLANNEL_ETCD_PREFIX}/subnets

m、查看節點flannel網絡信息

[root@k8s-01 ~]# ip addr show

n、檢查是否創建了 flannel 接口

[root@k8s-01 ~]# cd /opt/k8s/work

for node_ip in ${NODE_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh ${node_ip} "/usr/sbin/ip addr show flannel.1|grep -w inet"
  done

.
.
.

也可以參考這篇文章：https://i4t.com/4253.html

Kubernetes(k8s)1.14 離線版集羣 - 部署高可用的Etcd（3.3.17版）集羣並搭建Flannel網絡

1、Etcd集羣各節點的名稱和ip如下：

2、下載etcd

3、創建etcd證書和私鑰

4、啓動etcd服務

5、部署Flannel網絡

美團一面：項目中有 10000 個 if else 如何優化？想了半天，被問懵了！

京東面試：如何進行JVM調優？

Python 將PowerPoint (PPT/PPTX) 轉爲HTML

SQL優化-20231016

Kubernetes(k8s)1.14 離線版集羣 - 搭建環境配置

Spring boot2.0.x+SpringCloud(Finchley版本)分佈式架構--分佈式配置中心(Spring Cloud Config)（五）

Kubernetes(k8s)1.14 離線版集羣 - 部署高可用的Etcd（3.3.17版）集羣並搭建Flannel網絡

Kubernetes(k8s)1.14 離線版集羣 - 部署master節點

Spring boot2.0.x+SpringCloud(Finchley版本)分佈式架構--搭建服務註冊中心與發現服務（一）

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結