K8S 安裝文檔-倪淮濤
2018-05-22
一、環境準備
目前搭建仍然採用 5 臺虛擬機測試，基本環境如下

二、證書相關處理
2.1、證書說明
由於 Etcd 和 Kubernetes 全部採用 TLS 通訊，所以先要生成 TLS 證書，證書生成工具採用 cfssl，具體使用方法這裏不再詳細闡述，生成證書時可在任一節點完成，這裏在宿主機執行，證書列表如下
證書名稱配置文件用途

2.2、安裝 cfssl(一個master上做，然後傳到另外兩個master上)
這個安裝腳本使用的是我私人的 cdn，文件可能隨時刪除，想使用最新版本請自行從 Github clone 並編譯
wget https://mritdftp.b0.upaiyun.com/cfssl/cfssl.tar.gz
tar -zxvf cfssl.tar.gz
mv cfssl cfssljson /usr/local/bin
chmod +x /usr/local/bin/cfssl /usr/local/bin/cfssljson
rm -f cfssl.tar.gz
2.3、生成 Etcd 證書
etcd-csr.json
{
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"O": "etcd",
"OU": "etcd Security",
"L": "Beijing",
"ST": "Beijing",
"C": "CN"
}
],
"CN": "etcd",
"hosts": [
"127.0.0.1",
"localhost",
"192.168.182.181",
"192.168.182.182",
"192.168.182.183"
]}
etcd-gencert.json
{
"signing": {
"default": {
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
],
"expiry": "87600h"
}
}}
etcd-root-ca-csr.json
{
"key": {
"algo": "rsa",
"size": 4096
},
"names": [
{
"O": "etcd",
"OU": "etcd Security",
"L": "Beijing",
"ST": "Beijing",
"C": "CN"
}
],
"CN": "etcd-root-ca"}
生成證書
cfssl gencert --initca=true etcd-root-ca-csr.json | cfssljson --bare etcd-root-ca
cfssl gencert --ca etcd-root-ca.pem --ca-key etcd-root-ca-key.pem --config etcd-gencert.json etcd-csr.json | cfssljson --bare etcd
生成後如下

2.3、安裝 Etcd(三個master上做)
Etcd 這裏採用最新的 3.2.18 版本，安裝方式直接複製二進制文件、systemd service 配置即可，不過需要注意相關用戶權限問題，以下腳本配置等參考了 etcd rpm 安裝包
etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
EnvironmentFile=-/etc/etcd/etcd.conf
User=etcd# set GOMAXPROCS to number of processors
ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /usr/local/bin/etcd --name=\"${ETCD_NAME}\" --data-dir=\"${ETCD_DATA_DIR}\" --listen-client-urls=\"${ETCD_LISTEN_CLIENT_URLS}\""
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
etcd.conf

[member]

ETCD_NAME=etcd1
ETCD_DATA_DIR="/var/lib/etcd/etcd1.etcd"
ETCD_WAL_DIR="/var/lib/etcd/wal"
ETCD_SNAPSHOT_COUNT="100"
ETCD_HEARTBEAT_INTERVAL="100"
ETCD_ELECTION_TIMEOUT="1000"
ETCD_LISTEN_PEER_URLS="https://192.168.182.181:2380" //改成當前master的ip
ETCD_LISTEN_CLIENT_URLS="https://192.168.182.181:2379,http://127.0.0.1:2379"
ETCD_MAX_SNAPSHOTS="5"ETCD_MAX_WALS="5"#ETCD_CORS=""

[cluster]

ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.182.181:2380" //改成當前master的ip

if you use different ETCD_NAME (e.g. test), set ETCD_INITIAL_CLUSTER value for this name, i.e. "test=http://..."

ETCD_INITIAL_CLUSTER="etcd1=https://192.168.182.181:2380,etcd2=https://192.168.182.182:2380,etcd3=https://192.168.182.183:2380"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_ADVERTISE_CLIENT_URLS="192.168.182.181:2379" //改成當前master的ip
#ETCD_DISCOVERY=""#ETCD_DISCOVERY_SRV=""#ETCD_DISCOVERY_FALLBACK="proxy"#ETCD_DISCOVERY_PROXY=""#ETCD_STRICT_RECONFIG_CHECK="false"#ETCD_AUTO_COMPACTION_RETENTION="0"

[proxy]#ETCD_PROXY="off"#ETCD_PROXY_FAILURE_WAIT="5000"#ETCD_PROXY_REFRESH_INTERVAL="30000"#ETCD_PROXY_DIAL_TIMEOUT="1000"#ETCD_PROXY_WRITE_TIMEOUT="5000"#ETCD_PROXY_READ_TIMEOUT="0"

[security]

ETCD_CERT_FILE="/etc/etcd/ssl/etcd.pem"
ETCD_KEY_FILE="/etc/etcd/ssl/etcd-key.pem"
ETCD_CLIENT_CERT_AUTH="true"
ETCD_TRUSTED_CA_FILE="/etc/etcd/ssl/etcd-root-ca.pem"
ETCD_AUTO_TLS="true"ETCD_PEER_CERT_FILE="/etc/etcd/ssl/etcd.pem"
ETCD_PEER_KEY_FILE="/etc/etcd/ssl/etcd-key.pem"
ETCD_PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/ssl/etcd-root-ca.pem"
ETCD_PEER_AUTO_TLS="true"

[logging]#ETCD_DEBUG="false"# examples for -log-package-levels etcdserver=WARNING,security=DEBUG#ETCD_LOG_PACKAGE_LEVELS=""

install.sh
#!/bin/bash
set -e
ETCD_VERSION="3.2.18"
function download(){
if [ ! -f "etcd-v${ETCD_VERSION}-linux-amd64.tar.gz" ]; then
wget https://github.com/coreos/etcd/releases/download/v${ETCD_VERSION}/etcd-v${ETCD_VERSION}-linux-amd64.tar.gz
tar -zxvf etcd-v${ETCD_VERSION}-linux-amd64.tar.gz
fi}
function preinstall(){
getent group etcd >/dev/null || groupadd -r etcd
getent passwd etcd >/dev/null || useradd -r -g etcd -d /var/lib/etcd -s /sbin/nologin -c "etcd user" etcd}

function install(){
echo -e "\033[32mINFO: Copy etcd...\033[0m"
tar -zxvf etcd-v${ETCD_VERSION}-linux-amd64.tar.gz
cp etcd-v${ETCD_VERSION}-linux-amd64/etcd /usr/local/bin
rm -rf etcd-v${ETCD_VERSION}-linux-amd64
echo -e "\033[32mINFO: Copy etcd config...\033[0m"
cp -r conf /etc/etcd
chown -R etcd:etcd /etc/etcd
chmod -R 755 /etc/etcd/ssl
echo -e "\033[32mINFO: Copy etcd systemd config...\033[0m"
cp systemd/.service /lib/systemd/system
systemctl daemon-reload
}
function postinstall(){
if [ ! -d "/var/lib/etcd" ]; then
mkdir /var/lib/etcd
chown -R etcd:etcd /var/lib/etcd
Fi
}

download
Preinstall
Install
postinstall
腳本解釋如下:
· download: 從 Github 下載二進制文件並解壓
· preinstall: 爲 Etcd 安裝做準備，創建 etcd 用戶，並指定家目錄登錄 shell 等
· install: 將 etcd 二進制文件複製到安裝目錄(/usr/local/bin)，複製 conf 目錄到 /etc/etcd
· postinstall: 安裝後收尾工作，比如檢測 /var/lib/etcd 是否存在，糾正權限等
整體目錄結構如下
/etc/etcd
├── conf
│ ├── etcd.conf
│ └── ssl
│ ├── etcd.csr
│ ├── etcd-csr.json
│ ├── etcd-gencert.json
│ ├── etcd-key.pem
│ ├── etcd.pem
│ ├── etcd-root-ca.csr
│ ├── etcd-root-ca-csr.json
│ ├── etcd-root-ca-key.pem
│ └── etcd-root-ca.pem
├── etcd.service
└── install.sh
請自行創建 conf 目錄等，並放置好相關文件，保存上面腳本爲 install.sh，直接執行即可；在每臺機器上更改好對應的配置，如 etcd 名稱等，etcd 估計都是輕車熟路了，這裏不做過多闡述；安裝後啓動即可
systemctl start etcd
systemctl enable etcd
注意: 集羣 etcd 要 3 個一起啓動，集羣模式下單個啓動會卡半天最後失敗，不要傻等；啓動成功後測試如下
export ETCDCTL_API=3
etcdctl --cacert=/etc/etcd/ssl/etcd-root-ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem --endpoints=https://192.168.182.181:2379,https://192.168.182.182:2379,https://192.168.182.183:2379 endpoint health

三、安裝 Kubernets 集羣組件
注意：與以前文檔不同的是，這次不依賴 rpm 等特定安裝包，而是基於 hyperkube 二進制手動安裝，每個節點都會同時安裝 Master 與 Node 配置文件，具體作爲 Master 還是 Node 取決於服務開啓情況
3.1、生成 Kubernetes 證書
由於 kubelet 和 kube-proxy 用到的 kubeconfig 配置文件需要藉助 kubectl 來生成，所以需要先安裝一下 kubectl（所有master和node上都做）
wget https://storage.googleapis.com/kubernetes-release/release/v1.10.1/bin/linux/amd64/hyperkube -O hyperkube_1.10.1
chmod +x hyperkube_1.10.1
cp hyperkube_1.10.1 /usr/local/bin/hyperkube
ln -s /usr/local/bin/hyperkube /usr/local/bin/kubectl
以下生成證書和祕鑰在一臺master上做，傳到其他兩臺master上
admin-csr.json
{
"CN": "admin",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "system:masters",
"OU": "System"
}
]}
k8s-gencert.json
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"kubernetes": {
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
],
"expiry": "87600h"
}
}
}}
k8s-root-ca-csr.json
{
"CN": "kubernetes",
"key": {
"algo": "rsa",
"size": 4096
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "k8s",
"OU": "System"
}
]}
kube-apiserver-csr.json
注意: 在以前的文檔中這個配置叫 kubernetes-csr.json，爲了明確劃分職責，這個證書目前被重命名以表示其專屬於 apiserver 使用；加了一個 .kubernetes.master 域名以便內部私有 DNS 解析使用(可刪除)；至於很多人問過 kubernetes 這幾個能不能刪掉，答案是不可以的；因爲當集羣創建好後，default namespace 下會創建一個叫 kubenretes 的 svc，有一些組件會直接連接這個 svc 來跟 api 通訊的，證書如果不包含可能會出現無法連接的情況；其他幾個 kubernetes 開頭的域名作用相同
{
"CN": "kubernetes",
"hosts": [
"127.0.0.1",
"10.254.0.1",
"192.168.182.181",
"192.168.182.182",
"192.168.182.183",
"192.168.182.184",
"192.168.182.185",
".kubernetes.master",
"localhost",
"kubernetes",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.cluster",
"kubernetes.default.svc.cluster.local"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "k8s",
"OU": "System"
}
]}
kube-proxy-csr.json
{
"CN": "system:kube-proxy",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "k8s",
"OU": "System"
}
]}
生成證書及配置

生成 CA

cfssl gencert --initca=true k8s-root-ca-csr.json | cfssljson --bare k8s-root-ca

依次生成其他組件證書

for targetName in kube-apiserver admin kube-proxy;
do
cfssl gencert --ca k8s-root-ca.pem --ca-key k8s-root-ca-key.pem --config k8s-gencert.json --profile kubernetes $targetName-csr.json | cfssljson --bare $targetName
done

地址默認爲 127.0.0.1:6443# 如果在 master 上啓用 kubelet 請在生成後的 kubeconfig 中修改該地址爲當前MASTER_IP:6443，master不做kubelet時，master上直接用127的地址即可

KUBE_APISERVER="https://127.0.0.1:6443"
BOOTSTRAP_TOKEN=$(head -c 16 /dev/urandom | od -An -t x | tr -d ' ')
echo "Tokne: ${BOOTSTRAP_TOKEN}"

不要質疑 system:bootstrappers 用戶組是否寫錯了，有疑問請參考官方文檔

https://kubernetes.io/docs/admin/kubelet-tls-bootstrapping/

cat > token.csv <<EOF
${BOOTSTRAP_TOKEN},kubelet-bootstrap,10001,"system:bootstrappers"
EOF
echo "Create kubelet bootstrapping kubeconfig..."

設置集羣參數

kubectl config set-cluster kubernetes \
--certificate-authority=ssl/k8s-root-ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=bootstrap.kubeconfig

設置客戶端認證參數

kubectl config set-credentials kubelet-bootstrap \
--token=${BOOTSTRAP_TOKEN} \
--kubeconfig=bootstrap.kubeconfig

設置上下文參數

kubectl config set-context default \
--cluster=kubernetes \
--user=kubelet-bootstrap \
--kubeconfig=bootstrap.kubeconfig

設置默認上下文

kubectl config use-context default --kubeconfig=bootstrap.kubeconfig
echo "Create kube-proxy kubeconfig..."

設置集羣參數

kubectl config set-cluster kubernetes \
--certificate-authority=ssl/k8s-root-ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=kube-proxy.kubeconfig

設置客戶端認證參數

kubectl config set-credentials kube-proxy \
--client-certificate=ssl/kube-proxy.pem \
--client-key=ssl/kube-proxy-key.pem \
--embed-certs=true \
--kubeconfig=kube-proxy.kubeconfig

設置上下文參數

kubectl config set-context default \
--cluster=kubernetes \
--user=kube-proxy \
--kubeconfig=kube-proxy.kubeconfig

設置默認上下文

kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig

創建高級審計配置

cat >> audit-policy.yaml <<EOF

Log all requests at the Metadata level.

apiVersion: audit.k8s.io/v1beta1
kind: Policy
rules:

level: Metadata
EOF
生成後文件如下

3.2、準備 systemd 配置
所有組件的 systemd 配置如下
kube-apiserver.service
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target
After=etcd.service
[Service]
EnvironmentFile=-/etc/kubernetes/config
EnvironmentFile=-/etc/kubernetes/apiserver
User=kube
ExecStart=/usr/local/bin/hyperkube apiserver \
$KUBE_LOGTOSTDERR \
$KUBE_LOG_LEVEL \
$KUBE_ETCD_SERVERS \
$KUBE_API_ADDRESS \
$KUBE_API_PORT \
$KUBELET_PORT \
$KUBE_ALLOW_PRIV \
$KUBE_SERVICE_ADDRESSES \
$KUBE_ADMISSION_CONTROL \
$KUBE_API_ARGSRestart=on-failure
Type=notify
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
kube-controller-manager.service
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
[Service]
EnvironmentFile=-/etc/kubernetes/config
EnvironmentFile=-/etc/kubernetes/controller-manager
User=kube
ExecStart=/usr/local/bin/hyperkube controller-manager \
$KUBE_LOGTOSTDERR \
$KUBE_LOG_LEVEL \
$KUBE_MASTER \
$KUBE_CONTROLLER_MANAGER_ARGS
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
kubelet.service
[Unit]
Description=Kubernetes Kubelet Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=docker.service
Requires=docker.service
[Service]
WorkingDirectory=/var/lib/kubelet
EnvironmentFile=-/etc/kubernetes/config
EnvironmentFile=-/etc/kubernetes/kubelet
ExecStart=/usr/local/bin/hyperkube kubelet \
$KUBE_LOGTOSTDERR \
$KUBE_LOG_LEVEL \
$KUBELET_API_SERVER \
$KUBELET_ADDRESS \
$KUBELET_PORT \
$KUBELET_HOSTNAME \
$KUBE_ALLOW_PRIV \
$KUBELET_ARGS
Restart=on-failure
KillMode=process
[Install]
WantedBy=multi-user.target
kube-proxy.service
[Unit]
Description=Kubernetes Kube-Proxy Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target
[Service]
EnvironmentFile=-/etc/kubernetes/config
EnvironmentFile=-/etc/kubernetes/proxy
ExecStart=/usr/local/bin/hyperkube proxy \
$KUBE_LOGTOSTDERR \
$KUBE_LOG_LEVEL \
$KUBE_MASTER \
$KUBE_PROXY_ARGS
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
kube-scheduler.service
[Unit]
Description=Kubernetes Scheduler Plugin
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
[Service]
EnvironmentFile=-/etc/kubernetes/config
EnvironmentFile=-/etc/kubernetes/scheduler
User=kube
ExecStart=/usr/local/bin/hyperkube scheduler \
$KUBE_LOGTOSTDERR \
$KUBE_LOG_LEVEL \
$KUBE_MASTER \
$KUBE_SCHEDULER_ARGS
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
3.3、Master 節點配置
Master 節點主要會運行 3 各組件: kube-apiserver、kube-controller-manager、kube-scheduler，其中用到的配置文件如下
config
config 是一個通用配置文件，值得注意的是由於安裝時對於 Node、Master 節點都會包含該文件，在 Node 節點上請註釋掉 KUBE_MASTER 變量，因爲 Node 節點需要做 HA，要連接本地的 6443 加密端口；而這個變量將會覆蓋 kubeconfig 中指定的 127.0.0.1:6443 地址

kubernetes system config## The following values are used to configure various aspects of all# kubernetes services, including## kube-apiserver.service# kube-controller-manager.service# kube-scheduler.service# kubelet.service# kube-proxy.service# logging to stderr means we get it in the systemd journal

KUBE_LOGTOSTDERR="--logtostderr=true"

journal message level, 0 is debug

KUBE_LOG_LEVEL="--v=2"

Should this cluster be allowed to run privileged docker containers

KUBE_ALLOW_PRIV="--allow-privileged=true"

How the controller-manager, scheduler, and proxy find the apiserver

#KUBE_MASTER="--master=http://127.0.0.1:8080" //註釋掉
apiserver
apiserver 配置相對於 1.8 略有變動，其中准入控制器(admission control)選項名稱變爲了 --enable-admission-plugins，控制器列表也有相應變化，這裏採用官方推薦配置，具體請參考官方文檔

kubernetes system config## The following values are used to configure the kube-apiserver

The address on the local server to listen to.

KUBE_API_ADDRESS="--advertise-address=192.168.182.181 --bind-address=192.168.182.181" //改成本機的ip

The port on the local server to listen on.

KUBE_API_PORT="--secure-port=6443"

Port minions listen on# KUBELET_PORT="--kubelet-port=10250"

Comma separated list of nodes in the etcd cluster

KUBE_ETCD_SERVERS="--etcd-servers=https://192.168.182.181:2379,https://192.168.182.182:2379,https://192.168.182.183:2379"

Address range to use for services

KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=10.254.0.0/16"

default admission control policies

KUBE_ADMISSION_CONTROL="--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction"

Add your own!

KUBE_API_ARGS=" --anonymous-auth=false \
--apiserver-count=3 \
--audit-log-maxage=30 \
--audit-log-maxbackup=3 \
--audit-log-maxsize=100 \
--audit-log-path=/var/log/kube-audit/audit.log \
--audit-policy-file=/etc/kubernetes/audit-policy.yaml \
--authorization-mode=Node,RBAC \
--client-ca-file=/etc/kubernetes/ssl/k8s-root-ca.pem \
--enable-bootstrap-token-auth \
--enable-garbage-collector \
--enable-logs-handler \
--enable-swagger-ui \
--etcd-cafile=/etc/etcd/ssl/etcd-root-ca.pem \
--etcd-certfile=/etc/etcd/ssl/etcd.pem \
--etcd-keyfile=/etc/etcd/ssl/etcd-key.pem \
--etcd-compaction-interval=5m0s \
--etcd-count-metric-poll-period=1m0s \
--event-ttl=48h0m0s \
--kubelet-https=true \
--kubelet-timeout=3s \
--log-flush-frequency=5s \
--token-auth-file=/etc/kubernetes/token.csv \
--tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem \
--tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem \
--service-node-port-range=30000-50000 \
--service-account-key-file=/etc/kubernetes/ssl/k8s-root-ca.pem \
--storage-backend=etcd3 \
--enable-swagger-ui=true"
controller-manager
controller manager 配置默認開啓了證書輪換能力用於自動簽署 kueblet 證書，並且證書時間也設置了 10 年，可自行調整；增加了 --controllers 選項以指定開啓全部控制器

The following values are used to configure the kubernetes controller-manager

defaults from config and apiserver should be adequate

Add your own!KUBE_CONTROLLER_MANAGER_ARGS=" --bind-address=0.0.0.0 \

                            --cluster-name=kubernetes \
                            --cluster-signing-cert-file=/etc/kubernetes/ssl/k8s-root-ca.pem \
                            --cluster-signing-key-file=/etc/kubernetes/ssl/k8s-root-ca-key.pem \
                            --controllers=*,bootstrapsigner,tokencleaner \
                            --deployment-controller-sync-period=10s \
                            --experimental-cluster-signing-duration=86700h0m0s \
                            --leader-elect=true \
                            --node-monitor-grace-period=40s \
                            --node-monitor-period=5s \
                            --pod-eviction-timeout=5m0s \
                            --terminated-pod-gc-threshold=50 \
                            --root-ca-file=/etc/kubernetes/ssl/k8s-root-ca.pem \
                            --service-account-private-key-file=/etc/kubernetes/ssl/k8s-root-ca-key.pem \
                            --feature-gates=RotateKubeletServerCertificate=true"

scheduler

kubernetes scheduler config

default config should be adequate

Add your own!KUBE_SCHEDULER_ARGS=" --address=0.0.0.0 \

                    --leader-elect=true \
                    --algorithm-provider=DefaultProvider"

3.4、Node 節點配置
Node 節點上主要有 kubelet、kube-proxy 組件，用到的配置如下
kubelet
kubeket 默認也開啓了證書輪換能力以保證自動續簽相關證書，同時增加了 --node-labels 選項爲 node 打一個標籤，關於這個標籤最後部分會有討論，如果在 master 上啓動 kubelet，請將 node-role.kubernetes.io/k8s-node=true 修改爲 node-role.kubernetes.io/k8s-master=true

kubernetes kubelet (minion) config

The address for the info server to serve on (set to 0.0.0.0 or "" for all interfaces)

KUBELET_ADDRESS="--node-ip=192.168.182.181"

The port for the info server to serve on# KUBELET_PORT="--port=10250"

You may leave this blank to use the actual hostname

KUBELET_HOSTNAME="--hostname-override=k1.node" //修改成本機的主機名

location of the api-server# KUBELET_API_SERVER=""

Add your own!

KUBELET_ARGS=" --bootstrap-kubeconfig=/etc/kubernetes/bootstrap.kubeconfig \
--cert-dir=/etc/kubernetes/ssl \
--cgroup-driver=cgroupfs \
--cluster-dns=10.254.0.2 \
--cluster-domain=cluster.local. \
--fail-swap-on=false \
--feature-gates=RotateKubeletClientCertificate=true,RotateKubeletServerCertificate=true \
--node-labels=node-role.kubernetes.io/k8s-node=true \
--image-gc-high-threshold=70 \
--image-gc-low-threshold=50 \
--kube-reserved=cpu=500m,memory=512Mi,ephemeral-storage=1Gi \
--kubeconfig=/etc/kubernetes/kubelet.kubeconfig \
--system-reserved=cpu=1000m,memory=1024Mi,ephemeral-storage=1Gi \
--serialize-image-pulls=false \
--sync-frequency=30s \
--pod-infra-container-image=k8s.gcr.io/pause-amd64:3.0 \
--resolv-conf=/etc/resolv.conf \
--rotate-certificates"
proxy

kubernetes proxy config# default config should be adequate# Add your own!

KUBE_PROXY_ARGS="--bind-address=0.0.0.0 \
--hostname-override=k8s-node4 \
--kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig \
--cluster-cidr=10.254.0.0/16"
3.5、安裝集羣組件
上面已經準備好了相關配置文件，接下來將這些配置文件組織成如下目錄結構以便後續腳本安裝
k8s
├── conf
│ ├── apiserver
│ ├── audit-policy.yaml
│ ├── bootstrap.kubeconfig
│ ├── config
│ ├── controller-manager
│ ├── kubelet
│ ├── kube-proxy.kubeconfig
│ ├── proxy
│ ├── scheduler
│ ├── ssl
│ │ ├── admin.csr
│ │ ├── admin-csr.json
│ │ ├── admin-key.pem
│ │ ├── admin.pem
│ │ ├── k8s-gencert.json
│ │ ├── k8s-root-ca.csr
│ │ ├── k8s-root-ca-csr.json
│ │ ├── k8s-root-ca-key.pem
│ │ ├── k8s-root-ca.pem
│ │ ├── kube-apiserver.csr
│ │ ├── kube-apiserver-csr.json
│ │ ├── kube-apiserver-key.pem
│ │ ├── kube-apiserver.pem
│ │ ├── kube-proxy.csr
│ │ ├── kube-proxy-csr.json
│ │ ├── kube-proxy-key.pem
│ │ └── kube-proxy.pem
│ └── token.csv
├── hyperkube_1.10.1
├── install.sh
└── systemd
├── kube-apiserver.service
├── kube-controller-manager.service
├── kubelet.service
├── kube-proxy.service
└── kube-scheduler.service
其中 install.sh 內容如下（此腳本master和node上都要執行）
#!/bin/bash
set -e
KUBE_VERSION="1.10.1"
function downloadk8s(){
if [ ! -f "hyperkube${KUBE_VERSION}" ]; then
wget https://storage.googleapis.com/kubernetes-release/release/v${KUBE_VERSION}/bin/linux/amd64/hyperkube -O hyperkube_${KUBEVERSION}
chmod +x hyperkube${KUBE_VERSION}
fi}
function preinstall(){
getent group kube >/dev/null || groupadd -r kube
getent passwd kube >/dev/null || useradd -r -g kube -d / -s /sbin/nologin -c "Kubernetes user" kube}
function installk8s(){
echo -e "\033[32mINFO: Copy hyperkube...\033[0m"
cp hyperkube${KUBE_VERSION} /usr/local/bin/hyperkube

echo -e "\033[32mINFO: Create symbolic link...\033[0m"
ln -sf /usr/local/bin/hyperkube /usr/local/bin/kubectl

echo -e "\033[32mINFO: Copy kubernetes config...\033[0m"
cp -r conf /etc/kubernetes
if [ -d "/etc/kubernetes/ssl" ]; then
    chown -R kube:kube /etc/kubernetes/ssl
fi

echo -e "\033[32mINFO: Copy kubernetes systemd config...\033[0m"
cp systemd/*.service /lib/systemd/system
systemctl daemon-reload}

function postinstall(){
if [ ! -d "/var/log/kube-audit" ]; then
mkdir /var/log/kube-audit
fi

if [ ! -d "/var/lib/kubelet" ]; then
    mkdir /var/lib/kubelet
fi

if [ ! -d "/usr/libexec" ]; then
    mkdir /usr/libexec
fi
chown -R kube:kube /var/log/kube-audit /var/lib/kubelet /usr/libexec}

download_k8s
preinstall
install_k8s
postinstall
腳本解釋如下:
· download_k8s: 下載 hyperkube 二進制文件
· preinstall: 安裝前處理，同 etcd 一樣創建 kube 普通用戶指定家目錄、shell 等
· install_k8s: 複製 hyperkube 到安裝目錄，爲 kubectl 創建軟連接(爲啥創建軟連接就能執行請自行閱讀源碼)，複製相關配置到對應目錄，並處理權限
· postinstall: 收尾工作，創建日誌目錄等，並處理權限
最後執行此腳本安裝即可，此外，應確保每個節點安裝了 ipset、conntrack 兩個包，因爲 kube-proxy 組件會使用其處理 iptables 規則等
四、啓動 Kubernetes Master 節點
對於 master 節點啓動無需做過多處理，多個 master 只要保證 apiserver 等配置中的 ip 地址監聽沒問題後直接啓動即可
systemctl daemon-reload
systemctl start kube-apiserver
systemctl start kube-controller-manager
systemctl start kube-scheduler
systemctl enable kube-apiserver
systemctl enable kube-controller-manager
systemctl enable kube-scheduler

systemctl daemon-reload;systemctl start kube-apiserver;systemctl start kube-controller-manager;systemctl start kube-scheduler;systemctl enable kube-apiserver;systemctl enable kube-controller-manager;systemctl enable kube-scheduler

####################################################################################################
成功後截圖如下

五、啓動 Kubernetes Node 節點
由於 HA 等功能需要，對於 Node 需要做一些處理才能啓動，主要有以下兩個地方需要處理
5.1、nginx-proxy（node節點啓動）
在啓動 kubelet、kube-proxy 服務之前，需要在本地啓動 nginx 來 tcp 負載均衡 apiserver 6443 端口，nginx-proxy 使用 docker + systemd 啓動，配置如下
注意: 對於在 master 節點啓動 kubelet 來說，不需要 nginx 做負載均衡；可以跳過此步驟，並修改kubelet.kubeconfig、kube-proxy.kubeconfig 中的 apiserver 地址爲當前 master ip 6443 端口即可
· nginx-proxy.service
[Unit]
Description=kubernetes apiserver docker wrapper
Wants=docker.socket
After=docker.service
[Service]
User=root
PermissionsStartOnly=true
ExecStart=/usr/bin/docker run -p 127.0.0.1:6443:6443 \
-v /etc/nginx:/etc/nginx \
--name nginx-proxy \
--net=host \
--restart=on-failure:5 \
--memory=512M \
nginx:1.13.12-alpine
ExecStartPre=-/usr/bin/docker
rm -f nginx-proxy
ExecStop=/usr/bin/docker stop nginx-proxy
Restart=always
RestartSec=15s
TimeoutStartSec=30s
[Install]
WantedBy=multi-user.target
· nginx.conf
error_log stderr notice;

worker_processes auto;
events {
multi_accept on;
use epoll;
worker_connections 1024;}

stream {
upstream kube_apiserver {
least_conn;
server 192.168.182.181:6443;
server 192.168.182.182:6443;
server 192.168.182.183:6443;
}

server {
    listen        0.0.0.0:6443;
    proxy_pass    kube_apiserver;
    proxy_timeout 10m;
    proxy_connect_timeout 1s;
}}

啓動 apiserver 的本地負載均衡
mkdir /etc/nginx
cp nginx.conf /etc/nginx
cp nginx-proxy.service /lib/systemd/system
systemctl daemon-reload;systemctl start nginx-proxy;systemctl enable nginx-proxy
5.2、TLS bootstrapping
創建好 nginx-proxy 後不要忘記爲 TLS Bootstrap 創建相應的 RBAC 規則，這些規則能實現證自動簽署 TLS Bootstrap 發出的 CSR 請求，從而實現證書輪換(創建一次即可)；詳情請參考 Kubernetes TLS bootstrapping 那點事
· tls-bootstrapping-clusterrole.yaml(與 1.8 一樣)

A ClusterRole which instructs the CSR approver to approve a node requesting a# serving cert matching its client cert

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: system:certificates.k8s.io:certificatesigningrequests:selfnodeserver
rules:

apiGroups: ["certificates.k8s.io"]
resources: ["certificatesigningrequests/selfnodeserver"]
verbs: ["create"]
在 master 執行創建(在一臺master上執行即可)
給與 kubelet-bootstrap 用戶進行 node-bootstrapper 的權限

kubectl create clusterrolebinding kubelet-bootstrap \
--clusterrole=system:node-bootstrapper \
--user=kubelet-bootstrap

kubectl create -f tls-bootstrapping-clusterrole.yaml

自動批准 system:bootstrappers 組用戶 TLS bootstrapping 首次申請證書的 CSR 請求

kubectl create clusterrolebinding node-client-auto-approve-csr \
--clusterrole=system:certificates.k8s.io:certificatesigningrequests:nodeclient \
--group=system:bootstrappers

自動批准 system:nodes 組用戶更新 kubelet 自身與 apiserver 通訊證書的 CSR 請求

kubectl create clusterrolebinding node-client-auto-renew-crt \
--clusterrole=system:certificates.k8s.io:certificatesigningrequests:selfnodeclient \
--group=system:nodes

自動批准 system:nodes 組用戶更新 kubelet 10250 api 端口證書的 CSR 請求

kubectl create clusterrolebinding node-server-auto-renew-crt \
--clusterrole=system:certificates.k8s.io:certificatesigningrequests:selfnodeserver \
--group=system:nodes
5.3、執行啓動
多節點部署時先啓動好 nginx-proxy，然後修改好相應配置的 ip 地址等配置，最終直接啓動即可(master 上啓動 kubelet 不要忘了修改 kubeconfig 中的 apiserver 地址，還有對應的 kubelet 的 node label)
systemctl daemon-reload
systemctl start kubelet
systemctl start kube-proxy
systemctl enable kubelet
systemctl enable kube-proxy
最後啓動成功後如下

五、安裝 Calico
Calico 安裝仍然延續以前的方案，使用 Daemonset 安裝 cni 組件，使用 systemd 控制 calico-node 以確保 calico-node 能正確的拿到主機名等
5.1、修改 Calico 配置 (一臺master上執行即可)
wget https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/calico.yaml -O calico.example.yaml
ETCD_CERT=cat /etc/etcd/ssl/etcd.pem | base64 | tr -d '\n'
ETCD_KEY=cat /etc/etcd/ssl/etcd-key.pem | base64 | tr -d '\n'
ETCD_CA=cat /etc/etcd/ssl/etcd-root-ca.pem | base64 | tr -d '\n'
ETCD_ENDPOINTS="https://192.168.182.181:2379,https://192.168.182.182:2379,https://192.168.182.183:2379"
cp calico.example.yaml calico.yaml
sed -i "s@.etcd_endpoints:.@\ \ etcd_endpoints:\ \"${ETCD_ENDPOINTS}\"@gi" calico.yaml
sed -i "s@.etcd-cert:.@\ \ etcd-cert:\ ${ETCD_CERT}@gi" calico.yaml
sed -i "s@.etcd-key:.@\ \ etcd-key:\ ${ETCD_KEY}@gi" calico.yamlsed -i "s@.etcd-ca:.@\ \ etcd-ca:\ ${ETCD_CA}@gi" calico.yaml
sed -i 's@.etcd_ca:.@\ \ etcd_ca:\ "/calico-secrets/etcd-ca"@gi' calico.yaml
sed -i 's@.etcd_cert:.@\ \ etcd_cert:\ "/calico-secrets/etcd-cert"@gi' calico.yaml
sed -i 's@.etcd_key:.@\ \ etcd_key:\ "/calico-secrets/etcd-key"@gi' calico.yaml

註釋掉 calico-node 部分(由 Systemd 接管)

sed -i '123,219s@.*@#&@gi' calico.yaml
5.2、創建 Systemd 文件
注意: 創建 systemd service 配置文件要在每個節點上都執行
K8S_MASTER_IP="192.168.182.181" //修改成本機ip
HOSTNAME=cat /etc/hostname
ETCD_ENDPOINTS="https://192.168.182.181:2379,https://192.168.182.182:2379,https://192.168.182.183:2379"
cat > /lib/systemd/system/calico-node.service <<EOF
[Unit]
Description=calico node
After=docker.service
Requires=docker.service

[Service]
User=root
Environment=ETCD_ENDPOINTS=${ETCD_ENDPOINTS}
PermissionsStartOnly=true
ExecStart=/usr/bin/docker run --net=host --privileged --name=calico-node \
-e ETCD_ENDPOINTS=\${ETCD_ENDPOINTS} \
-e ETCD_CA_CERT_FILE=/etc/etcd/ssl/etcd-root-ca.pem \
-e ETCD_CERT_FILE=/etc/etcd/ssl/etcd.pem \
-e ETCD_KEY_FILE=/etc/etcd/ssl/etcd-key.pem \
-e NODENAME=${HOSTNAME} \
-e IP= \
-e IP_AUTODETECTION_METHOD=can-reach=${K8S_MASTER_IP} \
-e AS=64512 \
-e CLUSTER_TYPE=k8s,bgp \
-e CALICO_IPV4POOL_CIDR=10.20.0.0/16 \
-e CALICO_IPV4POOL_IPIP=always \
-e CALICO_LIBNETWORK_ENABLED=true \
-e CALICO_NETWORKING_BACKEND=bird \
-e CALICO_DISABLE_FILE_LOGGING=true \
-e FELIX_IPV6SUPPORT=false \
-e FELIX_DEFAULTENDPOINTTOHOSTACTION=ACCEPT \
-e FELIX_LOGSEVERITYSCREEN=info \
-e FELIX_IPINIPMTU=1440 \
-e FELIX_HEALTHENABLED=true \
-e CALICO_K8S_NODE_REF=${HOSTNAME} \
-v /etc/calico/etcd-root-ca.pem:/etc/etcd/ssl/etcd-root-ca.pem \
-v /etc/calico/etcd.pem:/etc/etcd/ssl/etcd.pem \
-v /etc/calico/etcd-key.pem:/etc/etcd/ssl/etcd-key.pem \
-v /lib/modules:/lib/modules \
-v /var/lib/calico:/var/lib/calico \
-v /var/run/calico:/var/run/calico \
quay.io/calico/node:v3.1.0
ExecStop=/usr/bin/docker rm -f calico-node
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF
對於以上腳本中的 K8S_MASTER_IP 變量，只需要填寫一個 master ip 即可，這個變量用於 calico 自動選擇 IP 使用；在宿主機有多張網卡的情況下，calcio node 會自動獲取一個 IP，獲取原則就是嘗試是否能夠聯通這個 master ip
由於 calico 需要使用 etcd 存儲數據，所以需要複製 etcd 證書到相關目錄，/etc/calico 需要在每個節點（master和node）都有
每臺機子上創建 /etc/calico目錄
Madir /etc/calico
cp -r /etc/etcd/ssl /etc/calico （master和node上都要有）
5.3、修改 kubelet 配置（node上操作）
使用 Calico 後需要修改 kubelet 配置增加 CNI 設置(--network-plugin=cni)，修改後配置如下

kubernetes kubelet (minion) config

The address for the info server to serve on (set to 0.0.0.0 or "" for all interfaces)

KUBELET_ADDRESS="--node-ip=192.168.1.61"

The port for the info server to serve on# KUBELET_PORT="--port=10250"

You may leave this blank to use the actual hostname

KUBELET_HOSTNAME="--hostname-override=k1.node"

location of the api-server# KUBELET_API_SERVER=""

Add your own!

KUBELET_ARGS=" --bootstrap-kubeconfig=/etc/kubernetes/bootstrap.kubeconfig \
--cert-dir=/etc/kubernetes/ssl \
--cgroup-driver=cgroupfs \
--network-plugin=cni \
--cluster-dns=10.254.0.2 \
--cluster-domain=cluster.local. \
--fail-swap-on=false \
--feature-gates=RotateKubeletClientCertificate=true,RotateKubeletServerCertificate=true \
--node-labels=node-role.kubernetes.io/k8s-master=true \
--image-gc-high-threshold=70 \
--image-gc-low-threshold=50 \
--kube-reserved=cpu=500m,memory=512Mi,ephemeral-storage=1Gi \
--kubeconfig=/etc/kubernetes/kubelet.kubeconfig \
--system-reserved=cpu=1000m,memory=1024Mi,ephemeral-storage=1Gi \
--serialize-image-pulls=false \
--sync-frequency=30s \
--pod-infra-container-image=k8s.gcr.io/pause-amd64:3.0 \
--resolv-conf=/etc/resolv.conf \
--rotate-certificates"
5.4、創建 Calico Daemonset(在一臺master上操作即可)

先創建 RBAC

kubectl apply -f \
https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/rbac.yaml

再創建 Calico Daemonset

kubectl create -f calico.yaml
#######################################################################################
刪除
kubectl apply -f https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/rbac.yaml
kubectl delete -f calico.yaml
########################################################################################
5.5、啓動 Calico Node
systemctl daemon-reload;systemctl restart calico-node;systemctl enable calico-node;ps ax|grep calico --color

等待 20s 拉取鏡像

sleep 20;systemctl restart kubelet
5.6、測試網絡
網絡測試與其他幾篇文章一樣，創建幾個 pod 測試即可

創建 deployment

cat << EOF >> demo.deploy.yml
apiVersion: apps/v1
kind: Deployment
metadata:
name: demo-deployment
spec:
replicas: 5
selector:
matchLabels:
app: demo
template:
metadata:
labels:
app: demo
spec:
containers:

name: demo
image: mritd/demo
imagePullPolicy: IfNotPresent
ports:
- containerPort: 80
  EOF
  kubectl create -f demo.deploy.yml
  測試結果如圖所示

六、部署集羣 DNS
6.1、部署 CoreDNS（在一臺master上部署即可）
CoreDNS 給出了標準的 deployment 配置，如下
· coredns.yaml.sed
apiVersion: v1
kind: ServiceAccount
metadata:
name: coredns
namespace: kube-system---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
labels:
kubernetes.io/bootstrapping: rbac-defaults
name: system:coredns
rules:

apiGroups:
- ""
  resources:
- endpoints
- services
- pods
- namespaces
  verbs:
- list
- watch---
  apiVersion: rbac.authorization.k8s.io/v1beta1
  kind: ClusterRoleBinding
  metadata:
  annotations:
  rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
  kubernetes.io/bootstrapping: rbac-defaults
  name: system:coredns
  roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:coredns
  subjects:
kind: ServiceAccount
name: coredns
namespace: kube-system---
apiVersion: v1
kind: ConfigMap
metadata:
name: coredns
namespace: kube-system
data:
Corefile: |
.:53 {
errors
health
kubernetes CLUSTER_DOMAIN REVERSE_CIDRS {
pods insecure
upstream
fallthrough in-addr.arpa ip6.arpa
}
prometheus :9153
proxy . /etc/resolv.conf
cache 30
}---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: coredns
namespace: kube-system
labels:
k8s-app: kube-dns
kubernetes.io/name: "CoreDNS"
spec:
replicas: 2
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
selector:
matchLabels:
k8s-app: kube-dns
template:
metadata:
labels:
k8s-app: kube-dns
spec:
serviceAccountName: coredns
tolerations:
- key: "CriticalAddonsOnly"
  operator: "Exists"
  containers:
  - name: coredns
    image: coredns/coredns:1.1.1
    imagePullPolicy: IfNotPresent
    args: [ "-conf", "/etc/coredns/Corefile" ]
    volumeMounts:
- name: config-volume
  mountPath: /etc/coredns
  ports:
- containerPort: 53
  name: dns
  protocol: UDP
- containerPort: 53
  name: dns-tcp
  protocol: TCP
- containerPort: 9153
  name: metrics
  protocol: TCP
  livenessProbe:
  httpGet:
  path: /health
  port: 8080
  scheme: HTTP
  initialDelaySeconds: 60
  timeoutSeconds: 5
  successThreshold: 1
  failureThreshold: 5
  dnsPolicy: Default
  volumes:
- name: config-volume
  configMap:
  name: coredns
  items:
  - key: Corefile
    path: Corefile---
    apiVersion: v1
    kind: Service
    metadata:
    name: kube-dns
    namespace: kube-system
    annotations:
    prometheus.io/scrape: "true"
    labels:
    k8s-app: kube-dns
    kubernetes.io/cluster-service: "true"
    kubernetes.io/name: "CoreDNS"
    spec:
    selector:
    k8s-app: kube-dns
    clusterIP: CLUSTER_DNS_IP
    ports:
    - name: dns
      port: 53
      protocol: UDP
    - name: dns-tcp
      port: 53
      protocol: TCP
      然後直接使用腳本替換即可(腳本變量我已經修改了)
      #!/bin/bash
      Deploys CoreDNS to a cluster currently running Kube-DNS.
      
      SERVICE_CIDR=${1:-10.254.0.0/16}
      POD_CIDR=${2:-10.20.0.0/16}
      CLUSTER_DNS_IP=${3:-10.254.0.2}
      CLUSTER_DOMAIN=${4:-cluster.local}
      YAML_TEMPLATE=${5:-pwd/coredns.yaml.sed}
      sed -e s/CLUSTER_DNS_IP/$CLUSTER_DNS_IP/g -e s/CLUSTER_DOMAIN/$CLUSTER_DOMAIN/g -e s?SERVICE_CIDR?$SERVICE_CIDR?g -e s?POD_CIDR?$POD_CIDR?g $YAML_TEMPLATE > coredns.yaml
      最後使用 kubectl 創建一下
      
      執行上面的替換腳本
      
      ./deploy.sh
      
      創建 CoreDNS
      
      kubectl create -f coredns.yaml
      測試截圖如下
      6.2、部署 DNS 自動擴容
      自動擴容跟以往一樣，yaml 創建一下就行
      · dns-horizontal-autoscaler.yaml
      
      Copyright 2016 The Kubernetes Authors.## Licensed under the Apache License, Version 2.0 (the "License");# you may not use this file except in compliance with the License.# You may obtain a copy of the License at## http://www.apache.org/licenses/LICENSE-2.0## Unless required by applicable law or agreed to in writing, software# distributed under the License is distributed on an "AS IS" BASIS,# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.# See the License for the specific language governing permissions and# limitations under the License.

kind: ServiceAccount
apiVersion: v1
metadata:
name: kube-dns-autoscaler
namespace: kube-system
labels:
addonmanager.kubernetes.io/mode: Reconcile---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: system:kube-dns-autoscaler
labels:
addonmanager.kubernetes.io/mode: Reconcile
rules:

apiGroups: [""]
resources: ["nodes"]
verbs: ["list"]
apiGroups: [""]
resources: ["replicationcontrollers/scale"]
verbs: ["get", "update"]
apiGroups: ["extensions"]
resources: ["deployments/scale", "replicasets/scale"]
verbs: ["get", "update"]# Remove the configmaps rule once below issue is fixed:# kubernetes-incubator/cluster-proportional-autoscaler#16
apiGroups: [""]
resources: ["configmaps"]
verbs: ["get", "create"]---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: system:kube-dns-autoscaler
labels:
addonmanager.kubernetes.io/mode: Reconcile
subjects:
kind: ServiceAccount
name: kube-dns-autoscaler
namespace: kube-system
roleRef:
kind: ClusterRole
name: system:kube-dns-autoscaler
apiGroup: rbac.authorization.k8s.io

apiVersion: apps/v1
kind: Deployment
metadata:
name: kube-dns-autoscaler
namespace: kube-system
labels:
k8s-app: kube-dns-autoscaler
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
spec:
selector:
matchLabels:
k8s-app: kube-dns-autoscaler
template:
metadata:
labels:
k8s-app: kube-dns-autoscaler
annotations:
scheduler.alpha.kubernetes.io/critical-pod: ''
spec:
priorityClassName: system-cluster-critical
containers:
- name: autoscaler
  image: k8s.gcr.io/cluster-proportional-autoscaler-amd64:1.1.2-r2
  resources:
  requests:
  cpu: "20m"
  memory: "10Mi"
  command:
  - /cluster-proportional-autoscaler
  - --namespace=kube-system
  - --configmap=kube-dns-autoscaler
    Should keep target in sync with cluster/addons/dns/kube-dns.yaml.base
  - --target=Deployment/kube-dns
    When cluster is using large nodes(with more cores), "coresPerReplica" should dominate.
    
    If using small nodes, "nodesPerReplica" should dominate.
  - --default-params={"linear":{"coresPerReplica":256,"nodesPerReplica":16,"preventSinglePointFailure":true}}
  - --logtostderr=true
  - --v=2
    tolerations:
- key: "CriticalAddonsOnly"
  operator: "Exists"
  serviceAccountName: kube-dns-autoscaler
  七、部署 heapster
  heapster 部署相對簡單的多，yaml 創建一下就可以了
  kubectl create -f https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/influxdb/grafana.yaml
  kubectl create -f https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/influxdb/heapster.yaml
  kubectl create -f https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/influxdb/influxdb.yaml
  kubectl create -f https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/rbac/heapster-rbac.yaml
  八、部署 Dashboard
  8.1、部署 Dashboard
  Dashboard 部署同 heapster 一樣，不過爲了方便訪問，我設置了 NodePort，還注意到一點是 yaml 拉取策略已經沒有比較傻的 Always 了
  wget https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard.yaml -O kubernetes-dashboard.yaml
  將最後部分的端口暴露修改如下
  ------------------- Dashboard Service -------------------
  
  kind: Service
  apiVersion: v1
  metadata:
  labels:
  k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kube-system
  spec:
  type: NodePort
  ports:
  - name: dashboard-tls
    port: 443
    targetPort: 8443
    nodePort: 30000
    protocol: TCP
    selector:
    k8s-app: kubernetes-dashboard
    然後執行 kubectl create -f kubernetes-dashboard.yaml 即可

從上可以看出，dashboard部署到了k8s0node4（192.168.182.184）上，採用的又是ndeport模式，那麼就可以通過https://192.168.182.184:30000的方式進行訪問了
8.2、創建 admin 賬戶
默認情況下部署成功後可以直接訪問 https://NODE_IP:30000 訪問，但是想要登錄進去查看的話需要使用 kubeconfig 或者 access token 的方式；實際上這個就是 RBAC 授權控制，以下提供一個創建 admin access token 的腳本，更細節的權限控制比如只讀用戶可以參考使用 RBAC 控制 kubectl 權限，RBAC 權限控制原理是一樣的

#!/bin/bash
if kubectl get sa dashboard-admin -n kube-system &> /dev/null;then
echo -e "\033[33mWARNING: ServiceAccount dashboard-admin exist!\033[0m"else
kubectl create sa dashboard-admin -n kube-system
kubectl create clusterrolebinding dashboard-admin --clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-admin
fi
kubectl describe secret -n kube-system $(kubectl get secrets -n kube-system | grep dashboard-admin | cut -f1 -d ' ') | grep -E '^token'
將以上腳本保存爲 create_dashboard_sa.sh 執行即可，成功後訪問截圖如下(如果訪問不了的話請檢查下 iptable FORWARD 默認規則是否爲 DROP，如果是將其改爲 ACCEPT 即可)

九、其他說明
9.1、選項 label 等說明
部署過程中注意到一些選項已經做了名稱更改，比如 --network-plugin-dir 變更爲 --cni-bin-dir 等，具體的那些選項做了變更請自行對比配置，以及查看官方文檔；
對於 Node label --node-labels=node-role.kubernetes.io/k8s-node=true 這個選項，它的作用只是在 kubectl get node 時 ROLES 欄顯示是什麼節點；不過需要注意 master 上的 kubelet 不要將node-role.kubernetes.io/k8s-master=true 更改成 node-role.kubernetes.io/master=xxxx；後面這個 node-role.kubernetes.io/master 是 kubeadm 用的，這個 label 會告訴 k8s 調度器當前節點爲 master，從而執行一些特定動作，比如 node-role.kubernetes.io/master:NoSchedule 此節點將不會被分配 pod；具體參見 kubespray issue 以及官方設計文檔
很多人可能會發現大約 1 小時候 kubectl get csr 看不到任何 csr 了，這是因爲最新版本增加了 csr 清理功能，默認對於 approved 和 denied 狀態的 csr 一小時後會被清理，對於 pending 狀態的 csr 24 小時後會被清理，想問時間從哪來的請看代碼；PR issue 我忘記了，增加這個功能的起因大致就是因爲當開啓了證書輪換後，csr 會不斷增加，所以需要增加一個清理功能
9.2、異常及警告說明
在部署過程中我記錄了一些異常警告等，以下做一下統一說明

https://github.com/kubernetes/kubernetes/issues/42158# 這個問題還沒解決，PR 沒有合併被關閉了，可以關注一下上面這個 issue，被關閉的 PR 在下面# https://github.com/kubernetes/kubernetes/pull/49567

Failed to update statusUpdateNeeded field in actual state of world: Failed to set statusUpdateNeeded to needed true, because nodeName=...

https://github.com/kubernetes/kubernetes/issues/59993# 這個似乎已經解決了，沒時間測試，PR 地址在下面，我大致 debug 一下好像是 cAdvisor 的問題# https://github.com/opencontainers/runc/pull/1722

Failed to get system container stats for "/kubepods": failed to get cgroup stats for "/kubepods": failed to get container info for "/kubepods": unknown containe "/kubepods"

https://github.com/kubernetes/kubernetes/issues/58217# 注意: 這個問題現在仍未解決，可關注上面的 issue，這個問題可能影響 node image gc# 強烈依賴於 kubelet 做宿主機 image gc 的需要注意一下

Image garbage collection failed once. Stats initialization may not have completed yet: failed to get imageFs info: unable to find data for container /

沒找到太多資料，不過感覺跟上面問題類似

failed to construct signal: "allocatableMemory.available" error: system container "pods" not found in metrics
轉載請註明出處，本文采用 CC4.0 協議授權

K8S 安裝文檔

[member]

[cluster]

if you use different ETCD_NAME (e.g. test), set ETCD_INITIAL_CLUSTER value for this name, i.e. "test=http://..."

[proxy]#ETCD_PROXY="off"#ETCD_PROXY_FAILURE_WAIT="5000"#ETCD_PROXY_REFRESH_INTERVAL="30000"#ETCD_PROXY_DIAL_TIMEOUT="1000"#ETCD_PROXY_WRITE_TIMEOUT="5000"#ETCD_PROXY_READ_TIMEOUT="0"

[security]

[logging]#ETCD_DEBUG="false"# examples for -log-package-levels etcdserver=WARNING,security=DEBUG#ETCD_LOG_PACKAGE_LEVELS=""

生成 CA

依次生成其他組件證書

地址默認爲 127.0.0.1:6443# 如果在 master 上啓用 kubelet 請在生成後的 kubeconfig 中修改該地址爲 當前MASTER_IP:6443，master不做kubelet時，master上直接用127的地址即可

不要質疑 system:bootstrappers 用戶組是否寫錯了，有疑問請參考官方文檔

設置集羣參數

設置客戶端認證參數

設置上下文參數

設置默認上下文

設置集羣參數

設置客戶端認證參數

設置上下文參數

設置默認上下文

創建高級審計配置

Log all requests at the Metadata level.

journal message level, 0 is debug

Should this cluster be allowed to run privileged docker containers

How the controller-manager, scheduler, and proxy find the apiserver

kubernetes system config## The following values are used to configure the kube-apiserver

The address on the local server to listen to.

The port on the local server to listen on.

Port minions listen on# KUBELET_PORT="--kubelet-port=10250"

Comma separated list of nodes in the etcd cluster

Address range to use for services

default admission control policies

Add your own!

The following values are used to configure the kubernetes controller-manager

defaults from config and apiserver should be adequate

Add your own!KUBE_CONTROLLER_MANAGER_ARGS=" --bind-address=0.0.0.0 \

kubernetes scheduler config

default config should be adequate

Add your own!KUBE_SCHEDULER_ARGS=" --address=0.0.0.0 \

kubernetes kubelet (minion) config

The address for the info server to serve on (set to 0.0.0.0 or "" for all interfaces)

The port for the info server to serve on# KUBELET_PORT="--port=10250"

You may leave this blank to use the actual hostname

location of the api-server# KUBELET_API_SERVER=""

Add your own!

kubernetes proxy config# default config should be adequate# Add your own!

A ClusterRole which instructs the CSR approver to approve a node requesting a# serving cert matching its client cert

給與 kubelet-bootstrap 用戶進行 node-bootstrapper 的權限

自動批准 system:bootstrappers 組用戶 TLS bootstrapping 首次申請證書的 CSR 請求

自動批准 system:nodes 組用戶更新 kubelet 自身與 apiserver 通訊證書的 CSR 請求

自動批准 system:nodes 組用戶更新 kubelet 10250 api 端口證書的 CSR 請求

註釋掉 calico-node 部分(由 Systemd 接管)

kubernetes kubelet (minion) config

The address for the info server to serve on (set to 0.0.0.0 or "" for all interfaces)

The port for the info server to serve on# KUBELET_PORT="--port=10250"

You may leave this blank to use the actual hostname

location of the api-server# KUBELET_API_SERVER=""

Add your own!

先創建 RBAC

再創建 Calico Daemonset

等待 20s 拉取鏡像

創建 deployment

Deploys CoreDNS to a cluster currently running Kube-DNS.

執行上面的替換腳本

創建 CoreDNS

kind: ServiceAccountname: kube-dns-autoscalernamespace: kube-systemroleRef:kind: ClusterRolename: system:kube-dns-autoscalerapiGroup: rbac.authorization.k8s.io

Should keep target in sync with cluster/addons/dns/kube-dns.yaml.base

When cluster is using large nodes(with more cores), "coresPerReplica" should dominate.

If using small nodes, "nodesPerReplica" should dominate.

------------------- Dashboard Service -------------------

https://github.com/kubernetes/kubernetes/issues/42158# 這個問題還沒解決，PR 沒有合併被關閉了，可以關注一下上面這個 issue，被關閉的 PR 在下面# https://github.com/kubernetes/kubernetes/pull/49567

https://github.com/kubernetes/kubernetes/issues/59993# 這個似乎已經解決了，沒時間測試，PR 地址在下面，我大致 debug 一下 好像是 cAdvisor 的問題# https://github.com/opencontainers/runc/pull/1722

https://github.com/kubernetes/kubernetes/issues/58217# 注意: 這個問題現在仍未解決，可關注上面的 issue，這個問題可能影響 node image gc# 強烈依賴於 kubelet 做 宿主機 image gc 的需要注意一下

沒找到太多資料，不過感覺跟上面問題類似

地址默認爲 127.0.0.1:6443# 如果在 master 上啓用 kubelet 請在生成後的 kubeconfig 中修改該地址爲當前MASTER_IP:6443，master不做kubelet時，master上直接用127的地址即可

kind: ServiceAccount
name: kube-dns-autoscaler
namespace: kube-system
roleRef:
kind: ClusterRole
name: system:kube-dns-autoscaler
apiGroup: rbac.authorization.k8s.io

https://github.com/kubernetes/kubernetes/issues/59993# 這個似乎已經解決了，沒時間測試，PR 地址在下面，我大致 debug 一下好像是 cAdvisor 的問題# https://github.com/opencontainers/runc/pull/1722

https://github.com/kubernetes/kubernetes/issues/58217# 注意: 這個問題現在仍未解決，可關注上面的 issue，這個問題可能影響 node image gc# 強烈依賴於 kubelet 做宿主機 image gc 的需要注意一下