k8s 集羣部署問題整理1

1、hostname “master” could not be reached
在host中沒有加解析

2、curl -sSL http://localhost:10248/healthz
curl: (7) Failed connect to localhost:10248; 拒絕連接 在host中沒有localhost的解析

3、Error starting daemon: SELinux is not supported with the overlay2 graph driver on this kernel. Either boot into a newer kernel or…abled=false)
vim /etc/ssconfig/docker --selinux-enabled=False

4、bridge-nf-call-iptables 固化的問題：
#下面的是關於bridge的配置： net.bridge.bridge-nf-call-ip6tables = 0 net.bridge.bridge-nf-call-iptables = 1 #意味着二層的網絡在轉發包的時候會被iptables的forward規則過濾 net.bridge.bridge-nf-call-arptables = 0

5、The connection to the server localhost:8080 was refused - did you specify the right host or port?
unable to recognize “kube-flannel.yml”: Get http://localhost:8080/api?timeout=32s: dial tcp [::1]:8080: connect: connection refused 下面如果在root用戶下執行的，就不會報錯 mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config
###6、error: unable to recognize “mycronjob.yml”: no matches for kind “CronJob” in version “batch/v2alpha1”
去kube-apiserver.yaml文件中添加： - --runtime-config=batch/v2alpha1=true，然後重啓kubelet服務，就可以了

7、Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized Unable to update cni config: No networks found in /etc/cni/net.d Failed to get system container stats for “/system.slice/kubelet.service”: failed to get cgroup stats for “/system.slice/kubelet.service”: failed to get container info for “/system.slice/kubelet.service”: unknown container “/system.slice/kubelet.service”

docker pull quay.io/coreos/flannel:v0.10.0-amd64 
mkdir -p /etc/cni/net.d/
cat <<EOF> /etc/cni/net.d/10-flannel.conf
{"name":"cbr0","type":"flannel","delegate": {"isDefaultGateway": true}}
EOF
mkdir /usr/share/oci-umount/oci-umount.d -p
mkdir /run/flannel/
cat <<EOF> /run/flannel/subnet.env
FLANNEL_NETWORK=172.100.0.0/16
FLANNEL_SUBNET=172.100.1.0/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=true
EOF
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/v0.9.1/Documentation/kube-flannel.yml

8、Unable to connect to the server: x509: certificate signed by unknown authority (possibly because of “crypto/rsa: verification error” while trying to verify candidate authority certificate “kubernetes”)
export KUBECONFIG=/etc/kubernetes/kubelet.conf

9、Failed to get system container stats for “/system.slice/docker.service”: failed to get cgroup stats for “/system.slice/docker.service”: failed to get container info for “/system.slice/docker.service”: unknown container “/system.slice/docker.service”
vim /etc/sysconfig/kubelet --runtime-cgroups=/systemd/system.slice --kubelet-cgroups=/systemd/system.slice systemctl restart kubelet

大概意思是Flag --cgroup-driver --kubelet-cgroups 驅動已經被禁用，這個參數應該通過kubelet 的配置指定配置文件來配置

10、The HTTP call equal to ‘curl -sSL http://localhost:10255/healthz’ failed with error: Get http://localhost:10255/healthz: dial tcp 127.0.0.1:10255: getsockopt: connection refused.
vim /etc/systemd/system/kubelet.service.d/10-kubeadm.conf Environment=“KUBELET_SYSTEM_PODS_ARGS=–pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true --fail-swap-on=false”

###11、failed to run Kubelet: failed to create kubelet: miscon figuration: kubelet cgroup driver: “systemd” is different from docker cgroup driver: “cgroupfs”
kubelet： Environment=“KUBELET_CGROUP_ARGS=–cgroup-driver=systemd” docker：　 　vi /lib/systemd/system/docker.service -exec-opt native.cgroupdriver=systemd

12、[ERROR CRI]: unable to check if the container runtime at “/var/run/dockershim.sock” is running: exit status 1
rm -f /usr/bin/crictl

13、 Warning FailedScheduling 2s (x7 over 33s) default-scheduler 0/4 nodes are available: 4 node(s) didn’t match node selector.
如果指定的label在所有node上都無法匹配，則創建Pod失敗，會提示無法調度：

14、kubeadm 生成的token過期後，集羣增加節點

kubeadm token create

openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | 
openssl dgst -sha256 -hex | sed 's/^.* //'

kubeadm join --token aa78f6.8b4cafc8ed26c34f --discovery-token-ca-cert-hash sha256:0fd95a9bc67a7bf0ef42da968a0d55d92e52898ec37c971bd77ee501d845b538  172.16.6.79:6443 --skip-preflight-checks

15、### systemctl status kubelet告警
cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d
May 29 06:30:28 fnode kubelet[4136]: E0529 06:30:28.935309 4136 kubelet.go:2130] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
刪除 /etc/systemd/system/kubelet.service.d/10-kubeadm.conf 的 KUBELET_NETWORK_ARGS，然後重啓kubelet服務 臨時解決。沒啥用
根本原因是缺少： k8s.gcr.io/pause-amd64:3.1

16 刪除flannel網絡：

ifconfig cni0 down
ifconfig flannel.1 down
ifconfig del flannel.1
ifconfig del cni0

ip link del flannel.1
ip link del cni0

yum install bridge-utils
brctl delbr  flannel.1
brctl delbr cni0
rm -rf /var/lib/cni/flannel/* && rm -rf /var/lib/cni/networks/cbr0/* && ip link delete cni0 &&  rm -rf /var/lib/cni/network/cni0/*

17、E0906 15:10:55.415662 1 leaderelection.go:234] error retrieving resource lock default/ceph.com-rbd: endpoints “ceph.com-rbd” is forbidden: User “system:serviceaccount:default:rbd-provisioner” cannot get endpoints in the namespace “default”
`在 添加下面的這一段 （會重新申請資源） kubectl apply -f ceph/rbd/deploy/rbac/clusterrole.yaml

apiGroups: [""]
resources: [“endpoints”]
verbs: [“get”, “list”, “watch”, “create”, “update”, “patch”]`

18、flannel指定網卡設備：

• –iface=eth0

21、 Failed create pod sandbox: rpc error: code = Unknown desc = [failed to set up sandbox container “957541888b8a0e5b9ad65da932f688eb02cc182808e10d1a89a6e8db2132c253” network for pod “coredns-7655b945bc-6hgj9”: NetworkPlugin cni failed to set up pod “coredns-7655b945bc-6hgj9_kube-system” network: failed to find plugin “loopback” in path [/opt/cni/bin], failed to clean up sandbox container “957541888b8a0e5b9ad65da932f688eb02cc182808e10d1a89a6e8db2132c253” network for pod “coredns-7655b945bc-6hgj9”: NetworkPlugin cni failed to teardown pod “coredns-7655b945bc-6hgj9_kube-system” network: failed to find plugin “portmap” in path [/opt/cni/bin]]
https://kubernetes.io/docs/setup/independent/troubleshooting-kubeadm/#coredns-pods-have-crashloopbackoff-or-error-state
如果您的網絡提供商不支持portmap CNI插件，您可能需要使用服務的NodePort功能或使用HostNetwork=true。

22、問題：kubelet設置了system-reserved（800m）、kube-reserved(500m)、eviction-hard(800)，其實集羣實際可用的內存是總內存-800m-800m-500m ，但是發現還 是會觸發系統級別kill進程，
排查：使用top查看前幾名的內存使用情況，發現etcd服務使用了內存達到500M以上，kubelet使用內存200m，ceph使用內存總和是200多m，加起來就已經900m了，這些都是k8s之外的系統開銷，已經完全超出了系統預留內存，因此可能會觸發系統級別的kill，

23、如何訪問api-server？
使用kubectl proxy功能

24、使用svc的endpoint代理集羣外部服務，經常出現endpoint丟失的問題
解決：去掉service.spec.selecter 標籤就好了。

25、集羣雪崩的一次問題處理，node節點偶爾出現noreading狀態，
排查：此node節點上cpu使用率過高。

1、沒有觸發node節點上的cpuPressure的狀態，判斷出來不是k8s所管理的cpu佔用過高的問題，應該是system、kube組件預留的cpu高導致的。
2、查看cpu和mem的cgroup分組，發現kubelet，都在system.sliec下面，因此判斷kube預留資源沒有生效導致的。
3、
--enforce-node-allocatable=pods,kube-reserved,system-reserved  #採用硬限制，超出限制就oom
--system-reserved-cgroup=/system.slice  #指定系統reserved-cgroup對那些cgroup限制。
--kube-reserved-cgroup=/system.slice/kubelet.service #指定kube-reserved-cgroup對那些服務的cgroup進行限制
--system-reserved=memory=1Gi,cpu=500m  
--kube-reserved=memory=500Mi,cpu=500m,ephemeral-storage=10Gi 

26、[etcd] Checking Etcd cluster health
etcd cluster is not healthy: context deadline exceeded