Tungsten Fabric實戰：基於K8s的部署踩坑

原創 TungstenFabric 2020-04-14 10:36

Tungsten Fabric(原名opencontrail),提供了可以與編排器(openstack/k8s/vCenter)協同工作的controller，和部署在計算節點/node上的vRouter受其管控，替代原有的linux-bridge/ovs進行通信。
作者：劉敬一

前言

研究一款開源控制器，最好的方法就是先部署一套，怎麼方便怎麼來。

先去TF的GitHub，無論是tf-devstack還是tf-dev-env裏面的run.sh，全都卡住。

[setup contrail git sources]
INFO: source env from /root/contrail/.env/tf-developer-sandbox.env
INFO: current folder is
100  2584  100  2584    0     0    934      0  0:00:02  0:00:02 --:--:--   933
INFO: Download repo tool
  1. 找到微信公衆號TF中文社區(CTFSDN)，加微信，被拉入TF討論羣
  2. 經過羣裏的大佬吳sir和楊sir的指導，開始按照以下文章來部署

第一篇：部署準備與初始狀態
第二篇：創建虛擬網絡
第三篇：創建安全策略
第四篇：創建隔離命名空間

實操記錄

初始準備

  • 創建三臺CentOS7.7的虛擬機
deployer 192.168.122.160
master01 192.168.122.96  <---內存至少8G
node01 192.168.122.250

#cat /etc/redhat-release 
CentOS Linux release 7.7.1908 (Core)

基於aliyun的pip加速

  • 各個節點設置pip加速
mkdir .pip && tee ~/.pip/pip.conf <<-'EOF'
[global]
trusted-host =  mirrors.aliyun.com
index-url = https://mirrors.aliyun.com/pypi/simple
EOF

基於aliyun的docker鏡像加速

  • 網上教程很多，下面的加速地址用**隱去
sudo mkdir -p /etc/docker
sudo tee /etc/docker/daemon.json <<-'EOF'
{
  "registry-mirrors": ["https://********.mirror.aliyuncs.com"]
}
EOF
sudo systemctl daemon-reload
sudo systemctl restart docker

一些源文件

  • 很多需要的安裝文件被放到了http://35.220.208.0/ 這個服務器上，可以根據實際鏈接來下發命令
mkdir pkg_python/
cd pkg_python/
wget http://35.220.208.0/packages_python/pip-19.3.1.tar.gz
easy_install pip-19.3.1.tar.gz
easy_install --upgrade --dry-run pip
wget http://35.220.208.0/packages_python/docker_compose-1.24.1-py2.py3-none-any.whl
pip2 install docker_compose-1.24.1-py2.py3-none-any.whl

mkdir /root/pkg_k8s
cd /root/pkg_k8s
wget http://35.220.208.0/k8s_v1.12.9/packages/auto_download.sh
chmod +x auto_download.sh
./auto_download.sh
  • 遇到下面的錯誤，但是貌似沒有什麼影響
[root@localhost pkg_python]# easy_install --upgrade --dry-run pip
Searching for pip
Reading https://pypi.python.org/simple/pip/
Best match: pip 20.0.2
Downloading https://files.pythonhosted.org/packages/8e/76/66066b7bc71817238924c7e4b448abdb17eb0c92d645769c223f9ace478f/pip-20.0.2.tar.gz#sha256=7db0c8ea4c7ea51c8049640e8e6e7fde949de672bfa4949920675563a5a6967f
Processing pip-20.0.2.tar.gz
Writing /tmp/easy_install-bm8Ztx/pip-20.0.2/setup.cfg
Running pip-20.0.2/setup.py -n -q bdist_egg --dist-dir /tmp/easy_install-bm8Ztx/pip-20.0.2/egg-dist-tmp-32s9sn
/usr/lib64/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'project_urls'
  warnings.warn(msg)
/usr/lib64/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'python_requires'
  warnings.warn(msg)
warning: no files found matching 'docs/docutils.conf'
warning: no previously-included files found matching '.coveragerc'
warning: no previously-included files found matching '.mailmap'
warning: no previously-included files found matching '.appveyor.yml'
warning: no previously-included files found matching '.travis.yml'
warning: no previously-included files found matching '.readthedocs.yml'
warning: no previously-included files found matching '.pre-commit-config.yaml'
warning: no previously-included files found matching 'tox.ini'
warning: no previously-included files found matching 'noxfile.py'
warning: no files found matching 'Makefile' under directory 'docs'
warning: no files found matching '*.bat' under directory 'docs'
warning: no previously-included files found matching 'src/pip/_vendor/six'
warning: no previously-included files found matching 'src/pip/_vendor/six/moves'
warning: no previously-included files matching '*.pyi' found under directory 'src/pip/_vendor'
no previously-included directories found matching '.github'
no previously-included directories found matching '.azure-pipelines'
no previously-included directories found matching 'docs/build'
no previously-included directories found matching 'news'
no previously-included directories found matching 'tasks'
no previously-included directories found matching 'tests'
no previously-included directories found matching 'tools'
warning: install_lib: 'build/lib' does not exist -- no Python modules to install

[root@localhost pkg_python]#

本地registry

  • 本地運行registry容器，宿主機的80端口映射爲容器的5000端口
[root@deployer ~]# docker run -d -p 80:5000 --restart=always --name registry registry:2
0c17a03ebdffe3cea98d7cec42c268c1117241f236f9f2443bbb1b77d34b0082
[root@deployer ~]# 
[root@deployer ~]# docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                  NAMES
0c17a03ebdff        registry:2          "/entrypoint.sh /etc…"   About an hour ago   Up About an hour    0.0.0.0:80->5000/tcp   registry
[root@deployer ~]#

設置yaml文件

  • 獲取到contrail-ansible-deployer之後，進入文件夾，修改instances.yaml
[root@deployer inventory]# vim  ../config/instances.yaml

provider_config:
  bms:
   ssh_pwd: Password
   ssh_user: root
   ssh_public_key: /root/.ssh/id_rsa.pub
   ssh_private_key: /root/.ssh/id_rsa
   domainsuffix: local
instances:
  bms1:
    provider: bms
    roles:
      config_database:
      config:
      control:
      analytics_database:
      analytics:
      webui:
      k8s_master:
      kubemanager:
    ip: 192.168.122.96
  bms2:
    provider: bms
    roles:
      vrouter:
      k8s_node:
    ip: 192.168.122.250
global_configuration:
  CONTAINER_REGISTRY: hub.juniper.net
contrail_configuration:
  CONTRAIL_VERSION: 1912-latest
  • CONTAINER_REGISTRY替換爲本地registry，contrail的版本設置爲1912-last與後面拉取鏡像retag保持一致

設置免密登錄

  • 需要設置從developer不輸入密碼就能登錄本機/master01/node01
#ssh-keygen -t rsa

#ssh-copy-id -i ~/.ssh/id_rsa.pub root@master01
#ssh-copy-id -i ~/.ssh/id_rsa.pub root@node01
#ssh-copy-id -i ~/.ssh/id_rsa.pub root@node02

ansible

  • deployer上執行ansible會有報錯
/usr/lib/python2.7/site-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.24.3) or chardet (2.2.1) doesn't match a supported version!
  RequestsDependencyWarning)

解決方法是

pip uninstall urllib3    
pip uninstall chardet
pip install requests

拉取鏡像

  • k8s的鏡像還好，有aliyun加速
  • contrail的源hub.juniper.net是需要Juniper的賬號，這個需要替換爲opencontrailnightly
  • 楊sir提供了腳本進行拉取和推送到本地registry，後續master/node就可以直接從deployer的registry拉取了
  • 如果是用最新的contrail-ansible-deployer代碼，還需要加上一個鏡像:contrail-provisioner
  • 但是執行之前，需要先將本地IP設置爲insecure-registry，就可以基於http而不是https下載了
  • 一種解決方法就是修改/etc/docker/daemon.json(如果沒有就自己加)
[root@node01 ~]# cat /etc/docker/daemon.json 
{
  "insecure-registries": [ "hub.juniper.net","k8s.gcr.io" ]
}
[root@node01 ~]#

然後

[root@deployer ~]# systemctl daemon-reload
[root@deployer ~]# systemctl restart docker
  • 腳本如下,已經修改爲deployer的IP
#準備Kubernetes離線鏡像，運行如下腳本

#!/bin/bash
#Author: Alex Yang <[email protected]>

set -e

REPOSITORIE="gcr.azk8s.cn/google_containers"
LOCAL_REPO="192.168.122.160"
IMAGES="kube-proxy:v1.12.9 kube-controller-manager:v1.12.9 kube-scheduler:v1.12.9 kube-apiserver:v1.12.9 coredns:1.2.2 coredns:1.2.6 pause:3.1 etcd:3.2.24 kubernetes-dashboard-amd64:v1.8.3"

for img in $IMAGES
do
  echo "===Pulling image: "$img
  docker pull $REPOSITORIE/$img
  echo "===Retag image ["$img"]"
  docker tag $REPOSITORIE/$img $LOCAL_REPO/$img
  echo "===Pushing image: "$LOCAL_REPO/$img
  docker push $LOCAL_REPO/$img
  docker rmi $REPOSITORIE/$img
done

#準備TungstenFabric離線鏡像，運行如下腳本

#!/bin/bash
#Author: Alex Yang <[email protected]>

set -e

REGISTRY_URL=opencontrailnightly
LOCAL_REGISTRY_URL=192.168.122.160
IMAGE_TAG=1912-latest
COMMON_IMAGES="contrail-node-init contrail-status contrail-nodemgr contrail-external-cassandra contrail-external-zookeeper contrail-external-kafka contrail-external-redis contrail-external-rabbitmq contrail-external-rsyslogd"
ANALYTICS_IMAGES="contrail-analytics-query-engine contrail-analytics-api contrail-analytics-collector contrail-analytics-snmp-collector contrail-analytics-snmp-topology contrail-analytics-alarm-gen"
CONTROL_IMAGES="contrail-controller-control-control contrail-controller-control-dns contrail-controller-control-named contrail-controller-config-api contrail-controller-config-devicemgr contrail-controller-config-schema contrail-controller-config-svcmonitor contrail-controller-config-stats contrail-controller-config-dnsmasq"
WEBUI_IMAGES="contrail-controller-webui-job contrail-controller-webui-web"
K8S_IMAGES="contrail-kubernetes-kube-manager contrail-kubernetes-cni-init"
VROUTER_IMAGES="contrail-vrouter-kernel-init contrail-vrouter-agent"

IMAGES=$COMMON_IMAGES" "$ANALYTICS_IMAGES" "$CONTROL_IMAGES" "$WEBUI_IMAGES" "$K8S_IMAGES" "$VROUTER_IMAGES

for image in $IMAGES
do
  echo "===Pulling image: "$image
  docker pull $REGISTRY_URL/$image:$IMAGE_TAG
  echo "===Retag image ["$image"]"
  docker tag $REGISTRY_URL/$image:$IMAGE_TAG $LOCAL_REGISTRY_URL/$image:$IMAGE_TAG
  echo "===Pushing image: "$LOCAL_REGISTRY_URL/$image:$IMAGE_TAG
  docker push $LOCAL_REGISTRY_URL/$image:$IMAGE_TAG
  docker rmi $REGISTRY_URL/$image:$IMAGE_TAG
done
  • 查看鏡像列表
[root@deployer ~]# docker image list


REPOSITORY                                              TAG                 IMAGE ID            CREATED             SIZE
ubuntu                                                  latest              72300a873c2c        3 weeks ago         64.2MB
registry                                                2                   708bc6af7e5e        7 weeks ago         25.8MB
registry                                                latest              708bc6af7e5e        7 weeks ago         25.8MB
192.168.122.160/contrail-vrouter-kernel-init            1912-latest         92e9cce315a5        3 months ago        581MB
192.168.122.160/contrail-vrouter-agent                  1912-latest         e8d9457d740e        3 months ago        729MB
192.168.122.160/contrail-status                         1912-latest         d2264c6741a5        3 months ago        513MB
192.168.122.160/contrail-nodemgr                        1912-latest         c3428aa7e9b7        3 months ago        523MB
192.168.122.160/contrail-node-init                      1912-latest         c846ff071cc8        3 months ago        506MB
192.168.122.160/contrail-kubernetes-kube-manager        1912-latest         983a6307731b        3 months ago        517MB
192.168.122.160/contrail-kubernetes-cni-init            1912-latest         45c88538c834        3 months ago        525MB
192.168.122.160/contrail-external-zookeeper             1912-latest         6937c72b866c        3 months ago        290MB
192.168.122.160/contrail-external-rsyslogd              1912-latest         812ba27a4e08        3 months ago        304MB
192.168.122.160/contrail-external-redis                 1912-latest         3dc79f0b6eb9        3 months ago        129MB
192.168.122.160/contrail-external-rabbitmq              1912-latest         a98ac91667b2        3 months ago        256MB
192.168.122.160/contrail-external-kafka                 1912-latest         7b5a2ce6a656        3 months ago        665MB
192.168.122.160/contrail-external-cassandra             1912-latest         20109c39696c        3 months ago        545MB
192.168.122.160/contrail-controller-webui-web           1912-latest         44054aa131c5        3 months ago        552MB
192.168.122.160/contrail-controller-webui-job           1912-latest         946e2bbd7451        3 months ago        552MB
192.168.122.160/contrail-controller-control-named       1912-latest         81ef8223a519        3 months ago        575MB
192.168.122.160/contrail-controller-control-dns         1912-latest         15c1ce0cf26e        3 months ago        575MB
192.168.122.160/contrail-controller-control-control     1912-latest         ec195cc75705        3 months ago        594MB
192.168.122.160/contrail-controller-config-svcmonitor   1912-latest         3d53781422be        3 months ago        673MB
192.168.122.160/contrail-controller-config-stats        1912-latest         46bc77cf1c87        3 months ago        506MB
192.168.122.160/contrail-controller-config-schema       1912-latest         75acb8ed961f        3 months ago        673MB
192.168.122.160/contrail-controller-config-dnsmasq      1912-latest         dc2980441d51        3 months ago        506MB
192.168.122.160/contrail-controller-config-devicemgr    1912-latest         c08868a27a0a        3 months ago        772MB
192.168.122.160/contrail-controller-config-api          1912-latest         f39ca251b475        3 months ago        706MB
192.168.122.160/contrail-analytics-snmp-topology        1912-latest         5ee37cbbd034        3 months ago        588MB
192.168.122.160/contrail-analytics-snmp-collector       1912-latest         29ae502fb74f        3 months ago        588MB
192.168.122.160/contrail-analytics-query-engine         1912-latest         b5f937d6b6e3        3 months ago        588MB
192.168.122.160/contrail-analytics-collector            1912-latest         ee1bdbcc460a        3 months ago        588MB
192.168.122.160/contrail-analytics-api                  1912-latest         ac5c8f7cef89        3 months ago        588MB
192.168.122.160/contrail-analytics-alarm-gen            1912-latest         e155b24a0735        3 months ago        588MB
192.168.10.10/kube-proxy                                v1.12.9             295526df163c        9 months ago        95.7MB
192.168.122.160/kube-proxy                              v1.12.9             295526df163c        9 months ago        95.7MB
192.168.122.160/kube-controller-manager                 v1.12.9             f473e8452c8e        9 months ago        164MB
192.168.122.160/kube-apiserver                          v1.12.9             8ea704c2d4a7        9 months ago        194MB
192.168.122.160/kube-scheduler                          v1.12.9             c79506ccc1bc        9 months ago        58.4MB
192.168.122.160/coredns                                 1.2.6               f59dcacceff4        16 months ago       40MB
192.168.122.160/etcd                                    3.2.24              3cab8e1b9802        18 months ago       220MB
192.168.122.160/coredns                                 1.2.2               367cdc8433a4        18 months ago       39.2MB
192.168.122.160/kubernetes-dashboard-amd64              v1.8.3              0c60bcf89900        2 years ago         102MB
192.168.122.160/pause                                   3.1                 da86e6ba6ca1        2 years ago         742kB
[root@deployer ~]#
  • 查看本地倉庫中的image
[root@deployer ~]# curl -X 


GET http://localhost/v2/_catalog | python -m json.tool
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1080  100  1080    0     0  18298      0 --:--:-- --:--:-- --:--:-- 18620
{
    "repositories": [
        "contrail-analytics-alarm-gen",
        "contrail-analytics-api",
        "contrail-analytics-collector",
        "contrail-analytics-query-engine",
        "contrail-analytics-snmp-collector",
        "contrail-analytics-snmp-topology",
        "contrail-controller-config-api",
        "contrail-controller-config-devicemgr",
        "contrail-controller-config-dnsmasq",
        "contrail-controller-config-schema",
        "contrail-controller-config-stats",
        "contrail-controller-config-svcmonitor",
        "contrail-controller-control-control",
        "contrail-controller-control-dns",
        "contrail-controller-control-named",
        "contrail-controller-webui-job",
        "contrail-controller-webui-web",
        "contrail-external-cassandra",
        "contrail-external-kafka",
        "contrail-external-rabbitmq",
        "contrail-external-redis",
        "contrail-external-rsyslogd",
        "contrail-external-zookeeper",
        "contrail-kubernetes-cni-init",
        "contrail-kubernetes-kube-manager",
        "contrail-node-init",
        "contrail-nodemgr",
        "contrail-status",
        "contrail-vrouter-agent",
        "contrail-vrouter-kernel-init",
        "coredns",
        "etcd",
        "kube-apiserver",
        "kube-controller-manager",
        "kube-proxy",
        "kube-scheduler",
        "kubernetes-dashboard-amd64",
        "pause"
    ]
}
[root@deployer ~]#
  • 至於master01和node01，就可以直接從developer上拉取k8s/contrail的鏡像了，速度槓槓的！(別忘了–insecure-registry=192.168.122.160)
#準備Kubernetes離線鏡像，運行如下腳本
#!/bin/bash
#Author: Alex Yang <[email protected]>

set -e

REPOSITORIE="k8s.gcr.io"
LOCAL_REPO="192.168.122.160"
IMAGES="kube-proxy:v1.12.9 kube-controller-manager:v1.12.9 kube-scheduler:v1.12.9 kube-apiserver:v1.12.9 coredns:1.2.2 coredns:1.2.6 pause:3.1 etcd:3.2.24 kubernetes-dashboard-amd64:v1.8.3"

for img in $IMAGES
do
  echo "===Pulling image: "$img
  docker pull $LOCAL_REPO/$img
  echo "===Retag image ["$img"]"
  docker tag $LOCAL_REPO/$img $REPOSITORIE/$img
  docker rmi $LOCAL_REPO/$img
done

#準備TungstenFabric離線鏡像，運行如下腳本

#!/bin/bash
#Author: Alex Yang <[email protected]>

set -e

REPOSITORIE=hub.juniper.net
LOCAL_REPO="192.168.122.160"
IMAGE_TAG=1912-latest
COMMON_IMAGES="contrail-node-init contrail-status contrail-nodemgr contrail-external-cassandra contrail-external-zookeeper contrail-external-kafka contrail-external-redis contrail-external-rabbitmq contrail-external-rsyslogd"
ANALYTICS_IMAGES="contrail-analytics-query-engine contrail-analytics-api contrail-analytics-collector contrail-analytics-snmp-collector contrail-analytics-snmp-topology contrail-analytics-alarm-gen"
CONTROL_IMAGES="contrail-controller-control-control contrail-controller-control-dns contrail-controller-control-named contrail-controller-config-api contrail-controller-config-devicemgr contrail-controller-config-schema contrail-controller-config-svcmonitor contrail-controller-config-stats contrail-controller-config-dnsmasq"
WEBUI_IMAGES="contrail-controller-webui-job contrail-controller-webui-web"
K8S_IMAGES="contrail-kubernetes-kube-manager contrail-kubernetes-cni-init"
VROUTER_IMAGES="contrail-vrouter-kernel-init contrail-vrouter-agent"

IMAGES=$COMMON_IMAGES" "$ANALYTICS_IMAGES" "$CONTROL_IMAGES" "$WEBUI_IMAGES" "$K8S_IMAGES" "$VROUTER_IMAGES

for img in $IMAGES
do
  echo "===Pulling image: "$img
  docker pull $LOCAL_REPO/$img:$IMAGE_TAG
  echo "===Retag image ["$img"]"
  docker tag $LOCAL_REPO/$img:$IMAGE_TAG $REPOSITORIE/$img:$IMAGE_TAG
  docker rmi $LOCAL_REPO/$img:$IMAGE_TAG
done

打開web

  • developer上執行過
ansible-playbook -e orchestrator=kubernetes -i inventory/ playbooks/install_k8s.yml
ansible-playbook -e orchestrator=kubernetes -i inventory/ playbooks/install_contrail.yml
  • web訪問master01的8143端口，默認進入的是monitor頁面
    在這裏插入圖片描述
  • 用戶名/密碼:admin/contrail123,domain不需要填，總算看到WebUI了
    在這裏插入圖片描述
  • 可以切換到config頁面
    在這裏插入圖片描述
    k8s狀態
  • node
[root@master01 ~]# kubectl get nodes
NAME       STATUS   ROLES    AGE    VERSION
master01   Ready    master   6h4m   v1.12.9
node01     Ready    <none>   6h3m   v1.12.9
[root@master01 ~]# 
[root@master01 ~]# kubectl get namespaces
NAME          STATUS   AGE
contrail      Active   80m
default       Active   6h20m
kube-public   Active   6h20m
kube-system   Active   6h20m
[root@master01 ~]#
  • pods
[root@master01 ~]# kubectl get pods -n kube-system 
NAME                                    READY   STATUS             RESTARTS   AGE
coredns-85c98899b4-4dzzx                0/1     ImagePullBackOff   0          6h2m
coredns-85c98899b4-w4bcs                0/1     ImagePullBackOff   0          6h2m
etcd-master01                           1/1     Running            5          28m
kube-apiserver-master01                 1/1     Running            4          28m
kube-controller-manager-master01        1/1     Running            5          28m
kube-proxy-dmmlh                        1/1     Running            5          6h2m
kube-proxy-ph9gx                        1/1     Running            1          6h2m
kube-scheduler-master01                 1/1     Running            5          28m
kubernetes-dashboard-76456c6d4b-x5lz4   0/1     ImagePullBackOff   0          6h2m

繼續排障

node01無法使用kubectrl命令

  • 問題如下
[root@node01 ~]# kubectl get pods -n kube-system -o wide
The connection to the server localhost:8080 was refused - did you specify the right host or port?
[root@node01 ~]# scp [email protected]:/etc/kubernetes/admin.conf /etc/kubernetes/admin.conf
[root@node01 ~]# echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> ~/.bash_profile
[root@node01 ~]# source ~/.bash_profile
[root@node01 ~]# kubectl get pods -n kube-system -o wide
NAME                                    READY   STATUS             RESTARTS   AGE     IP                NODE       NOMINATED NODE
coredns-85c98899b4-4dzzx                0/1     ImagePullBackOff   0          5h45m   10.47.255.252     node01     <none>
coredns-85c98899b4-w4bcs                0/1     ImagePullBackOff   0          5h45m   10.47.255.251     node01     <none>
etcd-master01                           1/1     Running            3          11m     192.168.122.96    master01   <none>
kube-apiserver-master01                 1/1     Running            3          11m     192.168.122.96    master01   <none>
kube-controller-manager-master01        1/1     Running            3          11m     192.168.122.96    master01   <none>
kube-proxy-dmmlh                        1/1     Running            3          5h45m   192.168.122.96    master01   <none>
kube-proxy-ph9gx                        1/1     Running            1          5h44m   192.168.122.250   node01     <none>
kube-scheduler-master01                 1/1     Running            3          11m     192.168.122.96    master01   <none>
kubernetes-dashboard-76456c6d4b-x5lz4   0/1     ImagePullBackOff   0          5h44m   192.168.122.250   node01     <none>
[root@node01 ~]#

ImagePullBackOff 的問題

  • 先看一下coredns的pod描述
[root@master01 ~]# kubectl describe pod coredns-85c98899b4-4dzzx -n kube-system
Name:               coredns-85c98899b4-4dzzx
Namespace:          kube-system
...
Events:
  Type     Reason                  Age                    From               Message
  ----     ------                  ----                   ----               -------
  Warning  FailedScheduling        75m (x281 over 4h40m)  default-scheduler  0/2 nodes are available: 2 node(s) had taints that the pod didn't tolerate.
  Warning  FailedCreatePodSandBox  71m                    kubelet, node01    Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "1af3fb24d906d5f82ad3bdcf6d65be328302d3c596e63fc79ed0c134390b4753" network for pod "coredns-85c98899b4-4dzzx": NetworkPlugin cni failed to set up pod "coredns-85c98899b4-4dzzx_kube-system" network: Failed in Poll VM-CFG. Error : Failed in PollVM. Error : Failed HTTP Get operation. Return code 404
  Normal   SandboxChanged          70m (x3 over 71m)      kubelet, node01    Pod sandbox changed, it will be killed and re-created.
  Normal   Pulling                 70m (x3 over 70m)      kubelet, node01    pulling image "k8s.gcr.io/coredns:1.2.6"
  Warning  Failed                  70m (x3 over 70m)      kubelet, node01    Failed to pull image "k8s.gcr.io/coredns:1.2.6": rpc error: code = Unknown desc = Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp 192.168.122.160:443: getsockopt: no route to host
  Warning  Failed                  70m (x3 over 70m)      kubelet, node01    Error: ErrImagePull
  Warning  Failed                  6m52s (x282 over 70m)  kubelet, node01    Error: ImagePullBackOff
  Normal   BackOff                 103s (x305 over 70m)   kubelet, node01    Back-off pulling image "k8s.gcr.io/coredns:1.2.6"
[root@master01 ~]#
  • 看來是啓動pod的時候，insecure-registry還沒有設置，強制重啓pod
[root@master01 ~]# kubectl get pod coredns-85c98899b4-4dzzx -n kube-system -o yaml | kubectl replace --force -f -
pod "coredns-85c98899b4-4dzzx" deleted
pod/coredns-85c98899b4-4dzzx replaced
[root@master01 ~]#
  • 發現還沒有up，繼續查看
[root@master01 ~]# kubectl describe pod coredns-85c98899b4-4dzzx -n kube-system
Events:
  Type     Reason                  Age                   From               Message
  ----     ------                  ----                  ----               -------
  Normal   Scheduled               6m29s                 default-scheduler  Successfully assigned kube-system/coredns-85c98899b4-fnpd7 to master01
  Warning  FailedCreatePodSandBox  6m26s                 kubelet, master01  Failed create pod sandbox: rpc error: code = Unknown desc = [failed to set up sandbox container "3074c719934789cef519eeae16d2eca4e272fb6bda1b157cee1dbdf2f597a59f" network for pod "coredns-85c98899b4-fnpd7": NetworkPlugin cni failed to set up pod "coredns-85c98899b4-fnpd7_kube-system" network: failed to find plugin "contrail-k8s-cni" in path [/opt/cni/bin], failed to clean up sandbox container "3074c719934789cef519eeae16d2eca4e272fb6bda1b157cee1dbdf2f597a59f" network for pod "coredns-85c98899b4-fnpd7": NetworkPlugin cni failed to teardown pod "coredns-85c98899b4-fnpd7_kube-system" network: failed to find plugin "contrail-k8s-cni" in path [/opt/cni/bin]]
  Normal   SandboxChanged          76s (x25 over 6m25s)  kubelet, master01  Pod sandbox changed, it will be killed and re-created.
  • 缺少contrail-k8s-cni,從node01複製一個過來
[root@master01 ~]# scp [email protected]:opt/cni/bin/contrail-k8s-cni /opt/cni/bin/
  • 再重建
[root@master01 ~]# kubectl get pod coredns-85c98899b4-fnpd7 -n kube-system -o yaml | kubectl replace --force -f -
pod "coredns-85c98899b4-fnpd7" deleted
pod/coredns-85c98899b4-fnpd7 replaced
[root@master01 ~]#
  • 可惜重啓之後還是有報錯
Events:
  Type     Reason                  Age                  From               Message
  ----     ------                  ----                 ----               -------
  Normal   Scheduled               18m                  default-scheduler  Successfully assigned kube-system/coredns-85c98899b4-8zq9h to master01
  Warning  FailedCreatePodSandBox  17m                  kubelet, master01  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "ffe9745c42750850e44035ee6413bf573148759738fc6131ce970537e03a5d13" network for pod "coredns-85c98899b4-8zq9h": NetworkPlugin cni failed to set up pod "coredns-85c98899b4-8zq9h_kube-system" network: Failed in Poll VM-CFG. Error : Failed in PollVM. Error : Get http://127.0.0.1:9091/vm-cfg/9bf51269-675b-11ea-ac43-525400c1ec4f: dial tcp 127.0.0.1:9091: connect: connection refused

隔天kebectl的命令都不能用了

  • 無論是在master01上還是在node01上
[root@master01 ~]# kubectl get nodes
The connection to the server 192.168.122.96:6443 was refused - did you specify the right host or port?
[root@master01 ~]#

多次重啓kubelet沒有用，雖然運行但是有報錯

[root@master01 ~]# journalctl -xe -u kubelet
3月 17 21:57:15 master01 kubelet[28722]: E0317 21:57:15.336303   28722 kubelet.go:2236] node "master01" not found
3月 17 21:57:15 master01 kubelet[28722]: E0317 21:57:15.425393   28722 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Node: Get https://192.168.122.96:6443/api/v1/nodes?fieldSelector=metadata.name%3Dma
3月 17 21:57:15 master01 kubelet[28722]: E0317 21:57:15.426388   28722 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:442: Failed to list *v1.Service: Get https://192.168.122.96:6443/api/v1/services?limit=500&resourceVersion=
3月 17 21:57:15 master01 kubelet[28722]: E0317 21:57:15.436468   28722 kubelet.go:2236] node "master01" not found
3月 17 21:57:15 master01 kubelet[28722]: E0317 21:57:15.536632   28722 kubelet.go:2236] node "master01" not found
3月 17 21:57:15 master01 kubelet[28722]: E0317 21:57:15.636848   28722 kubelet.go:2236] node "master01" not found
3月 17 21:57:15 master01 kubelet[28722]: E0317 21:57:15.636961   28722 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://192.168.122.96:6443/api/v1/pods?fieldSelector=spec.nodeNam
3月 17 21:57:15 master01 kubelet[28722]: E0317 21:57:15.737070   28722 kubelet.go:2236] node "master01" not found
3月 17 21:57:15 master01 kubelet[28722]: E0317 21:57:15.837781   28722 kubelet.go:2236] node "master01" not found
  • 搜索發現有很多人也遇到了這個問題
  • 據說可能是kube-apiserver沒有啓動造成的，但是當前環境無法啓動kube-apiserver
[root@master01 ~]# systemctl start kube-apiserver
Failed to start kube-apiserver.service: Unit not found.
[root@master01 ~]#

調用北向接口

  • 參考文檔戳這裏
  • 例如最簡單的獲取virtual-networks列表(使用最簡單用戶名/密碼認證方法)
[root@master01 ~]# curl -X GET -u "admin:contrail123" -H "Content-Type: application/json; charset=UTF-8" http://192.168.122.96:8082/virtual-networks
{"virtual-networks": [{"href": "http://192.168.122.96:8082/virtual-network/99c4144d-a7b7-4fb1-833e-887f21144320", "fq_name": ["default-domain", "default-project", "default-virtual-network"], "uuid": "99c4144d-a7b7-4fb1-833e-887f21144320"}, {"href": "http://192.168.122.96:8082/virtual-network/6e90abe8-91b6-48ad-99d2-fba6c9e29de4", "fq_name": ["default-domain", "k8s-default", "k8s-default-service-network"], "uuid": "6e90abe8-91b6-48ad-99d2-fba6c9e29de4"}, {"href": "http://192.168.122.96:8082/virtual-network/ab12e6dc-be52-407d-8f1d-37e6d29df0b1", "fq_name": ["default-domain", "default-project", "ip-fabric"], "uuid": "ab12e6dc-be52-407d-8f1d-37e6d29df0b1"}, {"href": "http://192.168.122.96:8082/virtual-network/915156f1-cec3-44eb-b15e-742452084d67", "fq_name": ["default-domain", "k8s-default", "k8s-default-pod-network"], "uuid": "915156f1-cec3-44eb-b15e-742452084d67"}, {"href": "http://192.168.122.96:8082/virtual-network/64a648ee-3ba6-4348-a543-07de6f225486", "fq_name": ["default-domain", "default-project", "dci-network"], "uuid": "64a648ee-3ba6-4348-a543-07de6f225486"}, {"href": "http://192.168.122.96:8082/virtual-network/82890bf9-a8e5-4c85-a32c-e307d9447a0a", "fq_name": ["default-domain", "default-project", "__link_local__"], "uuid": "82890bf9-a8e5-4c85-a32c-e307d9447a0a"}]}[root@master01 ~]# 
[root@master01 ~]#

重新部署

  • 下定決心，重新部署1-master/2-node的k8s場景，還是使用之前的deployer
  • 記錄
[root@deployer contrail-ansible-deployer]# cat install_k8s_3node.log 
...
PLAY RECAP **********************************************************************************************************************************************************************************************************************************
192.168.122.116            : ok=31   changed=15   unreachable=0    failed=0   
192.168.122.146            : ok=23   changed=8    unreachable=0    failed=0   
192.168.122.204            : ok=23   changed=8    unreachable=0    failed=0   
localhost                  : ok=62   changed=4    unreachable=0    failed=0  

[root@deployer contrail-ansible-deployer]# cat install_contrail_3node.log
...
PLAY RECAP **********************************************************************************************************************************************************************************************************************************
192.168.122.116            : ok=76   changed=45   unreachable=0    failed=0   
192.168.122.146            : ok=37   changed=17   unreachable=0    failed=0   
192.168.122.204            : ok=37   changed=17   unreachable=0    failed=0   
localhost                  : ok=66   changed=4    unreachable=0    failed=0

發現新的master的狀態是NotReady,查看狀態

[root@master02 ~]# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: active (running) since 三 2020-03-18 16:04:35 +08; 32min ago
     Docs: https://kubernetes.io/docs/
 Main PID: 18801 (kubelet)
    Tasks: 20
   Memory: 60.3M
   CGroup: /system.slice/kubelet.service
           └─18801 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=cgroupfs --network-plugin=cni

3月 18 16:36:51 master02 kubelet[18801]: W0318 16:36:51.929447   18801 cni.go:188] Unable to update cni config: No networks found in /etc/cni/net.d
3月 18 16:36:51 master02 kubelet[18801]: E0318 16:36:51.929572   18801 kubelet.go:2167] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready...fig uninitialized
3月 18 16:36:56 master02 kubelet[18801]: W0318 16:36:56.930736   18801 cni.go:188] Unable to update cni config: No networks found in /etc/cni/net.d
  • 發現master上確實沒有 /etc/cni/net.d這個目錄，所以將node02的拷貝過來
[root@master02 ~]# mkdir -p /etc/cni/net.d/
[root@master02 ~]# scp [email protected]:/etc/cni/net.d/10-contrail.conf /etc/cni/net.d/10-contrail.conf

[root@master02 ~]# systemctl restart kubelet

問題解決

[root@master02 ~]# kubectl get node
NAME                    STATUS   ROLES    AGE   VERSION
localhost.localdomain   Ready    <none>   35m   v1.12.9
master02                Ready    master   35m   v1.12.9
node03                  Ready    <none>   35m   v1.12.9
[root@master02 ~]# 
[root@master02 ~]# kubectl get pods -n kube-system -o wide
NAME                                    READY   STATUS    RESTARTS   AGE   IP                NODE                    NOMINATED NODE
coredns-85c98899b4-4vgk4                1/1     Running   0          69m   10.47.255.252     node03                  <none>
coredns-85c98899b4-thpz6                1/1     Running   0          69m   10.47.255.251     localhost.localdomain   <none>
etcd-master02                           1/1     Running   0          55m   192.168.122.116   master02                <none>
kube-apiserver-master02                 1/1     Running   0          55m   192.168.122.116   master02                <none>
kube-controller-manager-master02        1/1     Running   0          55m   192.168.122.116   master02                <none>
kube-proxy-6sp2n                        1/1     Running   0          69m   192.168.122.116   master02                <none>
kube-proxy-8gpgd                        1/1     Running   0          69m   192.168.122.204   node03                  <none>
kube-proxy-wtvhd                        1/1     Running   0          69m   192.168.122.146   localhost.localdomain   <none>
kube-scheduler-master02                 1/1     Running   0          55m   192.168.122.116   master02                <none>
kubernetes-dashboard-76456c6d4b-9s6vc   1/1     Running   0          69m   192.168.122.204   node03                  <none>
[root@master02 ~]#

在這裏插入圖片描述在這裏插入圖片描述

發表評論
所有評論
還沒有人評論，想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章

揭祕LOL背後的IT基礎架構丨SDN解鎖新基礎架構

歡迎來到Tungsten Fabric用戶案例系列文章，一起發現TF的更多應用場景。“揭祕LOL”系列的主人公是Tungsten Fabric用戶Riot Games遊戲公司，作爲LOL《英雄聯盟》的開發和運營商，Riot Ga

TungstenFabric 2020-07-03 08:19:33

揭祕LOL背後的IT基礎架構丨產品而非服務

歡迎來到Tungsten Fabric用戶案例系列文章，一起發現TF的更多應用場景。“揭祕LOL”系列的主人公是Tungsten Fabric用戶Riot Games遊戲公司，作爲LOL《英雄聯盟》的開發和運營商，Riot Ga

TungstenFabric 2020-07-03 08:19:33

揭祕LOL背後的IT基礎架構丨基礎設施即代碼

歡迎來到Tungsten Fabric用戶案例系列文章，一起發現TF的更多應用場景。“揭祕LOL”系列的主人公是Tungsten Fabric用戶Riot Games遊戲公司，作爲LOL《英雄聯盟》的開發和運營商，Riot Ga

TungstenFabric 2020-07-03 08:19:32

TF Live丨KK/建勳：多雲、SDN，還有網工進化論

一場疫情，改變了所有人的生產與生活，在時代的不確定性面前，我們應該怎麼做？ 昨晚，第一場【TF Live】如約而至，活動持續了兩個多小時，在線人數將近1000人，互動討論熱鬧十足。TF中文社區技術代表、瞻博網絡全國合作伙伴技術經

TungstenFabric 2020-07-03 08:19:32

一本SDN入門書籍讀書筆記——controller

Onix分佈式Controller 被設計用來控制大型網絡，有很強的擴展性； 通過引入Control Login（控制邏輯，可以認爲是特殊的應用程序）、Controller、和物理設備三層架構； 每個Controller只控制部分物理設備

gam0n 2020-07-08 11:53:38

Simulator和Emulator的區別

Simulator和Emulator的區別Simulator和Emulator異同相同之處不同之處使用決策何時使用Simulator何時使用Emulator結論參考 軟件測試種，經常會聽到Simulator和Emulator，兩者

BasilGuo 2020-07-08 01:18:26

opendaylight中的VTN模塊介紹

虛擬租戶網絡（VTN）介紹：

校嘻嘻 2020-07-06 21:25:46

golang etcd服務註冊與發現

產生場景 服務端架構最前面是 一臺網關節點 ，網關後面是多臺運行着一樣的 服務的計算節點，客戶端一般就是訪問網關 ，然後 網關 就把流量 轉發到 後面的計算節點上。如果後面的計算節點信息處理不過來的時候，我們需要加機器，最low的

猫小宝 2020-07-06 12:29:10

mininet無法正常連接RYU

根據控制器終端給的提示，需添加交換機版本

Ashley_Lee 2020-07-06 11:10:27

【SDN系列】之SDWAN產業地圖分析，拿走不謝

Link GAIA 2020-07-05 17:42:04

【IDC系列】之人爲“拔電源、剪光纖”等暴力“刪庫”事件真的不可控嗎？

不知道大家的朋友圈，有沒被近期運維人員惡意刪庫事件刷屏了。這次的故障已超過36小時，仍未完全恢復。據悉，該公司的業務系統數據庫（包括主備）遭遇其公司運維人員的刪除，目前技術團隊正在努力恢復數據中。 目前對新用戶服務已

Link GAIA 2020-07-05 17:42:04

【SDN系列】之“SDWAN和專線對比”爲什麼說SDWAN是新一代企業組網的最佳選擇？

首先，我們來看一下傳統企業組網常見的一些問題： 相信很多企業在組網過程以及體驗中都或多或少遇到過上訴問題，那爲什麼說SD-WAN是新一代企業組網的最佳選擇呢？ 我們收集了其中一家SD-WAN主流供應商的解決方案，來看看SD-WA

Link GAIA 2020-07-05 17:42:04

【直播預告】SDWAN+安全，如何幫助企業組建安全智能的自有網絡

企業數字轉型不僅僅是將工作流遷移到雲端和採納IoT，而是整個網絡的工具重構，令整個網絡變得更快、更高效、更靈活、更具成本有效性。同時，數字轉型還意味着應用敏捷軟件及應用開發，重新思考訪問與登錄，以及創建動態自適應網絡環境。SD-W

Link GAIA 2020-07-05 17:41:53

【SDN系列】之全球訪問Salesforce體驗太差怎麼辦？SEC+SDWAN高可用部署正流行

全球訪問Salesforce體驗太差怎麼辦？SEC+SDWAN高可用部署正流行 目前Salesforce在全球擁有200多萬用戶，連續五年被IDC評爲全球第一的CRM，已然成爲國際CRM巨頭。 隨着中國企業出海熱持續升溫，國際企業

Link GAIA 2020-07-05 17:41:53

理解Segment Routing和SDWAN

作爲一項新技術，Segment Routing(SR)最近幾年非常火爆，它和另一個概念SDN有效結合，成就了一些非常有意思的東西，比如說SR-TE。 但其實，SR理解起來其實是SDN/SDWAN發展到一定階段自然而然的思路。 SD

dog250 2020-07-05 04:06:44