ali lxcfs daemonset方式運行

• 剛開始按照相關文檔將apiserver ,kubelet節點的特權模式開啓--allow-privileged=true，再以ali的文檔執行，完全無法運行。參考github裏的issue得知，其實都是在問題爲啥運行不起來的，但回覆不詳，其中也確實提到需要在宿主機上支持fuse。

• 開始自行排錯：無法搜索到相關資料，下載源碼，編譯排查
  git clone https://github.com/denverdino/lxcfs-initializer.git

• 從Dockfile裏也可知，裏面的庫等文件並不能適合自身的版本需要，目前只是需要讓他運行起來，然後再裏面執行start.sh的腳本內容，查看具體出錯是在哪？

• 據自己環境變更lxcfs-image/Dockerfile 內容如下

FROM daocloud.io/centos:7.3.1611
RUN yum -y install fuse fuse-devel pam-devel wget install gcc automake autoconf libtool make
ENV LXCFS_VERSION 2.0.8
RUN wget https://linuxcontainers.org/downloads/lxcfs/lxcfs-$LXCFS_VERSION.tar.gz && \
mkdir /lxcfs && tar xzvf lxcfs-$LXCFS_VERSION.tar.gz -C /lxcfs --strip-components=1 && \
cd /lxcfs && ./configure && make && make install
STOPSIGNAL SIGINT
ADD start.sh /
CMD ["/bin/sleep","10000"]

• build lxcfs:sleep鏡像

  [root@ns-yun-020037 ~]# cd lxcfs-initializer/
  docker build -t lxcfs:sleep lxcfs-image

• 根據原始daemonSet的yaml文件將鏡像名改爲lxcfs:sleep即可

• 進入節點容器定位問題，根據/start.sh 腳本執行相關命令，可見在最後一步執行時無法找到lxcfs這個文件

[root@yun-020040 ~]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
4e1cb10dd73e xxx:80/test/lxcfs "/bin/sleep 10000" 52 seconds ago Up 51 seconds k8s_lxcfs_lxcfs-4m5g7_default_b1306fd2-3bd4-11e9-bb5d-ec388f7928b2_0

[root@yun-020040 ~]# docker exec -it 4e1cb10dd73e /bin/bash 
[root@lxcfs-4m5g7 /]#
[root@lxcfs-4m5g7 /]# nsenter -m/proc/1/ns/mnt fusermount -u /var/lib/lxcfs 2> /dev/null || true
[root@lxcfs-4m5g7 /]# nsenter -m/proc/1/ns/mnt [ -L /etc/mtab ] ||sed -i "/^lxcfs \/var\/lib\/lxcfs fuse.lxcfs/d" /etc/mtab
[root@lxcfs-4m5g7 /]# mkdir -p /usr/local/lib/lxcfs /var/lib/lxcfs
[root@lxcfs-4m5g7 /]# exec nsenter -m/proc/1/ns/mnt lxcfs /var/lib/lxcfs/
nsenter: failed to execute lxcfs: No such file or directory

• 根據Dockerfile的內容可以得知，其實容器應該是有該文件的
  https://github.com/denverdino/lxcfs-initializer/blob/master/lxcfs-image/Dockerfile

• 直接用docker的方式來啓動看下是否有問題，執行start.sh命令，能正常執行

  [root@yun-020040 ~]# docker  run --privileged=true -it lxcfs:sleep  /bin/bash
  [root@10ca4ad41ce4 /]# nsenter -m/proc/1/ns/mnt fusermount -u /var/lib/lxcfs 2> /dev/null || true
  [root@10ca4ad41ce4 /]# nsenter -m/proc/1/ns/mnt [ -L /etc/mtab ] ||sed -i "/^lxcfs \/var\/lib\/lxcfs fuse.lxcfs/d" /etc/mtab
  [root@10ca4ad41ce4 /]# mkdir -p /usr/local/lib/lxcfs /var/lib/lxcfs
  [root@10ca4ad41ce4 /]# exec nsenter -m/proc/1/ns/mnt lxcfs /var/lib/lxcfs/
  hierarchies:
  0: fd: 5: perf_event
  1: fd: 6: hugetlb
  2: fd: 7: pids
  3: fd: 8: cpuacct,cpu
  4: fd: 9: blkio
  5: fd: 10: devices
  6: fd: 11: cpuset
  7: fd: 12: memory
  8: fd: 13: freezer
  9: fd: 14: net_prio,net_cls
  10: fd: 15: name=systemd

• 回看k8s的yaml文件，裏面有把宿主機的/usr/local目錄掛載的，且爲宿主的文件，如下粗體所示

volumeMounts:
- name: cgroup
mountPath: /sys/fs/cgroup
- name: lxcfs
mountPath: /var/lib/lxcfs
mountPropagation: Bidirectional
- name: usr-local
**mountPath: /usr/local**
volumes:
- name: cgroup
hostPath:
path: /sys/fs/cgroup
**- name: usr-local**
hostPath:
path: /usr/local
- name: lxcfs
hostPath:
path: /var/lib/lxcfs
type: DirectoryOrCreate

• 將yaml 文件中的/usr/local的掛載去掉，看是否能正常使用容器內部的lxcfs文件，經驗證失敗

• 據此提示，在宿主機再次安裝部署lxcfs,再結合運行daemonSet，程序運行正常

測試結果：
此項目只是將宿主機啓動進程託管給daemonSet，方便統一管理，宿主機還得提供相關二進制文件lib庫等...

• 進一步分析他的init容器內容，main.go,方便了掛載目錄。

  flag.StringVar(&annotation, "annotation", defaultAnnotation, "The annotation to trigger initialization")
  flag.StringVar(&initializerName, "initializer-name", defaultInitializerName, "The initializer name")
  flag.StringVar(&namespace, "namespace", "default", "The configuration namespace")
  flag.BoolVar(&requireAnnotation, "require-annotation", true, "Require annotation for initialization")
  flag.Parse()
  
  log.Println("Starting the Kubernetes initializer...")
  log.Printf("Initializer name set to: %s", initializerName)
  
  clusterConfig, err := rest.InClusterConfig()
  if err != nil {
      log.Fatal(err.Error())
  }
  
  clientset, err := kubernetes.NewForConfig(clusterConfig)
  if err != nil {
      log.Fatal(err)
  }
  
  // -v /var/lib/lxcfs/proc/cpuinfo:/proc/cpuinfo:rw
  // -v /var/lib/lxcfs/proc/diskstats:/proc/diskstats:rw
  // -v /var/lib/lxcfs/proc/meminfo:/proc/meminfo:rw
  // -v /var/lib/lxcfs/proc/stat:/proc/stat:rw
  // -v /var/lib/lxcfs/proc/swaps:/proc/swaps:rw
  // -v /var/lib/lxcfs/proc/uptime:/proc/uptime:rw
  c := &config{
      volumeMounts: []corev1.VolumeMount{
          corev1.VolumeMount{
              Name:      "lxcfs-proc-cpuinfo",
              MountPath: "/proc/cpuinfo",
          },

參考資料：
https://www.alibabacloud.com/blog/kubernetes-demystified%3A-using-lxcfs-to-improve-container-resource-visibility_594109?spm=a2c41.12195345.0.0
https://github.com/denverdino/lxcfs-initializer