Tekton CI 之實戰篇（二): DinD方式構建鏡像

Tekton CI 之實戰篇（二）： DinD方式構建鏡像

背景

簡單介紹下什麼是dind？使用場景是什麼？
DinD即 Docker inside Docker, DinD在容器裏有一個完整的docker構建系統,可直接在容器中完成鏡像的構建,與之相對應的就是DooD ,通過掛載宿主機的docker.sock文件，調用宿主機的docker daemon去構建鏡像。他們的主要使用場景有很多，比較常見的就是CICD場景中了，CICD需要構建鏡像。我之前的文章使用的就是DooD的方式，通過掛載宿主機的docker.sock文件。而本講介紹DinD，進行鏡像構建並推送。簡單說下DooD和Dind的優勢與劣勢：

	DooD	DinD
優勢	同一臺宿主機上的緩存可以通過同一個 Docker daemon 共享，這樣構建鏡像會比較快	1、明顯支持併發，一個容器實例使用一個docker daemon進程； 2、容器實例之間相互隔離，不干擾，不使用緩存，比較乾淨 DinD劣勢：
劣勢	1、由於不同容器實例掛在同一個宿主機的 Docker daemon 進程，所有實例裏 docker 命令的權限也是共享的，也就是說不同容器實例可以查看甚至更新、刪除到同一個 Docker daemon 下別的容器實例構建產生的鏡像。這就是個安全問題； 2、DooD的並行效果不咋地，因爲多個容器實例使用同一個Docker daemon進程。可能存在排隊或者爭搶。 3、DooD的方式會導致宿主機本地的存儲被大量消耗，需要定期清理鏡像	速度上還是沒有DooD快，因爲不能使用緩存。每個容器構建完後即銷燬。

本講想使用DinD的方式，因爲我在實際開發中需要一個相互之間隔離，且每次使用鏡像時，都需要重新拉取的環境，所以我選在DinD的方式來構建鏡像。

DinD實戰

製作DinD鏡像

先去dockerhub上找下docker鏡像，裏面有dind的版本。然後你可以參考tekton的dind-sidecar來構建鏡像（參照參考文章[3]），但是由於構建鏡像時需要拉取dockerhub鏡像作爲基礎鏡像，就需要加上鏡像加速器，不然可能會拉取鏡像失敗導致構建鏡像不成功。所以,我們需要構建一個帶鏡像加速器的docker:dind的鏡像。參照 docker鏡像的github地址.你可以找到dind的構建所需要的Dockerfile和entrypoint.sh文件。下面我貼一下我的Dockerfile和entrypoint.sh文件：
Dockerfile:(就改了基礎鏡像，改成了18.05，其他都沒改)

FROM docker:18.05
# https://github.com/docker/docker/blob/master/project/PACKAGERS.md#runtime-dependencies
RUN set -eux; \
        apk add --no-cache \
                btrfs-progs \
                e2fsprogs \
                e2fsprogs-extra \
                iptables \
                openssl \
                shadow-uidmap \
                xfsprogs \
                xz \
# pigz: https://github.com/moby/moby/pull/35697 (faster gzip implementation)
                pigz \
        ; \
# only install zfs if it's available for the current architecture
# https://git.alpinelinux.org/cgit/aports/tree/main/zfs/APKBUILD?h=3.6-stable#n9 ("all !armhf !ppc64le" as of 2017-11-01)
# "apk info XYZ" exits with a zero exit code but no output when the package exists but not for this arch
        if zfs="$(apk info --no-cache --quiet zfs)" && [ -n "$zfs" ]; then \
                apk add --no-cache zfs; \
        fi

# TODO aufs-tools

# set up subuid/subgid so that "--userns-remap=default" works out-of-the-box
RUN set -x \
        && addgroup -S dockremap \
        && adduser -S -G dockremap dockremap \
        && echo 'dockremap:165536:65536' >> /etc/subuid \
        && echo 'dockremap:165536:65536' >> /etc/subgid

# https://github.com/docker/docker/tree/master/hack/dind
ENV DIND_COMMIT 37498f009d8bf25fbb6199e8ccd34bed84f2874b

RUN set -eux; \
        wget -O /usr/local/bin/dind "https://raw.githubusercontent.com/docker/docker/${DIND_COMMIT}/hack/dind"; \
        chmod +x /usr/local/bin/dind

COPY dockerd-entrypoint.sh /usr/local/bin/

VOLUME /var/lib/docker
EXPOSE 2375 2376

ENTRYPOINT ["dockerd-entrypoint.sh"]
CMD []

entrypony.sh:(增加了阿里雲鏡像加速地址)

#!/bin/sh
set -eu

_tls_ensure_private() {
        local f="$1"; shift
        [ -s "$f" ] || openssl genrsa -out "$f" 4096
}
_tls_san() {
        {
                ip -oneline address | awk '{ gsub(/\/.+$/, "", $4); print "IP:" $4 }'
                {
                        cat /etc/hostname
                        echo 'docker'
                        echo 'localhost'
                        hostname -f
                        hostname -s
                } | sed 's/^/DNS:/'
                [ -z "${DOCKER_TLS_SAN:-}" ] || echo "$DOCKER_TLS_SAN"
        } | sort -u | xargs printf '%s,' | sed "s/,\$//"
}
_tls_generate_certs() {
        local dir="$1"; shift

        # if ca/key.pem || !ca/cert.pem, generate CA public if necessary
        # if ca/key.pem, generate server public
        # if ca/key.pem, generate client public
        # (regenerating public certs every startup to account for SAN/IP changes and/or expiration)

        # https://github.com/FiloSottile/mkcert/issues/174
        local certValidDays='825'

        if [ -s "$dir/ca/key.pem" ] || [ ! -s "$dir/ca/cert.pem" ]; then
                # if we either have a CA private key or do *not* have a CA public key, then we should create/manage the CA
                mkdir -p "$dir/ca"
                _tls_ensure_private "$dir/ca/key.pem"
                openssl req -new -key "$dir/ca/key.pem" \
                        -out "$dir/ca/cert.pem" \
                        -subj '/CN=docker:dind CA' -x509 -days "$certValidDays"
        fi

        if [ -s "$dir/ca/key.pem" ]; then
                # if we have a CA private key, we should create/manage a server key
                mkdir -p "$dir/server"
                _tls_ensure_private "$dir/server/key.pem"
                openssl req -new -key "$dir/server/key.pem" \
                        -out "$dir/server/csr.pem" \
                        -subj '/CN=docker:dind server'
                cat > "$dir/server/openssl.cnf" <<-EOF
                        [ x509_exts ]
                        subjectAltName = $(_tls_san)
                EOF
                openssl x509 -req \
                                -in "$dir/server/csr.pem" \
                                -CA "$dir/ca/cert.pem" \
                                -CAkey "$dir/ca/key.pem" \
                                -CAcreateserial \
                                -out "$dir/server/cert.pem" \
                                -days "$certValidDays" \
                                -extfile "$dir/server/openssl.cnf" \
                                -extensions x509_exts
                cp "$dir/ca/cert.pem" "$dir/server/ca.pem"
                openssl verify -CAfile "$dir/server/ca.pem" "$dir/server/cert.pem"
        fi

        if [ -s "$dir/ca/key.pem" ]; then
                # if we have a CA private key, we should create/manage a client key
                mkdir -p "$dir/client"
                _tls_ensure_private "$dir/client/key.pem"
                chmod 0644 "$dir/client/key.pem" # openssl defaults to 0600 for the private key, but this one needs to be sared with arbitrary client contexts
                openssl req -new \
                                -key "$dir/client/key.pem" \
                                -out "$dir/client/csr.pem" \
                                -subj '/CN=docker:dind client'
                cat > "$dir/client/openssl.cnf" <<-'EOF'
                        [ x509_exts ]
                        extendedKeyUsage = clientAuth
                EOF
                openssl x509 -req \
                                -in "$dir/client/csr.pem" \
                                -CA "$dir/ca/cert.pem" \
                                -CAkey "$dir/ca/key.pem" \
                                -CAcreateserial \
                                -out "$dir/client/cert.pem" \
                                -days "$certValidDays" \
                                -extfile "$dir/client/openssl.cnf" \
                                -extensions x509_exts
                cp "$dir/ca/cert.pem" "$dir/client/ca.pem"
                openssl verify -CAfile "$dir/client/ca.pem" "$dir/client/cert.pem"
        fi
}

# no arguments passed
# or first arg is `-f` or `--some-option`
if [ "$#" -eq 0 ] || [ "${1#-}" != "$1" ]; then
        # set "dockerSocket" to the default "--host" *unix socket* value (for both standard or rootless)
        uid="$(id -u)"
        if [ "$uid" = '0' ]; then
                dockerSocket='unix:///var/run/docker.sock'
        else
                # if we're not root, we must be trying to run rootless
                : "${XDG_RUNTIME_DIR:=/run/user/$uid}"
                dockerSocket="unix://$XDG_RUNTIME_DIR/docker.sock"
        fi
        case "${DOCKER_HOST:-}" in
                unix://*)
                        dockerSocket="$DOCKER_HOST"
                        ;;
        esac

        # add our default arguments
        if [ -n "${DOCKER_TLS_CERTDIR:-}" ] \
                && _tls_generate_certs "$DOCKER_TLS_CERTDIR" \
                && [ -s "$DOCKER_TLS_CERTDIR/server/ca.pem" ] \
                && [ -s "$DOCKER_TLS_CERTDIR/server/cert.pem" ] \
                && [ -s "$DOCKER_TLS_CERTDIR/server/key.pem" ] \
        ; then
                # generate certs and use TLS if requested/possible (default in 19.03+)
                set -- dockerd \
                        --host="$dockerSocket" \
                        --host=tcp://0.0.0.0:2376 \
                        --tlsverify \
                        --tlscacert "$DOCKER_TLS_CERTDIR/server/ca.pem" \
                        --tlscert "$DOCKER_TLS_CERTDIR/server/cert.pem" \
                        --tlskey "$DOCKER_TLS_CERTDIR/server/key.pem" \
                        --registry-mirror=https://cduvuqsh.mirror.aliyuncs.com \
                        "$@"
                DOCKERD_ROOTLESS_ROOTLESSKIT_FLAGS="${DOCKERD_ROOTLESS_ROOTLESSKIT_FLAGS:-} -p 0.0.0.0:2376:2376/tcp"
        else
                # TLS disabled (-e DOCKER_TLS_CERTDIR='') or missing certs
                set -- dockerd \
                        --host="$dockerSocket" \
                        --host=tcp://0.0.0.0:2375 \
                        "$@"
                DOCKERD_ROOTLESS_ROOTLESSKIT_FLAGS="${DOCKERD_ROOTLESS_ROOTLESSKIT_FLAGS:-} -p 0.0.0.0:2375:2375/tcp"
        fi
fi

if [ "$1" = 'dockerd' ]; then
        # explicitly remove Docker's default PID file to ensure that it can start properly if it was stopped uncleanly (andthus didn't clean up the PID file)
        find /run /var/run -iname 'docker*.pid' -delete || :

        uid="$(id -u)"
        if [ "$uid" != '0' ]; then
                # if we're not root, we must be trying to run rootless
                if ! command -v rootlesskit > /dev/null; then
                        echo >&2 "error: attempting to run rootless dockerd but missing 'rootlesskit' (perhaps the 'docker:ind-rootless' image variant is intended?)"
                        exit 1
                fi
                user="$(id -un 2>/dev/null || :)"
                if ! grep -qE "^($uid${user:+|$user}):" /etc/subuid || ! grep -qE "^($uid${user:+|$user}):" /etc/subgid; thn
                        echo >&2 "error: attempting to run rootless dockerd but missing necessary entries in /etc/subuid an/or /etc/subgid for $uid"
                        exit 1
                fi
                : "${XDG_RUNTIME_DIR:=/run/user/$uid}"
                export XDG_RUNTIME_DIR
                if ! mkdir -p "$XDG_RUNTIME_DIR" || [ ! -w "$XDG_RUNTIME_DIR" ] || ! mkdir -p "$HOME/.local/share/docker" | [ ! -w "$HOME/.local/share/docker" ]; then
                        echo >&2 "error: attempting to run rootless dockerd but need writable HOME ($HOME) and XDG_RUNTIME_IR ($XDG_RUNTIME_DIR) for user $uid"
                        exit 1
                fi
                if [ -f /proc/sys/kernel/unprivileged_userns_clone ] && unprivClone="$(cat /proc/sys/kernel/unprivileged_usrns_clone)" && [ "$unprivClone" != '1' ]; then
                        echo >&2 "error: attempting to run rootless dockerd but need 'kernel.unprivileged_userns_clone' (/poc/sys/kernel/unprivileged_userns_clone) set to 1"
                        exit 1
                fi
                if [ -f /proc/sys/user/max_user_namespaces ] && maxUserns="$(cat /proc/sys/user/max_user_namespaces)" && [ $maxUserns" = '0' ]; then
                        echo >&2 "error: attempting to run rootless dockerd but need 'user.max_user_namespaces' (/proc/sys/ser/max_user_namespaces) set to a sufficiently large value"
                        exit 1
                fi
                # TODO overlay support detection?
                exec rootlesskit \
                        --net="${DOCKERD_ROOTLESS_ROOTLESSKIT_NET:-vpnkit}" \
                        --mtu="${DOCKERD_ROOTLESS_ROOTLESSKIT_MTU:-1500}" \
                        --disable-host-loopback \
                        --port-driver=builtin \
                        --copy-up=/etc \
                        --copy-up=/run \
                        ${DOCKERD_ROOTLESS_ROOTLESSKIT_FLAGS:-} \
                        "$@"
        elif [ -x '/usr/local/bin/dind' ]; then
                # if we have the (mostly defunct now) Docker-in-Docker wrapper script, use it
                set -- '/usr/local/bin/dind' "$@"
        fi
else
        # if it isn't `dockerd` we're trying to run, pass it through `docker-entrypoint.sh` so it gets `DOCKER_HOST` set apropriately too
        set -- docker-entrypoint.sh "$@"
fi

exec "$@"

我在entrypoint.sh中增加了registry-mirror的指定，寫了我的阿里雲鏡像加速地址。
運行

docker build -t registru.nanjun/tekton/docker:v18.05-dind

製作完dind鏡像，下面開始跑tekton的taskrun啦

task和taskrun

參考tekton的dind-sidecar的yaml，task.yaml如下：

apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
  name: docker-in-docker-demo
  namespace: nanjun
spec:
  steps:
  - image: docker
    name: client
    script: |
        #!/usr/bin/env sh
        cat > Dockerfile << EOF
        FROM ubuntu
        RUN apt-get update
        ENTRYPOINT ["echo", "hello"]
        EOF
        docker build -t hello . && docker run hello
        docker images
    volumeMounts:
      - mountPath: /var/run/
        name: dind-socket
  sidecars: #sidecar模式
  - image: registry.nanjun/tekton/docker:v18.05-dind
    name: server
    securityContext:
      privileged: true
    volumeMounts:
      - mountPath: /var/lib/docker
        name: dind-storage
      - mountPath: /var/run/
        name: dind-socket
  volumes:
  - name: dind-storage
    emptyDir: {}
  - name: dind-socket
    emptyDir: {}

你需要了解k8s的sidecar模式（邊車模式），我這邊的dind使用是每次運行taskrun，都會在pod中起一個sidecar的容器，作爲docker daemon，另一個client容器將構建請求發送到這個server容器，構建完，client本地運行一個容器並查看鏡像列表，然後退出，退出後pod的中所有容器都會銷燬，sidecar容器也會被銷燬。這樣就實現了在隔離的環境中構建鏡像，因爲下一次構建鏡像還是同樣的過程，但是和上一次的完全無關。
下面是taskrun.yaml:

apiVersion: tekton.dev/v1beta1
kind: TaskRun
metadata:
  name: docker-in-docker-demo
  namespace: nanjun
spec:
  taskRef:
    name: docker-in-docker-demo

運行完的結果如下：

總結

DinD可容入tekton CI中，作爲構建鏡像的一種選擇，用戶可自由選擇使用dood還是dind的方式，使用緩存就用dood，使用隔離，就用dind。本地使用文件將在github地址查看：https://github.com/fishingfly/sidecar-dind， 如使用有問題加我微信:nanjun_1224, 歡迎關注微信公衆號“雲原生書記”，我們一起成長！

參考文章
[1]:https://hub.docker.com/_/docker
[2]:https://github.com/docker-library/docker
[3]:https://github.com/tektoncd/pipeline/blob/master/examples/v1alpha1/taskruns/dind-sidecar.yaml