手把手教你使用K8S部署Apollo高可用

一、下載源碼

Apollo-git地址

二、創建數據庫

按照官網wiki創建數據庫

三、修改configservice配置

1. 修改config/apollo-configservice.conf

只修改下面內容，調整容器內日誌目錄

LOG_FOLDER=~/opt/logs/config/

2. 修改config/application-github.properties 【可改可不改，建議修改防止意外】

增加如下配置，禁用apollo的eureka，本次部署使用獨立eureka集羣

apollo.eureka.server.enabled=false

3. 修改resources/application.yml

修改端口號，避免端口衝突，修改日誌目錄爲config/apollo-configservice.conf中配置目錄，指定文件名

server:
  port: 28080

logging:
  file: ～/opt/logs/config/apollo-configservice.log

4. 修改bootstrap.yml

按註釋修改，其餘不用動

eureka:
  instance:
    # 修改此配置如下 30003端口號在第九步中說明 k8s-node-ip替換爲自己k8s節點ip
    homePageUrl: http://k8s-node-ip:30003
    # 修改此配置如下
    preferIpAddress: false

# 增加此屬性， 禁用apollo自帶的eureka
apollo:
  eureka:
    server:
      enabled: false

5. 修改configservice.properties

只修改這兩個屬性，與resources/application.yml對應

server.port= 28080
logging.file= ～/opt/logs/config/apollo-configservice.log

6. 修改scripts/startup.sh

只修改這兩個屬性，與上面對應

## Adjust log dir if necessary
LOG_DIR=~/opt/logs/config
## Adjust server port if necessary
SERVER_PORT=${SERVER_PORT:=28080}

四、修改adminservier配置

1. 修改config/apollo-adminservice.conf

日誌路徑修改爲當前用戶目錄下opt/logs/admin/文件夾內

LOG_FOLDER=~/opt/logs/portal/

2. 修改config/application-github.properties 【可改可不改，建議修改防止意外】

增加如下配置，禁用apollo的eureka，本次部署使用獨立eureka集羣

apollo.eureka.server.enabled=false

3. 修改resources/adminservice.properties

修改端口號，避免端口衝突，修改日誌目錄爲config/apollo-adminservice.conf中配置目錄，指定文件名

server.port= 28090
logging.file= ~/opt/logs/admin/apollo-adminservice.log

4. 修改resources/application.yml

修改端口號，避免端口衝突，修改日誌目錄爲config/apollo-adminservice.conf中配置目錄，指定文件名

server:
  port: 28090
  
logging:
  file: ~/opt/logs/admin/apollo-adminservice.log

5. 修改resources/bootstrap.yml

按註釋修改，其餘不用動

eureka:
  instance:
    # 修改此配置如下 28091端口號在第十步中說明 k8s-node-ip替換爲自己k8s節點ip
    homePageUrl: http://k8s-node-ip:28091
    # 修改此配置如下
    preferIpAddress: false
    
# 增加此屬性， 禁用apollo自帶的eureka
apollo:
  eureka:
    server:
      enabled: false

6. 修改scripts/startup.sh

只修改這兩個屬性，與上面對應

## Adjust log dir if necessary
LOG_DIR=~/opt/logs/admin
## Adjust server port if necessary
SERVER_PORT=${SERVER_PORT:=28090}

五、修改portal配置

1. 修改config/apollo-portal.conf

日誌路徑修改爲當前用戶目錄下opt/logs/portal/文件夾內

LOG_FOLDER=~/opt/logs/portal/

2. 修改resources/apollo-env.properties【可改可不改，建議修改】

修改這裏的端口號爲configservice服務的端口號

local.meta=http://localhost:28080

3. 修改resources/application.yml

修改端口號，避免端口衝突，修改日誌目錄爲config/apollo-portalservice.conf中配置目錄，指定文件名

server:
  port: 28070
  
logging:
  file: ~/opt/logs/portal/apollo-portal.log

4. 修改scripts/startup.sh

## Adjust log dir if necessary
LOG_DIR=~/opt/logs/portal
## Adjust server port if necessary
SERVER_PORT=${SERVER_PORT:=28070}

六、修改Dockerfile內容

1. 修改scripts/apollo-on/kubernetes/apollo-config-server/Dockerfile

只修改下面屬性值，這裏重新指定k8s中服務命名空間

ENV APOLLO_CONFIG_SERVICE_NAME=“service-apollo-config-server.apollo”

2. 修改scripts/apollo-on/kubernetes/apollo-config-server/scripts/startup-kubernetes.sh

只修改端口號與日誌存放目錄

## Adjust log dir if necessary
LOG_DIR=~/opt/logs/config
## Adjust server port if necessary
SERVER_PORT=28080

3. 修改scripts/apollo-on/kubernetes/apollo-admin-server/Dockerfile

只修改下面屬性值，這裏重新指定k8s中服務命名空間

ENV APOLLO_ADMIN_SERVICE_NAME=“service-apollo-admin-server.apollo”

4. 修改scripts/apollo-on/kubernetes/apollo-admin-server/scripts/startup-kubernetes.sh

只修改端口號與日誌存放目錄

## Adjust log dir if necessary
LOG_DIR=~/opt/logs/apollo-admin-server
## Adjust server port if necessary
SERVER_PORT=28090

5. 修改scripts/apollo-on/kubernetes/apollo-portal-server/Dockerfile

只修改下面屬性值，這裏重新指定k8s中服務命名空間

ENV APOLLO_PORTAL_SERVICE_NAME=“service-apollo-portal-server.apollo”

6. 修改scripts/apollo-on/kubernetes/apollo-portal-server/scripts/startup-kubernetes.sh

只修改端口號與日誌存放目錄

## Adjust log dir if necessary
LOG_DIR=~/opt/logs/apollo-portal-server
## Adjust server port if necessary
SERVER_PORT=28070

七、執行編譯腳本

在scripts/目錄下運行./build.sh

八、編譯鏡像並push到docker倉庫

1. copy並重命名jar包

1. 解壓apollo-configservice/target/apollo-configservice-1.6.0-SNAPSHOT-github.zip, 將apollo-configservice-1.6.0-SNAPSHOT.jar複製到scripts/apollo-on/kubernetes/apollo-config-server/，並重命名爲apollo-configservice.jar
2. 解壓apollo-adminservice/target/apollo-adminservice-1.6.0-SNAPSHOT-github.zip, 將apollo-adminservice-1.6.0-SNAPSHOT.jar複製到scripts/apollo-on/kubernetes/apollo-admin-server/，並重命名爲apollo-adminservice.jar
3. 解壓apollo-portalservice/target/apollo-portalservice-1.6.0-SNAPSHOT-github.zip, 將apollo-portalservice-1.6.0-SNAPSHOT.jar複製到scripts/apollo-on/kubernetes/apollo-portal-server/，並重命名爲apollo-portalservice.jar

2. 編譯鏡像並push

在以上jar包對應目錄下分別執行：

docker build -t imageName:tag .
docker push imageName:tag
注意，build命令後有個點

九、apollo-config-server.yml

# 在pod創建之前先創建configMap
#   pod可以通過三種方式使用configMap
#   1. 將configMap中的數據設置爲環境變量
#   2. 將configMap中的數據設置爲命令行參數
#   3. 使用volume將configMap作爲文件或目錄掛載
kind: ConfigMap
apiVersion: v1
metadata:
  namespace: apollo                                                 
  name: configmap-apollo-config-server-dev
data:        
  # 【注意這裏數據庫連接與eureka連接在不同環境都需要切換】目前由於資源問題只提供dev環境，
  # 數據庫通過ip連接 【改成自己的數據庫連接】
  # eureka通過config-server-pod-name-index.meta-server-service-name:prot/eureka/指定     【沒通過k8s部署的直接用自己的eureka服務】      
  application-github.properties: |
    spring.datasource.url = jdbc:mysql://mysql-server-ip:3306/ApolloConfigDB_Dev?characterEncoding=utf8   
    spring.datasource.username = username
    spring.datasource.password = password
    eureka.service.url = http://eureka-service-0.eureka-service.default:9008/eureka/,http://eureka-service-1.eureka-service.default:9008/eureka/,http://eureka-service-2.eureka-service.default:9008/eureka/

---
kind: Service
apiVersion: v1
metadata:
  namespace: apollo
  name: service-apollo-meta-server-dev
  labels:
    app: service-apollo-meta-server-dev
spec:
  ports:
    - protocol: TCP
      port: 28080                          # 設定Serivce對外提供服務的端口
      targetPort: 28080                    # 設定容器(Pod)的端口,即Pod網絡的端口
      nodePort: 30003                      # 將meta-server服務對節點外開發，讓開發機器也能訪問到k8s中的apollo
  selector:
    app: pod-apollo-config-server-dev
  type: NodePort
#  clusterIP: None
  sessionAffinity: ClientIP

---
kind: Service
apiVersion: v1
metadata:
  namespace: apollo
  name: service-apollo-config-server-dev
  labels:
    app: service-apollo-config-server-dev
spec:
  ports:
    - protocol: TCP
      port: 28080
      targetPort: 28080
      nodePort: 30002                     # 它僅在type爲NodePort時才需要指定
  selector:
    app: pod-apollo-config-server-dev 
  type: NodePort                          # NodePort類型的service, 可以讓k8s節點外的主機都能訪問到
  sessionAffinity: ClientIP


# RC、Deployment、DaemonSet都是面向無狀態的服務，它們所管理的Pod的IP、名字，啓停順序等都是隨機的
# StatefulSet是有狀態的集合，管理所有有狀態的服務
# StatefulSet本質上是Deployment的一種變體，在v1.9版本中已成爲GA版本，它爲了解決有狀態服務的問題，它所管理的Pod擁有固定的Pod名稱，啓停順序
# 在StatefulSet中，Pod名字稱爲網絡標識(hostname)，還必須要用到共享存儲
---
kind: StatefulSet
apiVersion: apps/v1
metadata:
  namespace: apollo
  name: statefulset-apollo-config-server-dev
  labels:
    app: statefulset-apollo-config-server-dev
spec:
  serviceName: service-apollo-meta-server-dev               # 聲明它屬於哪個Headless Service.
  replicas: 3
  selector:
    matchLabels:
      app: pod-apollo-config-server-dev                     # has to match .spec.template.metadata.labels
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: pod-apollo-config-server-dev                   # has to match .spec.selector.matchLabels
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - pod-apollo-config-server-dev
              topologyKey: kubernetes.io/hostname

      volumes:
        - name: volume-configmap-apollo-config-server-dev
          configMap:
            name: configmap-apollo-config-server-dev
            items:
              - key: application-github.properties
                path: application-github.properties
      
      containers:
        - image: imageName:tag   # 這裏需要改成自己打的鏡像
          securityContext:
            privileged: true
          imagePullPolicy: IfNotPresent
          name: container-apollo-config-server-dev
          ports:
            - protocol: TCP
              containerPort: 28080
          resources:
            limits:
              memory: 512Mi  # 限制內存爲512M
          volumeMounts:
            - name: volume-configmap-apollo-config-server-dev
              mountPath: /apollo-config-server/config/application-github.properties
              subPath: application-github.properties
          
          env:
            - name: APOLLO_CONFIG_SERVICE_NAME
              value: "service-apollo-config-server-dev.apollo"
          
          readinessProbe:
            tcpSocket:
              port: 28080
            initialDelaySeconds: 10
            periodSeconds: 5
          
          livenessProbe:
            tcpSocket:
              port: 28080
            initialDelaySeconds:  120
            periodSeconds: 10
          
      dnsPolicy: ClusterFirst
      restartPolicy: Always

十、apollo-admin-server.yml

# 在pod創建之前先創建configMap
#   pod可以通過三種方式使用configMap
#   1. 將configMap中的數據設置爲環境變量
#   2. 將configMap中的數據設置爲命令行參數
#   3. 使用volume將configMap作爲文件或目錄掛載
kind: ConfigMap
apiVersion: v1
metadata:
  namespace: apollo                                    
  name: configmap-apollo-admin-server-dev
data:
  # 【注意這裏數據庫連接與eureka連接在不同環境都需要切換】目前由於資源問題只提供dev環境，
  # 數據庫通過ip連接 【改成自己的數據庫連接】
  # eureka通過config-server-pod-name-index.meta-server-service-name:prot/eureka/指定     【沒通過k8s部署的直接用自己的eureka服務】          
  application-github.properties: |
    spring.datasource.url = jdbc:mysql://mysql-server-ip:3306/ApolloConfigDB_Dev?characterEncoding=utf8   
    spring.datasource.username = username
    spring.datasource.password = password
    eureka.service.url = http://eureka-service-0.eureka-service.default:9008/eureka/,http://eureka-service-1.eureka-service.default:9008/eureka/,http://eureka-service-2.eureka-service.default:9008/eureka/

---
kind: Service
apiVersion: v1
metadata:
  namespace: apollo
  name: service-apollo-admin-server-dev
  labels:
    app: service-apollo-admin-server-dev
spec:
  ports:
    - protocol: TCP
      port: 28090                          # 設定Serivce對外提供服務的端口
      targetPort: 28090                    # 設定容器(Pod)的端口,即Pod網絡的端口
      nodePort: 28091
  selector:
    app: pod-apollo-admin-server-dev
  type: NodePort
  # clusterIP: None
  sessionAffinity: ClientIP

# RC、Deployment、DaemonSet都是面向無狀態的服務，它們所管理的Pod的IP、名字，啓停順序等都是隨機的
# StatefulSet是有狀態的集合，管理所有有狀態的服務
# StatefulSet本質上是Deployment的一種變體，在v1.9版本中已成爲GA版本，它爲了解決有狀態服務的問題，它所管理的Pod擁有固定的Pod名稱，啓停順序
# 在StatefulSet中，Pod名字稱爲網絡標識(hostname)，還必須要用到共享存儲
---
kind: Deployment
apiVersion: apps/v1
metadata:
  namespace: apollo
  name: deployment-apollo-admin-server-dev
  labels:
    app: deployment-apollo-admin-server-dev
spec:
  replicas: 3
  selector:
    matchLabels:
      app: pod-apollo-admin-server-dev                     # has to match .spec.template.metadata.labels
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: pod-apollo-admin-server-dev                   # has to match .spec.selector.matchLabels
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - pod-apollo-admin-server-dev
              topologyKey: kubernetes.io/hostname

      volumes:
        - name: volume-configmap-apollo-admin-server-dev
          configMap:
            name: configmap-apollo-admin-server-dev
            items:
              - key: application-github.properties
                path: application-github.properties

      containers:
        - image: imageName:tag
          securityContext:
            privileged: true
          imagePullPolicy: IfNotPresent
          name: container-apollo-admin-server-dev
          ports:
            - protocol: TCP
              containerPort: 28090
          resources:
            limits:
              memory: 512Mi
          volumeMounts:
            - name: volume-configmap-apollo-admin-server-dev
              mountPath: /apollo-admin-server/config/application-github.properties
              subPath: application-github.properties
          
          env:
            - name: APOLLO_CONFIG_SERVICE_NAME
              value: "service-apollo-admin-server-dev.apollo"
          
          readinessProbe:
            tcpSocket:
              port: 28090
            initialDelaySeconds: 10
            periodSeconds: 5
          
          livenessProbe:
            tcpSocket:
              port: 28090
            initialDelaySeconds:  120
            periodSeconds: 10
          
      dnsPolicy: ClusterFirst
      restartPolicy: Always

十一、apollo-portal-server.yml

---
# configmap for apollo-portal-server
kind: ConfigMap
apiVersion: v1
metadata:
  namespace: apollo
  name: configmap-apollo-portal-server
data:
  # 數據庫通過ip連接 【改成自己的數據庫連接】
  application-github.properties: |
    spring.datasource.url = jdbc:mysql://mysql-server-ip:3306/ApolloPortalDB?characterEncoding=utf8   
    spring.datasource.username = username
    spring.datasource.password = password
  apollo-env.properties: |
    dev.meta=http://statefulset-apollo-config-server-dev-0.service-apollo-meta-server-dev:28080,http://statefulset-apollo-config-server-dev-1.service-apollo-meta-server-dev:28080,http://statefulset-apollo-config-server-dev-2.service-apollo-meta-server-dev:28080
#   資源限制，現在只使用dev環境
#   fat.meta=http://statefulset-apollo-config-server-fat-0.service-apollo-meta-server-fat:28080,http://statefulset-apollo-config-server-fat-1.service-apollo-meta-server-fat:28080,http://statefulset-apollo-config-server-fat-2.service-apollo-meta-server-fat:28080
#   uat.meta=http://service-apollo-config-server-test-beta.sre:8080
#   pro.meta=http://service-apollo-config-server-prod.sre:8080

---
kind: Service
apiVersion: v1
metadata:
  namespace: apollo
  name: service-apollo-portal-server
  labels:
    app: service-apollo-portal-server
spec:
  ports:
    - protocol: TCP
      port: 28070
      targetPort: 28070
      nodePort: 30001
  selector:
    app: pod-apollo-portal-server
  type: NodePort
  # portal session 保持
  sessionAffinity: ClientIP

# 這裏使用Deployment
---
kind: Deployment
apiVersion: apps/v1
metadata:
  namespace: apollo
  name: deployment-apollo-portal-server
  labels:
    app: deployment-apollo-portal-server
spec:
  # 3 個實例
  replicas: 3
  selector:
    matchLabels:
      app: pod-apollo-portal-server
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: pod-apollo-portal-server
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - pod-apollo-portal-server
              topologyKey: kubernetes.io/hostname
      
      volumes:
        - name: volume-configmap-apollo-portal-server
          configMap:
            name: configmap-apollo-portal-server
            items:
              - key: application-github.properties
                path: application-github.properties
              - key: apollo-env.properties
                path: apollo-env.properties
      
      containers:
        - image: imageName:tag    # 更改爲你的 docker registry 下的 image
          securityContext:
            privileged: true
          imagePullPolicy: IfNotPresent
          name: container-apollo-portal-server
          ports:
            - protocol: TCP
              containerPort: 28070
          resources:
            limits:
              memory: 512Mi          
          volumeMounts:
            - name: volume-configmap-apollo-portal-server
              mountPath: /apollo-portal-server/config/application-github.properties
              subPath: application-github.properties
            - name: volume-configmap-apollo-portal-server
              mountPath: /apollo-portal-server/config/apollo-env.properties
              subPath: apollo-env.properties
          
          env:
            - name: APOLLO_PORTAL_SERVICE_NAME
              value: "service-apollo-portal-server.apollo"
          
          readinessProbe:
            tcpSocket:
              port: 28070
            initialDelaySeconds: 10
            periodSeconds: 5
          
          livenessProbe:
            tcpSocket:
              port: 28070
            # 120s 內, server 未啓動則重啓 container
            initialDelaySeconds: 120
            periodSeconds: 15
          
      dnsPolicy: ClusterFirst
      restartPolicy: Always

十二、啓動服務

分別啓動configserver、adminserver、portalserver【必須嚴格按照此順序啓動，且在上一個服務啓動成功後再啓動下一個服務】， 命令如下:

kubectl create -f apollo-config-server.yml
kubectl create -f apollo-admin-server.yml
kubectl create -f apollo-portal-server.yml

啓動完成後可運行以下命令檢查pod啓動情況

kubectl get pods -n apollo

結果如下，所有pod都爲ready/running，則部署成功

查看某個pod日誌，先進入容器

kubectl exec -ti deployment-apollo-portal-server-59479cfddc-xf4fd /bin/sh -n apollo

再進入～/opt/logs目錄tail對應log文件

十三、訪問Apollo-portal服務

k8s任意節點ip:30001訪問，可在管理員工具-系統信息中查看服務狀態

到此，部署成功！