kubernetes1.13.1集羣結合ceph rbd部署最新版本jenkins

原創 三杯水 2019-02-23 00:15

參考文檔

https://blog.csdn.net/aixiaoyang168/article/details/79767649
https://github.com/jenkinsci/kubernetes-plugin/tree/master/src/main/kubernetes
https://www.cnblogs.com/cocowool/p/kubernetes_statefulset.html
https://www.cnblogs.com/cocowool/p/kubernetes_storage.html

文檔目錄

簡介

jenkins-kubernetes-plugin
Jenkins plugin to run dynamic agents in a Kubernetes cluster.
Based on the Scaling Docker with Kubernetes article, automates the scaling of Jenkins agents running in Kubernetes.
The plugin creates a Kubernetes Pod for each agent started, defined by the Docker image to run, and stops it after each build.
Agents are launched using JNLP, so it is expected that the image connects automatically to the Jenkins master. For that some environment variables are automatically injected:

  • JENKINS_URL: Jenkins web interface url
  • JENKINS_SECRET: the secret key for authentication
  • JENKINS_AGENT_NAME: the name of the Jenkins agent
  • JENKINS_NAME: the name of the Jenkins agent (Deprecated. Only here for backwards compatibility)

基本環境

k81集羣1.13.1版本

[root@elasticsearch01 ~]# kubectl get nodes
NAME        STATUS   ROLES    AGE   VERSION
10.2.8.34   Ready    <none>   25d   v1.13.1
10.2.8.65   Ready    <none>   25d   v1.13.1

ceph集羣 luminous版本

[root@ceph01 ~]# ceph -s
  services:
    mon: 3 daemons, quorum ceph01,ceph02,ceph03
    mgr: ceph03(active), standbys: ceph02, ceph01
    osd: 24 osds: 24 up, 24 in
    rgw: 3 daemons active

操作步驟

一、使用ceph rbd創建pv、pvc

官網使用的是自帶創建pv與pvc這裏使用的是手動創建
1、創建pv

[root@elasticsearch01 jenkins]# cat jenkins-pv.yaml 
apiVersion: v1
kind: PersistentVolume
metadata:
  name: jenkins-home-pv
spec:
  capacity:
    storage: 40Gi
  accessModes:
    - ReadWriteOnce
  rbd:
    monitors:
      - '10.0.4.10:6789'
      - '10.0.4.13:6789'
      - '10.0.4.15:6789'
    pool: rbd-k8s
    image: cephimage3
    user: admin
    secretRef:
      name: ceph-secret
    fsType: ext4
    readOnly: false
  persistentVolumeReclaimPolicy: Recycle

[root@elasticsearch01 jenkins]# kubectl create -f jenkins-pv.yaml 
persistentvolume/jenkins-home-pv created
[root@elasticsearch01 jenkins]# kubectl get pv
NAME              CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM                       STORAGECLASS   REASON   AGE
ceph-rbd-pv       20Gi       RWO            Recycle          Bound       default/ceph-rbd-pv-claim                           22h
jenkins-home-pv   40Gi       RWO            Recycle          Available                                                       4s

2、創建pvc

[root@elasticsearch01 jenkins]# cat jenkins-pvc.yaml 
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: jenkins-home-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi

[root@elasticsearch01 jenkins]# kubectl create -f jenkins-pvc.yaml 
persistentvolumeclaim/jenkins-home-pvc created

[root@elasticsearch01 jenkins]# kubectl get pvc
NAME                STATUS   VOLUME            CAPACITY   ACCESS MODES   STORAGECLASS   AGE
ceph-rbd-pv-claim   Bound    ceph-rbd-pv       20Gi       RWO                           22h
jenkins-home-pvc    Bound    jenkins-home-pv   40Gi       RWO                           3s
[root@elasticsearch01 jenkins]# kubectl get pv
NAME              CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                       STORAGECLASS   REASON   AGE
ceph-rbd-pv       20Gi       RWO            Recycle          Bound    default/ceph-rbd-pv-claim                           22h
jenkins-home-pv   40Gi       RWO            Recycle          Bound    default/jenkins-home-pvc                            77s

二、跟進實際情況修改jenkins.yml文件

主要修改的配置從上到下分別是：
1、拉取鏡像策略

          imagePullPolicy: IfNotPresent

2、自動存儲storage class改成voulumes的pvc方式實現

      volumes:
      - name: jenkins-home
        persistentVolumeClaim:
          claimName: jenkins-home-pvc

3、ingress的host改成實際的

    host: jenkins.minminmsn.com

4、ingres的tls證書改成實際的

  tls:
  - hosts:
    - jenkins.minminmsn.com
    secretName: ingress-secret

5、具體如下

[root@elasticsearch01 jenkins]# cat jenkins.yml 
---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: jenkins
  labels:
    name: jenkins
spec:
  serviceName: jenkins
  replicas: 1
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      name: jenkins
      labels:
        name: jenkins
    spec:
      terminationGracePeriodSeconds: 10
      serviceAccountName: jenkins
      containers:
        - name: jenkins
          image: jenkins/jenkins:lts-alpine
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8080
            - containerPort: 50000
          resources:
            limits:
              cpu: 1
              memory: 1Gi
            requests:
              cpu: 0.5
              memory: 500Mi
          env:
            - name: LIMITS_MEMORY
              valueFrom:
                resourceFieldRef:
                  resource: limits.memory
                  divisor: 1Mi
            - name: JAVA_OPTS
              # value: -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:MaxRAMFraction=1 -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85
              value: -Xmx$(LIMITS_MEMORY)m -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85
          volumeMounts:
            - name: jenkins-home
              mountPath: /var/jenkins_home
              readOnly: false
          livenessProbe:
            httpGet:
              path: /login
              port: 8080
            initialDelaySeconds: 60
            timeoutSeconds: 5
            failureThreshold: 12 # ~2 minutes
          readinessProbe:
            httpGet:
              path: /login
              port: 8080
            initialDelaySeconds: 60
            timeoutSeconds: 5
            failureThreshold: 12 # ~2 minutes
      securityContext:
        fsGroup: 1000
      volumes:
      - name: jenkins-home
        persistentVolumeClaim:
          claimName: jenkins-home-pvc

---
apiVersion: v1
kind: Service
metadata:
  name: jenkins
spec:
  # type: LoadBalancer
  selector:
    name: jenkins
  # ensure the client ip is propagated to avoid the invalid crumb issue when using LoadBalancer (k8s >=1.7)
  #externalTrafficPolicy: Local
  ports:
    -
      name: http
      port: 80
      targetPort: 8080
      protocol: TCP
    -
      name: agent
      port: 50000
      protocol: TCP

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: jenkins
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    kubernetes.io/tls-acme: "true"
    # "413 Request Entity Too Large" uploading plugins, increase client_max_body_size
    nginx.ingress.kubernetes.io/proxy-body-size: 50m
    nginx.ingress.kubernetes.io/proxy-request-buffering: "off"
    # For nginx-ingress controller < 0.9.0.beta-18
    ingress.kubernetes.io/ssl-redirect: "true"
    # "413 Request Entity Too Large" uploading plugins, increase client_max_body_size
    ingress.kubernetes.io/proxy-body-size: 50m
    ingress.kubernetes.io/proxy-request-buffering: "off"
spec:
  rules:
  - http:
      paths:
      - path: /
        backend:
          serviceName: jenkins
          servicePort: 80
    host: jenkins.minminmsn.com
  tls:
  - hosts:
    - jenkins.minminmsn.com
    secretName: ingress-secret

三、創建狀態集、svc、pod、ingress

1、創建rbac認證角色

[root@elasticsearch01 jenkins]# kubectl create -f service-account.yml 
serviceaccount/jenkins created
role.rbac.authorization.k8s.io/jenkins created
rolebinding.rbac.authorization.k8s.io/jenkins created

2、創建jenkins服務等

[root@elasticsearch01 jenkins]# kubectl create -f jenkins.yml 
statefulset.apps/jenkins created
service/jenkins created
ingress.extensions/jenkins created        4s
[root@elasticsearch01 jenkins]# kubectl get pods
NAME               READY   STATUS              RESTARTS   AGE
busybox            1/1     Running             454        18d
ceph-rbd-pv-pod1   1/1     Running             1          21h
jenkins-0          0/1     ContainerCreating   0          7s
[root@elasticsearch01 jenkins]# kubectl get pods
NAME               READY   STATUS    RESTARTS   AGE
busybox            1/1     Running   454        18d
ceph-rbd-pv-pod1   1/1     Running   1          21h
jenkins-0          1/1     Running   0          4m52s

四、通過ingress訪問

獲取ingress-nginx對外端口，https://jenkins.minminmsn.com:47215/訪問即可，需要配置dns解析到pod所在node的ip

[root@elasticsearch01 jenkins]# kubectl get svc -n ingress-nginx|grep ingress-nginx
ingress-nginx       LoadBalancer   10.254.125.151   <pending>     80:33003/TCP,443:47215/TCP   14d

五、初始化jenkins

1、查找密碼

[root@elasticsearch02 ~]# df -h|grep rbd
/dev/rbd0                  493G  163G  306G  35% /data
/dev/rbd1                   20G   45M   20G   1% /var/lib/kubelet/plugins/kubernetes.io/rbd/mounts/rbd-k8s-image-cephimage2
/dev/rbd2                   40G  138M   40G   1% /var/lib/kubelet/plugins/kubernetes.io/rbd/mounts/rbd-k8s-image-cephimage3
[root@elasticsearch02 ~]# cd //var/lib/kubelet/plugins/kubernetes.io/rbd/mounts/rbd-k8s-image-cephimage3
[root@elasticsearch02 rbd-k8s-image-cephimage3]# ls
config.xml                     init.groovy.d                        jobs              nodes                     secrets      war
copy_reference_file.log        jenkins.CLI.xml                      logs              plugins                   updates
hudson.model.UpdateCenter.xml  jenkins.install.UpgradeWizard.state  lost+found        secret.key                userContent
identity.key.enc               jenkins.telemetry.Correlator.xml     nodeMonitors.xml  secret.key.not-so-secret  users
[root@elasticsearch02 rbd-k8s-image-cephimage3]# cat secrets/initialAdminPassword 
92c145b796cc48b0af8b5ef0f7afce28

kubernetes1.13.1集羣結合ceph rbd部署最新版本jenkins

2、選擇安裝插件

kubernetes1.13.1集羣結合ceph rbd部署最新版本jenkins
kubernetes1.13.1集羣結合ceph rbd部署最新版本jenkins

3、創建初始管理賬號

kubernetes1.13.1集羣結合ceph rbd部署最新版本jenkins

4、設置jenkins url默認https://jenkins.minminmsn.com:47215/

kubernetes1.13.1集羣結合ceph rbd部署最新版本jenkins

5、開始使用jenkins

kubernetes1.13.1集羣結合ceph rbd部署最新版本jenkins

6、jenkins控制檯界面，主要配置都在系統管理中

kubernetes1.13.1集羣結合ceph rbd部署最新版本jenkins

總結

使用ceph rbd 這種只能讀寫一次的設備不能用在線上，線上應該使用分佈式存儲例如nfs，cephfs，glusterfs等，這裏只是測試jenkins結合ceph，pv，pvc完成有狀態pod的測試

發表評論
所有評論
還沒有人評論，想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章

『高級篇』docker之kubernetes基礎集羣附加功能kube-proxy和kube-dns（

IT人故事 2019-02-23 14:07:34

《空門》

王清甫 2019-02-24 13:32:45

《霧》

王清甫 2019-02-24 13:32:45

grep，sed，awk

曉晶 2019-02-23 13:58:27

linux命令學習記錄一

ssjywlb 2019-02-23 13:58:00

>>>>2007年IT業回顧之哀兵糗事篇：華爲帶頭鑽法律空子<轉>

ahgyyubo 2019-02-23 13:56:41

CCNA實驗二：手動建立靜態路由表

蝸牛001號 2019-02-23 13:53:33

C#實現的18位身份證格式驗證算法

sccdliangl 2019-02-23 13:53:12

安裝mvwareworstation

zhumingqiang 2019-02-23 13:52:48

ShareTech 防火牆 使用手冊(適用型號：LB-2225)

yzy747 2019-02-23 13:50:59

科技部網站是這樣回答無聊問題的....

張富貴 2019-02-23 13:49:41

zabbix3.0安裝

民工騎士 2019-02-23 13:46:35

關於HTML標記的

pengtingwei 2019-02-23 13:42:40

修改收藏夾的默認路徑

zongxn 2019-02-23 13:41:43

useradd

63332128 2019-02-23 13:41:40