基於Ansible的分佈式MPP數據庫Greenplum一鍵部署安裝包的設計實現

一、說在前面的廢話

最近在工作中研究分佈式MPP數據庫Greenplum的一鍵部署安裝包的製作，無意間在查看Greenplum的官網時發現了它基於ansible的相關文檔，於是開始深入瞭解ansible及ansible-playbook的使用，並順利實現了一個Greenplum的一鍵部署安裝包。

https://gpdb.docs.pivotal.io/6-7/install_guide/ansible-example.html

接下來介紹下ansible吧。

二、ansible概述

Ansible是一個開源配置管理工具，可以使用它來自動化任務，部署應用程序實現IT基礎架構。Ansible可以用來自動化日常任務，比如，服務器的初始化配置、安全基線配置、更新和打補丁系統，安裝軟件包等。Ansible架構相對比較簡單，僅需通過SSH連接客戶機執行任務即可：

ansible的工程目錄結構通常如下：

其優點總結如下：

• 無需客戶端

與Chef、Puppet以及Saltstack（現在也支持Agentless方式salt-ssh）不同，Ansible是無客戶端Agent的，所以無需在客戶機上安裝或配置任何程序，就可以運行Ansible任務。由於Ansible不會在客戶機上安裝任何軟件或運行監聽程序，因此消除了許多管理開銷，我們可以在即可上手使用Ansible管理服務器，同時Ansible的更新也不會影響任何客戶機。

• 使用SSH進行通訊

默認情況下，Ansible使用SSH協議在管理機和客戶機之間進行通信。可以使用SFTP與客戶機進行安全的文件傳輸。

• 並行執行

Ansible與客戶機並行通信，可以更快地運行自動化任務。默認情況下，forks值爲5，可以按需，在配置文件中增大該值。

三、基於ansible的Greenplum分佈式部署安裝包製作

經查閱Greenplum官方的安裝部署說明文檔，可以理順出greenplum在每臺機器節點上的安裝步驟，文檔地址：

https://gpdb.docs.pivotal.io/6-7/install_guide/install_guide.html

分析可知，絕大步驟在每臺機器上的配置基本相同，只有一些特定的安裝步驟需要在master節點上進行，大致內容如下：

• 【所有節點上】Greenplum各個主機節點的環境配置
• 【所有節點上】創建服務器Greenplum主機賬號
• 【所有節點上】Greenplum數據庫RPM包的安裝
• 【所有節點上】Greenplum數據目錄的配置
• 【所有節點上】賬號間免登錄配置
• 【master節點上】Greenplum數據庫的初始化與啓動

第一步驟：進行【所有節點上】的安裝與配置：

#!/usr/bin/env ansible-playbook
---

- hosts: all
  vars_files:
    - vars/gpdb.yml
  remote_user: root
  become: yes
  become_method: sudo
  connection: ssh
  gather_facts: yes
  tasks:
    - name: 01. stop and disable firewall service
      shell: '{{ item }}'
      with_items:
        - 'systemctl unmask firewalld'
        - 'systemctl start firewalld.service'
        - 'systemctl stop firewalld.service'
        - 'systemctl disable firewalld.service'
    - name: 02. be sure expect is installed
      yum: name=expect state=installed
    - name: 03. close selinux temporary
      shell: setenforce 0
      failed_when: false
    - name: 04. close selinux forever
      when: ansible_distribution == "CentOS" and ansible_distribution_major_version == "7"
      lineinfile:
        dest: /etc/selinux/config
        regexp: '^SELINUX='
        line: 'SELINUX=disabled'
    - name: 05. be sure ntp is installed
      yum: name=ntp state=installed
      tags: ntp
    - name: 06. configure sync time using aliyun server
      when: ansible_distribution == "CentOS" and ansible_distribution_major_version == "7"
      cron: name="sync time" minute='*/5' hour=* day=* month=* weekday=* job="/usr/sbin/ntpdate -u ntp1.aliyun.com >/dev/null 2>&1"
      ignore_errors: true
    - name: 07. update configure file for etc-hosts
      copy: src=gpnodes/hosts dest=/etc/hosts
    - name: 08. change host name to etc-hostname
      raw: 'echo {{hostname|quote}} > /etc/hostname'
    - name: 09. change host name by command hostname
      shell: hostname {{hostname|quote}}
    - name: 10. create greenplum admin user
      user:
        name: '{{ greenplum_admin_user }}'
        password: "{{ greenplum_admin_password | password_hash('sha512') }}"
    - name: 11. copy greeplum rpm package to host
      copy:
        src: '{{ package_path }}'
        dest: /tmp
    - name: 12. backing up sysctl
      copy:
        src: /etc/sysctl.conf
        remote_src: yes
        dest: /tmp/sysctl.conf.bak
        backup: yes
    - name: 13. get shmall 
      shell: echo $(expr $(getconf _PHYS_PAGES) / 2) 
      register: shmall
    - name: 14. get shmmax
      shell: echo $(expr $(getconf _PHYS_PAGES) / 2 \* $(getconf PAGE_SIZE))
      register: shmmax
    - name: 15. get min_free_kbytes
      shell: awk 'BEGIN {OFMT = "%.0f";} /MemTotal/ {print $2 * .03;}' /proc/meminfo
      register: min_free_kbytes
    - name: 16. set shmall
      sysctl:
        name: kernel.shmall
        value: '{{ shmall.stdout }}'
        reload: yes
    - name: 17. set shmmax
      sysctl:
        name: kernel.shmmax
        value: '{{ shmmax.stdout }}'
        reload: yes
    - name: 18. set min_free_kbytes
      sysctl:
        name: vm.min_free_kbytes
        value: '{{ min_free_kbytes.stdout }}'
        reload: yes
    - name: 19. configure sysctl
      sysctl:
        name: '{{ item.key }}'
        value: '{{ item.value }}'
        sysctl_set: yes
        state: present
        reload: yes
        ignoreerrors: yes
      with_dict:
        kernel.shmmni: 4096
        vm.overcommit_memory: 2
        vm.overcommit_ratio: 95
        net.ipv4.ip_local_port_range: 10000 65535
        kernel.sem: 500 2048000 200 40960
        kernel.sysrq: 1
        kernel.core_uses_pid: 1
        kernel.msgmnb: 65536
        kernel.msgmax: 65536
        kernel.msgmni: 2048
        net.ipv4.tcp_syncookies: 1
        net.ipv4.conf.default.accept_source_route: 0
        net.ipv4.tcp_max_syn_backlog: 4096
        net.ipv4.conf.all.arp_filter: 1
        net.core.netdev_max_backlog: 10000
        net.core.rmem_max: 2097152
        net.core.wmem_max: 2097152
        vm.swappiness: 0
        vm.zone_reclaim_mode: 0
        vm.dirty_expire_centisecs: 10
        vm.dirty_writeback_centisecs: 3
        vm.dirty_background_ratio: 10
        vm.dirty_ratio: 20
        vm.dirty_background_bytes: 0
        vm.dirty_bytes: 0 
    - name: 20. state PAM limits
      pam_limits:
        domain: '*'
        limit_type: '-'
        limit_item: '{{ item.key }}'
        value: '{{ item.value }}'
      with_dict:
        nofile: 655360
        nproc: 655360
        memlock: unlimited
        core: unlimited
    - name: 21. install package
      yum:
        name: '/tmp/{{ package_path | basename }}'
        # installroot: '{{ greenplum_install_directory }}'
        state: present
    - name: 22. cleanup package file from host
      file:
        path: '/tmp/{{ package_path | basename }}'
        state: absent
    - name: 23. find install directory
      find:
        paths: '{{ greenplum_install_directory }}'
        patterns: 'greenplum-db*'
        file_type: directory
      register: installed_dir
    - name: 24. change install directory ownership
      file:
        path: '{{ item.path }}'
        owner: '{{ greenplum_admin_user }}'
        group: '{{ greenplum_admin_user }}'
        recurse: yes
      with_items: '{{ installed_dir.files }}'
    - name: 25. update pam_limits
      pam_limits:
        domain: '{{ greenplum_admin_user }}'
        limit_type: '-'
        limit_item: '{{ item.key }}'
        value: '{{ item.value }}'
      with_dict:
        nofile: 524288
        nproc: 131072
    - name: 26. find installed greenplum version
      shell: . '{{ greenplum_install_directory }}'/greenplum-db/greenplum_path.sh && '{{ greenplum_install_directory }}'/greenplum-db/bin/postgres --gp-version
      register: postgres_gp_version
    - name: 27. fail if the correct greenplum version is not installed
      fail:
        msg: "Expected greenplum version {{ version }}, but found '{{ postgres_gp_version.stdout }}'"
      when: "version is not defined or version not in postgres_gp_version.stdout"
    - name: 28. Create data directory if it does not exist
      file:
        path: '{{ item }}'
        state: directory
        mode: '0755'
      with_items:
        - '{{ greenplum_data_directory }}/'
        - '{{ greenplum_data_directory }}/master/'
        - '{{ greenplum_data_directory }}/primary/'
        - '{{ greenplum_data_directory }}/mirror/'
    - name: 29. copy greenplum node temporary files
      copy:
        src: '{{ item }}'
        dest: '/home/{{ greenplum_admin_user }}/'
        remote_src: no
      with_items:
        - gpnodes/all_hosts
        - gpnodes/all_ips
        - gpnodes/master_hosts
        - gpnodes/standby_hosts
        - gpnodes/segment_hosts
    - name: 30. change data directory ownership
      file:
        path: '{{ greenplum_data_directory }}'
        owner: '{{ greenplum_admin_user }}'
        group: '{{ greenplum_admin_user }}'
        recurse: yes

第二步驟：進行【master節點上】的配置與數據庫初始化：

#!/usr/bin/env ansible-playbook
---

- hosts: all
  vars_files:
    - vars/gpdb.yml
  remote_user: root
  become: yes
  become_method: sudo
  connection: ssh
  gather_facts: yes
  tasks:
    - name: 31. copy files for initialize greenplum master
      copy:
        src: '{{ item }}'
        dest: '/home/{{ greenplum_admin_user }}/'
        remote_src: no
      with_items:
        - gpnodes/gpadmin_hosts
        - template/gpadmin_auto_ssh.sh
        - template/initdb_gpdb.sql
    - name: 32. replace greenplum admin user environment bash file
      template: src=template/gpadmin_bashrc.j2 dest=/home/{{ greenplum_admin_user }}/.bashrc
    - name: 33. copy and configure gpinitsystem config file
      template: src=template/gpinitsystem_config.j2 dest=/home/{{ greenplum_admin_user }}/gpinitsystem_config
    - name: 34. change data directory ownership
      file:
        path: '/home/{{ greenplum_admin_user }}/'
        owner: '{{ greenplum_admin_user }}'
        group: '{{ greenplum_admin_user }}'
        recurse: yes
    - name: 35. configure greenplum admin user auto login
      command: sh /home/{{ greenplum_admin_user }}/gpadmin_auto_ssh.sh /home/{{ greenplum_admin_user }}/gpadmin_hosts
      become: yes
      become_user: '{{ greenplum_admin_user }}'
    - name: 36. initialize greenplum master database
      shell: '{{ item }}'
      become: yes
      become_method: su
      become_flags: '-'
      become_user: '{{ greenplum_admin_user }}'
      with_items:
        - "gpinitsystem -a -c /home/{{ greenplum_admin_user }}/gpinitsystem_config -h /home/{{ greenplum_admin_user }}/segment_hosts -s smdw"
        - "psql -d postgres -U gpadmin -f /home/{{ greenplum_admin_user }}/initdb_gpdb.sql"
        - "echo \"host  all  all  0.0.0.0/0  password\" >> /home/{{ greenplum_admin_user }}/data/master/gpseg-1/pg_hba.conf"
        - "gpstop -u"

以上兩大步驟完成後，一個分佈式MPP數據庫Greenplum就很快搭建起來了。輔助編寫了很少的shell腳本即完成了一個靠人工大約要花費半天時間的工作量。

完整的安裝包製作項目源代碼請見：

項目地址：https://gitee.com/inrgihc/greenplum_installer

安裝文檔：https://gitee.com/inrgihc/greenplum_installer/wikis

使用起來也比較簡單：

git clone https://gitee.com/inrgihc/greenplum_installer.git
cd greenplum_installer/
make all
cd bin/
[root@localhost bin]# tree .
.
├── account.txt
└── greenplum6-centos7-release.bin

經過以上命令，即完成了一個CentOS7下Greenplum6.6.0分佈式安裝bin包的製作，接下來使用吧，先將greenplum6-centos7-release.bin文件上傳至10.101.1.10~10.101.1.13上任意一臺服務器上，然後以root身份按如下命令運行：

[root@localhost root]# cat account.txt 
10.101.1.10 root 123321
10.101.1.11 root 123321
10.101.1.12 root 123321
10.101.1.13 root 123321
[root@localhost bin]# tree .
.
├── account.txt
└── greenplum6-centos7-release.bin

0 directories, 2 files
[root@localhost bin]# sh ./greenplum6-centos7-release.bin ./account.txt  install

四、總結

ansible裏的知識還是很多的，當前也只是瞭解了最基本的使用，更多資料如下：

• (1) https://blog.csdn.net/workwithwebis3w/article/details/94617764
• (2) http://www.ansible.com.cn/index.html
• (3) https://docs.ansible.com/