實驗環境
OS: CentOS 6.6 soft: crmsh-2.1-1.6.x86_64.rpm drbd84-utils-8.9.1-1.el6.elrepo.x86_64.rpm kmod-drbd84-8.4.5-504.1.el6.x86_64.rpm pssh-debuginfo-2.3.1-2.el6.x86_64.rpm node1: hostname:node1.1inux.com IP: :172.16.66.81 node2: hostname:node2.1inux.com IP: :172.16.66.82
一、前期環境準備：
1、配置hostname
分別編輯node1 和node2 的 /etc/hosts 文件添加以下內容
172.16.66.81 node1.1inux.com node1 172.16.66.82 node2.1inux.com node2
2、設定 node1 、node2基於ssh祕鑰的認證
node1： [root@node1 ~]# ssh-keygen -t rsa -f /root/.ssh/id_rsa -P '' [root@node1 ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub [email protected] node2： [root@node2 ~]# ssh-keygen -t rsa -f /root/.ssh/id_rsa -P '' [root@node2 ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub [email protected]
3、設置時間同步
在node1上輸入以下命令：
[root@node1 ~]# ntpdate 172.16.0.1 ; ssh node2 'ntpdate 172.16.0.1' 31 May 22:43:40 ntpdate[1847]: adjust time server 172.16.0.1 offset 0.000252 sec 31 May 22:43:41 ntpdate[1907]: adjust time server 172.16.0.1 offset 0.000422 sec
驗證：
[root@node1 ~]# date; ssh node2 date Sun May 31 22:43:43 CST 2015 Sun May 31 22:43:43 CST 2015
4、分別在node1、node2上創建兩個大小都爲2G的分區作爲drbd設備 
讓內核讀取分區表：如圖3
[root@node1 ~]# partx -a /dev/sda //需多執行幾次
在node2上執行同樣的操作
二、安裝drbd軟件包
1、下載可信的安裝包，並安裝
drbd共有兩部分組成：內核模塊和用戶空間的管理工具。其中drbd內核模塊代碼已經整合進Linux內核2.6.33以後的版本中，因此，如果您的內核版本高於此版本的話，你只需要安裝管理工具即可；否則，您需要同時安裝內核模塊和管理工具兩個軟件包，並且此兩者的版本號一定要保持對應。
目前適用CentOS 5的drbd版本主要有8.0、8.2、8.3三個版本，其對應的rpm包的名字分別爲drbd, drbd82和drbd83，對應的內核模塊的名字分別爲kmod-drbd, kmod-drbd82和kmod-drbd83。而適用於CentOS 6的版本爲8.4，其對應的rpm包爲drbd和drbd-kmdl，但在實際選用時，要切記兩點：drbd和drbd-kmdl的版本要對應；另一個是drbd-kmdl的版本要與當前系統的內核版本相對應。
安裝包
drbd84-utils-8.9.1-1.el6.elrepo.x86_64.rpm
kmod-drbd84-8.4.5-504.1.el6.x86_64.rpm
node1: [root@node1 ~]# rpm -ivh drbd84-utils-8.9.1-1.el6.elrepo.x86_64.rpm kmod-drbd84-8.4.5-504.1.el6.x86_64.rpm node2: root@node2 ~]# rpm -ivh drbd84-utils-8.9.1-1.el6.elrepo.x86_64.rpm kmod-drbd84-8.4.5-504.1.el6.x86_64.rpm
2、 查看安裝生成的文件
------------------------------ [root@node1 ~]# rpm -ql drbd84-utils-8.9.1-1.el6.elrepo .... /etc/drbd.conf //配置文件 /etc/drbd.d /etc/drbd.d/global_common.conf //全局配置文件 ..... /sbin/drbdadm /sbin/drbdmeta /sbin/drbdsetup /usr/lib/drbd .... /usr/sbin/drbd-overview /usr/sbin/drbdadm /usr/sbin/drbdmeta /usr/sbin/drbdsetup /usr/share/cluster/drbd.metadata /usr/share/cluster/drbd.sh ..... /var/lib/drbd [root@node1 ~]# ------------------------------ [root@node1 ~]# rpm -ql kmod-drbd84-8.4.5-504.1.el6.x86_64 //安裝內核模塊 /etc/depmod.d/kmod-drbd84.conf /lib/modules/2.6.32-504.el6.x86_64 /lib/modules/2.6.32-504.el6.x86_64/extra /lib/modules/2.6.32-504.el6.x86_64/extra/drbd84 /lib/modules/2.6.32-504.el6.x86_64/extra/drbd84/drbd.ko //內核模塊 .... [root@node1 ~]#
三、配置DRBD
1、配置文件介紹
drbd的主配置文件爲/etc/drbd.conf；爲了管理的便捷性，目前通常會將些配置文件分成多個部分，且都保存至/etc/drbd.d/目錄中，主配置文件中僅使用"include"指令將這些配置文件片斷整合起來。通常，/etc/drbd.d目錄中的配置文件爲global_common.conf和所有以.res結尾的文件。其中global_common.conf中主要定義global段和common段，而每一個.res的文件用於定義一個資源。
在配置文件中，global段僅能出現一次，且如果所有的配置信息都保存至同一個配置文件中而不分開爲多個文件的話，global段必須位於配置文件的最開始處。目前global段中可以定義的參數僅有minor-count, dialog-refresh, disable-ip-verification和usage-count。
common段則用於定義被每一個資源默認繼承的參數，可以在資源定義中使用的參數都可以在common段中定義。實際應用中，common段並非必須，但建議將多個資源共享的參數定義爲common段中的參數以降低配置文件的複雜度。
resource段則用於定義drbd資源，每個資源通常定義在一個單獨的位於/etc/drbd.d目錄中的以.res結尾的文件中。資源在定義時必須爲其命名，名字可以由非空白的ASCII字符組成。每一個資源段的定義中至少要包含兩個host子段，以定義此資源關聯至的節點，其它參數均可以從common段或drbd的默認中進行繼承而無須定義。
/etc/drbd.conf
/etc/drbd.d/global_common.conf: 提供全局配置，及多個drbd設備相同的配置
/etc/drbd.d/*.res : 資源定義；
common:
*.res: 資源特有的配置
2、配置 /etc/drbd.d/global_common.conf
------------------
[root@node1 ~]# cat /etc/drbd.d/global_common.conf
---------------------
global { //全局屬性，定義drbd 自己的工作特性；
usage-count no; //此處更改爲no,大意是用戶體驗之類的，反饋信息，
# minor-count dialog-refresh disable-ip-verification
}
common { //通用屬性，定義多組drbd設備通用特性
handlers { //選擇默認
# These are EXAMPLE handlers only.
# They may have severe implications,
# like hard resetting the node under certain circumstances.
# Be careful when chosing your poison.
# pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
# pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
# local-io-error "/usr/lib/drbd/notify-io-error.sh; /usr/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger ; halt -f";
# fence-peer "/usr/lib/drbd/crm-fence-peer.sh";
# split-brain "/usr/lib/drbd/notify-split-brain.sh root";
# out-of-sync "/usr/lib/drbd/notify-out-of-sync.sh root";
# before-resync-target "/usr/lib/drbd/snapshot-resync-target-lvm.sh -p 15 -- -c 16k";
# after-resync-target /usr/lib/drbd/unsnapshot-resync-target-lvm.sh;
}
startup {
# wfc-timeout degr-wfc-timeout outdated-wfc-timeout wait-after-sb
}
options {
# cpu-mask on-no-data-accessible
}
disk {
on-io-error detach; //此處添加一行 ，意爲 一旦某個磁盤i/o發生錯誤 執行卸載
# size on-io-error fencing disk-barrier disk-flushes
# disk-drain md-flushes resync-rate resync-after al-extents
# c-plan-ahead c-delay-target c-fill-target c-max-rate
# c-min-rate disk-timeout
}
net { //定義網絡相關指令
protocol C;
cram-hmac-alg "sha1"; //消息校驗時使用的算法
shared-secret "iBtPDbodGmzDOKHTSa2h1g"; ////共享祕鑰 建議使用隨機生成數 {openssl rand -base64 16}
# protocol timeout max-epoch-size max-buffers unplug-watermark
# connect-int ping-int sndbuf-size rcvbuf-size ko-count
# allow-two-primaries cram-hmac-alg shared-secret after-sb-0pri
# after-sb-1pri after-sb-2pri always-asbp rr-conflict
# ping-timeout data-integrity-alg tcp-cork on-congestion
# congestion-fill congestion-extents csums-alg verify-alg
# use-rle
}
syncer { //定義同步速率
rate 500M ;
}
}
--------------------------------------------------3、定義一個res資源，配置如下
[root@node1 drbd.d]# vim mystore.res
resource mystore {
on node1.1inux.com { //定義在哪一個節點上
device /dev/drbd0; //定義設備文件名
disk /dev/sda3; //定義用哪個分區磁盤設備當做drbd設備
address 172.16.66.81:7789; //當前節點監聽的ip地址及端口
meta-disk internal; //定義源數據存放位置，internal表示磁盤自身
}
on node2.1inux.com {
device /dev/drbd0;
disk /dev/sda3;
address 172.16.66.82:7789;
meta-disk internal;
}
}
---------因爲res中相同選項時公共的，因此-也可以如下定義-------------
resource mystore {
device /dev/drbd0;
disk /dev/sda3;
meta-disk internal;
on node1.1inux.com {
address 172.16.66.81:7789; //drbd 默認端口 7789
}
on node2.1inux.com {
address 172.16.66.82:7789;
}
}
------------------------------------------以上文件在兩個節點上必須相同，因此，可以基於ssh將剛纔配置的文件全部同步至另外一個節點。
[root@node1 drbd.d]# scp /etc/drbd.d/* node2:/etc/drbd.d/
驗證是否copy成功：
[root@node1 drbd.d]# ssh node2 "ls -l /etc/drbd.d/" total 8 -rw-r--r-- 1 root root 2119 May 31 23:53 global_common.conf -rw-r--r-- 1 root root 207 May 31 23:53 mystore.res [root@node1 drbd.d]#
4、在兩個節點上初始化已定義的資源並啓動服務：
a）初始化資源，在Node1和Node2上分別執行：如圖4
[root@node1 drbd.d]# drbdadm create-md mystore
b）啓動服務，在Node1和Node2上分別執行：如圖5
/etc/init.d/drbd start
c)可以使用一下兩個命令查看資源狀態 如圖6
[root@node1 drbd.d]# cat /proc/drbd
[root@node1 drbd.d]# drbd-overview
------------- drbd-overview ------------------
0:mystore/0 Connected Secondary/Secondary Inconsistent/Inconsistent 
從上面的信息中可以看出此時兩個節點均處於Secondary狀態。於是，我們接下來需要將其中一個節點設置爲Primary。在要設置爲Primary的節點上執行如下命令：
# drbdadm primary --force mystore
也可以使用一下命令，將當前節點設置爲Primary
[root@node1 drbd.d]# drbdadm -- --overwrite-data-of-peer primary mystore
再次查看狀態 如圖7 ，發現節點已經有了主次
[root@node1 drbd.d]# drbd-overview 0:mystore/0 Connected Primary/Secondary UpToDate/UpToDate
5、創建文件系統
注：文件系統的掛載只能在Primary節點進行，因此，也只有在設置了主節點後才能對drbd設備進行格式化：
[root@node1 ~]# mke2fs -j -L DRBD /dev/drbd0
創建掛載點並掛載之
[root@node1 ~]# mkdir /mnt/drbd [root@node1 ~]# mount /dev/drbd0 /mnt/drbd [root@node1 ~]#
查看：
[root@node1 ~]# mount .... /dev/drbd0 on /mnt/drbd type ext3 (rw) //已成功掛載 [root@node1 ~]#
6、切換Primary和Secondary節點
對主Primary/Secondary模型的drbd服務來講，在某個時刻只能有一個節點爲Primary，因此，要切換兩個節點的角色，只能在先將原有的Primary節點設置爲Secondary後，才能將原來的Secondary節點設置爲Primary:
複製/etc/fstab 文件至/mnt/drbd 目錄下
[root@node1 ~]# cp /etc/fstab /mnt/drbd/ //爲了一會驗證數據 [root@node1 ~]# ls /mnt/drbd/ fstab lost+found
然後將node1 設置爲secondary
[root@node1 ~]# umount /mnt/drbd //注意設置爲secondary之前應先將其卸載 [root@node1 ~]# drbdadm secondary mystore [root@node1 ~]# drbd-overview 0:mystore/0 Connected Secondary/Secondary UpToDate/UpToDate //已經將爲Secondary [root@node1 ~]#
將node2 設置爲Primary
[root@node2 drbd.d]# mkdir /mnt/drbd [root@node2 drbd.d]# drbdadm primary mystore [root@node2 drbd.d]# drbd-overview 0:mystore/0 Connected Primary/Secondary UpToDate/UpToDate //OK ，已經更改爲Primary了 [root@node2 drbd.d]# mount /dev/drbd0 /mnt/drbd [root@node2 drbd.d]# ls /mnt/drbd //查看在node1上覆制的文件是否還在 fstab lost+found [root@node2 drbd.d]#
OK DRBD的基本配置已經完成 接下來我們安裝corosync
當一切正常後，設置不讓drbd服務自動啓動
[root@node1 ~]# chkconfig drbd off [root@node2 ~]# chkconfig drbd off
四、安裝配置 corosync 、pacemaker
1、按照corosync
在node1 上面安裝：
[root@node1 ~]# yum -y install corosync pacemaker
在node2上安裝
[root@node2 ~]# yum -y install corosync pacemaker
2、編輯配置文件
[root@node1 corosync]# cp corosync.conf.example corosync.conf
# vim corosync.conf
-------內容如下-----------------
# Please read the corosync.conf.5 manual page
compatibility: whitetank
totem {
version: 2
# secauth: Enable mutual node authentication. If you choose to
# enable this ("on"), then do remember to create a shared
# secret with "corosync-keygen".
secauth: on
threads: 0
# interface: define at least one interface to communicate
# over. If you define more than one interface stanza, you must
# also set rrp_mode.
interface {
# Rings must be consecutively numbered, starting at 0.
ringnumber: 0
# This is normally the *network* address of the
# interface to bind to. This ensures that you can use
# identical instances of this configuration file
# across all your cluster nodes, without having to
# modify this option.
bindnetaddr: 172.16.0.0
# However, if you have multiple physical network
# interfaces configured for the same subnet, then the
# network address alone is not sufficient to identify
# the interface Corosync should bind to. In that case,
# configure the *host* address of the interface
# instead:
# bindnetaddr: 192.168.1.1
# When selecting a multicast address, consider RFC
# 2365 (which, among other things, specifies that
# 239.255.x.x addresses are left to the discretion of
# the network administrator). Do not reuse multicast
# addresses across multiple Corosync clusters sharing
# the same network.
mcastaddr: 239.235.9.1
# Corosync uses the port you specify here for UDP
# messaging, and also the immediately preceding
# port. Thus if you set this to 5405, Corosync sends
# messages over UDP ports 5405 and 5404.
mcastport: 5405
# Time-to-live for cluster communication packets. The
# number of hops (routers) that this ring will allow
# itself to pass. Note that multicast routing must be
# specifically enabled on most network routers.
ttl: 1
}
}
logging {
# Log the source file and line where messages are being
# generated. When in doubt, leave off. Potentially useful for
# debugging.
fileline: off
# Log to standard error. When in doubt, set to no. Useful when
# running in the foreground (when invoking "corosync -f")
to_stderr: no
# Log to a log file. When set to "no", the "logfile" option
# must not be set.
to_logfile: yes
logfile: /var/log/cluster/corosync.log
# Log to the system log daemon. When in doubt, set to yes.
to_syslog: no
# Log debug messages (very verbose). When in doubt, leave off.
debug: off
# Log messages with time stamps. When in doubt, set to on
# (unless you are only logging to syslog, where double
# timestamps can be annoying).
timestamp: on
logger_subsys {
subsys: AMF
debug: off
}
}
service {
ver: 0
name: pacemaker
use_mgmtd: yes
}
aisexec {
user: root
group: root
}
------------------
3、生成corosync的密鑰文件
[root@node1 corosync]# corosync-keygen ... Press keys on your keyboard to generate entropy (bits = 1000). Writing corosync key to /etc/corosync/authkey.
3、查看網卡是否開啓了組播MULTICAST功能如果沒有開啓，要手動開啓
[root@node1 corosync]# ip addr show 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:e6:82:77 brd ff:ff:ff:ff:ff:ff [root@node1 corosync]#
4、將corosync.conf 和authkey複製到node2中
[root@node1 corosync]# scp authkey corosync.conf node2:/etc/corosync/
五、安裝crmsh
RHEL自6.4起不再提供集羣的命令行配置工具crmsh，轉而使用pcs；所以如果想使用crmsh可以自行安裝：
分別在node1和node2 上安裝crmsh和pssh
下載：
crmsh-2.1-1.6.x86_64.rpm
pssh-2.3.1-2.el6.x86_64.rpm
[root@node1 ~]# yum --nogpgcheck localinstall crmsh-2.1-1.6.x86_64.rpm pssh-2.3.1-2.el6.x86_64.rpm 在node2上執行： [root@node2 ~]# yum --nogpgcheck localinstall crmsh-2.1-1.6.x86_64.rpm pssh-2.3.1-2.el6.x86_64.rpm
六、驗證corosync 、crmsh 是否安裝成功
1、啓動
[root@node1 ~]# service corosync start Starting Corosync Cluster Engine (corosync): [ OK ]
2、查看端口
[root@node1 ~]# ss -tunl | grep :5405 udp UNCONN 0 0 172.16.66.81:5405 *:* udp UNCONN 0 0 239.235.9.1:5405 *:* [root@node1 ~]#
3、查看corosync引擎是否正常啓動：
[root@node1 ~]# grep -e "Corosync Cluster Engine" -e "configuration file" /var/log/cluster/corosync.log
Jun 01 01:10:24 corosync [MAIN ] Corosync Cluster Engine ('1.4.7'): started and ready to provide service.
Jun 01 01:10:24 corosync [MAIN ] Successfully read main configuration file '/etc/corosync/corosync.conf'.4、查看初始化成員節點通知是否正常發出：
[root@node1 ~]# grep TOTEM /var/log/cluster/corosync.log Jun 01 01:10:24 corosync [TOTEM ] Initializing transport (UDP/IP Multicast). Jun 01 01:10:24 corosync [TOTEM ] Initializing transmit/receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0). Jun 01 01:10:24 corosync [TOTEM ] The network interface [172.16.66.81] is now up. Jun 01 01:10:24 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
5、檢查啓動過程中是否有錯誤產生。下面的錯誤信息表示packmaker不久之後將不再作爲corosync的插件運行，因此，建議使用cman作爲集羣基礎架構服務；此處可安全忽略。
[root@node1 ~]# grep ERROR: /var/log/cluster/corosync.log | grep -v unpack_resources Jun 01 01:10:24 corosync [pcmk ] ERROR: process_ais_conf: You have configured a cluster using the Pacemaker plugin for Corosync. The plugin is not supported in this environment and will be removed very soon. Jun 01 01:10:24 corosync [pcmk ] ERROR: process_ais_conf: Please see Chapter 8 of 'Clusters from Scratch' (http://www.clusterlabs.org/doc) for details on using Pacemaker with CMAN Jun 01 01:10:25 corosync [pcmk ] ERROR: pcmk_wait_dispatch: Child process mgmtd exited (pid=12572, rc=100) [root@node1 ~]#
6、查看pacemaker是否正常啓動：
[root@node1 ~]# grep pcmk_startup /var/log/cluster/corosync.log Jun 01 01:10:24 corosync [pcmk ] info: pcmk_startup: CRM: Initialized Jun 01 01:10:24 corosync [pcmk ] Logging: Initialized pcmk_startup Jun 01 01:10:24 corosync [pcmk ] info: pcmk_startup: Maximum core file size is: 18446744073709551615 Jun 01 01:10:24 corosync [pcmk ] info: pcmk_startup: Service: 9 Jun 01 01:10:24 corosync [pcmk ] info: pcmk_startup: Local hostname: node1.1inux.com
7、如果上面命令執行均沒有問題，接着可以執行如下命令啓動node2上的corosync
[root@node1 ~]# ssh node2 -- /etc/init.d/corosync start Starting Corosync Cluster Engine (corosync): [ OK ]
注意：啓動node2需要在node1上使用如上命令進行，不要在node2節點上直接啓動。下面是node1上的相關日誌
[root@node1 ~]# tail /var/log/cluster/corosync.log Jun 01 01:15:31 [12570] node1.1inux.com pengine: info: determine_online_status: Node node2.1inux.com is online Jun 01 01:15:31 [12570] node1.1inux.com pengine: notice: stage6: Delaying fencing operations until there are resources to manage Jun 01 01:15:31 [12570] node1.1inux.com pengine: notice: process_pe_message: Calculated Transition 4: /var/lib/pacemaker/pengine/pe-input-4.bz2 Jun 01 01:15:31 [12570] node1.1inux.com pengine: notice: process_pe_message: Configuration ERRORs found during PE processing. Please run "crm_verify -L" to identify issues. Jun 01 01:15:31 [12571] node1.1inux.com crmd: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jun 01 01:15:31 [12571] node1.1inux.com crmd: info: do_te_invoke: Processing graph 4 (ref=pe_calc-dc-1433092531-26) derived from /var/lib/pacemaker/pengine/pe-input-4.bz2 Jun 01 01:15:31 [12571] node1.1inux.com crmd: notice: run_graph: Transition 4 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-4.bz2): Complete Jun 01 01:15:31 [12571] node1.1inux.com crmd: info: do_log: FSA: Input I_TE_SUCCESS from notify_crmd() received in state S_TRANSITION_ENGINE Jun 01 01:15:31 [12571] node1.1inux.com crmd: notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jun 01 01:15:36 [12566] node1.1inux.com cib: info: cib_process_ping: Reporting our current digest to node1.1inux.com: 636b6119a291743253db7001ca171e0d for 0.5.8 (0x254d3a0 0)
查看集羣節點的啓動狀態
[root@node1 ~]# crm status Last updated: Mon Jun 1 01:17:57 2015 Last change: Mon Jun 1 01:15:26 2015 Stack: classic openais (with plugin) Current DC: node1.1inux.com - partition with quorum Version: 1.1.11-97629de 2 Nodes configured, 2 expected votes 0 Resources configured Online: [ node1.1inux.com node2.1inux.com ] [root@node1 ~]#
禁用stonith [root@node1 ~]# crm configure property stonith-enabled=false 定義黏性值 [root@node1 ~]# crm configure property default-resource-stickiness=50 [root@node1 ~]# crm configure property no-quorum-policy=ignore
七、配置DRBD爲集羣服務
1、確保drbd服務不隨開機啓動，並且處於關閉狀態
[root@node2 ~]# chkconfig drbd off [root@node2 ~]# service drbd stop Stopping all DRBD resources: . [root@node2 ~]# drbd-overview 0:mystore/0 Unconfigured . . node1 操作同上
2、提供drbd的RA目前由OCF歸類爲linbit，其路徑爲/usr/lib/ocf/resource.d/linbit/drbd。我們可以使用如下命令來查看此RA及RA的meta信息：
# crm ra classes
heartbeat lsb ocf / heartbeat linbit pacemaker stonith
# crm ra list ocf linbit
drbd
# crm ra info ocf:linbit:drbd
This resource agent manages a DRBD resource as a master/slave resource. DRBD is a shared-nothing replicated storage device. (ocf:linbit:drbd) Master/Slave OCF Resource Agent for DRBD Parameters (* denotes required, [] the default): drbd_resource* (string): drbd resource name The name of the drbd resource from the drbd.conf file. drbdconf (string, [/etc/drbd.conf]): Path to drbd.conf Full path to the drbd.conf file. ..... Operations' defaults (advisory minimum): start timeout=240 promote timeout=90 demote timeout=90 notify timeout=90 stop timeout=100 monitor_Slave timeout=20 interval=20 monitor_Master timeout=20 interval=10
3、設置drbd爲集羣資源
drbd需要同時運行在兩個節點上，但只能有一個節點（primary/secondary模型）是Master，而另一個節點爲Slave；因此，它是一種比較特殊的集羣資源，其資源類型爲多態（Multi-state）clone類型，即主機節點有Master和Slave之分，且要求服務剛啓動時兩個節點都處於slave狀態。
configure# primitive mystor ocf:linbit:drbd params drbd_resource="mystore" op monitor role="Master" interval=10s timeout=20s op monitor role="Slave" interval=20s timeout=20s op start timeout=240s op stop timeout=100s
configure# ms ms_mystor mystor meta clone-max="2" clone-node-max="1" master-max="1" master-node-max="1" notify="true"
// 資源名： ms_mystor 、 源資源： mystor ； 屬性： clone-max="2" clone-node-max="1" master-max="1" master-node-max="1"
圖8 
4、查看狀態：
# crm status
圖9 
然後測試 讓node1 下線，查看狀態
# crm node standby
# crm status
如圖10 
5、掛載文件系統
分別在node1和node2上創建掛載點
#mkdir /mydata
crm(live)configure# primitive mydata ocf:heartbeat:Filesystem params device="/dev/drbd0" directory="/mydata" fstype="ext4" op monitor interval=20s timeout=40s op start timeout=60s op stop timeout=60s crm(live)configure# verify crm(live)configure# commit
創建約束
crm(live)configure# colocation mydata_with_ms_mystor_master inf: mydata ms_mystor:Master crm(live)configure# order mydata_after_ms_mystor_master Mandatory: ms_mystor:promote mydata:start crm(live)configure# verify crm(live)configure# commit
查看狀態：
[root@node1 ~]# crm status Last updated: Mon Jun 1 13:46:04 2015 Last change: Mon Jun 1 13:45:57 2015 Stack: classic openais (with plugin) Current DC: node1.1inux.com - partition with quorum Version: 1.1.11-97629de 2 Nodes configured, 2 expected votes 3 Resources configured Online: [ node1.1inux.com node2.1inux.com ] Master/Slave Set: ms_mystor [mystor] Masters: [ node2.1inux.com ] Slaves: [ node1.1inux.com ] mydata (ocf::heartbeat:Filesystem): Started node2.1inux.com [root@node1 ~]#
然後我們讓node2 下線，再次查看狀態
[root@node2 ~]# crm status
Last updated: Mon Jun 1 13:46:31 2015 Last change: Mon Jun 1 13:46:27 2015 Stack: classic openais (with plugin) Current DC: node1.1inux.com - partition with quorum Version: 1.1.11-97629de 2 Nodes configured, 2 expected votes 3 Resources configured Node node2.1inux.com: standby Online: [ node1.1inux.com ] Master/Slave Set: ms_mystor [mystor] Masters: [ node1.1inux.com ] Stopped: [ node2.1inux.com ] mydata (ocf::heartbeat:Filesystem): Started node1.1inux.com [root@node2 ~]#
在node1上查看掛載點
如圖11 
OK 現在基於corosync的drbd已經可以自動切換了
八、安裝mysql
1、創建mysql 用戶 【以下步驟在node1上操作】
[root@node1 ~]# groupadd -g 306 mysql && useradd -r -g 306 -u 306 mysql 驗證 [root@node1 ~]# grep mysql /etc/passwd mysql:x:306:306::/home/mysql:/bin/bash [root@node1 ~]#
2、創建目錄，並更改屬主、屬組
[root@node1 ~]# mkdir /mydata/data [root@node1 ~]# chown -R mysql:mysql /mydata/data
3、解壓並配置MariaDB
[root@node1 ~]# tar xf mariadb-5.5.43-linux-x86_64.tar.gz -C /usr/local/ [root@node1 local]# ln -sv mariadb-5.5.43-linux-x86_64/ mysql [root@node1 local]# cd mysql/ [root@node1 mysql]# chown -R root.mysql ./ [root@node1 local]# mkdir /etc/mysql [root@node1 mysql]# cp support-files/my-large.cnf /etc/mysql/my.cnf
[root@node1 mysql]# vim /etc/mysql/my.cnf
---------------- thread_concurrency = 8 datadir = /mydata/data innodb_file_per_table = on skip_name_resolve = on ---------------------
[root@node1 mysql]# cp support-files/mysql.server /etc/rc.d/init.d/mysqld [root@node1 mysql]# chmod +x /etc/rc.d/init.d/mysqld [root@node1 mysql]# chkconfig --add mysqld [root@node1 mysql]# chkconfig mysqld off
[root@node1 mysql]# echo "export PATH=/usr/local/mysql/bin:$PATH" > /etc/profile.d/mysql.sh [root@node1 mysql]# . /etc/profile.d/mysql.sh
[root@node1 mysql]# ln -sv /usr/local/mysql/include/ /usr/include/mysql [root@node1 mysql]# vim /etc/man.config 添加以下內容： MANPATH /usr/local/mysql/man
啓動MariaDB
[root@node2 mysql]# service mysqld start Starting MySQL.... [ OK ] [root@node2 mysql]#
4、初始化數據庫：
[root@node1 mysql]# ./scripts/mysql_install_db --user=mysql --datadir=/mydata/data
5、驗證數據庫是否初始化成功
如圖12
6、登陸並創建數據庫，如圖13
MariaDB [(none)]> GRANT ALL ON *.* TO "root"@"172.16.%.%" IDENTIFIED BY '1inux';
MariaDB [(none)]> FLUSH PRIVILEGES;
完成後退出數據庫，並下線node1，然後再node2上執行步驟1， 步驟3
完成後，在node2節點上連接數據庫，並查看，如圖14 ，在可以看到node1上創建的1inux數據庫
圖14 
退出mysql，並停止mysqld服務
#service mysqld stop #crm node syandby # crm node online
九、把MariaDB定義成高可用集羣的資源
crm(live)configure# primitive myip ocf:heartbeat:IPaddr params ip="172.16.66.88" op monitor interval=10s timeout=20s crm(live)configure# primitive myserver lsb:mysqld op monitor interval=20s timeout=20s crm(live)configure# verify crm(live)configure# commit crm(live)configure# colocation myip_with_ms_mystor_master inf: myip ms_mystor:Master crm(live)configure# colocation myserver_with_mydata inf: myserver mydata crm(live)configure# order myserver_after_mydata Mandatory: mydata:start myserver:start crm(live)configure# order myserver_after_myip Mandatory: myip:start myserver:start crm(live)configure# verify crm(live)configure# commit
crm(live)configure# show ---------------- node node1.1inux.com \ attributes standby=off node node2.1inux.com \ attributes standby=off primitive mydata Filesystem \ params device="/dev/drbd0" directory="/mydata" fstype=ext4 \ op monitor interval=20s timeout=40s \ op start timeout=60s interval=0 \ op stop timeout=60s interval=0 primitive myip IPaddr \ params ip=172.16.66.88 \ op monitor interval=10s timeout=20s primitive myserver lsb:mysqld \ op monitor interval=20s timeout=20s primitive mystor ocf:linbit:drbd \ params drbd_resource=mystore \ op monitor role=Master interval=10s timeout=20s \ op monitor role=Slave interval=20s timeout=20s \ op start timeout=240s interval=0 \ op stop timeout=100s interval=0 ms ms_mystor mystor \ meta clone-max=2 clone-node-max=1 master-max=1 master-node-max=1 notify=true colocation mydata_with_ms_mystor_master inf: mydata ms_mystor:Master colocation myip_with_ms_mystor_master inf: myip ms_mystor:Master colocation myserver_with_mydata inf: myserver mydata order mydata_after_ms_mystor_master Mandatory: ms_mystor:promote mydata:start order myserver_after_mydata Mandatory: mydata:start myserver:start order myserver_after_myip Mandatory: myip:start myserver:start property cib-bootstrap-options: \ dc-version=1.1.11-97629de \ cluster-infrastructure="classic openais (with plugin)" \ expected-quorum-votes=2 \ stonith-enabled=false \ default-resource-stickiness=50 \ no-quorum-policy=ignore \ last-lrm-refresh=1433127344
如圖15 
OK 基於corosync的 drbd高可用mysql 已經搭建完成
========試驗中錯誤信息及解決方案============================
查看HA 狀態信息
[root@node2 ~]# crm status
Last updated: Mon Jun 1 10:57:16 2015 Last change: Mon Jun 1 10:55:44 2015 Stack: classic openais (with plugin) Current DC: node1.1inux.com - partition with quorum Version: 1.1.11-97629de 2 Nodes configured, 2 expected votes 1 Resources configured Online: [ node1.1inux.com node2.1inux.com ] mystor (ocf::linbit:drbd): FAILED (unmanaged) [ node1.1inux.com node2.1inux.com ] Failed actions: mystor_stop_0 on node1.1inux.com 'not configured' (6): call=12, status=complete, last-rc-change='Mon Jun 1 10:55:44 2015', queued=0ms, exec=21ms mystor_stop_0 on node1.1inux.com 'not configured' (6): call=12, status=complete, last-rc-change='Mon Jun 1 10:55:44 2015', queued=0ms, exec=21ms mystor_monitor_0 on node2.1inux.com 'not configured' (6): call=5, status=complete, last-rc-change='Mon Jun 1 10:57:08 2015', queued=0ms, exec=46ms mystor_monitor_0 on node2.1inux.com 'not configured' (6): call=5, status=complete, last-rc-change='Mon Jun 1 10:57:08 2015', queued=0ms, exec=46ms [root@node2 ~]#
刪除資源
[root@node1 ~]# crm resource stop mystor [root@node1 ~]# crm configure delete mystor ERROR: resource mystor is running, can't delete it
顯示無法刪除
重啓corosync試試
重啓之前先清除一下節點狀態 【注 在node1 與node2 上都要清除並重啓 】
[root@node2 ~]# crm node clearstate node2.1inux.com Do you really want to drop state for node node2.1inux.com (y/n)? y [root@node2 ~]#
[root@node2 ~]# service corosync restart
Signaling Corosync Cluster Engine (corosync) to terminate: [ OK ] Waiting for corosync services to unload:...................[ OK ].................................................................. Starting Corosync Cluster Engine (corosync): [ OK ] [root@node2 ~]# [root@node2 ~]# crm status Last updated: Mon Jun 1 11:34:15 2015 Last change: Mon Jun 1 11:32:02 2015 Stack: classic openais (with plugin) Current DC: NONE 2 Nodes configured, 2 expected votes 1 Resources configured OFFLINE: [ node1.1inux.com node2.1inux.com ]
重啓後嘗試刪除節點 ：
[root@node2 ~]# crm configure delete mystor //OK 刪除成功 [root@node2 ~]#
接下來就可以重新配置了