構建Heartbeat+DRBD+NFS高可用文件共享存儲

實驗簡介：
本實驗部署DRBD + HEARDBEAT + NFS 環境，建立一個高可用(HA)的文件服務器集羣。在方案中，通過DRBD保證了服務器數據的完整性和一致性。DRBD類似於一個網絡RAID-1功能。當你將數據寫入本地文件系統時，數據還將會被髮送到網絡中另一臺主機上，以相同的形式記錄在一個另文件系統中。主節點與備節點的數據可以保證實時相互同步。當本地主服務器出現故障時，備份服務器上還會保留有一份相同的數據，可以繼續使用。在高可用(HA)中使用DRBD功能，可以代替使用一個共享盤陣。因爲數據同時存在於本地主服務器和備份服務器上。切換時，遠程主機只要使用它上面的那份備份數據，就可以繼續提供主服務器上相同的服務，並且client用戶對主服務器的故障無感知。
系統環境：
[root@localhost ~]# uname -r
2.6.18-164.el5
[root@localhost ~]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 5.4 (Tikanga)
拓撲圖：

一、DRBD的配置

node1主機配置
主機名配置
node1主機名配置
[root@localhost ~]# vim /etc/sysconfig/network
3 HOSTNAME=node1.a.com
[root@localhost ~]# hostname node1.a.com #配置後需註銷才能生效
[root@localhost ~]# hostname #查看主機名
node1.a.com
配置本地DNS解析
[root@node1 ~]# vim /etc/hosts
5 192.168.2.10 node1.a.com
6 192.168.2.11 node2.a.com

創建一個新的分區
查看分區
[root@node1 ~]# fdisk -l

Disk /dev/sda: 21.4 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id System
/dev/sda1   *           1          13      104391   83 Linux
/dev/sda2              14        1288    10241437+ 83 Linux
/dev/sda3            1289        1353      522112+ 82 Linux swap / Solaris
增加一個擴展分區
[root@node1 ~]# fdisk /dev/sda

The number of cylinders for this disk is set to 2610.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): p ##查看

Disk /dev/sda: 21.4 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id System
/dev/sda1   *           1          13      104391   83 Linux
/dev/sda2              14        1288    10241437+ 83 Linux
/dev/sda3            1289        1353      522112+ 82 Linux swap / Solaris

Command (m for help): n ##增加分區
Command action
e extended
p primary partition (1-4)
e ##擴展分區
Selected partition 4
First cylinder (1354-2610, default 1354):
Using default value 1354
Last cylinder or +size or +sizeM or +sizeK (1354-2610, default 2610):
Using default value 2610

Command (m for help): p ##查看

Disk /dev/sda: 21.4 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

增加一個邏輯分區
Command (m for help): n
First cylinder (1354-2610, default 1354):
Using default value 1354
Last cylinder or +size or +sizeM or +sizeK (1354-2610, default 2610): +1g

查看分區
Command (m for help): p

Disk /dev/sda: 21.4 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id System
/dev/sda1   *           1          13      104391   83 Linux
/dev/sda2              14        1288    10241437+ 83 Linux
/dev/sda3            1289        1353      522112+ 82 Linux swap / Solaris
/dev/sda4            1354        2610    10096852+   5 Extended
/dev/sda5            1354        1476      987966   83 Linux
保存退出
Command (m for help): w
內核重新識別分區
[root@node1 ~]# partprobe /dev/sda
[root@node1 ~]# cat /proc/partitions
major minor #blocks name

   8     0   20971520 sda
   8     1     104391 sda1
   8     2   10241437 sda2
   8     3     522112 sda3
   8     4          0 sda4
   8     5     987966 sda5
把下載好的rpm安裝包傳到node1上，如圖：

安裝drdb服務
[root@node1 ~]# rpm -ivh drbd83-8.3.8-1.el5.centos.i386.rpm
安裝內核模塊
[root@node1 ~]# rpm -ivh kmod-drbd83-8.3.8-1.el5.centos.i686.rpm
安裝之後形成的文件
[root@node1 ~]# rpm -ql drbd83 |less
[root@node1 ~]# rpm -ql kmod-drbd83 |less
將這兩個安裝包copy到node2，並安裝
[root@node1 ~]# scp *.rpm node2.a.com:/root/
The authenticity of host 'node2.a.com (192.168.2.11)' can't be established.
RSA key fingerprint is 3a:64:a0:6e:d0:21:14:6c:e9:1e:84:50:89:42:05:29.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'node2.a.com,192.168.2.11' (RSA) to the list of known hosts.
[email protected]'s password:
drbd83-8.3.8-1.el5.centos.i386.rpm                100% 217KB 216.7KB/s   00:00
kmod-drbd83-8.3.8-1.el5.centos.i686.rpm           100% 123KB 123.0KB/s   00:00

編輯配置文件
[root@node1 ~]# vim /etc/drbd.conf
#
# please have a a look at the example configuration file in
# /usr/share/doc/drbd83/drbd.conf
#
進入底行模式下，進行讀取那個文件
:read /usr/share/doc/drbd83-8.3.8/drbd.conf
可以看到
include "drbd.d/global_common.conf";通用配置的文件
include "drbd.d/*.res"; 定義資源的文件
下面編輯通用的配置文件
[root@node1 ~]# cd /etc/drbd.d/
爲了安全，先把文件做個備份
[root@node1 drbd.d]# cp global_common.conf global_common.conf.bak
[root@node1 drbd.d]# vim global_common.conf
編輯好的文件如下：

global {
usage-count yes;
# minor-count dialog-refresh disable-ip-verification
}

common {
protocol C;

        startup {
                wfc-timeout 120;
                degr-wfc-timeout 120;
         }
        disk {
                  on-io-error detach;
                  fencing resource-only;

          }
        net {
                cram-hmac-alg "sha1";
                shared-secret "mydrbdlab";
         }
        syncer {
                  rate 100M;
         }

}

編輯定義資源文件：
[root@node1 drbd.d]# vim web.res 名字自己隨意起
resource web {
        on node1.a.com {
        device   /dev/drbd0;
        disk    /dev/sda5;
        address 192.168.2.10:7789;
        meta-disk       internal;
        }

        on node2.a.com {
        device   /dev/drbd0;
        disk    /dev/sda5;
        address 192.168.2.11:7789;
        meta-disk       internal;
        }
}

然後將這些配置文件copy到node2上
[root@node1 drbd.d]# scp * node2.a.com:/etc/drbd.d/
[email protected]'s password:
global_common.conf                                100% 506     0.5KB/s   00:00
global_common.conf.bak                            100% 1418     1.4KB/s   00:00
web.res                                           100% 347     0.3KB/s   00:00
[root@node1 drbd.d]# scp /etc/drbd.conf node2.a.com:/etc/
[email protected]'s password:
drbd.conf                                         100% 100     0.1KB/s   00:00
[root@node1 drbd.d]#

node2主機配置
主機名配置
[root@localhost ~]# vim /etc/sysconfig/network
3 HOSTNAME=node2.a.com
[root@localhost ~]# hostname node2.a.com

#配置後需註銷才能生效
[root@localhost ~]# hostname #查看主機名
node1.a.com
配置本地DNS解析
[root@node2 ~]# vim /etc/hosts
5 192.168.2.10 node1.a.com
6 192.168.2.11 node2.a.com
增加分區
[root@node2 ~]# fdisk -l

Disk /dev/sda: 21.4 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id System
/dev/sda1   *           1          13      104391   83 Linux
/dev/sda2              14        1288    10241437+ 83 Linux
/dev/sda3            1289        1353      522112+ 82 Linux swap / Solaris
[root@node2 ~]# fdisk /dev/sda

Command (m for help): n
Command action
e extended
p primary partition (1-4)
e
Selected partition 4
First cylinder (1354-2610, default 1354):
Using default value 1354
Last cylinder or +size or +sizeM or +sizeK (1354-2610, default 2610):
Using default value 2610

Command (m for help): p

Disk /dev/sda: 21.4 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Command (m for help): n
First cylinder (1354-2610, default 1354):
Using default value 1354
Last cylinder or +size or +sizeM or +sizeK (1354-2610, default 2610): +1g

Command (m for help): p

Disk /dev/sda: 21.4 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table.
The new table will be used at the next reboot.
Syncing disks.
[root@node2 ~]# partprobe /dev/sda
[root@node2 ~]# cat /proc/partitions
major minor #blocks name

   8     0   20971520 sda
   8     1     104391 sda1
   8     2   10241437 sda2
   8     3     522112 sda3
   8     4          0 sda4
   8     5     987966 sda5
安裝drbd服務
[root@node2 ~]# rpm -ivh drbd83-8.3.8-1.el5.centos.i386.rpm
warning: drbd83-8.3.8-1.el5.centos.i386.rpm: Header V3 DSA signature: NOKEY, key ID e8562897
Preparing...                ########################################### [100%]
   1:drbd83                 ########################################### [100%]
[root@node2 ~]# rpm -ivh kmod-drbd83-8.3.8-1.el5.centos.i686.rpm
warning: kmod-drbd83-8.3.8-1.el5.centos.i686.rpm: Header V3 DSA signature: NOKEY, key ID e8562897
Preparing...                ########################################### [100%]
   1:kmod-drbd83            ########################################### [100%]

以上做完之後，在node1，node2上面初始化
[root@node1 drbd.d]# drbdadm create-md web
Writing meta data...
initializing activity log
NOT initialized bitmap
New drbd meta data block successfully created.

[root@node2 drbd.d]# drbdadm create-md web
Writing meta data...
initializing activity log
NOT initialized bitmap
New drbd meta data block successfully created.
在雙方的節點上啓動服務
[root@node1 drbd.d]# service drbd start
Starting DRBD resources: [
web
Found valid meta data in the expected location, 1011671040 bytes into /dev/sda5.
d(web) s(web) n(web) ]..........
***************************************************************
DRBD's startup script waits for the peer node(s) to appear.
- In case this node was already a degraded cluster before the
   reboot the timeout is 120 seconds. [degr-wfc-timeout]
- If the peer was available before the reboot the timeout will
   expire after 120 seconds. [wfc-timeout]
   (These values are for resource 'web'; 0 sec -> wait forever)
To abort waiting enter 'yes' [ 33]:

[root@node2 drbd.d]# service drbd start
Starting DRBD resources: [
web
Found valid meta data in the expected location, 1011671040 bytes into /dev/sda5.
d(web) s(web) n(web) ].

查看狀態
如圖：

可以看到兩個節點都是Secondary狀態，沒有同步
下面把node1設爲主設備，要在node1上執行下面命令
[root@node1 drbd.d]# drbdadm -- --overwrite-data-of-peer primary web
可以查看同步過程
[root@node1 drbd.d]# watch -n 1 'cat /proc/drbd'
查看狀態，可以看到node1爲Primary，node2爲Secondary
[root@node1 drbd.d]# drbd-overview
0:web Connected Primary/Secondary UpToDate/UpToDate C r----
[root@node2 drbd.d]# drbd-overview
0:web Connected Secondary/Primary UpToDate/UpToDate C r----

創建文件系統（在主節點上實現）
先把分區進行格式化，如圖：
[root@node1 drbd.d]# mkfs -t ext3 -L drbdweb /dev/drbd0

掛載分區
在node1上創建掛載點，並進行掛載
[root@node1 drbd.d]# mkdir /mnt/web
[root@node1 drbd.d]# mount /dev/drbd0 /mnt/web
[root@node1 drbd.d]# df -h
Filesystem            Size Used Avail Use% Mounted on
/dev/sda2             9.5G 2.6G 6.5G 29% /
/dev/sda1              99M   12M   83M 12% /boot
tmpfs                 125M     0 125M   0% /dev/shm
/dev/drbd0            950M   18M 885M   2% /mnt/web
在/mnt/web上寫個文件，測試同步
[root@node1 drbd.d]# cd /mnt/web/
[root@node1 web]# echo "hello" >index.html
現在要把node1變成Secondary，node2變成Primary
先卸載，再降級
[root@node1 ~]# umount /mnt/web
[root@node1 ~]# drbdadm secondary web
在node2上執行
[root@node2 drbd.d]# drbdadm   primary web
查看狀態
[root@node1 ~]# drbd-overview
0:web Connected Secondary/Primary UpToDate/UpToDate C r----
[root@node1 ~]#
[root@node2 drbd.d]# drbd-overview
0:web Connected Primary/Secondary UpToDate/UpToDate C r----
還可以用下面命令查看主從狀態
[root@node1 ~]# drbdadm role web
在node2上創建掛載點，並進行掛載
[root@node2 drbd.d]# mkdir /mnt/web
[root@node2 drbd.d]# mount /dev/drbd0 /mnt/web/
[root@node2 drbd.d]# df -h
Filesystem            Size Used Avail Use% Mounted on
/dev/sda2             9.5G 2.6G 6.5G 29% /
/dev/sda1              99M   12M   83M 12% /boot
tmpfs                 125M     0 125M   0% /dev/shm
/dev/drbd0            950M   18M 885M   2% /mnt/web
查看一下是否有剛纔創建的文件，如圖：

二、NFS的配置
兩臺服務器的nfs配置必須要一致
node1的配置
修改相關文件
[root@node1 ~]# vim /etc/exports
/mnt/web *(rw,sync,insecure,no_root_squash,no_wdelay)
執行相關操作
[root@node1 ~]# service portmap start && chkconfig portmap on
Starting portmap:                                          [ OK ]
[root@node1 ~]# service nfs start && chkconfig nfs on
Starting NFS services:                                     [ OK ]
Starting NFS quotas:                                       [ OK ]
Starting NFS daemon:                                       [ OK ]
Starting NFS mountd:                                       [ OK ]
修改nfs啓動腳本
[root@node1 ~]# vim /etc/init.d/nfs
122         killproc nfsd -9
node2的配置
修改相關文件
[root@node2 ~]# vim /etc/exports
/mnt/web *(rw,sync,insecure,no_root_squash,no_wdelay)
執行相關操作
[root@node2 ~]# service portmap start && chkconfig portmap on
Starting portmap:                                          [ OK ]
[root@node2 ~]# service nfs start && chkconfig nfs on
Starting NFS services:                                     [ OK ]
Starting NFS quotas:                                       [ OK ]
Starting NFS daemon:                                       [ OK ]
Starting NFS mountd:                                       [ OK ]
修改nfs啓動腳本
[root@node2 ~]# vim /etc/init.d/nfs
122         killproc nfsd -9

三、Heartbeat配置
Heartbeat的配置要求兩個節點的配置一致
在node1上進行如下操作

上傳相關軟件包，如圖：

安裝相關軟件包：（配置好yum環境）
[root@node1 ~]# yum install -y heartbeat-2.1.4-9.el5.i386.rpm heartbeat-pils-2.1.4-10.el5.i386.rpm heartbeat-stonith-2.1.4-10.el5.i386.rpm libnet-1.1.4-3.el5.i386.rpm perl-MailTools-1.77-1.el5.noarch.rpm --nogpgcheck
複製模版文件
[root@node1 ~]# cd /usr/share/doc/heartbeat-2.1.4/
[root@node1 heartbeat-2.1.4]# cp ha.cf haresources authkeys /etc/ha.d/
修改相關的配置信息
[root@node1 heartbeat-2.1.4]# cd /etc/ha.d/
[root@node1 ha.d]# vim ha.cf
24 debugfile /var/log/ha-debug
29 logfile /var/log/ha-log
34 logfacility local0
48 keepalive 2
56 deadtime 10
76 udpport 694
122 bcast eth0
158 auto_failback off
215 node node1.a.com
216 node node2.a.com
修改資源文件
[root@node1 ha.d]# echo "node1.a.com IPaddr::192.168.2.9/24/eth0/192.168.2.255 drbddisk::web Filesystem::/dev/drbd0::/mnt/web::ext3 killnfsd" >>/etc/ha.d/haresources
修改key文件
[root@node1 ha.d]# vim authkeys
23 auth 1
24 1 crc
手工創建文件
[root@node1 ha.d]# cd resource.d/
[root@node1 resource.d]# echo "killall -9 nfsd ; /etc/init.d/nfs restart ; exit 0" >>/etc/ha.d/resource.d/killnfsd
修改配置文件的權限
[root@node1 ha.d]# chmod 600 authkeys
[root@node1 ha.d]# chmod 755 resource.d/killnfsd
啓動服務
[root@node1 ha.d]# service heartbeat start

在node2上進行如下操作
安裝相關軟件包：（配置好yum環境）
[root@node2 ~]# yum install -y heartbeat-2.1.4-9.el5.i386.rpm heartbeat-pils-2.1.4-10.el5.i386.rpm heartbeat-stonith-2.1.4-10.el5.i386.rpm libnet-1.1.4-3.el5.i386.rpm perl-MailTools-1.77-1.el5.noarch.rpm --nogpgcheck
複製模版文件
[root@node2 ~]# cd /usr/share/doc/heartbeat-2.1.4/
[root@node2 heartbeat-2.1.4]# cp ha.cf haresources authkeys /etc/ha.d/
修改相關的配置信息
[root@node2 heartbeat-2.1.4]# cd /etc/ha.d/
[root@node2 ha.d]# vim ha.cf
24 debugfile /var/log/ha-debug
29 logfile /var/log/ha-log
34 logfacility local0
48 keepalive 2
56 deadtime 10
76 udpport 694
122 bcast eth0
158 auto_failback off
215 node node1.a.com
216 node node2.a.com
修改資源文件
[root@node2 ha.d]# echo "node1.a.com IPaddr::192.168.2.9/24/eth0/192.168.2.255 drbddisk::web Filesystem::/dev/drbd0::/mnt/web::ext3 killnfsd" >>/etc/ha.d/haresources
修改key文件
[root@node2 ha.d]# vim authkeys
23 auth 1
24 1 crc
手工創建文件
[root@node2 ha.d]# cd resource.d/
[root@node2 resource.d]# echo "killall -9 nfsd ; /etc/init.d/nfs restart ; exit 0" >>/etc/ha.d/resource.d/killnfsd
修改配置文件的權限
[root@node2 ha.d]# chmod 600 authkeys
[root@node2 ha.d]# chmod 755 resource.d/killnfsd
啓動服務
[root@node2 ha.d]# service heartbeat start

查看網絡信息，如圖：

在客戶機上測試：
新建掛載點，並掛載
[root@localhost ~]# mkdir /mnt/nfs
[root@localhost ~]# mount 192.168.2.9:/mnt/web /mnt/nfs
[root@localhost ~]# df -h
Filesystem            Size Used Avail Use% Mounted on
/dev/sda2             9.7G 2.6G 6.7G 28% /
/dev/sda1              99M   12M   83M 12% /boot
tmpfs                 125M     0 125M   0% /dev/shm
/dev/hdc              2.8G 2.8G     0 100% /mnt/cdrom
192.168.2.9:/mnt/web 950M   18M 885M   2% /mnt/nfs

查看磁盤上的文件：

現在node1，node2的狀態：
[root@node1 ~]# drbdadm role web
Primary/Secondary
[root@node2 ~]# drbdadm role web
Secondary/Primary

在客戶端寫個腳本文件測試：

[root@localhost ~]# vim /mnt/test.sh

while true
do
echo ------\>trying touch x:`date`
touch x
echo \<-----done touch x:`date`
echo
sleep 1
done

[root@localhost ~]# cd /mnt/nfs/
[root@localhost nfs]# bash /mnt/test.sh
------>trying touch x:Thu Oct 18 18:40:03 CST 2012
<-----done touch x:Thu Oct 18 18:40:03 CST 2012

------>trying touch x:Thu Oct 18 18:40:05 CST 2012
<-----done touch x:Thu Oct 18 18:40:05 CST 2012

------>trying touch x:Thu Oct 18 18:40:07 CST 2012
<-----done touch x:Thu Oct 18 18:40:07 CST 2012

------>trying touch x:Thu Oct 18 18:40:09 CST 2012
<-----done touch x:Thu Oct 18 18:40:09 CST 2012

下面在主節點node1上把heartbeat停止服務：
[root@node1 ~]# service heartbeat stop
Stopping High-Availability services:
[ OK ]

查看腳本執行情況：

------>trying touch x:Thu Oct 18 18:40:15 CST 2012
<-----done touch x:Thu Oct 18 18:40:15 CST 2012

------>trying touch x:Thu Oct 18 18:40:17 CST 2012
touch: cannot touch `x': Stale NFS file handle
<-----done touch x:Thu Oct 18 18:40:17 CST 2012

------>trying touch x:Thu Oct 18 18:41:07 CST 2012
<-----done touch x:Thu Oct 18 18:41:07 CST 2012

可以看到有丟失現象，但很快恢復正常

構建Heartbeat+DRBD+NFS高可用文件共享存儲

《Python進階》學習筆記

Leetcode 3161. 物塊放置查詢

leetcode 60 排列序列

一個docker容器暴露多個端口

微服務實踐之使用 Visual Studio 2022 調試Dapr 應用程序

wpf附加屬性理解 WPF附加屬性

用源代碼搭建lnmp環境

用pure-ftpd搭建ftp服務器

oracle下載（轉載）

SNMP的配置及應用

我的友情鏈接

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結