Linux下多路徑Multipath的簡單配置
1、啓用Multipath:
(1)啓動multipathd服務
#service multipathd start 或者 #/etc/init.d/multipathd start
(2)修改multipath配置文件/etc/multipath.conf:
a 默認情況下所以的設備都在multipath的黑名單中,所以即使啓動了multipathd服務並加在了內核模塊,multipath也不會對鏈路進行聚合,找到下面的3行並註釋掉(在行首加上#號)
#devnode_blacklist {
# devnode "*"
#}
b 默認情況下multipath生成dm設備之後,會同時在/dev/mapper/下生成以磁盤wwid爲名的符號鏈接指向對應的dm設備。如果想生成mpath設備,則需要打開user_friendly_names選項,將配置文件中下面3行的註釋取消(去掉行首的#號)
defaults {
user_friendly_names yes
}
(3)重啓multipathd服務(修改multipath.conf文件之後都應該重啓multipath服務)
(4)掃描磁盤
#multipath -v2
使用上面命令之後,系統中會出現鏈路聚合之後的dm設備,同時也會在/dev/mapper/、/dev/mpath/目錄下生成相應的設備。
查看multipath拓撲結構
#multipath -ll
另外一個重要的文件是/var/lib/multipath/bindings,這個文件中是磁盤的別名和wwid的對應關係,典型的例子是:
mpath0 3600508b4000c3af700008000002d0000
(5)需要注意的問題,multipath也會爲本地的磁盤生成相應的dm設備,所以需要在multipath.conf中將本地磁盤加入到黑名單,配置的方法可以參考下面的示例
devnode_blacklist {
wwid 36005076b08164f4011ae88b23efb95fe
devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
devnode "^hd[a-z]"
}
如上例所示,可以通過wwid或者設備名將本地磁盤加入到黑名單中。
2、固定multipath設備的命名:
通過wwid和設備別名一一對應的方式固定multipath設備的名稱,這些和別名對應的設備會被創建到/dev/mapper/目錄下,使用時直接使用這個目錄的的設備。
(1)通過/var/lib/multipath/bindings可以獲取所有磁盤的wwid,確定每個磁盤的別名之後,在/etc/multipath.conf中的multipaths段中加入相應的配置,如將wwid爲3600508b4000c3b910000c00000330000的磁盤命名爲etl01,wwid爲3600508b4000c3b910000c00000390000的磁盤命名爲etl02,配置文件如下所示
multipaths {
multipath {
wwid 3600508b4000c3b910000c00000330000
alias etl01
}
multipath {
wwid 3600508b4000c3b910000c00000390000
alias etl02
}
}
(2)配置完成之後,重啓multipathd服務,使用下面的命令清空已有的multipath記錄
#multipath -F
然後使用multipath -v2重新掃描設備,這時會在/dev/mapper/目錄下生成和別名對應的設備文件。
#ls /dev/mapper/
control etl01 eth02
(3)如果多臺服務器的存儲鏈路完全相同,並希望各服務器上同一磁盤的設備名相同,可以在一臺服務器上配置好別名綁定之後,將multipaths { }中間的配置複製到其他服務器,這樣各臺服務器/dev/mapper/下面的設備將會保持一致。
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
RHEL6與5在多路徑的配置上有一些小區別。scsi_id命令有些參數不再支持。
在RHEL 6中獲取磁盤的wwid可以通過–whitelist參數獲取:[root@db1 ~]# scsi_id --whitelist /dev/sdg (這個就是查看哪個iscsi磁盤wwid號的,也可以在/etc/multipath 下查看bindings wwids 這兩個文件可以看到wwid號)
可以通過一條shell命令,來獲取系統所有磁盤的wwid:
# for i in `cat /proc/partitions ¦ awk {'print $4'} ¦grep sd`; do echo "### $i: `scsi_id --whitelist /dev/$i`"; done在RHEL 5中,可以通過如下方式獲取磁盤wwid:
# for i in `cat /proc/partitions ¦ awk {'print $4'} ¦grep sd`; do echo "### $i: `scsi_id -g -u -s /block/$i`"; done另外,RHEL 6中,多路徑配置文件也出現變化:
- # multipath.conf written by anaconda
- defaults {
- user_friendly_names yes
- }
- blacklist {
- devnode "^(ramrawloopfdmddm-srscdst)[0-9]*"
- devnode "^hd[a-z]"
- devnode "^dcssblk[0-9]*"
- device {
- vendor "DGC"
- product "LUNZ"
- }
- device {
- vendor "IBM"
- product "S/390.*"
- }
- # don't count normal SATA devices as multipaths
- device {
- vendor "ATA"
- }
- # don't count 3ware devices as multipaths
- device {
- vendor "3ware"
- }
- device {
- vendor "AMCC"
- }
- # nor highpoint devices
- device {
- vendor "HPT"
- }
- wwid "20080930-1"
- wwid "20080930-1"
- device {
- vendor Cisco
- product Virtual_CD_DVD
- }
- wwid "*" //其實可以註釋這項,這樣就不需要單獨填寫blacklist_exceptions
- }
- blacklist_exceptions { //排除在黑名單之外的wwid
- wwid "360060160a2212f00a67e0b91f2dbe111"
- wwid "360060160a2212f0044a0fc6ef5eae111"
- }
- multipaths {
- multipath {
- uid 0 //磁盤讀所屬用戶uid
- gid 0 //磁盤所屬組gid
- wwid "360060160a2212f00a67e0b91f2dbe111" //wwid號
- mode 0600 //磁盤讀寫權限
- }
- multipath {
- wwid "360060160a2212f0044a0fc6ef5eae111"
- alias data //別名
- }
- ... ... //還可以根據實際情況,配置其它磁盤的別名、uid、gid、mode etc...
- }
配置完了之後,重啓multipathd服務,之後通過multipath -ll查看經過多路徑軟件綁定後的磁盤。
注意,如果要對磁盤進行格式化,請採用/dev/mapper/[alias]這類設備名進行fdisk。
****************************************************第二種*****************************************************
現在的Linux系統中,很多都會使用系統自帶的multipath多路徑軟件,在以前的版本中,我們一般通過multipath+udev或者multipath+rc.local來實現多路徑和權限設置,而在redhat 5.3-5.11的版本中multipath就直接可以實現多路徑聚合、設備持久化、用戶組設置
操作系統版本
[root@rac1
dev] #
uname -r 2.6.39-300.26.1.el5uek [root@rac1
dev] #
more /etc/issue Oracle
Linux Server release 5.9 Kernel
\r on an \m |
fdisk記錄
[root@rac1
dev] #
fdisk -l ………… Disk /dev/sdh :
134.2 GB, 134217728000 bytes 255
heads, 63 sectors /track ,
16317 cylinders Units
= cylinders of 16065 * 512 = 8225280 bytes Device
Boot Start End Blocks Id System Disk /dev/sdi :
33.5 GB, 33554432000 bytes 64
heads, 32 sectors /track ,
32000 cylinders Units
= cylinders of 2048 * 512 = 1048576 bytes Device
Boot Start End Blocks Id System |
multipath包
檢查安裝multipath相關包(該版本系統默認安裝)
[root@rac1
dev] #
rpm -aq|grep mapper device-mapper-multipath-libs-0.4.9-56.0.3.el5 device-mapper-event-1.02.67-2.el5 device-mapper-1.02.67-2.el5 device-mapper-multipath-0.4.9-56.0.3.el5 |
獲取wwid值
[root@rac1
dev] #
/sbin/scsi_id -g -u -s /block/sdh 14f504e46494c45527049754962662d395751372d68356743 [root@rac1
dev] #
/sbin/scsi_id -g -u -s /block/sdi 14f504e46494c4552484d486249782d464471382d354f4b58 |
獲取uid和gid
[root@rac1
dev] #
id grid uid=1100(grid)
gid=54321(oinstall) groups =54321(oinstall),1020(asmadmin),1021(asmdba) |
multipath.conf配置
[root@rac1
dev] #
vi /etc/multipath.conf defaults
{ user_friendly_names
no queue_without_daemon
no flush_on_last_del yes max_fds
max } blacklist
{ devnode "^hd[a-z]" devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*" devnode "^cciss.*" } devices
{ device
{ vendor "OPNFILER
" product "LUN" path_grouping_policy
group_by_prio features "3
queue_if_no_path pg_init_retries 50" getuid_callout "/sbin/scsi_id
-g -u -s /block/%n" path_checker
tur path_selector "round-robin
0" hardware_handler "1
alua" failback
immediate rr_weight
uniform rr_min_io
128 } } multipaths
{ multipath
{ wwid
14f504e46494c45527049754962662d395751372d68356743 #wwid alias xifenfei128 uid
1100 #uid gid
1020 #gid } multipath
{ wwid
14f504e46494c4552484d486249782d464471382d354f4b58 #wwid alias xifenfei32 uid
1100 #uid gid
1020 #gid } } |
啓動multipath
[root@rac1
dev] #
modprobe dm-multipath [root@rac1
dev] #
modprobe dm-round-robin [root@rac1
dev] #
chkconfig multipathd on [root@rac1
dev] #
service multipathd start Starting
multipathd daemon: [ OK ] [root@rac1
dev] #
multipath -F [root@rac1
dev] #
multipath -v2 create:
xifenfei128 (14f504e46494c45527049754962662d395751372d68356743) undef OPNFILER,VIRTUAL-DISK size=125G
features= '0' hwhandler= '0' wp=undef `-+-
policy= 'round-robin
0'
prio=1 status=undef `-
3:0:0:9 sdh 8:112 undef ready running create:
xifenfei32 (14f504e46494c4552484d486249782d464471382d354f4b58) undef OPNFILER,VIRTUAL-DISK size=31G
features= '0' hwhandler= '0' wp=undef `-+-
policy= 'round-robin
0'
prio=1 status=undef `-
3:0:0:10 sdi 8:128 undef ready running |
查看生成多路徑設備
注意設備名稱、組、用戶
[root@rac1
dev] #
ls -l /dev/mapper/xifenfei* brw-rw----
1 grid asmadmin 252, 2 Jan 7 21:21 /dev/mapper/xifenfei128 brw-rw----
1 grid asmadmin 252, 3 Jan 7 21:21 /dev/mapper/xifenfei32 |
補充Linux 6.x中udev設置所屬組和權限
對於linux 6.x,multipath不能設置磁盤所屬組和權限,可以通過udev進行實現,類似配置如下
[root@bxrac03
mapper] #cat
99-diskownership.rules SUBSYSTEM!= "block" ,
GOTO= "quickexit" KERNEL!= "dm-*" ,
GOTO= "quickexit" PROGRAM== "/sbin/dmsetup
info -c --noheadings -o name -m %m -j %M" RESULT== "*ocr*" ,
OWNER= "grid" ,
GROUP= "oinstall" ,
MODE= "0660" RESULT== "*oradata" ,
OWNER= "grid" ,
GROUP= "oinstall" ,
MODE= "0660" RESULT== "*backup" ,
OWNER= "grid" ,
GROUP= "oinstall" ,
MODE= "0660" LABEL= "quickexit" |
其中RESULT和dm的別名向匹配
#########################################################################################