Nginx+KeepAlive 高可用性實施

由於需要利用Nginx 在網通做一個電信的反代（動態網頁），目前公司所有的站點都在電信，那麼Nginx反代就需要做成高可用性，防止單點故障，於是就想到了做HA的強勁軟件 keepalived，keepalived做LVS健康檢測的資料倒是比較多，不過做雙機的資料不多，自己看了官方資料，利用VRRP做了雙機高可用，特此記錄：

1、服務器IP部署：
(1), Master（eth0:58.22.XXX.207/27接外網; eth1:10.2.1.207/8接內網）
(2), Backup（eth0:58.22.XXX.208/27接外網; eth1:10.2.1.208/8接內網）
VIP:58.22.XXX.202
網關：58.22.XXX.192

2、軟件下載解壓：
wget http://www.keepalived.org/software/keepalived-1.1.15.tar.gz
tar zxvf keepalived-1.1.15.tar.gz
cd keepalived-1.1.15

3、編譯安裝
./configure –prefix=/
make
make install

這裏需要動態鏈接庫的支持，因爲本例是使用Centos，所以可以直接使用yum 安裝：
libpopt-dev
libssl-dev


4、配置
A、主機Mater配置文件
vi /etc/keepalived/keepalived.conf

! Configuration File for keepalived

global_defs {
notification_email {
[email protected]
}
notification_email_from [email protected]
smtp_server smtp.139.com
smtp_connect_timeout 30
router_id LVS_DEVEL
}

vrrp_instance VI_1 {
state MASTER #(主機爲MASTER，備用機爲BACKUP)
interface eth0 #(HA監測網絡接口)
track_interface { #其他要監測狀態的接口
eth1
}
virtual_router_id 51 #(主、備機的virtual_router_id必須相同)
mcast_src_ip 58.22.XXX.207 #(多播的源IP，設置爲本機外網IP，與VIP同一網卡)

priority 500 #(主、備機取不同的優先級，主機值較大，備份機值較小,值越大優先級越高)
advert_int 1 #(VRRP Multicast廣播週期秒數)

authentication {
auth_type PASS #(VRRP認證方式)
auth_pass XXXXX #(密碼)
}
virtual_ipaddress {
58.22.XXX.202 #(VRRP HA虛擬地址)
}
}

這裏不做LVS，刪除後面的virtual_server內容部分

B、從機Backup配置文件

! Configuration File for keepalived

global_defs {
notification_email {
[email protected]
}
notification_email_from [email protected]
smtp_server smtp.139.com
smtp_connect_timeout 30
router_id LVS_DEVEL
}

vrrp_instance VI_1 {
state BACKUP #(與Master不同的地方)
interface eth0
track_interface {
eth1
}
virtual_router_id 51
mcast_src_ip 58.22.XXX.208
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass XXXX
}
virtual_ipaddress {
58.22.XXX.202
}
}

track_interface的意思是將Linux中你想監控的網絡接口卡監控起來，當其中的一塊出現故障是keepalived都將視爲路由器出現故障。

這裏請注意： virtual_router_id 51，同一組master/backup中，也就是如果你設置了多個VRRP，同一組這個ID必須相同，不然啓動的時候主從都會把VIP給啓動了，我就是在這裏鬱悶了很久，這裏是看了官方的反而把我搞得很鬱悶！！英文不夠強勁啊

5、啓動
在Master上

ip addr list
啓動keepalived 前

[root@L1-FC-XXX-207 keepalived-1.1.15]# ip addr list
1: lo: mtu 16436 qdisc noqueue
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 brd 127.255.255.255 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: mtu 1500 qdisc pfifo_fast qlen 100
link/ether 00:13:72:68:1c:d7 brd ff:ff:ff:ff:ff:ff
inet 58.22.XXX.207/27 brd 58.22.XXX.223 scope global eth0
inet6 fe80::213:72ff:fe68:1cd7/64 scope link
valid_lft forever preferred_lft forever
3: eth1: mtu 1500 qdisc pfifo_fast qlen 100
link/ether 00:13:72:68:1c:d8 brd ff:ff:ff:ff:ff:ff
inet 10.2.1.207/8 brd 10.255.255.255 scope global eth1
inet6 fe80::213:72ff:fe68:1cd8/64 scope link
valid_lft forever preferred_lft forever
4: sit0: mtu 1480 qdisc noop
link/sit 0.0.0.0 brd 0.0.0.0
啓動keepalived
/etc/init.d/keepalived start
啓動keepalived 後
1: lo: mtu 16436 qdisc noqueue
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 brd 127.255.255.255 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: mtu 1500 qdisc pfifo_fast qlen 100
link/ether 00:13:72:68:1c:d7 brd ff:ff:ff:ff:ff:ff
inet 58.22.XXX.207/27 brd 58.22.XXX.223 scope global eth0
inet 58.22.XXX.202/27 scope global secondary eth0
inet6 fe80::213:72ff:fe68:1cd7/64 scope link
valid_lft forever preferred_lft forever
3: eth1: mtu 1500 qdisc pfifo_fast qlen 100
link/ether 00:13:72:68:1c:d8 brd ff:ff:ff:ff:ff:ff
inet 10.2.1.207/8 brd 10.255.255.255 scope global eth1
inet6 fe80::213:72ff:fe68:1cd8/64 scope link
valid_lft forever preferred_lft forever
4: sit0: mtu 1480 qdisc noop
link/sit 0.0.0.0 brd 0.0.0.0

可以看到多了一個虛擬IP

inet 58.22.XXX.202/27 scope global secondary eth0

查看keepalived日誌

less /var/log/messages
Nov 19 14:27:59 FC-L103207 Keepalived: Terminating on signal
Nov 19 14:27:59 FC-L103207 Keepalived_vrrp: Terminating VRRP child process on signal
Nov 19 14:27:59 FC-L103207 Keepalived_vrrp: VRRP_Instance(VI_1) removing protocol VIPs.
Nov 19 14:27:59 FC-L103207 Keepalived: Stopping Keepalived v1.1.15 (11/19,2008)
Nov 19 14:27:59 FC-L103207 keepalived: keepalived shutdown succeeded
Nov 19 14:29:39 FC-L103207 keepalived: keepalived shutdown failed
Nov 19 14:29:39 FC-L103207 Keepalived: Starting Keepalived v1.1.15 (11/19,2008)
Nov 19 14:29:39 FC-L103207 Keepalived_vrrp: Using MII-BMSR NIC polling thread…
Nov 19 14:29:39 FC-L103207 Keepalived_vrrp: Registering Kernel netlink reflector
Nov 19 14:29:39 FC-L103207 Keepalived_vrrp: Registering Kernel netlink command channel
Nov 19 14:29:39 FC-L103207 Keepalived_vrrp: Registering gratutious ARP shared channel
Nov 19 14:29:39 FC-L103207 Keepalived: Starting VRRP child process, pid=7098
Nov 19 14:29:39 FC-L103207 Keepalived_vrrp: Opening file '/etc/keepalived/keepalived.conf'.
Nov 19 14:29:39 FC-L103207 Keepalived_vrrp: Configuration is using : 37126 Bytes
Nov 19 14:29:39 FC-L103207 keepalived: keepalived startup succeeded
Nov 19 14:29:39 FC-L103207 Keepalived_vrrp: VRRP sockpool: [ifindex(2), proto(112), fd(7,8)]
Nov 19 14:29:40 FC-L103207 Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE
Nov 19 14:29:41 FC-L103207 Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE
Nov 19 14:29:41 FC-L103207 Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs.
Nov 19 14:29:41 FC-L103207 Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 58.22.XXX.202

可以看出狀態爲MASTER

此時在BACKUP機上啓動keepalived，再查看日誌

Nov 19 14:29:56 FC-L103208 Keepalived: Terminating on signal
Nov 19 14:29:56 FC-L103208 Keepalived_vrrp: Terminating VRRP child process on signal
Nov 19 14:29:56 FC-L103208 Keepalived_vrrp: VRRP_Instance(VI_1) removing protocol VIPs.
Nov 19 14:29:56 FC-L103208 Keepalived: Stopping Keepalived v1.1.15 (11/19,2008)
Nov 19 14:29:56 FC-L103208 keepalived: keepalived shutdown succeeded
Nov 19 14:29:56 FC-L103208 Keepalived: Starting Keepalived v1.1.15 (11/19,2008)
Nov 19 14:29:56 FC-L103208 Keepalived_vrrp: Using MII-BMSR NIC polling thread…
Nov 19 14:29:56 FC-L103208 Keepalived: Starting VRRP child process, pid=20706
Nov 19 14:29:56 FC-L103208 Keepalived_vrrp: Registering Kernel netlink reflector
Nov 19 14:29:56 FC-L103208 Keepalived_vrrp: Registering Kernel netlink command channel
Nov 19 14:29:56 FC-L103208 Keepalived_vrrp: Registering gratutious ARP shared channel
Nov 19 14:29:56 FC-L103208 Keepalived_vrrp: Opening file '/etc/keepalived/keepalived.conf'.
Nov 19 14:29:56 FC-L103208 Keepalived_vrrp: Configuration is using : 37126 Bytes
Nov 19 14:29:56 FC-L103208 keepalived: keepalived startup succeeded
Nov 19 14:29:56 FC-L103208 Keepalived_vrrp: VRRP_Instance(VI_1) Entering BACKUP STATE
Nov 19 14:29:56 FC-L103208 Keepalived_vrrp: VRRP sockpool: [ifindex(2), proto(112), fd(7,8)]
此時如果停止MASTER上的keepalived

那麼在BACKUP的日誌會出現

Nov 19 14:33:55 FC-L103208 Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE
Nov 19 14:33:56 FC-L103208 Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE
Nov 19 14:33:56 FC-L103208 Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs.
Nov 19 14:33:56 FC-L103208 Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 58.22.103.202

即BACKUP接管了VIP
再啓動MASTER上的keepalived
日誌會出現：

Nov 19 14:34:01 FC-L103208 Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 58.22.103.202
Nov 19 14:35:40 FC-L103208 Keepalived_vrrp: VRRP_Instance(VI_1) Received higher prio advert
Nov 19 14:35:40 FC-L103208 Keepalived_vrrp: VRRP_Instance(VI_1) Entering BACKUP STATE
Nov 19 14:35:40 FC-L103208 Keepalived_vrrp: VRRP_Instance(VI_1) removing protocol VIPs.
則是說明有更高優先級的機器即MASTER啓動，去除VIP，還給更優先級的機器

這樣就實現瞭如果某一主機宕機，IP自動切換到備機上繼續運行

keepalived官方文檔：http://www.keepalived.org/documentation.html