keepalived+lvs實現高可用的負載均衡

###############################################

keepalived

keepalived+lvs實現高可用的負載均衡

測試

###############################################

keepalived

keepalived一款輕量級高可用軟件，工作於layer3, 4 & 5，不同於前幾篇博文中的Heartbeat、Corosync等軟件的實現機制不同，它採用虛擬路由冗餘協議（Virual Router Redundancy Protocal）來實現並且完美的與lvs結合，由於底層使用虛擬路由冗餘協議，因此Keepalived具有切換速度快的特點，工作在layer3的keepalived定期向服務器羣組中發送ICMP數據包宣告自己存活與否，工作在layer3的keepalived支持以檢測TCP端口狀態的方式來判定後臺Realserver故障與否，自動並將那些判定爲故障的後臺Realserver從ipvs規則中踢出，工作在layer5可以支持用戶自動以腳本來實現相應的智能操作。此lvs也可以結合ldirectord來實現對後臺realserver的動態監測，相對於keepalived來說ldirectord屬於重量級別的，部署和使用的靈活程度沒有前者方便，本文將介紹keepalived。

keepalived+lvs實現高可用的負載均衡

架構圖：

realserver端腳本

#!/bin/bash
#
# Script to start LVS DR real server.
# description: LVS DR real server
#
.  /etc/rc.d/init.d/functions
VIP=192.168.1.33
host=`/bin/hostname`
case "$1" in
start)
       # Start LVS-DR real server on this machine.
        /sbin/ifconfig lo down
        /sbin/ifconfig lo up
        echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
        echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
        echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
        echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
        /sbin/ifconfig lo:0 $VIP broadcast $VIP netmask 255.255.255.255 up
        /sbin/route add -host $VIP dev lo:0
;;
stop)
        # Stop LVS-DR real server loopback device(s).
        /sbin/ifconfig lo:0 down
        echo 0 > /proc/sys/net/ipv4/conf/lo/arp_ignore
        echo 0 > /proc/sys/net/ipv4/conf/lo/arp_announce
        echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore
        echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce
;;
status)
        # Status of LVS-DR real server.
        islothere=`/sbin/ifconfig lo:0 | grep $VIP`
        isrothere=`netstat -rn | grep "lo:0" | grep $VIP`
        if [ ! "$islothere" -o ! "isrothere" ];then
            # Either the route or the lo:0 device
            # not found.
            echo "LVS-DR real server Stopped."
        else
            echo "LVS-DR real server Running."
        fi
;;
*)
            # Invalid entry.
            echo "$0: Usage: $0 {start|status|stop}"
            exit 1
;;
esac

安裝httpd並建立測試頁面如下：

Director端配置

安裝ipvsadm和keepalived

yum install ipvsadm
rpm -ivh  keepalived-1.2.7-5.el5.i386.rpm

director_master的配置vim /etc/keepalived/keepalived.conf

! Configuration File for keepalived
global_defs {
   notification_email {
        root@localhost   #報警收件人地址
   }
   notification_email_from root@localhost  #報警發件人地址
   smtp_server 127.0.0.1                   #設置smtp服務地址
   smtp_connect_timeout 30                 #設置連接smtp服務的超時時間
   router_id LVS_DEVEL                     #發送郵件的主體信息
}
vrrp_script chk_schedown {                 #自定義腳本
   script "[ -e /etc/keepalived/down ] && exit 1 || exit 0"
   interval 1    #重試時間間隔
   weight -5     #減權重
   fall 2
   rise 1
}
vrrp_instance VI_1 {
    state MASTER             #制定keepalived角色
    interface eth0           #制定檢測網絡接口
    virtual_router_id 54     #虛擬路由標示碼
    priority 100             #權重，1-255之間
    advert_int 1             #設置同步檢查的時間間隔，單位是秒
    authentication {
        auth_type PASS       #驗證類型爲PASS
        auth_pass soulboy    #驗證密碼
    }
    virtual_ipaddress {
        192.168.1.33/24 dev eth0 label eth0:0  #設置虛擬IP
    }
     track_script {
        chk_schedown
    }
    notify_master "/etc/keepalived/notify.sh -n master -a 192.168.1.33"
    notify_backup "/etc/keepalived/notify.sh -n backup -a 192.168.1.33"
    notify_fault "/etc/keepalived/notify.sh -n fault -a 192.168.1.33"
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      
}
virtual_server 192.168.1.33 80 {  #定義虛擬服務器
    delay_loop 6                  #設置健康檢查時間
    lb_algo wrr                   #設置負載調度算法
    lb_kind DR                    #設置LVS工作模式
    nat_mask 255.255.255.0
    persistence_timeout 50  
    protocol TCP                  #設置轉發協議的類型
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      
    sorry_server 127.0.0.1 80     #設置緊急服務器
    real_server 192.168.1.10 80 {
        weight 1
        HTTP_GET {
            url {
              path /
                status_code 200
            }
            connect_timeout 2
            nb_get_retry 3
            delay_before_retry 1
        }
}
    real_server 192.168.1.20 80 {
        weight 1
        HTTP_GET {
            url {
              path /
                status_code 200
            }
            connect_timeout 2
            nb_get_retry 3
            delay_before_retry 1
                }
        }
    }
}

director_backup的配置vim /etc/keepalived/keepalived.conf

! Configuration File for keepalived
global_defs {
   notification_email {
        root@localhost
   }
   notification_email_from root@localhost
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id LVS_DEVEL
}
vrrp_script chk_schedown {
   script "[ -e /etc/keepalived/down ] && exit 1 || exit 0"
   interval 1
   weight -5
   fall 2
   rise 1
}
vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    virtual_router_id 54
    priority 99
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass soulboy
    }
    virtual_ipaddress {
        192.168.1.33/24 dev eth0 label eth0:0
    }
    track_script {
        chk_schedown
    }
    notify_master "/etc/keepalived/notify.sh -n master -a 192.168.1.33"
    notify_backup "/etc/keepalived/notify.sh -n backup -a 192.168.1.33"
    notify_fault "/etc/keepalived/notify.sh -n fault -a 192.168.1.33"
}
virtual_server 192.168.1.33 80 {
    delay_loop 6
    lb_algo wrr
    lb_kind DR
    nat_mask 255.255.255.0
    persistence_timeout 50
    protocol TCP
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
    sorry_server 127.0.0.1 80
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
    real_server 192.168.1.10 80 {
        weight 1
        HTTP_GET {
            url {
              path /
                status_code 200
            }
            connect_timeout 2
            nb_get_retry 3
            delay_before_retry 1
        }
}
    real_server 192.168.1.20 80 {
        weight 1
        HTTP_GET {
            url {
              path /
                status_code 200
            }
            connect_timeout 2
            nb_get_retry 3
            delay_before_retry 1
                }
        }
    }
}

通知腳本vim /etc/keepalived/notify.sh

#!/bin/bash
#
ifalias=${2:-eth0:0}
interface=$(echo $ifalias | awk -F: '{print $1}')
vip=$(ip addr show $interface | grep $ifalias | awk '{print $2}')
contact='root@localhost'
workspace=$(dirname $0)
notify() {
    subject="$ip change to $1"
    body="$ip change to $1 $(date '+%F %H:%M:%S')"
    echo $body | mail -s "$1 transition" $contact
}
case "$1" in
    master)
        notify master
        exit 0
    ;;
    backup)
        notify backup
        /etc/rc.d/init.d/httpd restart
        exit 0
    ;;
    fault)
        notify fault
        exit 0
    ;;
    *)
        echo 'Usage: $(basename $0) {master|backup|fault}'
        exit 1
    ;;
esac

測試

啓動director_master的keepalive服務並查看ipvs規則

#####查看ipvs規則
[root@master ~]# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.1.33:80 wrr
  -> 192.168.1.20:80              Route   1      0          0   
  -> 192.168.1.10:80              Route   1      0          0
#####查看網絡信息
[root@master ~]# ifconfig
eth0      Link encap:Ethernet  HWaddr 00:0C:29:C2:5E:01
          inet addr:192.168.1.61  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::20c:29ff:fec2:5e01/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:67996 errors:0 dropped:0 overruns:0 frame:0
          TX packets:116217 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:15418633 (14.7 MiB)  TX bytes:8387202 (7.9 MiB)
          Interrupt:67 Base address:0x2024
eth0:0    Link encap:Ethernet  HWaddr 00:0C:29:C2:5E:01
          inet addr:192.168.1.33  Bcast:0.0.0.0  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:67 Base address:0x2024

啓動director_backup的keepalive服務並查看ipvs規則

#####查看ipvs規則
[root@backup ~]# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.1.33:80 wrr
  -> 192.168.1.20:80              Route   1      0          0   
  -> 192.168.1.10:80              Route   1      0          0
#####查看網絡信息
[root@backup ~]# ifconfig
eth0      Link encap:Ethernet  HWaddr 00:0C:29:FA:52:D6
          inet addr:192.168.1.62  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::20c:29ff:fefa:52d6/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:115068 errors:0 dropped:0 overruns:0 frame:0
          TX packets:82940 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:19740061 (18.8 MiB)  TX bytes:6476242 (6.1 MiB)
          Interrupt:67 Base address:0x2024

使用客戶端訪問VIP

停止director_master的keepalived服務發現VIP消失

[root@master ~]# service keepalived stop
Stopping keepalived:                                       [  OK  ]
[root@master ~]# ifconfig
eth0      Link encap:Ethernet  HWaddr 00:0C:29:C2:5E:01
          inet addr:192.168.1.61  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::20c:29ff:fec2:5e01/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:69371 errors:0 dropped:0 overruns:0 frame:0
          TX packets:118587 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:15609985 (14.8 MiB)  TX bytes:8588490 (8.1 MiB)
          Interrupt:67 Base address:0x2024

在director_backup查看網絡信息，發現VIP已成功轉移

[root@backup ~]# ifconfig
eth0      Link encap:Ethernet  HWaddr 00:0C:29:FA:52:D6
          inet addr:192.168.1.62  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::20c:29ff:fefa:52d6/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:116816 errors:0 dropped:0 overruns:0 frame:0
          TX packets:84293 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:19932196 (19.0 MiB)  TX bytes:6597535 (6.2 MiB)
          Interrupt:67 Base address:0x2024
eth0:0    Link encap:Ethernet  HWaddr 00:0C:29:FA:52:D6
          inet addr:192.168.1.33  Bcast:0.0.0.0  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:67 Base address:0x2024

停止realserver_one的httpd服務

[root@realserver_one ~]# service httpd stop
Stopping httpd:                                            [  OK  ]

director_backup查看ipvs規則，發現realserver_one已經被踢出

[root@backup ~]# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.1.33:80 wrr
  -> 192.168.1.20:80              Route   1      0          0

客戶端訪問VIP發現頁面恆爲node2

停止realserver_two的httpd服務

[root@realserver_two ~]# service httpd stop
Stopping httpd:                                            [  OK  ]

director_backup查看ipvs規則，發現緊急站點生效

[root@backup ~]# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.1.33:80 wrr
  -> 127.0.0.1:80                 Local   1      0          0

客戶端訪問VIP發現頁面爲自定義警告頁面

分別啓動realserver_one和realserver_two的httpd服務

#####realserver_one
[root@realserver_one ~]# service httpd start
Starting httpd: httpd: apr_sockaddr_info_get() failed for realserver_one
httpd: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1 for ServerName
                                                           [  OK  ]
#####realserver_two
[root@realserver_two ~]# service httpd start
Starting httpd: httpd: apr_sockaddr_info_get() failed for realserver_two
httpd: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1 for ServerName
                                                           [  OK  ]

再次查看director_backup發現ipvs規則已經恢復

[root@backup ~]# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.1.33:80 wrr
  -> 192.168.1.20:80              Route   1      0          0   
  -> 192.168.1.10:80              Route   1      0          0

客戶端訪問VIP發現負載正常

啓動director_master的keepalived服務並查看網絡信息發現VIP成功轉移

[root@master ~]# service keepalived start
Starting keepalived:                                       [  OK  ]
[root@master ~]# ifconfig
eth0      Link encap:Ethernet  HWaddr 00:0C:29:C2:5E:01
          inet addr:192.168.1.61  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::20c:29ff:fec2:5e01/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:70394 errors:0 dropped:0 overruns:0 frame:0
          TX packets:118644 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:15679204 (14.9 MiB)  TX bytes:8593207 (8.1 MiB)
          Interrupt:67 Base address:0x2024
eth0:0    Link encap:Ethernet  HWaddr 00:0C:29:C2:5E:01
          inet addr:192.168.1.33  Bcast:0.0.0.0  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:67 Base address:0x2024

在director_backup查看網絡信息發現VIP消失

[root@backup ~]# ifconfig
eth0      Link encap:Ethernet  HWaddr 00:0C:29:FA:52:D6
          inet addr:192.168.1.62  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::20c:29ff:fefa:52d6/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:118485 errors:0 dropped:0 overruns:0 frame:0
          TX packets:87004 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:20112822 (19.1 MiB)  TX bytes:6791097 (6.4 MiB)
          Interrupt:67 Base address:0x2024
lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:6781 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6781 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:2122280 (2.0 MiB)  TX bytes:2122280 (2.0 MiB)

keepalived+lvs實現高可用的負載均衡

24-5-18 X

PKI相關概念及OpenSSL

BIND主從模式下實現View智能尋線

進程管理

sudo詳解

LVM

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結