如何用Nginx+Keepalived實現高可用的反向代理+負載均衡

keealived服務工作在3層(IP)、4層(TCP)、5(應用層)
nginx通過Virtual IP對外提供www服務，能通過算法實現後端web服務的負載均衡
下列操作徐預先安裝tengine，或者nginx。

【第一部分】配置一個虛擬IP地址，只向外界暴露這個VIP

以node1(192.168.100.151)爲例說明操作步驟。

【第一步】
添加http子模塊upstream，設置多個負載tomcat server（ip尾號152、153、154）
[root@node1 conf]# grep -v “#” nginx.conf
worker_processes 2;
events {
worker_connections 1024;
}

http {
include mime.types;
default_type application/octet-stream;
sendfile on;
keepalive_timeout 1;

upstream howareyou {   # 3臺tomcat服務器
  server 192.168.100.152:8080;
  server 192.168.100.153:8080;
  server 192.168.100.154:8080;
}

server {
    listen       8000;
    location /howareyou {
      proxy_pass http://howareyou/;
    }
    location / {
        root   html;
        index  index.html index.htm;
    }
    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   html;
    }
}

}

【第二步】在現有網卡上添加一個虛擬IP地址
[root@node1 ~]# ifconfig eth0:1 192.168.100.200 netmask 255.255.255.0 up
[root@node1 ~]# ifconfig -a | grep --color -A 3 “eth0:1”
eth0:1 Link encap:Ethernet HWaddr 00:0C:29:35:A8:FD
inet addr:192.168.100.200 Bcast:192.168.100.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

分別指定3臺tomcat服務器限定返回給web客戶端的內容——
[root@node2 ROOT]# pwd
/opt/apache-tomcat-8.0.53/webapps/ROOT
[root@node2 ROOT]# grep -vE “#|^\n” index.jsp
from node2Session=<%=session.getId()%>

[root@node3 ~]# cd /opt/apache-tomcat-8.0.53/webapps/ROOT
[root@node3 ROOT]# cat index.jsp
from node3Session=<%=session.getId()%>

[root@node4 ~]# cd /opt/apache-tomcat-8.0.53/webapps/ROOT/
[root@node4 ROOT]# cat index.jsp
from node4Session=<%=session.getId()%>

【第三步】
訪問虛擬IP的URL：http://192.168.100.200:8000/howareyou
結果：
首次：
from node4
Session=0248C9E8C73DA87237EFA89974430ECB
刷新：
from node2
Session=AEA89EF695781E1829517EEAFAC93A10
再刷新：
from node3
Session=42F8D3CD8C20B246E382D6F5A2B04B72
可以看出：三臺tomcat web服務器的權重相同時，每次http請求由nginx均衡地調度到3臺後端tomcat web服務器，http客戶端無需關注實際是哪一臺tomcat提供web服務，只需訪問VIP+port+URI 即可。

【第四步】把node1上的VIP換到node4上去
[root@node1 ~]# ifconfig eth0:1 down
[root@node1 ~]# ifconfig -a | grep --color -A 3 “eth0”
eth0 Link encap:Ethernet HWaddr 00:0C:29:35:A8:FD
inet addr:192.168.100.151 Bcast:192.168.100.255 Mask:255.255.255.0
inet6 addr: fe80::20c:29ff:fe35:a8fd/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

[root@node4 ROOT]# ifconfig eth1:192.168.100.200 netmask 255.255.255.0 up

【第五步】關閉node1的nginx服務，同時在node4啓動nginx服務
node4的nginx.conf與node1的一致

【第六步】訪問URL http://192.168.100.200:8000/howareyou
效果和上面的第三步一樣

【第二部分】配置高可用 Nginx+keepalived服務

【第1步】角色規劃
node1：state=MASTER、priority=100
node4：state=BACKUP、priority=50

【第2步】ndoe1、node4分別安裝keepalived
[root@node4 keepalived]# cat /etc/yum.repos.d/CentOS-Base.repo | grep -v “#”

[base]
name=CentOS- $releasever - Base mirrorlist=http://mirrorlist.centos.org/?release=$ releasever&arch= $KaTeX parse error: Expected 'EOF', got '&' at position 9: basearch&̲repo=os&infra=$ infra
baseurl=http://mirror.centos.org/centos/ $releasever/os/$ basearch/
gpgcheck=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6

yum clean all
yum makecache
yum install -y keepalived # 會默認安裝到目錄 /etc/keepalived/ ，該目錄下的keepalived.conf是配置文件

【第3步】按照第一步的角色規劃來配置node1、node4
3.1 分別修改nginx.conf、keepalived.conf
修改node1節點的nginx.conf，和node4的相同
[root@node1 ~]# grep -vE “#|^$” /opt/tengine-2.3.2/conf/nginx.conf
worker_processes 2;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
sendfile on;
keepalive_timeout 1;
upstream howareyou {
server 192.168.100.152:8080; # 後端3臺tomcat服務器
server 192.168.100.153:8080; # 3臺tomcat返回前端的內容請已在上文第一部分的【第二步】分別指定
server 192.168.100.154:8080;
}
server {
listen 8000;
location /howareyou {
proxy_pass http://howareyou/;
}
location / {
root html;
index index.html index.htm;
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}
}
}

修改node1節點的keepalived.conf
[root@node1 ~]# grep -v “^$” /etc/keepalived/keepalived.conf
#ConfigurationFile for keepalived
global_defs {
notification_email { #定義接受郵件的郵箱
[email protected]
}
notification_email_from [email protected] #定義發送郵件的郵箱
smtp_server mail.qq.com
smtp_connect_timeout 30
route_id nginx_backup
}
vrrp_script check_nginx { #定義監控nginx的腳本
script “/root/check_nginx.sh”
interval 2 #監控時間間隔
weight 2 #負載參數
}
vrrp_instance vrrptest { #定義vrrptest實例
state MASTER #服務器狀態
interface eth0 #當前進行vrrp通訊使用的網絡接口卡(當前centos的網卡)
virtual_router_id 51 #虛擬路由的標誌，(倆nginx服務器)同一組lvs的虛擬路由標識必須相同才能切換
priority 100 #服務啓動優先級,值越大優先級越高,BACKUP角色節點的該值不能大於MASTER
advert_int 1 #服務器之間的存活檢查時間
authentication {
auth_type PASS #認證類型
auth_pass ufsoft #認證密碼，一組lvs 服務器的認證密碼必須一致
}
track_script { #執行監控nginx進程的腳本
check_nginx
}
virtual_ipaddress { #虛擬IP地址
192.168.100.200
}
}
然後在我node1的eth0網卡下創建一個虛擬IP地址
[root@node1 ~]# ifconfig eth0:1 192.168.100.200 netmask 255.255.255.0 up

修改node4節點的nginx.conf
內容和node1節點的一樣，因爲二者提供相同的服務：爲web用戶提供反向代理、爲後端tomcat web服務器均衡負載。

修改node4節點的keepalived.conf
[root@node4 ~]# grep -v “^ $KaTeX parse error: Expected 'EOF', got '#' at position 35: \dotseepalived.conf #̲ConfigurationFi\dots$ (ps -ef | grep “nginx: master process”| grep -v grep )” == “” ]
then service keepalived stop
else echo “nginx is running”
該腳本功能：在nginx服務停止時關閉本機的keepalived服務，以便模擬該主機的nginx、keepalived服務不可用，那麼會將BACKUP角色切換成MASTER角色。

然後在我node1的eth1網卡下創建一個虛擬IP地址
[root@node1 ~]# ifconfig eth1:1 192.168.100.200 netmask 255.255.255.0 up

【第4步】分別啓動node1、node4上的nginx服務和keepalived服務
[root@node1 ~]# nginx
[root@node1 ~]# service keepalived start
[root@node4 ~]# nginx
[root@node4 ~]# service keepalived start
[root@node1 ~]# ps aux |grep --color nginx
[root@node1 ~]# service keepalived status
[root@node4 ~]# ps aux |grep --color nginx
[root@node4 ~]# service keepalived status

【第5步】主從 nginx+keepalived 的故障測試
停掉MASTER角色的keepalived服務所在主機的nginx服務。
[root@node4 ~]# grep --color -C 3 BACKUP /etc/keepalived/keepalived.conf
weight 2 #負載參數
}
vrrp_instance vrrptest { #定義vrrptest實例
state BACKUP #服務器狀態
interface eth1 #當前進行vrrp通訊使用的網絡接口卡(當前centos的網卡)
virtual_router_id 51 #虛擬路由的標誌，一組lvs的虛擬路由標識必須相同，這樣才能切換
priority 40 #服務啓動優先級,值越大優先級越高,BACKUP角色節點的該值不能大於MASTER
advert_int 1 #服務器之間的存活檢查時間
authentication {
auth_type PASS #認證類型

模擬node1上的nginx服務宕機，觀察keepalived服務能否自動將node4的keepalived服務從BACKUP切換成MASTER狀態
[root@node1 ~]# nginx -s stop # 那麼keepalived裏面的腳本check_nginx.sh 會將keealive服務停止
通過瀏覽器訪問VIP——

每刷新一下，可依次看到from node2、from node3、from node4字樣，說明MASTER角色能正常提供反向代理服務。

緊接着查看配置爲BACKUP狀態的node4上發生了什麼——
[root@node4 ~]# ps aux | grep -v grep | grep --color nginx
root 31787 0.0 0.1 47004 1160 ? Ss 18:14 0:00 nginx: master process nginx
nobody 31788 0.0 0.2 47388 2152 ? S 18:14 0:00 nginx: worker process
nobody 31789 0.0 0.2 47388 2124 ? S 18:14 0:00 nginx: worker process
[root@node4 ~]# service keepalived status
keepalived (pid 30901) is running…
[root@node4 ~]# tail /var/log/messages
Sep 9 17:56:54 node4 Keepalived_vrrp[30903]: Kernel is reporting: interface eth1 UP
Sep 9 17:56:55 node4 NetworkManager[1908]: (eth1): carrier now OFF (device state 8, deferring action for 4 seconds)
Sep 9 17:56:55 node4 kernel: e1000: eth1 NIC Link is Down
Sep 9 17:56:58 node4 Keepalived_vrrp[30903]: Kernel is reporting: interface eth1 DOWN
Sep 9 17:56:58 node4 Keepalived_vrrp[30903]: VRRP_Instance(vrrptest) Now in FAULT state
Sep 9 17:56:59 node4 kernel: e1000: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
Sep 9 17:56:59 node4 NetworkManager[1908]: (eth1): carrier now ON (device state 8)
Sep 9 17:57:02 node4 Keepalived_vrrp[30903]: Kernel is reporting: interface eth1 UP
Sep 9 17:57:05 node4 Keepalived_vrrp[30903]: VRRP_Instance(vrrptest) Transition to MASTER STATE
Sep 9 17:57:06 node4 Keepalived_vrrp[30903]: VRRP_Instance(vrrptest) Entering MASTER STATE
注意看這個日誌中的 “Entering MASTER STATE”記錄，意思是node4節點的Nginx服務角色正在被切換成MASTER狀態。
所以，node1上的nginx服務停掉以後，該節點的keepalived服務也被關停，會立即將node4節點的nginx角色變爲MASTER狀態。

【第6步】將原MASTER角色的nginx服務及其keepalived服務啓動，查看角色變化
在上一步中，我們關停了node1的nginx及其對應的keepalived服務。
現在我們把他們再次啓動，看看能恢復成MASTER角色嗎。
[root@node1 ~]# ps aux | grep -v grep |grep --color nginx
[root@node1 ~]# service keepalived status
keepalived is stopped
[root@node1 ~]# nginx
[root@node1 ~]# service keepalived start
Starting keepalived: [ OK ]
[root@node1 ~]# date
Mon Sep 9 19:36:14 CST 2019
[root@node1 ~]# ps aux | grep -v grep |grep --color nginx
root 24728 0.0 0.1 47016 1156 ? Ss 19:35 0:00 nginx: master process nginx
nobody 24729 0.0 0.2 47484 2124 ? S 19:35 0:00 nginx: worker process
nobody 24730 0.0 0.2 47484 2120 ? S 19:35 0:00 nginx: worker process
[root@node1 ~]# service keepalived status
keepalived (pid 24750) is running…

查看一下node1節點的keepalived有關日誌
[root@node1 ~]# tail -n 5 /var/log/messages
Sep 9 19:35:49 node1 Keepalived_vrrp[24753]: VRRP_Instance(vrrptest) Entering MASTER STATE
Sep 9 19:35:49 node1 Keepalived_vrrp[24753]: VRRP_Instance(vrrptest) setting protocol VIPs.
Sep 9 19:35:49 node1 Keepalived_vrrp[24753]: VRRP_Instance(vrrptest) Sending gratuitous ARPs on eth0 for 192.168.100.200
Sep 9 19:35:49 node1 Keepalived_healthcheckers[24752]: Netlink reflector reports IP 192.168.100.200 added
Sep 9 19:35:54 node1 Keepalived_vrrp[24753]: VRRP_Instance(vrrptest) Sending gratuitous ARPs on eth0 for 192.168.100.200

對應地，我們查看一下node4的keepalived日誌
[root@node4 ~]# date
Mon Sep 9 19:36:12 CST 2019
[root@node4 ~]# tail -n 5 /var/log/messages
Sep 9 17:57:02 node4 Keepalived_vrrp[30903]: Kernel is reporting: interface eth1 UP
Sep 9 17:57:05 node4 Keepalived_vrrp[30903]: VRRP_Instance(vrrptest) Transition to MASTER STATE
Sep 9 17:57:06 node4 Keepalived_vrrp[30903]: VRRP_Instance(vrrptest) Entering MASTER STATE
Sep 9 19:35:48 node4 Keepalived_vrrp[30903]: VRRP_Instance(vrrptest) Received higher prio advert
Sep 9 19:35:48 node4 Keepalived_vrrp[30903]: VRRP_Instance(vrrptest) Entering BACKUP STATE

可以看出，啓動Keepalived之前，配置的node1作爲MASTER狀態的nginx生效了，也就是node1的nginx關停以後再次啓動時，Node1的nginx恢復了MASTER狀態。

如何用Nginx+Keepalived實現高可用的反向代理+負載均衡

【第一部分】配置一個虛擬IP地址，只向外界暴露這個VIP

【第二部分】配置高可用 Nginx+keepalived服務

MySQL 核心模塊揭祕 | 18 期 | 鎖在內存里長什麼樣*

使用perf工具生成火焰圖

HttpSecurity 是如何組裝過濾器鏈的

數說海南——近6年海南各市縣人口簡單看

長序列中Transformers的高級注意力機制總結

WebStorm 創建 Vue 項目

大齡程序員思考

響應式界面控件DevExtreme * 更強的數據分析和可視化功能

如何在linux shell腳本中自動輸入密碼.

python面向對象代碼示例

python函數式編程示例

Pinpoint部署和踩坑記錄

遷移hbase的快照到新集羣后RegionServer無法啓動,報錯failed open of region

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結