Nginx不僅是一款優秀的WEB服務器,同時可以根據nginx的反代理可以配置成強大的負載均衡器.這裏就介紹如何把nginx配置成負載均衡器,並結合keepalived配置高可用的集羣.
一般集羣主要架構爲:
前端爲負載均衡器兩個:主/備,兩種工作方式,一種是備機待機狀態,主機故障時備機接管主機工作實現故障莊毅,在主機故障恢復完成時備機繼續僅需待機狀態,第二種是主備同時工作,一臺宕機另外一臺自動接管另一臺的工作實現故障轉移.
第一種方式可以通過將域名解析到一個虛擬ip(vip)上,主負載均衡器綁定虛擬ip,當主負載均衡器出現故障時,通過keepalived自動將vip綁定到備用負載均衡器上同時arping網關刷新MAC地址.,避免單點故障.
第二種方式主備同時綁定一個vip,把域名通過DNS輪詢的方式解析到這兩個服務器上,主機出現故障,備機就將主機綁定vip綁定到備機上,同時arping網關刷新MAC地址.實現故障轉移.
中間爲WEB服務器作爲real server,處理請求.
後端爲數據庫和分佈式文件系統.數據庫一般爲主從兩臺.分佈式文件系統有效解決WEB服務器之間的數據同步.有的還會將圖片服務器單獨分離出來放在後端.
本文使用環境:
CentOS 5.5 32位
nginx:nginx-1.0.11
keepalived:keepalived-1.1.19.tar.gz
主調度器:192.168.3.1
備調度器:192.168.3.2
real server:192.168.3.4/5/6
本文采用第一種方式來進行vip爲:192.168.3.253
一、在主備服務器上部署nginx
1.下載
wget http://nginx.org/download/nginx-1.0.11.tar.gz
2.安裝
install
zlib-devel pcre-devel openssl-devel #安裝依賴
tar -zxvf nginx-1.0.11.tar.gz
cd nginx-1.0.11
./configure --prefix=/usr/local/nginx --with-http_ssl_module --with-http_flv_module --with-http_gzip_static_module
make && make install
3.配置
配置主調度器的nginx,編輯nginx.conf
vi /usr/local/nginx/conf/nginx.conf
http {
include mime.types;
default_type application/octet-stream;
#log_format main '$remote_addr - $remote_user [$time_local] "$request" '
# '$status $body_bytes_sent "$http_referer" '
# '"$http_user_agent" "$http_x_forwarded_for"';
#access_log logs/access.log main;
sendfile on;
#tcp_nopush on;
#keepalive_timeout 0;
keepalive_timeout 65;
#gzip on;
# 添加一組真實的服務器地址池
# 供proxy_pass和fastcgi_pass指令中使用的代理服務器
upstream real_server_pool {
# 後臺如果有動態應用的時候,ip_hash指令可以通過hash算法
# 將客戶端請求定位到同一臺後端服務器上,解決session共享,
# 但建議用動態應用做session共享
# ip_hash;
# server用於指定一個後端服務器的名稱和參數
# weight代表權,重默認爲1,權重越高被分配的客戶端越多
# max_fails 指定時間內對後端請求失敗的次數
# fail_timeout 達到max_fails指定的失敗次數後暫停的時間
server 192.168.3.4:80 weight=1 max_fails=2 fail_timeout=30s;
# down參數用來標記爲離線,不參與負載均衡.在ip_hash下使用
# 在此做演示,後面測試會去掉
server 192.168.3.5:80 weight=1 max_fails=2 fail_timeout=30s down;
# backup僅僅在非backup服務器宕機或繁忙的時候使用
# (在此做演示,後面測試會去掉)
server 192.168.3.6:80 weight=1 max_fails=2 fail_timeout=30s backup;
}
server {
listen 192.168.3.1:80;
server_name localhost;
#charset koi8-r;
#access_log logs/host.access.log main;
location / {
#root html;
#index index.html index.htm;
# 使用upstream設置的一組代理服務器
# 如果後端服務器出現502或504等執行錯誤時,
# 將自動將請求轉發給負載均衡池中的另一臺服務器.
proxy_next_upstream http_502 http_504 error timeout invalid_header;
proxy_pass http://real_server_pool;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $remote_addr;
}
}
}
http {
include mime.types;
default_type application/octet-stream;
#log_format main '$remote_addr - $remote_user [$time_local] "$request" '
# '$status $body_bytes_sent "$http_referer" '
# '"$http_user_agent" "$http_x_forwarded_for"';
#access_log logs/access.log main;
sendfile on;
#tcp_nopush on;
#keepalive_timeout 0;
keepalive_timeout 65;
#gzip on;
upstream real_server_pool {
#ip_hash;
server 192.168.3.4:80 weight=1 max_fails=2 fail_timeout=30s;
server 192.168.3.5:80 weight=1 max_fails=2 fail_timeout=30s;
server 192.168.3.6:80 weight=1 max_fails=2 fail_timeout=30s;
}
server {
listen 192.168.3.2:80; # 監聽ip改爲本地ip
server_name localhost;
#charset koi8-r;
#access_log logs/host.access.log main;
location / {
#root html;
#index index.html index.htm;
proxy_next_upstream http_502 http_504 error timeout invalid_header;
proxy_pass http://real_server_pool;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $remote_addr;
}
然後啓動主備nginx:
/usr/local/nginx/sbin/nginx
二、在主備服務器上部署keepalived
安裝
安裝依賴:
yum -y install kernel-devel # 安裝依賴
開啓路由轉發:
vi /etc/sysctl.conf
net.ipv4.ip_forward = 1 # 此參數改爲1
sysctl -p # 使修改生效
首先安裝ipvs:
ln -s /usr/src/kernels/2.6.18-194.el5-i686/ /usr/src/linux # ipvs需要內核文件,做一個軟連接
# 下載
wget http://www.linuxvirtualserver.org/software/kernel-2.6/ipvsadm-1.24.tar.gz
tar -zxvf ipvsadm-1.24.tar.gz
cd ipvsadm-1.24
make
make install
然後安裝keepalived
# 下載
wget http://www.keepalived.org/software/keepalived-1.1.19.tar.gz
tar -zxvf keepalived-1.1.19.tar.gz
cd keepalived-1.1.19
./configure --prefix=/ \ # 安裝在默認位置(配置文件,二進制文件,啓動腳本放到默認位置)
--mandir=/usr/local/share/man/ \
--with-kernel-dir=/usr/src/kernels/2.6.18-194.el5-i686/ # 需要內核的頭文件
make && make install
配置keepalived
編輯主調度器配置文件/etc/keepalived/keepalived.conf
global_defs {
notification_email {
[email protected] # 定義通知郵箱,有多個可以換行添加
}
notification_email_from [email protected]# 定義發送郵件的郵箱
smtp_server www.linuxzen.com # 定義發件服務器
smtp_connect_timeout 30 # 定義連接smtp服務器超時時間
router_id LVS_DEVEL
}
vrrp_instance VI_1 {
state MASTER # 標示主備,備機上改爲BACKUP
interface eth0 # HA監測的端口
virtual_router_id 51 # 主備的virtual_router_id的值必須相同
priority 100 # 優先級,通常主要比備稍大
advert_int 1 # VRRP Multicast 廣播週期秒數
authentication { # 定義認證
auth_type PASS # 認證方式
auth_pass 1111 # 認證口令字
}
virtual_ipaddress { # 定義vip
192.168.3.253 # 多個可換行添加,一行一個
}
}
virtual_server 192.168.3.253 80 {
delay_loop 6 # 每隔 6 秒查詢 realserver 狀態
lb_algo rr
lb_kind NAT
nat_mask 255.255.255.0
persistence_timeout 50 # 同一IP 的連接50秒內被分配到同一臺realserver
protocol TCP # 用TCP監測realserver的狀態
real_server 192.168.3.1 80 {
weight 3 # 權重
TCP_CHECK {
connect_timeout 10 # 10秒無響應超時
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
real_server 192.168.3.2 80 {
weight 3
TCP_CHECK {
connect_timeout 3
delay_before_retry 3
connect_port 80
}
}
}
配置備用調度器的keepalived,只需要將state MASTER 改爲state BACKUP,降低priority 100 的值:
global_defs {
notification_email {
[email protected]
}
notification_email_from [email protected]
smtp_server www.linuxzen.com
smtp_connect_timeout 30
router_id LVS_DEVEL
}
vrrp_instance VI_1 {
state BACKUP # 備機上改爲BACKUP
interface eth0
virtual_router_id 51 # 主備的virtual_router_id的值必須相同
priority 99 # 備用優先級小於主調度器
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.3.253
}
}
virtual_server 192.168.3.253 80 {
delay_loop 6
lb_algo rr
lb_kind NAT
nat_mask 255.255.255.0
persistence_timeout 50
protocol TCP
real_server 192.168.3.1 80 {
weight 3
TCP_CHECK {
connect_timeout 10
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
real_server 192.168.3.2 80 {
weight 3
TCP_CHECK {
connect_timeout 3
delay_before_retry 3
connect_port 80
}
}
}
主備上啓動keepalived:
service keepalived start
三、測試—–部署後端服務器
在後端服務器安裝nginx,這裏僅部署一臺然後創建3個基於ip的虛擬主機供測試:
綁定ip:
ifconfig eth0:1 192.168.3.4/24
ifconfig eth0:2 192.168.3.5/24
ifconfig eth0:3 192.168.3.6/24
安裝nginx後編輯配置文件,在http塊裏添加:
http {
server {
listen 192.168.3.4:80;
server_name 192.168.3.4;
location / {
root html/s1;
index index.html index.htm;
}
}
server {
listen 192.168.3.5:80;
server_name 192.168.3.5;
location / {
root html/s2;
index index.html index.htm;
}
}
server {
listen 192.168.3.6:80;
server_name 192.168.3.5;
location / {
root html/s3;
index index.html index.htm;
}
}
}
創建虛擬主機根目錄,並創建不通的首頁文檔:
cd /usr/local/nginx/html/
mkdir s1 s2 s3
echo server1 > s1/index.html
echo server2 > s2/index.html
echo server3 > s3/index.html
啓動nginx:
/usr/local/nginx/sbin/nginx
打開瀏覽器訪問http://192.168.3.253
刷新會看到顯示不同的內容:server1,server2,server3(生產中的服務器應該是一樣的)
現在停掉主調度器的keepalived
pkill keepalived
查看備調度器的日誌:
cat /var/log/messages
Feb 10 16:36:27 cfhost Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE
Feb 10 16:36:28 cfhost Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE
Feb 10 16:36:28 cfhost Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs.
Feb 10 16:36:28 cfhost Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.3.253
Feb 10 16:36:28 cfhost Keepalived_vrrp: Netlink reflector reports IP 192.168.3.253 added
Feb 10 16:36:28 cfhost Keepalived_healthcheckers: Netlink reflector reports IP 192.168.3.253 added
Feb 10 16:36:33 cfhost Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.3.253
現在訪問http://192.168.3.253依然可以訪問.
大家也看到了備機keepalived只有檢測主機的keepalived停止的時候纔會切換vip,而不是檢測一臺real server的某一服務(比如檢測80端口的HTTP)切換vip,所以在nginx進程停止的時候,如果服務器沒有宕機這時候就無法實現故障轉移,所以我們編寫一個檢測nginx狀態的腳本結合keepalived實現故障轉移:
#!/bin/bash
#filename:nsc.sh
ps aux ¦ grep nginx ¦ grep -v grep 2> /dev/null 1>&2 # 過濾nginx進程
if [[ $? -eq 0 ]] # 如果過濾有nginx進程會返回0則認爲nginx存活
then
sleep 5 # 使腳本進入休眠
else
# 如果nginx沒有存活嘗試啓動nginx,如果失敗則殺死keepalived的進程
/usr/local/nginx/sbin/nginx
ps aux ¦ grep nginx ¦ grep -v grep 2> /dev/null 1>&2
if [[ $? -eq 0 ]]
then
pkill keepalived
fi
fi
然後後臺運行此腳本:
nohup sh nsc.sh &
這樣就實現了羣集的高可靠和高可用.