由於代理服務器上開啓了長連接後，可以減少每次與後端服務器的三次握手，提高後端服務器的效率。所以nginx在1.1.4版本中終於增加了upstream對keepalive的支持，而haproxy原本對後端是支持長連接的（但是這個時候由於不能把client的IP加到header轉發給後端服務器造成會丟失客戶端的IP）。今天簡單測試了一下。基本的結構是
A:
Client –> nginx(proxy) –>nginx(server)
B:
Client –> haproxy(proxy) –>nginx(server)
對比的結構
Client –>nginx(server)
先測試一下client直接和server端能否保持住長連接，我是直接在server端抓包觀察，確定OK後再進行nginx和haproxy的測試。
nginx proxy的配置如下：
server {
location / {
root html;
proxy_set_header Connection "";
proxy_http_version 1.1;
proxy_intercept_errors on;
proxy_set_header Host $http_host;
proxy_set_header ORIG_CLIENT_IP $remote_addr;
index index.html,index.htm;
proxy_pass http://httpd;
}
}
upstream httpd {
server 172.189.85.156:1080;
keepalive 4;
least_conn;
}
在server端tcpflow抓包後可以看到
0172.189.085.161.50970-0172.189.085.156.01080: GET /index.html HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.50970: HTTP/1.1 304 Not Modified
0172.189.085.161.50971-0172.189.085.156.01080: GET /favicon.ico HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.50971: HTTP/1.1 404 Not Found
0172.189.085.161.50972-0172.189.085.156.01080: GET /index.html HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.50972: HTTP/1.1 304 Not Modified
0172.189.085.161.50973-0172.189.085.156.01080: GET /favicon.ico HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.50973: HTTP/1.1 404 Not Found
0172.189.085.161.50974-0172.189.085.156.01080: GET /index.html HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.50974: HTTP/1.1 304 Not Modified
0172.189.085.161.50975-0172.189.085.156.01080: GET /favicon.ico HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.50975: HTTP/1.1 404 Not Found
0172.189.085.161.50976-0172.189.085.156.01080: GET /index.html HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.50976: HTTP/1.1 304 Not Modified
0172.189.085.161.50977-0172.189.085.156.01080: GET /favicon.ico HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.50977: HTTP/1.1 404 Not Found
0172.189.085.161.50978-0172.189.085.156.01080: GET /index.html HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.50978: HTTP/1.1 304 Not Modified
0172.189.085.161.50979-0172.189.085.156.01080: GET /favicon.ico HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.50979: HTTP/1.1 404 Not Found
0172.189.085.161.50980-0172.189.085.156.01080: GET /index.html HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.50980: HTTP/1.1 304 Not Modified
0172.189.085.161.50981-0172.189.085.156.01080: GET /favicon.ico HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.50981: HTTP/1.1 404 Not Found
0172.189.085.161.50982-0172.189.085.156.01080: GET /index.html HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.50982: HTTP/1.1 304 Not Modified
0172.189.085.161.50983-0172.189.085.156.01080: GET /favicon.ico HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.50983: HTTP/1.1 404 Not Found
0172.189.085.161.50984-0172.189.085.156.01080: GET /index.html HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.50984: HTTP/1.1 304 Not Modified
0172.189.085.161.50985-0172.189.085.156.01080: GET /favicon.ico HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.50985: HTTP/1.1 404 Not Found
0172.189.085.161.50986-0172.189.085.156.01080: GET /index.html HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.50986: HTTP/1.1 304 Not Modified
0172.189.085.161.50987-0172.189.085.156.01080: GET /favicon.ico HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.50987: HTTP/1.1 404 Not Found
0172.189.085.161.50988-0172.189.085.156.01080: GET /index.html HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.50988: HTTP/1.1 304 Not Modified
0172.189.085.161.50989-0172.189.085.156.01080: GET /favicon.ico HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.50989: HTTP/1.1 404 Not Found
0172.189.085.161.50990-0172.189.085.156.01080: GET /index.html HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.50990: HTTP/1.1 304 Not Modified
0172.189.085.161.50991-0172.189.085.156.01080: GET /favicon.ico HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.50991: HTTP/1.1 404 Not Found
從上面可以看到代理服務器每次連接後端服務器使用的端口都是不同的，這個時候我就鬱悶了，難道nginx對後端開啓不了長連接？
測試了一兩個小時我終於想到是不是加了最小連接數的那個算法導致的。果然把那行去掉後就OK了。
[最新的狀況是 把 keepalive 4;設置放在least_conn之後就可以了 ]
server {
location / {
root html;
proxy_set_header Connection "";
proxy_http_version 1.1;
proxy_intercept_errors on;
proxy_set_header Host $http_host;
proxy_set_header ORIG_CLIENT_IP $remote_addr;
index index.html,index.htm;
proxy_pass http://httpd;
}
}
upstream httpd {
server 172.189.85.156:1080;
keepalive 4;
}
0172.189.085.161.50998-0172.189.085.156.01080: GET /index.html HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.50998: HTTP/1.1 304 Not Modified
0172.189.085.161.50998-0172.189.085.156.01080: GET /favicon.ico HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.50998: HTTP/1.1 404 Not Found
0172.189.085.161.50998-0172.189.085.156.01080: GET /index.html HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.50998: HTTP/1.1 304 Not Modified
0172.189.085.161.50998-0172.189.085.156.01080: GET /favicon.ico HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.50998: HTTP/1.1 404 Not Found
0172.189.085.161.50998-0172.189.085.156.01080: GET /index.html HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.50998: HTTP/1.1 304 Not Modified
0172.189.085.161.50998-0172.189.085.156.01080: GET /favicon.ico HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.50998: HTTP/1.1 404 Not Found
0172.189.085.161.50998-0172.189.085.156.01080: GET /index.html HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.50998: HTTP/1.1 304 Not Modified
0172.189.085.161.50998-0172.189.085.156.01080: GET /favicon.ico HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.50998: HTTP/1.1 404 Not Found
0172.189.085.161.50998-0172.189.085.156.01080: GET /index.html HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.50998: HTTP/1.1 304 Not Modified
然後我又測試了一下haproxy。主要是之前對haproxy的理解有點誤區。一方面doc裏面說了
A last improvement in the communications is the pipelining mode. It still uses
keep-alive, but the client does not wait for the first response to send the
second request. This is useful for fetching large number of images composing a
page :
[CON] [REQ1] [REQ2] … [RESP1] [RESP2] [CLO] …
This can obviously have a tremendous benefit on performance because the network
latency is eliminated between subsequent requests. Many HTTP agents do not
correctly support pipelining since there is no way to associate a response with
the corresponding request in HTTP. For this reason, it is mandatory for the
server to reply in the exact same order as the requests were received.
By default HAProxy operates in a tunnel-like mode with regards to persistent
connections: for each connection it processes the first request and forwards
everything else (including additional requests) to selected server. Once
established, the connection is persisted both on the client and server
sides. Use “option http-server-close” to preserve client persistent connections
while handling every incoming request individually, dispatching them one after
another to servers, in HTTP close mode. Use “option httpclose” to switch both
sides to HTTP close mode. “option forceclose” and “option
http-pretend-keepalive” help working around servers misbehaving in HTTP close
mode.
另外一方面就是大家都知道haproxy對後端是沒有keepalive的，目前連測試版也不支持。。。這個就讓我覺得很是疑惑。
今天其實主要是爲了測試一下haproxy，配置文件如下
frontend httpf
bind *:1689
default_backend httpd
backend httpd
mode http
option forwardfor header ORIG_CLIENT_IP
balance roundrobin
server ser2 172.189.85.156:1080
frontend httpf2
bind *:1690
default_backend httpd2
backend httpd2
mode http
option httpclose
balance roundrobin
option forwardfor header ORIG_CLIENT_IP
server ser2 172.189.85.156:1080
不過抓包後直接看到了當訪問1689端口的時候對後端服務器的連接是一直保持的，但是顯示不出客戶端的IP（沒有轉發ORIG_CLIENT_IP）,
0172.189.085.156.01080-0172.189.085.161.51026: HTTP/1.1 304 Not Modified
0172.189.085.161.51026-0172.189.085.156.01080: GET /favicon.ico HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.51026: HTTP/1.1 404 Not Found
0172.189.085.161.51026-0172.189.085.156.01080: GET /index2.html HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.51026: HTTP/1.1 304 Not Modified
0172.189.085.161.51026-0172.189.085.156.01080: GET /favicon.ico HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.51026: HTTP/1.1 404 Not Found
0172.189.085.161.51026-0172.189.085.156.01080: GET /index2.html HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.51026: HTTP/1.1 304 Not Modified
0172.189.085.161.51026-0172.189.085.156.01080: GET /favicon.ico HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.51026: HTTP/1.1 404 Not Found
0172.189.085.161.51026-0172.189.085.156.01080: GET /index2.html HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.51026: HTTP/1.1 304 Not Modified
0172.189.085.161.51026-0172.189.085.156.01080: GET /favicon.ico HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.51026: HTTP/1.1 404 Not Found
0172.189.085.161.51026-0172.189.085.156.01080: GET /index2.html HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.51026: HTTP/1.1 304 Not Modified
0172.189.085.161.51026-0172.189.085.156.01080: GET /favicon.ico HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.51026: HTTP/1.1 404 Not Found
訪問1690端口，即是配置了httpclose的時候發現能顯示客戶端的IP，但是每個連接都在請求完成後就關閉了，抓包的記錄就不貼了。
看來有問題還是需要自己抓包來實際分析才知道究竟是怎麼個情況。
後記：
今天統計了一下做代理的nginx似乎每個鏈接對後端請求處理100個後就斷掉了，仔細查了一下wiki發現這個實際上是後端nginx的默認設置的問題。
可以吧後端的nginx的配置改成keepalive_requests 1000,這樣代理服務器向後端的連接能每個處理1000個再中斷。