開源解決方案:
squid:
varnish:默認端口:6081、127.20.0.1:6082
varnish官方站點: http://www.varnish-cache.org/
Community
Enterprise
This is Varnish Cache, a high-performance HTTP accelerator.
程序架構:
Manager進程
Cacher進程,包含多種類型的線程:
accept, worker, expiry, ...
shared memory log:
統計數據:計數器;
日誌區域:日誌記錄;
varnishlog, varnishncsa, varnishstat...
配置接口:VCL
Varnish Configuration Language,
vcl complier --> c complier --> shared object
varnish的程序環境:
/etc/varnish/varnish.params: 配置varnish服務進程的工作特性,例如監聽的地址和端口,緩存機制;
Unit File: EnvironmentFile=""
/etc/varnish/default.vcl:配置各Child/Cache線程的緩存策略;
VCL: dsl, subroutines, 子例程
主程序:
/usr/sbin/varnishd
CLI interface:
/usr/bin/varnishadm
Shared Memory Log交互工具:
/usr/bin/varnishhist
/usr/bin/varnishlog
/usr/bin/varnishncsa
/usr/bin/varnishstat
/usr/bin/varnishtop
測試工具程序:
/usr/bin/varnishtest
VCL配置文件重載程序:
/usr/sbin/varnish_reload_vcl
Systemd Unit File:
/usr/lib/systemd/system/varnish.service
varnish服務
/usr/lib/systemd/system/varnishlog.service
/usr/lib/systemd/system/varnishncsa.service
日誌持久的服務;
實例:將varnish服務設置內容http服務
126.180:httpd服務
yum -y install httpd
vim /var/www/html/index.html
not 1
systemctl start httpd.service
102.70:varnish服務器
yum -y install varnish
vim /etc/varnish/default.vcl
# Default backend definition. Set this to point to your content server.
//默認端定義。將其設置爲指向內容服務器
backend default {
.host = "172.20.126.180";
//內容服務器的ip
.port = "80";
//定義的端口
啓用:
systemctl start varnish
重新加載服務直接運行即可:
varnish_reload_vcl
測試:172.20.102.70:6081將顯示內容跳轉到126.180的httpd服務上;
varnish的緩存存儲機制( Storage Types):
-s [name=]type[,options]
· malloc[,size]
內存存儲,[,size]用於定義空間大小;重啓後所有緩存項失效;
· file[,path[,size[,granularity]]]
磁盤文件存儲,黑盒;重啓後所有緩存項失效;
· persistent,path,size
文件存儲,黑盒;重啓後所有緩存項有效;實驗;
varnish程序的選項:
程序選項:/etc/varnish/varnish.params文件
-a address[:port][,address[:port][...],默認爲6081端口;
-T address[:port],默認爲6082端口;
-s [name=]type[,options],定義緩存存儲機制;
-u user
-g group
-f config:VCL配置文件;
-F:運行於前臺;
...
運行時參數:/etc/varnish/varnish.params文件, DEAMON_OPTS
DAEMON_OPTS="-p thread_pool_min=5 -p thread_pool_max=500 -p thread_pool_timeout=300"
-p param=value:設定運行參數及其值; 可重複使用多次;
-r param[,param...]: 設定指定的參數爲只讀狀態;
重載vcl配置文件:
~ ]# varnish_reload_vcl
varnishadm
-S /etc/varnish/secret -T [ADDRESS:]PORT
help [<command>]
ping [<timestamp>]
auth <response>
quit
banner
status
start
stop
vcl.load <configname> <filename>
vcl.inline <configname> <quoted_VCLstring>
vcl.use <configname>
vcl.discard <configname>
vcl.list
param.show [-l] [<param>]
param.set <param> <value>
panic.show
panic.clear
storage.list
vcl.show [-v] <configname>
backend.list [<backend_expression>]
backend.set_health <backend_expression> <state>
ban <field> <operator> <arg> [&& <field> <oper> <arg>]...
ban.list
配置文件相關:
vcl.list:顯示類似於ls命令
vcl.load:裝載,加載並編譯;
vcl.use:激活;
vcl.discard:刪除;
vcl.show [-v] <configname>:查看指定的配置文件的詳細信息;
運行時參數:
param.show -l:顯示列表;
param.show <PARAM>
param.set <PARAM> <VALUE>
緩存存儲:
storage.list
後端服務器:
backend.list
VCL:
”域“專有類型的配置語言;
state engine:狀態引擎;
VCL有多個狀態引擎,狀態之間存在相關性,但狀態引擎彼此間互相隔離;每個狀態引擎可使用return(x)指明關聯至哪個下一級引擎;每個狀態引擎對應於vcl文件中的一個配置段,即爲subroutine
vcl_hash --> return(hit) --> vcl_hit
vcl_recv的默認配置:
sub vcl_recv {
if (req.method == "PRI") {
/* We do not support SPDY or HTTP/2.0 */
return (synth(405));
}
if (req.method != "GET" &&
req.method != "HEAD" &&
req.method != "PUT" &&
req.method != "POST" &&
req.method != "TRACE" &&
req.method != "OPTIONS" &&
req.method != "DELETE") {
/* Non-RFC2616 or CONNECT which is weird. */
return (pipe);
}
if (req.method != "GET" && req.method != "HEAD") {
/* We only deal with GET and HEAD by default */
return (pass);
}
if (req.http.Authorization || req.http.Cookie) {
/* Not cacheable by default */
return (pass);
}
return (hash);
}
}
Client Side:
vcl_recv, vcl_pass, vcl_hit, vcl_miss, vcl_pipe, vcl_purge, vcl_synth, vcl_deliver
vcl_recv:
hash:vcl_hash
pass: vcl_pass
pipe: vcl_pipe
synth: vcl_synth
purge: vcl_hash --> vcl_purge
vcl_hash:
lookup:
hit: vcl_hit
miss: vcl_miss
pass, hit_for_pass: vcl_pass
purge: vcl_purge
Backend Side:
vcl_backend_fetch, vcl_backend_response, vcl_backend_error
兩個特殊的引擎:
vcl_init:在處理任何請求之前要執行的vcl代碼:主要用於初始化VMODs;
vcl_fini:所有的請求都已經結束,在vcl配置被丟棄時調用;主要用於清理VMODs;
vcl的語法格式:
(1) VCL files start with vcl 4.0;
(2) //, # and /* foo */ for comments;
(3) Subroutines are declared with the sub keyword; 例如sub vcl_recv { ...};
//用sub關鍵詞定義子例程
(4) No loops, state-limited variables(受限於引擎的內建變量);
//不支持循環,
(5) Terminating statements with a keyword for next action as argument of the return() function, i.e.: return(action);用於實現狀態引擎轉換;
(6) Domain-specific;
//特定的領域
The VCL Finite State Machine
//VCL有限狀態機
(1) Each request is processed separately;
//每個請求是分開處理的
(2) Each request is independent from others at any given time;
//每個請求在任何給定的時間都是獨立於其他請求的
(3) States are related, but isolated;
//即便是有狀態、有關聯性,但彼此否是隔離的
(4) return(action); exits one state and instructs Varnish to proceed to the next state;
//決定下一跳是哪裏
(5) Built-in VCL code is always present and appended below your own VCL;
三類主要語法:
sub subroutine {
...
}
if CONDITION {
...
} else {
...
}
return(), hash_data()
VCL Built-in Functions and Keywords
函數:
regsub(str, regex, sub)
regsuball(str, regex, sub)
ban(boolean expression)
hash_data(input)
synthetic(str)
Keywords:
call subroutine, return(action),new,set,unset
操作符:
==, !=, ~, >, >=, <, <=
邏輯操作符:&&, ||, !
變量賦值:=
實例:obj.hits是內建變量,用於保存某緩存項的從緩存中命中的次
102.70:varnish
vim /etc/varnish/default.vcl /在最後結尾處定義
if (obj.hits > 0) {
set resp.http.X-Varnish = "Hit via " + server.ip;
} else {
set resp.http.X-Varnish = "Miss from " + server.ip;
}
用於加載default.val文件中添加的內容:
varnishadm -S /etc/varnish/secret -T 127.0.0.1:6082
//可獲取幫助:help
vcl.load conf1 default.vcl //編譯成功了 conf1:名字隨意 default.vcl:文件
200
VCL compiled.
vcl.use conf1 //使用加載conf1
200
VCL 'conf1' now active
vcl.discard //用於刪除 後面跟上要刪除的內容即可;
測試:google:172.20.102.70:6081
打開開發工具:顯示剛纔定義的內容
Via:1.1 varnish-v4
X-Varnish:Hit via 172.20.102.70
測試級:102.71
wget http://172.20.102.70:6081 //可將102.70的網站頁面能下載下來
wget -O - -q http://172.20.102.70:6081 //查看此網站的內容
問題:拒絕它訪問:
102.70:varnish
vim /etc/varnish/default.vcl //在 sub vcl_recv 處定義將它拒絕訪問
sub vcl_recv {
if(req.http.User-Agent ~ "curl") {
return(synth(403));
}
varnishadm -S /etc/varnish/secret -T 127.0.0.1:6082
varnish> vcl.load conf2 default.vcl
200
VCL compiled.
vcl.list
200
available 0 boot
active 0 conf1
available 0 conf2
vcl.use conf2
200
VCL 'conf2' now
測試機:102.71:
curl http://172.20.102.70:6081 //將會顯示拒絕訪問
問題:如有人訪問admin的URL時直接拒絕訪問:
126.180:http服務:
mkdir -pv /var/www/html/admin
vim /var/www/html/admin/index.html
admin
102.70:varnish
vim /etc/varnish/default.vcl //跟在剛纔定義的curl內容後面即可
if(req.url ~ "^/admin") {
return(synth(403));
}
varnishadm -S /etc/varnish/secret -T 127.0.0.1:6082
vcl.load conf2 default.vcl
200
VCL compiled.
vcl.use conf2
200
VCL 'conf2' now active
測試機:
wget -O - -q http://172.20.102.70:6081/admin //將無法使用admin的url的路徑
變量類型:
內建變量:
req.*:request,表示由客戶端發來的請求報文相關;
req.http.*
req.http.User-Agent, req.http.Referer, ...
bereq.*:由varnish發往BE主機的httpd請求相關;
bereq.http.*
beresp.*:由BE主機響應給varnish的響應報文相關;
beresp.http.*
resp.*:由varnish響應給client相關;
obj.*:存儲在緩存空間中的緩存對象的屬性;只讀;
常用變量:
bereq.*, req.*:
bereq.http.HEADERS
bereq.request, req.request:請求方法;
bereq.url, req.url:請求的url;
bereq.proto:請求的協議版本;
bereq.backend:指明要調用的後端主機;
req.http.Cookie:客戶端的請求報文中Cookie首部的值;
req.http.User-Agent ~ "chrome"
beresp.*, resp.*:
beresp.http.HEADERS
beresp.status, resp.status:響應的狀態碼;
reresp.proto, resp.proto:協議版本;
beresp.backend.name:BE主機的主機名;
beresp.ttl:BE主機響應的內容的餘下的可緩存時長;
obj.*
obj.hits:此對象從緩存中命中的次數;
obj.ttl:對象的ttl值
server.*
server.ip:varnish主機的IP;
server.hostname:varnish主機的Hostname;
client.*
client.ip:發請求至varnish主機的客戶端IP;
用戶自定義:
set
unset
示例1:強制對某類資源的請求不檢查緩存:
vcl_recv {
if (req.url ~ "(?i)^/(login|admin)") { //請求的url如果是login或者是admin
return(pass); //不檢查緩存
}
}
102.70:
vim /etc/varnish/default.vcl //修改內容
if(req.url ~ "^/admin") {
return(pass); //不緩存
}
varnishadm -S /etc/varnish/secret -T 127.0.0.1:6082
vcl.load conf6 default.vcl
vcl.use conf6
測試:Google:172.20.102.70:6081/admin 啓用開發工具
Via:1.1 varnish-v4
X-Varnish:Miss from 172.20.102.70 //沒有緩存
示例2:對於特定類型的資源,例如公開的圖片等,取消其私有標識,並強行設定其可以由varnish緩存的時長; 定義在vcl_backend_response中;
if (beresp.http.cache-control !~ "s-maxage") { //後端服務發的的響應報文 首部名稱:cache-control !~:不匹配不到 s-maxage
if (bereq.url ~ "(?i)\.(jpg|jpeg|png|gif|css|js)$") { //靜態內容
unset beresp.http.Set-Cookie; //取消後端服務器發來的Set-Cookie
set beresp.ttl = 3600s; //強行將靜態內容過期時間 爲1小時
}
}
注意:出現這樣情況需手動清理緩存
示例3:定義在vcl_recv中;
vim /etc/varnish/default.vcl
if (req.restarts == 0) { //請求被重啓的
if (req.http.X-Fowarded-For) { //請求報文中
set req.http.X-Forwarded-For = req.http.X-Forwarded-For + "," + client.ip;
} else {
set req.http.X-Forwarded-For = client.ip;
}
}
varnishadm -S /etc/varnish/secret -T 127.0.0.1:6082
vcl.load conf5 default.vcl
200
VCL compiled.
vcl.use conf5
200
VCL 'conf5' now active
126.180:http服務
vim /etc/httpd/conf/httpd.conf
<IfModule log_config_module>
LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
systemctl restart httpd.service
緩存對象的修剪:purge, ban
配置purge操作:
(1) 能執行purge操作
sub vcl_purge {
return (synth(200,"Purged"));
}
(2) 何時執行purge操作
sub vcl_recv {
if (req.method == "PURGE") {
return(purge);
}
...
}
實例:配置purge:
102.70:
vim /etc/varnish/default.vcl //sub vcl_recv中配置
if (req.method == "PURGE") {
return(purge);
}
sub vcl_purge {
return(synth(200,"Ok."));
}
varnishadm -S /etc/varnish/secret -T 127.0.0.1:6082
vcl.load conf6 default.vcl
200
VCL compiled.
vcl.use conf6
200
VCL 'conf6' now active
測試:curl -I http://172.20.102.70:6081
添加此類請求的訪問控制法則:
acl purgers {
"127.0.0.0"/8;
"10.1.0.0"/16;
}
sub vcl_recv {
if (req.method == "PURGE") {
if (!client.ip ~ purgers) {
return(synth(405,"Purging not allowed for " + client.ip));
}
return(purge);
}
...
}
實例:基於acl請求訪問控制:
acl不屬於任何子進程
102.70:varnish
vim /etc/varnish/default.vcl //在backend default { 中定義
acl purgers {
"127.0.0.0"/8;
}
if (req.method == "PURGE") {
if (client.ip ~ purgers) {
return(purge);
} else {
return(synth(403,"not allowed for " + client.ip));
}
}
varnishadm -S /etc/varnish/secret -T 127.0.0.1:6082
vcl.load conf7 default.vcl
200
VCL compiled.
vcl.use conf7
200
VCL 'conf7' now active
測試:客戶端:
curl -I http://172.20.102.70:6081
curl -X PURGE http://172.20.102.70:6081
<!DOCTYPE html>
<html>
<head>
<title>403 not allowed for 172.20.102.71</title>
</head>
<body>
<h1>Error 403 not allowed for 172.20.102.71</h1>
<p>not allowed for 172.20.102.71</p>
<h3>Guru Meditation:</h3>
<p>XID: 131133</p>
<hr>
<p>Varnish cache server</p>
</body>
</html>
varnish服務:
curl -X PURGE http://127.0.0.1:6081
<!DOCTYPE html>
<html>
<head>
<title>200 Ok.</title>
</head>
<body>
<h1>Error 200 Ok.</h1>
<p>Ok.</p>
<h3>Guru Meditation:</h3>
<p>XID: 131135</p>
<hr>
<p>Varnish cache server</p>
</body>
</html>
Banning:
(1) varnishadm:
ban <field> <operator> <arg>
示例:
ban req.url ~ (?i)^/javascripts
實例:不區分字符大小寫:
126.180:http
mkdir -pv /var/www/html/ja
vim /var/www/html/ja/j.txt
j.txt
102.70:varnish
varnishadm -S /etc/varnish/secret -T 127.0.0.1:6082
ban req.url ~ (?i)^/ja
200
(2) 在配置文件中定義,使用ban()函數;
示例:
if (req.method == "BAN") {
ban("req.http.host == " + req.http.host + " && req.url == " + req.url);
# Throw a synthetic page so the request won't go to the backend.
return(synth(200, "Ban added"));
}
curl -X BAN http://www.ilinux.io/test1.html
ban req.http.host==www.ilinux.io && req.url==/test1.html
如何設定使用多個後端主機:
backend default {
.host = "172.16.100.6";
.port = "80";
}
backend appsrv {
.host = "172.16.100.7";
.port = "80";
}
sub vcl_recv {
if (req.url ~ "(?i)\.php$") {
set req.backend_hint = appsrv;
} else {
set req.backend_hint = default;
}
...
}
nginx: proxy_pass
haproxy: use_backend
實例:使用負載均衡機制,實現多個後臺主機:
172.20.126.180:http
cd /var/www/html/
for i in {1..10}; do echo "test $i" > test$i.txt; done
scp -rp * [email protected]:/var/www/html/
systemctl restart httpd
172.20.128.17:http
systemctl start httpd
172.20.102.70:varnish
vim /etc/varnish/default.vcl
import directors;
# Default backend definition. Set this to point to your content server.
backend default {
.host = "172.20.126.180";
.port = "80";
}
backend srv1 {
.host = "172.20.126.180";
.port = "80";
}
backend srv2 {
.host = "172.20.128.17";
.port = "80";
}
sub vcl_init {
new websrvs = directors.round_robin();
websrvs.add_backend(srv1);
websrvs.add_backend(srv2);
}
acl purgers {
"127.0.0.0"/8;
"172.20.102.71"/32;
}
sub vcl_recv {
set req.backend_hint = websrvs.backend();
}
varnishadm -S /etc/varnish/secret -T 127.0.0.1:6082
vcl.load conf9 default.vcl
200
VCL compiled.
varnish> vcl.use conf9
200
VCL 'conf9' now active
測試:客戶端
for i in {1..10}; do curl -I -s http://172.20.102.70:6081/test$i.txt ; done //第一次無法生成緩存 第二次可以
Director:
varnish module;
使用前需要導入:
import directors;
示例:
import directors; # load the directors
backend server1 {
.host =
.port =
}
backend server2 {
.host =
.port =
}
sub vcl_init {
new GROUP_NAME = directors.round_robin();
GROUP_NAME.add_backend(server1);
GROUP_NAME.add_backend(server2);
}
sub vcl_recv {
# send all traffic to the bar director:
set req.backend_hint = GROUP_NAME.backend();
}
基於cookie的session sticky:
sub vcl_init {
new h = directors.hash();
h.add_backend(one, 1); // backend 'one' with weight '1'
h.add_backend(two, 1); // backend 'two' with weight '1'
}
sub vcl_recv {
// pick a backend based on the cookie header of the client
set req.backend_hint = h.backend(req.http.cookie);
}
基於隨機的調度方式,支持服務器權重:
sub vcl_init {
new websrvs = directors.random();
websrvs.add_backend(srv1,1);
websrvs.add_backend(srv2,2);
}
BE Health Check:
backend BE_NAME {
.host =
.port =
.probe = {
.url=
.timeout=
.interval=
.window=
.threshold=
}
}
10, 8
.probe:定義健康狀態檢測方法;
.request:發出的具體請求;
.request =
"GET /.healthtest.html HTTP/1.1"
"Host: www.magedu.com"
"Connection: close"
.window:基於最近的多少次檢查來判斷其健康狀態;
.threshold:最近.window中定義的這麼次檢查中至有.threshhold定義的次數是成功的;
.interval:檢測頻度;
.timeout:超時時長;
.expected_response:期望的響應碼,默認爲200;
健康狀態檢測的配置方式:
(1) probe PB_NAME { }
backend NAME = {
.probe = PB_NAME;
...
}
(2) backend NAME {
.probe = {
...
}
}
示例:
probe check {
.url = "/.healthcheck.html";
.window = 5;
.threshold = 4;
.interval = 2s;
.timeout = 1s;
}
backend default {
.host = "10.1.0.68";
.port = "80";
.probe = check;
}
backend appsrv {
.host = "10.1.0.69";
.port = "80";
.probe = check;
}
手動設定BE主機的狀態:
sick:管理down;
healthy:管理up;
auto:probe auto;
設置後端的主機屬性:
backend BE_NAME {
...
.connect_timeout = 0.5s;
.first_byte_timeout = 20s;
.between_bytes_timeout = 5s;
.max_connections = 50;
}
varnish的運行時參數:
線程模型:
cache-worker
cache-main
ban lurker
acceptor:
epoll/kqueue:
...
線程相關的參數:使用線程池機制管理線程;
在線程池內部,其每一個請求由一個線程來處理; 其worker線程的最大數決定了varnish的併發響應能力;
thread_pools:Number of worker thread pools. 最好小於或等於CPU核心數量;
thread_pool_max:The maximum number of worker threads in each pool. 每線程池的最大線程數;
thread_pool_min:The minimum number of worker threads in each pool. 額外意義爲“最大空閒線程數”;
最大併發連接數 = thread_pools * thread_pool_max
thread_pool_timeout:Thread idle threshold. Threads in excess of thread_pool_min, which have been idle for at least this long, will be destroyed.
thread_pool_add_delay:Wait at least this long after creating a thread.
thread_pool_destroy_delay:Wait this long after destroying a thread.
Timer相關的參數:
send_timeout:Send timeout for client connections. If the HTTP response hasn't been transmitted in this many seconds the session is closed.
timeout_idle:Idle timeout for client connections.
timeout_req: Max time to receive clients request headers, measured from first non-white-space character to double CRNL.
cli_timeout:Timeout for the childs replies to CLI requests from the mgt_param.
設置方式:
vcl.param
param.set
永久有效的方法:
varnish.params
DEAMON_OPTS="-p PARAM1=VALUE -p PARAM2=VALUE"
實例:
102.70:varnish
vim /etc/varnish/varnish.params
Other options, see the man page varnishd(1) //默認註釋
DAEMON_OPTS="-p thread_pools=4 -p thread_pool_min=50 -p thread_pool_max=2000 -p thread_pool_timeout=300"
varnishadm -S /etc/varnish/secret -T 127.0.0.1:6082
param.set thread_pools 4
200
varnish日誌區域:
shared memory log
計數器
日誌信息
1、varnishstat - Varnish Cache statistics
-1
-1 -f FILED_NAME
-l:可用於-f選項指定的字段名稱列表;
MAIN.cache_hit
MAIN.cache_miss
# varnishstat -1 -f MAIN.cache_hit -f MAIN.cache_miss
顯示指定參數的當前統計數據;
# varnishstat -l -f MAIN -f MEMPOOL
列出指定配置段的每個參數的意義;
2、varnishtop - Varnish log entry ranking
-1 Instead of a continously updated display, print the statistics once and exit.
-i taglist,可以同時使用多個-i選項,也可以一個選項跟上多個標籤;
-I <[taglist:]regex>:對指定的標籤的值基於regex進行過濾;
-x taglist:排除列表
-X <[taglist:]regex>:對指定的標籤的值基於regex進行過濾,符合條件的予以排除;
3、varnishlog - Display Varnish logs
4、 varnishncsa - Display Varnish logs in Apache / NCSA combined log format
內建函數:
hash_data():指明哈希計算的數據;減少差異,以提升命中率;
regsub(str,regex,sub):把str中被regex第一次匹配到字符串替換爲sub;主要用於URL Rewrite
regsuball(str,regex,sub):把str中被regex每一次匹配到字符串均替換爲sub;
return():
ban(expression)
ban_url(regex):Bans所有的其URL可以被此處的regex匹配到的緩存對象;
synth(status,"STRING"):生成響應報文;
總結:
varnish: state engine, vcl
varnish 4.0:
vcl_init
vcl_recv
vcl_hash
vcl_hit
vcl_pass
vcl_miss
vcl_pipe
vcl_waiting
vcl_purge
vcl_deliver
vcl_synth
vcl_fini
vcl_backend_fetch
vcl_backend_response
vcl_backend_error
sub VCL_STATE_ENGINE {
...
}
backend BE_NAME {}
probe PB_NAME {}
acl ACL_NAME {}