kubernetes flannel neutron calico 網絡性能測試分析
環境準備
- OS: Ubuntu 14.04
- kubernetes 版本：1.3.5
- flannel 版本： 0.5.5
- calico 版本: 1.6.0
- neutron 版本：6.0.0
- iperf 版本：2.0.5
- 物理機配置：
4 Intel(R) Core(TM) i5-4460S CPU @ 2.90GHz
8G memory千兆網卡
搭建kubernetes相關組件，並配置不同的網絡模式
|
節點類型
|
ip地址
|
備註
|
|
master/node
|
10.100.100.244
|
k8s的master跟node節點共用
|
|
node
|
10.100.100.245
|
|
|
neutron網絡節點
|
10.100.100.246
|
爲k8s提供neutron網絡
|
root@hc0:~# kubectl get nodeNAME STATUS AGE
10.100.100.244 Ready 6d
10.100.100.245 Ready 6d
性能評測指標：
- ping延遲: 用ping測試hosts之間和pods之間的延遲
- 帶寬測試: 用iperf測試hosts之間和pods之間的帶寬
- HTTP性能測試: 部署單進程nginx server並使用apache benchmark(ab)測試
物理機性能評測
ping延遲
登錄到10.100.100.244，運行ping 10.100.100.245測試延遲：
root@hc0:~# ping 10.100.100.245PING 10.100.100.245 (10.100.100.245) 56(84) bytes of data.
64 bytes from 10.100.100.245: icmp_seq=1 ttl=64 time=0.247 ms
64 bytes from 10.100.100.245: icmp_seq=2 ttl=64 time=0.221 ms
64 bytes from 10.100.100.245: icmp_seq=3 ttl=64 time=0.200 ms
64 bytes from 10.100.100.245: icmp_seq=4 ttl=64 time=0.234 ms
64 bytes from 10.100.100.245: icmp_seq=5 ttl=64 time=0.211 ms
64 bytes from 10.100.100.245: icmp_seq=6 ttl=64 time=0.212 ms
64 bytes from 10.100.100.245: icmp_seq=7 ttl=64 time=0.249 ms
64 bytes from 10.100.100.245: icmp_seq=8 ttl=64 time=0.227 ms
64 bytes from 10.100.100.245: icmp_seq=9 ttl=64 time=0.245 ms
64 bytes from 10.100.100.245: icmp_seq=10 ttl=64 time=0.278 ms
64 bytes from 10.100.100.245: icmp_seq=11 ttl=64 time=0.206 ms
平均值爲： 0.244842ms
注：可以通過下面的腳本例子計算ping延遲的平均值
root@hc0:~#
ping 10.100.100.245 | head -n 20 | gawk '/time/ {split($7, ss, "="); sum+=ss[2]; count+=1;} END{print sum/count "ms";}'0.244842ms
相互之間多次ping測試。延遲一般在0.22ms-0.26ms之間
iperf帶寬
apt-get install iperf 安裝iperf後，在10.100.100.245上啓動iperf server:
iperf -s，在10.100.100.244上啓動客戶端，執行下面命令:root@hc0:~# iperf -c 10.100.100.245 -i 1
------------------------------------------------------------
Client connecting to 10.100.100.245, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[ 3] local 10.100.100.244 port 49980 connected with 10.100.100.245 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 1.0 sec 113 MBytes 945 Mbits/sec
[ 3] 1.0- 2.0 sec 112 MBytes 942 Mbits/sec
[ 3] 2.0- 3.0 sec 112 MBytes 941 Mbits/sec
[ 3] 3.0- 4.0 sec 112 MBytes 942 Mbits/sec
[ 3] 4.0- 5.0 sec 112 MBytes 943 Mbits/sec
[ 3] 5.0- 6.0 sec 112 MBytes 943 Mbits/sec
[ 3] 6.0- 7.0 sec 112 MBytes 941 Mbits/sec
[ 3] 7.0- 8.0 sec 112 MBytes 942 Mbits/sec
[ 3] 8.0- 9.0 sec 112 MBytes 941 Mbits/sec
[ 3] 9.0-10.0 sec 112 MBytes 943 Mbits/sec
[ 3] 0.0-10.0 sec 1.10 GBytes 942 Mbits/sec
root@hc0:~# iperf -c 10.100.100.245 -u -t 60 -b 1000M
------------------------------------------------------------
Client connecting to 10.100.100.245, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size: 208 KByte (default)
------------------------------------------------------------
[ 3] local 10.100.100.244 port 53440 connected with 10.100.100.245 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-60.0 sec 5.67 GBytes 811 Mbits/sec
[ 3] Sent 4139127 datagrams
[ 3] Server Report:
[ 3] 0.0-60.0 sec 5.66 GBytes810 Mbits/sec 0.009 ms 7469/4139126 (0.18%)
[ 3] 0.0-60.0 sec 1 datagrams received out-of-order
root@hc1:~# iperf -c 10.100.100.244 -u -t 60 -b 2000M
------------------------------------------------------------
Client connecting to 10.100.100.244, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size: 208 KByte (default)
------------------------------------------------------------
[ 3] local 10.100.100.245 port 46402 connected with 10.100.100.244 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-60.0 sec 5.67 GBytes 811 Mbits/sec
[ 3] Sent 4138997 datagrams
[ 3] Server Report:
[ 3] 0.0-60.0 sec 5.66 GBytes810 Mbits/sec 0.043 ms 3929/4138996 (0.095%)
[ 3] 0.0-60.0 sec 1 datagrams received out-of-order
物理機iperf帶寬 942 Mbits/sec
nginx
benchmark
先在10.100.100.244上，執行
docker run --net=host nginx啓動一個單進程的nginx服務，然後在10.100.100.245上使用ab
-n 90000 -c 50 http://10.100.100.244/開始壓測，結果如下：root@hc1:~# ab -n 90000 -c 50 http://10.100.100.244/
This is ApacheBench, Version 2.3 <$Revision: 1528965 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking 10.100.100.244 (be patient)
Completed 9000 requests
Completed 18000 requests
Completed 27000 requests
Completed 36000 requests
Completed 45000 requests
Completed 54000 requests
Completed 63000 requests
Completed 72000 requests
Completed 81000 requests
Completed 90000 requests
Finished 90000 requests
Server Software: nginx/1.11.5
Server Hostname: 10.100.100.244
Server Port: 80
Document Path: /
Document Length: 612 bytes
Concurrency Level: 50
Time taken for tests: 7.235 seconds
Complete requests: 90000
Failed requests: 0
Total transferred: 76050000 bytes
HTML transferred: 55080000 bytes
Requests per second: 12439.61 [#/sec] (mean)
Time per request: 4.019 [ms] (mean)
Time per request: 0.080 [ms] (mean, across all concurrent requests)
Transfer rate: 10265.11 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.1 0 2
Processing: 1 4 1.0 4 9
Waiting: 1 4 1.0 4 9
Total: 1 4 0.9 4 9
Percentage of the requests served within a certain time (ms)
50% 4
66% 4
75% 4
80% 5
90% 5
95% 5
98% 5
99% 6
100% 9 (longest request)
neutron模式評測
ping延遲
在pod之間的ping延遲，啓動兩個iperf pod，啓動在不同的host上。然後執行下面的命令獲得ping延遲:
root@hc0:~/yaml# kubectl get pod -owide --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE
net1-sub1 net-test-4v9av 1/1 Running 0 29s 192.168.11.33 10.100.100.245
net1-sub1 net-test-ztu5r 1/1 Running 0 29s 192.168.11.34 10.100.100.244
root@hc0:~# kubectl exec -it net-test-ztu5r bash --namespace=net1-sub1
root@net-test-ztu5r:/#
ping
192.168.11.33 | head -n 20 | gawk '/time/ {split($7, ss, "="); sum+=ss[2]; count+=1;} END{print sum/count "ms";}'0.747789ms
相互之間多次ping測試，延遲一般在0.71ms-0.77ms之間
iperf帶寬
pod之間的iperf帶寬:
net-test-4v9av作爲server。然後從net-test-ztu5r訪問net-test-4v9av的IP來測試帶寬。
root@net-test-ztu5r:/# iperf -c 192.168.11.33 -u -t 60 -b 2000M
------------------------------------------------------------
Client connecting to 192.168.11.33, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size: 208 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.11.34 port 40863 connected with 192.168.11.33 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-60.0 sec 5.66 GBytes 810 Mbits/sec
[ 3] Sent 4131942 datagrams
[ 3] WARNING: did not receive ack of last datagram after 10 tries.
nginx benchmark
pod內的nginx性能：
用RC創建Nginx pod，以及service:
kubectl create -f nginx-rc.yamlkubectl create -f nginx-svc.yamlroot@hc0:~# kubectl get pod --namespace=net1-sub1 -owide
NAME READY STATUS RESTARTS AGE IP NODE
net-test-4v9av 1/1 Running 0 25m 192.168.11.33 10.100.100.245
net-test-ztu5r 1/1 Running 0 25m 192.168.11.34 10.100.100.244
nginx-0v5r8 1/1 Running 0 30s 192.168.11.36 10.100.100.244
nginx-pnxzt 1/1 Running 0 30s 192.168.11.35 10.100.100.245
net-test-4v9av pod使用podIP訪問
ab
-n 90000 -c 50 http://192.168.11.36/，性能相近。：root@net-test-4v9av:/# ab -n 90000 -c 50 http://192.168.11.36/
This is ApacheBench, Version 2.3 <$Revision: 1528965 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking 192.168.11.36 (be patient)
Completed 9000 requests
Completed 18000 requests
Completed 27000 requests
Completed 36000 requests
Completed 45000 requests
Completed 54000 requests
Completed 63000 requests
Completed 72000 requests
Completed 81000 requests
Completed 90000 requests
Finished 90000 requests
Server Software: nginx/1.11.5
Server Hostname: 192.168.11.36
Server Port: 80
Document Path: /
Document Length: 612 bytes
Concurrency Level: 50
Time taken for tests: 9.928 seconds
Complete requests: 90000
Failed requests: 0
Total transferred: 76050000 bytes
HTML transferred: 55080000 bytes
Requests per second: 9065.38 [#/sec] (mean)
Time per request: 5.515 [ms] (mean)
Time per request: 0.110 [ms] (mean, across all concurrent requests)
Transfer rate: 7480.71 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 1 1.3 0 12
Processing: 0 4 1.7 5 11
Waiting: 0 4 1.7 5 11
Total: 0 5 1.4 6 21
Percentage of the requests served within a certain time (ms)
50% 6
66% 6
75% 6
80% 6
90% 7
95% 7
98% 9
99% 9
100% 21 (longest request)
使用flannel vxlan模式評測：
pod之間的ping延遲：
操作步驟同上
root@hc0:~/yaml# kubectl get pod -owide --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE
net1-sub1 net-test-ipgc5 1/1 Running 0 9s 172.16.21.2 10.100.100.244
net1-sub1 net-test-iyfds 1/1 Running 0 9s 172.16.78.2 10.100.100.245
root@hc0:~# kubectl exec -it net-test-ipgc5
bash --namespace=net1-sub1
root@net-test-ipgc5:/#
ping 172.16.78.2
| head -n 20 | gawk '/time/ {split($7, ss, "="); sum+=ss[2]; count+=1;} END{print sum/count "ms";}'
0.379456ms
平均延遲：0.379456ms
pod之間的帶寬
操作步驟同上
root@net-test-iyfds:/#
iperf -c 172.16.21.2
-t 60------------------------------------------------------------
Client connecting to 172.16.21.2,
TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[ 3] local172.16.78.2
port 45646 connected with 192.168.202.4 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-60.0 sec 6.28 GBytes 910 Mbits/sec
flannel的iperf帶寬 910 Mbits/sec
nginx性能
操作步驟同上
root@net-test-b917n:/# ab -n 90000 -c 50 http://172.16.21.3/
This is ApacheBench, Version 2.3 <$Revision: 1528965 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking
172.16.21.3 (be patient)
Completed 9000 requests
Completed 18000 requests
Completed 27000 requests
Completed 36000 requests
Completed 45000 requests
Completed 54000 requests
Completed 63000 requests
Completed 72000 requests
Completed 81000 requests
Completed 90000 requests
Finished 90000 requests
Server Software: nginx/1.11.5
Server Hostname:
172.16.21.3
Server Port: 80
Document Path: /
Document Length: 612 bytes
Concurrency Level: 50
Time taken for tests: 8.945 seconds
Complete requests: 90000
Failed requests: 0
Total transferred: 76050000 bytes
HTML transferred: 55080000 bytes
Requests per second: 10804.44 [#/sec] (mean)
Time per request: 4.928 [ms] (mean)
Time per request: 0.093 [ms] (mean, across all concurrent requests)
Transfer rate: 8814.89 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.2 0 5
Processing: 1 4 1.3 5 10
Waiting: 1 4 1.3 5 9
Total: 2 5 1.3 5 10
Percentage of the requests served within a certain time (ms)
50% 5
66% 5
75% 5
80% 5
90% 5
95% 6
98% 6
99% 6
100% 10 (longest request)
Calico網絡性能
- 根據http://docs.projectcalico.org/v1.6/getting-started/kubernetes/installation/#configuring-kubernetes完成集羣的部署
ping延遲
評測方法同上
root@hc0:# kubectl get pod -owide --namespace=net1-sub1
NAME READY STATUS RESTARTS AGE IP NODE
net-test-b917n 1/1 Running 0 11m 192.168.145.132 10.100.100.245
net-test-w98x9 1/1 Running 0 11m 192.168.202.4 10.100.100.244
root@net-test-w98x9:/# ping 192.168.145.132 | head -n 20 | gawk '/time/ {split($7, ss, "="); sum+=ss[2]; count+=1;} END{print sum/count "ms";}'
0.267684ms
多次測試平均延遲：0.25ms-0.31ms之間
iperf帶寬
評測方法同上
tcp：
root@net-test-w98x9:/# iperf -s ------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[ 4] local 192.168.202.4 port 5001 connected with 192.168.145.132 port 54228
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-30.0 sec 3.29 GBytes 941 Mbits/sec
root@net-test-b917n:/# iperf -c 192.168.202.4 -t 60
------------------------------------------------------------
Client connecting to 192.168.202.4, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.145.132 port 54255 connected with 192.168.202.4 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-60.0 sec 6.58 GBytes 942 Mbits/sec
udp：
root@net-test-b917n:/# iperf -c 192.168.202.4 -u -b 1000M -t 10
------------------------------------------------------------
Client connecting to 192.168.202.4, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size: 208 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.145.132 port 60316 connected with 192.168.202.4 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 968 MBytes 812 Mbits/sec
[ 3] Sent 690167 datagrams
[ 3] Server Report:
[ 3] 0.0-10.0 sec 966 MBytes 811 Mbits/sec 0.028 ms 762/690166 (0.11%)
[ 3] 0.0-10.0 sec 1 datagrams received out-of-order
calico的iperf帶寬 941 Mbits/sec
nginx benchmark
評測方法同上
root@net-test-b917n:/# ab -n 90000 -c 50 http://192.168.202.6/
This is ApacheBench, Version 2.3 <$Revision: 1528965 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking 192.168.202.6 (be patient)
Completed 9000 requests
Completed 18000 requests
Completed 27000 requests
Completed 36000 requests
Completed 45000 requests
Completed 54000 requests
Completed 63000 requests
Completed 72000 requests
Completed 81000 requests
Completed 90000 requests
Finished 90000 requests
Server Software: nginx/1.11.5
Server Hostname: 192.168.202.6
Server Port: 80
Document Path: /
Document Length: 612 bytes
Concurrency Level: 50
Time taken for tests: 8.330 seconds
Complete requests: 90000
Failed requests: 0
Total transferred: 76050000 bytes
HTML transferred: 55080000 bytes
Requests per second: 10031.23 [#/sec] (mean)
Time per request: 4.886 [ms] (mean)
Time per request: 0.093 [ms] (mean, across all concurrent requests)
Transfer rate: 8915.77 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.2 0 5
Processing: 1 4 1.3 5 10
Waiting: 1 4 1.3 5 9
Total: 2 5 1.3 5 10
Percentage of the requests served within a certain time (ms)
50% 5
66% 5
75% 5
80% 5
90% 5
95% 6
98% 6
99% 6
100% 10 (longest request)
結論
| 網絡類型 | 延遲 | 帶寬 | nginx(QPS/延遲) |
| 物理 | 0.244842ms | 942Mb/s | 12439.61/4.019 |
| neutron | 0.747789ms | 907Mb/s | 9065.38/5.515 |
| flannel vxlan | 0.379456ms | 910Mb/s | 10031.23/4.886 |
| Calico | 0.267684ms | 941Mb/s | 10804.44/4.628 |
TODO
- ovs網絡性能測試
- ovn網絡性能測試
/***************************2月24日更新**************************/
ovs vxlan網絡性能
ping延遲
評測方法同上
root@hc0:# kubectl get pod -owide
NAME READY STATUS RESTARTS AGE IP NODE
net-test-278vs 1/1 Running 0 25s 172.16.75.2 10.100.100.244
net-test-vtva6 1/1 Running 0 25s 172.16.61.2 10.100.100.245
root@net-test-278vs:/# ping 172.16.61.2 | head -n 20 | gawk '/time/ {split($7, ss, "="); sum+=ss[2]; count+=1;} END{print sum/count "ms";}'
0.35475ms
iperf帶寬
評測方法同上
tcp：
root@net-test-278vs:/# iperf -c 172.16.61.2 -t 20
------------------------------------------------------------
Client connecting to 172.16.61.2, TCP port 5001
TCP window size: 45.0 KByte (default)
------------------------------------------------------------
[ 3] local 172.16.75.2 port 39574 connected with 172.16.61.2 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-20.0 sec 2.14 GBytes 917 Mbits/sec
ovs的iperf帶寬 917 Mbits/sec
nginx benchmark
評測方法同上
root@net-test-278vs:/# ab -n 90000 -c 50 http://172.16.61.3/
This is ApacheBench, Version 2.3 <$Revision: 1528965 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking 172.16.61.3 (be patient)
Completed 9000 requests
Completed 18000 requests
Completed 27000 requests
Completed 36000 requests
Completed 45000 requests
Completed 54000 requests
Completed 63000 requests
Completed 72000 requests
Completed 81000 requests
Completed 90000 requests
Finished 90000 requests
Server Software: nginx/1.11.5
Server Hostname: 172.16.80.2
Server Port: 80
Document Path: /
Document Length: 612 bytes
Concurrency Level: 50
Time taken for tests: 7.855 seconds
Complete requests: 90000
Failed requests: 0
Total transferred: 76050000 bytes
HTML transferred: 55080000 bytes
Requests per second:11457.32 [#/sec] (mean)
Time per request:4.364 [ms] (mean)
Time per request: 0.087 [ms] (mean, across all concurrent requests)
Transfer rate: 9454.53 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 2 0.3 2 4
Processing: 1 2 0.5 2 13
Waiting: 1 2 0.5 2 13
Total: 2 4 0.4 4 15
Percentage of the requests served within a certain time (ms)
50% 4
66% 4
75% 5
80% 5
90% 5
95% 5
98% 5
99% 5
100% 15 (longest request)
結論
| 網絡類型 | 延遲 | 帶寬 | nginx(QPS/延遲) |
|---|---|---|---|
| 物理 | 0.244842ms | 942Mb/s | 12439.61/4.019 |
| neutron | 0.747789ms | 907Mb/s | 9065.38/5.515 |
| flannel vxlan | 0.727158ms | 910Mb/s | 10031.23/4.886 |
| Calico | 0.267684ms | 941Mb/s | 10804.44/4.628 |
| ovs vxlan | 0.35475ms | 917Mb/s | 11457.32/4.364 |