參考鏈接：https://www.cnblogs.com/goldsunshine/p/11720310.html
1.默認的流表
當你啓動任何一個ryu app之後，交換機的流表就被設置爲默認的下面的內容
root@user-NF8480M5:/home/user/qyq/dragonflow/main# ovs-ofctl dump-flows ckltest-br
NXST_FLOW reply (xid=0x4):
cookie=0x0, duration=1.737s, table=0, n_packets=0, n_bytes=0, idle_age=1, priority=1,arp actions=CONTROLLER:65535
cookie=0x0, duration=1.737s, table=0, n_packets=0, n_bytes=0, idle_age=1, priority=1,ip actions=drop
cookie=0x0, duration=1.737s, table=0, n_packets=0, n_bytes=0, idle_age=1, priority=0 actions=NORMAL
2.默認流表下的流量轉發
以同一個ovs上的主機Aping主機B爲例，默認的話是ping不通的，僅僅增加arp表項的處理包個數
例如，下面的是192.168.1.2 ping 192.168.0.3,會首先尋找默認網關的mac地址
root@user-NF8480M5:/home/user# tcpdump -i tapq
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tapq, link-type EN10MB (Ethernet), capture size 262144 bytes
16:29:36.922760 ARP, Request who-has 192.168.1.1 tell 192.168.1.2, length 28
16:29:37.939530 ARP, Request who-has 192.168.1.1 tell 192.168.1.2, length 28
16:29:38.963532 ARP, Request who-has 192.168.1.1 tell 192.168.1.2, length 28
16:29:39.987755 ARP, Request who-has 192.168.1.1 tell 192.168.1.2, length 28
16:29:41.011541 ARP, Request who-has 192.168.1.1 tell 192.168.1.2, length 28
16:29:42.035539 ARP, Request who-has 192.168.1.1 tell 192.168.1.2, length 28
剛好被流表接受到6個流量包
NXST_FLOW reply (xid=0x4):
cookie=0x0, duration=574.422s, table=0, n_packets=6, n_bytes=252, idle_age=560, priority=1,arp actions=CONTROLLER:65535
cookie=0x0, duration=574.422s, table=0, n_packets=0, n_bytes=0, idle_age=574, priority=1,ip actions=drop
cookie=0x0, duration=574.422s, table=0, n_packets=0, n_bytes=0, idle_age=574, priority=0 actions=NORMAL
3.CONTROLLER:65535是什麼意思？
是指將流量包轉發給控制器的意思
4.192.168.1.2 ping 192.168.0.3的提交給控制器的流量包的分析
第一個包（arp表項中有網關的地址）
packet_in_handler!, header_list is
{
'ethernet':
ethernet(dst='fa:c0:29:b8:c6:77',
ethertype=2048,src='86:3c:7c:4b:f1:7a'),
'icmp': icmp(code=0,csum=31873,
data=echo(data='\xce,\xc6^\x00\x00\x00\x00\x05
\x0f\x0f\x00\x00\x00\x00\x00\x10\x11\x12\x13\x14
\x15\x16\x17\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f!"
#$%&\'()*+,-./01234567',id=5136,seq=1),type=8),
'ipv4':
ipv4(csum=13132,dst='192.168.0.3',flags=2,
header_length=5,identification=34055,offset=0,
option=None,proto=1,src='192.168.1.2',tos=0,
total_length=84,ttl=64,version=4)
}
dst_mac是網關的mac地址，
ipv4的源ip是192.168.1.2，目的ip是192.168.0.3
可以看一下網卡的mac
root@user-NF8480M5:/home/user/qyq/dragonflow/main# ip netn exec 82341 ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default
qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
1040: vtapq@if1041: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state
UP group default qlen 1000
link/ether 86:3c:7c:4b:f1:7a brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 192.168.1.2/24 scope global vtapq
valid_lft forever preferred_lft forever
root@user-NF8480M5:/home/user/qyq/dragonflow/main# ip netn exec 82341 arp -a
? (192.168.1.1) 位於 fa:c0:29:b8:c6:77 [ether] 在 vtapq
下面是第一個包的發送過程，192.168.1.2 發送到的192.168.0.3的icmp包，但是mac地址是網關的mac地址
1, packet_in_handler
dp_id is 38074417935428
Router, msg.data
packet_in_handler!, header_list is {'ethernet': ethernet(dst='fa:c0:29:b8:c6:77',ethertype=2048,
src='86:3c:7c:4b:f1:7a'), 'icmp': icmp(code=0,csum=3622,
data=echo(data='\x8dC\xc6^\x00\x00\x00\x00t\x9b\x06
\x00\x00\x00\x00\x00\x10\x11\x12\x13\x14\x15\x16\x17
\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f !"
#$%&\'()*+,-./01234567',id=23752,seq=1),type=8),
'ipv4':ipv4(csum=12866,dst='192.168.0.3',
flags=2,header_length=5,identification=34321,
offset=0,option=None,proto=1,
src='192.168.1.2',tos=0,total_length=84,ttl=64,version=4)}
IPV4 in header_list
[RT][INFO] switch_id=000022a0e427f444:
Receive IP packet from [192.168.1.2] to an internal host [192.168.0.3].
send_arp_request, src_ip is 192.168.0.1, dst_ip is 192.168.0.3
[RT][INFO] switch_id=000022a0e427f444: Send ARP request (flood)
注意看倒數第二行，這個時候192.168.0.1（OVS網橋）發送了ARP報文，這個也可以從抓包分析裏面得到結果。。
下面對192.168.0.3的網卡進行抓包
17:25:23.233879 ARP, Request who-has 192.168.0.1 tell 192.168.0.1, length 46
（初始化的時候）
17:25:23.241732 ARP, Request who-has 192.168.1.1 tell 192.168.1.1, length 46
（初始化的時候）
17:25:41.149465 ARP, Request who-has 192.168.0.3 tell 192.168.0.1, length 46
17:25:41.149489 ARP, Reply 192.168.0.3 is-at ca:4b:f7:92:f2:ef (oui Unknown), length 28
（第一次ping的時候，OVS需要知道192.168.0.3的MAC地址）
17:25:41.153425 IP 192.168.1.2 > 192.168.0.3: ICMP echo request, id 28032, seq 1, length 64
17:25:41.153453 IP 192.168.0.3 > 192.168.1.2: ICMP echo reply, id 28032, seq 1, length 64
（走下面的流表中被我標註的規則）
17:25:42.144964 IP 192.168.1.2 > 192.168.0.3: ICMP echo request, id 28032, seq 2, length 64
17:25:42.145003 IP 192.168.0.3 > 192.168.1.2: ICMP echo reply, id 28032, seq 2, length 64
下面是流表中的規則，其中我做記號的幾行是被實際流量包走的規則
cookie=0x1, duration=418.888s, table=0, n_packets=0, n_bytes=0,
idle_age=418, priority=1037,ip,nw_dst=192.168.0.1 actions=CONTROLLER:65535
cookie=0x2, duration=418.880s, table=0, n_packets=0, n_bytes=0,
idle_age=418, priority=1037,ip,nw_dst=192.168.1.1 actions=CONTROLLER:65535
# 192.168.1.2 ping 192.168.0.3是直接走的下面這兩行
cookie=0x2, duration=400.960s, table=0, n_packets=12, n_bytes=1176,
idle_timeout=1800, idle_age=2, priority=35,ip,
nw_dst=192.168.1.2 actions=dec_ttl,mod_dl_src:fa:c0:29:b8:c6:77,
mod_dl_dst:86:3c:7c:4b:f1:7a,output:1029
cookie=0x1, duration=0.402s, table=0, n_packets=12, n_bytes=1176,
idle_timeout=1800, idle_age=2, priority=35,ip,
nw_dst=192.168.0.3 actions=dec_ttl,mod_dl_src:46:0e:bf:4a:7c:11,
mod_dl_dst:ca:4b:f7:92:f2:ef,output:1028
cookie=0x1, duration=418.888s, table=0, n_packets=0, n_bytes=0,
idle_age=418, priority=36,ip,nw_src=192.168.0.0/24,nw_dst=192.168.0.0/24
actions=NORMAL
cookie=0x2, duration=418.880s, table=0, n_packets=0, n_bytes=0,
idle_age=418, priority=36,ip,nw_src=192.168.1.0/24,nw_dst=192.168.1.0/24
actions=NORMAL
cookie=0x1, duration=418.888s, table=0, n_packets=1, n_bytes=98,
idle_age=400, priority=2,ip,nw_dst=192.168.0.0/24 actions=CONTROLLER:65535
cookie=0x2, duration=418.880s, table=0, n_packets=1, n_bytes=98,
idle_age=400, priority=2,ip,nw_dst=192.168.1.0/24 actions=CONTROLLER:65535
# arp走的是下面的這行規則
cookie=0x0, duration=495.466s, table=0, n_packets=47, n_bytes=1974,
idle_age=0, priority=1,arp actions=CONTROLLER:65535
cookie=0x0, duration=495.466s, table=0, n_packets=9, n_bytes=756,
idle_age=4666, priority=1,ip actions=drop
cookie=0x0, duration=495.466s, table=0, n_packets=12, n_bytes=1062,
idle_age=2666, priority=0 actions=NORMAL
下面是第二個到達OVS的包，是192.168.0.3向192.168.0.1的arp響應報文
packet_in_handler!, header_list is {'arp': arp(dst_ip='192.168.0.1',dst_mac='46:0e:bf:4a:7c:11',
hlen=6,hwtype=1,opcode=2,plen=4,proto=2048,src_ip='192.168.0.3',
src_mac='ca:4b:f7:92:f2:ef'),
'ethernet': ethernet(dst='46:0e:bf:4a:7c:11',ethertype=2054,src='ca:4b:f7:92:f2:ef')
}
這個包是上一次緩存了ICMP包，並向所有網口洪泛了的ARP包，當192.168.0.3接收到ARP請求報文時，會發送ARP響應包給OVS
def _packetin_to_node(self, msg, header_list):
if len(self.packet_buffer) >= MAX_SUSPENDPACKETS:
self.logger.info('Packet is dropped, MAX_SUSPENDPACKETS exceeded.',
extra=self.sw_id)
return
# Send ARP request to get node MAC address.
in_port = self.ofctl.get_packetin_inport(msg)
src_ip = None
dst_ip = header_list[IPV4].dst
srcip = ip_addr_ntoa(header_list[IPV4].src)
dstip = ip_addr_ntoa(dst_ip)
address = self.address_data.get_data(ip=dst_ip)
if address is not None:
log_msg = 'Receive IP packet from [%s] to an internal host [%s].'
self.logger.info(log_msg, srcip, dstip, extra=self.sw_id)
src_ip = address.default_gw # 192.168.1.2, 192.168.1.1
else:
route = self.routing_tbl.get_data(dst_ip=dst_ip)
if route is not None:
log_msg = 'Receive IP packet from [%s] to [%s].'
self.logger.info(log_msg, srcip, dstip, extra=self.sw_id)
gw_address = self.address_data.get_data(ip=route.gateway_ip)
if gw_address is not None:
src_ip = gw_address.default_gw
dst_ip = route.gateway_ip
if src_ip is not None:
self.packet_buffer.add(in_port, header_list, msg.data)
self.send_arp_request(src_ip, dst_ip, in_port=in_port) // 這一行是重點
self.logger.info('Send ARP request (flood)', extra=self.sw_id)
下面是第二個到達OVS的包，是192.168.0.3向192.168.0.1的arp響應報文
packet_in_handler!, header_list is {'arp': arp(dst_ip='192.168.0.1',dst_mac='46:0e:bf:4a:7c:11',
hlen=6,hwtype=1,opcode=2,plen=4,proto=2048,src_ip='192.168.0.3',
src_mac='ca:4b:f7:92:f2:ef'),
'ethernet': ethernet(dst='46:0e:bf:4a:7c:11',ethertype=2054,src='ca:4b:f7:92:f2:ef')
}
處理程序是自己寫的
def packet_in_handler(self, msg, header_list):
# Check invalid TTL (for OpenFlow V1.2/1.3)
ofproto = self.dp.ofproto
if ofproto.OFP_VERSION == ofproto_v1_2.OFP_VERSION or \
ofproto.OFP_VERSION == ofproto_v1_3.OFP_VERSION:
if msg.reason == ofproto.OFPR_INVALID_TTL:
self._packetin_invalid_ttl(msg, header_list)
return
# Analyze event type.
if ARP in header_list:
self._packetin_arp(msg, header_list)
return
.............
self._packetin_arp
def _packetin_arp(self, msg, header_list):
src_addr = self.address_data.get_data(ip=header_list[ARP].src_ip)
if src_addr is None:
return
# case: Receive ARP from the gateway
# Update routing table.
# case: Receive ARP from an internal host
# Learning host MAC.
gw_flg = self._update_routing_tbl(msg, header_list)
if gw_flg is False:
self._learning_host_mac(msg, header_list)
..........
應該是receive ARP from an inernal host
def _learning_host_mac(self, msg, header_list):
# Set flow: routing to internal Host.
out_port = self.ofctl.get_packetin_inport(msg)
src_mac = header_list[ARP].src_mac
dst_mac = self.port_data[out_port].mac
src_ip = header_list[ARP].src_ip
gateways = self.routing_tbl.get_gateways()
if src_ip not in gateways:
address = self.address_data.get_data(ip=src_ip)
if address is not None:
cookie = self._id_to_cookie(REST_ADDRESSID, address.address_id)
priority = self._get_priority(PRIORITY_IMPLICIT_ROUTING)
self.ofctl.set_routing_flow(cookie, priority,
out_port, dl_vlan=self.vlan_id,
src_mac=dst_mac, dst_mac=src_mac,
nw_dst=src_ip,
idle_timeout=IDLE_TIMEOUT,
dec_ttl=True)
self.logger.info('Set implicit routing flow [cookie=0x%x]',
cookie, extra=self.sw_id)
下面是第三個到達OVS的包，是192.168.0.3到192.168.1.2的icmp echo response報文
packet_in_handler!,
header_list is
{'ethernet': ethernet(dst='46:0e:bf:4a:7c:11',ethertype=2048,src='ca:4b:f7:92:f2:ef'), 'icmp':
icmp(code=0,csum=16750,
data=echo(data='\xdbg\xc7^\x00\x00\x00\x00\xb7\x07
\x0f\x00\x00\x00\x00\x00\x10\x11\x12\x13\x14\x15
\x16\x17\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f !"#$%&\'()*+,
-./01234567',id=38639,seq=1),type=0),
'ipv4':
ipv4(csum=12313,dst='192.168.1.2',
flags=0,
header_length=5,identification=51258,
offset=0,option=None,
proto=1,
src='192.168.0.3',
tos=0,total_length=84,ttl=64,version=4)}
這個包的處理程序也是自己寫的
下面第四個到達OVS的包，是192.168.1.2向192.168.1.1的arp響應報文
packet_in_handler!, header_list is
{
'arp':
arp(dst_ip='192.168.1.1',dst_mac='fa:c0:29:b8:c6:77',
hlen=6,hwtype=1,opcode=2,plen=4,proto=2048,
src_ip='192.168.1.2',src_mac='86:3c:7c:4b:f1:7a'),
'ethernet':
ethernet(dst='fa:c0:29:b8:c6:77',
ethertype=2054,
src='86:3c:7c:4b:f1:7a')
}
5.新增加的規則分析
cookie=0x3, duration=1.625s, table=0, n_packets=1,
n_bytes=98, idle_timeout=1800, idle_age=6,
priority=35,ip,nw_dst=192.168.0.3 actions=dec_ttl, // ttl減少一個，爲0的時候丟棄
mod_dl_src:46:0e:bf:4a:7c:11, // tapq1的veth的mac
mod_dl_dst:ca:4b:f7:92:f2:ef, // vtapq1的veth的mac
output:1028 // tapq1
cookie=0x4, duration=1.624s, table=0, n_packets=1,
n_bytes=98, idle_timeout=1800,
idle_age=6,
priority=35,ip,nw_dst=192.168.1.2
actions=dec_ttl,
mod_dl_src:fa:c0:29:b8:c6:77,
mod_dl_dst:86:3c:7c:4b:f1:7a,output:1029
這是爲什麼呢，由於我們是1.2 ping 0.3，先看1.2的mac地址
1040: vtapq@if1041: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500
qdisc noqueue state UP group default qlen 1000
link/ether 86:3c:7c:4b:f1:7a brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 192.168.1.2/24 scope global vtapq
valid_lft forever preferred_lft forever
1.2的mac地址是86:3c:7c:4b:f1:7a
0.3的mac地址
1038: vtapq1@if1039: <BROADCAST,MULTICAST,UP,LOWER_UP>
mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether ca:4b:f7:92:f2:ef brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 192.168.0.3/24 scope global vtapq1
valid_lft forever preferred_lft forever
看一看他們的veth設備的mac地址
1039: tapq1@if1038: <BROADCAST,MULTICAST,UP,LOWER_UP>
mtu 1500 qdisc noqueue master ovs-system state UP group default qlen 1000
link/ether 46:0e:bf:4a:7c:11 brd ff:ff:ff:ff:ff:ff link-netnsid 2
inet6 fe80::440e:bfff:fe4a:7c11/64 scope link
valid_lft forever preferred_lft forever
1041: tapq@if1040: <BROADCAST,MULTICAST,UP,LOWER_UP>
mtu 1500 qdisc noqueue master ovs-system state UP group default qlen 1000
link/ether fa:c0:29:b8:c6:77 brd ff:ff:ff:ff:ff:ff link-netnsid 3
inet6 fe80::f8c0:29ff:feb8:c677/64 scope link
valid_lft forever preferred_lft forever
看看端口
# ovs-ofctl show ckltest-br
OFPT_FEATURES_REPLY (xid=0x2): dpid:000022a0e427f444
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: output enqueue set_vlan_vid set_vlan_pcp
strip_vlan mod_dl_src mod_dl_dst mod_nw_src mod_nw_dst
mod_nw_tos mod_tp_src mod_tp_dst
1028(tapq1): addr:46:0e:bf:4a:7c:11
config: 0
state: 0
current: 10GB-FD COPPER
speed: 10000 Mbps now, 0 Mbps max
1029(tapq): addr:fa:c0:29:b8:c6:77
config: 0
state: 0
current: 10GB-FD COPPER
speed: 10000 Mbps now, 0 Mbps max
LOCAL(ckltest-br): addr:22:a0:e4:27:f4:44
config: 0
state: 0
speed: 0 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0
6.192.168.1.2對應的tapq上實際捕獲的流量分析
ICMP ECHO REQUEST
ICMP ECHO REPLY
7.192.168.0.3對應的tapq1上面實際捕獲的流量的分析
ICMP ECHO REQUEST
ICMP ECHO REPLY
8.TTL分析
192.168.1.2 ping 192.168.0.3 第一次到達tapq的時候，如下
然後經過OVS之後，TTL-1 ,MAC地址被修改並轉發給tapq1，如下
這樣就實現了轉發
8.ping默認網關
192.168.0.3 ping 192.168.0.1
代碼如下
if IPV4 in header_list:
rt_ports = self.address_data.get_default_gw() # 這個是網關的ip
if header_list[IPV4].dst in rt_ports:
# Packet to router's port.
if ICMP in header_list:
if header_list[ICMP].type == icmp.ICMP_ECHO_REQUEST:
self._packetin_icmp_req(msg, header_list)
return
elif TCP in header_list or UDP in header_list:
self._packetin_tcp_udp(msg, header_list)
return
else:
# Packet to internal host or gateway router.
self._packetin_to_node(msg, header_list)
return
self._packetin_icmp_req:發送ICMP_ECHO_REPLY
def _packetin_icmp_req(self, msg, header_list):
# Send ICMP echo reply.
in_port = self.ofctl.get_packetin_inport(msg)
self.ofctl.send_icmp(in_port, header_list, self.vlan_id,
icmp.ICMP_ECHO_REPLY,
icmp.ICMP_ECHO_REPLY_CODE,
icmp_data=header_list[ICMP].data)
srcip = ip_addr_ntoa(header_list[IPV4].src)
dstip = ip_addr_ntoa(header_list[IPV4].dst)
log_msg = 'Receive ICMP echo request from [%s] to router port [%s].'
self.logger.info(log_msg, srcip, dstip, extra=self.sw_id)
self.logger.info('Send ICMP echo reply to [%s].', srcip,
extra=self.sw_id)
self.packetin_tcp_udp:
響應ICMP_PORT_UNREACH_ERROR
def _packetin_tcp_udp(self, msg, header_list):
# Send ICMP port unreach error.
in_port = self.ofctl.get_packetin_inport(msg)
self.ofctl.send_icmp(in_port, header_list, self.vlan_id,
icmp.ICMP_DEST_UNREACH,
icmp.ICMP_PORT_UNREACH_CODE,
msg_data=msg.data)
srcip = ip_addr_ntoa(header_list[IPV4].src)
dstip = ip_addr_ntoa(header_list[IPV4].dst)
self.logger.info('Receive TCP/UDP from [%s] to router port [%s].',
srcip, dstip, extra=self.sw_id)
self.logger.info('Send ICMP destination unreachable to [%s].', srcip,
extra=self.sw_id)
9.日誌分析
第一個icmp包
[RT][INFO] switch_id=000022a0e427f444: Receive IP packet
from [192.168.1.2] to an internal host [192.168.0.3].
[RT][INFO] switch_id=000022a0e427f444: Send ARP request (flood)
收到icmp包，向所有端口洪泛arp
第一個arp包
[RT][INFO] switch_id=000022a0e427f444: Set implicit routing flow [cookie=0x1]
[RT][INFO] switch_id=000022a0e427f444: Receive ARP reply from [192.168.0.3]
to router port [192.168.0.1].
[RT][INFO] switch_id=000022a0e427f444: Send suspend packet to [192.168.0.3].
收到arp包，直接設置點對點的轉發規則，然後將上一個icmp包發送到192.168.0.3
返程的報文分析同理
ping一個網關
[RT][INFO] switch_id=000022a0e427f444: Receive ICMP echo request from [192.168.0.3]
to router port [192.168.0.1].
[RT][INFO] switch_id=000022a0e427f444: Send ICMP echo reply to [192.168.0.3].
10.從流量中觀察流表下發過程
8,9,10號包？
轉發所有的arp報文
其中有個send_arp函數
def send_arp_request(self, src_ip, dst_ip, in_port=None):
# Send ARP request from all ports.
for send_port in self.port_data.values():
if in_port is None or in_port != send_port.port_no:
src_mac = send_port.mac
dst_mac = mac_lib.BROADCAST_STR
arp_target_mac = mac_lib.DONTCARE_STR
inport = self.ofctl.dp.ofproto.OFPP_CONTROLLER
output = send_port.port_no
self.ofctl.send_arp(arp.ARP_REQUEST, self.vlan_id,
src_mac, dst_mac, src_ip, dst_ip,
arp_target_mac, inport, output)
由於連接着OVS的都是veth設備，所以相當於每個veth設備都向自己的對端發送了arp報文，然後如果對端是對應的ip地址的話，對端會返回arp報文。