Utility of the month: ofproto/trace
Anyone tried to debug neutron openvswitch rules knows, it’s almost impossible.
If you have access to both sides of a link you may get away by sending packets and using tcpdump on different interfaces to see what’s going in and out/ But even then, it’s tedious and error-prone process. And if you are not so lucky and do not have access to one (or both) ends of the link, you are at a mercy of random traffic. If there is no traffic and you need to figure out your customer’s complain: ‘why can’t I ssh into my server via my private network?’ — you are screwed up. Well, you may try to ping server from DHCP or L3 namespace… What if there is no DHCP? Or what if you are not allowed to inject random traffic into tenants’ private networks?
Well, rejoice. There is a solution: ovs-appctl ofproto/trace.
It allows to track packet flow inside openvswitch without sending actual packets. You may compose any packet headers you want (even impossible to send or malicious) and simulate how it passes across all rules in OVS. There is no real packet — only report what would have happen if such packet appears.
Example:
$ ovs-appctl ofproto/trace br-int in_port=122,arp,arp_spa=192.163.3.54,dl_src=fa:16:3e:04:e8:43eOutput (very long, skip it):
Bridge: br-int
Flow: arp,metadata=0,in_port=122,vlan_tci=0x0000,dl_src=fa:16:3e:04:e8:3e,dl_dst=00:00:00:00:00:00,arp_spa=192.163.3.54,arp_tpa=0.0.0.0,arp_sha=00:00:00:00:00:00,arp_tha=00:00:00:00:00:00
Rule: table=0 cookie=0x81b28fc521fc6432 priority=0
OpenFlow actions=NORMAL
no learned MAC for destination, floodingResubmitted flow: arp,metadata=0,in_port=9,dl_vlan=4,dl_vlan_pcp=0,dl_src=fa:16:3e:04:e8:3e,dl_dst=00:00:00:00:00:00,arp_spa=192.163.3.54,arp_tpa=0.0.0.0,arp_sha=00:00:00:00:00:00,arp_tha=00:00:00:00:00:00
Resubmitted regs: reg0=0x0 reg1=0x0 reg2=0x0 reg3=0x0 reg4=0x0 reg5=0x0 reg6=0x0 reg7=0x0
Resubmitted odp: push_vlan(vid=4,pcp=0),9,pop_vlan,12,15
Resubmitted megaflow: recirc_id=0,skb_priority=0,arp,in_port=122,vlan_tci=0x0000,dl_src=fa:16:3e:04:e8:3e,dl_dst=00:00:00:00:00:00
Rule: table=0 cookie=0xaf800fade633357c priority=4,in_port=9,dl_vlan=4
OpenFlow actions=mod_vlan_vid:1005,NORMAL
no learned MAC for destination, floodingResubmitted flow: unchanged
Resubmitted regs: reg0=0x0 reg1=0x0 reg2=0x0 reg3=0x0 reg4=0x0 reg5=0x0 reg6=0x0 reg7=0x0
Resubmitted odp: push_vlan(vid=4,pcp=0),9,pop_vlan,12,15,push_vlan(vid=1005,pcp=0),6,7
Resubmitted megaflow: recirc_id=0,skb_priority=0,arp,in_port=122,vlan_tci=0x0000,dl_src=fa:16:3e:04:e8:3e,dl_dst=00:00:00:00:00:00
Rule: table=0 cookie=0x9a6a31cd21388929 priority=2,in_port=9
OpenFlow actions=dropFinal flow: unchanged
Megaflow: recirc_id=0,skb_priority=0,arp,in_port=122,vlan_tci=0x0000,dl_src=fa:16:3e:04:e8:3e,dl_dst=00:00:00:00:00:00
Datapath actions: push_vlan(vid=4,pcp=0),9,pop_vlan,12,15,push_vlan(vid=1005,pcp=0),6,7
Look at the last line! Datapath actions: push_vlan(vid=4,pcp=0),9,pop_vlan,12,15,push_vlan(vid=1005,pcp=0),6,7 — this is what going to happen with our packet.
Another example:
ovs-appctl ofproto/trace br-int in_port=44
Bridge: br-int
Flow: metadata=0,in_port=44,vlan_tci=0x0000,dl_src=00:00:00:00:00:00,dl_dst=00:00:00:00:00:00,dl_type=0x0000
Rule: table=0 cookie=0x81b28fc521fc6432 priority=0
OpenFlow actions=NORMAL
no input bundle, droppingFinal flow: unchanged
Megaflow: recirc_id=0,skb_priority=0,in_port=44,vlan_tci=0x0000/0x1fff,dl_src=00:00:00:00:00:00,dl_dst=00:00:00:00:00:00,dl_type=0x0000
Datapath actions: drop
As you can see final action is ‘drop’. And you can find why it <s>is dropped</s> would be dropped:
Rule: table=0 cookie=0x81b28fc521fc6432 priority=0
OpenFlow actions=NORMAL
no input bundle, droppingBasically, we have no rule for port 44, so it will be send to /dev/null.
A bit more complicated example for modern port security in the Neutron (Mitaka). Again, extremely long:
ovs-appctl ofproto/trace br-int arp,in_port=749,dl_src=fa:16:3e:f2:31:e8,arp_spa=192.168.0.4Bridge: br-int
Flow: arp,in_port=749,vlan_tci=0x0000,dl_src=fa:16:3e:f2:31:e8,dl_dst=00:00:00:00:00:00,arp_spa=192.168.0.4,arp_tpa=0.0.0.0,arp_op=0,arp_sha=00:00:00:00:00:00,arp_tha=00:00:00:00:00:00Rule: table=0 cookie=0x8db2ecbaa59987a5 priority=10,arp,in_port=749
OpenFlow actions=resubmit(,24)Resubmitted flow: arp,in_port=749,vlan_tci=0x0000,dl_src=fa:16:3e:f2:31:e8,dl_dst=00:00:00:00:00:00,arp_spa=192.168.0.4,arp_tpa=0.0.0.0,arp_op=0,arp_sha=00:00:00:00:00:00,arp_tha=00:00:00:00:00:00
Resubmitted regs: reg0=0x0 reg1=0x0 reg2=0x0 reg3=0x0 reg4=0x0 reg5=0x0 reg6=0x0 reg7=0x0
Resubmitted odp: drop
Resubmitted megaflow: recirc_id=0,arp,in_port=749,arp_spa=192.168.0.4
Rule: table=24 cookie=0x8db2ecbaa59987a5 priority=2,arp,in_port=749,arp_spa=192.168.0.4
OpenFlow actions=resubmit(,25)Resubmitted flow: unchanged
Resubmitted regs: reg0=0x0 reg1=0x0 reg2=0x0 reg3=0x0 reg4=0x0 reg5=0x0 reg6=0x0 reg7=0x0
Resubmitted odp: drop
Resubmitted megaflow: recirc_id=0,arp,in_port=749,dl_src=fa:16:3e:f2:31:e8,arp_spa=192.168.0.4
Rule: table=25 cookie=0x8db2ecbaa59987a5 priority=2,in_port=749,dl_src=fa:16:3e:f2:31:e8
OpenFlow actions=NORMAL
no learned MAC for destination, floodingResubmitted flow: arp,in_port=17,dl_vlan=15,dl_vlan_pcp=0,dl_src=fa:16:3e:f2:31:e8,dl_dst=00:00:00:00:00:00,arp_spa=192.168.0.4,arp_tpa=0.0.0.0,arp_op=0,arp_sha=00:00:00:00:00:00,arp_tha=00:00:00:00:00:00
Resubmitted regs: reg0=0x0 reg1=0x0 reg2=0x0 reg3=0x0 reg4=0x0 reg5=0x0 reg6=0x0 reg7=0x0
Resubmitted odp: push_vlan(vid=15,pcp=0),9,pop_vlan,250,272,274,256,290,266,282,263,278,306,308,310
Resubmitted megaflow: recirc_id=0,arp,in_port=749,vlan_tci=0x0000,dl_src=fa:16:3e:f2:31:e8,dl_dst=00:00:00:00:00:00,arp_spa=192.168.0.4
Rule: table=0 cookie=0xb2e6e32fdb00ecba priority=4,in_port=17,dl_vlan=15
OpenFlow actions=mod_vlan_vid:247,NORMAL
no learned MAC for destination, floodingResubmitted flow: arp,in_port=16,dl_vlan=15,dl_vlan_pcp=0,dl_src=fa:16:3e:f2:31:e8,dl_dst=00:00:00:00:00:00,arp_spa=192.168.0.4,arp_tpa=0.0.0.0,arp_op=0,arp_sha=00:00:00:00:00:00,arp_tha=00:00:00:00:00:00
Resubmitted regs: reg0=0x0 reg1=0x0 reg2=0x0 reg3=0x0 reg4=0x0 reg5=0x0 reg6=0x0 reg7=0x0
Resubmitted odp: push_vlan(vid=15,pcp=0),9,pop_vlan,250,272,274,256,290,266,282,263,278,306,308,310,push_vlan(vid=247,pcp=0),1,2
Resubmitted megaflow: recirc_id=0,arp,in_port=749,vlan_tci=0x0000,dl_src=fa:16:3e:f2:31:e8,dl_dst=00:00:00:00:00:00,arp_spa=192.168.0.4
Rule: table=0 cookie=0x9c992dff7ea95ca8 priority=2,in_port=16
OpenFlow actions=dropFinal flow: unchanged
Megaflow: recirc_id=0,arp,in_port=749,vlan_tci=0x0000,dl_src=fa:16:3e:f2:31:e8,dl_dst=00:00:00:00:00:00,arp_spa=192.168.0.4
Datapath actions: push_vlan(vid=15,pcp=0),9,pop_vlan,250,272,274,256,290,266,282,263,278,306,308,310,push_vlan(vid=247,pcp=0),1,2
Yes, it is huge. But it’s real, and it works. Just image how long it would take to debug THIS in real ovs-ofctl dump-flows output with few thousands rules.
Let’s try to ‘spoof’ mac address. I’ll spare you all output and will cite only relevant lines:
ovs-appctl ofproto/trace br-int arp,in_port=749,dl_src=fa:16:3e:f2:31:e8,arp_spa=192.168.0.5(IP was changed from 192.168.0.4 to 192.168.0.5 compare to previous example)
...Rule: table=0 priority=10,arp,in_port=749
OpenFlow actions=resubmit(,24)
... Rule: table=24 priority=0
OpenFlow actions=drop
...Datapath actions: drop
Anti-spoofing at work!
Where to get in_port?
It’s a bit more work.
nova interface-list a390f667–2d4c-4192–8d71-c0408d110cc6 #instance UUID
8d764afd-7643-4967-9e16-b934a900f140 | 813cd43a-276e-461a-99b6-d0e37356d3d8 | 88.212.238.136 | fa:16:3e:7f:0e:56 |4ffaadfa-3645-4333-5e16-6634900f141 | 134ca3d6-b6f8-4bae-aadc-4c13219b7b2d | 192.168.12.1 | fa:16:3e:b5:4d:ea |
Go to a host where the instance is. (We can see host where instance is in the nova show UUID command).
Use virsh domiflist command to see local tap names for interfaces. You need to use instance_name from OS-EXT-ATTR:instance_name
Example:
virsh domiflist instance-00001
Interface Type Source Model MAC
— — — — — — — — — — — — — — — — — — — — — — — — — — — -
tap20e4753a-84 bridge br-int virtio fa:16:3e:7f:0e:56
tap82dfe025–32 bridge br-int virtio fa:16:3e:b5:4d:eaIf you got more than one interface, choose needed via MAC address (you can see it in the nova interface-list output).
As ovs-vsctl about that interface:
ovs-vsctl list interface tap82dfe025–32
ovs-vsctl list interface tap82dfe025-32
_uuid : 6070c7f0-3fed-44ec-b64a-05c6bde1c87e
...
external_ids : {attached-mac="fa:16:3e:b5:4d:ea", iface-id="82dfe025-3268-43bb-8356-9d53e9256312", iface-status=active, vm-id="7a254818-ea6f-4807-9a74-5237b32df476"}
...
mac_in_use : "fe:16:3e:b5:4d:ea"
name : "tap82dfe025-32"
ofport : 1004
...ofport — is the port number for in_port field for out requests.
If you want debug VLAN part of OVS you may ask for tag settings for interface:
ovs-vs_uuid : 7a2e164f-baed-4c92-9c95-33fc9099ec0a
...
interfaces : [6070c7f0-3fed-44ec-b64a-05c6bde1c87e]
...
name : "tap82dfe025-32"
other_config : {net_uuid="134ca3d6-b6f8-4bae-aadc-4c13219b7b2d", network_type=vlan, physical_network=internet, segmentation_id="30", tag="57"}
tag : 206
...
vlan_mode : []ctl list port tap82dfe025-32You can see tag field — this is a ‘host-local-vlan’. It is not the ‘VLAN’ you get outside of the host.
If you are using mitaka+ you can see other_config field with ‘segmentation_id’ — this is an external VLAN number (or GRE, or VXLAN).
You may use this commands with physical interfaces plugged to bridges with same success. Just ask OVS for ofport number for given physical inteface.
Example:
ovs-vsctl list interface eth4
_uuid : 595edb81-30bb-49d5-ab6f-7c1f68e1c7c2
admin_state : up
...
mac_in_use : "a0:36:9f:83:ef:f0"
mtu : 8000
name : "eth4"
ofport : 1
...
status : {driver_name=ixgbe, driver_version="3.15.1-k", firmware_version="0x8000059e"}
And you may track output of traffic via that port:
ovs-appctl ofproto/trace br_internet ip,in_port=1,dl_vlan=22
Datapath actions: sample(sample=0.1%,actions(userspace(pid=4294962961,sFlow(vid=22,pcp=0,output=2147483650),actions))),3,5You can see all ports in the system via ovs-dpctl show command. It dumps all ports from OVS database and you can see their linkage into real hardware (or software).
Other fields
If you need to construct complicated query, all possible values and syntax is described in man ovs-ofctl. One thing many find non-trivial, that if you want do deviate from tcp/udp, you need to clarify underlaying protocols (usually, ip).
