TL;DR: ip -4 rule del priority <priority> table <table-id> type unicast will delete the first matching rule it encounters: if there are two rules with the same priority it will just kill the first one it finds.
OpenStack Queens from UCA (xenial, GA kernel, deployed via OpenStack charms), 2 external subnets (one routed provider network), 2 tenant subnets all in the same address scope to trigger "fast exit".
2 tenant networks attached (subnets 192.168.100.0/24 and 192.168.200.0/24) to a DVR:
# 2 rules as expected
ip netns exec qrouter-4f9ca9ef-303b-4082-abbc-e50782d9b800 ip rule
0: from all lookup local
32766: from all lookup main
32767: from all lookup default
80000: from 192.168.100.0/24 lookup 16
80000: from 192.168.200.0/24 lookup 16
# remove 192.168.200.0/24 sometimes deletes an incorrect policy rule
openstack router remove subnet pubrouter othertenantsubnet
ip netns exec qrouter-4f9ca9ef-303b-4082-abbc-e50782d9b800 ip rule
0: from all lookup local
32766: from all lookup main
32767: from all lookup default
80000: from 192.168.100.0/24 lookup 16
80000: from 192.168.200.0/24 lookup 16
# try to delete a rule manually to see what is going on
ip netns exec qrouter-4f9ca9ef-303b-4082-abbc-e50782d9b800 ip rule ; ip netns exec qrouter-4f9ca9ef-303b-4082-abbc-e50782d9b800 ip -4 rule del priority 80000 table 16 type unicast ; ip netns exec qrouter-4f9ca9ef-303b-4082-abbc-e50782d9b800 ip rule
0: from all lookup local
32766: from all lookup main
32767: from all lookup default
80000: from 192.168.100.0/24 lookup 16
80000: from 192.168.200.0/24 lookup 16
0: from all lookup local
32766: from all lookup main
32767: from all lookup default
80000: from 192.168.200.0/24 lookup 16
# ^^ 192.168.100.0/24 rule got deleted instead of 192.168.200.0/24
# add the rule back manually
ip netns exec qrouter-4f9ca9ef-303b-4082-abbc-e50782d9b800 ip rule add from 192.168.100.0/24 priority 80000 table 16 type unicast
# different order now - 192.168.200.0/24 is first
ip netns exec qrouter-4f9ca9ef-303b-4082-abbc-e50782d9b800 ip rule
0: from all lookup local
32766: from all lookup main
32767: from all lookup default
80000: from 192.168.200.0/24 lookup 16
80000: from 192.168.100.0/24 lookup 16
# now 192.168.200.0/24 got deleted because it was first to match
ip netns exec qrouter-4f9ca9ef-303b-4082-abbc-e50782d9b800 ip rule ; ip netns exec qrouter-4f9ca9ef-303b-4082-abbc-e50782d9b800 ip -4 rule del priority 80000 table 16 type unicast ; ip netns exec qrouter-4f9ca9ef-303b-4082-abbc-e50782d9b800 ip rule
0: from all lookup local
32766: from all lookup main
32767: from all lookup default
80000: from 192.168.200.0/24 lookup 16
80000: from 192.168.100.0/24 lookup 16
0: from all lookup local
32766: from all lookup main
32767: from all lookup default
80000: from 192.168.100.0/24 lookup 16
TL;DR: ip -4 rule del priority <priority> table <table-id> type unicast will delete the first matching rule it encounters: if there are two rules with the same priority it will just kill the first one it finds.
The original setup is described here: /bugs.launchpad .net/ubuntu/ +source/ neutron/ +bug/1759918
https:/
OpenStack Queens from UCA (xenial, GA kernel, deployed via OpenStack charms), 2 external subnets (one routed provider network), 2 tenant subnets all in the same address scope to trigger "fast exit".
2 tenant networks attached (subnets 192.168.100.0/24 and 192.168.200.0/24) to a DVR:
# 2 rules as expected 4f9ca9ef- 303b-4082- abbc-e50782d9b8 00 ip rule
ip netns exec qrouter-
0: from all lookup local
32766: from all lookup main
32767: from all lookup default
80000: from 192.168.100.0/24 lookup 16
80000: from 192.168.200.0/24 lookup 16
# remove 192.168.200.0/24 sometimes deletes an incorrect policy rule
openstack router remove subnet pubrouter othertenantsubnet
# ip route del contains the cidr agent.linux. utils [-] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/ rootwrap. conf', 'ip', 'ne dc45-4237- 9ce0-a9e1977735 eb', 'ip', '-4', 'route', 'del', '192.168.200.0/24', 'via', '169.254.93.94', 'dev', 'fpr-4f9ca9ef-3' python2. 7/dist- packages/ neutron/ agent/linux/ utils.py: 92
2018-03-29 20:09:52.946 2083594 DEBUG neutron.
tns', 'exec', 'fip-d0f008fc-
] create_process /usr/lib/
# ip rule delete is not that specific agent.linux. utils [-] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/ rootwrap. conf', 'ip', 'netns', 'exec', 'qrouter- 4f9ca9ef- 303b-4082- abbc-e50782d9b8 00', 'ip', '-4', 'rule', 'del', 'priority', '80000', 'table', '16', 'type', 'unicast'] create_pr python2. 7/dist- packages/ neutron/ agent/linux/ utils.py: 92
2018-03-29 20:09:53.195 2083594 DEBUG neutron.
ocess /usr/lib/
2018-03-29 20:15:59.210 2083594 DEBUG neutron. agent.linux. utils [-] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/ rootwrap. conf', 'ip', 'netns', 'exec', 'qrouter- 4f9ca9ef- 303b-4082- abbc-e50782d9b8 00', 'ip', '-4', 'rule', 'show'] create_process /usr/lib/ python2. 7/dist- packages/ neutron/ agent/linux/ utils.py: 92 agent.linux. utils [-] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/ rootwrap. conf', 'ip', 'netns', 'exec', 'qrouter- 4f9ca9ef- 303b-4082- abbc-e50782d9b8 00', 'ip', '-4', 'rule', 'add', 'from', '192.168.100.0/24', 'priority', '80000', 'table', '16', 'type', 'unicast'] create_process /usr/lib/ python2. 7/dist- packages/ neutron/ agent/linux/ utils.py: 92
2018-03-29 20:15:59.455 2083594 DEBUG neutron.
~~~~
ip netns exec qrouter- 4f9ca9ef- 303b-4082- abbc-e50782d9b8 00 ip rule
0: from all lookup local
32766: from all lookup main
32767: from all lookup default
80000: from 192.168.100.0/24 lookup 16
80000: from 192.168.200.0/24 lookup 16
# try to delete a rule manually to see what is going on
ip netns exec qrouter- 4f9ca9ef- 303b-4082- abbc-e50782d9b8 00 ip rule ; ip netns exec qrouter- 4f9ca9ef- 303b-4082- abbc-e50782d9b8 00 ip -4 rule del priority 80000 table 16 type unicast ; ip netns exec qrouter- 4f9ca9ef- 303b-4082- abbc-e50782d9b8 00 ip rule
0: from all lookup local
32766: from all lookup main
32767: from all lookup default
80000: from 192.168.100.0/24 lookup 16
80000: from 192.168.200.0/24 lookup 16
0: from all lookup local
32766: from all lookup main
32767: from all lookup default
80000: from 192.168.200.0/24 lookup 16
# ^^ 192.168.100.0/24 rule got deleted instead of 192.168.200.0/24
# add the rule back manually 4f9ca9ef- 303b-4082- abbc-e50782d9b8 00 ip rule add from 192.168.100.0/24 priority 80000 table 16 type unicast
ip netns exec qrouter-
# different order now - 192.168.200.0/24 is first 4f9ca9ef- 303b-4082- abbc-e50782d9b8 00 ip rule
ip netns exec qrouter-
0: from all lookup local
32766: from all lookup main
32767: from all lookup default
80000: from 192.168.200.0/24 lookup 16
80000: from 192.168.100.0/24 lookup 16
# now 192.168.200.0/24 got deleted because it was first to match
ip netns exec qrouter- 4f9ca9ef- 303b-4082- abbc-e50782d9b8 00 ip rule ; ip netns exec qrouter- 4f9ca9ef- 303b-4082- abbc-e50782d9b8 00 ip -4 rule del priority 80000 table 16 type unicast ; ip netns exec qrouter- 4f9ca9ef- 303b-4082- abbc-e50782d9b8 00 ip rule
0: from all lookup local
32766: from all lookup main
32767: from all lookup default
80000: from 192.168.200.0/24 lookup 16
80000: from 192.168.100.0/24 lookup 16
0: from all lookup local
32766: from all lookup main
32767: from all lookup default
80000: from 192.168.100.0/24 lookup 16
Code:
_dvr_internal_ network_ removed /github. com/openstack/ neutron/ blob/stable/ queens/ neutron/ agent/l3/ dvr_local_ router. py#L431- L443
https:/
_delete_ interface_ routing_ rule_in_ router_ ns /github. com/openstack/ neutron/ blob/stable/ queens/ neutron/ agent/l3/ dvr_local_ router. py#L642- L648 IPRule( namespace= self.ns_ name) port['subnets' ]:
rtr_ port_cidr = subnet['cidr']
ip_ rule.rule. delete( ip=rtr_ port_cidr,
table= dvr_fip_ ns.FIP_ RT_TBL,
priority= dvr_fip_ ns.FAST_ PATH_EXIT_ PR)
https:/
ip_rule = ip_lib.
for subnet in router_
IpRuleCommand /github. com/openstack/ neutron/ blob/master/ neutron/ agent/linux/ ip_lib. py#L486- L494
https:/
# TODO(Carl) ip ignored in delete, okay in general?
He-he, experience shows that definitely not.
We need to use the most specific rule description to avoid ordering issues.
ip -4 rule del from 192.168.200.0/24 priority 80000 table 16 type unicast
With a fix it looks like this:
2018-03-29 20:58:57.023 192084 DEBUG neutron. agent.linux. utils [-] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/ rootwrap. conf', 'ip', 'netns', 'exec', 'qrouter- 4f9ca9ef- 303b-4082- abbc-e50782d9b8 00', 'ip', '-4', 'rule', 'del', 'from', '192.168.200.0/24', 'priority', '80000', 'table', '16', 'type', 'unicast'] create_process /usr/lib/ python2. 7/dist- packages/ neutron/ agent/linux/ utils.py: 92