toggling explicitly_egress_direct from true to false does not clean the openflow flows on the integration bridge

Triaged looking at code

I did some tests so far.

1, confirm delete_accepted_egress_direct_flow [1] is ok

see https://pastebin.ubuntu.com/p/J2HTFPr4sY/

2, pdb shows delete_accepted_egress_direct_flow was triggered

snat-xxx port is a little special, so try to use the following command to trigger delete_accepted_egress_direct_flow

openstack router set --disable provider-router && sleep 30 && openstack router set --enable provider-router

see https://pastebin.ubuntu.com/p/48ZXFdhgST/

3, log shows delete_accepted_egress_direct_flow was NOT triggered

see https://pastebin.ubuntu.com/p/JNnsd6TgYs/

[1] https://review.opendev.org/c/openstack/neutron/+/704506/1/neutron/agent/linux/openvswitch_firewall/firewall.py#1140

Successfully find a better workaround to take advantage of delete_accepted_egress_direct_flow in _unbind_distributed_router_interface_port [1].

# eg: mac of the old snat-xxx port is fa:16:3e:7a:11:7d
neutron router-interface-delete provider-router $(openstack subnet show private_subnet -cid -fvalue)
# eg: mac of the new snat-xxx port is fa:16:3e:e6:f9:b2
neutron router-interface-add provider-router $(openstack subnet show private_subnet -cid -fvalue)
openstack port list --device-owner network:router_centralized_snat

The code path is:

process_deleted_ports -> port_unbound -> unbind_port_from_dvr -> _unbind_centralized_snat_port_on_dvr_subnet -> delete_accepted_egress_direct_flow

The egress direct flow for the old snat-xxx port won't disappear

# ovs-ofctl dump-flows br-int |grep fa:16:3e:7a:11:7d |grep -E 'priority=12|priority=10'
 cookie=0x59874eed7c9fa42a, duration=76882.302s, table=94, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=12,reg6=0x1,dl_dst=fa:16:3e:7a:11:7d actions=output:16
 cookie=0x59874eed7c9fa42a, duration=76882.302s, table=94, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=10,reg6=0x1,dl_src=fa:16:3e:7a:11:7d,dl_dst=00:00:00:00:00:00/01:00:00:00:00:00 actions=mod_vlan_vid:1,output:2

but the egress direct flow won't produce as well

# ovs-ofctl dump-flows br-int |grep 'fa:16:3e:e6:f9:b2' |grep -E 'priority=12|priority=10'

So north-south traffic will resume to work again.

# ip netns exec snat-10140acd-28e6-4110-ae67-76115b72b37c ping -c1 192.168.21.114
PING 192.168.21.114 (192.168.21.114) 56(84) bytes of data.
64 bytes from 192.168.21.114: icmp_seq=1 ttl=64 time=1.86 ms

[1] https://review.opendev.org/c/openstack/neutron/+/704506/1/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_dvr_neutron_agent.py#678