br-int bridge in one compute can't learn MAC addresses of VMs in other compute nodes
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
neutron |
New
|
Medium
|
Unassigned |
Bug Description
In Openstack Queens release, we noticed a very serious issue, br-int bridge in one compute node can't learn MAC addresses of VMs in other compute nodes, so after launched many VMs, VM-to-VM network performance will decrease linearly, especially ovs will broadcast packets because it doesn't learn target VM MAC address, other VMs in same subnet in same compute node can receive these broadcast packets, therefore, the corresponding vhost kernel threads are receiving these packets and wasting CPU resources. More VMs, more serious the issue, worse the performance, no matter UDP or TCP performance.
We have checked several Queens deployments, they have same issues, but Openstack Rocky doesn't have this issue. Here is the flow I dumped:
recirc_
MAC address of target VM wasn't learnt by br-int
$ sudo ovs-appctl fdb/show br-int | grep "fa:16:3e:a7:0a:3a"
By the way, we used linuxbridge for security group.
Yi Yang (yangyi01) wrote : | #1 |
Bence Romsics (bence-romsics) wrote : | #2 |
Hi Yi,
Could this be a duplicate of https:/
If no, could you please give us details on how to reproduce the issue?
Changed in neutron: | |
status: | New → Incomplete |
Yi Yang (yangyi01) wrote : | #3 |
Bence, thank you so much for information,but we used linux bridge for security group, i.e firewall_driver = iptables_hybrid. I will tried the fix for bug #1732067 and check if it can fix my issue.
By the way, Rocky also has same issue, we tried to narrow issue domain, if we don't add subnet of VMs into a router, MAC can be learnt, but MAC can't be learnt if we add subnet of VMs to a router. We used DVR and distributed DHCP, every VM's tap interface is attached into a linux bridge which is attached into br-int by veth pair.
Yi Yang (yangyi01) wrote : | #4 |
bug #1732067 is same issue as this one.
Yi Yang (yangyi01) wrote : | #5 |
But fix for bug #1732067 is just a workaround, MAC learning issue isn't fixed, it will be better if somebody can fix MAC learning issue and make sure br-int can always VMs' MACs.
LIU Yulong (dragon889) wrote : | #6 |
bug #1732067 is about the egress flooding in the intergration bridge (br-int), and its fix is aiming to that. So have you enable the L2pop for your deployment? Paste your config file may be usefull for the upstream team.
LIU Yulong (dragon889) wrote : | #7 |
Remove the duplicated mark since flooding and mac learning are not same. And you could use "ovs-appctl ofproto/trace" to find out the flows hit by the packets. Typically, the packets hit the "NORMAL" flow will let the bridge learn the mac to the fdb.
Yi Yang (yangyi01) wrote : | #8 |
@LIU Yulong, yes, we enabled l2pop and arpresponder, and use DVR, firewall_dirver is iptables_hybrid, to be clarified, your fix patch can't fix our issue because your fix patch is openvswitch firewall only.
From ovs-ofctl -Oopenflow13 dump-flows br-int, we can see VM-to-VM traffic hits NORMAL action, ovs-appctl dpif/dump-flows can see the traffic is output to many ofports, what is very weird is br-int can learn MAC once the subnet VMs host is removed from router.
The below is neutron openvswitch agent config in our compute node.
cat /etc/neutron/
[DEFAULT]
[agent]
tunnel_types =vxlan
vxlan_udp_port = 4789
l2_population = True
arp_responder = True
enable_
drop_flows_on_start = False
[network_log]
[ovs]
integration_bridge = br-int
tunnel_bridge = br-tun
local_ip = xxx.xxx.xxx.xxx
bridge_mappings = physnet1:
[securitygroup]
firewall_driver = iptables_hybrid
enable_
Yi Yang (yangyi01) wrote : | #9 |
@LIU Yulong, BTW, after we switch to openvswitch firewall, your patch can work, fix our issue, but br-int can't learn local MACs and remote MACs, do you have such issue in your deployment? but when we use iptables_hybrid firewall driver, local MACs can be learnt, remote MACs can't be learn. My point is if br-int can learn remote MACs, it won't have flood issue, we also do have flood issue, that is root cause which results in super low pps performance.
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (master) | #10 |
Related fix proposed to branch: master
Review: https:/
Yi Yang (yangyi01) wrote : | #11 |
@LIU Yulong, I added a fix commit in https:/
Changed in neutron: | |
status: | Incomplete → In Progress |
importance: | Undecided → Medium |
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master) | #12 |
Fix proposed to branch: master
Review: https:/
Changed in neutron: | |
assignee: | nobody → Darragh O'Reilly (darragh-oreilly) |
Changed in neutron: | |
assignee: | Darragh O'Reilly (darragh-oreilly) → jacy lee (jacylees) |
Changed in neutron: | |
assignee: | jacy lee (jacylees) → nobody |
Changed in neutron: | |
assignee: | nobody → jacy lee (jacylees) |
Changed in neutron: | |
assignee: | jacy lee (jacylees) → Li YaJie (yjmango) |
OpenStack Infra (hudson-openstack) wrote : | #13 |
Fix proposed to branch: master
Review: https:/
Jacy Lee (yjmango) wrote : | #14 |
In DVR mode, this bug not only caused Layer 2 flooding, but also caused Layer 3 flooding.
* For example L2 flooding, vm1(sub1, host1), vm2(sub1, host1), vm3(sub1, host2).
Ping from vm1 to vm3, the vm2 can also grab packets.
* For example L3 flooding, vm1(sub1, host1), vm4(sub2, host1), vm5(sub2, host2).
Ping from vm1 to vm5, the vm4 can also grab packets.
I add a fix commit in [1], it add a flow table 59 which process the packets with src_mac is gateway_mac.This table contains three types of flows:
* table=59, n_packets=0, n_bytes=0, priority=
* table=59, n_packets=8, n_bytes=672, priority=
* table=59, n_packets=3, n_bytes=294, priority=
Yi Yang (yangyi01) wrote : | #15 |
I'm confused by several proposed patches. Per my understanding, they are duplicate.
Yi Yang (yangyi01) wrote : | #16 |
@Li Yajie, please check if L3 flood you're saying is still there after applied Darragh O'Reilly (darragh-oreilly) https:/
Per my check, Liu Yulong's fix patch plus my fix patch for iptables_hybrid won't have L2 flood and L3 flood you're saying. So now we have at least 3 proposed ways to fix it, which one is better? Welcome comments.
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (master) | #17 |
Change abandoned by Darragh O'Reilly (<email address hidden>) on branch: master
Review: https:/
Reason: https:/
LIU Yulong (dragon889) wrote : | #18 |
Hi, sorry for the late reply.
The fix of bug #1732067 has a config option ``explicitly_
Please have a try to add the following config option to your compute nodes to see if the flooding still happen:
[agent]
...
explicitly_
The fix has a flow which direct the unicast from br-int to tunnel/physical bridge, something like this:
able=61, n_packets=0, n_bytes=0, priority=
table=61, n_packets=0, n_bytes=0, priority=
table=61, n_packets=0, n_bytes=0, priority=
table=61, n_packets=0, n_bytes=0, priority=
table=61, n_packets=0, n_bytes=0, priority=
table=61, n_packets=0, n_bytes=0, priority=
so these test cases should not happen:
* For example L2 flooding, vm1(sub1, host1), vm2(sub1, host1), vm3(sub1, host2).
Ping from vm1 to vm3, the vm2 can also grab packets.
* For example L3 flooding, vm1(sub1, host1), vm4(sub2, host1), vm5(sub2, host2).
Ping from vm1 to vm5, the vm4 can also grab packets.
Yi Yang (yangyi01) wrote : | #19 |
@LIU Yulong, your fix can't work for firewall_
Can you help review Yajie's solution? What do you think about it?
Arjun Baindur (abaindur) wrote : | #20 |
Is there any idea what caused this breakage, and for how long? We are NOT seeing it on Pike, but we are seeing it in Rocky.
We thought it was this commit, https:/
Kaustubh Phatak (kphatak-pf9) wrote : | #21 |
This is affecting us in large setups.
Arjun Baindur (abaindur) wrote : | #22 |
Correction - we see this in Pike as well.
Darragh O'Reilly (darragh-oreilly) wrote : | #23 |
@abaindur - yes I believe https:/
Yi Yang (yangyi01) wrote : | #24 |
Hi, folks, I think https:/
@Darragh O'Reilly (darragh-oreilly), for the case you mentioned (a subnet which isn't added a router), it also doesn't have flood, https:/
So so far, I propose you folks use https:/
Arjun Baindur (abaindur) wrote : | #25 |
Yi, what about https:/
We're on Rocky, and do not already have Yulong's fix (666991) as you pointed out. So in that case we'd have to backport in two patches. Do you recommend abandoning 714519 and instead backporting the two fixes you mentioned?
Also, you said on review: "Another is found by Darragh O'Reilly, if subnet has no router
connected, it will case the a new "ingress" flooding. This will be
addressed in another bug or patch."
I was under the impression the original issue is only hit in DVR cases, and when subnet is attached to a router. What about for provider networks? Does this patch introduce a new flooding issue for that? Because provider networks can never be attached to a DVR router.
Arjun Baindur (abaindur) wrote : | #26 |
Is this truly fixed? The patch doesn't seem to fix the underlying issue at hand - ingress MACs (coming from br-vlan or br-tun) are NOT learned by br-int.
Also what are the implications of turning on this setting? The commit states:
"A new config option ``explicitly_
was added for the aim of distinguishing clouds which are running the
network node mixed with compute services, upstream neutron CI should be
an example. In such situation, this ``explicitly_
set to False, because there are numerous cases from HA routers which can
not be covered, particularly when you have centralized floating IPs running
in such mixed hosts.
Otherwise, set ``explicitly_
One more note is if your network nodes are for networing services only, we
recommand you disable all the security_group to get a higher performance."
I take that to say on a network node mixed with compute services it should be set to False. But isn't that the case for every node in DVR? The compute node hosts qrouter, fip router, and can also host dhcp services.
Or does it mean we not supposed to enable this setting on hosts with dvr_snat mode for l3-agent? The message states something about centralized fips - which are hosting on the snat router/node.
Arjun Baindur (abaindur) wrote : | #27 |
This fix seems to break certain other scenarios by causing flooding in opposite direction: https:/
Arjun Baindur (abaindur) wrote : | #28 |
We are seeing ingress flooding occur due to enabling this fix on other networks. We have a VLAN based network, both attached and unattached to a DVR router. In both cases, on ingress the NORMAL action at table=60 is hit.
This is because for a provider network, there are no dvr MAC rules with an explicit output: action to a local port. In DVR subnet, these rules are added with the EXTERNAL vlan/segmentation ID. Not with the local vlan. However, there is a table=0, priority=3 rule which converts from the external VLAN to host local VLAN.
NORMAL action at table=60 for ingress, requires the local VM MACs to be learned. But these are never learned, because the fix resubmits egress packets to table 61, which has a direct output: action northbound.
So now we have flooding in opposite direction. But we still need this fix because in other scenarios, we see the egress flooding.
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (master) | #29 |
Related fix proposed to branch: master
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (master) | #30 |
Change abandoned by Slawek Kaplonski (<email address hidden>) on branch: master
Review: https:/
Reason: This review is > 4 weeks without comment, and failed Zuul jobs the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.
OpenStack Infra (hudson-openstack) wrote : | #31 |
Change abandoned by Slawek Kaplonski (<email address hidden>) on branch: master
Review: https:/
Reason: This review is > 4 weeks without comment, and failed Zuul jobs the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.
Slawek Kaplonski (slaweq) wrote : auto-abandon-script | #32 |
This bug has had a related patch abandoned and has been automatically un-assigned due to inactivity. Please re-assign yourself if you are continuing work or adjust the state as appropriate if it is no longer valid.
Changed in neutron: | |
assignee: | Li YaJie (yjmango) → nobody |
status: | In Progress → New |
tags: | added: timeout-abandon |
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (master) | #33 |
Change abandoned by Slawek Kaplonski (<email address hidden>) on branch: master
Review: https:/
Reason: This review is > 4 weeks without comment, and failed Zuul jobs the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (master) | #34 |
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: master
commit 959d8b6d73e2a6a
Author: LIU Yulong <email address hidden>
Date: Fri Jul 10 17:25:15 2020 +0800
Local mac direct flow for non-openflow firewall
When there is no openflow firewall, aka the ovs agent security group
is disabled or Noop/HybridIptable, this patch will introduce a different
ingress pipeline for bridge ports which will avoid ingress flood:
(1) table=0, in_port=
(2) table=60, in_port=
(3) table=61, dl_dst=
And changes the local ports pipeline:
(1) table=0, in_port=
(2) table=25, in_port=
(3) table=60, in_port=
(4) table=61, dl_dst=
Closes-Bug: #1884708
Closes-Bug: #1881070
Related-Bug: #1732067
Related-Bug: #1866445
Related-Bug: #1883321
Change-Id: Iecf9cffaf02616
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (master) | #35 |
Change abandoned by Yi Yang (<email address hidden>) on branch: master
Review: https:/
Reason: https:/
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/victoria) | #36 |
Related fix proposed to branch: stable/victoria
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/ussuri) | #37 |
Related fix proposed to branch: stable/ussuri
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/train) | #38 |
Related fix proposed to branch: stable/train
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/stein) | #39 |
Related fix proposed to branch: stable/stein
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/rocky) | #40 |
Related fix proposed to branch: stable/rocky
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/queens) | #41 |
Related fix proposed to branch: stable/queens
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/ussuri) | #42 |
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: stable/ussuri
commit ef14d258eea91ef
Author: LIU Yulong <email address hidden>
Date: Fri Jul 10 17:25:15 2020 +0800
Local mac direct flow for non-openflow firewall
When there is no openflow firewall, aka the ovs agent security group
is disabled or Noop/HybridIptable, this patch will introduce a different
ingress pipeline for bridge ports which will avoid ingress flood:
(1) table=0, in_port=
(2) table=60, in_port=
(3) table=61, dl_dst=
And changes the local ports pipeline:
(1) table=0, in_port=
(2) table=25, in_port=
(3) table=60, in_port=
(4) table=61, dl_dst=
Closes-Bug: #1884708
Closes-Bug: #1881070
Related-Bug: #1732067
Related-Bug: #1866445
Related-Bug: #1883321
Change-Id: Iecf9cffaf02616
(cherry picked from commit 959d8b6d73e2a6a
tags: | added: in-stable-ussuri |
tags: | added: in-stable-train |
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/train) | #43 |
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: stable/train
commit c06895e8e78de06
Author: LIU Yulong <email address hidden>
Date: Fri Jul 10 17:25:15 2020 +0800
Local mac direct flow for non-openflow firewall
When there is no openflow firewall, aka the ovs agent security group
is disabled or Noop/HybridIptable, this patch will introduce a different
ingress pipeline for bridge ports which will avoid ingress flood:
(1) table=0, in_port=
(2) table=60, in_port=
(3) table=61, dl_dst=
And changes the local ports pipeline:
(1) table=0, in_port=
(2) table=25, in_port=
(3) table=60, in_port=
(4) table=61, dl_dst=
Closes-Bug: #1884708
Closes-Bug: #1881070
Related-Bug: #1732067
Related-Bug: #1866445
Related-Bug: #1883321
Change-Id: Iecf9cffaf02616
(cherry picked from commit 959d8b6d73e2a6a
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/stein) | #44 |
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: stable/stein
commit 7c757ad3372b5fe
Author: LIU Yulong <email address hidden>
Date: Fri Jul 10 17:25:15 2020 +0800
Local mac direct flow for non-openflow firewall
When there is no openflow firewall, aka the ovs agent security group
is disabled or Noop/HybridIptable, this patch will introduce a different
ingress pipeline for bridge ports which will avoid ingress flood:
(1) table=0, in_port=
(2) table=60, in_port=
(3) table=61, dl_dst=
And changes the local ports pipeline:
(1) table=0, in_port=
(2) table=25, in_port=
(3) table=60, in_port=
(4) table=61, dl_dst=
Closes-Bug: #1884708
Closes-Bug: #1881070
Related-Bug: #1732067
Related-Bug: #1866445
Related-Bug: #1883321
Change-Id: Iecf9cffaf02616
(cherry picked from commit 959d8b6d73e2a6a
tags: | added: in-stable-stein |
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/victoria) | #45 |
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: stable/victoria
commit 88bbb58c80b9c88
Author: LIU Yulong <email address hidden>
Date: Fri Jul 10 17:25:15 2020 +0800
Local mac direct flow for non-openflow firewall
When there is no openflow firewall, aka the ovs agent security group
is disabled or Noop/HybridIptable, this patch will introduce a different
ingress pipeline for bridge ports which will avoid ingress flood:
(1) table=0, in_port=
(2) table=60, in_port=
(3) table=61, dl_dst=
And changes the local ports pipeline:
(1) table=0, in_port=
(2) table=25, in_port=
(3) table=60, in_port=
(4) table=61, dl_dst=
Closes-Bug: #1884708
Closes-Bug: #1881070
Related-Bug: #1732067
Related-Bug: #1866445
Related-Bug: #1883321
Change-Id: Iecf9cffaf02616
(cherry picked from commit 959d8b6d73e2a6a
tags: | added: in-stable-victoria |
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/rocky) | #46 |
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: stable/rocky
commit 7ce65c94786d2d1
Author: LIU Yulong <email address hidden>
Date: Fri Jul 10 17:25:15 2020 +0800
Local mac direct flow for non-openflow firewall
When there is no openflow firewall, aka the ovs agent security group
is disabled or Noop/HybridIptable, this patch will introduce a different
ingress pipeline for bridge ports which will avoid ingress flood:
(1) table=0, in_port=
(2) table=60, in_port=
(3) table=61, dl_dst=
And changes the local ports pipeline:
(1) table=0, in_port=
(2) table=25, in_port=
(3) table=60, in_port=
(4) table=61, dl_dst=
Closes-Bug: #1884708
Closes-Bug: #1881070
Related-Bug: #1732067
Related-Bug: #1866445
Related-Bug: #1883321
Conflicts:
Change-Id: Iecf9cffaf02616
(cherry picked from commit 959d8b6d73e2a6a
tags: | added: in-stable-rocky |
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/queens) | #47 |
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: stable/queens
commit 7fe3e3d1e04d206
Author: LIU Yulong <email address hidden>
Date: Fri Jul 10 17:25:15 2020 +0800
Local mac direct flow for non-openflow firewall
When there is no openflow firewall, aka the ovs agent security group
is disabled or Noop/HybridIptable, this patch will introduce a different
ingress pipeline for bridge ports which will avoid ingress flood:
(1) table=0, in_port=
(2) table=60, in_port=
(3) table=61, dl_dst=
And changes the local ports pipeline:
(1) table=0, in_port=
(2) table=25, in_port=
(3) table=60, in_port=
(4) table=61, dl_dst=
Closes-Bug: #1884708
Closes-Bug: #1881070
Related-Bug: #1732067
Related-Bug: #1866445
Related-Bug: #1883321
Conflicts:
Change-Id: Iecf9cffaf02616
(cherry picked from commit 959d8b6d73e2a6a
tags: | added: in-stable-queens |
Top output after I launched many VMs in the same subnet.