OVN: HA chassis group priority is different than gateway chassis priority

I don't have a deployment with baremetal servers but this looks like a legit bug.

Hello Michal:

It is quite difficult for me to create an environment with baremetal ports. Can you check what Numan is suggesting in the mail?

"That could be the issue. You can perhaps arp for the router ip from
your bare metal machine and see if you get 2 arp replies - one from
the controller which binds the external port and one from the gateway
chassis controller."

If I'm not wrong, the HA chassis group will assign the highest priority chassis in "HA_Chassis" table and will detect failovers. Other ports (not external ones) should use the same chassis. In your case you are using VLAN that implies we are explicitly sending this traffic to a centralized router port, that is in the chassis hosting the distributed GW port [1]. I would need to check this issue with Lucas.

Regards.

[1]https://github.com/openvswitch/ovs/commit/85706c34d53d4810f54bec1de662392a3c06a996

After talking to Lucas:
* HA group priority selection is done in [1].
* LRP scheduling is done in the l3_ovn_scheduler module [2].

We should find a way to sync both mechanisms.

[1]https://github.com/openstack/neutron/blob/9e2a0ac058305dd16d6dedb883d305cd162f5fe4/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py#L453-L475
[2]https://github.com/openstack/neutron/blob/9e2a0ac058305dd16d6dedb883d305cd162f5fe4/neutron/scheduler/l3_ovn_scheduler.py#L1

Hello again:

This issue you are hitting here is similar to [1]. For distributed VLAN traffic, what you need to do is to configure the GW chassis to define this [2]. "external_ids:ovn-chassis-mac-mappings" is a list of key-pairs. The key is the physnet, the value is the MAC address. The OVN controller will replace the local LRP with the defined MAC address if a packet is for a distributed port. That will solve the issue you have in your deployment using VLAN networks.

Please let me know if that helped you.

Regards.

[1]https://bugzilla.redhat.com/show_bug.cgi?id=1766930
[2]https://github.com/ovn-org/ovn/blob/main/controller/ovn-controller.8.xml#L239

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/864510

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/864510
Committed: https://opendev.org/openstack/kolla-ansible/commit/8bf8656dbad3def707eca2d8ddd2c9bfed389b86
Submitter: "Zuul (22348)"
Branch: master

commit 8bf8656dbad3def707eca2d8ddd2c9bfed389b86
Author: Bartosz Bezak <email address hidden>
Date: Tue Nov 15 11:08:15 2022 +0100

    Generate ovn-chassis-mac-mappings on ovn-controller group

    Previously ovn-chassis-mac-mappings [1] has been added only to
    ovn-controller-compute group. However external ports are being
    scheduled on network nodes, therefore we need also do that there.

    Closes-Bug: 1995078

    [1] https://github.com/ovn-org/ovn/blob/v22.09.0/controller/ovn-controller.8.xml#L239

    Change-Id: Ie62e9220bad56262cad602ca1480e6ca65827819

Hello Michal:

Can we consider this bug in Neutron as not valid? So far, I see the problem you had was the definition of the "ovn-chassis-mac-mappings" in the wrong group [1]; this is the parameter commented in c#4 that should be added to the GW chassis. Did that solved the issue?

Regards.

[1]https://review.opendev.org/c/openstack/kolla-ansible/+/864510

Hi Rodolfo,

At the moment it seems that it has fixed the issue. Basically we added that in the past not thinking about external ports.

I marked it as invalid in Neutron, if we'll see any issues - we'll reopen in future.
Thanks for your help!

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/864482

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/864483

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/864484

Hello Michal:

Nice to read that!

Regards.

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/864482
Committed: https://opendev.org/openstack/kolla-ansible/commit/86e2f2df428cc8e8157942b8776600fa9674ff99
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 86e2f2df428cc8e8157942b8776600fa9674ff99
Author: Bartosz Bezak <email address hidden>
Date: Tue Nov 15 11:08:15 2022 +0100

    Generate ovn-chassis-mac-mappings on ovn-controller group

    Previously ovn-chassis-mac-mappings [1] has been added only to
    ovn-controller-compute group. However external ports are being
    scheduled on network nodes, therefore we need also do that there.

    Closes-Bug: 1995078

    [1] https://github.com/ovn-org/ovn/blob/v22.09.0/controller/ovn-controller.8.xml#L239

    Change-Id: Ie62e9220bad56262cad602ca1480e6ca65827819
    (cherry picked from commit 8bf8656dbad3def707eca2d8ddd2c9bfed389b86)

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/864483
Committed: https://opendev.org/openstack/kolla-ansible/commit/55fb10b7820fa6c5de45774ca793427bc27a3e38
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 55fb10b7820fa6c5de45774ca793427bc27a3e38
Author: Bartosz Bezak <email address hidden>
Date: Tue Nov 15 11:08:15 2022 +0100

    Generate ovn-chassis-mac-mappings on ovn-controller group

    Previously ovn-chassis-mac-mappings [1] has been added only to
    ovn-controller-compute group. However external ports are being
    scheduled on network nodes, therefore we need also do that there.

    Closes-Bug: 1995078

    [1] https://github.com/ovn-org/ovn/blob/v22.09.0/controller/ovn-controller.8.xml#L239

    Change-Id: Ie62e9220bad56262cad602ca1480e6ca65827819
    (cherry picked from commit 8bf8656dbad3def707eca2d8ddd2c9bfed389b86)

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/864484
Committed: https://opendev.org/openstack/kolla-ansible/commit/40a4240e88573b43e3b7a7b98a9949f8029a1d18
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 40a4240e88573b43e3b7a7b98a9949f8029a1d18
Author: Bartosz Bezak <email address hidden>
Date: Tue Nov 15 11:08:15 2022 +0100

    Generate ovn-chassis-mac-mappings on ovn-controller group

    Previously ovn-chassis-mac-mappings [1] has been added only to
    ovn-controller-compute group. However external ports are being
    scheduled on network nodes, therefore we need also do that there.

    Closes-Bug: 1995078

    [1] https://github.com/ovn-org/ovn/blob/v22.09.0/controller/ovn-controller.8.xml#L239

    Change-Id: Ie62e9220bad56262cad602ca1480e6ca65827819
    (cherry picked from commit 8bf8656dbad3def707eca2d8ddd2c9bfed389b86)

After doing all those changes in kolla-ansible - we found that data traffic bandwidth for External VLAN ports is kilobytes per second (around 30-100 kbytes/sec) compared to 200 megabytes/sec before.
Probably that's related to MAC flooding issues in OVN.

If we remove ovn-chassis-mac-mappings on network nodes - and ha_chassis_group active chassis and gateway_chassis active chassis are the same - everything works fine (except HA).

Any other ideas?

This issue was fixed in the openstack/kolla-ansible 15.0.0.0rc1 release candidate.

This issue was fixed in the openstack/kolla-ansible 13.7.0 release.

This issue was fixed in the openstack/kolla-ansible 14.7.0 release.

In Centralised configuration (no DVR), this problem still persist: i.e. Traffic from VLAN external ports (Baremetal) is not reaching router as external port HA chassis is scheduled on different active chassis than gateway chassis (external vlan network) active chassis.

Traffic starts to go normally when chassis priorities got manually altered.

Tested on yoga with OVN 22.09

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/872023

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/872033

Reviewed: https://review.opendev.org/c/openstack/neutron/+/872023
Committed: https://opendev.org/openstack/neutron/commit/ac231c817473c018dde8fa31594b1c9a78a36c13
Submitter: "Zuul (22348)"
Branch: master

commit ac231c817473c018dde8fa31594b1c9a78a36c13
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Jan 23 16:17:46 2023 +0100

    Improve "sync_ha_chassis_group" method

    The method "sync_ha_chassis_group" now creates (or retrieves) a
    HA Chassis Group register and updates the needed HA Chassis registers
    in a single transaction. That is possible using the new ovsdbapp
    release 2.2.1 (check the depends-on patch).

    Depends-On: https://review.opendev.org/c/openstack/ovsdbapp/+/871836

    Related-Bug: #1995078
    Change-Id: I936855214c635de0e89d5d13a86562f5b282633c

Change abandoned by "Slawek Kaplonski <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/872033
Reason: This review is > 4 weeks without comment, and failed Zuul jobs the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

This issue was fixed in the openstack/kolla-ansible wallaby-eol release.

Change abandoned by "Slawek Kaplonski <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/872033
Reason: This review is > 4 weeks without comment, and failed Zuul jobs the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.