OpenStack Charm Guide

MTU mismatch between o-hm0 and lb-mgmt-net

Bug #2018998 reported by Giuseppe Petralia
38
This bug affects 6 people
Affects Status Importance Assigned to Milestone
OpenStack Charm Guide
Triaged
Undecided
Unassigned
OpenStack Octavia Charm
Fix Released
High
Edward Hope-Morley
2023.1
Fix Released
Undecided
Unassigned
Yoga
Fix Released
Undecided
Unassigned
Zed
Fix Released
Undecided
Unassigned

Bug Description

lb-mgmt-net is created by the charm with mtu 8942 while on o-hm0 charm sets mtu 1500. This is causing heartbeats of certain LBs to be dropped forcing octavia to fail over them.

This seem related to https://storyboard.openstack.org/#!/story/2009734

Some LBs, because of the number of members/listeners/pools, may have an health status message longer then 1500.

i.e.
12:06:56.706585 fa:16:3e:56:a6:05 > fa:16:3e:cb:ba:9e, ethertype IPv6 (0x86dd), length 1596: fc00:51be:95ec:ac07:f816:3eff:fe56:a605.63575 > fc00:51be:95ec:ac07:f816:3eff:fecb:ba9e.5555: UDP, length 1534

See 1534 > 1500 and o-hm0 never receives them.

We worked around that running on all octavia:

ovs-vsctl set interface o-hm0 mtu_request=8942

Revision history for this message
Pedro Victor Lourenço Fragola (pedrovlf) wrote :

I had the same situation when the lb-mgmt-net was set to use 9042 MTU and o-hm0 is set to use 1500 MTU.

The Octavia units will show the drop packets:

[2065317.692684] o-hm0: dropped over-mtu packet: 1744 > 1500
[2065339.573539] o-hm0: dropped over-mtu packet: 1654 > 1500
[2065347.747633] o-hm0: dropped over-mtu packet: 1744 > 1500
[2065375.092206] o-hm0: dropped over-mtu packet: 1653 > 1500
[2065405.176370] o-hm0: dropped over-mtu packet: 1655 > 1500

If the lb-mgmt-net is set to use the same MTU as o-hm0, for example, 1500 it will work using defragment.
During the problem, Octavia Health starts the failover of the amphoras after 60 seconds.

In this case, the LB has more than 60 members.

[Workaround]
The workaround described on this LP works, but the final solution would be the charm check the MTU config lb-mgmt-net and set the same config for o-hm0 interface.

Revision history for this message
Edward Hope-Morley (hopem) wrote :

I think this is reasonably high priority given that it is not uncommon for a k8s deployment to have sufficient members to hit this.

Changed in charm-octavia:
importance: Undecided → High
assignee: nobody → Edward Hope-Morley (hopem)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-octavia (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-octavia/+/883868

Changed in charm-octavia:
status: New → In Progress
Revision history for this message
Felipe Reyes (freyes) wrote :

Please, document this new behavior in the charm-guide's release note.

Changed in charm-guide:
status: New → Triaged
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-octavia (master)

Reviewed: https://review.opendev.org/c/openstack/charm-octavia/+/883868
Committed: https://opendev.org/openstack/charm-octavia/commit/f326ef267aa3e671b608b823bfcaf938868cba45
Submitter: "Zuul (22348)"
Branch: master

commit f326ef267aa3e671b608b823bfcaf938868cba45
Author: Edward Hope-Morley <email address hidden>
Date: Mon May 22 20:05:29 2023 +0100

    Ensure mgmt network mtu

    This will ensure that mtu is set whenever setup_hm_port()
    is called regardless of if the port has just been created
    or not. Also checks mtu in update-status hook.

    Change-Id: I80b5cea812becd1724fcfe2a8a232253eac76735
    Closes-Bug: #2018998

Changed in charm-octavia:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-octavia (stable/2023.1)

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/charm-octavia/+/886091

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-octavia (stable/zed)

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/charm-octavia/+/886261

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-octavia (stable/yoga)

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/charm-octavia/+/886262

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-octavia (stable/2023.1)

Reviewed: https://review.opendev.org/c/openstack/charm-octavia/+/886091
Committed: https://opendev.org/openstack/charm-octavia/commit/3ebb523f89cda314b52805443fb4782a16888c66
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit 3ebb523f89cda314b52805443fb4782a16888c66
Author: Edward Hope-Morley <email address hidden>
Date: Mon May 22 20:05:29 2023 +0100

    Ensure mgmt network mtu

    This will ensure that mtu is set whenever setup_hm_port()
    is called regardless of if the port has just been created
    or not. Also checks mtu in update-status hook.

    Change-Id: I80b5cea812becd1724fcfe2a8a232253eac76735
    Closes-Bug: #2018998
    (cherry picked from commit f326ef267aa3e671b608b823bfcaf938868cba45)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-octavia (stable/zed)

Reviewed: https://review.opendev.org/c/openstack/charm-octavia/+/886261
Committed: https://opendev.org/openstack/charm-octavia/commit/1fa839826f5009d1cbf8bcc97234afa1fbd5dfb0
Submitter: "Zuul (22348)"
Branch: stable/zed

commit 1fa839826f5009d1cbf8bcc97234afa1fbd5dfb0
Author: Edward Hope-Morley <email address hidden>
Date: Mon May 22 20:05:29 2023 +0100

    Ensure mgmt network mtu

    This will ensure that mtu is set whenever setup_hm_port()
    is called regardless of if the port has just been created
    or not. Also checks mtu in update-status hook.

    Change-Id: I80b5cea812becd1724fcfe2a8a232253eac76735
    Closes-Bug: #2018998
    (cherry picked from commit f326ef267aa3e671b608b823bfcaf938868cba45)
    (cherry picked from commit 3ebb523f89cda314b52805443fb4782a16888c66)

Edward Hope-Morley (hopem)
Changed in charm-octavia:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.