VMs don't get ip from dhcp after compute restart

@Darragh,

Given https://bugs.launchpad.net/neutron/+bug/1853582, it seems you assumption was that ovs_all_ports set to False prevented the deletion of ports created by Nova. But as I indicated in that bug, you assumption is not valid. Do you still believe we have to pursue this bug? If yes, how should the report be updated?

@Miguel,

Yes I think we should pursue this bug. I mentioned the ovs-cleanup here because it's needed to reproduce the problem in devstack.

The problem was first seen on a production system installed by a distro, and I was able to reproduce with the distro. But I was having difficulty reproducing on devstack. Unlike the distro, devstack was not running ovs-cleanup at boot, but I didn't think this was relevant because I assumed it was not deleting nova ports (ovs_all_ports was false). After I found that assumption was incorrect, I added a systemd script to run ovs-cleanup, changed the timings so q-agt started 4 sec after ovs-cleanup, and n-cpu started 4 sec after q-agt, and then the problem happened.

I don't think ovs-cleanup is the problem here. The problem was introduced with https://opendev.org/openstack/neutron/commit/62fe7852bbd70a24174853997096c52ee015e269 , but I'm not sure how it should be fixed.

Fix proposed to branch: master
Review: https://review.opendev.org/697655

I just started digesting this bug, but I may have found a previous occurence which seems to have a quite deep analysis. May this be a duplicate of the following bug?

https://bugs.launchpad.net/neutron/+bug/1681979

@Bence

No, that bug is from 2017. This problem is a consequence of agent_boot_time no longer being used, which was added this year: https://opendev.org/openstack/neutron/commit/a5244d6d44d2b66de27dc77efa7830fa657260be

Steps to reproduce using multinode devstack:

- Boot 4 or more vms to same compute and same vxlan net.
- Wait until they are fully working.
- check the flood to tun flow is present:
ovs-ofctl dump-flows br-tun table=22 | grep priority=1
- On compute disable services so they won't start automatically on boot:
- systemctl disable <email address hidden> <email address hidden> <email address hidden>
- Reboot compute
- Run neutron-ovs-cleanup
- systemctl start <email address hidden>
- journalctl -f -u <email address hidden>
- wait until loop iteration > 0
- systemctl start <email address hidden>
- vms are SHUTOFF - start them - nova start vm1 vm2 vm3 vm4
- wait until vms show ACTIVE in nova list
- check the flood to tun flow is missing:
- ovs-ofctl dump-flows br-tun table=22 | grep priority=1

Reviewed: https://review.opendev.org/697655
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=93e9dc5426764b791ac69e62c6d60be7591c16ab
Submitter: Zuul
Branch: master

commit 93e9dc5426764b791ac69e62c6d60be7591c16ab
Author: Darragh O'Reilly <email address hidden>
Date: Fri Dec 6 10:06:21 2019 +0000

    ovs agent: signal to plugin if tunnel refresh needed

    Currently the ovs agent calls update_device_list with the
    agent_restarted flag set only on the first loop iteration. Then the
    server knows to send the l2pop flooding entries for the network to
    the agent. But when a compute node with many instances on many
    networks reboots, it takes time to readd all the active devices and
    some may be readded after the first loop iteration. Then the server
    can fail to send the flooding entries which means there will be no
    flood_to_tuns flow and broadcasts like dhcp will fail.

    This patch fixes that by renaming the agent_restarted flag to
    refresh_tunnels and setting it if the agent has not received the
    flooding entries for the network.

    Change-Id: I607aa8fa399e72b037fd068ad4f02b6210e57e91
    Closes-Bug: #1853613

Fix proposed to branch: stable/train
Review: https://review.opendev.org/704173

Reviewed: https://review.opendev.org/704173
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=bc0ab0fcd721d3fb01fd83291269a586f50efa0e
Submitter: Zuul
Branch: stable/train

commit bc0ab0fcd721d3fb01fd83291269a586f50efa0e
Author: Darragh O'Reilly <email address hidden>
Date: Fri Jan 24 16:19:30 2020 +0000

    ovs agent: signal to plugin if tunnel refresh needed

    Patch https://review.opendev.org/#/c/697655/ cannot be backported
    because it includes an RPC version change. This patch is for the
    stable branches.

    Currently the ovs agent calls update_device_list with the
    agent_restarted flag set only on the first loop iteration. Then the
    server knows to send the l2pop flooding entries for the network to
    the agent. But when a compute node with many instances on many
    networks reboots, it takes time to readd all the active devices and
    some may be readded after the first loop iteration. Then the server
    can fail to send the flooding entries which means there will be no
    flood_to_tuns flow and broadcasts like dhcp will fail.

    This patch fixes that by also setting the agent_restarted flag if
    the agent has not received the flooding entries for a network.

    Change-Id: Iccc4fe4a785ee042fd76a663d0e76a27facd1809
    Closes-Bug: #1853613

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/708748

This issue was fixed in the openstack/neutron 16.0.0.0b1 development milestone.

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/709445

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/709446

Reviewed: https://review.opendev.org/708748
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=aee87e72b1d456da66d719ccf19054ac1f285a7b
Submitter: Zuul
Branch: stable/stein

commit aee87e72b1d456da66d719ccf19054ac1f285a7b
Author: Darragh O'Reilly <email address hidden>
Date: Fri Jan 24 16:19:30 2020 +0000

    ovs agent: signal to plugin if tunnel refresh needed

    Patch https://review.opendev.org/#/c/697655/ cannot be backported
    because it includes an RPC version change. This patch is for the
    stable branches.

    Currently the ovs agent calls update_device_list with the
    agent_restarted flag set only on the first loop iteration. Then the
    server knows to send the l2pop flooding entries for the network to
    the agent. But when a compute node with many instances on many
    networks reboots, it takes time to readd all the active devices and
    some may be readded after the first loop iteration. Then the server
    can fail to send the flooding entries which means there will be no
    flood_to_tuns flow and broadcasts like dhcp will fail.

    This patch fixes that by also setting the agent_restarted flag if
    the agent has not received the flooding entries for a network.

    Change-Id: Iccc4fe4a785ee042fd76a663d0e76a27facd1809
    Closes-Bug: #1853613
    (cherry picked from commit bc0ab0fcd721d3fb01fd83291269a586f50efa0e)

Fix proposed to branch: stable/pike
Review: https://review.opendev.org/709683

Reviewed: https://review.opendev.org/709445
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=d1e2b840b57ba9425d5ea710a0509b677eefcc08
Submitter: Zuul
Branch: stable/rocky

commit d1e2b840b57ba9425d5ea710a0509b677eefcc08
Author: Darragh O'Reilly <email address hidden>
Date: Fri Jan 24 16:19:30 2020 +0000

    ovs agent: signal to plugin if tunnel refresh needed

    Patch https://review.opendev.org/#/c/697655/ cannot be backported
    because it includes an RPC version change. This patch is for the
    stable branches.

    Currently the ovs agent calls update_device_list with the
    agent_restarted flag set only on the first loop iteration. Then the
    server knows to send the l2pop flooding entries for the network to
    the agent. But when a compute node with many instances on many
    networks reboots, it takes time to readd all the active devices and
    some may be readded after the first loop iteration. Then the server
    can fail to send the flooding entries which means there will be no
    flood_to_tuns flow and broadcasts like dhcp will fail.

    This patch fixes that by also setting the agent_restarted flag if
    the agent has not received the flooding entries for a network.

    Change-Id: Iccc4fe4a785ee042fd76a663d0e76a27facd1809
    Closes-Bug: #1853613
    (cherry picked from commit bc0ab0fcd721d3fb01fd83291269a586f50efa0e)
    (cherry picked from commit aee87e72b1d456da66d719ccf19054ac1f285a7b)

Reviewed: https://review.opendev.org/709446
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=576e76f209ce69d99606ce126cfd4a96d99f8e88
Submitter: Zuul
Branch: stable/queens

commit 576e76f209ce69d99606ce126cfd4a96d99f8e88
Author: Darragh O'Reilly <email address hidden>
Date: Fri Jan 24 16:19:30 2020 +0000

    ovs agent: signal to plugin if tunnel refresh needed

    Patch https://review.opendev.org/#/c/697655/ cannot be backported
    because it includes an RPC version change. This patch is for the
    stable branches.

    Currently the ovs agent calls update_device_list with the
    agent_restarted flag set only on the first loop iteration. Then the
    server knows to send the l2pop flooding entries for the network to
    the agent. But when a compute node with many instances on many
    networks reboots, it takes time to readd all the active devices and
    some may be readded after the first loop iteration. Then the server
    can fail to send the flooding entries which means there will be no
    flood_to_tuns flow and broadcasts like dhcp will fail.

    This patch fixes that by also setting the agent_restarted flag if
    the agent has not received the flooding entries for a network.

    Change-Id: Iccc4fe4a785ee042fd76a663d0e76a27facd1809
    Closes-Bug: #1853613
    (cherry picked from commit bc0ab0fcd721d3fb01fd83291269a586f50efa0e)
    (cherry picked from commit aee87e72b1d456da66d719ccf19054ac1f285a7b)
    (cherry picked from commit d1e2b840b57ba9425d5ea710a0509b677eefcc08)

This issue was fixed in the openstack/neutron 13.0.7 release.

Thanks for uploading the fix for this bug report to -proposed. However, when reviewing the package in -proposed and the details of this bug report I noticed that the bug description is missing information required for the SRU process. You can find full details at http://wiki.ubuntu.com/StableReleaseUpdates#Procedure but essentially this bug is missing some of the following: a statement of impact, a test case and details regarding the regression potential. Thanks in advance!

> bug description is missing information required for the SRU process

sorry, I updated the description to refer this bug to the sru template in bug 1869808 as that sru info applies to all other bugs in the upload.

Hello Darragh, or anyone else affected,

Accepted neutron into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/neutron/2:12.1.1-0ubuntu4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

All SRU verification completed and performed in https://bugs.launchpad.net/neutron/+bug/1869808 so please refer to that LP for the results.

The verification of the Stable Release Update for neutron has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package neutron - 2:12.1.1-0ubuntu4

---------------
neutron (2:12.1.1-0ubuntu4) bionic; urgency=medium

  * Fix interrupt of VLAN traffic on reboot of neutron-ovs-agent:
  - d/p/0001-ovs-agent-signal-to-plugin-if-tunnel-refresh-needed.patch (LP: #1853613)
  - d/p/0002-Do-not-block-connection-between-br-int-and-br-phys-o.patch (LP: #1869808)
  - d/p/0003-Ensure-that-stale-flows-are-cleaned-from-phys_bridge.patch (LP: #1864822)
  - d/p/0004-DVR-Reconfigure-re-created-physical-bridges-for-dvr-.patch (LP: #1864822)
  - d/p/0005-Ensure-drop-flows-on-br-int-at-agent-startup-for-DVR.patch (LP: #1887148)
  - d/p/0006-Don-t-check-if-any-bridges-were-recrected-when-OVS-w.patch (LP: #1864822)
  - d/p/0007-Not-remove-the-running-router-when-MQ-is-unreachable.patch (LP: #1871850)

 -- Edward Hope-Morley <email address hidden> Mon, 22 Feb 2021 16:55:40 +0000

This issue was fixed in the openstack/neutron pike-eol release.

This issue was fixed in the openstack/neutron queens-eol release.