Need to audit when notifications are sent during live migration

Bug #1763051 reported by Matt Riedemann
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
In Progress
Medium
Matt Riedemann
Queens
In Progress
Medium
Artom Lifshitz
Rocky
In Progress
Medium
Artom Lifshitz

Bug Description

We do a pretty good job of testing that notifications are sent during certain operations, like live migration, but not so great a job at making sure that notifications are sent at expected times, like that start and end notifications actually happen at the start and end of a method (seems we should really use a decorator function for something like this for future proofing...).

This is a follow on from bug 1762876 where I thought about relying on the "live_migration._post.end" notification to be able to tell when a migration record should be 'completed' but that notification is sent *before* we change the status on the migration record:

https://github.com/openstack/nova/blob/fe976dcc559d059589a9ccf953a28e855abf50fb/nova/compute/manager.py#L6323

If you look at the beginning of the same method, the start notification is sent well after we've already started doing some work:

https://github.com/openstack/nova/blob/fe976dcc559d059589a9ccf953a28e855abf50fb/nova/compute/manager.py#L6261

So this bug is primarily meant to be an audit of at least the live migration flows where the methods are big and hairy so it's easy to see how over the years, the notifications got pushed into weird spots in those methods, and should be moved back to the appropriate start/end locations (or write a decorator to handle this).

Matt Riedemann (mriedem)
tags: added: low-hanging-fruit
Revision history for this message
Balazs Gibizer (balazs-gibizer) wrote :

I agree to fix this. I will also open a blueprint to do the audit on the rest of the start/end notification as it feels we have more such places.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/610739

Changed in nova:
assignee: nobody → Matt Riedemann (mriedem)
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/610739
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=7a2228142b8f8f015a554e2b25198ab37811a058
Submitter: Zuul
Branch: master

commit 7a2228142b8f8f015a554e2b25198ab37811a058
Author: Matt Riedemann <email address hidden>
Date: Mon Oct 15 18:16:40 2018 -0400

    Move live_migration.pre.start to the start of the method

    This simply moves the pre_live_migration start notification
    back to the start of the method before creating new volume
    attachments. While in here, the list of bdms are passed to
    the notify_about_instance_action method so if
    [notifications]/bdms_in_notifications is True we don't have
    to pull them out of the database again for each versioned
    notification.

    Change-Id: I372eddb9e356b02328f937917a856bc631691f53
    Partial-Bug: #1763051

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/612714

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/612773

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/rocky)

Reviewed: https://review.openstack.org/612714
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=ae2da62955e5c856fec71eebb4f3458fb4c7fb79
Submitter: Zuul
Branch: stable/rocky

commit ae2da62955e5c856fec71eebb4f3458fb4c7fb79
Author: Matt Riedemann <email address hidden>
Date: Mon Oct 15 18:16:40 2018 -0400

    Move live_migration.pre.start to the start of the method

    This simply moves the pre_live_migration start notification
    back to the start of the method before creating new volume
    attachments. While in here, the list of bdms are passed to
    the notify_about_instance_action method so if
    [notifications]/bdms_in_notifications is True we don't have
    to pull them out of the database again for each versioned
    notification.

    Change-Id: I372eddb9e356b02328f937917a856bc631691f53
    Partial-Bug: #1763051
    (cherry picked from commit 7a2228142b8f8f015a554e2b25198ab37811a058)

tags: added: in-stable-rocky
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/queens)

Reviewed: https://review.openstack.org/612773
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=6b065dad8015ea4fdacb0aedc763b38c5c8a9130
Submitter: Zuul
Branch: stable/queens

commit 6b065dad8015ea4fdacb0aedc763b38c5c8a9130
Author: Matt Riedemann <email address hidden>
Date: Mon Oct 15 18:16:40 2018 -0400

    Move live_migration.pre.start to the start of the method

    This simply moves the pre_live_migration start notification
    back to the start of the method before creating new volume
    attachments. While in here, the list of bdms are passed to
    the notify_about_instance_action method so if
    [notifications]/bdms_in_notifications is True we don't have
    to pull them out of the database again for each versioned
    notification.

    Conflicts in nova/compute/manager.py due to absence of comment from
    f0b9d5204f3.

    Change-Id: I372eddb9e356b02328f937917a856bc631691f53
    Partial-Bug: #1763051
    (cherry picked from commit 7a2228142b8f8f015a554e2b25198ab37811a058)
    (cherry picked from commit ae2da62955e5c856fec71eebb4f3458fb4c7fb79)

tags: added: in-stable-queens
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.