OpenStack Compute (nova)

misleading power state logging in _sync_instance_power_state

Bug #1439223 reported by Matt Riedemann
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
Matt Riedemann
OpenStack Compute (nova) 2015.1.0 "kilo"
Juno
Fix Released
Undecided
Unassigned
OpenStack Compute (nova) 2014.2.4

Bug Description

Commit aa1792eb4c1d10e9a192142ce7e20d37871d916a added more verbose logging of the various database and hypervisor states when _sync_instance_power_state is called (which can be called from handle_lifecycle_event - triggered by the libvirt driver, or from the _sync_power_states periodic task).

The current instance power_state from the DB's POV and the power state from the hypervisor's POV (via handle_lifecycle_event) can be different and if they are different, the database is updated with the power_state from the hypervisor and the local db_power_state variable is updated to be the same as the vm_power_state (from the hypervisor).

Then later, the db_power_state value is used to log the different states when we have conditions like the database says an instance is running / active but the hypervisor says it's stopped, so we call compute_api.stop().

We should be logging the original database power state and the power_state from the hypervisor to more accurately debug when we're out of sync.

This is already fixed on master: https://review.openstack.org/#/c/159263/

I'm reporting the bug so it this can be backported to stable/juno.

Matt Riedemann (mriedem)
tags: added: juno-backport-potential
Changed in nova:
assignee: nobody → Matt Riedemann (mriedem)
status: New → Fix Committed
importance: Undecided → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/juno)

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/169782

Thierry Carrez (ttx)
Changed in nova:
milestone: none → kilo-rc1
status: Fix Committed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/juno)

Reviewed: https://review.openstack.org/169782
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=729483f41ddfe55afbbf39089d9e4d93177f1189
Submitter: Jenkins
Branch: stable/juno

commit 729483f41ddfe55afbbf39089d9e4d93177f1189
Author: Matt Riedemann <email address hidden>
Date: Wed Feb 25 13:45:49 2015 -0800

    Better power_state logging in _sync_instance_power_state

    Commit aa1792eb4c1d10e9a192142ce7e20d37871d916a added more verbose
    logging of the various database and hypervisor states when
    _sync_instance_power_state is called (which can be called from
    handle_lifecycle_event - triggered by the libvirt driver, or from the
    _sync_power_states periodic task).

    This change adds logging of when the power_state in the database does
    not match the power_state from the hypervisor, and then the power_state
    in the database is updated to match what's in the hypervisor.

    Also, we save the original database power_state for logging when we have
    cases of stopping an active instance due to vm_state / power_state
    conflicts, or stopping a "stopped" instance but the hypervisor says is
    still running.

    This is all needed because debugging the various things that can hit
    this code at different times and cause problems is, well, terrible.

    Conflicts:
            nova/compute/manager.py

    NOTE(mriedem): The conflict is due to commit 1e8df2f00 on master
    which changed LOG.warn to LOG.warning.

    Closes-Bug: #1439223

    Change-Id: I297ca0037f43535c80afa8a4b5086ccb30558dc3
    (cherry picked from commit 26424b883e3cbd4c1abc1738b417eb6823b0e3cd)

tags: added: in-stable-juno
Thierry Carrez (ttx)
Changed in nova:
milestone: kilo-rc1 → 2015.1.0
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.