Unversioned notifications not being sent.

Bug #1721843 reported by Charles Volzka
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
Balazs Gibizer
Pike
Fix Committed
High
Matt Riedemann

Bug Description

Description
===========
After a vm moves from state 'building' to 'error' an unversioned notification is no longer sent if CONF.notifications.notification_format is set to 'unversioned'.

Steps to reproduce
==================
In nova.conf set
[notifications]
notification_format = unversioned

Setup environment so VM deploy fails.
To reproduce easily in my environment I raised a generic Exception just after the call to spawn in orchestrator's start_deploy_simple()
Attempt to deploy VM.
Wait for deploy to fail.

Expected result
===============
When the vm_state changes to 'error' an unversioned notification should be sent.

Actual result
=============
The unversioned notification is not sent.

Environment
===========
(pike)nova-compute/now 10:16.0.0-201710030907

Additional Info:
================
Problem seems to stem from this change: https://github.com/openstack/nova/commit/29cb8f1c459e6d23dd9303fb570cee773d9c4d02 at:
        if (NOTIFIER.is_enabled() and
                CONF.notifications.notification_format in ('both',
                                                           'versioned')):
Because 'unversioned' is not in the list, the @rpc.if_notifications_enabled decorator causes send_instance_update_notification() as well as _send_versioned_instance_update() to effectively be skipped. The name of the decorator and the comment describing it's functionality make it hard to determine is precise intended purpose. The decorator name implies it's checking if notifications are enabled at all. The comment in the decorator states it's specificly checking if versioned notifications are enabled and is in fact what it appears to be doing. Since the decorator was applied to send_instance_update_notification it's effectively blocking unversioned notifications if versioned notifications are not enabled.

Changed in nova:
status: New → Confirmed
status: Confirmed → New
tags: added: notifications
Revision history for this message
Balazs Gibizer (balazs-gibizer) wrote :

I think the observation is correct. The if_notifications_enabled decorator is applied to early in the send_instance_update_notification call chain.

Changed in nova:
status: New → Confirmed
Revision history for this message
Balazs Gibizer (balazs-gibizer) wrote :

A possible solution is to remove the decorator from send_instance_update_notification() as the called _send_versioned_instance_update() already has the decorator. This way the legacy notification is still emitted from send_instance_update_notification in every case.

This also shows the lack of our legacy notification test coverage. But I guess it is not something to want to improve now as we are focusing more on the versioned notifications.

Revision history for this message
Balazs Gibizer (balazs-gibizer) wrote :

To be clear the regression was introduced in I03c8e8225e51fd80580772752c0b292987e34218 not in https://github.com/openstack/nova/commit/29cb8f1c459e6d23dd9303fb570cee773d9c4d02

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/510603

Changed in nova:
assignee: nobody → Balazs Gibizer (balazs-gibizer)
status: Confirmed → In Progress
Revision history for this message
Matt Riedemann (mriedem) wrote :

Marking as high severity since this is a regression introduced in Pike.

Changed in nova:
importance: Undecided → Medium
importance: Medium → High
Revision history for this message
Matt Riedemann (mriedem) wrote :

The regression was introduced with this change: https://review.openstack.org/#/c/459923/

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/pike)

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/510957

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/510603
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=0ffe03a2650a698de1b83289098ed535734b2360
Submitter: Jenkins
Branch: master

commit 0ffe03a2650a698de1b83289098ed535734b2360
Author: Balazs Gibizer <email address hidden>
Date: Mon Oct 9 16:29:09 2017 +0200

    Fix sending legacy instance.update notification

    The if_notifications_enabled decorator skips the execution of the
    decorated function if the versioned notifications are not configured
    to be emitted. The send_instance_update_notification() call was wrongly
    decorated with this decorator as it not only sends versioned
    notification but also send the legacy compute.instance.update
    notification as well. This caused that the legacy instance.update
    notification was not emitted when the notification_format config option
    was set to unversioned.

    As the _send_versioned_instance_update() call already has the decorator
    the solution is simply to remove the decorator from the
    send_instance_update_notification() call.

    Closes-Bug: #1721843
    Change-Id: I9904adeb3de60cff4e29f1ab3c95399bbe9ff2e7

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/pike)

Reviewed: https://review.openstack.org/510957
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=38dbb0595072092f13c4017dc4edf811fa70fa21
Submitter: Zuul
Branch: stable/pike

commit 38dbb0595072092f13c4017dc4edf811fa70fa21
Author: Balazs Gibizer <email address hidden>
Date: Mon Oct 9 16:29:09 2017 +0200

    Fix sending legacy instance.update notification

    The if_notifications_enabled decorator skips the execution of the
    decorated function if the versioned notifications are not configured
    to be emitted. The send_instance_update_notification() call was wrongly
    decorated with this decorator as it not only sends versioned
    notification but also send the legacy compute.instance.update
    notification as well. This caused that the legacy instance.update
    notification was not emitted when the notification_format config option
    was set to unversioned.

    As the _send_versioned_instance_update() call already has the decorator
    the solution is simply to remove the decorator from the
    send_instance_update_notification() call.

    Closes-Bug: #1721843
    Change-Id: I9904adeb3de60cff4e29f1ab3c95399bbe9ff2e7
    (cherry picked from commit 0ffe03a2650a698de1b83289098ed535734b2360)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 17.0.0.0b1

This issue was fixed in the openstack/nova 17.0.0.0b1 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 16.0.2

This issue was fixed in the openstack/nova 16.0.2 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.