nova-manage cell_v2 verify_instance returns a valid instance mapping even after the instance is deleted/archived

Bug #1724621 reported by Surya Seetharaman
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
Surya Seetharaman
Pike
Fix Committed
Medium
Matt Riedemann

Bug Description

Although nova-manage cell_v2 verify_instance is used to check if the provided instance is correctly mapped to a cell or not, this should not be returning a valid mapping message if the instance itself is deleted. It should return an error message saying 'The instance does not exist'.

Steps to reproduce :

1. Create an instance :

-> nova boot --image 831bb8a0-9305-4cd7-b985-cbdadfb5d3db --flavor m1.nano test
-> nova list
+--------------------------------------+--------+--------+------------+-------------+---------------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+--------+--------+------------+-------------+---------------------------------+
| aec6eb34-6aaf-4883-8285-348d40fdac87 | test | ACTIVE | - | Running | public=2001:db8::4, 172.24.4.9 |
+--------------------------------------+--------+--------+------------+-------------+---------------------------------+

2. Delete the instance :

-> nova delete test
Request to delete server test has been accepted.
-> nova list
+--------------------------------------+--------+--------+------------+-------------+---------------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+--------+--------+------------+-------------+---------------------------------+
+--------------------------------------+--------+--------+------------+-------------+---------------------------------+

3. Verify Instance :

-> nova-manage cell_v2 verify_instance --uuid aec6eb34-6aaf-4883-8285-348d40fdac87
Instance aec6eb34-6aaf-4883-8285-348d40fdac87 is in cell: cell5 (c5ccba5d-1a45-4739-a5dd-d665a1b19301)

Basically the message that we get is misleading for a deleted instance. This is because verify_instance queries the instance_mappings table which maintains a mapping of the deleted instances as well.

Revision history for this message
Surya Seetharaman (tssurya) wrote :

There are other related bugs like below dealing with the existence of information about deleted instances in the instance_mapping table :

1. https://bugs.launchpad.net/nova/+bug/1722451
2. https://bugs.launchpad.net/nova/+bug/1679941

However, even if we decide to do a purge operation (https://bugs.launchpad.net/nova/+bug/1722451/comments/1) , the output of verify_instance for a deleted instance should not be dependent on the purge, IMHO.

Revision history for this message
melanie witt (melwitt) wrote :

End users of the REST API (with appropriate policy permission) can 'nova list' deleted instances. So it's not straightforward to decide whether or not to show that the instance is mapped. It *is* mapped and it can be seen by the end user if they can list deleted instances.

summary: nova-manage cell_v2 verify_instance returns a valid instance mapping
- even after the instance is deleted
+ even after the instance is deleted/archived
Revision history for this message
Surya Seetharaman (tssurya) wrote :

Following the discussion on the IRC channel, we have thus decided to do a follow up deletion of those instance_mappings from the nova-api database which is not done currently, once the corresponding rows are archived from the instances table using nova-manage db archive_deleted_rows.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/515034

Changed in nova:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/515034
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=32fd58813f8247641a6b574b5f01528b29d48b76
Submitter: Zuul
Branch: master

commit 32fd58813f8247641a6b574b5f01528b29d48b76
Author: Surya Seetharaman <email address hidden>
Date: Wed Oct 25 13:43:43 2017 +0200

    cleanup mapping/reqspec after archive instance

    This patch aims at deleting the records of the archived instances from
    the instance_mappings and request_specs tables in the API database
    immediately following their archival from instances to shadow_instances
    table. So upon running the 'nova-manage db archive_deleted_rows' command
    the records of the archived instances will be automatically removed from
    the instance_mappings and request_specs tables as well. A warning has
    also been added to fix the issue of 'nova-manage verify_instance'
    returning a valid instance mapping even after the instance is deleted.

    The patch also adds InstanceMappingList.destory_bulk() and
    RequestSpec.destroy_bulk() methods for ease of bulk deletion of records.

    Change-Id: I483701a55576c245d091ff086b32081b392f746e
    Closes-Bug: #1724621
    Closes-Bug: #1678056

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 17.0.0.0b2

This issue was fixed in the openstack/nova 17.0.0.0b2 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/pike)

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/580775

Matt Riedemann (mriedem)
Changed in nova:
importance: Undecided → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/pike)

Reviewed: https://review.openstack.org/580775
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=70de423255fd01822188bb9082a0c0cc1c8ec2d0
Submitter: Zuul
Branch: stable/pike

commit 70de423255fd01822188bb9082a0c0cc1c8ec2d0
Author: Surya Seetharaman <email address hidden>
Date: Wed Oct 25 13:43:43 2017 +0200

    cleanup mapping/reqspec after archive instance

    This patch aims at deleting the records of the archived instances from
    the instance_mappings and request_specs tables in the API database
    immediately following their archival from instances to shadow_instances
    table. So upon running the 'nova-manage db archive_deleted_rows' command
    the records of the archived instances will be automatically removed from
    the instance_mappings and request_specs tables as well. A warning has
    also been added to fix the issue of 'nova-manage verify_instance'
    returning a valid instance mapping even after the instance is deleted.

    The patch also adds InstanceMappingList.destory_bulk() and
    RequestSpec.destroy_bulk() methods for ease of bulk deletion of records.

    Change-Id: I483701a55576c245d091ff086b32081b392f746e
    Closes-Bug: #1724621
    Closes-Bug: #1678056
    (cherry picked from commit 32fd58813f8247641a6b574b5f01528b29d48b76)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 16.1.5

This issue was fixed in the openstack/nova 16.1.5 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.