Upgrade from Ocata to Pike caused instance listing to fail

Bug #1809260 reported by Andrea Ieri
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Charms Deployment Guide
Fix Released
High
Unassigned
OpenStack Nova Cloud Controller Charm
Triaged
Medium
Unassigned

Bug Description

This really seems awfully similar to LP#1809210 (and it happened in the same site): after the upgrade listing all instances yielded the following failure:

This service is older (v9) than the minimum (v16) version of the rest of the deployment. Unable to continue. (HTTP 400)

Looking at the nova database, indeed there were some stale entries with version=9:

https://pastebin.canonical.com/p/DDs4GFrNxw/

The juju-machine-<n>-lxc-<m> hosts did not even exist anymore in the model, but setting them to disabled (manually, `openstack host set --disable <hostname>` did not work) did not help.

Instance listing resumed working only after all version=9 entries were deleted.

Andrea Ieri (aieri)
summary: - Upgrade from Newton to Ocata caused instance listing to fail
+ Upgrade from Ocata to Pike caused instance listing to fail
Revision history for this message
James Page (james-page) wrote :

@aieri

I'm assuming that the v9 based services and the machines hosting them where around at some point in time; the 'Service' db code checks the version and raises this exception for everything that does not meet the minimum requirement hence why deleting the entries resolved the issue.

I think we at a minimum need to add some pre-flight check notes to the charm deployment guide to purge any down/old service entries from nova prior to upgrades. This can probably be done in advance of an upgrade but obviously can't be done post upgrade as the API will continue to throw these errors.

Changed in charm-deployment-guide:
status: New → Triaged
importance: Undecided → High
Changed in charm-nova-cloud-controller:
status: New → Triaged
Revision history for this message
James Page (james-page) wrote :

a "openstack compute service delete" might be possible to clean up the old, versioned services.

Revision history for this message
James Page (james-page) wrote :

deployment guide update: https://review.openstack.org/631209

James Page (james-page)
Changed in charm-nova-cloud-controller:
importance: Undecided → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-deployment-guide (master)

Reviewed: https://review.openstack.org/631209
Committed: https://git.openstack.org/cgit/openstack/charm-deployment-guide/commit/?id=5b82aa481447bdd033d8250f851422642ee46db3
Submitter: Zuul
Branch: master

commit 5b82aa481447bdd033d8250f851422642ee46db3
Author: James Page <email address hidden>
Date: Wed Jan 16 14:42:21 2019 +0200

    upgrades: purge old compute service entries

    Add pre-upgrade step to discover and purge and old compute service
    entries; in later openstack releases, the DB model will throw
    exceptions for service entries with versions less than the minimum
    requirement, causing API requests to error post upgrade.

    Change-Id: Id739f0f23aaadf08dff216bb9f17552deee1b0a4
    Closes-Bug: 1809260

Changed in charm-deployment-guide:
status: Triaged → Fix Released
tags: added: openstack-upgrade
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.