pre-cells_v2 nova-osapi_compute service in database breaks instance lookup
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Compute (nova) |
Fix Released
|
Medium
|
Matt Riedemann | ||
Ocata |
Confirmed
|
Medium
|
Unassigned | ||
Pike |
Confirmed
|
Medium
|
Unassigned | ||
Queens |
Fix Committed
|
Medium
|
Matt Riedemann |
Bug Description
This was encoutered on Ocata after and upgrade from Newton, but affects master to the best of my knowledge.
During our upgrade from Newton -> Ocata after we finished cells_v2 migration and map'd instances accordingly, `nova show $uuid` no longer worked. Returning the error:
{"itemNotFound": {"message": "Instance 0e1e6038-
After much probing and with the complete lack of logs/warnings I discovered that the 'nova-osapi_
With version less than 15 and cells_v2, the instance lookup will not work since it never reaches the code path needed to talk to cells_v2 things. [1]
The solution was to service delete the old services.
My suggestion moving forward is to do one or more of the following:
* place a WARN in the linked nova code [1]
* add a check to `nova-status upgrade check` to look for old service entries
[0] http://
[1] https:/
Changed in nova: | |
assignee: | nobody → Matt Riedemann (mriedem) |
The code in [1] was added in Newton, and I think we'd be OK to add a warning if you're not using cells v1 and the osapi_compute minimum version is < 15 in that code as a breadcrumb at least, and we could backport that through to queens, pike and ocata.
For nova-status, we'd likely add a check that queries the minimum nova-osapi_compute service version across all cells (API services should really only be in once cell though) and if < 15 we'd emit a warning. The thing about the nova-status check would be, if you had older nova-osapi_compute services in your nova (cell) database from before upgrading to ocata where cells v2 was required, and then you re-configured the API to point the [database] /connection at the nova_cell0 database and created a new 'current' service version, the cross-cell min version check would give a warning for a cell table entry you don't actually care about. The resolution would just be to delete that entry though I think. Alternatively, we could just not look across cells in nova-status and just rely on [database] /connection being set (or at least look in cell0).