os_nova: upgrade_levels/compute=auto failure on master
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Compute (nova) |
Fix Released
|
Medium
|
git-harry | ||
Queens |
In Progress
|
Medium
|
Matt Riedemann | ||
openstack-ansible |
Fix Released
|
Undecided
|
Unassigned |
Bug Description
It looks like a recent change [1] in nova, to remove RPC 4.x support, has exposed a bug when using upgrade_
Tempest tests are failing, the following in nova-conductor.log shows the failure:
```
2018-03-03 05:13:23.679 9771 ERROR oslo_messaging.
2018-03-03 05:13:23.679 9771 ERROR oslo_messaging.
2018-03-03 05:13:23.679 9771 ERROR oslo_messaging.
2018-03-03 05:13:23.679 9771 ERROR oslo_messaging.
2018-03-03 05:13:23.679 9771 ERROR oslo_messaging.
2018-03-03 05:13:23.679 9771 ERROR oslo_messaging.
2018-03-03 05:13:23.679 9771 ERROR oslo_messaging.
2018-03-03 05:13:23.679 9771 ERROR oslo_messaging.
2018-03-03 05:13:23.679 9771 ERROR oslo_messaging.
2018-03-03 05:13:23.679 9771 ERROR oslo_messaging.
2018-03-03 05:13:23.679 9771 ERROR oslo_messaging.
2018-03-03 05:13:23.679 9771 ERROR oslo_messaging.
2018-03-03 05:13:23.679 9771 ERROR oslo_messaging.
2018-03-03 05:13:23.679 9771 ERROR oslo_messaging.
2018-03-03 05:13:23.679 9771 ERROR oslo_messaging.
2018-03-03 05:13:23.679 9771 ERROR oslo_messaging.
2018-03-03 05:13:23.679 9771 ERROR oslo_messaging.
2018-03-03 05:13:23.679 9771 ERROR oslo_messaging.
```
When openstack-
```
2018-03-02 17:25:55.954 19495 DEBUG nova.compute.rpcapi [req-97c173ed-
```
The reference to Mitaka is caused by [2], it looks to be intended to set the version cap to be as permissive as possible (N to N+1 upgrades) but it appears it hasn't been updated since it was first added for newton.
Restarting the services addresses the issue, observed by:
```
2018-03-04 21:42:14.367 21270 INFO nova.compute.rpcapi [req-95678c1e-
```
It seems like there may be two issues exposed here, one is the bug in nova setting the minimum version and the other is how OSA handles the deployment. With the default OSA deployment the minimum RPC version will change with a restart, it would seem that has the potential to cause failures if the order of the restarts is not controlled given those restarts are not triggered by the deployment process.
[1] https:/
[2] https:/
Changed in nova: | |
assignee: | git-harry (git-harry) → nobody |
assignee: | nobody → new (cloudie) |
Changed in openstack-ansible: | |
status: | New → In Progress |
Changed in openstack-ansible: | |
status: | In Progress → Fix Committed |
status: | Fix Committed → Fix Released |
Changed in nova: | |
assignee: | new (cloudie) → git-harry (git-harry) |
Changed in nova: | |
importance: | Undecided → Medium |
Fix proposed to branch: master /review. openstack. org/549737
Review: https:/