When upgrading to Stein, nova-consoleauth service is deprecated and should be removed. However if nova-consoleauth service is located on the same host with nova-compute, corresponding row in compute_nodes table is soft-deleted, making nova-compute service report in log:
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager [req-f0255008-c398-406c-bca0-12cdc34fc0b4 - - - - -] Error updating resources for node vzstor1.vstoragedomain.: ResourceProviderCreationFailed: Failed to create resource provider vzstor1.vstoragedomain
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager Traceback (most recent call last):
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7399, in update_available_resource_for_node
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager rt.update_available_resource(context, nodename)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 689, in update_available_resource
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager self._update_available_resource(context, resources)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager return f(*args, **kwargs)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 713, in _update_available_resource
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager self._init_compute_node(context, resources)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 562, in _init_compute_node
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager self._update(context, cn)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 887, in _update
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager inv_data,
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 68, in set_inventory_for_provider
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager parent_provider_uuid=parent_provider_uuid,
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 37, in __run_method
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager return getattr(self.instance, __name)(*args, **kwargs)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 1106, in set_inventory_for_provider
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager parent_provider_uuid=parent_provider_uuid)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 667, in _ensure_resource_provider
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager parent_provider_uuid=parent_provider_uuid)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 66, in wrapper
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager return f(self, *a, **k)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 614, in _create_resource_provider
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager raise exception.ResourceProviderCreationFailed(name=name)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager ResourceProviderCreationFailed: Failed to create resource provider vzstor1.vstoragedomain
Steps to reproduce
==================
# nova service-list
+--------------------------------------+------------------+------------------------+----------+---------+-------+----------------------------+-----------------+-------------+
| Id | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason | Forced down |
+--------------------------------------+------------------+------------------------+----------+---------+-------+----------------------------+-----------------+-------------+
| 57885a23-6bc2-4400-a199-a7e0defe2e00 | nova-conductor | vzstor1.vstoragedomain | internal | enabled | up | 2019-11-18T12:33:36.904174 | - | False |
| 0b08d3ea-1b4a-4076-a51e-fbd892241b9b | nova-scheduler | vzstor1.vstoragedomain | internal | enabled | up | 2019-11-18T12:33:40.111038 | - | False |
| 7d7f3dc6-da81-41b4-be30-c2cc451b560a | nova-consoleauth | vzstor1.vstoragedomain | internal | enabled | up | - | - | False |
| 367a4591-cce5-4b7b-ad7c-69135aa803aa | nova-compute | vzstor1.vstoragedomain | nova | enabled | up | 2019-11-18T12:33:43.500922 | - | False |
+--------------------------------------+------------------+------------------------+----------+---------+-------+----------------------------+-----------------+-------------+
# nova service-delete 7d7f3dc6-da81-41b4-be30-c2cc451b560a <-- this is nova-consoleauth service
nova=# select uuid,deleted_at from compute_nodes; uuid | deleted_at
--------------------------------------+----------------------------
13c1fbd5-fbc1-4301-8a6e-9d50bde6826f | 2019-11-18 12:19:30.080625 <-- compute_node with the same host is also deleted
Expected result
===============
compute_node is not deleted from db if nova-consoleauth service is removed
Actual result
=============
compute_node is deleted from db if nova-consoleauth service is removed
When upgrading to Stein, nova-consoleauth service is deprecated and should be removed. However if nova-consoleauth service is located on the same host with nova-compute, corresponding row in compute_nodes table is soft-deleted, making nova-compute service report in log:
2019-11-18 16:03:20.069 7 ERROR nova.compute. manager [req-f0255008- c398-406c- bca0-12cdc34fc0 b4 - - - - -] Error updating resources for node vzstor1. vstoragedomain. : ResourceProvide rCreationFailed : Failed to create resource provider vzstor1. vstoragedomain manager Traceback (most recent call last): manager File "/usr/lib/ python2. 7/site- packages/ nova/compute/ manager. py", line 7399, in update_ available_ resource_ for_node manager rt.update_ available_ resource( context, nodename) manager File "/usr/lib/ python2. 7/site- packages/ nova/compute/ resource_ tracker. py", line 689, in update_ available_ resource manager self._update_ available_ resource( context, resources) manager File "/usr/lib/ python2. 7/site- packages/ oslo_concurrenc y/lockutils. py", line 274, in inner manager return f(*args, **kwargs) manager File "/usr/lib/ python2. 7/site- packages/ nova/compute/ resource_ tracker. py", line 713, in _update_ available_ resource manager self._init_ compute_ node(context, resources) manager File "/usr/lib/ python2. 7/site- packages/ nova/compute/ resource_ tracker. py", line 562, in _init_compute_node manager self._update( context, cn) manager File "/usr/lib/ python2. 7/site- packages/ nova/compute/ resource_ tracker. py", line 887, in _update manager inv_data, manager File "/usr/lib/ python2. 7/site- packages/ nova/scheduler/ client/ __init_ _.py", line 68, in set_inventory_ for_provider manager parent_ provider_ uuid=parent_ provider_ uuid, manager File "/usr/lib/ python2. 7/site- packages/ nova/scheduler/ client/ __init_ _.py", line 37, in __run_method manager return getattr( self.instance, __name)(*args, **kwargs) manager File "/usr/lib/ python2. 7/site- packages/ nova/scheduler/ client/ report. py", line 1106, in set_inventory_ for_provider manager parent_ provider_ uuid=parent_ provider_ uuid) manager File "/usr/lib/ python2. 7/site- packages/ nova/scheduler/ client/ report. py", line 667, in _ensure_ resource_ provider manager parent_ provider_ uuid=parent_ provider_ uuid) manager File "/usr/lib/ python2. 7/site- packages/ nova/scheduler/ client/ report. py", line 66, in wrapper manager return f(self, *a, **k) manager File "/usr/lib/ python2. 7/site- packages/ nova/scheduler/ client/ report. py", line 614, in _create_ resource_ provider manager raise exception. ResourceProvide rCreationFailed (name=name) manager ResourceProvide rCreationFailed : Failed to create resource provider vzstor1. vstoragedomain
2019-11-18 16:03:20.069 7 ERROR nova.compute.
2019-11-18 16:03:20.069 7 ERROR nova.compute.
2019-11-18 16:03:20.069 7 ERROR nova.compute.
2019-11-18 16:03:20.069 7 ERROR nova.compute.
2019-11-18 16:03:20.069 7 ERROR nova.compute.
2019-11-18 16:03:20.069 7 ERROR nova.compute.
2019-11-18 16:03:20.069 7 ERROR nova.compute.
2019-11-18 16:03:20.069 7 ERROR nova.compute.
2019-11-18 16:03:20.069 7 ERROR nova.compute.
2019-11-18 16:03:20.069 7 ERROR nova.compute.
2019-11-18 16:03:20.069 7 ERROR nova.compute.
2019-11-18 16:03:20.069 7 ERROR nova.compute.
2019-11-18 16:03:20.069 7 ERROR nova.compute.
2019-11-18 16:03:20.069 7 ERROR nova.compute.
2019-11-18 16:03:20.069 7 ERROR nova.compute.
2019-11-18 16:03:20.069 7 ERROR nova.compute.
2019-11-18 16:03:20.069 7 ERROR nova.compute.
2019-11-18 16:03:20.069 7 ERROR nova.compute.
2019-11-18 16:03:20.069 7 ERROR nova.compute.
2019-11-18 16:03:20.069 7 ERROR nova.compute.
2019-11-18 16:03:20.069 7 ERROR nova.compute.
2019-11-18 16:03:20.069 7 ERROR nova.compute.
2019-11-18 16:03:20.069 7 ERROR nova.compute.
2019-11-18 16:03:20.069 7 ERROR nova.compute.
2019-11-18 16:03:20.069 7 ERROR nova.compute.
2019-11-18 16:03:20.069 7 ERROR nova.compute.
Steps to reproduce
==================
# nova service-list ------- ------- ------- ------- ----+-- ------- ------- --+---- ------- ------- ------+ ------- ---+--- ------+ ------- +------ ------- ------- ------- -+----- ------- -----+- ------- -----+ ------- ------- ------- ------- ----+-- ------- ------- --+---- ------- ------- ------+ ------- ---+--- ------+ ------- +------ ------- ------- ------- -+----- ------- -----+- ------- -----+ 6bc2-4400- a199-a7e0defe2e 00 | nova-conductor | vzstor1. vstoragedomain | internal | enabled | up | 2019-11- 18T12:33: 36.904174 | - | False | 1b4a-4076- a51e-fbd892241b 9b | nova-scheduler | vzstor1. vstoragedomain | internal | enabled | up | 2019-11- 18T12:33: 40.111038 | - | False | da81-41b4- be30-c2cc451b56 0a | nova-consoleauth | vzstor1. vstoragedomain | internal | enabled | up | - | - | False | cce5-4b7b- ad7c-69135aa803 aa | nova-compute | vzstor1. vstoragedomain | nova | enabled | up | 2019-11- 18T12:33: 43.500922 | - | False | ------- ------- ------- ------- ----+-- ------- ------- --+---- ------- ------- ------+ ------- ---+--- ------+ ------- +------ ------- ------- ------- -+----- ------- -----+- ------- -----+
+------
| Id | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason | Forced down |
+------
| 57885a23-
| 0b08d3ea-
| 7d7f3dc6-
| 367a4591-
+------
nova=# select uuid,deleted_at from compute_nodes;
uuid | deleted_at ------- ------- ------- ------- ---+--- ------- ------- ------- ---- fbc1-4301- 8a6e-9d50bde682 6f |
-------
13c1fbd5-
# nova service-delete 7d7f3dc6- da81-41b4- be30-c2cc451b56 0a <-- this is nova-consoleauth service
nova=# select uuid,deleted_at from compute_nodes;
uuid | deleted_at ------- ------- ------- ------- ---+--- ------- ------- ------- ---- fbc1-4301- 8a6e-9d50bde682 6f | 2019-11-18 12:19:30.080625 <-- compute_node with the same host is also deleted
-------
13c1fbd5-
Expected result
===============
compute_node is not deleted from db if nova-consoleauth service is removed
Actual result
=============
compute_node is deleted from db if nova-consoleauth service is removed
Environment
===========
Queens, Stein