| 2019-11-18 12:13:35 |
Pavel Gluschak |
bug |
|
|
added bug |
| 2019-11-18 12:13:43 |
Pavel Gluschak |
nova: assignee |
|
Pavel Gluschak (scsnow) |
|
| 2019-11-18 12:22:03 |
Pavel Gluschak |
description |
When upgrading to Stein, nova-consoleauth service is deprecated and should be removed. However if nova-consoleauth service is located on the same host with nova-compute, corresponding row in compute_nodes table is soft-deleted, making nova-compute service report in log:
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager [req-f0255008-c398-406c-bca0-12cdc34fc0b4 - - - - -] Error updating resources for node vzstor1.vstoragedomain.: ResourceProviderCreationFailed: Failed to create resource provider vzstor1.vstoragedomain
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager Traceback (most recent call last):
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7399, in update_available_resource_for_node
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager rt.update_available_resource(context, nodename)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 689, in update_available_resource
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager self._update_available_resource(context, resources)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager return f(*args, **kwargs)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 713, in _update_available_resource
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager self._init_compute_node(context, resources)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 562, in _init_compute_node
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager self._update(context, cn)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 887, in _update
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager inv_data,
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 68, in set_inventory_for_provider
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager parent_provider_uuid=parent_provider_uuid,
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 37, in __run_method
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager return getattr(self.instance, __name)(*args, **kwargs)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 1106, in set_inventory_for_provider
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager parent_provider_uuid=parent_provider_uuid)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 667, in _ensure_resource_provider
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager parent_provider_uuid=parent_provider_uuid)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 66, in wrapper
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager return f(self, *a, **k)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 614, in _create_resource_provider
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager raise exception.ResourceProviderCreationFailed(name=name)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager ResourceProviderCreationFailed: Failed to create resource provider vzstor1.vstoragedomain
Steps to reproduce
==================
# nova service-list
+--------------------------------------+------------------+------------------------+----------+---------+-------+----------------------------+-----------------+-------------+
| Id | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason | Forced down |
+--------------------------------------+------------------+------------------------+----------+---------+-------+----------------------------+-----------------+-------------+
| 57885a23-6bc2-4400-a199-a7e0defe2e00 | nova-conductor | vzstor1.vstoragedomain | internal | enabled | up | 2019-11-18T12:33:36.904174 | - | False |
| 0b08d3ea-1b4a-4076-a51e-fbd892241b9b | nova-scheduler | vzstor1.vstoragedomain | internal | enabled | up | 2019-11-18T12:33:40.111038 | - | False |
| 7d7f3dc6-da81-41b4-be30-c2cc451b560a | nova-consoleauth | vzstor1.vstoragedomain | internal | enabled | up | - | - | False |
| 367a4591-cce5-4b7b-ad7c-69135aa803aa | nova-compute | vzstor1.vstoragedomain | nova | enabled | up | 2019-11-18T12:33:43.500922 | - | False |
+--------------------------------------+------------------+------------------------+----------+---------+-------+----------------------------+-----------------+-------------+
nova=# select uuid,deleted_at from compute_nodes;
uuid | deleted_at
--------------------------------------+----------------------------
13c1fbd5-fbc1-4301-8a6e-9d50bde6826f |
# nova service-delete 7d7f3dc6-da81-41b4-be30-c2cc451b560a <-- this is nova-consoleauth service
nova=# select uuid,deleted_at from compute_nodes;
uuid | deleted_at
--------------------------------------+----------------------------
13c1fbd5-fbc1-4301-8a6e-9d50bde6826f | 2019-11-18 12:19:30.080625 <-- compute_node with the same host is also deleted
Expected result
===============
compute_node is not deleted from db if nova-consoleauth service is removed
Actual result
=============
compute_node is deleted from db if nova-consoleauth service is removed
Environment
===========
Queens, Stein |
When upgrading to Stein, nova-consoleauth service is deprecated and should be removed. However if nova-consoleauth service is located on the same host with nova-compute, corresponding row in compute_nodes table is soft-deleted as well, making nova-compute service report in log:
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager [req-f0255008-c398-406c-bca0-12cdc34fc0b4 - - - - -] Error updating resources for node vzstor1.vstoragedomain.: ResourceProviderCreationFailed: Failed to create resource provider vzstor1.vstoragedomain
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager Traceback (most recent call last):
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7399, in update_available_resource_for_node
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager rt.update_available_resource(context, nodename)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 689, in update_available_resource
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager self._update_available_resource(context, resources)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager return f(*args, **kwargs)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 713, in _update_available_resource
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager self._init_compute_node(context, resources)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 562, in _init_compute_node
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager self._update(context, cn)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 887, in _update
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager inv_data,
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 68, in set_inventory_for_provider
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager parent_provider_uuid=parent_provider_uuid,
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 37, in __run_method
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager return getattr(self.instance, __name)(*args, **kwargs)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 1106, in set_inventory_for_provider
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager parent_provider_uuid=parent_provider_uuid)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 667, in _ensure_resource_provider
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager parent_provider_uuid=parent_provider_uuid)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 66, in wrapper
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager return f(self, *a, **k)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 614, in _create_resource_provider
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager raise exception.ResourceProviderCreationFailed(name=name)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager ResourceProviderCreationFailed: Failed to create resource provider vzstor1.vstoragedomain
Steps to reproduce
==================
# nova service-list
+--------------------------------------+------------------+------------------------+----------+---------+-------+----------------------------+-----------------+-------------+
| Id | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason | Forced down |
+--------------------------------------+------------------+------------------------+----------+---------+-------+----------------------------+-----------------+-------------+
| 57885a23-6bc2-4400-a199-a7e0defe2e00 | nova-conductor | vzstor1.vstoragedomain | internal | enabled | up | 2019-11-18T12:33:36.904174 | - | False |
| 0b08d3ea-1b4a-4076-a51e-fbd892241b9b | nova-scheduler | vzstor1.vstoragedomain | internal | enabled | up | 2019-11-18T12:33:40.111038 | - | False |
| 7d7f3dc6-da81-41b4-be30-c2cc451b560a | nova-consoleauth | vzstor1.vstoragedomain | internal | enabled | up | - | - | False |
| 367a4591-cce5-4b7b-ad7c-69135aa803aa | nova-compute | vzstor1.vstoragedomain | nova | enabled | up | 2019-11-18T12:33:43.500922 | - | False |
+--------------------------------------+------------------+------------------------+----------+---------+-------+----------------------------+-----------------+-------------+
nova=# select uuid,deleted_at from compute_nodes;
uuid | deleted_at
--------------------------------------+----------------------------
13c1fbd5-fbc1-4301-8a6e-9d50bde6826f |
# nova service-delete 7d7f3dc6-da81-41b4-be30-c2cc451b560a <-- this is nova-consoleauth service
nova=# select uuid,deleted_at from compute_nodes;
uuid | deleted_at
--------------------------------------+----------------------------
13c1fbd5-fbc1-4301-8a6e-9d50bde6826f | 2019-11-18 12:19:30.080625 <-- compute_node with the same host is also deleted
Expected result
===============
compute_node is not deleted from db if nova-consoleauth service is removed
Actual result
=============
compute_node is deleted from db if nova-consoleauth service is removed
Environment
===========
Queens, Stein |
|
| 2019-11-18 12:22:24 |
OpenStack Infra |
nova: status |
New |
In Progress |
|
| 2019-11-18 12:24:15 |
Pavel Gluschak |
description |
When upgrading to Stein, nova-consoleauth service is deprecated and should be removed. However if nova-consoleauth service is located on the same host with nova-compute, corresponding row in compute_nodes table is soft-deleted as well, making nova-compute service report in log:
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager [req-f0255008-c398-406c-bca0-12cdc34fc0b4 - - - - -] Error updating resources for node vzstor1.vstoragedomain.: ResourceProviderCreationFailed: Failed to create resource provider vzstor1.vstoragedomain
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager Traceback (most recent call last):
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7399, in update_available_resource_for_node
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager rt.update_available_resource(context, nodename)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 689, in update_available_resource
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager self._update_available_resource(context, resources)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager return f(*args, **kwargs)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 713, in _update_available_resource
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager self._init_compute_node(context, resources)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 562, in _init_compute_node
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager self._update(context, cn)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 887, in _update
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager inv_data,
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 68, in set_inventory_for_provider
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager parent_provider_uuid=parent_provider_uuid,
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 37, in __run_method
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager return getattr(self.instance, __name)(*args, **kwargs)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 1106, in set_inventory_for_provider
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager parent_provider_uuid=parent_provider_uuid)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 667, in _ensure_resource_provider
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager parent_provider_uuid=parent_provider_uuid)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 66, in wrapper
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager return f(self, *a, **k)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 614, in _create_resource_provider
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager raise exception.ResourceProviderCreationFailed(name=name)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager ResourceProviderCreationFailed: Failed to create resource provider vzstor1.vstoragedomain
Steps to reproduce
==================
# nova service-list
+--------------------------------------+------------------+------------------------+----------+---------+-------+----------------------------+-----------------+-------------+
| Id | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason | Forced down |
+--------------------------------------+------------------+------------------------+----------+---------+-------+----------------------------+-----------------+-------------+
| 57885a23-6bc2-4400-a199-a7e0defe2e00 | nova-conductor | vzstor1.vstoragedomain | internal | enabled | up | 2019-11-18T12:33:36.904174 | - | False |
| 0b08d3ea-1b4a-4076-a51e-fbd892241b9b | nova-scheduler | vzstor1.vstoragedomain | internal | enabled | up | 2019-11-18T12:33:40.111038 | - | False |
| 7d7f3dc6-da81-41b4-be30-c2cc451b560a | nova-consoleauth | vzstor1.vstoragedomain | internal | enabled | up | - | - | False |
| 367a4591-cce5-4b7b-ad7c-69135aa803aa | nova-compute | vzstor1.vstoragedomain | nova | enabled | up | 2019-11-18T12:33:43.500922 | - | False |
+--------------------------------------+------------------+------------------------+----------+---------+-------+----------------------------+-----------------+-------------+
nova=# select uuid,deleted_at from compute_nodes;
uuid | deleted_at
--------------------------------------+----------------------------
13c1fbd5-fbc1-4301-8a6e-9d50bde6826f |
# nova service-delete 7d7f3dc6-da81-41b4-be30-c2cc451b560a <-- this is nova-consoleauth service
nova=# select uuid,deleted_at from compute_nodes;
uuid | deleted_at
--------------------------------------+----------------------------
13c1fbd5-fbc1-4301-8a6e-9d50bde6826f | 2019-11-18 12:19:30.080625 <-- compute_node with the same host is also deleted
Expected result
===============
compute_node is not deleted from db if nova-consoleauth service is removed
Actual result
=============
compute_node is deleted from db if nova-consoleauth service is removed
Environment
===========
Queens, Stein |
When upgrading to Stein, nova-consoleauth service is deprecated and should be removed. However if nova-consoleauth service is located on the same host with nova-compute, matching row in compute_nodes table is soft-deleted as well, making nova-compute service report in log because of stale resource provider exists in placement:
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager [req-f0255008-c398-406c-bca0-12cdc34fc0b4 - - - - -] Error updating resources for node vzstor1.vstoragedomain.: ResourceProviderCreationFailed: Failed to create resource provider vzstor1.vstoragedomain
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager Traceback (most recent call last):
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7399, in update_available_resource_for_node
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager rt.update_available_resource(context, nodename)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 689, in update_available_resource
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager self._update_available_resource(context, resources)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager return f(*args, **kwargs)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 713, in _update_available_resource
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager self._init_compute_node(context, resources)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 562, in _init_compute_node
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager self._update(context, cn)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 887, in _update
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager inv_data,
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 68, in set_inventory_for_provider
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager parent_provider_uuid=parent_provider_uuid,
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 37, in __run_method
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager return getattr(self.instance, __name)(*args, **kwargs)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 1106, in set_inventory_for_provider
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager parent_provider_uuid=parent_provider_uuid)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 667, in _ensure_resource_provider
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager parent_provider_uuid=parent_provider_uuid)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 66, in wrapper
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager return f(self, *a, **k)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 614, in _create_resource_provider
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager raise exception.ResourceProviderCreationFailed(name=name)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager ResourceProviderCreationFailed: Failed to create resource provider vzstor1.vstoragedomain
Steps to reproduce
==================
# nova service-list
+--------------------------------------+------------------+------------------------+----------+---------+-------+----------------------------+-----------------+-------------+
| Id | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason | Forced down |
+--------------------------------------+------------------+------------------------+----------+---------+-------+----------------------------+-----------------+-------------+
| 57885a23-6bc2-4400-a199-a7e0defe2e00 | nova-conductor | vzstor1.vstoragedomain | internal | enabled | up | 2019-11-18T12:33:36.904174 | - | False |
| 0b08d3ea-1b4a-4076-a51e-fbd892241b9b | nova-scheduler | vzstor1.vstoragedomain | internal | enabled | up | 2019-11-18T12:33:40.111038 | - | False |
| 7d7f3dc6-da81-41b4-be30-c2cc451b560a | nova-consoleauth | vzstor1.vstoragedomain | internal | enabled | up | - | - | False |
| 367a4591-cce5-4b7b-ad7c-69135aa803aa | nova-compute | vzstor1.vstoragedomain | nova | enabled | up | 2019-11-18T12:33:43.500922 | - | False |
+--------------------------------------+------------------+------------------------+----------+---------+-------+----------------------------+-----------------+-------------+
nova=# select uuid,deleted_at from compute_nodes;
uuid | deleted_at
--------------------------------------+----------------------------
13c1fbd5-fbc1-4301-8a6e-9d50bde6826f |
# nova service-delete 7d7f3dc6-da81-41b4-be30-c2cc451b560a <-- this is nova-consoleauth service
nova=# select uuid,deleted_at from compute_nodes;
uuid | deleted_at
--------------------------------------+----------------------------
13c1fbd5-fbc1-4301-8a6e-9d50bde6826f | 2019-11-18 12:19:30.080625 <-- compute_node with the same host is also deleted
Expected result
===============
compute_node is not deleted from db if nova-consoleauth service is removed
Actual result
=============
compute_node is deleted from db if nova-consoleauth service is removed
Environment
===========
Queens, Stein |
|
| 2019-11-18 12:24:52 |
Pavel Gluschak |
description |
When upgrading to Stein, nova-consoleauth service is deprecated and should be removed. However if nova-consoleauth service is located on the same host with nova-compute, matching row in compute_nodes table is soft-deleted as well, making nova-compute service report in log because of stale resource provider exists in placement:
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager [req-f0255008-c398-406c-bca0-12cdc34fc0b4 - - - - -] Error updating resources for node vzstor1.vstoragedomain.: ResourceProviderCreationFailed: Failed to create resource provider vzstor1.vstoragedomain
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager Traceback (most recent call last):
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7399, in update_available_resource_for_node
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager rt.update_available_resource(context, nodename)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 689, in update_available_resource
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager self._update_available_resource(context, resources)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager return f(*args, **kwargs)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 713, in _update_available_resource
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager self._init_compute_node(context, resources)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 562, in _init_compute_node
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager self._update(context, cn)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 887, in _update
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager inv_data,
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 68, in set_inventory_for_provider
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager parent_provider_uuid=parent_provider_uuid,
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 37, in __run_method
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager return getattr(self.instance, __name)(*args, **kwargs)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 1106, in set_inventory_for_provider
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager parent_provider_uuid=parent_provider_uuid)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 667, in _ensure_resource_provider
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager parent_provider_uuid=parent_provider_uuid)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 66, in wrapper
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager return f(self, *a, **k)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 614, in _create_resource_provider
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager raise exception.ResourceProviderCreationFailed(name=name)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager ResourceProviderCreationFailed: Failed to create resource provider vzstor1.vstoragedomain
Steps to reproduce
==================
# nova service-list
+--------------------------------------+------------------+------------------------+----------+---------+-------+----------------------------+-----------------+-------------+
| Id | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason | Forced down |
+--------------------------------------+------------------+------------------------+----------+---------+-------+----------------------------+-----------------+-------------+
| 57885a23-6bc2-4400-a199-a7e0defe2e00 | nova-conductor | vzstor1.vstoragedomain | internal | enabled | up | 2019-11-18T12:33:36.904174 | - | False |
| 0b08d3ea-1b4a-4076-a51e-fbd892241b9b | nova-scheduler | vzstor1.vstoragedomain | internal | enabled | up | 2019-11-18T12:33:40.111038 | - | False |
| 7d7f3dc6-da81-41b4-be30-c2cc451b560a | nova-consoleauth | vzstor1.vstoragedomain | internal | enabled | up | - | - | False |
| 367a4591-cce5-4b7b-ad7c-69135aa803aa | nova-compute | vzstor1.vstoragedomain | nova | enabled | up | 2019-11-18T12:33:43.500922 | - | False |
+--------------------------------------+------------------+------------------------+----------+---------+-------+----------------------------+-----------------+-------------+
nova=# select uuid,deleted_at from compute_nodes;
uuid | deleted_at
--------------------------------------+----------------------------
13c1fbd5-fbc1-4301-8a6e-9d50bde6826f |
# nova service-delete 7d7f3dc6-da81-41b4-be30-c2cc451b560a <-- this is nova-consoleauth service
nova=# select uuid,deleted_at from compute_nodes;
uuid | deleted_at
--------------------------------------+----------------------------
13c1fbd5-fbc1-4301-8a6e-9d50bde6826f | 2019-11-18 12:19:30.080625 <-- compute_node with the same host is also deleted
Expected result
===============
compute_node is not deleted from db if nova-consoleauth service is removed
Actual result
=============
compute_node is deleted from db if nova-consoleauth service is removed
Environment
===========
Queens, Stein |
When upgrading to Stein, nova-consoleauth service is deprecated and should be removed. However if nova-consoleauth service is located on the same host with nova-compute, matching row in compute_nodes table is soft-deleted as well, making nova-compute service report in log, that stale resource provider exists in placement:
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager [req-f0255008-c398-406c-bca0-12cdc34fc0b4 - - - - -] Error updating resources for node vzstor1.vstoragedomain.: ResourceProviderCreationFailed: Failed to create resource provider vzstor1.vstoragedomain
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager Traceback (most recent call last):
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7399, in update_available_resource_for_node
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager rt.update_available_resource(context, nodename)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 689, in update_available_resource
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager self._update_available_resource(context, resources)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager return f(*args, **kwargs)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 713, in _update_available_resource
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager self._init_compute_node(context, resources)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 562, in _init_compute_node
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager self._update(context, cn)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 887, in _update
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager inv_data,
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 68, in set_inventory_for_provider
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager parent_provider_uuid=parent_provider_uuid,
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 37, in __run_method
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager return getattr(self.instance, __name)(*args, **kwargs)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 1106, in set_inventory_for_provider
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager parent_provider_uuid=parent_provider_uuid)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 667, in _ensure_resource_provider
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager parent_provider_uuid=parent_provider_uuid)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 66, in wrapper
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager return f(self, *a, **k)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 614, in _create_resource_provider
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager raise exception.ResourceProviderCreationFailed(name=name)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager ResourceProviderCreationFailed: Failed to create resource provider vzstor1.vstoragedomain
Steps to reproduce
==================
# nova service-list
+--------------------------------------+------------------+------------------------+----------+---------+-------+----------------------------+-----------------+-------------+
| Id | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason | Forced down |
+--------------------------------------+------------------+------------------------+----------+---------+-------+----------------------------+-----------------+-------------+
| 57885a23-6bc2-4400-a199-a7e0defe2e00 | nova-conductor | vzstor1.vstoragedomain | internal | enabled | up | 2019-11-18T12:33:36.904174 | - | False |
| 0b08d3ea-1b4a-4076-a51e-fbd892241b9b | nova-scheduler | vzstor1.vstoragedomain | internal | enabled | up | 2019-11-18T12:33:40.111038 | - | False |
| 7d7f3dc6-da81-41b4-be30-c2cc451b560a | nova-consoleauth | vzstor1.vstoragedomain | internal | enabled | up | - | - | False |
| 367a4591-cce5-4b7b-ad7c-69135aa803aa | nova-compute | vzstor1.vstoragedomain | nova | enabled | up | 2019-11-18T12:33:43.500922 | - | False |
+--------------------------------------+------------------+------------------------+----------+---------+-------+----------------------------+-----------------+-------------+
nova=# select uuid,deleted_at from compute_nodes;
uuid | deleted_at
--------------------------------------+----------------------------
13c1fbd5-fbc1-4301-8a6e-9d50bde6826f |
# nova service-delete 7d7f3dc6-da81-41b4-be30-c2cc451b560a <-- this is nova-consoleauth service
nova=# select uuid,deleted_at from compute_nodes;
uuid | deleted_at
--------------------------------------+----------------------------
13c1fbd5-fbc1-4301-8a6e-9d50bde6826f | 2019-11-18 12:19:30.080625 <-- compute_node with the same host is also deleted
Expected result
===============
compute_node is not deleted from db if nova-consoleauth service is removed
Actual result
=============
compute_node is deleted from db if nova-consoleauth service is removed
Environment
===========
Queens, Stein |
|
| 2019-11-19 15:26:21 |
Matt Riedemann |
nova: importance |
Undecided |
Medium |
|
| 2019-11-19 15:26:28 |
Matt Riedemann |
tags |
|
db |
|
| 2019-11-19 15:26:37 |
Matt Riedemann |
nominated for series |
|
nova/train |
|
| 2019-11-19 15:26:37 |
Matt Riedemann |
bug task added |
|
nova/train |
|
| 2019-11-19 15:26:37 |
Matt Riedemann |
nominated for series |
|
nova/rocky |
|
| 2019-11-19 15:26:37 |
Matt Riedemann |
bug task added |
|
nova/rocky |
|
| 2019-11-19 15:26:37 |
Matt Riedemann |
nominated for series |
|
nova/stein |
|
| 2019-11-19 15:26:37 |
Matt Riedemann |
bug task added |
|
nova/stein |
|
| 2019-11-19 15:26:45 |
Matt Riedemann |
nova/rocky: status |
New |
Confirmed |
|
| 2019-11-19 15:26:50 |
Matt Riedemann |
nova/stein: status |
New |
Confirmed |
|
| 2019-11-19 15:26:54 |
Matt Riedemann |
nova/stein: importance |
Undecided |
Medium |
|
| 2019-11-19 15:27:02 |
Matt Riedemann |
nova/train: importance |
Undecided |
Medium |
|
| 2019-11-19 15:27:04 |
Matt Riedemann |
nova/rocky: importance |
Undecided |
Medium |
|
| 2019-11-19 15:27:12 |
Matt Riedemann |
nova/train: status |
New |
Confirmed |
|
| 2019-11-20 08:46:34 |
OpenStack Infra |
nova/train: status |
Confirmed |
In Progress |
|
| 2019-11-20 08:46:34 |
OpenStack Infra |
nova/train: assignee |
|
Sylvain Bauza (sylvain-bauza) |
|
| 2019-11-20 10:40:03 |
OpenStack Infra |
nova: status |
In Progress |
Fix Released |
|
| 2019-11-20 13:37:30 |
OpenStack Infra |
nova/train: assignee |
Sylvain Bauza (sylvain-bauza) |
Matt Riedemann (mriedem) |
|
| 2019-11-20 13:49:28 |
Matt Riedemann |
nova/train: assignee |
Matt Riedemann (mriedem) |
Sylvain Bauza (sylvain-bauza) |
|
| 2019-11-21 00:42:54 |
OpenStack Infra |
nova/train: status |
In Progress |
Fix Committed |
|
| 2019-11-21 09:45:31 |
OpenStack Infra |
nova/stein: status |
Confirmed |
In Progress |
|
| 2019-11-21 09:45:31 |
OpenStack Infra |
nova/stein: assignee |
|
Sylvain Bauza (sylvain-bauza) |
|
| 2019-11-21 09:49:29 |
OpenStack Infra |
nova/rocky: status |
Confirmed |
In Progress |
|
| 2019-11-21 09:49:29 |
OpenStack Infra |
nova/rocky: assignee |
|
Sylvain Bauza (sylvain-bauza) |
|
| 2019-11-22 06:39:06 |
OpenStack Infra |
nova/stein: status |
In Progress |
Fix Committed |
|
| 2019-11-22 19:53:02 |
OpenStack Infra |
nova/rocky: status |
In Progress |
Fix Committed |
|
| 2019-11-25 19:50:27 |
OpenStack Infra |
tags |
db |
db in-stable-queens |
|
