OpenStack Compute (nova)

An unknown error has occurred when trying to get the list of nodes from the Ironic inventory. Error: StrictVersion instance has no attribute 'version'

Bug #1793766 reported by Matt Riedemann on 2018-09-21

This bug affects 3 people

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Compute (nova)	Confirmed	Low	Unassigned

Bug Description

Seen in Ironic CI jobs since Rocky when version discovery was added to the ironic virt driver in nova-compute:

http://logs.openstack.org/27/602127/2/check/ironic-tempest-dsvm-ipa-wholedisk-bios-agent_ipmitool-tinyipa/4238d0f/controller/logs/screen-n-cpu.txt.gz?level=TRACE#_Sep_20_21_52_53_685133

Sep 20 21:52:03.587629 ubuntu-xenial-inap-mtl01-0002177186 nova-compute[14241]: ERROR oslo_service.periodic_task Traceback (most recent call last):
Sep 20 21:52:03.587816 ubuntu-xenial-inap-mtl01-0002177186 nova-compute[14241]: ERROR oslo_service.periodic_task File "/usr/local/lib/python2.7/dist-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks
Sep 20 21:52:03.588019 ubuntu-xenial-inap-mtl01-0002177186 nova-compute[14241]: ERROR oslo_service.periodic_task task(self, context)
Sep 20 21:52:03.588182 ubuntu-xenial-inap-mtl01-0002177186 nova-compute[14241]: ERROR oslo_service.periodic_task File "/opt/stack/nova/nova/compute/manager.py", line 7462, in _sync_power_states
Sep 20 21:52:03.588344 ubuntu-xenial-inap-mtl01-0002177186 nova-compute[14241]: ERROR oslo_service.periodic_task num_vm_instances = self.driver.get_num_instances()
Sep 20 21:52:03.588517 ubuntu-xenial-inap-mtl01-0002177186 nova-compute[14241]: ERROR oslo_service.periodic_task File "/opt/stack/nova/nova/virt/driver.py", line 183, in get_num_instances
Sep 20 21:52:03.588714 ubuntu-xenial-inap-mtl01-0002177186 nova-compute[14241]: ERROR oslo_service.periodic_task return len(self.list_instances())
Sep 20 21:52:03.588885 ubuntu-xenial-inap-mtl01-0002177186 nova-compute[14241]: ERROR oslo_service.periodic_task File "/opt/stack/nova/nova/virt/ironic/driver.py", line 624, in list_instances
Sep 20 21:52:03.589071 ubuntu-xenial-inap-mtl01-0002177186 nova-compute[14241]: ERROR oslo_service.periodic_task fields=['instance_uuid'], limit=0)
Sep 20 21:52:03.589263 ubuntu-xenial-inap-mtl01-0002177186 nova-compute[14241]: ERROR oslo_service.periodic_task File "/opt/stack/nova/nova/virt/ironic/driver.py", line 611, in _get_node_list
Sep 20 21:52:03.589447 ubuntu-xenial-inap-mtl01-0002177186 nova-compute[14241]: ERROR oslo_service.periodic_task raise exception.VirtDriverNotReady()
Sep 20 21:52:03.589602 ubuntu-xenial-inap-mtl01-0002177186 nova-compute[14241]: ERROR oslo_service.periodic_task VirtDriverNotReady: Virt driver is not ready.
Sep 20 21:52:03.589770 ubuntu-xenial-inap-mtl01-0002177186 nova-compute[14241]: ERROR oslo_service.periodic_task
Sep 20 21:52:53.625466 ubuntu-xenial-inap-mtl01-0002177186 nova-compute[14241]: ERROR nova.compute.manager [None req-2339900e-55df-468b-95ff-b904d73d5728 None None] No compute node record for host ubuntu-xenial-inap-mtl01-0002177186: ComputeHostNotFound_Remote: Compute host ubuntu-xenial-inap-mtl01-0002177186 could not be found.
Sep 20 21:52:53.685133 ubuntu-xenial-inap-mtl01-0002177186 nova-compute[14241]: ERROR nova.virt.ironic.driver [None req-2339900e-55df-468b-95ff-b904d73d5728 None None] An unknown error has occurred when trying to get the list of nodes from the Ironic inventory. Error: StrictVersion instance has no attribute 'version'
Sep 20 21:52:53.685496 ubuntu-xenial-inap-mtl01-0002177186 nova-compute[14241]: WARNING nova.compute.manager [None req-2339900e-55df-468b-95ff-b904d73d5728 None None] Virt driver is not ready.: VirtDriverNotReady: Virt driver is not ready.

The virt driver not ready error is normal/expected while ironic is coming up after nova-compute, but the StrictVersion thing should be handled probably.

Tags:

Revision history for this message

David Hill (david-hill-ubisoft) wrote on 2019-05-18:

Download full text (7.8 KiB)

I do have this error looping every minute or so.

2019-05-17 21:33:39.463 8 DEBUG ironicclient.common.http [req-f1d152cb-d19c-4946-b0b0-baaf2f9e4804 - - - - -] No version header in response, requesting from server negotiate_version /usr/lib/python3.6/site-packages/ironicclient/common/http.py:157
2019-05-17 21:33:39.473 8 ERROR nova.virt.ironic.driver [req-f1d152cb-d19c-4946-b0b0-baaf2f9e4804 - - - - -] An unknown error has occurred when trying to get the list of nodes from the Ironic inventory. Error: 'StrictVersion' object has no attribute 'version'
2019-05-17 21:33:39.473 8 WARNING nova.compute.manager [req-f1d152cb-d19c-4946-b0b0-baaf2f9e4804 - - - - -] Virt driver is not ready.: nova.exception.VirtDriverNotReady: Virt driver is not ready.
2019-05-17 21:33:40.473 8 DEBUG oslo_service.periodic_task [req-f1d152cb-d19c-4946-b0b0-baaf2f9e4804 - - - - -] Running periodic task ComputeManager._heal_instance_info_cache run_periodic_tasks /usr/lib/python3.6/site-packages/oslo_service/periodic_task.py:217
2019-05-17 21:33:40.473 8 DEBUG nova.compute.manager [req-f1d152cb-d19c-4946-b0b0-baaf2f9e4804 - - - - -] Starting heal instance info cache _heal_instance_info_cache /usr/lib/python3.6/site-packages/nova/compute/manager.py:7247
2019-05-17 21:33:40.474 8 DEBUG nova.compute.manager [req-f1d152cb-d19c-4946-b0b0-baaf2f9e4804 - - - - -] Rebuilding the list of instances to heal _heal_instance_info_cache /usr/lib/python3.6/site-packages/nova/compute/manager.py:7251
2019-05-17 21:33:40.487 8 DEBUG nova.compute.manager [req-f1d152cb-d19c-4946-b0b0-baaf2f9e4804 - - - - -] Didn't find any instances for network info cache update. _heal_instance_info_cache /usr/lib/python3.6/site-packages/nova/compute/manager.py:7333
2019-05-17 21:33:40.487 8 DEBUG oslo_service.periodic_task [req-f1d152cb-d19c-4946-b0b0-baaf2f9e4804 - - - - -] Running periodic task ComputeManager._poll_unconfirmed_resizes run_periodic_tasks /usr/lib/python3.6/site-packages/oslo_service/periodic_task.py:217
2019-05-17 21:33:40.488 8 DEBUG oslo_service.periodic_task [req-f1d152cb-d19c-4946-b0b0-baaf2f9e4804 - - - - -] Running periodic task ComputeManager._reclaim_queued_deletes run_periodic_tasks /usr/lib/python3.6/site-packages/oslo_service/periodic_task.py:217
2019-05-17 21:33:40.488 8 DEBUG nova.compute.manager [req-f1d152cb-d19c-4946-b0b0-baaf2f9e4804 - - - - -] CONF.reclaim_instance_interval <= 0, skipping... _reclaim_queued_deletes /usr/lib/python3.6/site-packages/nova/compute/manager.py:7946
2019-05-17 21:33:41.422 8 DEBUG oslo_service.periodic_task [req-f1d152cb-d19c-4946-b0b0-baaf2f9e4804 - - - - -] Running periodic task ComputeManager._poll_rebooting_instances run_periodic_tasks /usr/lib/python3.6/site-packages/oslo_service/periodic_task.py:217
2019-05-17 21:33:41.423 8 DEBUG oslo_service.periodic_task [req-f1d152cb-d19c-4946-b0b0-baaf2f9e4804 - - - - -] Running periodic task ComputeManager._poll_rescued_instances run_periodic_tasks /usr/lib/python3.6/site-packages/oslo_service/periodic_task.py:217
2019-05-17 21:33:43.422 8 DEBUG oslo_service.periodic_task [req-f1d152cb-d19c-4946-b0b0-baaf2f9e4804 - - - - -] Running periodic task ComputeManager._instance_usage_audit run_periodic_tasks /...

I do have this error looping every minute or so.

and some error in mistral when trying to introspect overcloud nodes:

2019-05-17 21:21:09.091 8 INFO workflow_trace [req-d462b3f8-acde-4224-9977-877545feed2e 022b15c9e0b14450b55c71d23bce3253 561ad084af1941b5931dc317db563e12 - default default] Task 'cell_v2_discover_hosts' (ff79c16c-b48e-495a-b2cc-38d9e92df08f) [RUNNING -> ERROR, msg=Failure caused by error in tasks: wait_for_nova_resources

wait_for_nova_resources [task_ex_id=ec65be73-4f36-425c-9bd4-6684456c3b32] -> One or more actions had failed.
    [action_ex_id=15a7c7e2-558b-4548-b73b-a78d4b7be594, idx=0]: The action raised an exception [action_ex_id=15a7c7e2-558b-4548-b73b-a78d4b7be594, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{'client_method_name': 'hypervisors.find'}', params='{'hypervisor_hostname': '005d8935-9991-40bc-95be-dd0eb0e3830e'}']
 NovaAction.hypervisors.find failed: No Hypervisor matching {'hypervisor_hostname': '005d8935-9991-40bc-95be-dd0eb0e3830e'}. (HTTP 404)
    [action_ex_id=266a8a2b-4218-441d-902c-3f7da984d912, idx=1]: The action raised an exception [action_ex_id=266a8a2b-4218-441d-902c-3f7da984d912, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{'client_method_name': 'hypervisors.find'}', params='{'hypervisor_hostname': '4b0e84d7-7810-4a42-81d5-6b14579362c8'}']
 NovaAction.hypervisors.find failed: No Hypervisor matching {'hypervisor_hostname': '4b0e84d7-7810-4a42-81d5-6b14579362c8'}. (HTTP 404)
    [action_ex_id=4f6e7bea-55ef-4b92-8539-76f2d2decfa4, idx=2]: The action raised an exception [action_ex_id=4f6e7bea-55ef-4b92-8539-76f2d2decfa4, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{'client_method_name': 'hypervisors.find'}', params='{'hypervisor_hostname': 'c586ead4-07b0-4e3c-983e-ad1ece3ffc1e'}']
 NovaAction.hypervisors.find failed: No Hypervisor matching {'hypervisor_hostname': 'c586ead4-07b0-4e3c-983e-ad1ece3ffc1e'}. (HTTP 404)
    [action_ex_id=64e5320d-fb64-49b0-8cb0-eb62f04f4170, idx=3]: The action raised an exception [action_ex_id=64e5320d-fb64-49b0-8cb0-eb62f04f4170, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{'client_method_name': 'hypervisors.find'}', params='{'hypervisor_hostname': '6bf2c92f-8eaf-4596-982c-87cef89337b7'}']
 NovaAction.hypervisors.find failed: No Hypervisor matching {'hypervisor_hostname': '6bf2c92f-8eaf-4596-982c-87cef89337b7'}. (HTTP 404)
    [action_ex_id=83e97b9c-2775-4752-962d-d19cc728ff0c, idx=4]: The action raised an exception [action_ex_id=83e97b9c-2775-4752-962d-d19cc728ff0c, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{'client_method_name': 'hypervisors.find'}', params='{'hypervisor_hostname': '6f539c86-3905-4014-8973-f515343c6c4b'}']
 NovaAction.hypervisors.find failed: No Hypervisor matching {'hypervisor_hostname': '6f539c86-3905-4014-8973-f515343c6c4b'}. (HTTP 404)
    [action_ex_id=b7881d3d-78c7-4898-a8cd-36a8b25112a5, idx=5]: The action raised an exception [action_ex_id=b7881d3d-78c7-4898-a8cd-36a8b25112a5, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{'client_method_name': 'hypervisors.find'}', params='{'hypervisor_hostname': '82cdb49e-a248-4221-8206-62813b6deb26'}']
 NovaAction.hypervisors.find failed: No Hypervisor matching {'hypervisor_hostname': '82cdb49e-a248-4221-8206-62813b6deb26'}. (HTTP 404)
    [action_ex_id=c0aca431-e692-45e0-98ef-2f5fcf8fa623, idx=6]: The action raised an exception [action_ex_id=c0aca431-e692-45e0-98ef-2f5fcf8fa623, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{'client_method_name': 'hypervisors.find'}', params='{'hypervisor_hostname': 'd1e608f4-9e95-4d85-8d8b-ad8ee73c01d5'}']
 NovaAction.hypervisors.find failed: No Hypervisor matching {'hypervisor_hostname': 'd1e608f4-9e95-4d85-8d8b-ad8ee73c01d5'}. (HTTP 404)
] (execution_id=e0b1ab58-9b03-4088-a008-a92614dcd892)
(END)

Revision history for this message

Rajini Karthik (rajini-karthik) wrote on 2019-06-18:

I see this error too

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.