Juniper Openstack

Mainline 2333:Nova runtime error while running tests in parallel

Bug #1371592 reported by Sandip Dey
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R2.0
Fix Committed
High
Hampapur Ajay
Trunk
Fix Committed
High
Hampapur Ajay

Bug Description

Main line : 2333 Centos 64

Getting the following error while running sanity in parallel

This could be affecting the sanity result.

logs saved at /cs-shared/bugs/<bug-id> on any blr shell server (ex nodeb6)

Logs
===
2014-09-19 04:41:00.165 4355 TRACE nova.compute.manager [instance: fcae8094-4bf6-4401-b081-0d3fd6221f7f] chunk = self.read(sz - have)
2014-09-19 04:41:00.165 4355 TRACE nova.compute.manager [instance: fcae8094-4bf6-4401-b081-0d3fd6221f7f] File "/usr/lib64/python2.6/site-packages/thrift/transport/TSocket.py", line 103, in read
2014-09-19 04:41:00.165 4355 TRACE nova.compute.manager [instance: fcae8094-4bf6-4401-b081-0d3fd6221f7f] buff = self.handle.recv(sz)
2014-09-19 04:41:00.165 4355 TRACE nova.compute.manager [instance: fcae8094-4bf6-4401-b081-0d3fd6221f7f] File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 264, in recv
2014-09-19 04:41:00.165 4355 TRACE nova.compute.manager [instance: fcae8094-4bf6-4401-b081-0d3fd6221f7f] timeout_exc=socket.timeout("timed out"))
2014-09-19 04:41:00.165 4355 TRACE nova.compute.manager [instance: fcae8094-4bf6-4401-b081-0d3fd6221f7f] File "/usr/lib/python2.6/site-packages/eventlet/hubs/__init__.py", line 151, in trampoline
2014-09-19 04:41:00.165 4355 TRACE nova.compute.manager [instance: fcae8094-4bf6-4401-b081-0d3fd6221f7f] listener = hub.add(hub.READ, fileno, current.switch)
2014-09-19 04:41:00.165 4355 TRACE nova.compute.manager [instance: fcae8094-4bf6-4401-b081-0d3fd6221f7f] File "/usr/lib/python2.6/site-packages/eventlet/hubs/epolls.py", line 48, in add
2014-09-19 04:41:00.165 4355 TRACE nova.compute.manager [instance: fcae8094-4bf6-4401-b081-0d3fd6221f7f] listener = BaseHub.add(self, evtype, fileno, cb)
2014-09-19 04:41:00.165 4355 TRACE nova.compute.manager [instance: fcae8094-4bf6-4401-b081-0d3fd6221f7f] File "/usr/lib/python2.6/site-packages/eventlet/hubs/hub.py", line 126, in add
2014-09-19 04:41:00.165 4355 TRACE nova.compute.manager [instance: fcae8094-4bf6-4401-b081-0d3fd6221f7f] evtype, fileno, evtype))
2014-09-19 04:41:00.165 4355 TRACE nova.compute.manager [instance: fcae8094-4bf6-4401-b081-0d3fd6221f7f] RuntimeError: Second simultaneous read on fileno 9 detected. Unless you really know what you're doing, make sure that only one greenthread can read any particular socket. Consider using a pools.Pool. If you do know what you're doing and want to disable this error, call eventlet.debug.hub_prevent_multiple_readers(False)
2014-09-19 04:41:00.165 4355 TRACE nova.compute.manager [instance: fcae8094-4bf6-4401-b081-0d3fd6221f7f]
2014-09-19 04:41:02.193 4355 ERROR nova.virt.libvirt.driver [-] [instance: fcae8094-4bf6-4401-b081-0d3fd6221f7f] During wait destroy, instance disappeared.
2014-09-19 04:41:02.548 4355 ERROR nova.virt.driver [-] Exception dispatching event <nova.virt.event.LifecycleEvent object at 0x49a0c90>: Info cache for instance d5e30215-51ad-4f11-b641-afb9c4a873e3 could not be found.

See original description
Sandip Dey (sandipd)
no longer affects: toontowninfinite
Nagabhushana R (bhushana)
tags: added: nova openstack
Revision history for this message
Sandip Dey (sandipd) wrote :
Download full text (7.9 KiB)

Someone should look into this bug.VM spawn failing due to this

2014-09-23 05:15:09.974 5897 ERROR nova.compute.manager [req-5a9a7595-e37b-44fb-aa67-85a16fb90b57 4aa1c9b2265f427fa3a56dbd25706bcf fe88547b4886406aac0330e855a72a15] [instance: 3d9284a7-4144-4ad6-b1cb-afd5c38bd44e] Instance failed to spawn
2014-09-23 05:15:09.974 5897 TRACE nova.compute.manager [instance: 3d9284a7-4144-4ad6-b1cb-afd5c38bd44e] Traceback (most recent call last):
2014-09-23 05:15:09.974 5897 TRACE nova.compute.manager [instance: 3d9284a7-4144-4ad6-b1cb-afd5c38bd44e] File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1410, in _spawn
2014-09-23 05:15:09.974 5897 TRACE nova.compute.manager [instance: 3d9284a7-4144-4ad6-b1cb-afd5c38bd44e] block_device_info)
2014-09-23 05:15:09.974 5897 TRACE nova.compute.manager [instance: 3d9284a7-4144-4ad6-b1cb-afd5c38bd44e] File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 2071, in spawn
2014-09-23 05:15:09.974 5897 TRACE nova.compute.manager [instance: 3d9284a7-4144-4ad6-b1cb-afd5c38bd44e] block_device_info, context=context)
2014-09-23 05:15:09.974 5897 TRACE nova.compute.manager [instance: 3d9284a7-4144-4ad6-b1cb-afd5c38bd44e] File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 3222, in _create_domain_and_network
2014-09-23 05:15:09.974 5897 TRACE nova.compute.manager [instance: 3d9284a7-4144-4ad6-b1cb-afd5c38bd44e] self.plug_vifs(instance, network_info)
2014-09-23 05:15:09.974 5897 TRACE nova.compute.manager [instance: 3d9284a7-4144-4ad6-b1cb-afd5c38bd44e] File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 748, in plug_vifs
2014-09-23 05:15:09.974 5897 TRACE nova.compute.manager [instance: 3d9284a7-4144-4ad6-b1cb-afd5c38bd44e] self.vif_driver.plug(instance, vif)
2014-09-23 05:15:09.974 5897 TRACE nova.compute.manager [instance: 3d9284a7-4144-4ad6-b1cb-afd5c38bd44e] File "/usr/lib/python2.6/site-packages/nova_contrail_vif/contrailvif.py", line 128, in plug
2014-09-23 05:15:09.974 5897 TRACE nova.compute.manager [instance: 3d9284a7-4144-4ad6-b1cb-afd5c38bd44e] **kwargs)
2014-09-23 05:15:09.974 5897 TRACE nova.compute.manager [instance: 3d9284a7-4144-4ad6-b1cb-afd5c38bd44e] File "/usr/lib/python2.6/site-packages/contrail_vrouter_api/vrouter_api.py", line 123, in add_port
2014-09-23 05:15:09.974 5897 TRACE nova.compute.manager [instance: 3d9284a7-4144-4ad6-b1cb-afd5c38bd44e] result = self._client.AddPort([data])
2014-09-23 05:15:09.974 5897 TRACE nova.compute.manager [instance: 3d9284a7-4144-4ad6-b1cb-afd5c38bd44e] File "/usr/lib/python2.6/site-packages/contrail_vrouter_api/gen_py/instance_service/InstanceService.py", line 150, in AddPort
2014-09-23 05:15:09.974 5897 TRACE nova.compute.manager [instance: 3d9284a7-4144-4ad6-b1cb-afd5c38bd44e] result = self._client.AddPort([data])
2014-09-23 05:15:09.974 5897 TRACE nova.compute.manager [instance: 3d9284a7-4144-4ad6-b1cb-afd5c38bd44e] File "/usr/lib/python2.6/site-packages/contrail_vrouter_api/gen_py/instance_service/InstanceService.py", line 150, in AddPort
2014-09-23 05:15:09.974 5897 TRACE nova.compute.manager [instance: 3d9284a7-4144-4ad6-b1cb-afd5c38bd44...

Read more...

Ashish Ranjan (aranjan-n)
Changed in juniperopenstack:
assignee: nobody → Hampapur Ajay (hajay)
Nagabhushana R (bhushana)
no longer affects: juniperopenstack/r1.1
tags: added: blocker
Nagabhushana R (bhushana)
summary: - Mainline 2233:Nova runtime error while running tests in parallel
+ Mainline 2333:Nova runtime error while running tests in parallel
Revision history for this message
Hampapur Ajay (hajay) wrote :

This backport to neutronclient should fix it
 https://github.com/Juniper/python-neutronclient/pull/3

Vedamurthy Joshi (vedujoshi)
description: updated
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.