I saw several rpc call error from powervc neutron agent:
2014-11-03 17:18:26.687 3600 INFO powervc.neutron.api.powervc_rpc [req-22c031c8-3bb5-4692-a769-c50dc5529c28 None] Neutron Agent RPC: get_pvc_port_uuid:
2014-11-03 17:18:26.687 3600 INFO powervc.neutron.api.powervc_rpc [req-22c031c8-3bb5-4692-a769-c50dc5529c28 None] - local_port_id: 61245c6b-dbf7-405f-8e22-0991d31cdbf3
2014-11-03 17:18:26.691 3600 INFO powervc.neutron.api.powervc_rpc [req-22c031c8-3bb5-4692-a769-c50dc5529c28 None] - pvc_port_id: None
2014-11-03 17:18:26.730 3600 INFO powervc.neutron.api.powervc_rpc [req-732b37f6-9b8a-454e-9637-fc389c554d4a None] Neutron Agent RPC: get_pvc_port_uuid:
2014-11-03 17:18:26.730 3600 INFO powervc.neutron.api.powervc_rpc [req-732b37f6-9b8a-454e-9637-fc389c554d4a None] - local_port_id: a244d8c8-8ec7-4d27-8fe7-47063c74fa86
2014-11-03 17:18:26.735 3600 INFO powervc.neutron.api.powervc_rpc [req-732b37f6-9b8a-454e-9637-fc389c554d4a None] - pvc_port_id: None
2014-11-03 17:18:26.744 3600 INFO powervc.neutron.api.powervc_rpc [req-146283a8-5d7b-4796-8e14-172b81447f2a None] Neutron Agent RPC: get_pvc_port_uuid:
2014-11-03 17:18:26.745 3600 INFO powervc.neutron.api.powervc_rpc [req-146283a8-5d7b-4796-8e14-172b81447f2a None] - local_port_id: 48a4dec2-74e3-47d4-a227-769b7c1cc816
2014-11-03 17:18:26.749 3600 INFO powervc.neutron.api.powervc_rpc [req-5189c2ac-0a76-4ac5-82f0-6c58ae8e38e5 None] Neutron Agent RPC: get_pvc_port_uuid:
2014-11-03 17:18:26.750 3600 INFO powervc.neutron.api.powervc_rpc [req-5189c2ac-0a76-4ac5-82f0-6c58ae8e38e5 None] - local_port_id: ab3f8a9f-03b0-4473-845a-1f997015d71d
2014-11-03 17:18:26.755 3600 INFO powervc.neutron.api.powervc_rpc [req-5189c2ac-0a76-4ac5-82f0-6c58ae8e38e5 None] - pvc_port_id: None
2014-11-03 17:18:26.757 3600 INFO powervc.neutron.api.powervc_rpc [req-5189c2ac-0a76-4ac5-82f0-6c58ae8e38e5 None] - pvc_port_id: None
2014-11-03 17:18:26.757 3600 ERROR oslo.messaging.rpc.dispatcher [-] Exception during message handling: _oslo_messaging_localcontext_a5bb97ccf6064400ac9107698d48b016
2014-11-03 17:18:26.757 3600 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2014-11-03 17:18:26.757 3600 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
2014-11-03 17:18:26.757 3600 TRACE oslo.messaging.rpc.dispatcher incoming.message))
2014-11-03 17:18:26.757 3600 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 179, in _dispatch
2014-11-03 17:18:26.757 3600 TRACE oslo.messaging.rpc.dispatcher localcontext.clear_local_context()
2014-11-03 17:18:26.757 3600 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/oslo/messaging/localcontext.py", line 55, in clear_local_context
2014-11-03 17:18:26.757 3600 TRACE oslo.messaging.rpc.dispatcher delattr(_STORE, _KEY)
2014-11-03 17:18:26.757 3600 TRACE oslo.messaging.rpc.dispatcher AttributeError: _oslo_messaging_localcontext_a5bb97ccf6064400ac9107698d48b016
2014-11-03 17:18:26.757 3600 TRACE oslo.messaging.rpc.dispatcher
2014-11-03 17:18:26.758 3600 ERROR oslo.messaging._drivers.common [-] Returning exception _oslo_messaging_localcontext_a5bb97ccf6064400ac9107698d48b016 to caller
There is some known community bugs for such error:
https://bugs.launchpad.net/mistral/+bug/1337705
https://bugs.launchpad.net/oslo.messaging/+bug/1288878
people said:
monkey_patch thread is needed when using oslo.message. the behavior is unknown and in risk if thread is set to False.
and I saw they are fixing them using:
eventlet.monkey_patch(socket=True, select=True)
From other neutron agent/service, they are directly using
event.monkey_patch(),
but for current powervc neutron agent:
import eventlet
eventlet.patcher.monkey_patch(os=False, thread=False)
It's explicitly un-patch the thread module... I think that's wrong from oslo.messaging rpc call, we need to monkey_patch the eventlet like neutron..
Reviewed: https:/ /review. openstack. org/132756 /git.openstack. org/cgit/ stackforge/ powervc- driver/ commit/ ?id=64139a45b79 f1a4bf70ee8c5cd 1e84c13fec0a15
Committed: https:/
Submitter: Jenkins
Branch: master
commit 64139a45b79f1a4 bf70ee8c5cd1e84 c13fec0a15
Author: terryyao <email address hidden>
Date: Wed Nov 5 14:54:25 2014 +0800
fix eventlet monkey_patch for rpc call
In PowerVC neutron agent, currently, it's monkey_patch eventlet with
setting thread=False, that will result in oslo.messaging RPC call failed
in concurrent execution. Just refer neutron code call monkey_patch() to
patch all modules
Closes-Bug: #1389468 1d3d2e125c8d088 b4d6ef1b40e
Change-Id: Ia669c37a555bec