neutron

Neutron OVS agent fails to start when neutron-server is not available

Bug #1694524 reported by Ihar Hrachyshka
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
neutron
Invalid
High
Kevin Benton
neutron pike-2 "p-2"
tripleo
Fix Released
Critical
Unassigned
tripleo pike-2 "pike-2"

Bug Description

2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp [req-34115ed3-3043-4fcb-ba3f-ab0e4eb0e83c - - - - -] Agent main thread died of an exception
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp Traceback (most recent call last):
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ovs_ryuapp.py", line 40, in agent_main_wrapper
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp ovs_agent.main(bridge_classes)
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2166, in main
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp agent = OVSNeutronAgent(bridge_classes, cfg.CONF)
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 180, in __init__
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp self.setup_rpc()
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 153, in wrapper
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp return f(*args, **kwargs)
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 362, in setup_rpc
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp self.plugin_rpc = OVSPluginApi(topics.PLUGIN)
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp File "/usr/lib/python2.7/site-packages/neutron/agent/rpc.py", line 182, in __init__
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp self.remote_resource_cache = create_cache_for_l2_agent()
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp File "/usr/lib/python2.7/site-packages/neutron/agent/rpc.py", line 174, in create_cache_for_l2_agent
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp rcache.bulk_flood_cache()
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp File "/usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py", line 55, in bulk_flood_cache
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp for resource in puller.bulk_pull(context, rtype):
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp File "/usr/lib/python2.7/site-packages/oslo_log/helpers.py", line 48, in wrapper
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp return method(*args, **kwargs)
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp File "/usr/lib/python2.7/site-packages/neutron/api/rpc/handlers/resources_rpc.py", line 109, in bulk_pull
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp version=resource_type_cls.VERSION, filter_kwargs=filter_kwargs)
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp File "/usr/lib/python2.7/site-packages/neutron/common/rpc.py", line 174, in call
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp time.sleep(wait)
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp self.force_reraise()
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp six.reraise(self.type_, self.value, self.tb)
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp File "/usr/lib/python2.7/site-packages/neutron/common/rpc.py", line 151, in call
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp return self._original_context.call(ctxt, method, **kwargs)
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in call
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp retry=self.retry)
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 123, in _send
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp timeout=timeout, retry=retry)
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 505, in send
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp retry=retry)
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 494, in _send
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp result = self._waiter.wait(msg_id, timeout)
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 386, in wait
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp message = self.waiters.get(msg_id, timeout=timeout)
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 275, in get
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp 'to message ID %s' % msg_id)
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp MessagingTimeout: Timed out waiting for a reply to message ID c79afd6fe556408bbaca568e599feada
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp
2017-05-30 18:58:02.051 27929 INFO oslo_rootwrap.client [-] Stopping rootwrap daemon process with pid=28388

https://logs.rdoproject.org/weirdo-generic-puppet-openstack-scenario001/1754/weirdo-project/logs/neutron/openvswitch-agent.txt.gz

This is probably because of push-notifications work.

Tags: ovs
Ihar Hrachyshka (ihar-hrachyshka)
Changed in neutron:
importance: Undecided → High
tags: added: ovs
Changed in neutron:
status: New → Confirmed
milestone: none → pike-2
Kevin Benton (kevinbenton)
Changed in neutron:
assignee: nobody → Kevin Benton (kevinbenton)
Emilien Macchi (emilienm)
Changed in tripleo:
status: New → Triaged
milestone: none → pike-2
importance: Undecided → Critical
tags: added: alert ci promotion-blocker
Revision history for this message
Alan Pevec (apevec) wrote :

Is this duplicate of bug 1694505 or followup work after https://review.openstack.org/#/q/Ic86b5eeae25a916c3c51f21e6820f5b0212dd5f8,n,z is merged?

Revision history for this message
Ihar Hrachyshka (ihar-hrachyshka) wrote :

Alan, it's a separate bug. It's just that without bug 1694505 TripleO gate may have been unaffected.

Emilien Macchi (emilienm)
Changed in tripleo:
status: Triaged → Fix Released
tags: removed: alert ci promotion-blocker
Rodolfo Alonso (rodolfo-alonso-hernandez)
Changed in neutron:
status: Confirmed → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.