Need retry mechanism when connecting to etcdserver

2017-04-18 16:41:22.522 33 DEBUG etcd.client [-] Writing None to key /networking-vpp/state ttl=None dir=True append=False write /usr/lib/python2.7/site-packages/etcd/client.py:472
2017-04-18 16:41:29.524 33 DEBUG oslo_concurrency.lockutils [-] Lock "manager" released by "neutron.manager._create_instance" :: held 12.088s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2017-04-18 16:41:29.525 33 ERROR neutron.service [-] Unrecoverable error: please check log for details.
2017-04-18 16:41:29.525 33 ERROR neutron.service Traceback (most recent call last):
2017-04-18 16:41:29.525 33 ERROR neutron.service File "/usr/lib/python2.7/site-packages/neutron/service.py", line 87, in serve_wsgi
2017-04-18 16:41:29.525 33 ERROR neutron.service service.start()
2017-04-18 16:41:29.525 33 ERROR neutron.service File "/usr/lib/python2.7/site-packages/neutron/service.py", line 63, in start
2017-04-18 16:41:29.525 33 ERROR neutron.service self.wsgi_app = _run_wsgi(self.app_name)
2017-04-18 16:41:29.525 33 ERROR neutron.service File "/usr/lib/python2.7/site-packages/neutron/service.py", line 289, in _run_wsgi
2017-04-18 16:41:29.525 33 ERROR neutron.service app = config.load_paste_app(app_name)
2017-04-18 16:41:29.525 33 ERROR neutron.service File "/usr/lib/python2.7/site-packages/neutron/common/config.py", line 125, in load_paste_app
2017-04-18 16:41:29.525 33 ERROR neutron.service app = loader.load_app(app_name)
2017-04-18 16:41:29.525 33 ERROR neutron.service File "/usr/lib/python2.7/site-packages/oslo_service/wsgi.py", line 353, in load_app
2017-04-18 16:41:29.525 33 ERROR neutron.service return deploy.loadapp("config:%s" % self.config_path, name=name)
2017-04-18 16:41:29.525 33 ERROR neutron.service File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 247, in loadapp
2017-04-18 16:41:29.525 33 ERROR neutron.service return loadobj(APP, uri, name=name, **kw)
2017-04-18 16:41:29.525 33 ERROR neutron.service File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 272, in loadobj
2017-04-18 16:41:29.525 33 ERROR neutron.service return context.create()
2017-04-18 16:41:29.525 33 ERROR neutron.service File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create
2017-04-18 16:41:29.525 33 ERROR neutron.service return self.object_type.invoke(self)
2017-04-18 16:41:29.525 33 ERROR neutron.service File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 144, in invoke
2017-04-18 16:41:29.525 33 ERROR neutron.service **context.local_conf)
2017-04-18 16:41:29.525 33 ERROR neutron.service File "/usr/lib/python2.7/site-packages/paste/deploy/util.py", line 55, in fix_call
2017-04-18 16:41:29.525 33 ERROR neutron.service val = callable(*args, **kw)
2017-04-18 16:41:29.525 33 ERROR neutron.service File "/usr/lib/python2.7/site-packages/paste/urlmap.py", line 25, in urlmap_factory
2017-04-18 16:41:29.525 33 ERROR neutron.service app = loader.get_app(app_name, global_conf=global_conf)
2017-04-18 16:41:29.525 33 ERROR neutron.service File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 350, in get_app
2017-04-18 16:41:29.525 33 ERROR neutron.service name=name, global_conf=global_conf).create()
2017-04-18 16:41:29.525 33 ERROR neutron.service File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create
2017-04-18 16:41:29.525 33 ERROR neutron.service return self.object_type.invoke(self)
2017-04-18 16:41:29.525 33 ERROR neutron.service File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 144, in invoke
2017-04-18 16:41:29.525 33 ERROR neutron.service **context.local_conf)
2017-04-18 16:41:29.525 33 ERROR neutron.service File "/usr/lib/python2.7/site-packages/paste/deploy/util.py", line 55, in fix_call
2017-04-18 16:41:29.525 33 ERROR neutron.service val = callable(*args, **kw)
2017-04-18 16:41:29.525 33 ERROR neutron.service File "/usr/lib/python2.7/site-packages/neutron/auth.py", line 71, in pipeline_factory
2017-04-18 16:41:29.525 33 ERROR neutron.service app = loader.get_app(pipeline[-1])
2017-04-18 16:41:29.525 33 ERROR neutron.service File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 350, in get_app
2017-04-18 16:41:29.525 33 ERROR neutron.service name=name, global_conf=global_conf).create()
2017-04-18 16:41:29.525 33 ERROR neutron.service File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create
2017-04-18 16:41:29.525 33 ERROR neutron.service return self.object_type.invoke(self)
2017-04-18 16:41:29.525 33 ERROR neutron.service File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 146, in invoke
2017-04-18 16:41:29.525 33 ERROR neutron.service return fix_call(context.object, context.global_conf, **context.local_conf)
2017-04-18 16:41:29.525 33 ERROR neutron.service File "/usr/lib/python2.7/site-packages/paste/deploy/util.py", line 55, in fix_call
2017-04-18 16:41:29.525 33 ERROR neutron.service val = callable(*args, **kw)
2017-04-18 16:41:29.525 33 ERROR neutron.service File "/usr/lib/python2.7/site-packages/neutron/api/v2/router.py", line 72, in factory
2017-04-18 16:41:29.525 33 ERROR neutron.service return cls(**local_config)
2017-04-18 16:41:29.525 33 ERROR neutron.service File "/usr/lib/python2.7/site-packages/neutron/api/v2/router.py", line 76, in __init__
2017-04-18 16:41:29.525 33 ERROR neutron.service plugin = manager.NeutronManager.get_plugin()
2017-04-18 16:41:29.525 33 ERROR neutron.service File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 244, in get_plugin
2017-04-18 16:41:29.525 33 ERROR neutron.service return weakref.proxy(cls.get_instance().plugin)
2017-04-18 16:41:29.525 33 ERROR neutron.service File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 238, in get_instance
2017-04-18 16:41:29.525 33 ERROR neutron.service cls._create_instance()
2017-04-18 16:41:29.525 33 ERROR neutron.service File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 271, in inner
2017-04-18 16:41:29.525 33 ERROR neutron.service return f(*args, **kwargs)
2017-04-18 16:41:29.525 33 ERROR neutron.service File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 224, in _create_instance
2017-04-18 16:41:29.525 33 ERROR neutron.service cls._instance = cls()
2017-04-18 16:41:29.525 33 ERROR neutron.service File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 126, in __init__
2017-04-18 16:41:29.525 33 ERROR neutron.service plugin_provider)
2017-04-18 16:41:29.525 33 ERROR neutron.service File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 160, in _get_plugin_instance
2017-04-18 16:41:29.525 33 ERROR neutron.service return plugin_class()
2017-04-18 16:41:29.525 33 ERROR neutron.service app = loader.get_app(app_name, global_conf=global_conf)
2017-04-18 16:41:29.525 33 ERROR neutron.service File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 350, in get_app
2017-04-18 16:41:29.525 33 ERROR neutron.service name=name, global_conf=global_conf).create()
2017-04-18 16:41:29.525 33 ERROR neutron.service File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create
2017-04-18 16:41:29.525 33 ERROR neutron.service return self.object_type.invoke(self)
2017-04-18 16:41:29.525 33 ERROR neutron.service File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 144, in invoke
2017-04-18 16:41:29.525 33 ERROR neutron.service **context.local_conf)
2017-04-18 16:41:29.525 33 ERROR neutron.service File "/usr/lib/python2.7/site-packages/paste/deploy/util.py", line 55, in fix_call
2017-04-18 16:41:29.525 33 ERROR neutron.service val = callable(*args, **kw)
2017-04-18 16:41:29.525 33 ERROR neutron.service File "/usr/lib/python2.7/site-packages/neutron/auth.py", line 71, in pipeline_factory
2017-04-18 16:41:29.525 33 ERROR neutron.service app = loader.get_app(pipeline[-1])
2017-04-18 16:41:29.525 33 ERROR neutron.service File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 350, in get_app
2017-04-18 16:41:29.525 33 ERROR neutron.service name=name, global_conf=global_conf).create()
2017-04-18 16:41:29.525 33 ERROR neutron.service File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create
2017-04-18 16:41:29.525 33 ERROR neutron.service return self.object_type.invoke(self)
2017-04-18 16:41:29.525 33 ERROR neutron.service File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 146, in invoke
2017-04-18 16:41:29.525 33 ERROR neutron.service return fix_call(context.object, context.global_conf, **context.local_conf)
2017-04-18 16:41:29.525 33 ERROR neutron.service File "/usr/lib/python2.7/site-packages/paste/deploy/util.py", line 55, in fix_call
2017-04-18 16:41:29.525 33 ERROR neutron.service val = callable(*args, **kw)
2017-04-18 16:41:29.525 33 ERROR neutron.service File "/usr/lib/python2.7/site-packages/neutron/api/v2/router.py", line 72, in factory
2017-04-18 16:41:29.525 33 ERROR neutron.service return cls(**local_config)
2017-04-18 16:41:29.525 33 ERROR neutron.service File "/usr/lib/python2.7/site-packages/neutron/api/v2/router.py", line 76, in __init__
2017-04-18 16:41:29.525 33 ERROR neutron.service plugin = manager.NeutronManager.get_plugin()
2017-04-18 16:41:29.525 33 ERROR neutron.service File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 244, in get_plugin
2017-04-18 16:41:29.525 33 ERROR neutron.service return weakref.proxy(cls.get_instance().plugin)
2017-04-18 16:41:29.525 33 ERROR neutron.service File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 238, in get_instance
2017-04-18 16:41:29.525 33 ERROR neutron.service cls._create_instance()
2017-04-18 16:41:29.525 33 ERROR neutron.service File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 271, in inner
2017-04-18 16:41:29.525 33 ERROR neutron.service return f(*args, **kwargs)
2017-04-18 16:41:29.525 33 ERROR neutron.service File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 224, in _create_instance
2017-04-18 16:41:29.525 33 ERROR neutron.service cls._instance = cls()
2017-04-18 16:41:29.525 33 ERROR neutron.service File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 126, in __init__
2017-04-18 16:41:29.525 33 ERROR neutron.service plugin_provider)
2017-04-18 16:41:29.525 33 ERROR neutron.service File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 160, in _get_plugin_instance
2017-04-18 16:41:29.525 33 ERROR neutron.service return plugin_class()
2017-04-18 16:41:29.525 33 ERROR neutron.service File "/usr/lib/python2.7/site-packages/neutron/quota/resource_registry.py", line 127, in wrapper
2017-04-18 16:41:29.525 33 ERROR neutron.service return f(*args, **kwargs)
2017-04-18 16:41:29.525 33 ERROR neutron.service File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py", line 163, in __init__
2017-04-18 16:41:29.525 33 ERROR neutron.service self.mechanism_manager.initialize()
2017-04-18 16:41:29.525 33 ERROR neutron.service File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/managers.py", line 399, in initialize
2017-04-18 16:41:29.525 33 ERROR neutron.service driver.obj.initialize()
2017-04-18 16:41:29.525 33 ERROR neutron.service File "/usr/lib/python2.7/site-packages/networking_vpp/mech_vpp.py", line 89, in initialize
2017-04-18 16:41:29.525 33 ERROR neutron.service self.communicator = EtcdAgentCommunicator(self.port_bind_complete)
2017-04-18 16:41:29.525 33 ERROR neutron.service File "/usr/lib/python2.7/site-packages/networking_vpp/mech_vpp.py", line 464, in __init__
2017-04-18 16:41:29.525 33 ERROR neutron.service etcd_helper.ensure_dir(self.state_key_space)
2017-04-18 16:41:29.525 33 ERROR neutron.service File "/usr/lib/python2.7/site-packages/networking_vpp/agent/utils.py", line 45, in ensure_dir
2017-04-18 16:41:29.525 33 ERROR neutron.service self.etcd_client.write(path, None, dir=True)
2017-04-18 16:41:29.525 33 ERROR neutron.service File "/usr/lib/python2.7/site-packages/etcd/client.py", line 500, in write
2017-04-18 16:41:29.525 33 ERROR neutron.service response = self.api_execute(path, method, params=params)
2017-04-18 16:41:29.525 33 ERROR neutron.service File "/usr/lib/python2.7/site-packages/etcd/client.py", line 907, in wrapper
2017-04-18 16:41:29.525 33 ERROR neutron.service return self._handle_server_response(response)
2017-04-18 16:41:29.525 33 ERROR neutron.service File "/usr/lib/python2.7/site-packages/etcd/client.py", line 987, in _handle_server_response
2017-04-18 16:41:29.525 33 ERROR neutron.service etcd.EtcdError.handle(r)
2017-04-18 16:41:29.525 33 ERROR neutron.service File "/usr/lib/python2.7/site-packages/etcd/__init__.py", line 306, in handle
2017-04-18 16:41:29.525 33 ERROR neutron.service raise exc(msg, payload)
2017-04-18 16:41:29.525 33 ERROR neutron.service EtcdException: Raft Internal Error : etcdserver: request timed out, possibly due to previous leader failure

Looks like neutron-server is coming up, and loading all ML2 plugins. networking-vpp created an etcdclient, but failed to connect to etcdserver. The server at this moment is probably still booting up, so better we have a retry mechanism to be more reliable.