zaza.openstack.utilities.exceptions.NeutronBGPSpeakerMissing: No BGP Speaker appeared on agent

Gate test failing with "zaza.openstack.utilities.exceptions.NeutronBGPSpeakerMissing: No BGP Speaker appeared on agent." The charm state is good, quagga has an AS listed.

Observed on xenial-pike, but may not be limited to that.

neutron-dynamic-routing/0* active idle 3 172.17.115.13 Unit is ready

quagga/0* active idle 4 172.17.115.6 Ready (AS Number 65285)

Artifact index:
https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_full/openstack/charm-neutron-dynamic-routing/677906/1/3660/index.html

Juju status:
https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_full/openstack/charm-neutron-dynamic-routing/677906/1/3660/test_charm_func_full_6934/juju-status-zaza-3680dc9b891b.txt

Juju crashdump:
https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_full/openstack/charm-neutron-dynamic-routing/677906/1/3660/test_charm_func_full_6934/juju-crashdump-137bc80d-0c7e-42ee-830c-43e3a8c79313.tar.xz

Test console output:
https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_full/openstack/charm-neutron-dynamic-routing/677906/1/3660/consoleText.test_charm_func_full_6934.txt

2019-08-22 13:05:15 [INFO] Deploying bundle './tests/bundles/xenial-pike-functional.yaml' on to 'zaza-3680dc9b891b' model
2019-08-22 13:05:15 [INFO] Rendered template '<Template memory:7fb690c2eda0>' to file '/tmp/tmp_x05hxui/local-charm-overlay.yaml'
2019-08-22 13:05:15 [INFO] Deploying overlay '/tmp/tmp_x05hxui/local-charm-overlay.yaml' on to 'zaza-3680dc9b891b' model
2019-08-22 13:05:16 [INFO] Resolving charm: cs:~openstack-charmers-next/xenial/keystone
2019-08-22 13:05:16 [INFO] Resolving charm: cs:~openstack-charmers-next/xenial/percona-cluster
2019-08-22 13:05:16 [INFO] Resolving charm: cs:~openstack-charmers-next/xenial/neutron-api
2019-08-22 13:05:16 [INFO] Resolving charm: cs:~openstack-charmers-next/xenial/quagga
2019-08-22 13:05:16 [INFO] Resolving charm: cs:~openstack-charmers-next/xenial/rabbitmq-server
2019-08-22 13:05:16 [INFO] Executing changes:
2019-08-22 13:05:16 [INFO] - upload charm cs:~openstack-charmers-next/keystone-453 for series xenial
2019-08-22 13:05:17 [INFO] - deploy application keystone on xenial using cs:~openstack-charmers-next/keystone-453
2019-08-22 13:05:18 [INFO] - upload charm cs:~openstack-charmers-next/percona-cluster-353 for series xenial
2019-08-22 13:05:19 [INFO] - deploy application mysql on xenial using cs:~openstack-charmers-next/percona-cluster-353
2019-08-22 13:05:20 [INFO] - upload charm cs:~openstack-charmers-next/neutron-api-413 for series xenial
2019-08-22 13:05:20 [INFO] - deploy application neutron-api on xenial using cs:~openstack-charmers-next/neutron-api-413
2019-08-22 13:05:22 [INFO] - upload charm /var/lib/jenkins/workspace/test_charm_func_full/asset/neutron-dynamic-routing/tests/bundles/../../../neutron-dynamic-routing for series xenial
2019-08-22 13:05:23 [INFO] - deploy application neutron-dynamic-routing on xenial using /var/lib/jenkins/workspace/test_charm_func_full/asset/neutron-dynamic-routing/tests/bundles/../../../neutron-dynamic-routing
2019-08-22 13:05:24 [INFO] - upload charm cs:~openstack-charmers-next/quagga-6 for series xenial
2019-08-22 13:05:26 [INFO] - deploy application quagga on xenial using cs:~openstack-charmers-next/quagga-6
2019-08-22 13:05:27 [INFO] - upload charm cs:~openstack-charmers-next/rabbitmq-server-348 for series xenial
2019-08-22 13:05:27 [INFO] - deploy application rabbitmq-server on xenial using cs:~openstack-charmers-next/rabbitmq-server-348
2019-08-22 13:05:28 [INFO] - add relation neutron-dynamic-routing - rabbitmq-server
2019-08-22 13:05:29 [INFO] - add relation neutron-dynamic-routing - quagga
2019-08-22 13:05:29 [INFO] - add relation keystone - mysql
2019-08-22 13:05:29 [INFO] - add relation neutron-api - mysql
2019-08-22 13:05:29 [INFO] - add relation neutron-api - rabbitmq-server
2019-08-22 13:05:29 [INFO] - add relation neutron-api - keystone
2019-08-22 13:05:29 [INFO] - add unit keystone/0 to new machine 0
2019-08-22 13:05:30 [INFO] - add unit mysql/0 to new machine 1
2019-08-22 13:05:31 [INFO] - add unit neutron-api/0 to new machine 2
2019-08-22 13:05:32 [INFO] - add unit neutron-dynamic-routing/0 to new machine 3
2019-08-22 13:05:34 [INFO] - add unit quagga/0 to new machine 4
2019-08-22 13:05:35 [INFO] - add unit rabbitmq-server/0 to new machine 5
2019-08-22 13:05:36 [INFO] Deploy of bundle completed.
2019-08-22 13:05:36 [INFO] Waiting for environment to settle
2019-08-22 13:05:36 [INFO] Waiting for a unit to appear
2019-08-22 13:05:36 [INFO] Waiting for all units to be idle
2019-08-22 13:17:13 [INFO] Checking workload status of keystone/0
2019-08-22 13:17:13 [INFO] Checking workload status message of keystone/0
2019-08-22 13:17:13 [INFO] Checking workload status of neutron-api/0
2019-08-22 13:17:13 [INFO] Checking workload status message of neutron-api/0
2019-08-22 13:17:13 [INFO] Checking workload status of quagga/0
2019-08-22 13:17:13 [INFO] Checking workload status message of quagga/0
2019-08-22 13:17:13 [INFO] Checking workload status of mysql/0
2019-08-22 13:17:13 [INFO] Checking workload status message of mysql/0
2019-08-22 13:17:13 [INFO] Checking workload status of rabbitmq-server/0
2019-08-22 13:17:13 [INFO] Checking workload status message of rabbitmq-server/0
2019-08-22 13:17:13 [INFO] Checking workload status of neutron-dynamic-routing/0
2019-08-22 13:17:13 [INFO] Checking workload status message of neutron-dynamic-routing/0
2019-08-22 13:17:16 [INFO] Using keystone API V2 for overcloud auth
2019-08-22 13:17:20 [INFO] Configuring overcloud network
2019-08-22 13:17:21 [INFO] Configuring external network
2019-08-22 13:17:21 [INFO] Creating new external network definition: ext_net
2019-08-22 13:17:22 [INFO] New external network created: 3b063a5d-48d5-4445-ad17-7d1d9b222f2c
2019-08-22 13:17:22 [INFO] Creating new subnet
2019-08-22 13:17:22 [INFO] New subnet created: 0637caff-d5ec-4d14-9748-ea163b24f22a
2019-08-22 13:17:22 [INFO] Creating provider router for external network access
2019-08-22 13:17:22 [INFO] New router created: cd13cb6e-f592-4187-8da4-0105d88e148f
2019-08-22 13:17:22 [INFO] Plugging router into ext_net
2019-08-22 13:17:25 [INFO] Router connected
2019-08-22 13:17:25 [INFO] Creating public address scope
2019-08-22 13:17:25 [INFO] New address scope created: cfb0f793-5ddd-49c5-b73b-213c18136dd0
2019-08-22 13:17:25 [INFO] Creating subnetpool: pooled_subnets
2019-08-22 13:17:25 [INFO] Creating network: private
2019-08-22 13:17:25 [INFO] Creating subnet
2019-08-22 13:17:26 [INFO] Updating dns_nameservers (10.5.0.2) for subnet
2019-08-22 13:17:27 [INFO] Adding interface from subnet to provider-router
2019-08-22 13:17:29 [INFO] Adding ssh security group rule
2019-08-22 13:17:29 [INFO] Adding ping security group rule
2019-08-22 13:17:37 [INFO] Setting up BGP speaker
2019-08-22 13:17:37 [INFO] Creating BGP Speaker
2019-08-22 13:17:37 [INFO] Advertising BGP routes
2019-08-22 13:17:37 [INFO] Advertising ext_net network on BGP Speaker bgp-speaker
2019-08-22 13:17:37 [INFO] Advertising private network on BGP Speaker bgp-speaker
2019-08-22 13:17:38 [INFO] Setting up BGP peer
2019-08-22 13:17:38 [INFO] Creating BGP Peer
2019-08-22 13:17:38 [INFO] Adding BGP peer to BGP speaker
2019-08-22 13:17:38 [INFO] Adding peer quagga on BGP Speaker bgp-speaker
2019-08-22 13:17:38 [INFO] Creating floating IP to advertise
2019-08-22 13:17:38 [INFO] Creating port: FIP TEST
2019-08-22 13:17:39 [INFO] Creating floatingip
2019-08-22 13:17:42 [INFO] Advertised floating IP: 172.17.115.200
2019-08-22 13:17:42 [INFO] Waiting for Neutron agent 'neutron-bgp-dragent' to appear...
2019-08-22 13:17:47 [INFO] Using keystone API V2 for overcloud auth
2019-08-22 13:17:50 [INFO] Waiting for BGP speaker to appear on agent "02830902-75ac-4eb3-840d-f10b337294d8"...
Traceback (most recent call last):
  File "/tmp/tmp.zngYgJyU0i/func/bin/functest-run-suite", line 10, in <module>
    sys.exit(main())
  File "/tmp/tmp.zngYgJyU0i/func/lib/python3.5/site-packages/zaza/charm_lifecycle/func_test_runner.py", line 140, in main
    bundle=args.bundle)
  File "/tmp/tmp.zngYgJyU0i/func/lib/python3.5/site-packages/zaza/charm_lifecycle/func_test_runner.py", line 71, in func_test_runner
    config_steps.get(model_alias, []))
  File "/tmp/tmp.zngYgJyU0i/func/lib/python3.5/site-packages/zaza/charm_lifecycle/configure.py", line 48, in configure
    run_configure_list(functions)
  File "/tmp/tmp.zngYgJyU0i/func/lib/python3.5/site-packages/zaza/charm_lifecycle/configure.py", line 37, in run_configure_list
    utils.get_class(func)()
  File "/tmp/tmp.zngYgJyU0i/func/lib/python3.5/site-packages/zaza/openstack/charm_tests/dragent/configure.py", line 92, in setup
    keystone_session=keystone_session)
  File "/tmp/tmp.zngYgJyU0i/func/lib/python3.5/site-packages/zaza/openstack/configure/bgp_speaker.py", line 119, in setup_bgp_speaker
    neutron_client, agent_id)
  File "/tmp/tmp.zngYgJyU0i/func/lib/python3.5/site-packages/tenacity/__init__.py", line 292, in wrapped_f
    return self.call(f, *args, **kw)
  File "/tmp/tmp.zngYgJyU0i/func/lib/python3.5/site-packages/tenacity/__init__.py", line 358, in call
    do = self.iter(retry_state=retry_state)
  File "/tmp/tmp.zngYgJyU0i/func/lib/python3.5/site-packages/tenacity/__init__.py", line 331, in iter
    raise retry_exc.reraise()
  File "/tmp/tmp.zngYgJyU0i/func/lib/python3.5/site-packages/tenacity/__init__.py", line 167, in reraise
    raise self.last_attempt.result()
  File "/usr/lib/python3.5/concurrent/futures/_base.py", line 398, in result
    return self.__get_result()
  File "/usr/lib/python3.5/concurrent/futures/_base.py", line 357, in __get_result
    raise self._exception
  File "/tmp/tmp.zngYgJyU0i/func/lib/python3.5/site-packages/tenacity/__init__.py", line 361, in call
    result = fn(*args, **kwargs)
  File "/tmp/tmp.zngYgJyU0i/func/lib/python3.5/site-packages/zaza/openstack/utilities/openstack.py", line 2147, in neutron_bgp_speaker_appears_on_agent
    ''.format(agent_id))
zaza.openstack.utilities.exceptions.NeutronBGPSpeakerMissing: No BGP Speaker appeared on agent "02830902-75ac-4eb3-840d-f10b337294d8"