While fixing lp:1915512 [1] we implement a new test_update_controller_ip_port_list functional test [2] that starts by adding a 4th octavia unit.
Unfortunately occasionally the new unit will end up in error state:
[INFO] ERROR: test_update_controller_ip_port_list (zaza.openstack.charm_tests.octavia.tests.CharmOperationTest)
[INFO] Test update_controller_ip_port_list.
[INFO] ----------------------------------------------------------------------
[INFO] Traceback (most recent call last):
[INFO] File "/home/ubuntu/src/review.opendev.org/openstack/charm-octavia/build/builds/octavia/.tox/func-target/lib/python3.6/site-packages/zaza/openstack/charm_tests/octavia/tests.py", line 140, in test_update_controller_ip_port_list
[INFO] zaza.model.wait_for_application_states()
[INFO] File "/home/ubuntu/src/review.opendev.org/openstack/charm-octavia/build/builds/octavia/.tox/func-target/lib/python3.6/site-packages/zaza/__init__.py", line 48, in _wrapper
[INFO] return run(_run_it())
[INFO] File "/home/ubuntu/src/review.opendev.org/openstack/charm-octavia/build/builds/octavia/.tox/func-target/lib/python3.6/site-packages/zaza/__init__.py", line 36, in run
[INFO] return task.result()
[INFO] File "/home/ubuntu/src/review.opendev.org/openstack/charm-octavia/build/builds/octavia/.tox/func-target/lib/python3.6/site-packages/zaza/__init__.py", line 47, in _run_it
[INFO] return await f(*args, **kwargs)
[INFO] File "/home/ubuntu/src/review.opendev.org/openstack/charm-octavia/build/builds/octavia/.tox/func-target/lib/python3.6/site-packages/zaza/model.py", line 1386, in async_wait_for_application_states
[INFO] model, unit, check_wl_statuses)
[INFO] File "/home/ubuntu/src/review.opendev.org/openstack/charm-octavia/build/builds/octavia/.tox/func-target/lib/python3.6/site-packages/zaza/model.py", line 1136, in check_unit_workload_status
[INFO] check_model_for_hard_errors(model)
[INFO] File "/home/ubuntu/src/review.opendev.org/openstack/charm-octavia/build/builds/octavia/.tox/func-target/lib/python3.6/site-packages/zaza/model.py", line 1114, in check_model_for_hard_errors
[INFO] raise UnitError(errored_units)
[INFO] zaza.model.UnitError: Units octavia/3 in error state
[INFO] ----------------------------------------------------------------------
[INFO] Ran 2 tests in 883.158s
[INFO] FAILED
[INFO] (errors=1)
[ERROR] {'default_alias': 'zaza-21268a29090d'}
[ERROR] Model default_alias (zaza-21268a29090d)
[ERROR] Applications in error state: octavia
[ERROR] Units in error state: octavia/3
[ERROR] Juju log for octavia/3
[ERROR] unit-octavia-3.log: 2021-11-14 13:32:53 WARNING neutron-api-relation-changed if api_crud.setup_hm_port(
[ERROR] unit-octavia-3.log: 2021-11-14 13:32:53 WARNING neutron-api-relation-changed File "/var/lib/juju/agents/unit-octavia-3/charm/lib/charm/openstack/api_crud.py", line 524, in setup_hm_port
[ERROR] unit-octavia-3.log: 2021-11-14 13:32:53 WARNING neutron-api-relation-changed subprocess.check_call(
[ERROR] unit-octavia-3.log: 2021-11-14 13:32:53 WARNING neutron-api-relation-changed File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
[ERROR] unit-octavia-3.log: 2021-11-14 13:32:53 WARNING neutron-api-relation-changed raise CalledProcessError(retcode, cmd)
[ERROR] unit-octavia-3.log: 2021-11-14 13:32:53 WARNING neutron-api-relation-changed subprocess.CalledProcessError: Command '['ovs-vsctl', '--', 'add-port', 'br-int', 'o-hm0', '--', 'set', 'Interface', 'o-hm0', 'type=internal', '--', 'set', 'Interface', 'o-hm0', 'external-ids:iface-status=active', '--', 'set', 'Interface', 'o-hm0', 'external-ids:attached-mac=fa:16:3e:0c:74:ec', '--', 'set', 'Interface', 'o-hm0', 'external-ids:iface-id=d1c2cabd-9247-42c8-ab77-7d5dc8fc0c1c', '--', 'set', 'Interface', 'o-hm0', 'external-ids:skip_cleanup=true']' returned non-zero exit status 1.
[1]: https://review.opendev.org/c/openstack/charm-octavia/+/787700
[2]: https://github.com/openstack-charmers/zaza-openstack-tests/pull/600
