I'm starting in on this MIR and I'm surprised how many errors there are in the buildlogs. Both a version I built locally and some buildd build logs are noisy:

https://launchpad.net/ubuntu/+source/octavia/1:7.0.0+git2021012713.fbbc5f90-0ubuntu1/+build/20944559/+files/buildlog_ubuntu-hirsute-amd64.octavia_1%3A7.0.0+git2021012713.fbbc5f90-0ubuntu1_BUILDING.txt.gz


/usr/lib/python3/dist-packages/pbr/core.py:131: UserWarning: Unknown distribution option: 'requires_python'
  warnings.warn(msg)

Missing keepalived PID file /tmp/test/vrrp/octavia-keepalived.pid, skipping health heartbeat.
octavia.tests.unit.api.drivers.amphora_driver.v2.test_driver.TestAmphoraDriver.test_validate_flavor
octavia.tests.unit.api.drivers.amphora_driver.v2.test_driver.TestAmphoraDriver.test_validate_flavor ... ok
Failed to check keepalived and haproxy status due to exception , skipping health heartbeat.
Traceback (most recent call last):
  File "/<<BUILDDIR>>/octavia-7.0.0+git2021012713.fbbc5f90/octavia/amphorae/backends/health_daemon/health_daemon.py", line 121, in run_sender
    with open(keepalived_pid_path, 'r') as pid_file:
  File "/usr/lib/python3.9/unittest/mock.py", line 1093, in __call__
    return self._mock_call(*args, **kwargs)
  File "/usr/lib/python3.9/unittest/mock.py", line 1097, in _mock_call
    return self._execute_mock_call(*args, **kwargs)
  File "/usr/lib/python3.9/unittest/mock.py", line 1152, in _execute_mock_call
    raise effect
OSError
Failed to check keepalived and haproxy status due to exception invalid literal for int() with base 10: 'foo', skipping health heartbeat.
Traceback (most recent call last):
  File "/<<BUILDDIR>>/octavia-7.0.0+git2021012713.fbbc5f90/octavia/amphorae/backends/health_daemon/health_daemon.py", line 122, in run_sender
    pid = int(pid_file.readline())
ValueError: invalid literal for int() with base 10: 'foo'
Keepalived is configured but not running, skipping health heartbeat.
Failed to check keepalived and haproxy status due to exception , skipping health heartbeat.
Traceback (most recent call last):
  File "/<<BUILDDIR>>/octavia-7.0.0+git2021012713.fbbc5f90/octavia/amphorae/backends/health_daemon/health_daemon.py", line 123, in run_sender
    os.kill(pid, 0)
  File "/usr/lib/python3.9/unittest/mock.py", line 1093, in __call__
    return self._mock_call(*args, **kwargs)
  File "/usr/lib/python3.9/unittest/mock.py", line 1097, in _mock_call
    return self._execute_mock_call(*args, **kwargs)
  File "/usr/lib/python3.9/unittest/mock.py", line 1152, in _execute_mock_call
    raise effect
OSError
calculated hmac(hex=True): 64656665373265366363366664376235343632313263373937653933396336393236353035626462386566356533323138306561626637343036323965356366 not equal to msg hmac: 31393335326537356662646134393735346263633839633634376133386135343664626439613137323464316539333261363962303038653862323232383734 dropping packet
calculated hmac(hex=False): cb3d8c27bc9975ba1f9efc5a542a9a5cbf421f8aa28c79ec8f293647340cc0ae not equal to msg hmac: 3664626439613137323464316539333261363962303038653862323232383734 dropping packet
calculated hmac(hex=True): 38666134663061636636383463666538643637633536643562633830623461386265316664316633303265343038643832356264393461613134303835393039 not equal to msg hmac: 33303065373165333064306465636362656131353363376534396436326431313065643462326639343837333631646664333235636263613663313931633730 dropping packet
octavia.tests.unit.amphorae.backends.health_daemon.test_health_daemon.TestHealthDaemon.test_run_sender

[there's 39 of these hmacs not equal messages]

octavia.tests.unit.api.drivers.driver_agent.test_driver_updater.TestDriverUpdater.test_update_listener_statistics
octavia.tests.unit.api.drivers.driver_agent.test_driver_updater.TestDriverUpdater.test_update_listener_statistics ... ok
Unable to send address advertisement for address: "not an IP", error: failed to detect a valid IP address from 'not an IP'. Skipping
Unknown IP version for address: "2001:db8::33". Skipping
octavia.tests.unit.amphorae.drivers.haproxy.test_rest_api_driver_0_5.TestHaproxyAmphoraLoadBalancerDriverTest.test__process_pool_certs
octavia.tests.unit.amphorae.drivers.haproxy.test_rest_api_driver_0_5.TestHaproxyAmphoraLoadBalancerDriverTest.test__process_pool_certs ... ok


octavia.tests.unit.api.v2.types.test_l7rule.TestL7RulePOST.test_value_mandatory
octavia.tests.unit.api.v2.types.test_l7rule.TestL7RulePOST.test_value_mandatory ... ok
Could not connect to instance. Retrying.
Could not connect to instance. Retrying.
Connection retries (currently set to 2) exhausted.  The amphora is unavailable. Reason:$
Could not connect to instance. Retrying.
Could not connect to instance. Retrying.
Could not connect to instance. Retrying.
Connection retries (currently set to 3) exhausted.  The amphora is unavailable. Reason:$
Amphora sample_amphora_id_1 for loadbalancer sample_loadbalancer_id_1 needs upgrade to single process mode.
octavia.tests.unit.amphorae.drivers.haproxy.test_rest_api_driver_0_5.TestHaproxyAmphoraLoadBalancerDriverTest.test_delete_second_listener
octavia.tests.unit.amphorae.drivers.haproxy.test_rest_api_driver_0_5.TestHaproxyAmphoraLoadBalancerDriverTest.test_delete_second_listener ... ok

octavia.tests.unit.amphorae.drivers.haproxy.test_rest_api_driver_0_5.TestHaproxyAmphoraLoadBalancerDriverTest.test_populate_amphora_api_version
octavia.tests.unit.amphorae.drivers.haproxy.test_rest_api_driver_0_5.TestHaproxyAmphoraLoadBalancerDriverTest.test_populate_amphora_api_version ... ok
Error storing certificate data:$
octavia.tests.unit.amphorae.drivers.haproxy.test_rest_api_driver_0_5.TestAmphoraAPIClientTest.test_upload_udp_invalid_config
octavia.tests.unit.amphorae.drivers.haproxy.test_rest_api_driver_0_5.TestAmphoraAPIClientTest.test_upload_udp_invalid_config ... ok

octavia.tests.unit.amphorae.drivers.haproxy.test_rest_api_driver_1_0.TestAmphoraAPIClientTest.test_get_info
octavia.tests.unit.amphorae.drivers.haproxy.test_rest_api_driver_1_0.TestAmphoraAPIClientTest.test_get_info ... ok
Amphora <MagicMock name='mock.id' id='139637539497056'> for loadbalancer 6e66d974-f116-4126-b36c-e0671f01ba29 needs upgrade to single process mode.
octavia.tests.unit.cmd.test_health_manager.TestHealthManagerCMD.test_handle_mutate_config
octavia.tests.unit.cmd.test_health_manager.TestHealthManagerCMD.test_handle_mutate_config ... ok
octavia.tests.unit.amphorae.drivers.haproxy.test_rest_api_driver_0_5.TestHaproxyAmphoraLoadBalancerDriverTest.test_reload
octavia.tests.unit.amphorae.drivers.haproxy.test_rest_api_driver_0_5.TestHaproxyAmphoraLoadBalancerDriverTest.test_reload ... ok
octavia.tests.unit.amphorae.drivers.haproxy.test_rest_api_driver_1_0.TestAmphoraAPIClientTest.test_get_info_missing
octavia.tests.unit.amphorae.drivers.haproxy.test_rest_api_driver_1_0.TestAmphoraAPIClientTest.test_get_info_missing ... ok
octavia.tests.unit.cmd.test_health_manager.TestHealthManagerCMD.test_hm_health_check
octavia.tests.unit.cmd.test_health_manager.TestHealthManagerCMD.test_hm_health_check ... ok
Health Manager listener experienced unknown error: break
octavia.tests.unit.amphorae.drivers.haproxy.test_rest_api_driver_1_0.TestAmphoraAPIClientTest.test_get_info_server_error
octavia.tests.unit.amphorae.drivers.haproxy.test_rest_api_driver_1_0.TestAmphoraAPIClientTest.test_get_info_server_error ... ok
octavia.tests.unit.cmd.test_health_manager.TestHealthManagerCMD.test_hm_listener
octavia.tests.unit.cmd.test_health_manager.TestHealthManagerCMD.test_hm_listener ... ok
octavia.tests.unit.amphorae.drivers.haproxy.test_rest_api_driver_1_0.TestAmphoraAPIClientTest.test_get_info_service_unavailable
octavia.tests.unit.amphorae.drivers.haproxy.test_rest_api_driver_1_0.TestAmphoraAPIClientTest.test_get_info_service_unavailable ... ok
octavia.tests.unit.amphorae.drivers.haproxy.test_rest_api_driver_1_0.TestHaproxyAmphoraLoadBalancerDriverTest.test_delete_last_listener
octavia.tests.unit.amphorae.drivers.haproxy.test_rest_api_driver_1_0.TestHaproxyAmphoraLoadBalancerDriverTest.test_delete_last_listener ... ok
octavia.tests.unit.cmd.test_health_manager.TestHealthManagerCMD.test_main
octavia.tests.unit.cmd.test_health_manager.TestHealthManagerCMD.test_main ... ok
octavia.tests.unit.amphorae.drivers.haproxy.test_rest_api_driver_1_0.TestAmphoraAPIClientTest.test_get_info_unauthorized
octavia.tests.unit.amphorae.drivers.haproxy.test_rest_api_driver_1_0.TestAmphoraAPIClientTest.test_get_info_unauthorized ... ok
Amphora agent returned unexpected result code 500 with response FAIL
octavia.tests.unit.cmd.test_health_manager.TestHealthManagerCMD.test_main_keyboard_interrupt
octavia.tests.unit.cmd.test_health_manager.TestHealthManagerCMD.test_main_keyboard_interrupt ... ok
octavia.tests.unit.amphorae.drivers.haproxy.test_rest_api_driver_1_0.TestAmphoraAPIClientTest.test_get_interface
octavia.tests.unit.amphorae.drivers.haproxy.test_rest_api_driver_1_0.TestAmphoraAPIClientTest.test_get_interface ... ok
octavia.tests.unit.common.jinja.haproxy.split_listeners.test_jinja_cfg.TestHaproxyCfg.test_escape_haproxy_config_string
octavia.tests.unit.common.jinja.haproxy.split_listeners.test_jinja_cfg.TestHaproxyCfg.test_escape_haproxy_config_string ... ok
octavia.tests.unit.amphorae.drivers.haproxy.test_rest_api_driver_1_0.TestAmphoraAPIClientTest.test_plug_network
octavia.tests.unit.amphorae.drivers.haproxy.test_rest_api_driver_1_0.TestAmphoraAPIClientTest.test_plug_network ... ok
octavia.tests.unit.amphorae.drivers.haproxy.test_rest_api_driver_1_0.TestAmphoraAPIClientTest.test_plug_vip
octavia.tests.unit.amphorae.drivers.haproxy.test_rest_api_driver_1_0.TestAmphoraAPIClientTest.test_plug_vip ... ok
octavia.tests.unit.common.jinja.haproxy.split_listeners.test_jinja_cfg.TestHaproxyCfg.test_get_template
octavia.tests.unit.common.jinja.haproxy.split_listeners.test_jinja_cfg.TestHaproxyCfg.test_get_template ... ok
octavia.tests.unit.amphorae.drivers.haproxy.test_rest_api_driver_1_0.TestHaproxyAmphoraLoadBalancerDriverTest.test_delete_second_listener
octavia.tests.unit.amphorae.drivers.haproxy.test_rest_api_driver_1_0.TestHaproxyAmphoraLoadBalancerDriverTest.test_delete_second_listener ... ok
Could not connect to instance. Retrying.
Could not connect to instance. Retrying.
Connection retries (currently set to 2) exhausted.  The amphora is unavailable. Reason:$
Could not connect to instance. Retrying.
Could not connect to instance. Retrying.
Could not connect to instance. Retrying.
Connection retries (currently set to 3) exhausted.  The amphora is unavailable. Reason:$
octavia.tests.unit.amphorae.drivers.haproxy.test_rest_api_driver_1_0.TestAmphoraAPIClientTest.test_plug_vip_api_not_ready
octavia.tests.unit.amphorae.drivers.haproxy.test_rest_api_driver_1_0.TestAmphoraAPIClientTest.test_plug_vip_api_not_ready ... ok

octavia.tests.unit.amphorae.drivers.haproxy.test_rest_api_driver_1_0.TestHaproxyAmphoraLoadBalancerDriverTest.test_populate_amphora_api_version
octavia.tests.unit.amphorae.drivers.haproxy.test_rest_api_driver_1_0.TestHaproxyAmphoraLoadBalancerDriverTest.test_populate_amphora_api_version ... ok
calculated hmac(hex=True): 65333961653566333233336635363131363238333639383965303530633162363162613965336435656233643634643462333730316539356662353835306131 not equal to msg hmac: 78daab562a492d2ec94ead54b252500a710d0e51aa050041b506245806e5c1971e79951818394ea6e71ad989ff950945f9573f4ab6f83e25db8ed7 dropping packet
octavia.tests.unit.common.jinja.haproxy.split_listeners.test_jinja_cfg.TestHaproxyCfg.test_render_template_unlimited_connections
octavia.tests.unit.common.jinja.haproxy.split_listeners.test_jinja_cfg.TestHaproxyCfg.test_render_template_unlimited_connections ... ok

octavia.tests.unit.amphorae.drivers.haproxy.test_rest_api_driver_0_5.TestHaproxyAmphoraLoadBalancerDriverTest.test_update_amphora_agent_config
octavia.tests.unit.amphorae.drivers.haproxy.test_rest_api_driver_0_5.TestHaproxyAmphoraLoadBalancerDriverTest.test_update_amphora_agent_config ... Member member-id-4 reported status of DRAINING
ok
Listener listener-id-5 reported status of bogus
Pool pool-id-5 reported status of bogus
Member member-id-5 reported status of bogus
octavia.tests.unit.common.jinja.haproxy.split_listeners.test_jinja_cfg.TestHaproxyCfg.test_transform_loadbalancer_with_l7
octavia.tests.unit.common.jinja.haproxy.split_listeners.test_jinja_cfg.TestHaproxyCfg.test_transform_loadbalancer_with_l7 ... ok

octavia.tests.unit.amphorae.drivers.haproxy.test_rest_api_driver_1_0.TestHaproxyAmphoraLoadBalancerDriverTest.test_start_with_amphora
octavia.tests.unit.amphorae.drivers.haproxy.test_rest_api_driver_1_0.TestHaproxyAmphoraLoadBalancerDriverTest.test_start_with_amphora ... ok
Unable to update listener sample_listener_id_1 due to "". Skipping this listener.
Traceback (most recent call last):
  File "/<<BUILDDIR>>/octavia-7.0.0+git2021012713.fbbc5f90/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 174, in update_amphora_listeners
    self._process_tls_certificates(
  File "/<<BUILDDIR>>/octavia-7.0.0+git2021012713.fbbc5f90/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 458, in _process_tls_certificates
    data = cert_parser.load_certificates_data(
  File "/usr/lib/python3.9/unittest/mock.py", line 1093, in __call__
    return self._mock_call(*args, **kwargs)
  File "/usr/lib/python3.9/unittest/mock.py", line 1097, in _mock_call
    return self._execute_mock_call(*args, **kwargs)
  File "/usr/lib/python3.9/unittest/mock.py", line 1156, in _execute_mock_call
    raise result
Exception
octavia.tests.unit.compute.drivers.test_nova_driver.TestNovaClient.test_attach_network_or_port
octavia.tests.unit.compute.drivers.test_nova_driver.TestNovaClient.test_attach_network_or_port ... ok

and so on. There's 19 lines matching the regex '^Exception', showing a stack trace from python.

Are we sure this works? Does it work well enough to support it? This is one of the noisiest build logs I've seen, it's not encouraging.

If these failures are because the builds are run as unprivileged users, that's one thing, but it'd be nice if the test suite would skip those gracefully with a message indicating that privilege is required, etc.

Thanks