While investigating gate failure in https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_full/openstack/charm-octavia/751663/1/6876/consoleText.test_charm_func_full_9986.txt I found: Sep 14 21:06:32 juju-860e2a-zaza-4546bf52d844-10 systemd[1]: message repeated 4 times: [ Reloading.] Sep 14 21:06:56 juju-860e2a-zaza-4546bf52d844-10 systemd[1]: Stopping The Apache HTTP Server... Sep 14 21:06:58 juju-860e2a-zaza-4546bf52d844-10 systemd[1]: Stopped The Apache HTTP Server. Sep 14 21:06:58 juju-860e2a-zaza-4546bf52d844-10 systemd[1]: Stopping OpenStack Octavia Health Manager (octavia-health-manager)... Sep 14 21:06:58 juju-860e2a-zaza-4546bf52d844-10 systemd[1]: Stopped OpenStack Octavia Health Manager (octavia-health-manager). Sep 14 21:06:58 juju-860e2a-zaza-4546bf52d844-10 systemd[1]: Stopping OpenStack Octavia Housekeeping Manager (octavia-housekeeping)... Which appear to correlate with: 2020-09-14 21:06:14 INFO juju-log Reactive main running for hook update-status 2020-09-14 21:06:17 INFO juju-log Initializing Leadership Layer (is follower) 2020-09-14 21:06:17 DEBUG juju-log tracer> tracer: starting handler dispatch, 127 flags set tracer: set flag amqp.available tracer: set flag amqp.connected tracer: set flag certificates.available tracer: set flag certificates.batch.cert.available tracer: set flag certificates.ca.available tracer: set flag certificates.certs.available tracer: set flag certificates.client.cert.available tracer: set flag certificates.connected tracer: set flag certificates.server.cert.available tracer: set flag certificates.server.certs.available tracer: set flag charm.installed tracer: set flag charm.octavia.enable-ovn-driver tracer: set flag charms.openstack.do-default-amqp.connected tracer: set flag charms.openstack.do-default-certificates.available tracer: set flag charms.openstack.do-default-charm.installed tracer: set flag charms.openstack.do-default-cluster.available tracer: set flag charms.openstack.do-default-config.changed tracer: set flag charms.openstack.do-default-identity-service.connected tracer: set flag charms.openstack.do-default-shared-db.connected tracer: set flag charms.openstack.do-default-update-status tracer: set flag charms.openstack.do-default-upgrade-charm tracer: set flag cluster.available tracer: set flag cluster.connected tracer: set flag config.default.action-managed-upgrade tracer: set flag config.default.amp-image-owner-id tracer: set flag config.default.amp-image-tag tracer: set flag config.default.amp-ssh-key-name tracer: set flag config.default.amp-ssh-pub-key tracer: set flag config.default.create-mgmt-network tracer: set flag config.default.custom-amp-flavor-id tracer: set flag config.default.dns-ha tracer: set flag config.default.haproxy-client-timeout tracer: set flag config.default.haproxy-connect-timeout tracer: set flag config.default.haproxy-queue-timeout tracer: set flag config.default.haproxy-server-timeout tracer: set flag config.default.nagios_context tracer: set flag config.default.nagios_servicegroups tracer: set flag config.default.os-admin-hostname tracer: set flag config.default.os-admin-network tracer: set flag config.default.os-internal-hostname tracer: set flag config.default.os-internal-network tracer: set flag config.default.os-public-hostname tracer: set flag config.default.os-public-network tracer: set flag config.default.region tracer: set flag config.default.ssl_ca tracer: set flag config.default.ssl_cert tracer: set flag config.default.ssl_key tracer: set flag config.default.use-internal-endpoints tracer: set flag config.default.use-policyd-override tracer: set flag config.default.vip_cidr tracer: set flag config.default.vip_iface tracer: set flag config.default.worker-multiplier tracer: set flag config.rendered tracer: set flag config.set.amp-image-tag tracer: set flag config.set.create-mgmt-network tracer: set flag config.set.debug tracer: set flag config.set.lb-mgmt-controller-cacert tracer: set flag config.set.lb-mgmt-controller-cert tracer: set flag config.set.lb-mgmt-issuing-ca-key-passphrase tracer: set flag config.set.lb-mgmt-issuing-ca-private-key tracer: set flag config.set.lb-mgmt-issuing-cacert tracer: set flag config.set.loadbalancer-topology tracer: set flag config.set.nagios_context tracer: set flag config.set.nagios_servicegroups tracer: set flag config.set.openstack-origin tracer: set flag config.set.region tracer: set flag config.set.spare-pool-size tracer: set flag config.set.vip tracer: set flag config.set.vip_cidr tracer: set flag config.set.vip_iface tracer: set flag db.synced tracer: set flag endpoint.certificates.changed tracer: set flag endpoint.certificates.changed.ca tracer: set flag endpoint.certificates.changed.chain tracer: set flag endpoint.certificates.changed.client.cert tracer: set flag endpoint.certificates.changed.client.key tracer: set flag endpoint.certificates.changed.egress-subnets tracer: set flag endpoint.certificates.changed.ingress-address tracer: set flag endpoint.certificates.changed.octavia_0.server.cert tracer: set flag endpoint.certificates.changed.octavia_0.server.key tracer: set flag endpoint.certificates.changed.octavia_1.server.cert tracer: set flag endpoint.certificates.changed.octavia_1.server.key tracer: set flag endpoint.certificates.changed.octavia_2.server.cert tracer: set flag endpoint.certificates.changed.octavia_2.server.key tracer: set flag endpoint.certificates.changed.private-address tracer: set flag endpoint.certificates.joined tracer: set flag endpoint.neutron-api.changed tracer: set flag endpoint.neutron-api.changed.egress-subnets tracer: set flag endpoint.neutron-api.changed.ingress-address tracer: set flag endpoint.neutron-api.changed.private-address tracer: set flag endpoint.neutron-api.joined tracer: set flag endpoint.ovsdb-cms.changed tracer: set flag endpoint.ovsdb-cms.changed.bound-address tracer: set flag endpoint.ovsdb-cms.changed.egress-subnets tracer: set flag endpoint.ovsdb-cms.changed.ingress-address tracer: set flag endpoint.ovsdb-cms.changed.private-address tracer: set flag endpoint.ovsdb-cms.joined tracer: set flag endpoint.ovsdb-subordinate.changed tracer: set flag endpoint.ovsdb-subordinate.changed.chassis-name tracer: set flag endpoint.ovsdb-subordinate.changed.egress-subnets tracer: set flag endpoint.ovsdb-subordinate.changed.ingress-address tracer: set flag endpoint.ovsdb-subordinate.changed.ovn-configured tracer: set flag endpoint.ovsdb-subordinate.changed.private-address tracer: set flag endpoint.ovsdb-subordinate.joined tracer: set flag ha.available tracer: set flag ha.connected tracer: set flag haproxy.stat.password tracer: set flag identity-service.available tracer: set flag identity-service.available.auth tracer: set flag identity-service.connected tracer: set flag leadership.set.amp-boot-network-list tracer: set flag leadership.set.amp-flavor-id tracer: set flag leadership.set.amp-secgroup-list tracer: set flag leadership.set.controller-ip-port-list tracer: set flag leadership.set.db-sync-done tracer: set flag leadership.set.heartbeat-key tracer: set flag neutron-api.available tracer: set flag octavia-installed tracer: set flag ovsdb-cms.available tracer: set flag ovsdb-cms.connected tracer: set flag ovsdb-subordinate.available tracer: set flag ovsdb-subordinate.connected tracer: set flag sdn-subordinate.available tracer: set flag sdn-subordinate.connected tracer: set flag shared-db.available tracer: set flag shared-db.connected tracer: set flag ssl.enabled 2020-09-14 21:06:17 DEBUG juju-log tracer> tracer: hooks phase, 1 handlers queued tracer: ++ queue handler reactive/layer_openstack.py:59:default_update_status 2020-09-14 21:06:17 INFO juju-log Invoking reactive handler: reactive/layer_openstack.py:59:default_update_status 2020-09-14 21:06:17 DEBUG juju-log tracer: set flag run-default-update-status 2020-09-14 21:06:17 DEBUG juju-log tracer: set flag is-update-status-hook 2020-09-14 21:06:18 DEBUG juju-log tracer> tracer: main dispatch loop, 16 handlers queued tracer: ++ queue handler hooks/relations/ovsdb-cms/requires.py:43:joined:ovsdb-cms tracer: ++ queue handler hooks/relations/ovsdb-subordinate/requires.py:129:joined:ovsdb-subordinate tracer: ++ queue handler hooks/relations/tls-certificates/requires.py:79:joined:certificates tracer: ++ queue handler reactive/layer_openstack.py:121:default_request_certificates tracer: ++ queue handler reactive/layer_openstack.py:77:check_really_is_update_status tracer: ++ queue handler reactive/layer_openstack.py:88:run_default_update_status tracer: ++ queue handler reactive/layer_openstack_api.py:20:default_setup_database tracer: ++ queue handler reactive/layer_openstack_api.py:37:default_setup_endpoint_connection tracer: ++ queue handler reactive/layer_openstack_api.py:54:default_update_peers tracer: ++ queue handler reactive/layer_openstack_api.py:6:default_amqp_connection tracer: ++ queue handler reactive/octavia_handlers.py:102:setup_neutron_lbaas_proxy tracer: ++ queue handler reactive/octavia_handlers.py:123:setup_hm_port tracer: ++ queue handler reactive/octavia_handlers.py:181:render tracer: ++ queue handler reactive/octavia_handlers.py:47:sdn_joined tracer: ++ queue handler reactive/octavia_handlers.py:66:maybe_enable_ovn_driver tracer: ++ queue handler reactive/octavia_handlers.py:76:setup_endpoint_connection 2020-09-14 21:06:18 INFO juju-log Invoking reactive handler: reactive/layer_openstack.py:77:check_really_is_update_status 2020-09-14 21:06:18 INFO juju-log Invoking reactive handler: reactive/layer_openstack.py:88:run_default_update_status 2020-09-14 21:06:18 DEBUG juju-log tracer> tracer: cleared flag run-default-update-status tracer: -- dequeue handler reactive/layer_openstack.py:88:run_default_update_status 2020-09-14 21:06:18 INFO juju-log Invoking reactive handler: reactive/layer_openstack.py:121:default_request_certificates 2020-09-14 21:06:20 INFO juju-log Invoking reactive handler: reactive/layer_openstack_api.py:6:default_amqp_connection 2020-09-14 21:06:20 INFO juju-log Invoking reactive handler: reactive/layer_openstack_api.py:20:default_setup_database 2020-09-14 21:06:21 INFO juju-log Invoking reactive handler: reactive/layer_openstack_api.py:37:default_setup_endpoint_connection 2020-09-14 21:06:22 INFO juju-log Invoking reactive handler: reactive/layer_openstack_api.py:54:default_update_peers 2020-09-14 21:06:22 INFO juju-log Invoking reactive handler: reactive/octavia_handlers.py:47:sdn_joined 2020-09-14 21:06:22 INFO juju-log Invoking reactive handler: reactive/octavia_handlers.py:66:maybe_enable_ovn_driver 2020-09-14 21:06:23 INFO juju-log Installing [] with options: ['--option=Dpkg::Options::=--force-confold'] 2020-09-14 21:06:23 DEBUG update-status Reading package lists... 2020-09-14 21:06:23 DEBUG update-status Building dependency tree... 2020-09-14 21:06:23 DEBUG update-status Reading state information... 2020-09-14 21:06:24 DEBUG update-status The following packages were automatically installed and are no longer required: 2020-09-14 21:06:24 DEBUG update-status grub-pc-bin python3-xdg 2020-09-14 21:06:24 DEBUG update-status Use 'apt autoremove' to remove them. 2020-09-14 21:06:24 DEBUG update-status 0 upgraded, 0 newly installed, 0 to remove and 8 not upgraded. 2020-09-14 21:06:24 DEBUG update-status Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2020-09-14 21:06:24 DEBUG update-status Hit:2 http://nova.clouds.archive.ubuntu.com/ubuntu bionic InRelease 2020-09-14 21:06:24 DEBUG update-status Hit:3 http://nova.clouds.archive.ubuntu.com/ubuntu bionic-updates InRelease 2020-09-14 21:06:24 DEBUG update-status Hit:4 http://nova.clouds.archive.ubuntu.com/ubuntu bionic-backports InRelease 2020-09-14 21:06:24 DEBUG update-status Hit:5 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-proposed/ussuri InRelease 2020-09-14 21:06:26 DEBUG update-status Reading package lists... 2020-09-14 21:06:31 INFO juju-log Running maybe_do_policyd_overrides 2020-09-14 21:06:31 DEBUG juju-log Cleaning path: /etc/octavia/policy.d 2020-09-14 21:06:31 INFO juju-log Adding user systemd-network to group octavia 2020-09-14 21:06:31 DEBUG update-status Adding user systemd-network to group octavia 2020-09-14 21:06:31 DEBUG update-status inactive 2020-09-14 21:06:31 DEBUG update-status Synchronizing state of octavia-api.service with SysV service script with /lib/systemd/systemd-sysv-install. 2020-09-14 21:06:31 DEBUG update-status Executing: /lib/systemd/systemd-sysv-install disable octavia-api 2020-09-14 21:06:32 DEBUG update-status Unit /etc/systemd/system/octavia-api.service is masked, ignoring. 2020-09-14 21:06:33 INFO juju-log Invoking reactive handler: reactive/octavia_handlers.py:76:setup_endpoint_connection 2020-09-14 21:06:34 INFO juju-log Invoking reactive handler: reactive/octavia_handlers.py:102:setup_neutron_lbaas_proxy 2020-09-14 21:06:34 INFO juju-log Invoking reactive handler: reactive/octavia_handlers.py:123:setup_hm_port 2020-09-14 21:06:40 INFO juju-log toggling port 943c3336-8299-4126-98fc-f35f369a6463 (admin_state_up: True status: DOWN) 2020-09-14 21:06:47 INFO juju-log Invoking reactive handler: reactive/octavia_handlers.py:181:render 2020-09-14 21:06:50 DEBUG juju-log Changing permissions on existing content: 33184 -> 416 2020-09-14 21:06:51 INFO juju-log Making dir /etc/octavia/certs root:octavia 750 2020-09-14 21:06:52 DEBUG juju-log Changing permissions on existing content: 33056 -> 288 2020-09-14 21:06:52 INFO juju-log Making dir /etc/octavia/certs root:octavia 750 2020-09-14 21:06:52 DEBUG juju-log Changing permissions on existing content: 33056 -> 288 2020-09-14 21:06:52 INFO juju-log Making dir /etc/octavia/certs root:octavia 750 2020-09-14 21:06:52 DEBUG juju-log Changing permissions on existing content: 33056 -> 288 2020-09-14 21:06:52 INFO juju-log Making dir /etc/octavia/certs root:octavia 750 2020-09-14 21:06:52 DEBUG juju-log Changing permissions on existing content: 33056 -> 288 2020-09-14 21:06:52 INFO juju-log Making dir /etc/octavia/certs root:octavia 750 2020-09-14 21:06:53 DEBUG juju-log Changing permissions on existing content: 33056 -> 288 2020-09-14 21:06:54 DEBUG juju-log Writing file /etc/octavia/octavia.conf root:octavia 640 2020-09-14 21:06:54 DEBUG update-status none 2020-09-14 21:06:54 DEBUG juju-log Changing permissions on existing content: 33184 -> 416 2020-09-14 21:06:54 DEBUG juju-log Changing permissions on existing content: 33184 -> 416 2020-09-14 21:06:55 WARNING juju-log Not adding haproxy listen stanza for octavia-api_int port is already in use 2020-09-14 21:06:55 WARNING juju-log Not adding haproxy listen stanza for octavia-api_public port is already in use 2020-09-14 21:06:55 DEBUG juju-log Changing permissions on existing content: 33184 -> 416 2020-09-14 21:06:56 DEBUG juju-log Changing permissions on existing content: 33184 -> 416