Intermittent service restart during update-status hook

While investigating gate failure in https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_full/openstack/charm-octavia/751663/1/6876/consoleText.test_charm_func_full_9986.txt

I found:
Sep 14 21:06:32 juju-860e2a-zaza-4546bf52d844-10 systemd[1]: message repeated 4 times: [ Reloading.]
Sep 14 21:06:56 juju-860e2a-zaza-4546bf52d844-10 systemd[1]: Stopping The Apache HTTP Server...
Sep 14 21:06:58 juju-860e2a-zaza-4546bf52d844-10 systemd[1]: Stopped The Apache HTTP Server.
Sep 14 21:06:58 juju-860e2a-zaza-4546bf52d844-10 systemd[1]: Stopping OpenStack Octavia Health Manager (octavia-health-manager)...
Sep 14 21:06:58 juju-860e2a-zaza-4546bf52d844-10 systemd[1]: Stopped OpenStack Octavia Health Manager (octavia-health-manager).
Sep 14 21:06:58 juju-860e2a-zaza-4546bf52d844-10 systemd[1]: Stopping OpenStack Octavia Housekeeping Manager (octavia-housekeeping)...

Which appear to correlate with a update-status hook run (see below). However, attempts to reproduce by actually running the update-status hook manually are not successful.

2020-09-14 21:06:14 INFO juju-log Reactive main running for hook update-status
2020-09-14 21:06:17 INFO juju-log Initializing Leadership Layer (is follower)
2020-09-14 21:06:17 DEBUG juju-log tracer>
tracer: starting handler dispatch, 127 flags set
tracer: set flag amqp.available
tracer: set flag amqp.connected
tracer: set flag certificates.available
tracer: set flag certificates.batch.cert.available
tracer: set flag certificates.ca.available
tracer: set flag certificates.certs.available
tracer: set flag certificates.client.cert.available
tracer: set flag certificates.connected
tracer: set flag certificates.server.cert.available
tracer: set flag certificates.server.certs.available
tracer: set flag charm.installed
tracer: set flag charm.octavia.enable-ovn-driver
tracer: set flag charms.openstack.do-default-amqp.connected
tracer: set flag charms.openstack.do-default-certificates.available
tracer: set flag charms.openstack.do-default-charm.installed
tracer: set flag charms.openstack.do-default-cluster.available
tracer: set flag charms.openstack.do-default-config.changed
tracer: set flag charms.openstack.do-default-identity-service.connected
tracer: set flag charms.openstack.do-default-shared-db.connected
tracer: set flag charms.openstack.do-default-update-status
tracer: set flag charms.openstack.do-default-upgrade-charm
tracer: set flag cluster.available
tracer: set flag cluster.connected
tracer: set flag config.default.action-managed-upgrade
tracer: set flag config.default.amp-image-owner-id
tracer: set flag config.default.amp-image-tag
tracer: set flag config.default.amp-ssh-key-name
tracer: set flag config.default.amp-ssh-pub-key
tracer: set flag config.default.create-mgmt-network
tracer: set flag config.default.custom-amp-flavor-id
tracer: set flag config.default.dns-ha
tracer: set flag config.default.haproxy-client-timeout
tracer: set flag config.default.haproxy-connect-timeout
tracer: set flag config.default.haproxy-queue-timeout
tracer: set flag config.default.haproxy-server-timeout
tracer: set flag config.default.nagios_context
tracer: set flag config.default.nagios_servicegroups
tracer: set flag config.default.os-admin-hostname
tracer: set flag config.default.os-admin-network
tracer: set flag config.default.os-internal-hostname
tracer: set flag config.default.os-internal-network
tracer: set flag config.default.os-public-hostname
tracer: set flag config.default.os-public-network
tracer: set flag config.default.region
tracer: set flag config.default.ssl_ca
tracer: set flag config.default.ssl_cert
tracer: set flag config.default.ssl_key
tracer: set flag config.default.use-internal-endpoints
tracer: set flag config.default.use-policyd-override
tracer: set flag config.default.vip_cidr
tracer: set flag config.default.vip_iface
tracer: set flag config.default.worker-multiplier
tracer: set flag config.rendered
tracer: set flag config.set.amp-image-tag
tracer: set flag config.set.create-mgmt-network
tracer: set flag config.set.debug
tracer: set flag config.set.lb-mgmt-controller-cacert
tracer: set flag config.set.lb-mgmt-controller-cert
tracer: set flag config.set.lb-mgmt-issuing-ca-key-passphrase
tracer: set flag config.set.lb-mgmt-issuing-ca-private-key
tracer: set flag config.set.lb-mgmt-issuing-cacert
tracer: set flag config.set.loadbalancer-topology
tracer: set flag config.set.nagios_context
tracer: set flag config.set.nagios_servicegroups
tracer: set flag config.set.openstack-origin
tracer: set flag config.set.region
tracer: set flag config.set.spare-pool-size
tracer: set flag config.set.vip
tracer: set flag config.set.vip_cidr
tracer: set flag config.set.vip_iface
tracer: set flag db.synced
tracer: set flag endpoint.certificates.changed
tracer: set flag endpoint.certificates.changed.ca
tracer: set flag endpoint.certificates.changed.chain
tracer: set flag endpoint.certificates.changed.client.cert
tracer: set flag endpoint.certificates.changed.client.key
tracer: set flag endpoint.certificates.changed.egress-subnets
tracer: set flag endpoint.certificates.changed.ingress-address
tracer: set flag endpoint.certificates.changed.octavia_0.server.cert
tracer: set flag endpoint.certificates.changed.octavia_0.server.key
tracer: set flag endpoint.certificates.changed.octavia_1.server.cert
tracer: set flag endpoint.certificates.changed.octavia_1.server.key
tracer: set flag endpoint.certificates.changed.octavia_2.server.cert
tracer: set flag endpoint.certificates.changed.octavia_2.server.key
tracer: set flag endpoint.certificates.changed.private-address
tracer: set flag endpoint.certificates.joined
tracer: set flag endpoint.neutron-api.changed
tracer: set flag endpoint.neutron-api.changed.egress-subnets
tracer: set flag endpoint.neutron-api.changed.ingress-address
tracer: set flag endpoint.neutron-api.changed.private-address
tracer: set flag endpoint.neutron-api.joined
tracer: set flag endpoint.ovsdb-cms.changed
tracer: set flag endpoint.ovsdb-cms.changed.bound-address
tracer: set flag endpoint.ovsdb-cms.changed.egress-subnets
tracer: set flag endpoint.ovsdb-cms.changed.ingress-address
tracer: set flag endpoint.ovsdb-cms.changed.private-address
tracer: set flag endpoint.ovsdb-cms.joined
tracer: set flag endpoint.ovsdb-subordinate.changed
tracer: set flag endpoint.ovsdb-subordinate.changed.chassis-name
tracer: set flag endpoint.ovsdb-subordinate.changed.egress-subnets
tracer: set flag endpoint.ovsdb-subordinate.changed.ingress-address
tracer: set flag endpoint.ovsdb-subordinate.changed.ovn-configured
tracer: set flag endpoint.ovsdb-subordinate.changed.private-address
tracer: set flag endpoint.ovsdb-subordinate.joined
tracer: set flag ha.available
tracer: set flag ha.connected
tracer: set flag haproxy.stat.password
tracer: set flag identity-service.available
tracer: set flag identity-service.available.auth
tracer: set flag identity-service.connected
tracer: set flag leadership.set.amp-boot-network-list
tracer: set flag leadership.set.amp-flavor-id
tracer: set flag leadership.set.amp-secgroup-list
tracer: set flag leadership.set.controller-ip-port-list
tracer: set flag leadership.set.db-sync-done
tracer: set flag leadership.set.heartbeat-key
tracer: set flag neutron-api.available
tracer: set flag octavia-installed
tracer: set flag ovsdb-cms.available
tracer: set flag ovsdb-cms.connected
tracer: set flag ovsdb-subordinate.available
tracer: set flag ovsdb-subordinate.connected
tracer: set flag sdn-subordinate.available
tracer: set flag sdn-subordinate.connected
tracer: set flag shared-db.available
tracer: set flag shared-db.connected
tracer: set flag ssl.enabled
2020-09-14 21:06:17 DEBUG juju-log tracer>
tracer: hooks phase, 1 handlers queued
tracer: ++ queue handler reactive/layer_openstack.py:59:default_update_status
2020-09-14 21:06:17 INFO juju-log Invoking reactive handler: reactive/layer_openstack.py:59:default_update_status
2020-09-14 21:06:17 DEBUG juju-log tracer: set flag run-default-update-status
2020-09-14 21:06:17 DEBUG juju-log tracer: set flag is-update-status-hook
2020-09-14 21:06:18 DEBUG juju-log tracer>
tracer: main dispatch loop, 16 handlers queued
tracer: ++ queue handler hooks/relations/ovsdb-cms/requires.py:43:joined:ovsdb-cms
tracer: ++ queue handler hooks/relations/ovsdb-subordinate/requires.py:129:joined:ovsdb-subordinate
tracer: ++ queue handler hooks/relations/tls-certificates/requires.py:79:joined:certificates
tracer: ++ queue handler reactive/layer_openstack.py:121:default_request_certificates
tracer: ++ queue handler reactive/layer_openstack.py:77:check_really_is_update_status
tracer: ++ queue handler reactive/layer_openstack.py:88:run_default_update_status
tracer: ++ queue handler reactive/layer_openstack_api.py:20:default_setup_database
tracer: ++ queue handler reactive/layer_openstack_api.py:37:default_setup_endpoint_connection
tracer: ++ queue handler reactive/layer_openstack_api.py:54:default_update_peers
tracer: ++ queue handler reactive/layer_openstack_api.py:6:default_amqp_connection
tracer: ++ queue handler reactive/octavia_handlers.py:102:setup_neutron_lbaas_proxy
tracer: ++ queue handler reactive/octavia_handlers.py:123:setup_hm_port
tracer: ++ queue handler reactive/octavia_handlers.py:181:render
tracer: ++ queue handler reactive/octavia_handlers.py:47:sdn_joined
tracer: ++ queue handler reactive/octavia_handlers.py:66:maybe_enable_ovn_driver
tracer: ++ queue handler reactive/octavia_handlers.py:76:setup_endpoint_connection
2020-09-14 21:06:18 INFO juju-log Invoking reactive handler: reactive/layer_openstack.py:77:check_really_is_update_status
2020-09-14 21:06:18 INFO juju-log Invoking reactive handler: reactive/layer_openstack.py:88:run_default_update_status
2020-09-14 21:06:18 DEBUG juju-log tracer>
tracer: cleared flag run-default-update-status
tracer: -- dequeue handler reactive/layer_openstack.py:88:run_default_update_status
2020-09-14 21:06:18 INFO juju-log Invoking reactive handler: reactive/layer_openstack.py:121:default_request_certificates
2020-09-14 21:06:20 INFO juju-log Invoking reactive handler: reactive/layer_openstack_api.py:6:default_amqp_connection
2020-09-14 21:06:20 INFO juju-log Invoking reactive handler: reactive/layer_openstack_api.py:20:default_setup_database
2020-09-14 21:06:21 INFO juju-log Invoking reactive handler: reactive/layer_openstack_api.py:37:default_setup_endpoint_connection
2020-09-14 21:06:22 INFO juju-log Invoking reactive handler: reactive/layer_openstack_api.py:54:default_update_peers
2020-09-14 21:06:22 INFO juju-log Invoking reactive handler: reactive/octavia_handlers.py:47:sdn_joined
2020-09-14 21:06:22 INFO juju-log Invoking reactive handler: reactive/octavia_handlers.py:66:maybe_enable_ovn_driver
2020-09-14 21:06:23 INFO juju-log Installing [] with options: ['--option=Dpkg::Options::=--force-confold']
2020-09-14 21:06:23 DEBUG update-status Reading package lists...
2020-09-14 21:06:23 DEBUG update-status Building dependency tree...
2020-09-14 21:06:23 DEBUG update-status Reading state information...
2020-09-14 21:06:24 DEBUG update-status The following packages were automatically installed and are no longer required:
2020-09-14 21:06:24 DEBUG update-status grub-pc-bin python3-xdg
2020-09-14 21:06:24 DEBUG update-status Use 'apt autoremove' to remove them.
2020-09-14 21:06:24 DEBUG update-status 0 upgraded, 0 newly installed, 0 to remove and 8 not upgraded.
2020-09-14 21:06:24 DEBUG update-status Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease
2020-09-14 21:06:24 DEBUG update-status Hit:2 http://nova.clouds.archive.ubuntu.com/ubuntu bionic InRelease
2020-09-14 21:06:24 DEBUG update-status Hit:3 http://nova.clouds.archive.ubuntu.com/ubuntu bionic-updates InRelease
2020-09-14 21:06:24 DEBUG update-status Hit:4 http://nova.clouds.archive.ubuntu.com/ubuntu bionic-backports InRelease
2020-09-14 21:06:24 DEBUG update-status Hit:5 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-proposed/ussuri InRelease
2020-09-14 21:06:26 DEBUG update-status Reading package lists...
2020-09-14 21:06:31 INFO juju-log Running maybe_do_policyd_overrides
2020-09-14 21:06:31 DEBUG juju-log Cleaning path: /etc/octavia/policy.d
2020-09-14 21:06:31 INFO juju-log Adding user systemd-network to group octavia
2020-09-14 21:06:31 DEBUG update-status Adding user systemd-network to group octavia
2020-09-14 21:06:31 DEBUG update-status inactive
2020-09-14 21:06:31 DEBUG update-status Synchronizing state of octavia-api.service with SysV service script with /lib/systemd/systemd-sysv-install.
2020-09-14 21:06:31 DEBUG update-status Executing: /lib/systemd/systemd-sysv-install disable octavia-api
2020-09-14 21:06:32 DEBUG update-status Unit /etc/systemd/system/octavia-api.service is masked, ignoring.
2020-09-14 21:06:33 INFO juju-log Invoking reactive handler: reactive/octavia_handlers.py:76:setup_endpoint_connection
2020-09-14 21:06:34 INFO juju-log Invoking reactive handler: reactive/octavia_handlers.py:102:setup_neutron_lbaas_proxy
2020-09-14 21:06:34 INFO juju-log Invoking reactive handler: reactive/octavia_handlers.py:123:setup_hm_port
2020-09-14 21:06:40 INFO juju-log toggling port 943c3336-8299-4126-98fc-f35f369a6463 (admin_state_up: True status: DOWN)
2020-09-14 21:06:47 INFO juju-log Invoking reactive handler: reactive/octavia_handlers.py:181:render
2020-09-14 21:06:50 DEBUG juju-log Changing permissions on existing content: 33184 -> 416
2020-09-14 21:06:51 INFO juju-log Making dir /etc/octavia/certs root:octavia 750
2020-09-14 21:06:52 DEBUG juju-log Changing permissions on existing content: 33056 -> 288
2020-09-14 21:06:52 INFO juju-log Making dir /etc/octavia/certs root:octavia 750
2020-09-14 21:06:52 DEBUG juju-log Changing permissions on existing content: 33056 -> 288
2020-09-14 21:06:52 INFO juju-log Making dir /etc/octavia/certs root:octavia 750
2020-09-14 21:06:52 DEBUG juju-log Changing permissions on existing content: 33056 -> 288
2020-09-14 21:06:52 INFO juju-log Making dir /etc/octavia/certs root:octavia 750
2020-09-14 21:06:52 DEBUG juju-log Changing permissions on existing content: 33056 -> 288
2020-09-14 21:06:52 INFO juju-log Making dir /etc/octavia/certs root:octavia 750
2020-09-14 21:06:53 DEBUG juju-log Changing permissions on existing content: 33056 -> 288
2020-09-14 21:06:54 DEBUG juju-log Writing file /etc/octavia/octavia.conf root:octavia 640
2020-09-14 21:06:54 DEBUG update-status none
2020-09-14 21:06:54 DEBUG juju-log Changing permissions on existing content: 33184 -> 416
2020-09-14 21:06:54 DEBUG juju-log Changing permissions on existing content: 33184 -> 416
2020-09-14 21:06:55 WARNING juju-log Not adding haproxy listen stanza for octavia-api_int port is already in use
2020-09-14 21:06:55 WARNING juju-log Not adding haproxy listen stanza for octavia-api_public port is already in use
2020-09-14 21:06:55 DEBUG juju-log Changing permissions on existing content: 33184 -> 416
2020-09-14 21:06:56 DEBUG juju-log Changing permissions on existing content: 33184 -> 416