Comment 4 for bug 1884449

Revision history for this message
Alex Kavanagh (ajkavanagh) wrote :

After a quick initial look at the logs, it stalled during the unpacking of the wheelhouse; it seems that the extract/install of the wheelhouse items took a LONG time (some 5-10 minutes each). Octavia was being installed in a LXD container.

It looks like each ceph-osd "update-status" hook 'stalled' the octavia install:

unit-octavia-0: 09:54:02 DEBUG unit.octavia/0.install Processing ./wheelhouse/python-neutronclient-7.1.0.tar.gz
unit-ceph-osd-0: 09:56:40 DEBUG unit.ceph-osd/0.juju-log Hardening function 'update_status'
unit-ceph-osd-0: 09:56:41 DEBUG unit.ceph-osd/0.juju-log No hardening applied to 'update_status'
unit-ceph-osd-0: 09:56:41 INFO unit.ceph-osd/0.juju-log Updating status.
unit-ceph-osd-0: 09:56:41 INFO juju.worker.uniter.operation ran "update-status" hook
unit-octavia-0: 10:00:11 DEBUG unit.octavia/0.install Processing ./wheelhouse/netifaces-0.10.9.tar.gz
unit-octavia-0: 10:00:12 DEBUG unit.octavia/0.install Processing ./wheelhouse/oslo.config-7.0.0.tar.gz
unit-ceph-osd-0: 10:02:34 DEBUG unit.ceph-osd/0.juju-log Hardening function 'update_status'
unit-ceph-osd-0: 10:02:34 DEBUG unit.ceph-osd/0.juju-log No hardening applied to 'update_status'
unit-ceph-osd-0: 10:02:34 INFO unit.ceph-osd/0.juju-log Updating status.
unit-ceph-osd-0: 10:02:35 INFO juju.worker.uniter.operation ran "update-status" hook
unit-octavia-0: 10:06:21 DEBUG unit.octavia/0.install Processing ./wheelhouse/pecan-1.3.3.tar.gz
unit-octavia-0: 10:06:21 DEBUG unit.octavia/0.install Processing ./wheelhouse/charms.reactive-1.3.0.tar.gz
unit-octavia-0: 10:06:22 DEBUG unit.octavia/0.install Processing ./wheelhouse/cffi-1.14.0.tar.gz
unit-octavia-0: 10:06:22 DEBUG unit.octavia/0.install Processing ./wheelhouse/osc-lib-2.0.0.tar.gz
unit-ceph-osd-0: 10:07:37 DEBUG unit.ceph-osd/0.juju-log Hardening function 'update_status'
unit-ceph-osd-0: 10:07:37 DEBUG unit.ceph-osd/0.juju-log No hardening applied to 'update_status'
unit-ceph-osd-0: 10:07:37 INFO unit.ceph-osd/0.juju-log Updating status.
unit-ceph-osd-0: 10:07:38 INFO juju.worker.uniter.operation ran "update-status" hook
unit-octavia-0: 10:12:31 DEBUG unit.octavia/0.install Processing ./wheelhouse/monotonic-1.5.tar.gz
unit-octavia-0: 10:12:31 DEBUG unit.octavia/0.install Processing ./wheelhouse/jmespath-0.10.0.tar.gz
unit-octavia-0: 10:12:32 DEBUG unit.octavia/0.install Processing ./wheelhouse/os-service-types-1.7.0.tar.gz
unit-ceph-osd-0: 10:13:02 DEBUG unit.ceph-osd/0.juju-log Hardening function 'update_status'
unit-ceph-osd-0: 10:13:02 DEBUG unit.ceph-osd/0.juju-log No hardening applied to 'update_status'
unit-ceph-osd-0: 10:13:03 INFO unit.ceph-osd/0.juju-log Updating status.
unit-ceph-osd-0: 10:13:03 INFO juju.worker.uniter.operation ran "update-status" hook
unit-ceph-osd-0: 10:18:33 DEBUG unit.ceph-osd/0.juju-log Hardening function 'update_status'
unit-ceph-osd-0: 10:18:33 DEBUG unit.ceph-osd/0.juju-log No hardening applied to 'update_status'
unit-ceph-osd-0: 10:18:33 INFO unit.ceph-osd/0.juju-log Updating status.
unit-ceph-osd-0: 10:18:34 INFO juju.worker.uniter.operation ran "update-status" hook
unit-octavia-0: 10:18:41 DEBUG unit.octavia/0.install Processing ./wheelhouse/PasteDeploy-2.1.0.tar.gz
unit-octavia-0: 10:18:41 DEBUG unit.octavia/0.install Processing ./wheelhouse/pytest-runner-5.2.tar.gz
unit-octavia-0: 10:18:41 DEBUG unit.octavia/0.install Installing build dependencies: started
unit-octavia-0: 10:18:42 DEBUG unit.octavia/0.install Installing build dependencies: finished with status 'done'
unit-octavia-0: 10:18:42 DEBUG unit.octavia/0.install Getting requirements to build wheel: started
unit-octavia-0: 10:18:42 DEBUG unit.octavia/0.install Getting requirements to build wheel: finished with status 'done'
unit-octavia-0: 10:18:42 DEBUG unit.octavia/0.install Preparing wheel metadata: started
unit-octavia-0: 10:18:43 DEBUG unit.octavia/0.install Preparing wheel metadata: finished with status 'done'
unit-octavia-0: 10:18:43 DEBUG unit.octavia/0.install Processing ./wheelhouse/chardet-3.0.4.tar.gz
unit-octavia-0: 10:18:43 DEBUG unit.octavia/0.install Processing ./wheelhouse/sqlalchemy-migrate-0.13.0.tar.gz
unit-ceph-osd-0: 10:23:10 DEBUG unit.ceph-osd/0.juju-log Hardening function 'update_status'
unit-ceph-osd-0: 10:23:10 DEBUG unit.ceph-osd/0.juju-log No hardening applied to 'update_status'
unit-ceph-osd-0: 10:23:10 INFO unit.ceph-osd/0.juju-log Updating status.
unit-ceph-osd-0: 10:23:11 INFO juju.worker.uniter.operation ran "update-status" hook
unit-octavia-0: 10:24:52 DEBUG unit.octavia/0.install Processing ./wheelhouse/testtools-2.4.0.tar.gz

Obviously, ceph is on bare-metal and octavia is in an LXD on the same host, in the configuration that's been provided.

Which version of Juju was being used to deploy, please?
Please could you attach full bundles that demonstrate the issue for the bionic and focal variables?

Thanks.