After a quick initial look at the logs, it stalled during the unpacking of the wheelhouse; it seems that the extract/install of the wheelhouse items took a LONG time (some 5-10 minutes each). Octavia was being installed in a LXD container.
It looks like each ceph-osd "update-status" hook 'stalled' the octavia install:
unit-octavia-0: 09:54:02 DEBUG unit.octavia/0.install Processing ./wheelhouse/python-neutronclient-7.1.0.tar.gz
unit-ceph-osd-0: 09:56:40 DEBUG unit.ceph-osd/0.juju-log Hardening function 'update_status'
unit-ceph-osd-0: 09:56:41 DEBUG unit.ceph-osd/0.juju-log No hardening applied to 'update_status'
unit-ceph-osd-0: 09:56:41 INFO unit.ceph-osd/0.juju-log Updating status.
unit-ceph-osd-0: 09:56:41 INFO juju.worker.uniter.operation ran "update-status" hook
unit-octavia-0: 10:00:11 DEBUG unit.octavia/0.install Processing ./wheelhouse/netifaces-0.10.9.tar.gz
unit-octavia-0: 10:00:12 DEBUG unit.octavia/0.install Processing ./wheelhouse/oslo.config-7.0.0.tar.gz
unit-ceph-osd-0: 10:02:34 DEBUG unit.ceph-osd/0.juju-log Hardening function 'update_status'
unit-ceph-osd-0: 10:02:34 DEBUG unit.ceph-osd/0.juju-log No hardening applied to 'update_status'
unit-ceph-osd-0: 10:02:34 INFO unit.ceph-osd/0.juju-log Updating status.
unit-ceph-osd-0: 10:02:35 INFO juju.worker.uniter.operation ran "update-status" hook
unit-octavia-0: 10:06:21 DEBUG unit.octavia/0.install Processing ./wheelhouse/pecan-1.3.3.tar.gz
unit-octavia-0: 10:06:21 DEBUG unit.octavia/0.install Processing ./wheelhouse/charms.reactive-1.3.0.tar.gz
unit-octavia-0: 10:06:22 DEBUG unit.octavia/0.install Processing ./wheelhouse/cffi-1.14.0.tar.gz
unit-octavia-0: 10:06:22 DEBUG unit.octavia/0.install Processing ./wheelhouse/osc-lib-2.0.0.tar.gz
unit-ceph-osd-0: 10:07:37 DEBUG unit.ceph-osd/0.juju-log Hardening function 'update_status'
unit-ceph-osd-0: 10:07:37 DEBUG unit.ceph-osd/0.juju-log No hardening applied to 'update_status'
unit-ceph-osd-0: 10:07:37 INFO unit.ceph-osd/0.juju-log Updating status.
unit-ceph-osd-0: 10:07:38 INFO juju.worker.uniter.operation ran "update-status" hook
unit-octavia-0: 10:12:31 DEBUG unit.octavia/0.install Processing ./wheelhouse/monotonic-1.5.tar.gz
unit-octavia-0: 10:12:31 DEBUG unit.octavia/0.install Processing ./wheelhouse/jmespath-0.10.0.tar.gz
unit-octavia-0: 10:12:32 DEBUG unit.octavia/0.install Processing ./wheelhouse/os-service-types-1.7.0.tar.gz
unit-ceph-osd-0: 10:13:02 DEBUG unit.ceph-osd/0.juju-log Hardening function 'update_status'
unit-ceph-osd-0: 10:13:02 DEBUG unit.ceph-osd/0.juju-log No hardening applied to 'update_status'
unit-ceph-osd-0: 10:13:03 INFO unit.ceph-osd/0.juju-log Updating status.
unit-ceph-osd-0: 10:13:03 INFO juju.worker.uniter.operation ran "update-status" hook
unit-ceph-osd-0: 10:18:33 DEBUG unit.ceph-osd/0.juju-log Hardening function 'update_status'
unit-ceph-osd-0: 10:18:33 DEBUG unit.ceph-osd/0.juju-log No hardening applied to 'update_status'
unit-ceph-osd-0: 10:18:33 INFO unit.ceph-osd/0.juju-log Updating status.
unit-ceph-osd-0: 10:18:34 INFO juju.worker.uniter.operation ran "update-status" hook
unit-octavia-0: 10:18:41 DEBUG unit.octavia/0.install Processing ./wheelhouse/PasteDeploy-2.1.0.tar.gz
unit-octavia-0: 10:18:41 DEBUG unit.octavia/0.install Processing ./wheelhouse/pytest-runner-5.2.tar.gz
unit-octavia-0: 10:18:41 DEBUG unit.octavia/0.install Installing build dependencies: started
unit-octavia-0: 10:18:42 DEBUG unit.octavia/0.install Installing build dependencies: finished with status 'done'
unit-octavia-0: 10:18:42 DEBUG unit.octavia/0.install Getting requirements to build wheel: started
unit-octavia-0: 10:18:42 DEBUG unit.octavia/0.install Getting requirements to build wheel: finished with status 'done'
unit-octavia-0: 10:18:42 DEBUG unit.octavia/0.install Preparing wheel metadata: started
unit-octavia-0: 10:18:43 DEBUG unit.octavia/0.install Preparing wheel metadata: finished with status 'done'
unit-octavia-0: 10:18:43 DEBUG unit.octavia/0.install Processing ./wheelhouse/chardet-3.0.4.tar.gz
unit-octavia-0: 10:18:43 DEBUG unit.octavia/0.install Processing ./wheelhouse/sqlalchemy-migrate-0.13.0.tar.gz
unit-ceph-osd-0: 10:23:10 DEBUG unit.ceph-osd/0.juju-log Hardening function 'update_status'
unit-ceph-osd-0: 10:23:10 DEBUG unit.ceph-osd/0.juju-log No hardening applied to 'update_status'
unit-ceph-osd-0: 10:23:10 INFO unit.ceph-osd/0.juju-log Updating status.
unit-ceph-osd-0: 10:23:11 INFO juju.worker.uniter.operation ran "update-status" hook
unit-octavia-0: 10:24:52 DEBUG unit.octavia/0.install Processing ./wheelhouse/testtools-2.4.0.tar.gz
Obviously, ceph is on bare-metal and octavia is in an LXD on the same host, in the configuration that's been provided.
Which version of Juju was being used to deploy, please?
Please could you attach full bundles that demonstrate the issue for the bionic and focal variables?
After a quick initial look at the logs, it stalled during the unpacking of the wheelhouse; it seems that the extract/install of the wheelhouse items took a LONG time (some 5-10 minutes each). Octavia was being installed in a LXD container.
It looks like each ceph-osd "update-status" hook 'stalled' the octavia install:
unit-octavia-0: 09:54:02 DEBUG unit.octavia/ 0.install Processing ./wheelhouse/ python- neutronclient- 7.1.0.tar. gz osd/0.juju- log Hardening function 'update_status' osd/0.juju- log No hardening applied to 'update_status' osd/0.juju- log Updating status. uniter. operation ran "update-status" hook 0.install Processing ./wheelhouse/ netifaces- 0.10.9. tar.gz 0.install Processing ./wheelhouse/ oslo.config- 7.0.0.tar. gz osd/0.juju- log Hardening function 'update_status' osd/0.juju- log No hardening applied to 'update_status' osd/0.juju- log Updating status. uniter. operation ran "update-status" hook 0.install Processing ./wheelhouse/ pecan-1. 3.3.tar. gz 0.install Processing ./wheelhouse/ charms. reactive- 1.3.0.tar. gz 0.install Processing ./wheelhouse/ cffi-1. 14.0.tar. gz 0.install Processing ./wheelhouse/ osc-lib- 2.0.0.tar. gz osd/0.juju- log Hardening function 'update_status' osd/0.juju- log No hardening applied to 'update_status' osd/0.juju- log Updating status. uniter. operation ran "update-status" hook 0.install Processing ./wheelhouse/ monotonic- 1.5.tar. gz 0.install Processing ./wheelhouse/ jmespath- 0.10.0. tar.gz 0.install Processing ./wheelhouse/ os-service- types-1. 7.0.tar. gz osd/0.juju- log Hardening function 'update_status' osd/0.juju- log No hardening applied to 'update_status' osd/0.juju- log Updating status. uniter. operation ran "update-status" hook osd/0.juju- log Hardening function 'update_status' osd/0.juju- log No hardening applied to 'update_status' osd/0.juju- log Updating status. uniter. operation ran "update-status" hook 0.install Processing ./wheelhouse/ PasteDeploy- 2.1.0.tar. gz 0.install Processing ./wheelhouse/ pytest- runner- 5.2.tar. gz 0.install Installing build dependencies: started 0.install Installing build dependencies: finished with status 'done' 0.install Getting requirements to build wheel: started 0.install Getting requirements to build wheel: finished with status 'done' 0.install Preparing wheel metadata: started 0.install Preparing wheel metadata: finished with status 'done' 0.install Processing ./wheelhouse/ chardet- 3.0.4.tar. gz 0.install Processing ./wheelhouse/ sqlalchemy- migrate- 0.13.0. tar.gz osd/0.juju- log Hardening function 'update_status' osd/0.juju- log No hardening applied to 'update_status' osd/0.juju- log Updating status. uniter. operation ran "update-status" hook 0.install Processing ./wheelhouse/ testtools- 2.4.0.tar. gz
unit-ceph-osd-0: 09:56:40 DEBUG unit.ceph-
unit-ceph-osd-0: 09:56:41 DEBUG unit.ceph-
unit-ceph-osd-0: 09:56:41 INFO unit.ceph-
unit-ceph-osd-0: 09:56:41 INFO juju.worker.
unit-octavia-0: 10:00:11 DEBUG unit.octavia/
unit-octavia-0: 10:00:12 DEBUG unit.octavia/
unit-ceph-osd-0: 10:02:34 DEBUG unit.ceph-
unit-ceph-osd-0: 10:02:34 DEBUG unit.ceph-
unit-ceph-osd-0: 10:02:34 INFO unit.ceph-
unit-ceph-osd-0: 10:02:35 INFO juju.worker.
unit-octavia-0: 10:06:21 DEBUG unit.octavia/
unit-octavia-0: 10:06:21 DEBUG unit.octavia/
unit-octavia-0: 10:06:22 DEBUG unit.octavia/
unit-octavia-0: 10:06:22 DEBUG unit.octavia/
unit-ceph-osd-0: 10:07:37 DEBUG unit.ceph-
unit-ceph-osd-0: 10:07:37 DEBUG unit.ceph-
unit-ceph-osd-0: 10:07:37 INFO unit.ceph-
unit-ceph-osd-0: 10:07:38 INFO juju.worker.
unit-octavia-0: 10:12:31 DEBUG unit.octavia/
unit-octavia-0: 10:12:31 DEBUG unit.octavia/
unit-octavia-0: 10:12:32 DEBUG unit.octavia/
unit-ceph-osd-0: 10:13:02 DEBUG unit.ceph-
unit-ceph-osd-0: 10:13:02 DEBUG unit.ceph-
unit-ceph-osd-0: 10:13:03 INFO unit.ceph-
unit-ceph-osd-0: 10:13:03 INFO juju.worker.
unit-ceph-osd-0: 10:18:33 DEBUG unit.ceph-
unit-ceph-osd-0: 10:18:33 DEBUG unit.ceph-
unit-ceph-osd-0: 10:18:33 INFO unit.ceph-
unit-ceph-osd-0: 10:18:34 INFO juju.worker.
unit-octavia-0: 10:18:41 DEBUG unit.octavia/
unit-octavia-0: 10:18:41 DEBUG unit.octavia/
unit-octavia-0: 10:18:41 DEBUG unit.octavia/
unit-octavia-0: 10:18:42 DEBUG unit.octavia/
unit-octavia-0: 10:18:42 DEBUG unit.octavia/
unit-octavia-0: 10:18:42 DEBUG unit.octavia/
unit-octavia-0: 10:18:42 DEBUG unit.octavia/
unit-octavia-0: 10:18:43 DEBUG unit.octavia/
unit-octavia-0: 10:18:43 DEBUG unit.octavia/
unit-octavia-0: 10:18:43 DEBUG unit.octavia/
unit-ceph-osd-0: 10:23:10 DEBUG unit.ceph-
unit-ceph-osd-0: 10:23:10 DEBUG unit.ceph-
unit-ceph-osd-0: 10:23:10 INFO unit.ceph-
unit-ceph-osd-0: 10:23:11 INFO juju.worker.
unit-octavia-0: 10:24:52 DEBUG unit.octavia/
Obviously, ceph is on bare-metal and octavia is in an LXD on the same host, in the configuration that's been provided.
Which version of Juju was being used to deploy, please?
Please could you attach full bundles that demonstrate the issue for the bionic and focal variables?
Thanks.