unit is in waiting state for about 5 minutes

Bug #1995450 reported by Hemanth Nakkina
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juju Charmed Operator - Keystone K8S
New
Undecided
Unassigned

Bug Description

Deployed bundle https://github.com/openstack-snaps/snap-microstack/blob/main/bundles/etc/bundles/control-plane.yaml

The keystone unit is in waiting state with message "pebble not ready" for about 5 minutes.
The unit is changed to Active state only when update-status event is triggered.
Expectation is juju status should get updated sooner during initialisation/config change as and when events happen.

Log from keystone unit:

$ juju debug-log -m openstack -i unit-keystone-0 --replay
unit-keystone-0: 12:09:37 INFO juju.cmd running containerAgent [3.0.1 c3c40cda223fb79c9dae916d4958f06426584a97 gc go1.19.2]
unit-keystone-0: 12:09:37 INFO juju.cmd.containeragent.unit start "unit"
unit-keystone-0: 12:09:37 INFO juju.worker.upgradesteps upgrade steps for 3.0.1 have already been run.
unit-keystone-0: 12:09:37 INFO juju.worker.probehttpserver starting http server on 127.0.0.1:65301
unit-keystone-0: 12:09:37 INFO juju.api connection established to "wss://controller-service.controller-microk8s-localhost.svc.cluster.local:17070/model/a746fafe-7684-4bd1-8617-3d80479affe0/api"
unit-keystone-0: 12:09:37 INFO juju.worker.apicaller [a746fa] "unit-keystone-0" successfully connected to "controller-service.controller-microk8s-localhost.svc.cluster.local:17070"
unit-keystone-0: 12:09:38 INFO juju.worker.migrationminion migration phase is now: NONE
unit-keystone-0: 12:09:38 INFO juju.worker.logger logger worker started
unit-keystone-0: 12:09:38 INFO juju.worker.caasupgrader abort check blocked until version event received
unit-keystone-0: 12:09:38 INFO juju.worker.caasupgrader unblocking abort check
unit-keystone-0: 12:09:38 WARNING juju.worker.proxyupdater unable to set snap core settings [proxy.http= proxy.https= proxy.store=]: exec: "snap": executable file not found in $PATH, output: ""
unit-keystone-0: 12:09:38 INFO juju.worker.leadership keystone/0 promoted to leadership of keystone
unit-keystone-0: 12:09:38 INFO juju.agent.tools ensure jujuc symlinks in /var/lib/juju/tools/unit-keystone-0
unit-keystone-0: 12:09:38 INFO juju.worker.uniter unit "keystone/0" started
unit-keystone-0: 12:09:38 INFO juju.worker.uniter resuming charm install
unit-keystone-0: 12:09:38 INFO juju.worker.uniter.charm downloading ch:amd64/focal/keystone-k8s-67 from API server
unit-keystone-0: 12:09:38 INFO juju.downloader downloading from ch:amd64/focal/keystone-k8s-67
unit-keystone-0: 12:09:38 INFO juju.downloader download complete ("ch:amd64/focal/keystone-k8s-67")
unit-keystone-0: 12:09:38 INFO juju.downloader download verified ("ch:amd64/focal/keystone-k8s-67")
unit-keystone-0: 12:10:19 INFO juju.worker.uniter hooks are retried true
unit-keystone-0: 12:10:19 INFO juju.worker.uniter found queued "install" hook
unit-keystone-0: 12:10:24 INFO unit.keystone/0.juju-log Running legacy hooks/install.
unit-keystone-0: 12:10:28 INFO unit.keystone/0.juju-log Kubernetes service 'keystone' patched successfully
unit-keystone-0: 12:10:30 INFO juju.worker.uniter.operation ran "install" hook (via hook dispatching script: dispatch)
unit-keystone-0: 12:10:37 INFO juju.worker.uniter.operation ran "identity-service-relation-created" hook (via hook dispatching script: dispatch)
unit-keystone-0: 12:10:41 INFO juju.worker.uniter.operation ran "identity-service-relation-created" hook (via hook dispatching script: dispatch)
unit-keystone-0: 12:10:45 INFO juju.worker.uniter.operation ran "identity-service-relation-created" hook (via hook dispatching script: dispatch)
unit-keystone-0: 12:10:49 INFO juju.worker.uniter.operation ran "identity-service-relation-created" hook (via hook dispatching script: dispatch)
unit-keystone-0: 12:10:53 INFO juju.worker.uniter.operation ran "identity-credentials-relation-created" hook (via hook dispatching script: dispatch)
unit-keystone-0: 12:10:56 INFO unit.keystone/0.juju-log peers:11: Peers on_created
unit-keystone-0: 12:10:57 INFO juju.worker.uniter.operation ran "peers-relation-created" hook (via hook dispatching script: dispatch)
unit-keystone-0: 12:11:01 INFO juju.worker.uniter.operation ran "database-relation-created" hook (via hook dispatching script: dispatch)
unit-keystone-0: 12:11:04 INFO juju.worker.uniter.operation ran "ingress-public-relation-created" hook (via hook dispatching script: dispatch)
unit-keystone-0: 12:11:04 INFO juju.worker.uniter found queued "leader-elected" hook
unit-keystone-0: 12:11:07 INFO juju.worker.uniter.operation ran "leader-elected" hook (via hook dispatching script: dispatch)
unit-keystone-0: 12:11:10 INFO juju.worker.uniter.operation ran "credential-keys-storage-attached" hook (via hook dispatching script: dispatch)
unit-keystone-0: 12:11:12 INFO juju.worker.uniter.operation ran "fernet-keys-storage-attached" hook (via hook dispatching script: dispatch)
unit-keystone-0: 12:11:15 INFO unit.keystone/0.juju-log Relations {'ingress-public', 'database'} incomplete
unit-keystone-0: 12:11:15 INFO juju.worker.uniter.operation ran "config-changed" hook (via hook dispatching script: dispatch)
unit-keystone-0: 12:11:15 INFO juju.worker.uniter found queued "start" hook
unit-keystone-0: 12:11:17 INFO unit.keystone/0.juju-log Running legacy hooks/start.
unit-keystone-0: 12:11:19 INFO juju.worker.uniter.operation ran "start" hook (via hook dispatching script: dispatch)
unit-keystone-0: 12:11:22 INFO juju.worker.uniter.operation ran "ingress-public-relation-joined" hook (via hook dispatching script: dispatch)
unit-keystone-0: 12:11:24 INFO juju.worker.uniter.operation ran "identity-service-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-0: 12:11:27 INFO juju.worker.uniter.operation ran "identity-service-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-0: 12:11:29 INFO juju.worker.uniter.operation ran "identity-service-relation-joined" hook (via hook dispatching script: dispatch)
unit-keystone-0: 12:11:32 INFO juju.worker.uniter.operation ran "identity-service-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-0: 12:11:34 INFO juju.worker.uniter.operation ran "identity-credentials-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-0: 12:11:36 INFO juju.worker.uniter.operation ran "identity-service-relation-joined" hook (via hook dispatching script: dispatch)
unit-keystone-0: 12:11:39 INFO juju.worker.uniter.operation ran "ingress-public-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-0: 12:11:41 INFO juju.worker.uniter.operation ran "identity-service-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-0: 12:11:44 INFO juju.worker.uniter.operation ran "database-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-0: 12:11:46 INFO juju.worker.uniter.operation ran "identity-credentials-relation-joined" hook (via hook dispatching script: dispatch)
unit-keystone-0: 12:11:48 INFO juju.worker.uniter.operation ran "identity-service-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-0: 12:11:51 INFO juju.worker.uniter.operation ran "identity-credentials-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-0: 12:11:53 INFO juju.worker.uniter.operation ran "identity-service-relation-joined" hook (via hook dispatching script: dispatch)
unit-keystone-0: 12:11:55 INFO juju.worker.uniter.operation ran "identity-service-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-0: 12:11:58 INFO juju.worker.uniter.operation ran "database-relation-joined" hook (via hook dispatching script: dispatch)
unit-keystone-0: 12:12:00 INFO juju.worker.uniter.operation ran "database-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-0: 12:12:02 INFO juju.worker.uniter.operation ran "identity-service-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-0: 12:12:05 INFO juju.worker.uniter.operation ran "identity-service-relation-joined" hook (via hook dispatching script: dispatch)
unit-keystone-0: 12:12:07 INFO juju.worker.uniter.operation ran "identity-service-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-0: 12:12:10 INFO juju.worker.uniter.operation ran "database-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-0: 12:13:36 INFO unit.keystone/0.juju-log database:13: endpoints changed on 2022-11-02 06:43:36.769827
unit-keystone-0: 12:13:36 INFO unit.keystone/0.juju-log database:13: Received data: {'data': '{"database": "keystone"}', 'endpoints': 'mysql-0.mysql-endpoints:3306', 'read-only-endpoints': 'mysql-0.mysql-endpoints:3306'}
unit-keystone-0: 12:13:36 INFO unit.keystone/0.juju-log database:13: Relations {'database', 'ingress-public'} incomplete
unit-keystone-0: 12:13:38 INFO juju.worker.uniter.operation ran "database-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-0: 12:13:58 INFO unit.keystone/0.juju-log database:13: database created at 2022-11-02 06:43:58.907046
unit-keystone-0: 12:13:59 INFO unit.keystone/0.juju-log database:13: Received data: {'data': '{"database": "keystone"}', 'endpoints': 'mysql-0.mysql-endpoints:3306', 'password': '7ySwxCa8lBQsFkEtd5I3nIBC', 'read-only-endpoints': 'mysql-0.mysql-endpoints:3306', 'username': 'relation-13', 'version': '8.0.30-0ubuntu0.22.04.1'}
unit-keystone-0: 12:13:59 INFO unit.keystone/0.juju-log database:13: Relations {'ingress-public'} incomplete
unit-keystone-0: 12:13:59 INFO juju.worker.uniter.operation ran "database-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-0: 12:16:14 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch)
unit-keystone-0: 12:19:24 INFO juju.worker.uniter.operation ran "ingress-public-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-0: 12:19:27 INFO juju.worker.uniter.operation ran "identity-service-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-0: 12:19:29 INFO juju.worker.uniter.operation ran "identity-service-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-0: 12:19:32 INFO juju.worker.uniter.operation ran "identity-service-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-0: 12:19:34 INFO juju.worker.uniter.operation ran "identity-service-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-0: 12:20:52 WARNING unit.keystone/0.juju-log Not DB sync ran. Charm does not specify self.db_sync_cmds
unit-keystone-0: 12:20:52 INFO unit.keystone/0.juju-log Entering guarded section: 'Initializing Keystone'
unit-keystone-0: 12:20:52 INFO unit.keystone/0.juju-log Syncing database...
unit-keystone-0: 12:21:01 INFO unit.keystone/0.juju-log Setting up fernet tokens...
unit-keystone-0: 12:21:02 INFO unit.keystone/0.juju-log Setting up credentials...
unit-keystone-0: 12:21:04 INFO unit.keystone/0.juju-log Bootstrapping keystone service
unit-keystone-0: 12:21:07 INFO unit.keystone/0.juju-log Completed guarded section fully: 'Initializing Keystone'
unit-keystone-0: 12:21:07 INFO unit.keystone/0.juju-log Entering guarded section: 'Setting up initial projects and users'
unit-keystone-0: 12:21:24 INFO unit.keystone/0.juju-log Completed guarded section fully: 'Setting up initial projects and users'
unit-keystone-0: 12:21:25 INFO juju.worker.uniter.operation ran "keystone-pebble-ready" hook (via hook dispatching script: dispatch)
unit-keystone-0: 12:22:24 INFO unit.keystone/0.juju-log peers:11: Peers on_changed
unit-keystone-0: 12:22:25 INFO juju.worker.uniter.operation ran "peers-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-0: 12:22:28 INFO unit.keystone/0.juju-log peers:11: Peers on_changed
unit-keystone-0: 12:22:30 INFO juju.worker.uniter.operation ran "peers-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-0: 12:27:02 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch)

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.