Comment 0 for bug 1891076

This is on focal with jujud 2.8.1.

heat-mysql-router/1 has been stuck in executing "(install) installing charm software" for over 5 hours:

ubuntu@production-cpe-07d59674-95e9-4406-9d05-8f25282eb455:~/project$ juju status heat-mysql-router
Model Controller Cloud/Region Version SLA Timestamp
openstack foundations-maas maas_cloud/default 2.8.1 unsupported 18:54:25Z

App Version Status Scale Charm Store Rev OS Notes
hacluster-heat waiting 0 hacluster jujucharms 143 ubuntu
heat 14.0.0 waiting 3 heat jujucharms 357 ubuntu
heat-mysql-router 8.0.21 waiting 3 mysql-router jujucharms 21 ubuntu
logrotate waiting 0 logrotate-charm jujucharms 5 ubuntu
public-policy-routing waiting 0 advanced-routing jujucharms 5 ubuntu

Unit Workload Agent Machine Public address Ports Message
heat/0 waiting idle 3/lxd/5 10.244.40.226 8000/tcp,8004/tcp Incomplete relations: database, identity
  hacluster-heat/1 waiting allocating 10.244.40.226 agent initializing
  heat-mysql-router/1 maintenance executing 10.244.40.226 (install) installing charm software
  logrotate/42 waiting allocating 10.244.40.226 agent initializing
  public-policy-routing/34 waiting allocating 10.244.40.226 agent initializing
heat/1 waiting idle 4/lxd/11 10.244.41.2 8000/tcp,8004/tcp Incomplete relations: database, identity
  hacluster-heat/2 waiting idle 10.244.41.2 Resource: res_heat_8afed2c_vip not yet configured
  heat-mysql-router/2 waiting idle 10.244.41.2 'db-router' incomplete, MySQL Router not yet bootstrapped
  logrotate/50 active idle 10.244.41.2 Unit is ready.
  public-policy-routing/41 active idle 10.244.41.2 Unit is ready
heat/2* waiting idle 5/lxd/6 10.244.40.205 8000/tcp,8004/tcp Incomplete relations: database, identity
  hacluster-heat/0* waiting idle 10.244.40.205 Resource: res_heat_8afed2c_vip not yet configured
  heat-mysql-router/0* waiting idle 10.244.40.205 'db-router' incomplete, MySQL Router not yet bootstrapped
  logrotate/12 active idle 10.244.40.205 Unit is ready.
  public-policy-routing/3 active idle 10.244.40.205 Unit is ready

Machine State DNS Inst id Series AZ Message
3 started 10.244.40.108 armaldo focal zone1 Deployed
3/lxd/5 started 10.244.40.226 juju-05dab1-3-lxd-5 focal zone1 Container started
4 started 10.244.40.111 spearow focal zone2 Deployed
4/lxd/11 started 10.244.41.2 juju-05dab1-4-lxd-11 focal zone2 Container started
5 started 10.244.40.106 beartic focal zone3 Deployed
5/lxd/6 started 10.244.40.205 juju-05dab1-5-lxd-6 focal zone3 Container started

ubuntu@production-cpe-07d59674-95e9-4406-9d05-8f25282eb455:~/project$ juju show-status-log heat-mysql-router/1
Time Type Status Message
07 Aug 2020 13:06:30Z workload waiting waiting for machine
07 Aug 2020 13:06:30Z juju-unit allocating
07 Aug 2020 13:06:30Z workload waiting installing agent
07 Aug 2020 13:06:33Z workload waiting agent initializing
07 Aug 2020 13:06:46Z workload maintenance installing charm software
07 Aug 2020 13:06:46Z juju-unit executing running install hook
ubuntu@production-cpe-07d59674-95e9-4406-9d05-8f25282eb455:~/project$ date
Mon Aug 10 18:56:18 UTC 2020

However, there is no jujud for it:
ubuntu@juju-05dab1-3-lxd-5:~$ ps auxf | grep jujud
ubuntu 33523 0.0 0.0 8164 660 pts/1 S+ 18:56 0:00 \_ grep --color=auto jujud
root 313 0.0 0.0 8696 3424 ? Ss 14:59 0:00 bash /etc/systemd/system/jujud-machine-3-lxd-5-exec-start.sh
root 347 0.0 0.0 828256 78016 ? SLl 14:59 0:05 \_ /var/lib/juju/tools/machine-3-lxd-5/jujud machine --data-dir /var/lib/juju --machine-id 3/lxd/5 --debug
root 315 0.0 0.0 8696 3432 ? Ss 14:59 0:00 bash /etc/systemd/system/jujud-unit-heat-0-exec-start.sh
root 342 0.0 0.0 829024 80224 ? SLl 14:59 0:13 \_ /var/lib/juju/tools/unit-heat-0/jujud unit --data-dir /var/lib/juju --unit-name heat/0 --debug

And no logs either.