Landscape Charm

Operator charm fails to install with `"meter-status-changed": charm missing from disk`

Bug #1994961 reported by Alexandre Gomes on 2022-10-27

This bug affects 3 people

Affects		Status	Importance	Assigned to	Milestone
	Landscape Charm	New	Undecided	Unassigned

Bug Description

channel: latest/edge
revision: 58

When deploying the charm the landscape-server units get stuck in "(install) Installing apt packages".

The logs indicate `"meter-status-changed": charm missing from disk`. Full logs: https://pastebin.canonical.com/p/5WTwNJwKB7/

We managed to work around it by restarting jujud-machine agent and the charm seems to proceed correctly after that.

To deploy, we used this bundle[1] with this override[2].

[1] https://bazaar.launchpad.net/~sajoupa/canonical-mojo-specs/landscape-3/view/head:/landscape-internal/bundle.yaml
[2] https://bazaar.launchpad.net/~sajoupa/canonical-mojo-specs/landscape-3/view/head:/landscape-internal/ci/overlay.yaml

Revision history for this message

Haw Loeung (hloeung) wrote on 2022-10-28:

I don't think the issue is to do with the "charm missing from disk" as that's common if a hook isn't implemented. I think more work needs to be done to debug what's going on in legacy hooks/install. Maybe apt logs would help?

Anything in "ps" output at the time to see what processes were forked/spawned?

Revision history for this message

Jeff Hillman (jhillman) wrote on 2023-04-13:

Download full text (6.9 KiB)

I am having this same issue. This is the entirety of the unit log when it gets to this point.

2023-04-13 17:11:25 INFO juju unit_agent.go:289 Starting unit workers for "landscape-server/0"
2023-04-13 17:11:25 INFO juju.worker.apicaller connect.go:163 [ccba07] "unit-landscape-server-0" successfully connected to "10.141.15.103:17070"
2023-04-13 17:11:25 INFO juju.worker.apicaller connect.go:260 [ccba07] password changed for "unit-landscape-server-0"
2023-04-13 17:11:26 INFO juju.worker.apicaller connect.go:163 [ccba07] "unit-landscape-server-0" successfully connected to "10.141.15.128:17070"
2023-04-13 17:11:26 INFO juju.worker.migrationminion worker.go:142 migration phase is now: NONE
2023-04-13 17:11:26 INFO juju.worker.logger logger.go:120 logger worker started
2023-04-13 17:11:26 INFO juju.worker.upgrader upgrader.go:216 no waiter, upgrader is done
2023-04-13 17:11:26 ERROR juju.worker.meterstatus runner.go:93 error running "meter-status-changed": charm missing from disk
2023-04-13 17:11:26 INFO juju.worker.uniter uniter.go:326 unit "landscape-server/0" started
2023-04-13 17:11:26 INFO juju.worker.uniter uniter.go:634 resuming charm install
2023-04-13 17:11:26 INFO juju.worker.uniter.charm bundles.go:78 downloading ch:amd64/focal/landscape-server-87 from API server
2023-04-13 17:11:31 INFO juju.worker.uniter uniter.go:344 hooks are retried true
2023-04-13 17:11:31 INFO juju.worker.uniter.storage resolver.go:127 initial storage attachments ready
2023-04-13 17:11:31 INFO juju.worker.uniter resolver.go:159 found queued "install" hook
2023-04-13 17:11:31 INFO unit.landscape-server/0.juju-log server.go:316 Running legacy hooks/install.

This is the ps auxwww output (relevant part)

I am having this same issue.  This is the entirety of the unit log when it gets to this point.

This is the ps auxwww output (relevant part)

root         679  0.0  0.2 241032  9260 ?        Ssl  17:10   0:00 /usr/lib/accountsservice/accounts-daemon
root         683  0.0  0.0   8540  2800 ?        Ss   17:10   0:00 /usr/sbin/cron -f
message+     687  0.5  0.1   7888  5164 ?        Ss   17:10   0:00 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
root         691  0.0  0.0  81892  3732 ?        Ssl  17:10   0:00 /usr/sbin/irqbalance --foreground
root         694  0.0  0.4  29868 18196 ?        Ss   17:10   0:00 /usr/bin/python3 /usr/bin/networkd-dispatcher --run-startup-triggers
root         695  0.0  0.2 236420  8988 ?        Ssl  17:10   0:00 /usr/lib/policykit-1/polkitd --no-debug
root         702  0.4  0.9 948996 38396 ?        Ssl  17:10   0:00 /usr/lib/snapd/snapd
root         703  0.1  0.1  17300  7644 ?        Ss   17:10   0:00 /lib/systemd/systemd-logind
root         705  0.0  0.3 395472 13720 ?        Ssl  17:10   0:00 /usr/lib/udisks2/udisksd
daemon       706  0.0  0.0   3796  2156 ?        Ss   17:10   0:00 /usr/sbin/atd -f
root         759  0.0  0.3 245088 13512 ?        Ssl  17:10   0:00 /usr/sbin/ModemManager
root         764  0.0  0.5 108120 20496 ?        Ssl  17:10   0:00 /usr/bin/python3 /usr/share/unattended-upgrades/unattended-upgrade-shutdown --wait-for-signal
root         778  0.0  0.0   7352  2416 ttyS0    Ss+  17:10   0:00 /sbin/agetty -o -p -- \u --keep-baud 115200,38400,9600 ttyS0 vt220
root         779  0.0  0.0      0     0 ?        I    17:10   0:00 [kworker/0:5-events]
root         785  0.0  0.0   5828  1984 tty1     Ss+  17:10   0:00 /sbin/agetty -o -p -- \u --noclear tty1 linux
root         854  0.0  0.1  12180  7400 ?        Ss   17:10   0:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
root        1661  0.0  0.3 283816 15776 ?        Ssl  17:10   0:00 /usr/lib/packagekit/packagekitd
root        3528  0.0  0.0      0     0 ?        I    17:10   0:00 [kworker/1:5-cgroup_destroy]
root        3603  0.2  0.1   9664  5556 ?        Ss   17:10   0:00 /lib/systemd/systemd-udevd
systemd+    4547  0.0  0.1  27276  6144 ?        Ss   17:11   0:00 /lib/systemd/systemd-networkd
systemd+    4570  0.0  0.3  24420 12188 ?        Ss   17:11   0:00 /lib/systemd/systemd-resolved
root        4576  0.2  0.3  43864 13760 ?        S<s  17:11   0:00 /lib/systemd/systemd-journald
syslog      4801  0.0  0.1 224492  4500 ?        Ssl  17:11   0:00 /usr/sbin/rsyslogd -n -iNONE
root       16149  0.0  0.0      0     0 ?        I<   17:11   0:00 [xfsalloc]
root       16151  0.0  0.0      0     0 ?        I<   17:11   0:00 [xfs_mru_cache]
root       16964  0.0  0.0   8696  3476 ?        Ss   17:11   0:00 bash /etc/systemd/system/jujud-machine-8-exec-start.sh
root       16969  1.6  2.3 858808 96096 ?        Sl   17:11   0:00 /var/lib/juju/tools/machine-8/jujud machine --data-dir /var/lib/juju --machine-id 8 --debug
root       17047  0.3  0.6  32884 25204 ?        S    17:11   0:00 python3 ./src/charm.py
root       17607  1.1  1.7  79644 71776 ?        S    17:11   0:00 apt-get -y --option=Dpkg::Options::=--force-confold install landscape-server=23.03+7-0landscape0
root       18985  0.2  0.1   8312  5108 pts/0    Ss+  17:12   0:00 /usr/bin/dpkg --force-confold --status-fd 43 --configure --pending
root       20135  0.0  0.2  13804  8868 ?        Ss   17:12   0:00 sshd: ubuntu [priv]
ubuntu     20149  1.0  0.2  19084  9572 ?        Ss   17:12   0:00 /lib/systemd/systemd --user
ubuntu     20151  0.0  0.1 170596  4576 ?        S    17:12   0:00 (sd-pam)
ubuntu     20304  0.0  0.1  13940  5960 ?        S    17:12   0:00 sshd: ubuntu@pts/1
ubuntu     20307  1.0  0.1  10140  5228 pts/1    Ss   17:12   0:00 -bash
root       20348  0.0  0.0   2608  1648 pts/0    S+   17:12   0:00 /bin/sh /var/lib/dpkg/info/ntp.postinst configure
root       20416  2.0  0.2  13340 10588 pts/0    S+   17:12   0:00 /usr/bin/perl /usr/bin/deb-systemd-helper enable ntp.service
root       20417  0.0  0.0   6708  3756 pts/0    S+   17:12   0:00 /bin/systemctl --system --preset-mode=enable-only preset ntp.service
root       20420  0.0  0.1 169384  6420 ?        S    17:12   0:00 (sd-executor)
root       20422  0.0  0.0      0     0 ?        Z    17:12   0:00 [friendly-recove] <defunct>
root       20424  0.0  0.0      0     0 ?        Z    17:12   0:00 [netplan] <defunct>
root       20426  0.0  0.0   2608   596 ?        S    17:12   0:00 /bin/sh /usr/lib/systemd/system-generators/postfix-instance-generator /run/systemd/generator /run/systemd/generator.early /run/systemd/generator.late
root       20427  0.0  0.0      0     0 ?        Z    17:12   0:00 [snapd-generator] <defunct>
root       20432  0.0  0.0      0     0 ?        Z    17:12   0:00 [systemd-getty-g] <defunct>
root       20433  0.0  0.0      0     0 ?        Z    17:12   0:00 [systemd-gpt-aut] <defunct>
root       20437  0.0  0.0      0     0 ?        Z    17:12   0:00 [systemd-system-] <defunct>
root       20439  0.0  0.0      0     0 ?        Z    17:12   0:00 [systemd-veritys] <defunct>
root       20448  0.0  0.0  41932  3612 ?        S    17:12   0:00 postconf -h multi_instance_directories
ubuntu     20449  0.0  0.0  10616  3360 pts/1    R+   17:12   0:00 ps auxwww

charm rev 87 from stable, landscape server landscape/self-hosted-23.03 juju 2.9.42 focal

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.