Single lxd machine stuck on pending:
------------------------------------------
Machine State DNS Inst id Series AZ Message
0 started 10.244.8.128 azurill focal zone1 Deployed
0/lxd/0 started 10.244.8.176 juju-7c9753-0-lxd-0 focal zone1 Container started
0/lxd/1 started 10.246.65.92 juju-7c9753-0-lxd-1 focal zone1 Container started
0/lxd/2 started 10.244.8.178 juju-7c9753-0-lxd-2 focal zone1 Container started
0/lxd/3 pending pending focal Creating container
0/lxd/4 started 10.246.65.79 juju-7c9753-0-lxd-4 focal zone1 Container started
0/lxd/5 started 10.244.8.177 juju-7c9753-0-lxd-5 focal zone1 Container started
0/lxd/6 started 10.246.65.89 juju-7c9753-0-lxd-6 focal zone1 Container started
0/lxd/7 started 10.244.8.179 juju-7c9753-0-lxd-7 focal zone1 Container started
0/lxd/8 started 10.244.8.167 juju-7c9753-0-lxd-8 focal zone1 Container started
0/lxd/9 started 10.244.8.175 juju-7c9753-0-lxd-9 focal zone1 Container started
------------------------------------------
In the logs:
------------------------------------------
var/log/kern.log:Jan 8 01:10:42 azurill kernel: [ 716.131356] audit: type=1400 audit(1641604242.710:332): apparmor="DENIED" operation="file_inherit" namespace="root//lxd-juju-7c9753-0-lxd-3_<var-snap-lxd-common-lxd>" profile="/snap/snapd/14295/usr/lib/snapd/snap-confine" pid=32816 comm="snap-confine" family="netlink" sock_type="raw" protocol=15 requested_mask="send receive" denied_mask="send receive"
var/log/kern.log:Jan 8 01:10:42 azurill kernel: [ 716.193780] audit: type=1400 audit(1641604242.774:333): apparmor="DENIED" operation="file_inherit" namespace="root//lxd-juju-7c9753-0-lxd-3_<var-snap-lxd-common-lxd>" profile="snap-update-ns.lxd" name="/apparmor/.null" pid=32874 comm="6" requested_mask="wr" denied_mask="wr" fsuid=1000000 ouid=0
var/log/kern.log-Jan 8 01:10:46 azurill kernel: [ 720.362843] kauditd_printk_skb: 13 callbacks suppressed
------------------------------------------
Similar messages show up for the different containers on machine 0, but they are not quite the same.
Testrun:
https://solutions.qa.canonical.com/testruns/testRun/ab94b749-b99f-473b-8997-afa48c6815dd
Links to crashdumps:
https://oil-jenkins.canonical.com/artifacts/ab94b749-b99f-473b-8997-afa48c6815dd/index.html
Future occurrences of this bug can be found here:
https://solutions.qa.canonical.com/bugs/bugs/bug/1956981
This issue seems to be very active on jammy as well