Apparmor 3.0.0 does not load profiles in containers anymore

It seems once fixed the system is ok and I can't get into the bad state again :/

I tried on another bad system (withotu changing back to the former version)
1. A restart of the service
2. Trying to force capabilities reset (remove cache) + service restart

None of these got it into the good case, so I might be able to debug here what happens when probing.

for (i = 0; sec_managers[i]; i++) {
...
   VIR_DEBUG("Initialized caps for security driver \"%s\" with "

Good:
- apparmor
- dac

Bad:
- none
- dac

In function virQEMUDriverCreateCapabilities.
So it isn't probing apparmor because it isn't even in the list.

That list is from "qemuSecurityGetNested"
qemuSecurityGetNested == virSecurityManagerGetNested
-> virSecurityStackGetNested(mgr)

The latter iterates on the list priv->itemsHead which is from the security manager.

That in turn is from driver->securityManager of
virQEMUDriverGetCapabilities(driver)
 virCapsPtr virQEMUDriverCreateCapabilities(virQEMUDriverPtr driver)

(gdb) bt
#0 virSecurityStackGetNested (mgr=mgr@entry=0x7f8b0c00dde0) at ../../../src/security/security_stack.c:613
#1 0x00007f8b5704f2b8 in virSecurityManagerGetNested (mgr=0x7f8b0c00dde0) at ../../../src/security/security_manager.c:1035
#2 0x00007f8b50133970 in virQEMUDriverCreateCapabilities (driver=0x7f8b0c051550) at ../../../src/qemu/qemu_conf.c:1344
#3 0x00007f8b50133c18 in virQEMUDriverGetCapabilities (driver=0x7f8b0c051550, refresh=<optimized out>) at ../../../src/qemu/qemu_conf.c:1397
#4 0x00007f8b5019e0b8 in qemuConnectGetCapabilities (conn=<optimized out>) at ../../../src/qemu/qemu_driver.c:1328
#5 0x00007f8b57171953 in virConnectGetCapabilities (conn=0x7f8b28004050) at ../../../src/libvirt-host.c:467
#6 0x00005555a51f16ec in remoteDispatchConnectGetCapabilities (server=0x5555a5c1d080, msg=0x5555a5c2bc80, ret=0x7f8b48000e60, rerr=0x7f8b51be6920, client=0x5555a5c2c070)
    at ./remote/remote_daemon_dispatch_stubs.h:766
#7 remoteDispatchConnectGetCapabilitiesHelper (server=0x5555a5c1d080, client=0x5555a5c2c070, msg=0x5555a5c2bc80, rerr=0x7f8b51be6920, args=0x0, ret=0x7f8b48000e60)
    at ./remote/remote_daemon_dispatch_stubs.h:748
#8 0x00007f8b5707d470 in virNetServerProgramDispatchCall (msg=0x5555a5c2bc80, client=0x5555a5c2c070, server=0x5555a5c1d080, prog=0x5555a5c25810)
    at ../../../src/rpc/virnetserverprogram.c:430
#9 virNetServerProgramDispatch (prog=0x5555a5c25810, server=server@entry=0x5555a5c1d080, client=0x5555a5c2c070, msg=0x5555a5c2bc80) at ../../../src/rpc/virnetserverprogram.c:302
#10 0x00007f8b570825a8 in virNetServerProcessMsg (msg=<optimized out>, prog=<optimized out>, client=<optimized out>, srv=0x5555a5c1d080) at ../../../src/rpc/virnetserver.c:137
#11 virNetServerHandleJob (jobOpaque=0x5555a5bf97f0, opaque=0x5555a5c1d080) at ../../../src/rpc/virnetserver.c:154
#12 0x00007f8b56f901e2 in virThreadPoolWorker (opaque=<optimized out>) at ../../../src/util/virthreadpool.c:163
#13 0x00007f8b56f8f769 in virThreadHelper (data=<optimized out>) at ../../../src/util/virthread.c:233
#14 0x00007f8b56c61590 in start_thread (arg=0x7f8b51be7640) at pthread_create.c:463
#15 0x00007f8b56b6c223 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Good:
(gdb) p *((virSecurityStackDataPtr)(((virQEMUDriverPtr)conn->privateData )->securityManager->privateData))->itemsHead->securityManager
$7 = {parent = {parent = {parent_instance = {g_type_instance = {g_class = 0x7f430805ddf0}, ref_count = 1, qdata = 0x0}}, lock = {lock = {__data = {__lock = 0, __count = 0, __owner = 0,
          __nusers = 0, __kind = 512, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 17 times>, "\002", '\000' <repeats 21 times>,
        __align = 0}}}, drv = 0x7f435aadfae0 <virAppArmorSecurityDriver>, flags = 10, virtDriver = 0x7f43541e71b2 "QEMU", privateData = 0x0}
(gdb) p *((virSecurityStackDataPtr)(((virQEMUDriverPtr)conn->privateData )->securityManager->privateData))->itemsHead->next->securityManager
$8 = {parent = {parent = {parent_instance = {g_type_instance = {g_class = 0x7f430805ddf0}, ref_count = 1, qdata = 0x0}}, lock = {lock = {__data = {__lock = 0, __count = 0, __owner = 0,
          __nusers = 0, __kind = 512, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 17 times>, "\002", '\000' <repeats 21 times>,
        __align = 0}}}, drv = 0x7f435aadf7c0 <virSecurityDriverDAC>, flags = 10, virtDriver = 0x7f43541e71b2 "QEMU", privateData = 0x7f430807b180}

Bad:
(gdb) p *((virSecurityStackDataPtr)(((virQEMUDriverPtr)conn->privateData )->securityManager->privateData))->itemsHead->securityManager
$9 = {parent = {parent = {parent_instance = {g_type_instance = {g_class = 0x7f8b0c0259e0}, ref_count = 1, qdata = 0x0}}, lock = {lock = {__data = {__lock = 0, __count = 0, __owner = 0,
          __nusers = 0, __kind = 512, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 17 times>, "\002", '\000' <repeats 21 times>,
        __align = 0}}}, drv = 0x7f8b572d24c0 <virSecurityDriverNop>, flags = 8, virtDriver = 0x7f8b501d91b2 "QEMU", privateData = 0x0}
(gdb) p *((virSecurityStackDataPtr)(((virQEMUDriverPtr)conn->privateData )->securityManager->privateData))->itemsHead->next->securityManager
$10 = {parent = {parent = {parent_instance = {g_type_instance = {g_class = 0x7f8b0c0259e0}, ref_count = 1, qdata = 0x0}}, lock = {lock = {__data = {__lock = 0, __count = 0, __owner = 0,
          __nusers = 0, __kind = 512, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 17 times>, "\002", '\000' <repeats 21 times>,
        __align = 0}}}, drv = 0x7f8b572d27c0 <virSecurityDriverDAC>, flags = 10, virtDriver = 0x7f8b501d91b2 "QEMU", privateData = 0x7f8b0c07add0}

See virSecurityDriverNop vs virAppArmorSecurityDriver in the above output

Need to check the init of the bunch in qemuSecurityInit and qemuSecurityNew.
But that happens at daemon start and not later when probing caps.

virQEMUDriverConfigLoadSecurityEntry load this from config and it includes apparmor in both:
/etc/libvirt/qemu.conf:# security_driver = [ "selinux", "apparmor" ]

So the initialization must go wrong in the bad case.

virSecurityManagerNew loooks up the driver via virSecurityDriverLookup(name, virtDriver);
Then it calls virSecurityManagerNewDriver

Already differs here:
bad:
Thread 17 "daemon-init" hit Breakpoint 1, virSecurityManagerNew (name=name@entry=0x0, virtDriver=virtDriver@entry=0x7fffea6ae1b2 "QEMU", flags=flags@entry=10)
    at ../../../src/security/security_manager.c:180
180 ../../../src/security/security_manager.c: No such file or directory.
(gdb) c
Continuing.

Thread 17 "daemon-init" hit Breakpoint 2, virSecurityDriverLookup (name=name@entry=0x0, virtDriver=virtDriver@entry=0x7fffea6ae1b2 "QEMU") at ../../../src/security/security_driver.c:50
50 ../../../src/security/security_driver.c: No such file or directory.
(gdb) c
Continuing.

Thread 17 "daemon-init" hit Breakpoint 3, virSecurityManagerNewDriver (drv=0x7ffff7fad4c0 <virSecurityDriverNop>, virtDriver=virtDriver@entry=0x7fffea6ae1b2 "QEMU", flags=8)
    at ../../../src/security/security_manager.c:78
78 ../../../src/security/security_manager.c: No such file or directory.
(gdb) c
Continuing.

Thread 17 "daemon-init" hit Breakpoint 3, virSecurityManagerNewDriver (drv=0x7ffff7fad640 <virSecurityDriverStack>, virtDriver=0x7fffea6ae1b2 "QEMU", flags=flags@entry=8)
    at ../../../src/security/security_manager.c:78
78 in ../../../src/security/security_manager.c

Good:
Thread 17 "daemon-init" hit Breakpoint 1, virSecurityManagerNew (name=name@entry=0x0, virtDriver=virtDriver@entry=0x7f694365e1b2 "QEMU", flags=flags@entry=10)
    at ../../../src/security/security_manager.c:180
180 ../../../src/security/security_manager.c: No such file or directory.
(gdb) c
Continuing.

Thread 17 "daemon-init" hit Breakpoint 2, virSecurityDriverLookup (name=name@entry=0x0, virtDriver=virtDriver@entry=0x7f694365e1b2 "QEMU") at ../../../src/security/security_driver.c:50
50 ../../../src/security/security_driver.c: No such file or directory.
(gdb) c
Continuing.

Thread 17 "daemon-init" hit Breakpoint 3, virSecurityManagerNewDriver (drv=0x7f694ff5cae0 <virAppArmorSecurityDriver>, virtDriver=virtDriver@entry=0x7f694365e1b2 "QEMU", flags=10)
    at ../../../src/security/security_manager.c:78
78 ../../../src/security/security_manager.c: No such file or directory.
(gdb) c
Continuing.

Thread 17 "daemon-init" hit Breakpoint 3, virSecurityManagerNewDriver (drv=0x7f694ff5c640 <virSecurityDriverStack>, virtDriver=0x7f694365e1b2 "QEMU", flags=flags@entry=10)
    at ../../../src/security/security_manager.c:78
78 in ../../../src/security/security_manager.c

P.S. I might need a debug build going further yet I'm unsure if installing that might change the bug conditions.

Lookup fails:

(gdb) fin
Run till exit from #0 virSecurityDriverLookup (name=name@entry=0x0, virtDriver=virtDriver@entry=0x7fffd26ae1b2 "QEMU") at ../../../src/security/security_driver.c:50
virSecurityManagerNew (name=name@entry=0x0, virtDriver=virtDriver@entry=0x7fffd26ae1b2 "QEMU", flags=flags@entry=10) at ../../../src/security/security_manager.c:182
182 ../../../src/security/security_manager.c: No such file or directory.
Value returned is $2 = (virSecurityDriver *) 0x7ffff7fad4c0 <virSecurityDriverNop>

This goes via
AppArmorSecurityManagerProbe

Good:
Value returned is $3 = 0

Bad:
Value returned is $5 = -2

This is the failing function

 221 /* returns -1 on error or profile for libvirtd is unconfined, 0 if complain
 222 * mode and 1 if enforcing. This is required because at present you cannot
 223 * aa_change_profile() from a process that is unconfined.
 224 */
 225 static int
 226 use_apparmor(void)
 227 {
 228 int rc = -1;
 229 char *libvirt_daemon = NULL;
 230
 231 if (virFileResolveLink("/proc/self/exe", &libvirt_daemon) < 0) {
 232 virReportError(VIR_ERR_INTERNAL_ERROR,
 233 "%s", _("could not find libvirtd"));
 234 return rc;
 235 }
 236
 237 /* If libvirt_lxc is calling us, then consider apparmor is used
 238 * and enforced. */
 239 if (strstr(libvirt_daemon, "libvirt_lxc"))
 240 return 1;
 241
 242 if (access(APPARMOR_PROFILES_PATH, R_OK) != 0)
 243 goto cleanup;
 244
 245 /* First check profile status using full binary path. If that fails
 246 * check using profile name.
 247 */
 248 rc = profile_status(libvirt_daemon, 1);
 249 if (rc < 0) {
 250 rc = profile_status("libvirtd", 1);
 251 /* Error or unconfined should all result in -1 */
 252 if (rc < 0)
 253 rc = -1;
 254 }
 255
 256 cleanup:
 257 VIR_FREE(libvirt_daemon); ...

Sorry my system broke down in various way stalling debugging of this for a few days.
Back on it ...

Yeah and the comment above this function pointed the right way:

Good case (libvirt is enforced):
oot@testkvm-groovy-to:~# aa-status
apparmor module is loaded.
31 profiles are loaded.
31 profiles are in enforce mode.
   /snap/snapd/9279/usr/lib/snapd/snap-confine
   /snap/snapd/9279/usr/lib/snapd/snap-confine//mount-namespace-capture-helper
   /usr/bin/man
   /usr/lib/NetworkManager/nm-dhcp-client.action
   /usr/lib/NetworkManager/nm-dhcp-helper
   /usr/lib/connman/scripts/dhclient-script
   /usr/lib/snapd/snap-confine
   /usr/lib/snapd/snap-confine//mount-namespace-capture-helper
   /{,usr/}sbin/dhclient
   libvirtd
   libvirtd//qemu_bridge_helper
   lsb_release
   man_filter
   man_groff
   nvidia_modprobe
   nvidia_modprobe//kmod
   snap-update-ns.lxd
   snap.lxd.activate
   snap.lxd.benchmark
   snap.lxd.buginfo
   snap.lxd.check-kernel
   snap.lxd.daemon
   snap.lxd.hook.configure
   snap.lxd.hook.install
   snap.lxd.hook.remove
   snap.lxd.lxc
   snap.lxd.lxc-to-lxd
   snap.lxd.lxd
   snap.lxd.migrate
   tcpdump
   virt-aa-helper
0 profiles are in complain mode.
0 profiles are in kill mode.
0 profiles are in unconfined mode.
1 processes have profiles defined.
1 processes are in enforce mode.
   /usr/sbin/libvirtd (14751) libvirtd
0 processes are in complain mode.
0 processes are unconfined but have a profile defined.
0 processes are in mixed mode.
0 processes are in kill mode.

Bad case libvirt (and plenty of other things) are not confined:
# aa-status
apparmor module is loaded.
15 profiles are loaded.
15 profiles are in enforce mode.
   /snap/snapd/9279/usr/lib/snapd/snap-confine
   /snap/snapd/9279/usr/lib/snapd/snap-confine//mount-namespace-capture-helper
   snap-update-ns.lxd
   snap.lxd.activate
   snap.lxd.benchmark
   snap.lxd.buginfo
   snap.lxd.check-kernel
   snap.lxd.daemon
   snap.lxd.hook.configure
   snap.lxd.hook.install
   snap.lxd.hook.remove
   snap.lxd.lxc
   snap.lxd.lxc-to-lxd
   snap.lxd.lxd
   snap.lxd.migrate
0 profiles are in complain mode.
0 profiles are in kill mode.
0 profiles are in unconfined mode.
0 processes have profiles defined.
0 processes are in enforce mode.
0 processes are in complain mode.
0 processes are unconfined but have a profile defined.
0 processes are in mixed mode.
0 processes are in kill mode.

As if only snap profiles are loaded.

This gets me back to a working system
  $ aa-enforce /etc/apparmor.d/usr.sbin.libvirtd
  $ systemctl restart libvirtd

And this also explains why on the system where I re-installed libvirt things might have worked.
The re-install runs dh_apparmor which has loaded and enforced it.

Wild _guess_/hint that could explain the behaviour you see: Do you have (snap?) profiles that have rules with "peer=libvirtd", and fail if libvirtd is running unconfined (which would need "peer=unconfined" in the other profile)?

I knew from my former tests:
1. apparmor 3.0 = bad
2. downgrading to 2.13.3-7ubuntu6 and back up to 3.0 = good
3. aa-enforce + service restart = good

I checked the logs on the affected systems how this got into the bad state:

$ grep -E 'configure (lib)?(apparmor|libvirt)' /var/log/dpkg.log
2020-09-16 05:56:09 configure libapparmor1:amd64 3.0.0~beta1-0ubuntu1 <none>
2020-09-16 05:56:18 configure apparmor:amd64 3.0.0~beta1-0ubuntu1 <none>
2020-09-16 05:57:31 configure libvirt-daemon-system-systemd:amd64 6.6.0-1ubuntu2 <none>
2020-09-16 05:57:31 configure libvirt0:amd64 6.6.0-1ubuntu2 <none>
2020-09-16 05:57:33 configure libvirt-clients:amd64 6.6.0-1ubuntu2 <none>
2020-09-16 05:57:36 configure libvirt-daemon:amd64 6.6.0-1ubuntu2 <none>
2020-09-16 05:57:36 configure libvirt-daemon-driver-qemu:amd64 6.6.0-1ubuntu2 <none>
2020-09-16 05:57:36 configure libvirt-daemon-system:amd64 6.6.0-1ubuntu2 <none>
2020-09-16 05:58:05 configure apparmor-utils:amd64 3.0.0~beta1-0ubuntu1 <none>
2020-09-17 14:04:17 configure libvirt-daemon-system-dbgsym:amd64 6.6.0-1ubuntu2 <none>
2020-09-17 14:04:17 configure libvirt0-dbgsym:amd64 6.6.0-1ubuntu2 <none>
2020-09-17 14:04:17 configure libvirt-daemon-driver-qemu-dbgsym:amd64 6.6.0-1ubuntu2 <none>
2020-09-17 14:04:17 configure libvirt-clients-dbgsym:amd64 6.6.0-1ubuntu2 <none>
2020-09-17 14:04:17 configure libvirt-daemon-dbgsym:amd64 6.6.0-1ubuntu2 <none>
2020-09-22 06:56:34 configure apparmor:amd64 3.0.0~beta1-0ubuntu5 <none>

It seems I had:
1. groovy container
2. upgrade to proposed (including libapparmor1 / apparmor 3.0)
3. install libvirt

I was trying to recreate the above with a new container as of today:
1. groovy container (2.13.3-7ubuntu6, all still confined)
2. upgrade to proposed (3.0.0~beta1-0ubuntu5, all still confined)
3. install libvirt (confinement working well)

Hmm, something must have been different.
I know I have used container snapshots when I ran into that - I need to sort out in what order that happened and if it would occur again.

Hi Christian Bolz o/
I'd have such rules but this isn't the problem here as that would matter only much later.
I libvirtd itself isn't confined it refuses to go on confining the guests and that is here the problem.

The current question really comes down to "how did I manage to have everything but snaps loose enforce mode"?

Ok, I have definitely a snapshot left that has "conserved" the bad state.

$ lxc stop testkvm-groovy-from
$ lxc restore testkvm-groovy-from orig
$ lxc start testkvm-groovy-from
$ lxc exec testkvm-groovy-from
# aa-status
apparmor module is loaded.
15 profiles are loaded.
15 profiles are in enforce mode.
   /snap/snapd/9279/usr/lib/snapd/snap-confine
   /snap/snapd/9279/usr/lib/snapd/snap-confine//mount-namespace-capture-helper
   snap-update-ns.lxd
   snap.lxd.activate
   snap.lxd.benchmark
   snap.lxd.buginfo
   snap.lxd.check-kernel
   snap.lxd.daemon
   snap.lxd.hook.configure
   snap.lxd.hook.install
   snap.lxd.hook.remove
   snap.lxd.lxc
   snap.lxd.lxc-to-lxd
   snap.lxd.lxd
   snap.lxd.migrate
0 profiles are in complain mode.
0 profiles are in kill mode.
0 profiles are in unconfined mode.
0 processes have profiles defined.
0 processes are in enforce mode.
0 processes are in complain mode.
0 processes are unconfined but have a profile defined.
0 processes are in mixed mode.
0 processes are in kill mode.

While silly this gets me back to normal from here
# aa-enforce /etc/apparmor.d/*
# aa-status
apparmor module is loaded.
32 profiles are loaded.
32 profiles are in enforce mode.
...

You see that we now have more than twice as much loaded

I have backed up this container and its snapshot for later and re-run the whole automation which got me that bad state.

That allowed me to run my automation again without removing this container (in case we need it for debugging later). So I ran everything again to check if it would happen again with the version now in groovy proposed.

Ok it ran into the same issues again so it is reproducible with the current version in proposed.
Since in the tests have plenty of systems involved I need to cut it down and simplify it to just one ...

FYI - other testing might miss this as "starting a guest on groovy" works with the new versions, but it will be without apparmor. Migrating from focal or a pre-upgrade groovy shows the issues broken by apparmor not being enabled.

It seems it comes down to a change in /lib/apparmor/apparmor.systemd which now refuses to load profiles when running in a container.

Example with 3.0:
$ /lib/apparmor/apparmor.systemd reload
Not starting AppArmor in container

Example with 2.x
 /lib/apparmor/apparmor.systemd reload
Restarting AppArmor
Reloading AppArmor profiles

This also explains why snap profiles work, the are loaded by snapd and not by apparmor.service.

I'll attach a repro script and full logs of good and bad case.

Bad case:

$ ./repro.sh bad
+ '[' bad == bad ']'
+ echo 'Bad case: Using apparmor from proposed'
Bad case: Using apparmor from proposed
+ BADCASE=1
+ lxc stop --force testguest-apparmor-bad
+ lxc delete --force testguest-apparmor-bad
+ lxc launch ubuntu-daily:groovy/amd64 testguest-apparmor-bad --profile default --profile kvm
Creating testguest-apparmor-bad
Starting testguest-apparmor-bad
+ sleep 30s
+ lxc exec testguest-apparmor-bad runlevel
N 5
+ lxc exec testguest-apparmor-bad -- bash -c 'H=`cat /etc/hostname`; if [ -f /var/lib/cloud/instance/boot-finished ]; then echo "LXD container $H ready"; else echo "LXD container $H not ready yet"; exit 2; fi'
LXD container testguest-apparmor-bad ready
+ lxc exec testguest-apparmor-bad --env DEBIAN_FRONTEND=noninteractive -- bash -c 'apt-get --allow-unauthenticated --assume-yes -o Dpkg::Options::='\''--force-confdef'\'' -o Dpkg::Options::='\''--force-confold'\'' install apparmor-utils'
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following package was automatically installed and is no longer required:
  libfreetype6
Use 'apt autoremove' to remove it.
The following additional packages will be installed:
  python3-apparmor python3-libapparmor
Suggested packages:
  vim-addon-manager
The following NEW packages will be installed:
  apparmor-utils python3-apparmor python3-libapparmor
0 upgraded, 3 newly installed, 0 to remove and 0 not upgraded.
Need to get 157 kB of archives.
After this operation, 966 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu groovy/main amd64 python3-libapparmor amd64 2.13.3-7ubuntu6 [26.7 kB]
Get:2 http://archive.ubuntu.com/ubuntu groovy/main amd64 python3-apparmor amd64 2.13.3-7ubuntu6 [78.6 kB]
Get:3 http://archive.ubuntu.com/ubuntu groovy/main amd64 apparmor-utils amd64 2.13.3-7ubuntu6 [51.4 kB]
Fetched 157 kB in 0s (385 kB/s)
Selecting previously unselected package python3-libapparmor.
(Reading database ... 31714 files and directories currently installed.)
Preparing to unpack .../python3-libapparmor_2.13.3-7ubuntu6_amd64.deb ...
Unpacking python3-libapparmor (2.13.3-7ubuntu6) ...
Selecting previously unselected package python3-apparmor.
Preparing to unpack .../python3-apparmor_2.13.3-7ubuntu6_amd64.deb ...
Unpacking python3-apparmor (2.13.3-7ubuntu6) ...
Selecting previously unselected package apparmor-utils.
Preparing to unpack .../apparmor-utils_2.13.3-7ubuntu6_amd64.deb ...
Unpacking apparmor-utils (2.13.3-7ubuntu6) ...
Setting up python3-libapparmor (2.13.3-7ubuntu6) ...
Setting up python3-apparmor (2.13.3-7ubuntu6) ...
Setting up apparmor-utils (2.13.3-7ubuntu6) ...
Processing triggers for man-db (2.9.3-2) ...
+ lxc exec testguest-apparmor-bad -- aa-status
apparmor module is loaded.
28 profiles are loaded.
28 profiles are in enforce mode.
   /snap/snapd/9279/usr/lib/snapd/snap-confine
   /snap/snapd/9279/usr/lib/snapd/snap-confine//mount-namespace-capture-helper
   /usr/bin/man
   /usr/lib/NetworkManager/nm-dhcp-client.action
   /usr/lib/NetworkManager/nm-dhcp-helper
   /usr/lib/connman/scripts/dhclient-script
   /usr/lib/snapd/snap-confine
   /usr/lib/snapd/snap-confine//mo...

Still chasing this down

The apparmor.systemd file is unchanged from focal.

The change is in rc.apparmor.functions which is a dependency of apparmor.systemd.

Isn't that "Not starting AppArmor in container" message just in:

/lib/apparmor/apparmor.systemd
   -> /lib/apparmor/rc.apparmor.functions
      -> function is_container_with_internal_policy()

That looks unchanged (except a comment) but it behaves differently:

root@testguest-apparmor-good:~# . /usr/lib/apparmor/rc.apparmor.functions
root@testguest-apparmor-good:~# is_container_with_internal_policy
root@testguest-apparmor-good:~# echo $?
0

root@testguest-apparmor-bad:~# . /usr/lib/apparmor/rc.apparmor.functions
root@testguest-apparmor-bad:~# is_container_with_internal_policy
root@testguest-apparmor-bad:~# echo $?
1

Looking into what happens in detail ...

good:
+ SFS_MOUNTPOINT=/sys/kernel/security/apparmor
+ local ns_stacked_path=/sys/kernel/security/apparmor/.ns_stacked

bad:
+ SFS_MOUNTPOINT=/sys/kernel/security/
+ local ns_stacked_path=/sys/kernel/security//.ns_stacked

Once we know that we can see that it is missing in the bad case

good:
root@testguest-apparmor-good:~# grep MODULE /usr/lib/apparmor/rc.apparmor.functions
MODULE=apparmor
 SFS_MOUNTPOINT="${SECURITYFS}/${MODULE}"
 if [ -f "${SECURITYFS}/${MODULE}/profiles" ]; then
  SFS_MOUNTPOINT="${SECURITYFS}/${MODULE}"
  MODULE=apparmor
 /sbin/modprobe -qr $MODULE

bad:
root@testguest-apparmor-bad:~# grep MODULE /usr/lib/apparmor/rc.apparmor.functions
 SFS_MOUNTPOINT="${SECURITYFS}/${MODULE}"

So whatever took away the modprobe from /usr/lib/apparmor/rc.apparmor.functions also removed the variable, but that has broken function is_container_with_internal_policy

And we are back with Christian Bolz :-)

commit 61c27d8808f0589beb6a319cc04073e8bb32d860
Author: Christian Boltz <email address hidden>
Date: Fri Jun 21 19:22:15 2019 +0200

    Fix and simplify setting SFS_MOUNTPOINT

The question is why isn't this in the apparmor 3.0 package in groovy-proposed ?

https://gitlab.com/apparmor/apparmor/-/commit/61c27d8808f0589beb6a319cc04073e8bb32d860

As refrence, it is a re-occurrence of https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1824812 , look who filed that bug :-)

That patch by Christian Bolz is already applied (which seems reasonable after that much time),
but when merging 3.0 the old patch for bug 1824812 should have been dropped.

Tested the change - works as expected, prepping an MP

I uploaded 3.0.0~beta1-0ubuntu6 just now that should address this issue. Thanks Christian for your debugging!

Christian - thanks for your work on debugging this - can you please remove the block-proposed tag if you are happy that 3.0.0~beta1-0ubuntu6 resolves this issue?

FYI, I removed the block-proposed tag since ubuntu6 fixes this bug.

This bug was fixed in the package apparmor - 3.0.0~beta1-0ubuntu6

---------------
apparmor (3.0.0~beta1-0ubuntu6) groovy; urgency=medium

  * Drop d/p/lp1824812.patch: this patch was only needed with 2.13 and not
    3.0. With AppArmor 3, the patch ends up setting SFS_MOUNTPOINT to the
    wrong directory in is_container_with_internal_policy(), which causes
    policy to always fail to load in containers. Thanks to Christian Ehrhardt
    for the analysis. (LP: #1895967)

apparmor (3.0.0~beta1-0ubuntu5) groovy; urgency=medium

  [ John Johansen ]
  * d/p/fix-parser-to-emit-proc-attr-access-for-all-situations.patch:
    fix-automatic-adding-of-rule-for-change-hat-iface.patch fixed the
    parser to emit rules needed for change_hat in the hat profiles but
    broke the rule being emitted for the parent profile, this fixes it for
    both so that it is emitted for any profile that is a hat or that
    contains a hat.
  * d/p/fix-change-profile-stack-abstraction.patch: fix the change_profile
    abstraction so that it allows access to the apparmor attribute paths
    under LSM stacking.

apparmor (3.0.0~beta1-0ubuntu2) groovy; urgency=medium

  [ John Johansen ]
  * d/p/fix-automatic-adding-of-rule-for-change-hat-iface.patch: fix
    parser not adding a rule to profiles if they are a hat or contain hats
    granting write access to the kernel interfaces.

apparmor (3.0.0~beta1-0ubuntu1) groovy; urgency=medium

  [ John Johansen ]
  * New upstream release (LP: #1895060, LP: #1887577, LP: #1880841)
  * Drop all patches backported from upstream: applied in 3.0
  * d/p/policy-provide-example-and-base-abi-to-pin-pre-3.0-p.patch: provide
    example and base abi to pin pre 3.0 policy
  * d/p/ubuntu/enable-pinning-of-pre-AppArmor-3.x-poli.patch: enable pinning
    of pre AppArmor 3.x policy
  * drop d/p/debian/dont-include-site-local-with-dovecot.patch: no longer
    needed with upstream 'include if exists'

  [ Steve Beattie ]
  * d/p/parser-fix_cap_match.patch: fix cap match to work correctly, important
    now that groovy has a 5.8 kernel.
  * d/apparmor-profiles.install:
    + adjust for renamed postfix profiles
    + add usr.bin.dumpcap and usr.bin.mlmmj-receive to extra-profiles
    + remove usr.sbin.nmbd and usr.sbin.smbd from extra-profiles (already in
      apparmor-profiles)
  * d/apparmor.install: include abi/ directory and tunables/etc.
  * d/apparmor.manpages: add apparmor_xattrs.7 manpage
  * d/control:
    + apparmor-utils: no more shipped perl tools, drop perl dependency
    + apparmor-notify: aa-notify was converted to python3 from perl; adjust
      -notify dependencies to compensate
  * d/p/fix-tests-regression-apparmor-prologue-inc-settest.patch:
    fix sed expression in settest()

  [ Emilia Torino ]
  * Removing Ubuntu specific chromium-browser profile. This is safe to do
    since groovy's chromium-browser deb installs the snap. If apparmor3
    is backported to 18.04 or earlier, the profile will need to be taken
    into consideration
    - d/profiles/chromium-browser: remove chromium-browser profile
    - d/apparmor-profiles.postinst: remove postinst script as it only
      contains chromium-browser related functionallity.
    ...