Comment 0 for bug 1964636

We've recently noticed a lot of the LXD CI jobs failing because of apparmor related snapd issues.

The way this usually manifests is:
 - lxc launch images:ubuntu/20.04 c1
 - lxc exec c1 -- apt install snapd
 - lxc exec c1 -- snap install distrobuilder --edge --classic
 - lxc exec c1 -- distrobuilder

This all works as expected, `systemctl --failed` is clean and `apparmor_status` gets me:
```
0 loaded units listed.
root@v1:~# lxc exec c1 -- apparmor_status
apparmor module is loaded.
11 profiles are loaded.
10 profiles are in enforce mode.
   /usr/lib/NetworkManager/nm-dhcp-client.action
   /usr/lib/NetworkManager/nm-dhcp-helper
   /usr/lib/connman/scripts/dhclient-script
   /usr/lib/snapd/snap-confine
   /usr/lib/snapd/snap-confine//mount-namespace-capture-helper
   /{,usr/}sbin/dhclient
   lsb_release
   nvidia_modprobe
   nvidia_modprobe//kmod
   snap-update-ns.distrobuilder
1 profiles are in complain mode.
   snap.distrobuilder.distrobuilder
0 processes have profiles defined.
0 processes are in enforce mode.
0 processes are in complain mode.
0 processes are unconfined but have a profile defined.
```

Now to break things:
 - lxc restart c1
 - lxc exec c1 -- distrobuilder

```
root@v1:~# lxc exec c1 -- distrobuilder
cannot change profile for the next exec call: No such file or directory
```

Looking around, we see:
```
root@c1:~# find /var/lib/snapd/apparmor/
/var/lib/snapd/apparmor/
/var/lib/snapd/apparmor/snap-confine
/var/lib/snapd/apparmor/snap-confine/cap-bpf
/var/lib/snapd/apparmor/profiles
/var/lib/snapd/apparmor/profiles/snap.distrobuilder.distrobuilder
/var/lib/snapd/apparmor/profiles/snap-update-ns.distrobuilder
/var/lib/snapd/apparmor/profiles/snap-confine.snapd.14978
root@c1:~# cat /var/lib/snapd/apparmor/snap-confine/cap-bpf

capability bpf,
root@c1:~# systemctl --failed
  UNIT LOAD ACTIVE SUB DESCRIPTION
● snapd.apparmor.service loaded failed failed Load AppArmor profiles managed internally by snapd

LOAD = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB = The low-level unit activation state, values depend on unit type.

1 loaded units listed.
```

The error listed is:
```
Mar 11 19:54:58 c1 systemd[1]: Starting Load AppArmor profiles managed internally by snapd...
Mar 11 19:54:58 c1 snapd-apparmor[163]: /usr/lib/snapd/snapd-apparmor: 47: ns_stacked: not found
Mar 11 19:54:58 c1 snapd-apparmor[163]: /usr/lib/snapd/snapd-apparmor: 48: ns_name: not found
Mar 11 19:54:58 c1 snapd-apparmor[172]: AppArmor parser error for /var/lib/snapd/apparmor/profiles/snap-confine.snapd.14978 in /var/lib/snapd/apparmor/snap-confine/cap-bpf at line 2: Invalid capability bpf.
Mar 11 19:54:58 c1 systemd[1]: snapd.apparmor.service: Main process exited, code=exited, st
```

One can workaround it with:
```
> /var/lib/snapd/apparmor/snap-confine/cap-bpf
systemctl restart snapd.apparmor.service
```

Now for the bits I didn't quite figure out:
 - Why does snapd think that the parser supports `bpf` when it in fact doesn't?
 - Why does this only seem to hit with `distrobuilder`, testing with `hello-world` doesn't hit this issue though we've seen similar behavior from the `go` snap, is it a bug that only triggers on classic snaps?