Incorrect handling of apparmor `bpf` capability

Thanks for the report, I just tried to reproduce this on both updated impish and updated jammy and couldn't reproduce. I notice on impish specifically that my focal container does not get the /var/lib/snapd/apparmor/snap-confine/cap-bpf file which is expected because the apparmor_parser for focal does not know about the bpf capability. What's confusing is how/why your container saw this, because again snapd actually tries to compile a program with apparmor_parser with "capability bpf," in it and only if that succeeds will it generate that snippet to include in snap-confine's policy.

So for this to have happened to you, the apparmor_parser that snapd sees inside the focal container must have been able to successfully compile with that snippet.

You mentioned on IRC that this was a privileged container, is there anyway that in addition to being a privileged container somehow it had a newer apparmor_parser in the container too?

This is failing because the apparmor_parser before the 3.0 release would build its capability list from the installed kernel headers. The apparmor_parser here was built against a kernel without support for cap 'bpf'

root@priv:~# apparmor_parser -V
AppArmor parser version 2.13.3
Copyright (C) 1999-2008 Novell Inc.
Copyright 2009-2018 Canonical Ltd.

This was fixed in 3.0 by having a static caps list (with full mapping info) and the dynamic auto-generated list (against the kernel headers) that is used to check that the static list has not become stale. In addition the parser can pull kernel supported caps straight from the apparmor kernel module (it will however be missing the mapping info).

To fix similar issues upstream a limited version of the 3.0 fix was backported to 2.13. And release in 2.13.5.

Ideally we would pull 2.13.6 in but at a minimum we need to pick
  726c3fc1 parser: Make sure apparmor can build on old kernels
  3f8cfac3 parser/Makefile: fix generated cap comparison against known list
  ad45b807 parser: add CAP_BPF/PERFMON; convert to pregenerated cap list
  cd4a1613 Add CAP_BPF and CAP_PERFMON to severity.db
  60007d3f parser: Add warning to capability_table about the need to update the Makefile
  ef8d5141 parser/Makefile: use LC_ALL=C when invoking sed
  4e194b2f parser: unify capability name handling
  ed61e482 parser: cleanup capability_table generation by dropping cap sys_log
  efb6952e parser: Move to a pre-generated cap_names.h

Now for the bits I didn't quite figure out:
 - Why does snapd think that the parser supports `bpf` when it in fact doesn't?

My guess is that its checking the kernel for support of bpf and not the parser.

 - Why does this only seem to hit with `distrobuilder`, testing with `hello-world` doesn't hit this issue though we've seen similar behavior from the `go` snap, is it a bug that only triggers on classic snaps?

my guess is that it will depend on what connections (plugs) are used. Classic snaps in general shouldn't trigger this because they are run unconfined, but LXD triggers it because it has its own profile. Only snaps that plug a connection that requires bpf would trigger this.

Snapd does not check for the bpf feature through the kernel, it very explicitly only checks the apparmor_parser support, see https://github.com/snapcore/snapd/blob/master/sandbox/apparmor/apparmor.go#L331-L372 for full details of how we check for features

The steps in the description reproduce this issue for me on a clean Ubuntu 20.04 server install on either 5.4.0 (GA) or 5.13.0 (HWE) kernels.

The container doesn't need to be privileged. The main steps really seem to be the installation of a classic snap followed by restart of the container.

As part of digging into this issue, I found this error:

```
Warning from stdin (line 1): apparmor_parser: Warning capping number of jobs to 0 * # of cpus == '4'
```

Which started appearing since we released LXCFS 5.0.0 (available in LXD edge and candidate).
The root cause is https://github.com/lxc/lxcfs/issues/522

This only causes an apparmor warning and the parser is otherwise functional, but this may also be interacting with snapd somehow, causing it to mis-detect available or missing features somehow?

I pulled a clean 20.04 cloud image VM from https://cloud-images.ubuntu.com/focal/current/

root@ubuntu:/home/guest# grep PRETTY /etc/os-release
PRETTY_NAME="Ubuntu 20.04.4 LTS"
root@ubuntu:/home/guest# uname -a
Linux ubuntu 5.4.0-104-generic #118-Ubuntu SMP Wed Mar 2 19:02:41 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

root@ubuntu:/home/guest# echo 'profile snap-test { capability bpf, }' | apparmor_parser --preprocess
AppArmor parser error, in stdin line 1: Invalid capability bpf.

as expected:

root@ubuntu:/home/guest# cat /var/lib/snapd/apparmor/snap-confine/cap-bpf
cat: /var/lib/snapd/apparmor/snap-confine/cap-bpf: No such file or directory

root@ubuntu:/home/guest# snap list lxd
Name Version Rev Tracking Publisher Notes
lxd 4.0.9 22526 4.0/stable/… canonical✓ -
root@ubuntu:/home/guest# lxd init --auto
root@ubuntu:/home/guest# lxc launch images:ubuntu/20.04 c1
Creating c1
Starting c1
root@ubuntu:/home/guest# lxc exec c1 -- apt install snapd -y
..
root@ubuntu:/home/guest# lxc exec c1 -- snap list
No snaps are installed yet. Try 'snap install hello-world'.

As expected bpf isn't supported by apparmor_parser:

root@c1:~# echo 'profile snap-test { capability bpf, }' | apparmor_parser --preprocess

restarted the guest:
root@ubuntu:/home/guest# lxc restart c1

and it's still the same:
root@ubuntu:/home/guest# lxc exec c1 -t /bin/bash
root@c1:~# echo 'profile snap-test { capability bpf, }' | apparmor_parser --preprocess
AppArmor parser error, in stdin line 1: Invalid capability bpf.
profile snap-test { capability bpfroot@c1:~#
root@c1:~#
root@c1:~# cat /var/lib/snapd/apparmor/snap-confine/cap-bpf
cat: /var/lib/snapd/apparmor/snap-confine/cap-bpf: No such file or directory

The only difference is that I didn't install or run distrobuilder. So I proceeded to do it.

root@c1:~# snap install distrobuilder --edge --classic
2022-03-12T09:17:52Z INFO Waiting for automatic snapd restart...
distrobuilder (edge) git-f883431 from Stéphane Graber (stgraber) installed
root@c1:~# echo 'profile snap-test { capability bpf, }' | apparmor_parser --preprocess
AppArmor parser error, in stdin line 1: Invalid capability bpf.
profile snap-test { capability bpf
root@c1:~# cat /var/lib/snapd/apparmor/snap-confine/cap-bpf
cat: /var/lib/snapd/apparmor/snap-confine/cap-bpf: No such file or directory

and restart:

root@c1:~# exit
root@ubuntu:/home/guest# lxc restart c1
root@ubuntu:/home/guest# lxc exec c1 -t /bin/bash
root@c1:~# echo 'profile snap-test { capability bpf, }' | apparmor_parser --preprocess
AppArmor parser error, in stdin line 1: Invalid capability bpf.
profile snap-test { capability bpfroo...

Yeah, that's because you're using LXD 4.23 which has the older LXCFS and doesn't trigger the apparmor warning message which then confuses everything else (per previous comment).

If you `snap install lxd --channel=latest/edge`, you should start getting the broken behavior.
Note however that we did fix LXCFS so that broken behavior on an apparmor warning will soon go away as the warning in question will no longer be issued by apparmor.

@stgraber while I know snapd currently isn't vendoring the apparmor_parser, is the LXD snap vendoring apparmor?

Does changing the sysctl kernel.unprivileged_bpf_disabled to 0 fix the issue?

This has been recently changed in all kernels supporting ebpf to default to 2, even on those kernels that do not support CAP_BPF.

Cascardo.

@jjohansen nope, we use the apparmor_parser that ships in core20.

@cascardo that sysctl does indeed change when/if bpf_capable() is called, so a possibility to explore.

We start seeing the same behavior in the Anbox Cloud CI for a few days now. What tests (via spread) primarily do:

1. ssh to an existing arm64 VM
2. Install a fresh LXD from latest/edge and configure it with the following preseed (setting security.nesting to true or false doesn't make a difference):

```
  config:
    cluster.https_address: $addr:8443
    core.https_address: $addr:8443
  cluster:
    enabled: false
    server_name: lxd0
  networks:
  - name: lxdbr0
    type: bridge
    config:
      ipv4.nat: true
      ipv4.dhcp.expiry: infinite
      ipv4.address: $LXD_SUBNET
      ipv6.address: none
  profiles:
  - name: default
    config:
      security.nesting: true
    devices:
      root:
        path: /
        pool: default
        type: disk
      eth0:
        type: nic
        nictype: bridged
        parent: lxdbr0
  storage_pools:
  - name: default
    driver: zfs
    config:
      size: 20GB
```

3. Now juju starts to bootstrap a controller on top of LXD and then we deploy our charms.
4. At some point the tests run the following:

11:08:00 ++++ timeout -s KILL 5m sudo -u root -H /snap/bin/juju ssh ams/0 -o 'ConnectionAttempts 30' -- /snap/bin/amc image add bionic:android10:arm64 /home/ubuntu/anbox-lxd-image.tar.xz
11:08:00 snap-confine has elevated permissions and is not confined but should be. Refusing to continue to avoid permission escalation attacks

This seems to be consis11:08:00 ++++ timeout -s KILL 5m sudo -u root -H /snap/bin/juju ssh ams/0 -o 'ConnectionAttempts 30' -- /snap/bin/amc image add bionic:android10:arm64 /home/ubuntu/anbox-lxd-image.tar.xz
11:08:00 snap-confine has elevated permissions and is not confined but should be. Refusing to continue to avoid permission escalation attacks

This doesn't always happen but I haven't yet checked if it's only happening on one particular machine. The VMs are all running 20.04

I can easily reproduce with stgraber's instructions on an arm64 VM running 20.04 with the 5.4 kernel.

Maciej asked me to provide the output of the following commands:

in the LXD container:

root@c1:~# echo 'profile snap-test { capability bpf, }' | apparmor_parser --preprocess ; echo $?
Warning from stdin (line 1): apparmor_parser: Warning capping number of jobs to 0 * # of cpus == '16'AppArmor parser error, in stdin line 1: Invalid capability bpf.
0

on the host:

ubuntu@anbox-streaming-stack-0:~$ echo 'profile snap-test { capability bpf, }' | apparmor_parser --preprocess ; echo $?
AppArmor parser error, in stdin line 1: Invalid capability bpf.
profile snap-test { capability bpf1

It appears that the same version of apparmor parser (2.13.3 as reported by Simon), behaves differently when running in the container. Specifically, the command that snapd also executes did not fail in the container, while it should have. Actually there's even an error message but the exit code is still 0. Since it did not fail, then snapd assumes that bpf is supported and generates a snippet for snap-confine apparmor profile.

I think the first step here, is to figure out why apparmor_parser did not fail.

The warning message:
"""
Warning from stdin (line 1): apparmor_parser: Warning capping number of jobs to 0 * # of cpus == '16'
"""

Is caused by a LXCFS bug which we've since fixed. Systems still showing this message are in need of a snap refresh and host reboot.

It's quite likely that the warning is somehow causing apparmor_parser to exit non-zero which would be the core of the issue here.

"""
Warning from stdin (line 1): apparmor_parser: Warning capping number of jobs to 0 * # of cpus == '16'
"""

Does not cause any change in return codes.

@maciek-borzecki the parser can change its behavior based on a few things.
1. the kernel its built against. This would not change behavior when run in a container vs at system level.

2. If a feature-file is specified, via --features-file, --policy-features, or --kernel-features. This allows overriding the normal policy and kernel examination that the parser does when compiling policy.

3. If /sys/kernel/security/apparmor/features is not available. The parser will fallback to an old set of features available in a kernel before the kernel module started exporting what the kernel module supports on the running kernel.

seeing more capability bpf failures in this LP as well
https://bugs.launchpad.net/cloud-archive/+bug/1988270

This is missing the SRU template information - there seems to be a test case there, but the rest is missing.

Description updated with the SRU template information.

Hello Stéphane, or anyone else affected,

Accepted apparmor into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apparmor/2.13.3-7ubuntu5.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

I didn't want to block this from getting into focal-proposed because I think it would be good to get some initial testing already, especially that the package has been sponsored by someone from the security team. But I'm not entirely happy with the test/regression-potential stories for this bug, so maybe it would be nice to re-visit that. Both of those seem to be copied from comments under the long thread in the bug here, which is fine, but I'd like to make sure this is all that needs testing.

I don't know apparmor code, but I'd like if someone could re-visit the cap* patches added to this upload and check if there's anything that we could regress *besides* in the actual feature-set. There's a lot of patches here touching a lot of code, so there's surely a risk that we might influence existing code in some additional way? Or are those three cases the only possible risk?

For such a change, is there also some other testing we could do to make sure things work properly?

All autopkgtests for the newly accepted apparmor (2.13.3-7ubuntu5.2) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

libreoffice/1:6.4.7-0ubuntu0.20.04.6 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#apparmor

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Verification done. The autopkgtest failure for libreoffice was a temporary issue with the test infrastructure that passed when it was retriggered.

I did a quick test with apparmor focal-yoga and -proposed this morning to verify it fixes https://bugs.launchpad.net/cloud-archive/+bug/1988270

# testing with focal-yoga
Apparmor version tested - 2.13.3-7ubuntu5.2

generate focal-yoga instance
juju ssh nova-compute/0

sudo apt-cache policy apparmor
sudo vim /etc/apt/sources.list
# add -proposed
deb http://nova.clouds.archive.ubuntu.com/ubuntu/ focal-proposed main universe
# save and exit
sudo apt-get upgrade apparmor
sudo systemctl restart apparmor
tail -n 1000 /var/log/syslog

# no errors are thrown by apparmor
Jan 9 15:27:40 juju-3151fe-testapparmor-9 apparmor.systemd[62260]: Restarting AppArmor
Jan 9 15:27:40 juju-3151fe-testapparmor-9 apparmor.systemd[62260]: Reloading AppArmor profiles
Jan 9 15:27:40 juju-3151fe-testapparmor-9 apparmor.systemd[62274]: Skipping profile in /etc/apparmor.d/disable: usr.bin.nova-compute
Jan 9 15:27:40 juju-3151fe-testapparmor-9 kernel: [ 1440.404862] audit: type=1400 audit(1673278060.118:74): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="virt-aa-helper" pid=62273 comm="apparmor_parser"
Jan 9 15:27:40 juju-3151fe-testapparmor-9 kernel: [ 1440.406888] audit: type=1400 audit(1673278060.118:75): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="/usr/bin/man" pid=62275 comm="apparmor_parser"
Jan 9 15:27:40 juju-3151fe-testapparmor-9 kernel: [ 1440.406890] audit: type=1400 audit(1673278060.118:76): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="man_filter" pid=62275 comm="apparmor_parser"
Jan 9 15:27:40 juju-3151fe-testapparmor-9 kernel: [ 1440.406892] audit: type=1400 audit(1673278060.118:77): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="man_groff" pid=62275 comm="apparmor_parser"
Jan 9 15:27:40 juju-3151fe-testapparmor-9 kernel: [ 1440.408833] audit: type=1400 audit(1673278060.122:78): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="/usr/lib/snapd/snap-confine" pid=62276 comm="apparmor_parser"
Jan 9 15:27:40 juju-3151fe-testapparmor-9 kernel: [ 1440.408838] audit: type=1400 audit(1673278060.122:79): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="/usr/lib/snapd/snap-confine//mount-namespace-capture-helper" pid=62276 comm="apparmor_parser"
Jan 9 15:27:40 juju-3151fe-testapparmor-9 kernel: [ 1440.415917] audit: type=1400 audit(1673278060.130:80): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="/usr/lib/NetworkManager/nm-dhcp-client.action" pid=62277 comm="apparmor_parser"
Jan 9 15:27:40 juju-3151fe-testapparmor-9 kernel: [ 1440.415921] audit: type=1400 audit(1673278060.130:81): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="/usr/lib/NetworkManager/nm-dhcp-helper" pid=62277 comm="apparmor_parser"
Jan 9 15:27:40 juju-3151f...

In regards to @sil2100 questions, I can review the patches(~24) and double check any use cases.

Lukasz is correct, we should be diligent in backporting the upstream patches. In regards to testing it's important to ensure apparmor and its new features work as intended with no errors in logs.
As well as apparmor not quitting after a restart. All three effected LP's should be thoroughly tested.

There are over 20 patches being backported from upstream apparmor3.0
They fall into 3 categories.
1. capabilities improvements (maintain and generate the capabilities list used by apparmor)
2. abi [0]
3. mqueue

[0] https://gitlab.com/apparmor/apparmor/-/wikis/AppArmorpolicyfeaturesabi#why-were-feature-abi-rules-added

Most of the cap* patches are around generating and maintaining a list of supported capabilities.
the first 2 caps (cap1 & cap2) introduce new scripts to generate a list of current capabilities and
apparmor-bash related profiles.

# cap1-Generate-CAPABILITIES-in-a-script-due-to-make-4.3.patch

there is a new command under /common
./list_capabilities.sh
# code that generates a list of capabilities
CAP_AUDIT_CONTROL
CAP_AUDIT_READ
CAP_AUDIT_WRITE
...
CAP_CHOWN

# new python script to create vim profiles with
python create-apparmor.vim.py
# generates a new file called apparmor.vim.in

# cap2-parser-Move-to-a-pre-generated-cap_names.h.patch
use a pre-generated list of capabilities so that all capabilities are
supported even when building against older kernels.

The rest of the cap* patches are code cleanup related.

@sli2100 I hope that answers some of the concern about capabilities patches.

I will work on testing the other 3 affected LP's (1988270, 1728130, 1993353).
So a total of 4 Lp's will be addressed.

Please let me know if I/someone else can elaborate on the testing that needs to happen before approval.

> # new python script to create vim profiles with
>
> python create-apparmor.vim.py

For the records: create-apparmor.vim.py exists since years, and ...

> # generates a new file called apparmor.vim.in

... it uses apparmor.vim.in as _input_ and generates the apparmor.vim file (syntax highlighting for vim), but (unless the Ubuntu packaging does this) this file does _not_ get installed in a location where vim finds it. (In openSUSE, I regularly submit it to the vim package manually. No idea if/how this is handled in Ubuntu.)

Is there something blocking the release for focal?

Focal is in soft freeze atm, once the soft freeze is lifted this should land

ack thanks!

This bug was fixed in the package apparmor - 2.13.3-7ubuntu5.2

---------------
apparmor (2.13.3-7ubuntu5.2) focal; urgency=medium

  * Add capability upstream patches to fix LP: #1964636
    - u/cap1-Generate-CAPABILITIES-in-a-script-due-to-make-4.3.patch: move
    code that generates a list of capabilities to a script in common/
    - u/cap2-parser-Move-to-a-pre-generated-cap_names.h.patch: use a
    pre-generated list of capabilities so that all capabilities are
    supported even when building against older kernels.
    - u/cap3-parser-cleanup-capability_table-generation-by-droppi.patch: drop
    sys_log static declaration because it's already in the generated list.
    - u/cap4-parser-unify-capability-name-handling.patch: drop internal
    hardcoded capability table.
    - u/cap5-parser-Makefile-use-LC_ALL-C-when-invoking-sed.patch: use
    LC_ALL=C when invoking sed.
    - u/cap6-parser-Add-warning-to-capability_table-about-the-nee.patch: add
    warning to capability_table about the need to update the Makefile.
    - u/cap7-Add-CAP_BPF-and-CAP_PERFMON-to-severity.db.patch: add
    support for cap_bpf and cap_perfmon
    - u/cap8-parser-Makefile-fix-generated-cap-comparison-against.patch: fix
    generated cap comparison against known list
  * Add upstream patches for abi support. LP: #1728130
    - u/abi1-parser-feature-abi-setup-parser-to-intersect-policy-.patch: add
    the ability to intersect parser and kernel features in the parser.
    - u/abi2-parser-add-basic-support-for-feature-abis.patch: add support
    to specify a feature abi.
    - u/abi3-pin-abi-2.13.patch: add and pin a policy abi for 2.13
    - u/abi4-parser-fix-abi-rule-and-pinned-feature-file-interact.patch: fix
    abi rule and pinned feature file interaction
    - apparmor.install: add 2.13 abi file to be installed in /etc/apparmor.d/abi/
  * Add mqueue patches. LP: #1993353
    - u/mqueue1-parser-add-parser-support-for-message-queue-mediatio.patch:
    add parser support for mqueue mediation
    - u/mqueue2-tests-add-posix-message-queue-regression-tests.patch: add
    posix mqueue regression tests
    - u/mqueue3-utils-add-message-queue-rules-parsing-in-python-tool.patch:
    add support in python tools to parse mqueue rules
    - u/mqueue4-parser-add-parser-simple-tests-for-mqueue-rules.patch: add
    parser simple tests for mqueue
    - u/mqueue5-parser-place-perm-on-name-as-well-as-name-label-comb.patch:
    add permissions on name and also on name + label
    - u/mqueue6-libapparmor-add-support-for-requested-and-denied-on-.patch:
    add parsing support for "denied" and "requested" from audit logs
    - u/mqueue7-libapparmor-add-support-for-class-in-logparsing.patch: add
    parsing support for "class" from audit logs
    - u/mqueue8-utils-add-logparser-support-for-mqueue.patch: add logparser
    support for mqueue rules
    - u/mqueue9-tests-add-sysv-message-queue-regression-tests.patch: add
    sysv mqueue regression tests
    - u/mqueue10-parser-enable-mqueue-rules-when-abi-is-not-set.patch:
    override pinned features for mqueue rules when abi is not set in policy.
    - debian/rules: create mqueue testcase empty files for libapparmor tests.
  * Closes LP...

The verification of the Stable Release Update for apparmor has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.