Ubuntu
nova-compute package

When attaching multiattach volumes apparmor nova-compute profile blocks some operations

Bug #2039161 reported by Giuseppe Petralia
18
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Nova Compute Charm
In Progress
Undecided
Unassigned
nova-compute (Ubuntu)
Invalid
Undecided
Unassigned

Bug Description

This is happening in jammy nova-compute version 3:25.2.0-0ubuntu1

When attaching multiattach volumes the following warnings are logged:

2023-10-12 09:45:25.723 3906368 WARNING os_brick.initiator.connectors.nvmeof [req-9bfe2fce-9a78-4df3-8200-0e5901016e72 db682b8df0304e36b91345b7ce594aff 12b52497ff32492f888ae0ba837c2ae6 - 16b5db126dbb42fd804296790a3f9f6a 16b5db126dbb42fd804296790a3f9f6a] Process execution error in _get_host_uuid: [Errno 13] Permission denied
Command: blkid /dev/mapper/vg0-lvroot -s UUID -o value
Exit code: -
Stdout: None
Stderr: None: oslo_concurrency.processutils.ProcessExecutionError: [Errno 13] Permission denied
2023-10-12 09:45:25.800 3906368 WARNING os_brick.initiator.connectors.nvmeof [req-9bfe2fce-9a78-4df3-8200-0e5901016e72 db682b8df0304e36b91345b7ce594aff 12b52497ff32492f888ae0ba837c2ae6 - 16b5db126dbb42fd804296790a3f9f6a 16b5db126dbb42fd804296790a3f9f6a] Unknown error when checking presence of nvme: [Errno 13] Permission denied: 'nvme': PermissionError: [Errno 13] Permission denied: 'nvme'
2023-10-12 09:45:25.806 958579 WARNING os_brick.privileged.nvmeof [-] Could not generate host nqn: [Errno 13] Permission denied: 'nvme'

and in syslog I can see:

Oct 12 09:13:59 machine1 kernel: [18324599.319817] audit: type=1400 audit(1697102039.363:696303): apparmor="DENIED" operation="exec" class="file" profile="/usr/bin/nova-compute" name="/usr/sbin/blkid" pid=4085668 comm="privsep-helper" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
Oct 12 09:13:59 machine1 kernel: [18324599.319844] audit: type=1400 audit(1697102039.363:696304): apparmor="DENIED" operation="exec" class="file" profile="/usr/bin/nova-compute" name="/usr/sbin/blkid" pid=4085668 comm="privsep-helper" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
Oct 12 09:13:59 machine1 kernel: [18324599.346662] audit: type=1400 audit(1697102039.387:696305): apparmor="DENIED" operation="exec" class="file" profile="/usr/bin/nova-compute" name="/usr/sbin/nvme" pid=4085671 comm="nova-compute" requested_mask="x" denied_mask="x" fsuid=64060 ouid=0
Oct 12 09:13:59 machine1 kernel: [18324599.346745] audit: type=1400 audit(1697102039.387:696306): apparmor="DENIED" operation="exec" class="file" profile="/usr/bin/nova-compute" name="/usr/sbin/nvme" pid=4085671 comm="nova-compute" requested_mask="x" denied_mask="x" fsuid=64060 ouid=0
Oct 12 09:13:59 machine1 kernel: [18324599.364823] audit: type=1400 audit(1697102039.407:696307): apparmor="DENIED" operation="open" class="file" profile="/usr/bin/nova-compute" name="/etc/nvme/hostnqn" pid=1343874 comm="nova-compute" requested_mask="r" denied_mask="r" fsuid=64060 ouid=0
Oct 12 09:13:59 machine1 kernel: [18324599.369262] audit: type=1400 audit(1697102039.411:696308): apparmor="DENIED" operation="exec" class="file" profile="/usr/bin/nova-compute" name="/usr/sbin/nvme" pid=4085672 comm="privsep-helper" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
Oct 12 09:13:59 machine1 kernel: [18324599.369299] audit: type=1400 audit(1697102039.411:696309): apparmor="DENIED" operation="exec" class="file" profile="/usr/bin/nova-compute" name="/usr/sbin/nvme" pid=4085672 comm="privsep-helper" requested_mask="x" denied_mask="x" fsuid=0 ouid=0

Warnings are gone if I set the apparmor profile to complain with:

aa-complain /etc/apparmor.d/usr.bin.nova-compute

See original description
Giuseppe Petralia (peppepetra)
description: updated
Giuseppe Petralia (peppepetra)
description: updated
Revision history for this message
Nobuto Murata (nobuto) wrote :

Fwiw, when the last time I looked into, access denial for blkid and nvme wasn't in the critical path although it was a valid issue for NVMEoF use case.
https://bugs.launchpad.net/charm-nova-compute/+bug/1979812

Revision history for this message
Felipe Reyes (freyes) wrote :

Hi Giuseppe, can you check the version of the charms you are running?, because the bug Nobuto pointed out has been fixed and backported all the way back to Ussuri - https://review.opendev.org/q/message:1979812

I'm marking the "nova" package task as invalid since the apparmor is owned by the charm.

Changed in nova-compute (Ubuntu):
status: New → Invalid
Changed in charm-nova-compute:
status: New → Incomplete
Revision history for this message
Giuseppe Petralia (peppepetra) wrote (last edit ):

Hi Felipe, we are using nova-compute from yoga/stable rev. 664

rev. 664 has the update to apparmor from https://review.opendev.org/q/message:1979812

Checked on one machine

```
root@machine5:~# grep scsi/device /etc/apparmor.d/usr.bin.nova-compute
  /sys/bus/scsi/devices/ r,

root@machine5:~# grep run/lock/nova /etc/apparmor.d/usr.bin.nova-compute
  /run/lock/nova/* wk,

```

I checked latest yoga/stable and I can't see any update related to blkid

Checked with
```
mkdir nova-compute
juju download ch:nova-compute --channel yoga/stable
unzip nova-compute_02670ee.charm -d nova-compute
grep blkid nova-compute/templates/usr.bin.nova-compute
```

can you clarify if latest yoga/stable will allow the Denied in the description and if we need them to have multiattach to work?

Thanks

Revision history for this message
Giuseppe Petralia (peppepetra) wrote :

I did some tests and apparently even with the warnings multiattach volumes work as expected.

Felipe Reyes (freyes)
Changed in charm-nova-compute:
status: Incomplete → New
Revision history for this message
Felipe Reyes (freyes) wrote :

it looks like os-brick has a fallback mechanism - https://opendev.org/openstack/os-brick/src/branch/master/os_brick/privileged/nvmeof.py#L33 - but I will enable these in the AA profile anyway.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-nova-compute (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-nova-compute/+/898127

Changed in charm-nova-compute:
status: New → In Progress
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.