OpenStack Nova Compute Charm

Nova fails to attach Cinder volume when using LVM backend

Bug #1916487 reported by Chris MacNaughton
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Nova Compute Charm
In Progress
Undecided
Chris MacNaughton

Bug Description

When an admin attempts to attach a volume to an instance, Nova attempts to do so and then fails, see reproducer steps below, followed by error stack trace:

$ openstack server create --image=bionic --flavor=m1.small --network=private test
+-------------------------------------+-----------------------------------------------+
| Field | Value |
+-------------------------------------+-----------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | |
| OS-EXT-SRV-ATTR:host | None |
| OS-EXT-SRV-ATTR:hypervisor_hostname | None |
| OS-EXT-SRV-ATTR:instance_name | |
| OS-EXT-STS:power_state | NOSTATE |
| OS-EXT-STS:task_state | scheduling |
| OS-EXT-STS:vm_state | building |
| OS-SRV-USG:launched_at | None |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| addresses | |
| adminPass | AAfgZY8XK826 |
| config_drive | |
| created | 2021-02-22T11:19:05Z |
| flavor | m1.small (2) |
| hostId | |
| id | 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1 |
| image | bionic (661671ab-1d6c-46b8-9bed-60dc30b1f1c4) |
| key_name | key |
| name | test |
| progress | 0 |
| project_id | 576934a931a5460fa08ac6f7d3484364 |
| properties | |
| security_groups | name='default' |
| status | BUILD |
| updated | 2021-02-22T11:19:04Z |
| user_id | a3c4707778ed4775b7f70d37b4246136 |
| volumes_attached | |
+-------------------------------------+-----------------------------------------------+

$ openstack volume create --size 5 test-volume
+---------------------+--------------------------------------+
| Field | Value |
+---------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2021-02-22T11:23:24.000000 |
| description | None |
| encrypted | False |
| id | 79d0f7c7-ce75-42b8-b862-cbb5205afb42 |
| migration_status | None |
| multiattach | False |
| name | test-volume |
| properties | |
| replication_status | None |
| size | 5 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| type | __DEFAULT__ |
| updated_at | None |
| user_id | a3c4707778ed4775b7f70d37b4246136 |
+---------------------+--------------------------------------+
$ openstack server show 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1
+-------------------------------------+----------------------------------------------------------+
| Field | Value |
+-------------------------------------+----------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | juju-186993-zaza-589aa03f7694-10 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | juju-186993-zaza-589aa03f7694-10 |
| OS-EXT-SRV-ATTR:instance_name | instance-00000001 |
| OS-EXT-STS:power_state | Running |
| OS-EXT-STS:task_state | None |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2021-02-22T11:21:02.000000 |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| addresses | private=192.168.0.205 |
| config_drive | |
| created | 2021-02-22T11:19:04Z |
| flavor | m1.small (2) |
| hostId | 37491a934108306544ce5d65a9d42984a38e442e477b70a6eedf39be |
| id | 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1 |
| image | bionic (661671ab-1d6c-46b8-9bed-60dc30b1f1c4) |
| key_name | key |
| name | test |
| progress | 0 |
| project_id | 576934a931a5460fa08ac6f7d3484364 |
| properties | |
| security_groups | name='default' |
| status | ACTIVE |
| updated | 2021-02-22T11:21:03Z |
| user_id | a3c4707778ed4775b7f70d37b4246136 |
| volumes_attached | id='79d0f7c7-ce75-42b8-b862-cbb5205afb42' |
+-------------------------------------+----------------------------------------------------------+
$ # wait a minute
$ openstack server show 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1
+-------------------------------------+----------------------------------------------------------+
| Field | Value |
+-------------------------------------+----------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | juju-186993-zaza-589aa03f7694-10 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | juju-186993-zaza-589aa03f7694-10 |
| OS-EXT-SRV-ATTR:instance_name | instance-00000001 |
| OS-EXT-STS:power_state | Running |
| OS-EXT-STS:task_state | None |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2021-02-22T11:21:02.000000 |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| addresses | private=192.168.0.205 |
| config_drive | |
| created | 2021-02-22T11:19:04Z |
| flavor | m1.small (2) |
| hostId | 37491a934108306544ce5d65a9d42984a38e442e477b70a6eedf39be |
| id | 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1 |
| image | bionic (661671ab-1d6c-46b8-9bed-60dc30b1f1c4) |
| key_name | key |
| name | test |
| progress | 0 |
| project_id | 576934a931a5460fa08ac6f7d3484364 |
| properties | |
| security_groups | name='default' |
| status | ACTIVE |
| updated | 2021-02-22T11:21:03Z |
| user_id | a3c4707778ed4775b7f70d37b4246136 |
| volumes_attached | |
+-------------------------------------+----------------------------------------------------------+

The volumes_attached key goes from showing the new volume, to showing nothing, and the nova compute logs show:

2021-02-22 11:23:43.873 38520 DEBUG os_brick.initiator.connectors.iscsi [req-f6d5f877-4284-4116-bb81-2b37dd272183 a3c4707778ed4775b7f70d37b4246136 576934a931a5460fa08ac6f7d3484364 - d90e8021bde343bcb4ef1270c512a
a60 d90e8021bde343bcb4ef1270c512aa60] <== connect_volume: exception (17ms) ProcessExecutionError(None, None, None, 'iscsiadm -m node', '[Errno 13] Permission denied') trace_logging_wrapper /usr/lib/python3/dist-
packages/os_brick/utils.py:153
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [req-f6d5f877-4284-4116-bb81-2b37dd272183 a3c4707778ed4775b7f70d37b4246136 576934a931a5460fa08ac6f7d3484364 - d90e8021bde343bcb4ef1270c512aa60 d90e8021b
de343bcb4ef1270c512aa60] [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] Driver failed to attach volume 79d0f7c7-ce75-42b8-b862-cbb5205afb42 at /dev/vdb: oslo_concurrency.processutils.ProcessExecutionError: [Er
rno 13] Permission denied
Command: iscsiadm -m node
Exit code: -
Stdout: None
Stderr: None
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] Traceback (most recent call last):
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] File "/usr/lib/python3/dist-packages/os_brick/privileged/rootwrap.py", line 169, in execute
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] return execute_root(*cmd, **kwargs)
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] File "/usr/lib/python3/dist-packages/oslo_privsep/priv_context.py", line 247, in _wrap
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] return self.channel.remote_call(name, args, kwargs)
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] File "/usr/lib/python3/dist-packages/oslo_privsep/daemon.py", line 204, in remote_call
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] raise exc_type(*result[2])
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] PermissionError: [Errno 13] Permission denied
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1]
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] During handling of the above exception, another exception occurred:
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1]
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] Traceback (most recent call last):
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] File "/usr/lib/python3/dist-packages/os_brick/initiator/connectors/iscsi.py", line 513, in connect_vo
lume
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] return self._connect_single_volume(connection_properties)
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] File "/usr/lib/python3/dist-packages/os_brick/utils.py", line 61, in _wrapper
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] return r.call(f, *args, **kwargs)
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] File "/usr/lib/python3/dist-packages/retrying.py", line 206, in call
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] return attempt.get(self._wrap_exception)
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] six.reraise(self.value[0], self.value[1], self.value[2])
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] File "/usr/lib/python3/dist-packages/six.py", line 703, in reraise
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] raise value
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] File "/usr/lib/python3/dist-packages/os_brick/initiator/connectors/iscsi.py", line 569, in _connect_single_volume
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] self._connect_vol(self.device_scan_attempts, props, data)
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] File "/usr/lib/python3/dist-packages/os_brick/initiator/connectors/iscsi.py", line 626, in _connect_vol
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] session, manual_scan = self._connect_to_iscsi_portal(props)
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] File "/usr/lib/python3/dist-packages/os_brick/initiator/connectors/iscsi.py", line 1035, in _connect_to_iscsi_portal
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] out, err = self._run_iscsiadm(connection_properties, (),
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] File "/usr/lib/python3/dist-packages/os_brick/initiator/connectors/iscsi.py", line 985, in _run_iscsiadm
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] (out, err) = self._execute('iscsiadm', '-m', 'node', '-T',
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] File "/usr/lib/python3/dist-packages/os_brick/executor.py", line 52, in _execute
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] result = self.__execute(*args, **kwargs)
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] File "/usr/lib/python3/dist-packages/os_brick/privileged/rootwrap.py", line 186, in execute
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] raise putils.ProcessExecutionError(
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] oslo_concurrency.processutils.ProcessExecutionError: [Errno 13] Permission denied
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] Command: iscsiadm -m node -T iqn.2010-10.org.openstack:volume-79d0f7c7-ce75-42b8-b862-cbb5205afb42 -p 10.5.0.6:3260
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] Exit code: -
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] Stdout: None
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] Stderr: None
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1]
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] During handling of the above exception, another exception occurred:
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1]
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] Traceback (most recent call last):
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] File "/usr/lib/python3/dist-packages/os_brick/privileged/rootwrap.py", line 169, in execute
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] return execute_root(*cmd, **kwargs)
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] File "/usr/lib/python3/dist-packages/oslo_privsep/priv_context.py", line 247, in _wrap
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] return self.channel.remote_call(name, args, kwargs)
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] File "/usr/lib/python3/dist-packages/oslo_privsep/daemon.py", line 204, in remote_call
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] raise exc_type(*result[2])
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] PermissionError: [Errno 13] Permission denied
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1]
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] During handling of the above exception, another exception occurred:
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1]
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] Traceback (most recent call last):
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] File "/usr/lib/python3/dist-packages/nova/virt/block_device.py", line 595, in _volume_attach
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] virt_driver.attach_volume(
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 1842, in attach_volume
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] self._connect_volume(context, connection_info, instance,
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 1623, in _connect_volume
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] vol_driver.connect_volume(connection_info, instance)
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/volume/iscsi.py", line 64, in connect_volume
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] device_info = self.connector.connect_volume(connection_info['data'])
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] File "/usr/lib/python3/dist-packages/os_brick/utils.py", line 150, in trace_logging_wrapper
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] result = f(*args, **kwargs)
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] File "/usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py", line 359, in inner
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] return f(*args, **kwargs)
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] File "/usr/lib/python3/dist-packages/os_brick/initiator/connectors/iscsi.py", line 519, in connect_volume
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] self._cleanup_connection(connection_properties, force=True)
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] File "/usr/lib/python3/dist-packages/os_brick/initiator/connectors/iscsi.py", line 897, in _cleanup_connection
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] devices_map = self._get_connection_devices(connection_properties,
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] File "/usr/lib/python3/dist-packages/os_brick/initiator/connectors/iscsi.py", line 815, in _get_connection_devices
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] nodes = self._get_iscsi_nodes()
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] File "/usr/lib/python3/dist-packages/os_brick/initiator/connectors/iscsi.py", line 145, in _get_iscsi_nodes
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] out, err = self._execute('iscsiadm', '-m', 'node', run_as_root=True,
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] File "/usr/lib/python3/dist-packages/os_brick/executor.py", line 52, in _execute
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] result = self.__execute(*args, **kwargs)
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] File "/usr/lib/python3/dist-packages/os_brick/privileged/rootwrap.py", line 186, in execute
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] raise putils.ProcessExecutionError(
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] oslo_concurrency.processutils.ProcessExecutionError: [Errno 13] Permission denied
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] Command: iscsiadm -m node
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] Exit code: -
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] Stdout: None
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] Stderr: None
2021-02-22 11:23:43.874 38520 ERROR nova.virt.block_device [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1]
2021-02-22 11:23:44.389 38520 DEBUG oslo_messaging._drivers.amqpdriver [req-f6d5f877-4284-4116-bb81-2b37dd272183 a3c4707778ed4775b7f70d37b4246136 576934a931a5460fa08ac6f7d3484364 - d90e8021bde343bcb4ef1270c512aa60 d90e8021bde343bcb4ef1270c512aa60] CALL msg_id: 548cfebe499a4109afb7f53c6a33a82a exchange 'nova' topic 'conductor' _send /usr/lib/python3/dist-packages/oslo_messaging/_drivers/amqpdriver.py:633
2021-02-22 11:23:44.425 38520 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: 548cfebe499a4109afb7f53c6a33a82a __call__ /usr/lib/python3/dist-packages/oslo_messaging/_drivers/amqpdriver.py:480
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [req-f6d5f877-4284-4116-bb81-2b37dd272183 a3c4707778ed4775b7f70d37b4246136 576934a931a5460fa08ac6f7d3484364 - d90e8021bde343bcb4ef1270c512aa60 d90e8021bde343bcb4ef1270c512aa60] [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] Failed to attach 79d0f7c7-ce75-42b8-b862-cbb5205afb42 at /dev/vdb: oslo_concurrency.processutils.ProcessExecutionError: [Errno 13] Permission denied
Command: iscsiadm -m node
Exit code: -
Stdout: None
Stderr: None
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] Traceback (most recent call last):
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] File "/usr/lib/python3/dist-packages/os_brick/privileged/rootwrap.py", line 169, in execute
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] return execute_root(*cmd, **kwargs)
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] File "/usr/lib/python3/dist-packages/oslo_privsep/priv_context.py", line 247, in _wrap
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] return self.channel.remote_call(name, args, kwargs)
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] File "/usr/lib/python3/dist-packages/oslo_privsep/daemon.py", line 204, in remote_call
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] raise exc_type(*result[2])
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] PermissionError: [Errno 13] Permission denied
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1]
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] During handling of the above exception, another exception occurred:
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1]
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] Traceback (most recent call last):
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] File "/usr/lib/python3/dist-packages/os_brick/initiator/connectors/iscsi.py", line 513, in connect_volume
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] return self._connect_single_volume(connection_properties)
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] File "/usr/lib/python3/dist-packages/os_brick/utils.py", line 61, in _wrapper
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] return r.call(f, *args, **kwargs)
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] File "/usr/lib/python3/dist-packages/retrying.py", line 206, in call
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] return attempt.get(self._wrap_exception)
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] six.reraise(self.value[0], self.value[1], self.value[2])
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] File "/usr/lib/python3/dist-packages/six.py", line 703, in reraise
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] raise value
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] File "/usr/lib/python3/dist-packages/os_brick/initiator/connectors/iscsi.py", line 569, in _connect_single_volume
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] self._connect_vol(self.device_scan_attempts, props, data)
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] File "/usr/lib/python3/dist-packages/os_brick/initiator/connectors/iscsi.py", line 626, in _connect_vol
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] session, manual_scan = self._connect_to_iscsi_portal(props)
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] File "/usr/lib/python3/dist-packages/os_brick/initiator/connectors/iscsi.py", line 1035, in _connect_to_iscsi_portal
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] out, err = self._run_iscsiadm(connection_properties, (),
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] File "/usr/lib/python3/dist-packages/os_brick/initiator/connectors/iscsi.py", line 985, in _run_iscsiadm
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] (out, err) = self._execute('iscsiadm', '-m', 'node', '-T',
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] File "/usr/lib/python3/dist-packages/os_brick/executor.py", line 52, in _execute
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] result = self.__execute(*args, **kwargs)
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] File "/usr/lib/python3/dist-packages/os_brick/privileged/rootwrap.py", line 186, in execute
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] raise putils.ProcessExecutionError(
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] oslo_concurrency.processutils.ProcessExecutionError: [Errno 13] Permission denied
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] Command: iscsiadm -m node -T iqn.2010-10.org.openstack:volume-79d0f7c7-ce75-42b8-b862-cbb5205afb42 -p 10.5.0.6:3260
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] Exit code: -
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] Stdout: None
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] Stderr: None
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1]
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] During handling of the above exception, another exception occurred:
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1]
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] Traceback (most recent call last):
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] File "/usr/lib/python3/dist-packages/os_brick/privileged/rootwrap.py", line 169, in execute
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] return execute_root(*cmd, **kwargs)
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] File "/usr/lib/python3/dist-packages/oslo_privsep/priv_context.py", line 247, in _wrap
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] return self.channel.remote_call(name, args, kwargs)
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] File "/usr/lib/python3/dist-packages/oslo_privsep/daemon.py", line 204, in remote_call
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] raise exc_type(*result[2])
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] PermissionError: [Errno 13] Permission denied
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1]
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] During handling of the above exception, another exception occurred:
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1]
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] Traceback (most recent call last):
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 6999, in _attach_volume
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] bdm.attach(context, instance, self.volume_api, self.driver,
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] File "/usr/lib/python3/dist-packages/nova/virt/block_device.py", line 46, in wrapped
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] ret_val = method(obj, context, *args, **kwargs)
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] File "/usr/lib/python3/dist-packages/nova/virt/block_device.py", line 672, in attach
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] self._do_attach(context, instance, volume, volume_api,
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] File "/usr/lib/python3/dist-packages/nova/virt/block_device.py", line 657, in _do_attach
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] self._volume_attach(context, volume, connector, instance,
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] File "/usr/lib/python3/dist-packages/nova/virt/block_device.py", line 606, in _volume_attach
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] volume_api.attachment_delete(context,
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] self.force_reraise()
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] six.reraise(self.type_, self.value, self.tb)
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] File "/usr/lib/python3/dist-packages/six.py", line 703, in reraise
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] raise value
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] File "/usr/lib/python3/dist-packages/nova/virt/block_device.py", line 595, in _volume_attach
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] virt_driver.attach_volume(
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 1842, in attach_volume
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] self._connect_volume(context, connection_info, instance,
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 1623, in _connect_volume
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] vol_driver.connect_volume(connection_info, instance)
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/volume/iscsi.py", line 64, in connect_volume
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] device_info = self.connector.connect_volume(connection_info['data'])
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] File "/usr/lib/python3/dist-packages/os_brick/utils.py", line 150, in trace_logging_wrapper
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] result = f(*args, **kwargs)
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] File "/usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py", line 359, in inner
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] return f(*args, **kwargs)
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] File "/usr/lib/python3/dist-packages/os_brick/initiator/connectors/iscsi.py", line 519, in connect_volume
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] self._cleanup_connection(connection_properties, force=True)
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] File "/usr/lib/python3/dist-packages/os_brick/initiator/connectors/iscsi.py", line 897, in _cleanup_connection
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] devices_map = self._get_connection_devices(connection_properties,
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] File "/usr/lib/python3/dist-packages/os_brick/initiator/connectors/iscsi.py", line 815, in _get_connection_devices
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] nodes = self._get_iscsi_nodes()
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] File "/usr/lib/python3/dist-packages/os_brick/initiator/connectors/iscsi.py", line 145, in _get_iscsi_nodes
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] out, err = self._execute('iscsiadm', '-m', 'node', run_as_root=True,
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] File "/usr/lib/python3/dist-packages/os_brick/executor.py", line 52, in _execute
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] result = self.__execute(*args, **kwargs)
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] File "/usr/lib/python3/dist-packages/os_brick/privileged/rootwrap.py", line 186, in execute
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] raise putils.ProcessExecutionError(
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] oslo_concurrency.processutils.ProcessExecutionError: [Errno 13] Permission denied
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] Command: iscsiadm -m node
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] Exit code: -
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] Stdout: None
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1] Stderr: None
2021-02-22 11:23:44.426 38520 ERROR nova.compute.manager [instance: 7aa8f1f0-9e72-4ded-924c-113e1b6c16d1]

It looks like the nova-compute / os-brick configuration for rootwrap doesn't allow running the iscsiadm command that's necessary to mount the Cinder LVM managed block devices (which are provided over ISCSI)

Revision history for this message
Alex Kavanagh (ajkavanagh) wrote :

Hi Chris, please could you add ubuntu/openstack versions, just to narrow the focus? Thanks.

Changed in charm-nova-compute:
status: New → Incomplete
Revision history for this message
Chris MacNaughton (chris.macnaughton) wrote :

This seems to be due to an apparmor rule pointing at the wrong path:

root@juju-5df37e-zaza-0dd13ef7f5f4-10:~# which iscsiadm
/usr/sbin/iscsiadm
root@juju-5df37e-zaza-0dd13ef7f5f4-10:~# grep iscsiadm /etc/apparmor.d/usr.bin.nova-compute
  /sbin/iscsiadm rix,

Revision history for this message
Chris MacNaughton (chris.macnaughton) wrote :

This has been specifically seen on Focal Ussuri, but is likely present on all release combinations. A fix for the apparmor profile has been proposed at: https://review.opendev.org/c/openstack/charm-nova-compute/+/776950

Changed in charm-nova-compute:
status: Incomplete → In Progress
assignee: nobody → Chris MacNaughton (chris.macnaughton)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on charm-nova-compute (master)

Change abandoned by "Chris MacNaughton <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/charm-nova-compute/+/776950
Reason: Superseded by https://review.opendev.org/c/openstack/charm-nova-compute/+/788000

Revision history for this message
Andre Ruiz (andre-ruiz) wrote :

Is this the same as https://bugs.launchpad.net/charm-nova-compute/+bug/1963922 ?

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.