Attach volume test failed（LINSTOR+iSCSI protocol）
Environment：
1.ARM64
2.root@openstcontroller:~# uname -a
Linux openstcontroller 4.15.0-130-generic #134-Ubuntu SMP Tue Jan 5 20:43:12 UTC 2021
3.root@openstcontroller:~# cat /proc/version
Linux version 4.15.0-130-generic (buildd@bos02-arm64-033) (gcc version 7.5.0 (Ubuntu/Linaro 7.5.0-3ubuntu1~18.04)) #134-Ubuntu SMP Tue Jan 5 20:43:12 UTC 2021
4.openstack version：Stein
5.LINSTOR version:linstor 1.6.1; GIT-hash: df9121a4e86502351bd7373446dc0f9233ab3bb5
6.DRBD version：DRBDADM_BUILDTAG=GIT-hash:\ baaca8a080dc54652f57da4bafb2dce51dfe9f68\ build\ by\ buildd@bos02-arm64-030,\ 2020-09-28\ 08:05:14DRBDADM_API_VERSION=2DRBD_KERNEL_VERSION_CODE=0x090019DRBD_KERNEL_VERSION=9.0.25DRBDADM_VERSION_CODE=0x090f00DRBDADM_VERSION=9.15.0
Architecture diagram：Please view the first comment picture
Cinder configuration （/etc/cinder/cinder.conf ）
root@node88:~# cat /etc/cinder/cinder.conf
[DEFAULT]
rootwrap_config = /etc/cinder/rootwrap.conf
api_paste_confg = /etc/cinder/api-paste.ini
iscsi_helper = tgtadm
volume_name_template = volume-%s
volume_group = cinder-volumes
verbose = True
state_path = /var/lib/cinder
lock_path = /var/lock/cinder
volumes_dir = /var/lib/cinder/volumes
transport_url = rabbit://openstack:password@openstcontroller
auth_strategy = keystone
my_ip = 192.168.7.88
enabled_backends = lvm,linstor,linstor_iscsi
glance_api_servers = http://openstcontroller:9292
[database]
connection = mysql+pymysql://cinder:password@openstcontroller/cinder
#connection = sqlite:////var/lib/cinder/cinder.sqlite
[keystone_authtoken]
www_authenticate_uri = http://openstcontroller:5000
auth_url = http://openstcontroller:5000
memcached_servers = openstcontroller:11211
auth_type = password
project_domain_name = default
user_domain_name = default
project_name = service
username = cinder
password = password
[linstor_iscsi]
volume_backend_name = linstor_iscsi
volume_driver = cinder.volume.drivers.linstordrv.LinstorIscsiDriver
image_volume_cache_enabled = True
linstor_default_volume_group_name=drbdpool
linstor_default_uri=linstor://192.168.7.43
linstor_default_storage_pool_name=DfltStorPool
linstor_default_resource_size=1
linstor_volume_downsize_factor=4096
linstor_controller_diskless=False
iscsi_helper=tgtadm
[lvm]
volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver
volume_group = cinder-volumes
target_protocol = iscsi
target_helper = tgtadm
[oslo_concurrency]
lock_path = /var/lib/cinder/tmp
Check status on OpenStack controller node
openstcontroller@openstcontroller:~$ openstack server list
+--------------------------------------+-----------+--------+-----------------------+-------+---------------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+-----------+--------+-----------------------+-------+---------------+
| 3910f0c6-9c65-4b09-91dc-ab39d1d88418 | cirrorARM | ACTIVE | provider=192.168.7.15 | | m1.nano |
| 7c4c32b1-13c9-4a9a-b82c-63fbae582503 | ubuntu | ACTIVE | provider=192.168.7.14 | | ubuntu50.nano |
+--------------------------------------+-----------+--------+-----------------------+-------+---------------+
openstcontroller@openstcontroller:~$ cinder service-list
+------------------+----------------------+------+----------+-------+----------------------------+---------+-----------------+---------------+
| Binary | Host | Zone | Status | State | Updated_at | Cluster | Disabled Reason | Backend State |
+------------------+----------------------+------+----------+-------+----------------------------+---------+-----------------+---------------+
| cinder-scheduler | openstcontroller | nova | enabled | up | 2021-02-05T06:19:33.000000 | - | - | |
| cinder-volume | node88@linstor_iscsi | nova | enabled | up | 2021-02-05T06:19:26.000000 | - | - | up |
| cinder-volume | node88@lvm | nova | enabled | up | 2021-02-05T06:19:26.000000 | - | - | up |
| cinder-volume | openstcontroller@lvm | nova | disabled | down | 2021-01-25T07:27:39.000000 | - | - | - |
+------------------+----------------------+------+----------+-------+----------------------------+---------+-----------------+---------------+
openstcontroller@openstcontroller:~$ openstack volume list
+--------------------------------------+----------------+-----------+------+------------------------------------+
| ID | Name | Status | Size | Attached to |
+--------------------------------------+----------------+-----------+------+------------------------------------+
| 89a5d0ed-990e-4976-aa21-03162761c849 | linstoriscsi8g | available | 8 | |
| 9d0e19dd-a909-4290-b783-0819b956c925 | volume3 | in-use | 10 | Attached to ubuntu on /dev/vdb |
| 5ac5f5d3-c77e-4dea-b70f-465e9bfb391b | volume2 | in-use | 2 | Attached to cirrorARM on /dev/vda |
| 23c6a114-47d5-4e0d-92ba-feb68c726852 | vollume1 | in-use | 35 | Attached to ubuntu on /dev/vda |
+--------------------------------------+----------------+-----------+------+------------------------------------+
Create volume successful！
Check status on cinder node(node88)
root@node88:~# linstor r l
╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ ResourceName ┊ Node ┊ Port ┊ Usage ┊ Conns ┊ State ┊ CreatedOn ┊
╞═════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ CV_89a5d0ed-990e-4976-aa21-03162761c849 ┊ linstora ┊ 7000 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2021-02-05 01:20:41 ┊
┊ CV_89a5d0ed-990e-4976-aa21-03162761c849 ┊ linstorb ┊ 7000 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2021-02-05 01:20:46 ┊
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
root@node88:~# drbdadm status
CV_89a5d0ed-990e-4976-aa21-03162761c849 role:Secondary
disk:Diskless
node43 role:Secondary
peer-disk:UpToDate
ubuntu role:Secondary
peer-disk:UpToDate
Attach volume to virtual machine failed
openstcontroller@openstcontroller:~$ openstack server add volume ubuntu linstoriscsi8g
openstcontrolle@openstcontroller:~$
openstcontroller@openstcontroller:~$ openstack volume list
+--------------------------------------+----------------+-----------+------+------------------------------------+
| ID | Name | Status | Size | Attached to |
+--------------------------------------+----------------+-----------+------+------------------------------------+
| 89a5d0ed-990e-4976-aa21-03162761c849 | linstoriscsi8g | available | 8 | |
| 9d0e19dd-a909-4290-b783-0819b956c925 | volume3 | in-use | 10 | Attached to ubuntu on /dev/vdb |
| 5ac5f5d3-c77e-4dea-b70f-465e9bfb391b | volume2 | in-use | 2 | Attached to cirrorARM on /dev/vda |
| 23c6a114-47d5-4e0d-92ba-feb68c726852 | vollume1 | in-use | 35 | Attached to ubuntu on /dev/vda |
+--------------------------------------+----------------+-----------+------+------------------------------------+
Volume not in use，attach failed！
Cinder log:
root@node88:~# cat /var/log/cinder/cinder-volume.log
2021-02-05 01:20:38.457 35844 INFO cinder.volume.flows.manager.create_volume [req-fee6b110-8e6e-499d-9b93-ada17413ba39 2a19dea7fcb848eba9d25af19fa038e0 f27b37f7b0954575828bf6157ab11ce6 - default default] Volume 89a5d0ed-990e-4976-aa21-03162761c849: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-89a5d0ed-990e-4976-aa21-03162761c849', 'volume_size': 8}
2021-02-05 01:20:47.028 35844 INFO cinder.volume.flows.manager.create_volume [req-fee6b110-8e6e-499d-9b93-ada17413ba39 2a19dea7fcb848eba9d25af19fa038e0 f27b37f7b0954575828bf6157ab11ce6 - default default] Volume volume-89a5d0ed-990e-4976-aa21-03162761c849 (89a5d0ed-990e-4976-aa21-03162761c849): created successfully
2021-02-05 01:20:47.041 35844 INFO cinder.volume.manager [req-fee6b110-8e6e-499d-9b93-ada17413ba39 2a19dea7fcb848eba9d25af19fa038e0 f27b37f7b0954575828bf6157ab11ce6 - default default] Created volume successfully.
2021-02-05 01:23:09.817 35844 WARNING cinder.volume.targets.tgt [req-43f970fe-c691-4303-829b-33050cc189b0 2a19dea7fcb848eba9d25af19fa038e0 f27b37f7b0954575828bf6157ab11ce6 - default default] Attempting recreate of backing lun...
2021-02-05 01:23:19.837 35844 ERROR cinder.volume.targets.tgt [req-43f970fe-c691-4303-829b-33050cc189b0 2a19dea7fcb848eba9d25af19fa038e0 f27b37f7b0954575828bf6157ab11ce6 - default default] Failed recovery attempt to create iscsi backing lun for Volume ID:iqn.2010-10.org.openstack:volume-89a5d0ed-990e-4976-aa21-03162761c849: Unexpected error while running command.
Command: tgtadm --lld iscsi --op new --mode logicalunit --tid 7 --lun 1 -b None
Exit code: 22
Stdout: ''
Stderr: 'tgtadm: invalid request\n': oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
2021-02-05 01:23:19.928 35844 ERROR cinder.volume.manager [req-43f970fe-c691-4303-829b-33050cc189b0 2a19dea7fcb848eba9d25af19fa038e0 f27b37f7b0954575828bf6157ab11ce6 - default default] Create export for volume failed (Failed to create iscsi target for volume volume-89a5d0ed-990e-4976-aa21-03162761c849.).: cinder.exception.ISCSITargetCreateFailed: Failed to create iscsi target for volume volume-89a5d0ed-990e-4976-aa21-03162761c849.
2021-02-05 01:23:19.928 35844 ERROR cinder.volume.manager Traceback (most recent call last):
2021-02-05 01:23:19.928 35844 ERROR cinder.volume.manager File "/usr/lib/python3/dist-packages/cinder/volume/manager.py", line 4445, in _connection_create
2021-02-05 01:23:19.928 35844 ERROR cinder.volume.manager volume, connector)
2021-02-05 01:23:19.928 35844 ERROR cinder.volume.manager File "/usr/lib/python3/dist-packages/cinder/volume/drivers/linstordrv.py", line 1051, in create_export
2021-02-05 01:23:19.928 35844 ERROR cinder.volume.manager volume_path)
2021-02-05 01:23:19.928 35844 ERROR cinder.volume.manager File "/usr/lib/python3/dist-packages/cinder/volume/targets/iscsi.py", line 210, in create_export
2021-02-05 01:23:19.928 35844 ERROR cinder.volume.manager **portals_config)
2021-02-05 01:23:19.928 35844 ERROR cinder.volume.manager File "/usr/lib/python3/dist-packages/cinder/utils.py", line 818, in _wrapper
2021-02-05 01:23:19.928 35844 ERROR cinder.volume.manager File "/usr/lib/python3/dist-packages/cinder/utils.py", line 818, in _wrapper
2021-02-05 01:23:19.928 35844 ERROR cinder.volume.manager return r.call(f, *args, **kwargs)
2021-02-05 01:23:19.928 35844 ERROR cinder.volume.manager File "/usr/lib/python3/dist-packages/retrying.py", line 206, in call
2021-02-05 01:23:19.928 35844 ERROR cinder.volume.manager return attempt.get(self._wrap_exception)
2021-02-05 01:23:19.928 35844 ERROR cinder.volume.manager File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get
2021-02-05 01:23:19.928 35844 ERROR cinder.volume.manager six.reraise(self.value[0], self.value[1], self.value[2])
2021-02-05 01:23:19.928 35844 ERROR cinder.volume.manager File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise
2021-02-05 01:23:19.928 35844 ERROR cinder.volume.manager raise value
2021-02-05 01:23:19.928 35844 ERROR cinder.volume.manager File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call
2021-02-05 01:23:19.928 35844 ERROR cinder.volume.manager attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
2021-02-05 01:23:19.928 35844 ERROR cinder.volume.manager File "/usr/lib/python3/dist-packages/cinder/volume/targets/tgt.py", line 224, in create_iscsi_target
2021-02-05 01:23:19.928 35844 ERROR cinder.volume.manager raise exception.ISCSITargetCreateFailed(volume_id=vol_id)
2021-02-05 01:23:19.928 35844 ERROR cinder.volume.manager cinder.exception.ISCSITargetCreateFailed: Failed to create iscsi target for volume volume-89a5d0ed-990e-4976-aa21-03162761c849.
2021-02-05 01:23:19.928 35844 ERROR cinder.volume.manager
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server [req-43f970fe-c691-4303-829b-33050cc189b0 2a19dea7fcb848eba9d25af19fa038e0 f27b37f7b0954575828bf6157ab11ce6 - default default] Exception during message handling: cinder.exception.VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Create export for volume failed (Failed to create iscsi target for volume volume-89a5d0ed-990e-4976-aa21-03162761c849.).
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/volume/manager.py", line 4445, in _connection_create
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server volume, connector)
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/volume/drivers/linstordrv.py", line 1051, in create_export
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server volume_path)
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/volume/targets/iscsi.py", line 210, in create_export
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server **portals_config)
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/utils.py", line 818, in _wrapper
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server return r.call(f, *args, **kwargs)
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/retrying.py", line 206, in call
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server return attempt.get(self._wrap_exception)
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server six.reraise(self.value[0], self.value[1], self.value[2])
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server raise value
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/volume/targets/tgt.py", line 224, in create_iscsi_target
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server raise exception.ISCSITargetCreateFailed(volume_id=vol_id)
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server cinder.exception.ISCSITargetCreateFailed: Failed to create iscsi target for volume volume-89a5d0ed-990e-4976-aa21-03162761c849.
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/volume/manager.py", line 4517, in attachment_update
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server connector)
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server connector)
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/volume/manager.py", line 4449, in _connection_create
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server raise exception.VolumeBackendAPIException(data=err_msg)
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server cinder.exception.VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Create export for volume failed (Failed to create iscsi target for volume volume-89a5d0ed-990e-4976-aa21-03162761c849.).
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server
root@node88:~#
if anyone need more information,please let me know.
Architecture diagram on my environment