Openstack version: wallaby
Cinder storage backend: ceph
keymanager for nova and cinder: barbican
Reproduce step:
1. Create the volume type LUKS
```
openstack volume type create --encryption-provider luks \
--encryption-cipher aes-xts-plain64 --encryption-key-size 256 --encryption-control-location front-end LUKS
```
2. Create unencrypted volume
```
openstack volume create --size 1 'unencrypted volume' -c id -f value
```
3. Retype
```
cinder retype --migration-policy on-demand cca3582f-c8b4-4175-bb44-ae4bf49be566 LUKS
```
The output error log of c-vol is;
```
2021-10-29 19:29:16.436 233 INFO oslo.privsep.daemon [-] privsep daemon starting
2021-10-29 19:29:16.443 233 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0
2021-10-29 19:29:16.446 233 INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_SYS_ADMIN/CAP_SYS_ADMIN/none
2021-10-29 19:29:16.447 233 INFO oslo.privsep.daemon [-] privsep daemon running as pid 233
2021-10-29 19:29:16.670 9 WARNING os_brick.initiator.connectors.iscsi [req-b8e226cf-cf35-4969-bae0-de3a2deda8fd e08aeb12768b4021a04002b6a1cbf3c5 00cc2ee8b0d24b4799b2319d8cc315da - - -] Could not find the iSCSI Initiator File /etc/iscsi/initiatorname.iscsi: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
2021-10-29 19:29:16.687 9 WARNING os_brick.initiator.connectors.nvmeof [req-b8e226cf-cf35-4969-bae0-de3a2deda8fd e08aeb12768b4021a04002b6a1cbf3c5 00cc2ee8b0d24b4799b2319d8cc315da - - -] Process execution error in _get_host_uuid: Unexpected error while running command.
Command: blkid overlay -s UUID -o value
Exit code: 2
Stdout: ''
Stderr: '': oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
2021-10-29 19:29:16.699 9 WARNING os_brick.initiator.connectors.nvmeof [req-b8e226cf-cf35-4969-bae0-de3a2deda8fd e08aeb12768b4021a04002b6a1cbf3c5 00cc2ee8b0d24b4799b2319d8cc315da - - -] Could not generate host nqn: Unexpected error while running command.
Command: mkdir -m 755 -p /etc/nvme
Exit code: 1
Stdout: ''
Stderr: 'mkdir: cannot create directory ‘/etc/nvme’: Read-only file system\n': oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
2021-10-29 19:29:17.247 9 INFO cinder.volume.manager [req-b8e226cf-cf35-4969-bae0-de3a2deda8fd e08aeb12768b4021a04002b6a1cbf3c5 00cc2ee8b0d24b4799b2319d8cc315da - - -] Initialize volume connection completed successfully.
2021-10-29 19:29:17.319 9 ERROR os_brick.encryptors [req-b8e226cf-cf35-4969-bae0-de3a2deda8fd e08aeb12768b4021a04002b6a1cbf3c5 00cc2ee8b0d24b4799b2319d8cc315da - - -] Error instantiating os_brick.encryptors.luks.LuksEncryptor: expected str, bytes or os.PathLike object, not RBDVolumeIOWrapper
2021-10-29 19:29:17.319 9 ERROR cinder.volume.manager [req-b8e226cf-cf35-4969-bae0-de3a2deda8fd e08aeb12768b4021a04002b6a1cbf3c5 00cc2ee8b0d24b4799b2319d8cc315da - - -] Failed to attach volume encryptor 92ec3065-ed0a-4742-8f1b-1d30cd416684.
2021-10-29 19:29:17.346 9 INFO cinder.volume.manager [req-b8e226cf-cf35-4969-bae0-de3a2deda8fd e08aeb12768b4021a04002b6a1cbf3c5 00cc2ee8b0d24b4799b2319d8cc315da - - -] Terminate volume connection completed successfully.
2021-10-29 19:29:17.377 9 ERROR cinder.volume.manager [req-b8e226cf-cf35-4969-bae0-de3a2deda8fd e08aeb12768b4021a04002b6a1cbf3c5 00cc2ee8b0d24b4799b2319d8cc315da - - -] Failed to copy volume cca3582f-c8b4-4175-bb44-ae4bf49be566 to 92ec3065-ed0a-4742-8f1b-1d30cd416684: TypeError: expected str, bytes or os.PathLike object, not RBDVolumeIOWrapper
2021-10-29 19:29:17.377 9 ERROR cinder.volume.manager Traceback (most recent call last):
2021-10-29 19:29:17.377 9 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.8/site-packages/cinder/volume/manager.py", line 2344, in _migrate_volume_generic
2021-10-29 19:29:17.377 9 ERROR cinder.volume.manager self._copy_volume_data(ctxt, volume, new_volume, remote='dest')
2021-10-29 19:29:17.377 9 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.8/site-packages/cinder/volume/manager.py", line 2217, in _copy_volume_data
2021-10-29 19:29:17.377 9 ERROR cinder.volume.manager dest_attach_info = self._attach_volume(
2021-10-29 19:29:17.377 9 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.8/site-packages/cinder/volume/manager.py", line 2161, in _attach_volume
2021-10-29 19:29:17.377 9 ERROR cinder.volume.manager self._detach_volume(ctxt, attach_info, volume, properties,
2021-10-29 19:29:17.377 9 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2021-10-29 19:29:17.377 9 ERROR cinder.volume.manager self.force_reraise()
2021-10-29 19:29:17.377 9 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2021-10-29 19:29:17.377 9 ERROR cinder.volume.manager raise self.value
2021-10-29 19:29:17.377 9 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.8/site-packages/cinder/volume/manager.py", line 2154, in _attach_volume
2021-10-29 19:29:17.377 9 ERROR cinder.volume.manager volume_utils.brick_attach_volume_encryptor(ctxt,
2021-10-29 19:29:17.377 9 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.8/site-packages/cinder/volume/volume_utils.py", line 1360, in brick_attach_volume_encryptor
2021-10-29 19:29:17.377 9 ERROR cinder.volume.manager encryptor = brick_get_encryptor(connection_info,
2021-10-29 19:29:17.377 9 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.8/site-packages/cinder/volume/volume_utils.py", line 1350, in brick_get_encryptor
2021-10-29 19:29:17.377 9 ERROR cinder.volume.manager return encryptors.get_volume_encryptor(root_helper=root_helper,
2021-10-29 19:29:17.377 9 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.8/site-packages/os_brick/encryptors/__init__.py", line 88, in get_volume_encryptor
2021-10-29 19:29:17.377 9 ERROR cinder.volume.manager encryptor = importutils.import_object(
2021-10-29 19:29:17.377 9 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.8/site-packages/oslo_utils/importutils.py", line 44, in import_object
2021-10-29 19:29:17.377 9 ERROR cinder.volume.manager return import_class(import_str)(*args, **kwargs)
2021-10-29 19:29:17.377 9 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.8/site-packages/os_brick/encryptors/luks.py", line 57, in __init__
2021-10-29 19:29:17.377 9 ERROR cinder.volume.manager super(LuksEncryptor, self).__init__(
2021-10-29 19:29:17.377 9 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.8/site-packages/os_brick/encryptors/cryptsetup.py", line 62, in __init__
2021-10-29 19:29:17.377 9 ERROR cinder.volume.manager self.dev_name = 'crypt-%s' % os.path.basename(self.symlink_path)
2021-10-29 19:29:17.377 9 ERROR cinder.volume.manager File "/usr/lib/python3.8/posixpath.py", line 142, in basename
2021-10-29 19:29:17.377 9 ERROR cinder.volume.manager p = os.fspath(p)
2021-10-29 19:29:17.377 9 ERROR cinder.volume.manager TypeError: expected str, bytes or os.PathLike object, not RBDVolumeIOWrapper
2021-10-29 19:29:17.377 9 ERROR cinder.volume.manager
2021-10-29 19:29:17.415 9 INFO cinder.volume.manager [req-b8e226cf-cf35-4969-bae0-de3a2deda8fd e08aeb12768b4021a04002b6a1cbf3c5 00cc2ee8b0d24b4799b2319d8cc315da - - -] Remove volume export completed successfully.
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server [req-b8e226cf-cf35-4969-bae0-de3a2deda8fd e08aeb12768b4021a04002b6a1cbf3c5 00cc2ee8b0d24b4799b2319d8cc315da - - -] Exception during message handling: TypeError: expected str, bytes or os.PathLike object, not RBDVolumeIOWrapper
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server File "/var/lib/openstack/lib/python3.8/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server File "/var/lib/openstack/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server File "/var/lib/openstack/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server File "/var/lib/openstack/lib/python3.8/site-packages/cinder/volume/manager.py", line 3040, in retype
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server _retype_error(context, volume, old_reservations,
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server File "/var/lib/openstack/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server self.force_reraise()
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server File "/var/lib/openstack/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server raise self.value
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server File "/var/lib/openstack/lib/python3.8/site-packages/cinder/volume/manager.py", line 3036, in retype
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server self.migrate_volume(context, volume, host,
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server File "/var/lib/openstack/lib/python3.8/site-packages/cinder/volume/manager.py", line 2619, in migrate_volume
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server volume.save()
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server File "/var/lib/openstack/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server self.force_reraise()
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server File "/var/lib/openstack/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server raise self.value
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server File "/var/lib/openstack/lib/python3.8/site-packages/cinder/volume/manager.py", line 2612, in migrate_volume
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server self._migrate_volume_generic(ctxt, volume, host, new_type_id)
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server File "/var/lib/openstack/lib/python3.8/site-packages/cinder/volume/manager.py", line 2365, in _migrate_volume_generic
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server self._clean_temporary_volume(ctxt, volume,
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server File "/var/lib/openstack/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server self.force_reraise()
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server File "/var/lib/openstack/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server raise self.value
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server File "/var/lib/openstack/lib/python3.8/site-packages/cinder/volume/manager.py", line 2344, in _migrate_volume_generic
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server self._copy_volume_data(ctxt, volume, new_volume, remote='dest')
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server File "/var/lib/openstack/lib/python3.8/site-packages/cinder/volume/manager.py", line 2217, in _copy_volume_data
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server dest_attach_info = self._attach_volume(
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server File "/var/lib/openstack/lib/python3.8/site-packages/cinder/volume/manager.py", line 2161, in _attach_volume
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server self._detach_volume(ctxt, attach_info, volume, properties,
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server File "/var/lib/openstack/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server self.force_reraise()
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server File "/var/lib/openstack/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server raise self.value
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server File "/var/lib/openstack/lib/python3.8/site-packages/cinder/volume/manager.py", line 2154, in _attach_volume
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server volume_utils.brick_attach_volume_encryptor(ctxt,
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server File "/var/lib/openstack/lib/python3.8/site-packages/cinder/volume/volume_utils.py", line 1360, in brick_attach_volume_encryptor
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server encryptor = brick_get_encryptor(connection_info,
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server File "/var/lib/openstack/lib/python3.8/site-packages/cinder/volume/volume_utils.py", line 1350, in brick_get_encryptor
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server return encryptors.get_volume_encryptor(root_helper=root_helper,
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server File "/var/lib/openstack/lib/python3.8/site-packages/os_brick/encryptors/__init__.py", line 88, in get_volume_encryptor
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server encryptor = importutils.import_object(
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server File "/var/lib/openstack/lib/python3.8/site-packages/oslo_utils/importutils.py", line 44, in import_object
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server return import_class(import_str)(*args, **kwargs)
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server File "/var/lib/openstack/lib/python3.8/site-packages/os_brick/encryptors/luks.py", line 57, in __init__
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server super(LuksEncryptor, self).__init__(
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server File "/var/lib/openstack/lib/python3.8/site-packages/os_brick/encryptors/cryptsetup.py", line 62, in __init__
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server self.dev_name = 'crypt-%s' % os.path.basename(self.symlink_path)
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.8/posixpath.py", line 142, in basename
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server p = os.fspath(p)
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server TypeError: expected str, bytes or os.PathLike object, not RBDVolumeIOWrapper
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server
2021-10-29 19:29:17.642 9 INFO cinder.volume.manager [req-b8e226cf-cf35-4969-bae0-de3a2deda8fd e08aeb12768b4021a04002b6a1cbf3c5 00cc2ee8b0d24b4799b2319d8cc315da - - -] Deleted volume successfully.
```
So I think the main error is the following; rpc.server TypeError: expected str, bytes or os.PathLike object, not RBDVolumeIOWrapper /opendev. org/openstack/ os-brick/ src/branch/ stable/ wallaby/ os_brick/ initiator/ connectors/ rbd.py# L230-L245
```
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.
```
https:/
All `os_brick. initiator. connectors` have `connect_volume` and `disconnect_volume` methods which returns `device_path`. While retyping, the `device_path` is used to attach the volume. But as you can see the above link, `connect_volume` of rbd connector returns `linuxrbd. RBDVolumeIOWrap per` and returns str when only `do_local_attach` is set true.
But do_local_attach is set false.