| 2022-11-30 18:05:06 |
Alexander Deiter |
description |
Description of problem: [NFS] Creating an encrypted volume from an image overwrites the volume file with a broken symlink
Steps to Reproduce:
* Configure Devstack environment with generic NFS driver and barbican
[nfs]
volume_driver = cinder.volume.drivers.nfs.NfsDriver
volume_backend_name = nfs
nas_host = 192.168.153.222
nas_share_path = /tank2/nfs
* Create a volume type with enabled LUKS
$ openstack volume type create \
--property volume_backend_name=nfs \
--encryption-provider luks \
--encryption-cipher aes-xts-plain64 \
--encryption-key-size 256 \
--encryption-control-location front-end test-type
$ cinder extra-specs-list
+--------------------------------------+-------------+-------------------------------------+
| ID | Name | extra_specs |
+--------------------------------------+-------------+-------------------------------------+
| 524a6e8d-9457-43e8-9117-9247b5cdbf03 | luks | {'volume_backend_name': 'nfs'} |
+--------------------------------------+-------------+-------------------------------------+
$ cinder encryption-type-list
+--------------------------------------+----------+-----------------+----------+------------------+
| Volume Type ID | Provider | Cipher | Key Size | Control Location |
+--------------------------------------+----------+-----------------+----------+------------------+
| 524a6e8d-9457-43e8-9117-9247b5cdbf03 | luks | aes-xts-plain64 | 256 | front-end |
+--------------------------------------+----------+-----------------+----------+------------------+
* Try to create an encrypted volume from an image
$ glance image-list
+--------------------------------------+--------------------------+
| ID | Name |
+--------------------------------------+--------------------------+
| 60c97eaa-2064-4ade-8ee1-b3550df24fa3 | cirros-0.5.2-x86_64-disk |
+--------------------------------------+--------------------------+
$ openstack volume create --image cirros-0.5.2-x86_64-disk --type test-type --size 1 test-volume
Expected results
Success
Actual results
No volume file (Broken symlink for the volume file) and error:
$ mount -t nfs4
192.168.153.222:/tank2/nfs on /opt/stack/data/cinder/mnt/5f80077cb99fe69d9946fce44709af16 type nfs4 (rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.153.78,local_lock=none,addr=192.168.153.222)
$ ls -l /opt/stack/data/cinder/mnt/5f80077cb99fe69d9946fce44709af16/
total 1
lrwxrwxrwx 1 root root 61 Nov 30 17:13 volume-8d218ca3-4542-4140-b052-c525a00c203b -> /dev/mapper/crypt-volume-8d218ca3-4542-4140-b052-c525a00c203b
$ stat /opt/stack/data/cinder/mnt/5f80077cb99fe69d9946fce44709af16/volume-8d218ca3-4542-4140-b052-c525a00c203b
File: /opt/stack/data/cinder/mnt/5f80077cb99fe69d9946fce44709af16/volume-8d218ca3-4542-4140-b052-c525a00c203b -> /dev/mapper/crypt-volume-8d218ca3-4542-4140-b052-c525a00c203b
Size: 61 Blocks: 1 IO Block: 1048576 symbolic link
Device: 3ah/58d Inode: 21 Links: 1
Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2022-11-30 17:13:55.262969050 +0000
Modify: 2022-11-30 17:13:55.262969050 +0000
Change: 2022-11-30 17:13:55.266257870 +0000
Birth: -
$ ls -l /dev/mapper/crypt-volume-8d218ca3-4542-4140-b052-c525a00c203b
ls: cannot access '/dev/mapper/crypt-volume-8d218ca3-4542-4140-b052-c525a00c203b': No such file or directory
$ ls -l /dev/mapper
total 0
crw------- 1 root root 10, 236 Nov 29 20:33 control
$ cinder list
+--------------------------------------+--------+-------------+------+-------------+----------+-------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+--------+-------------+------+-------------+----------+-------------+
| 8d218ca3-4542-4140-b052-c525a00c203b | error | test-volume | 1 | test-type | false | |
+--------------------------------------+--------+-------------+------+-------------+----------+-------------+
cinder-volume log file:
DEBUG oslo_concurrency.processutils [None req-f1c87143-0ec3-4474-b525-610baa3d07a7 admin None] CMD "qemu-img create -f qcow2 -o encrypt.format=luks,encrypt.key-secret=*** --object secret,id=sec1,format=raw,file=/opt/stack/data/cinder/conversion/tmphmolpcwm /opt/stack/data/cinder/mnt/5f80077cb99fe69d9946fce44709af16/volume-8d218ca3-4542-4140-b052-c525a00c203b 1073741824" returned: 0 in 4.911s {{execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
...
DEBUG cinder.volume.volum
e_utils [None req-f1c87143-0ec3-4474-b525-610baa3d07a7 admin None] Attempting download of
60c97eaa-2064-4ade-8ee1-b3550df24fa3 ((None, None)) to volume 8d218ca3-4542-4140-b052-c5
25a00c203b. {{copy_image_to_volume /opt/stack/cinder/cinder/volume/volume_utils.py:1190}}
...
DEBUG oslo_concurrency.processutils [None req-f1c87143-0ec3-4474-b525-610baa3d07a7 admin None] Running cmd (subprocess): /usr/bin/python3.8 -m oslo_concurrency.prlimit --as=1073741824 --cpu=60 -- sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info --output=json --force-share /opt/stack/data/cinder/mnt/5f80077cb99fe69d9946fce44709af16/volume-8d218ca3-4542-4140-b052-c525a00c203b {{execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
...
DEBUG cinder.volume.drivers.nfs [None req-f1c87143-0ec3-4474-b525-610baa3d07a7 admin None] NfsDriver: conn_info: {'driver_volume_type': 'nfs', 'data': {'export': '192.168.153.222:/tank2/nfs', 'name': 'volume-8d218ca3-4542-4140-b052-c525a00c203b', 'options': None, 'format': 'qcow2'}, 'mount_point_base': '/opt/stack/data/cinder/mnt'} {{initialize_connection /opt/stack/cinder/cinder/volume/drivers/nfs.py:164}}
...
DEBUG os_brick.initiator.connectors.remotefs [None req-f1c87143-0ec3-4474-b525-610baa3d07a7 admin None] ==> connect_volume: call "{'self': <os_brick.initiator.connectors.remotefs.RemoteFsConnector object at 0x7ff74e51d640>, 'connection_properties': {'export': '192.168.153.222:/tank2/nfs', 'name': 'volume-8d218ca3-4542-4140-b052-c525a00c203b', 'options': None, 'format': 'qcow2', 'encrypted': True}}" {{trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:174}}
...
DEBUG os_brick.initiator.connectors.remotefs [None req-f1c87143-0ec3-4474-b525-610baa3d07a7 admin None] <== connect_volume: return (0ms) {'path': '/opt/stack/data/cinder/mnt/5f80077cb99fe69d9946fce44709af16/volume-8d218ca3-4542-4140-b052-c525a00c203b'} {{trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:201}}
...
DEBUG os_brick.encryptors [None req-f1c87143-0ec3-4474-b525-610baa3d07a7 admin None] Using volume encryptor '<os_brick.encryptors.luks.LuksEncryptor object at 0x7ff74e6228e0>' for connection: {'driver_volume_type': 'nfs', 'data': {'export': '192.168.153.222:/tank2/nfs', 'name': 'volume-8d218ca3-4542-4140-b052-c525a00c203b', 'options': None, 'format': 'qcow2', 'encrypted': True, 'device_path': '/opt/stack/data/cinder/mnt/5f80077cb99fe69d9946fce44709af16/volume-8d218ca3-4542-4140-b052-c525a00c203b'}, 'mount_point_base': '/opt/stack/data/cinder/mnt'} {{get_volume_encryptor /usr/local/lib/python3.8/dist-packages/os_brick/encryptors/__init__.py:107}}
...
DEBUG os_brick.encryptors.luks [None req-f1c87143-0ec3-4474-b525-610baa3d07a7 admin None] opening encrypted volume /opt/stack/data/cinder/mnt/5f80077cb99fe69d9946fce44709af16/volume-8d218ca3-4542-4140-b052-c525a00c203b {{_open_volume /usr/local/lib/python3.8/dist-packages/os_brick/encryptors/luks.py:111}}
...
INFO os_brick.encryptors.luks [None req-f1c87143-0ec3-4474-b525-610baa3d07a7 admin None] /opt/stack/data/cinder/mnt/5f80077cb99fe69d9946fce44709af16/volume-8d218ca3-4542-4140-b052-c525a00c203b is not a valid LUKS device; formatting device for first use
...
DEBUG os_brick.encryptors.luks [None req-f1c87143-0ec3-4474-b525-610baa3d07a7 admin None] formatting encrypted volume /opt/stack/data/cinder/mnt/5f80077cb99fe69d9946fce44709af16/volume-8d218ca3-4542-4140-b052-c525a00c203b {{_format_luks_volume /usr/local/lib/python3.8/dist-packages/os_brick/encryptors/luks.py:84}}
...
DEBUG oslo_concurrency.processutils [-] CMD "cryptsetup luksOpen --key-file=- /opt/stack/data/cinder/mnt/5f80077cb99fe69d9946fce44709af16/volume-8d218ca3-4542-4140-b052-c525a00c203b crypt-volume-8d218ca3-4542-4140-b052-c525a00c203b" returned: 0 in 2.231s {{execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
...
DEBUG oslo_concurrency.processutils [-] CMD "ln --symbolic --force /dev/mapper/crypt-volume-8d218ca3-4542-4140-b052-c525a00c203b /opt/stack/data/cinder/mnt/5f80077cb99fe69d9946fce44709af16/volume-8d218ca3-4542-4140-b052-c525a00c203b" returned: 0 in 0.011s {{execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
...
DEBUG oslo_concurrency.processutils [-] CMD "cryptsetup luksClose crypt-volume-8d218ca3-4542-4140-b052-c525a00c203b" returned: 0 in 0.066s {{execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
...
DEBUG cinder.volume.driver [None req-f1c87143-0ec3-4474-b525-610baa3d07a7 admin None] volume 8d218ca3-4542-4140-b052-c525a00c203b: removing export {{_detach_volume /opt/stack/cinder/cinder/volume/driver.py:543}}
...
Nov 30 17:13:56 openstack-master-ns5-nfs cinder-volume[176211]: ERROR cinder.volume.volume_utils [None req-f1c87143-0ec3-4474-b525-610baa3d07a7 admin None] Failed to copy image 60c97eaa-2064-4ade-8ee1-b3550df24fa3 to volume: 8d218ca3-4542-4140-b052-c525a00c203b: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf qemu-img convert -O raw -t none -f qcow2 /opt/stack/data/cinder/conversion/image_fetch_60c97eaa-2064-4ade-8ee1-b3550df24fa3_n8610k83openstack-master-ns5-nfs@nfs /opt/stack/data/cinder/mnt/5f80077cb99fe69d9946fce44709af16/volume-8d218ca3-4542-4140-b052-c525a00c203b
Exit code: 1
Stdout: ''
Stderr: 'qemu-img: /opt/stack/data/cinder/mnt/5f80077cb99fe69d9946fce44709af16/volume-8d218ca3-4542-4140-b052-c525a00c203b: error while converting raw: Device is too small\n' |
Description of problem: [NFS] Creating an encrypted volume from an image overwrites the volume file with a broken symlink
Steps to Reproduce:
* Configure Devstack environment with generic NFS driver and barbican
[nfs]
volume_driver = cinder.volume.drivers.nfs.NfsDriver
volume_backend_name = nfs
nas_host = 192.168.153.222
nas_share_path = /tank2/nfs
* Create a volume type with enabled LUKS
$ openstack volume type create \
--property volume_backend_name=nfs \
--encryption-provider luks \
--encryption-cipher aes-xts-plain64 \
--encryption-key-size 256 \
--encryption-control-location front-end test-type
$ cinder extra-specs-list
+--------------------------------------+-------------+-------------------------------------+
| ID | Name | extra_specs |
+--------------------------------------+-------------+-------------------------------------+
| 524a6e8d-9457-43e8-9117-9247b5cdbf03 | luks | {'volume_backend_name': 'nfs'} |
+--------------------------------------+-------------+-------------------------------------+
$ cinder encryption-type-list
+--------------------------------------+----------+-----------------+----------+------------------+
| Volume Type ID | Provider | Cipher | Key Size | Control Location |
+--------------------------------------+----------+-----------------+----------+------------------+
| 524a6e8d-9457-43e8-9117-9247b5cdbf03 | luks | aes-xts-plain64 | 256 | front-end |
+--------------------------------------+----------+-----------------+----------+------------------+
* Try to create an encrypted volume from an image
$ glance image-list
+--------------------------------------+--------------------------+
| ID | Name |
+--------------------------------------+--------------------------+
| 60c97eaa-2064-4ade-8ee1-b3550df24fa3 | cirros-0.5.2-x86_64-disk |
+--------------------------------------+--------------------------+
$ openstack volume create --image cirros-0.5.2-x86_64-disk --type test-type --size 1 test-volume
Expected results
Success
Actual results
No volume file (Broken symlink for the volume file) and error:
$ mount -t nfs4
192.168.153.222:/tank2/nfs on /opt/stack/data/cinder/mnt/5f80077cb99fe69d9946fce44709af16 type nfs4 (rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.153.78,local_lock=none,addr=192.168.153.222)
$ ls -l /opt/stack/data/cinder/mnt/5f80077cb99fe69d9946fce44709af16/
total 1
lrwxrwxrwx 1 root root 61 Nov 30 17:13 volume-8d218ca3-4542-4140-b052-c525a00c203b -> /dev/mapper/crypt-volume-8d218ca3-4542-4140-b052-c525a00c203b
$ stat /opt/stack/data/cinder/mnt/5f80077cb99fe69d9946fce44709af16/volume-8d218ca3-4542-4140-b052-c525a00c203b
File: /opt/stack/data/cinder/mnt/5f80077cb99fe69d9946fce44709af16/volume-8d218ca3-4542-4140-b052-c525a00c203b -> /dev/mapper/crypt-volume-8d218ca3-4542-4140-b052-c525a00c203b
Size: 61 Blocks: 1 IO Block: 1048576 symbolic link
Device: 3ah/58d Inode: 21 Links: 1
Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2022-11-30 17:13:55.262969050 +0000
Modify: 2022-11-30 17:13:55.262969050 +0000
Change: 2022-11-30 17:13:55.266257870 +0000
Birth: -
$ ls -l /dev/mapper/crypt-volume-8d218ca3-4542-4140-b052-c525a00c203b
ls: cannot access '/dev/mapper/crypt-volume-8d218ca3-4542-4140-b052-c525a00c203b': No such file or directory
$ ls -l /dev/mapper
total 0
crw------- 1 root root 10, 236 Nov 29 20:33 control
$ cinder list
+--------------------------------------+--------+-------------+------+-------------+----------+-------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+--------+-------------+------+-------------+----------+-------------+
| 8d218ca3-4542-4140-b052-c525a00c203b | error | test-volume | 1 | test-type | false | |
+--------------------------------------+--------+-------------+------+-------------+----------+-------------+
cinder-volume log file:
DEBUG oslo_concurrency.processutils [None req-f1c87143-0ec3-4474-b525-610baa3d07a7 admin None] CMD "qemu-img create -f qcow2 -o encrypt.format=luks,encrypt.key-secret=*** --object secret,id=sec1,format=raw,file=/opt/stack/data/cinder/conversion/tmphmolpcwm /opt/stack/data/cinder/mnt/5f80077cb99fe69d9946fce44709af16/volume-8d218ca3-4542-4140-b052-c525a00c203b 1073741824" returned: 0 in 4.911s {{execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
...
DEBUG cinder.volume.volum
e_utils [None req-f1c87143-0ec3-4474-b525-610baa3d07a7 admin None] Attempting download of
60c97eaa-2064-4ade-8ee1-b3550df24fa3 ((None, None)) to volume 8d218ca3-4542-4140-b052-c5
25a00c203b. {{copy_image_to_volume /opt/stack/cinder/cinder/volume/volume_utils.py:1190}}
...
DEBUG oslo_concurrency.processutils [None req-f1c87143-0ec3-4474-b525-610baa3d07a7 admin None] Running cmd (subprocess): /usr/bin/python3.8 -m oslo_concurrency.prlimit --as=1073741824 --cpu=60 -- sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info --output=json --force-share /opt/stack/data/cinder/mnt/5f80077cb99fe69d9946fce44709af16/volume-8d218ca3-4542-4140-b052-c525a00c203b {{execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
...
DEBUG cinder.volume.drivers.nfs [None req-f1c87143-0ec3-4474-b525-610baa3d07a7 admin None] NfsDriver: conn_info: {'driver_volume_type': 'nfs', 'data': {'export': '192.168.153.222:/tank2/nfs', 'name': 'volume-8d218ca3-4542-4140-b052-c525a00c203b', 'options': None, 'format': 'qcow2'}, 'mount_point_base': '/opt/stack/data/cinder/mnt'} {{initialize_connection /opt/stack/cinder/cinder/volume/drivers/nfs.py:164}}
...
DEBUG os_brick.initiator.connectors.remotefs [None req-f1c87143-0ec3-4474-b525-610baa3d07a7 admin None] ==> connect_volume: call "{'self': <os_brick.initiator.connectors.remotefs.RemoteFsConnector object at 0x7ff74e51d640>, 'connection_properties': {'export': '192.168.153.222:/tank2/nfs', 'name': 'volume-8d218ca3-4542-4140-b052-c525a00c203b', 'options': None, 'format': 'qcow2', 'encrypted': True}}" {{trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:174}}
...
DEBUG os_brick.initiator.connectors.remotefs [None req-f1c87143-0ec3-4474-b525-610baa3d07a7 admin None] <== connect_volume: return (0ms) {'path': '/opt/stack/data/cinder/mnt/5f80077cb99fe69d9946fce44709af16/volume-8d218ca3-4542-4140-b052-c525a00c203b'} {{trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:201}}
...
DEBUG os_brick.encryptors [None req-f1c87143-0ec3-4474-b525-610baa3d07a7 admin None] Using volume encryptor '<os_brick.encryptors.luks.LuksEncryptor object at 0x7ff74e6228e0>' for connection: {'driver_volume_type': 'nfs', 'data': {'export': '192.168.153.222:/tank2/nfs', 'name': 'volume-8d218ca3-4542-4140-b052-c525a00c203b', 'options': None, 'format': 'qcow2', 'encrypted': True, 'device_path': '/opt/stack/data/cinder/mnt/5f80077cb99fe69d9946fce44709af16/volume-8d218ca3-4542-4140-b052-c525a00c203b'}, 'mount_point_base': '/opt/stack/data/cinder/mnt'} {{get_volume_encryptor /usr/local/lib/python3.8/dist-packages/os_brick/encryptors/__init__.py:107}}
...
DEBUG os_brick.encryptors.luks [None req-f1c87143-0ec3-4474-b525-610baa3d07a7 admin None] opening encrypted volume /opt/stack/data/cinder/mnt/5f80077cb99fe69d9946fce44709af16/volume-8d218ca3-4542-4140-b052-c525a00c203b {{_open_volume /usr/local/lib/python3.8/dist-packages/os_brick/encryptors/luks.py:111}}
...
INFO os_brick.encryptors.luks [None req-f1c87143-0ec3-4474-b525-610baa3d07a7 admin None] /opt/stack/data/cinder/mnt/5f80077cb99fe69d9946fce44709af16/volume-8d218ca3-4542-4140-b052-c525a00c203b is not a valid LUKS device; formatting device for first use
...
DEBUG os_brick.encryptors.luks [None req-f1c87143-0ec3-4474-b525-610baa3d07a7 admin None] formatting encrypted volume /opt/stack/data/cinder/mnt/5f80077cb99fe69d9946fce44709af16/volume-8d218ca3-4542-4140-b052-c525a00c203b {{_format_luks_volume /usr/local/lib/python3.8/dist-packages/os_brick/encryptors/luks.py:84}}
...
DEBUG oslo_concurrency.processutils [-] CMD "cryptsetup luksOpen --key-file=- /opt/stack/data/cinder/mnt/5f80077cb99fe69d9946fce44709af16/volume-8d218ca3-4542-4140-b052-c525a00c203b crypt-volume-8d218ca3-4542-4140-b052-c525a00c203b" returned: 0 in 2.231s {{execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
...
DEBUG oslo_concurrency.processutils [-] CMD "ln --symbolic --force /dev/mapper/crypt-volume-8d218ca3-4542-4140-b052-c525a00c203b /opt/stack/data/cinder/mnt/5f80077cb99fe69d9946fce44709af16/volume-8d218ca3-4542-4140-b052-c525a00c203b" returned: 0 in 0.011s {{execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
...
DEBUG oslo_concurrency.processutils [-] CMD "cryptsetup luksClose crypt-volume-8d218ca3-4542-4140-b052-c525a00c203b" returned: 0 in 0.066s {{execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
...
DEBUG cinder.volume.driver [None req-f1c87143-0ec3-4474-b525-610baa3d07a7 admin None] volume 8d218ca3-4542-4140-b052-c525a00c203b: removing export {{_detach_volume /opt/stack/cinder/cinder/volume/driver.py:543}}
...
ERROR cinder.volume.volume_utils [None req-f1c87143-0ec3-4474-b525-610baa3d07a7 admin None] Failed to copy image 60c97eaa-2064-4ade-8ee1-b3550df24fa3 to volume: 8d218ca3-4542-4140-b052-c525a00c203b: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf qemu-img convert -O raw -t none -f qcow2 /opt/stack/data/cinder/conversion/image_fetch_60c97eaa-2064-4ade-8ee1-b3550df24fa3_n8610k83openstack-master-nfs@nfs /opt/stack/data/cinder/mnt/5f80077cb99fe69d9946fce44709af16/volume-8d218ca3-4542-4140-b052-c525a00c203b
Exit code: 1
Stdout: ''
Stderr: 'qemu-img: /opt/stack/data/cinder/mnt/5f80077cb99fe69d9946fce44709af16/volume-8d218ca3-4542-4140-b052-c525a00c203b: error while converting raw: Device is too small\n' |
|
