OpenStack Compute (nova)

Retype of non-encrypted root volume to encrypted fails

Bug #1925818 reported by Jan Wasilewski on 2021-04-23

This bug affects 2 people

	Status	Importance	Assigned to
Cinder	New	High	Unassigned
OpenStack Compute (nova)	Incomplete	Undecided	Unassigned
os-brick	New	High	Unassigned

Bug Description

Description
===========
I tried to retype volume from unencrypted to encrypted one. That works fine for non-root volumes, however it doesn't work when volume was/is root volume.

Steps to reproduce
==================
1. Create an instance with unencrypted root volume(select do not delete volume on termination)
2. Delete instance
3. Retype left volume from deleted instance to encrypted one and wait till retype will finish successfully
4. Launch an instance from this encrypted volume(increase original size by +2GB)

Expected result
===============
New instance is started from encrypted volume

Actual result
=============
Instance fails to start. At horizon we can see an error message:
Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 610, in build_instances filter_properties, instances[0].uuid) File "/usr/lib/python2.7/dist-packages/nova/scheduler/utils.py", line 680, in populate_retry raise exception.MaxRetriesExceeded(reason=msg) MaxRetriesExceeded: Exceeded maximum number of retries. Exceeded max scheduling attempts 3 for instance d19504b4-8c05-423b-abaa-b1b8c0658fa8. Last exception: Unexpected error while running command. Command: cryptsetup --batch-mode luksFormat --key-file=- --cipher aes-xts-plain64 --key-size 256 /dev/dm-19 Exit code: 5 Stdout: u'' Stderr: u'Cannot use device /dev/dm-19 which is in use (already mapped or mounted).\n'

Environment
===========
1. Version of cinder:
ii cinder-api 2:13.0.9-0ubuntu1~cloud1.1 all Cinder storage service - API server
ii cinder-backup 2:13.0.9-0ubuntu1~cloud1.1 all Cinder storage service - Scheduler server
ii cinder-common 2:13.0.9-0ubuntu1~cloud1.1 all Cinder storage service - common files
ii cinder-scheduler 2:13.0.9-0ubuntu1~cloud1.1 all Cinder storage service - Scheduler server
ii cinder-volume 2:13.0.9-0ubuntu1~cloud1.1 all Cinder storage service - Volume server
ii python-cinder 2:13.0.9-0ubuntu1~cloud1.1 all Cinder Python 2 libraries
ii python-cinderclient 1:3.5.0-0ubuntu1 all Python bindings to the OpenStack Volume API - Python 2.x

Version of barbican:

ii barbican-api 1:10.0.0-0ubuntu0.20.04.1~cloud0 all OpenStack Key Management Service - API Server
ii barbican-common 1:10.0.0-0ubuntu0.20.04.1~cloud0 all OpenStack Key Management Service - common files
ii barbican-keystone-listener 1:10.0.0-0ubuntu0.20.04.1~cloud0 all OpenStack Key Management Service - Keystone Listener
ii barbican-worker 1:10.0.0-0ubuntu0.20.04.1~cloud0 all OpenStack Key Management Service - Worker Node
ii python-barbican 1:6.0.1-0ubuntu1 all OpenStack Key Management Service - Python files
ii python3-barbican 1:10.0.0-0ubuntu0.20.04.1~cloud0 all OpenStack Key Management Service - Python 3 files
ii python3-barbicanclient 4.10.0-0ubuntu1~cloud0 all OpenStack Key Management API client - Python 3.x

Version of nova:
ii nova-api 2:18.3.0-0ubuntu1~cloud1 all OpenStack Compute - API frontend
ii nova-common 2:18.3.0-0ubuntu1~cloud1 all OpenStack Compute - common files
ii nova-conductor 2:18.3.0-0ubuntu1~cloud1 all OpenStack Compute - conductor service
ii nova-consoleauth 2:18.3.0-0ubuntu1~cloud1 all OpenStack Compute - Console Authenticator
ii nova-novncproxy 2:18.3.0-0ubuntu1~cloud1 all OpenStack Compute - NoVNC proxy
ii nova-placement-api 2:18.3.0-0ubuntu1~cloud1 all OpenStack Compute - placement API frontend
ii nova-scheduler 2:18.3.0-0ubuntu1~cloud1 all OpenStack Compute - virtual machine scheduler
ii python-nova 2:18.3.0-0ubuntu1~cloud1 all OpenStack Compute Python 2 libraries
ii python-novaclient 2:11.0.0-0ubuntu1~cloud0 all client library for OpenStack Compute API - Python 2.7

2. We use iSCSI storage(Huawei dorado)

Logs & Configs
==============
Cinder volume logs:
2021-04-23 16:33:43.288 17298 INFO cinder.volume.drivers.huawei.huawei_driver [req-19f3af30-022a-4c96-ab28-2c3aaa91764c 7d52a2a770e4402982390eb658425083 ab5b748ab5e04fa58c8b4aad9f744e2b - default default] Initialize iscsi connection for volume de2c38b7-bf67-42b7-ae51-76fca6c7c05b, connector info {u'initiator': u'iqn.1993-08.org.debian:01:3a7188e5b55f', u'ip': u'10.10.0.248', u'platform': u'x86_64', u'host': u'comp-c08', u'do_local_attach': False, u'mountpoint': u'/dev/vda', u'os_type': u'linux2', u'multipath': True}.
2021-04-23 16:34:05.883 17298 INFO cinder.volume.drivers.huawei.huawei_driver [req-969c4e48-91a7-4c46-824d-63b924503c34 7d52a2a770e4402982390eb658425083 ab5b748ab5e04fa58c8b4aad9f744e2b - default default] Terminate iscsi connection for volume de2c38b7-bf67-42b7-ae51-76fca6c7c05b, connector info {u'initiator': u'iqn.1993-08.org.debian:01:3a7188e5b55f', u'ip': u'10.10.0.248', u'platform': u'x86_64', u'host': u'comp-c08', u'do_local_attach': False, u'mountpoint': u'/dev/vda', u'os_type': u'linux2', u'multipath': True}.

2021-04-23 16:34:12.452 17298 INFO cinder.volume.drivers.huawei.huawei_driver [req-314b1423-2650-4a06-9f2e-945bd648e70f 7d52a2a770e4402982390eb658425083 ab5b748ab5e04fa58c8b4aad9f744e2b - default default] Initialize iscsi connection for volume de2c38b7-bf67-42b7-ae51-76fca6c7c05b, connector info {u'initiator': u'iqn.1993-08.org.debian:01:a1edf5552e8', u'ip': u'10.10.0.253', u'platform': u'x86_64', u'host': u'comp-a09', u'do_local_attach': False, u'mountpoint': u'/dev/vda', u'os_type': u'linux2', u'multipath': True}.

2021-04-23 16:34:58.111 17298 INFO cinder.volume.drivers.huawei.huawei_driver [req-77231901-21bf-42c3-99a3-5a2e0b13a066 7d52a2a770e4402982390eb658425083 ab5b748ab5e04fa58c8b4aad9f744e2b - default default] Terminate iscsi connection for volume de2c38b7-bf67-42b7-ae51-76fca6c7c05b, connector info {u'initiator': u'iqn.1993-08.org.debian:01:36b34c8cbdd8', u'ip': u'10.10.0.148', u'platform': u'x86_64', u'host': u'comp-a03', u'do_local_attach': False, u'mountpoint': u'/dev/vda', u'os_type': u'linux2', u'multipath': True}.

In case some additional logs are needed, just let me know.

Tags:

Revision history for this message

Jan Wasilewski (janwasilewski) wrote on 2021-04-27:

error.log Edit (38.5 KiB, text/plain)

I'm attaching error code from nova-compute logs. It is a bit weird, before attachment dm-55 is not in-use, so this error message doesn't make a sense.

Revision history for this message

Lee Yarwood (lyarwood) wrote on 2021-04-29:

I've added cinder and os-brick here as AFAICT this isn't a Nova issue given the description in c#0 and logs in c#1.

The attempt by Cinder to retype the detached volume doesn't appear to have encrypted anything.

This might be the result of another issue in os-brick when connecting the volume to the underlying host but that isn't obvious from the compute logs.

Can you reproduce this with DEBUG logs on the compute so we can see what os-brick is doing?

Changed in nova:
status:	New → Incomplete

Revision history for this message

Jan Wasilewski (janwasilewski) wrote on 2021-05-04:

new_part_logs.txt Edit (26.1 KiB, text/plain)

I changed nova logging.conf to below values:

[logger_nova]
level = DEBUG
handlers = stderr
qualname = nova

But I do not know if there is something os-brick specific. If there is another way to provide you requested logs - please let me know, I will try to deliver the best possible logs.

You will find new sort of logs in provided attachment.

Sofia Enriquez (lsofia-enriquez) on 2021-05-05

Changed in cinder:
importance:	Undecided → High
tags:	added: encryption retype volumes
tags:	added: multipath
tags:	added: cryptsetup

Revision history for this message

Sofia Enriquez (lsofia-enriquez) wrote on 2021-05-05:

Hi Jan Wasilewski, hope this msg finds you well.

Looks like this is duplicate of https://bugs.launchpad.net/cinder/+bug/1687880

Just to double check, this works with fine with non-root volumes iSCSI storage(Huawei dorado). Have you tried Huawei without iSCI?

This is a known issue and Cinder team discussed this topic last month in Xena PTG, for more context of the problem please check: https://wiki.openstack.org/wiki/CinderXenaPTGSummary#How_to_handle_retyping.2Fmigrating_nonencrypted_volumes_to_encrypted_volumes_of_the_same_size

Cheers,
Sofi

Revision history for this message

Jan Wasilewski (janwasilewski) wrote on 2021-05-05:

Hi Sofi,

thanks for your input here. Unfortunately I have to try with iSCSI storage(Huawei dorado), as I do not have anything else in my environment.
In the meantime when I'm trying to deeply looking to materials shared by you, I can disagree that it looks like duplicate, because retype of volume finished successfully(at least from what I can see in OS), but attaching this volume to an instance fails with the error message shared by me.

I was already informed at #openstack-cinder channel that problem with retype the same size partition is known, so from the beginning I was adding a bit more(around +2GB) of extra space to avoid such issue.

After all I see that topic of this bug can sounds a bit misleading, as it is related to problem with attaching encrypted volume which was retyped from non-encrypted volume to an instance(just to mention: volume has to be filled with some data before, in my use case it was around 50% of partition size. Empty partition was always fine).

Anyway, if there is any logs which can be useful here, I'm more than happy to collect this and try to help with solution.

Best regards,
Jan

Sofia Enriquez (lsofia-enriquez) on 2021-05-05

Changed in os-brick:
importance:	Undecided → High

Revision history for this message

Sofia Enriquez (lsofia-enriquez) wrote on 2021-05-19:

Hi Jan,
Do you mind adding c-vol logs to this bug report?
Thanks

Revision history for this message

Jan Wasilewski (janwasilewski) wrote on 2021-05-19:

logs_cinder.tar.gz Edit (137.4 KiB, application/x-tar)

Hi,

please find attached. At first try it failed with too less space during retype, but second try finished successfully, howerver attachment failed with the same story as previously. If anything else is needed - just let me know.

Revision history for this message

Jan Wasilewski (janwasilewski) wrote on 2022-03-24:

Hi,

any news related to such failure, as I'm still hitting this issue. I believe problem is a bit serious, as there is no nice way to make unencrypted volume easily encrypted.