available volume from no-encrytion to luks retype failed

Bug #1824321 reported by yenai
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Cinder
In Progress
Undecided
Yeshwanth Allampati

Bug Description

[stack@localhost devstack]$ cinder encryption-type-show luks-type
+--------------------------------------+----------+-----------------+----------+------------------+
| Volume Type ID | Provider | Cipher | Key Size | Control Location |
+--------------------------------------+----------+-----------------+----------+------------------+
| 0cfcee9d-20cb-4552-a440-d8203f8e11ad | luks | aes-xts-plain64 | 512 | front-end |
+--------------------------------------+----------+-----------------+----------+------------------+

[stack@localhost devstack]$ cinder show 9c91062c-a16d-44d4-b01d-7e45aa37797e
+--------------------------------+-----------------------------------------------+
| Property | Value |
+--------------------------------+-----------------------------------------------+
| attached_servers | [] |
| attachment_ids | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2019-01-13T08:55:32.000000 |
| description | |
| encrypted | False |
| id | 9c91062c-a16d-44d4-b01d-7e45aa37797e |
| metadata | |
| migration_status | None |
| multiattach | False |
| name | yenai-001 |
| os-vol-host-attr:host | localhost.localdomain@lvmdriver-1#lvmdriver-1 |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | b984b88d70dd4cbb8f99efa4a48ebfe9 |
| replication_status | None |
| size | 1 |
| snapshot_id | None |
| source_volid | None |
| status | available |
| updated_at | 2019-01-13T08:55:33.000000 |
| user_id | edffe81b43534dfea48effced3ac8e53 |
| volume_type | lvmdriver-1 |
+--------------------------------+-----------------------------------------------+

[stack@localhost devstack]$ cinder retype --migration-policy on-demand 9c91062c-a16d-44d4-b01d-7e45aa37797e luks-type

ERROR MSG:
Jan 13 19:00:46 localhost.localdomain cinder-volume[46318]: ERROR cinder.volume.manager [req-7a0380ea-987a-4803-a76b-c1edbb5a4ca8 req-f2187717-b025-4fb9-8037-f10b95d4695b admin None] Failed to copy volume 9c91062c-a16d-44d4-b01d-7e45aa37797e to bc12e2ee-cecc-4e5a-b6b0-8fc7678a66f3: ProcessExecutionError: Unexpected error while running command.
Jan 13 19:00:46 localhost.localdomain cinder-volume[46318]: Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf dd if=/dev/sdh of=/dev/disk/by-id/scsi-36001405d4d948676c98448f9d10a2217 count=1073741824 bs=1M iflag=count_bytes,direct oflag=direct conv=sparse
Jan 13 19:00:46 localhost.localdomain cinder-volume[46318]: Exit code: 1
Jan 13 19:00:46 localhost.localdomain cinder-volume[46318]: Stdout: u''
Jan 13 19:00:46 localhost.localdomain cinder-volume[46318]: Stderr: '/bin/dd: error writing \xe2\x80\x98/dev/disk/by-id/scsi-36001405d4d948676c98448f9d10a2217\xe2\x80\x99: No space left on device\n1023+0 records in\n1022+0 records out\n1071644672 bytes (1.1 GB) copied, 3.06053 s, 350 MB/s\n'
Jan 13 19:00:46 localhost.localdomain cinder-volume[46318]: ERROR cinder.volume.manager Traceback (most recent call last):
Jan 13 19:00:46 localhost.localdomain cinder-volume[46318]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/manager.py", line 2166, in _migrate_volume_generic
Jan 13 19:00:46 localhost.localdomain cinder-volume[46318]: ERROR cinder.volume.manager self._copy_volume_data(ctxt, volume, new_volume, remote='dest')
Jan 13 19:00:46 localhost.localdomain cinder-volume[46318]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/manager.py", line 2081, in _copy_volume_data
Jan 13 19:00:46 localhost.localdomain cinder-volume[46318]: ERROR cinder.volume.manager {'src': src_vol['id'], 'dest': dest_vol['id']})
Jan 13 19:00:46 localhost.localdomain cinder-volume[46318]: ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
Jan 13 19:00:46 localhost.localdomain cinder-volume[46318]: ERROR cinder.volume.manager self.force_reraise()
Jan 13 19:00:46 localhost.localdomain cinder-volume[46318]: ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
Jan 13 19:00:46 localhost.localdomain cinder-volume[46318]: ERROR cinder.volume.manager six.reraise(self.type_, self.value, self.tb)
Jan 13 19:00:46 localhost.localdomain cinder-volume[46318]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/manager.py", line 2077, in _copy_volume_data
Jan 13 19:00:46 localhost.localdomain cinder-volume[46318]: ERROR cinder.volume.manager sparse=sparse_copy_volume)
Jan 13 19:00:46 localhost.localdomain cinder-volume[46318]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/utils.py", line 602, in copy_volume
Jan 13 19:00:46 localhost.localdomain cinder-volume[46318]: ERROR cinder.volume.manager sparse=sparse)
Jan 13 19:00:46 localhost.localdomain cinder-volume[46318]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/utils.py", line 486, in _copy_volume_with_path
Jan 13 19:00:46 localhost.localdomain cinder-volume[46318]: ERROR cinder.volume.manager execute(*cmd, run_as_root=True)
Jan 13 19:00:46 localhost.localdomain cinder-volume[46318]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/utils.py", line 128, in execute
Jan 13 19:00:46 localhost.localdomain cinder-volume[46318]: ERROR cinder.volume.manager return processutils.execute(*cmd, **kwargs)
Jan 13 19:00:46 localhost.localdomain cinder-volume[46318]: ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 424, in execute
Jan 13 19:00:46 localhost.localdomain cinder-volume[46318]: ERROR cinder.volume.manager cmd=sanitized_cmd)
Jan 13 19:00:46 localhost.localdomain cinder-volume[46318]: ERROR cinder.volume.manager ProcessExecutionError: Unexpected error while running command.
Jan 13 19:00:46 localhost.localdomain cinder-volume[46318]: ERROR cinder.volume.manager Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf dd if=/dev/sdh of=/dev/disk/by-id/scsi-36001405d4d948676c98448f9d10a2217 count=1073741824 bs=1M iflag=count_bytes,direct oflag=direct conv=sparse
Jan 13 19:00:46 localhost.localdomain cinder-volume[46318]: ERROR cinder.volume.manager Exit code: 1
Jan 13 19:00:46 localhost.localdomain cinder-volume[46318]: ERROR cinder.volume.manager Stdout: u''
Jan 13 19:00:46 localhost.localdomain cinder-volume[46318]: ERROR cinder.volume.manager Stderr: '/bin/dd: error writing \xe2\x80\x98/dev/disk/by-id/scsi-36001405d4d948676c98448f9d10a2217\xe2\x80\x99: No space left on device\n1023+0 records in\n1022+0 records out\n1071644672 bytes (1.1 GB) copied, 3.06053 s, 350 MB/s\n'
Jan 13 19:00:46 localhost.localdomain cinder-volume[46318]: ERROR cinder.volume.manager

====================================================================================================
The reason is luks-volume contains LUKS header in the device, it costs 2M (512bytes * 4096sectors).

[root@localhost ~]# cryptsetup status /dev/mapper/crypt-scsi-36001405eb821aa8e66e44ecbfec83dfa
/dev/mapper/crypt-scsi-36001405eb821aa8e66e44ecbfec83dfa is active and is in use.
  type: LUKS1
  cipher: aes-xts-plain64
  keysize: 512 bits
  key location: dm-crypt
  device: /dev/sde
  sector size: 512
  offset: 4096 sectors
  size: 2093056 sectors
  mode: read/write

[root@localhost ~]# cryptsetup luksDump /dev/sde
LUKS header information for /dev/sde

Version: 1
Cipher name: aes
Cipher mode: xts-plain64
Hash spec: sha256
Payload offset: 4096
MK bits: 512
MK digest: 5d 6a d7 86 41 d0 f4 0a dd 55 c7 c6 ff f5 ed 63 76 e9 61 fd
MK salt: 77 c2 88 b6 a0 7e cb 18 f2 79 63 09 19 14 dc 76
                40 8b 0f d5 eb aa 0d ef ba 85 e3 58 75 48 bc df
MK iterations: 15784
UUID: 83431299-e4c5-4999-b007-973425725ace

Key Slot 0: ENABLED
        Iterations: 279470
        Salt: 8e 57 e5 c6 d2 65 d2 da 94 bb cb a7 65 e3 2c 6d
                                d1 46 7d 30 0e 1f d1 33 89 39 f9 97 30 54 aa 17
        Key material offset: 8
        AF stripes: 4000
Key Slot 1: DISABLED
Key Slot 2: DISABLED
Key Slot 3: DISABLED
Key Slot 4: DISABLED
Key Slot 5: DISABLED
Key Slot 6: DISABLED
Key Slot 7: DISABLED

yenai (yenai2008)
description: updated
Revision history for this message
yenai (yenai2008) wrote :

When use plain, it's OK.

[root@localhost ~]# cryptsetup status "/dev/disk/by-id/scsi-3600140548df84c9a1704122a77658534"
/dev/disk/by-id/scsi-3600140548df84c9a1704122a77658534 is active and is in use.
  type: PLAIN
  cipher: aes-xts-plain64
  keysize: 512 bits
  key location: dm-crypt
  device: /dev/sdf
  sector size: 512
  offset: 0 sectors
  size: 2097152 sectors
  mode: read/write

Revision history for this message
Eric Harney (eharney) wrote :
Revision history for this message
Yeshwanth Allampati (yeshwanth-allampati) wrote :

Hi... bug #1720885 went on well on my Devstack setup. But I was able to reproduce this one.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.opendev.org/682339

Changed in cinder:
assignee: nobody → Yeshwanth Allampati (yeshwanth-allampati)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/682355

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on cinder (stable/stein)

Change abandoned by Sean McGinnis (<email address hidden>) on branch: stable/stein
Review: https://review.opendev.org/682355
Reason: Abandoning since there are issues here that are just going to cause test failures, and needs to be proposed to master first before backporting to stein.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on cinder (master)

Change abandoned by "Yeshwanth Allampati <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/cinder/+/682339

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.