Cinder

Out of space when migrating encrypted volume between two LVM backends

Bug #1720885 reported by Eric Harney
20
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Cinder
Confirmed
High
Unassigned

Bug Description

Deployed devstack w/ Barbican.

$ cinder show d0d68a2e-7687-4d8f-a376-20d870e7708a
+--------------------------------+-------------------------------------------------+
| Property | Value |
+--------------------------------+-------------------------------------------------+
| attached_servers | [] |
| attachment_ids | [] |
| availability_zone | nova |
| bootable | true |
| consistencygroup_id | None |
| created_at | 2017-10-02T21:12:10.000000 |
| description | None |
| encrypted | True |
| id | d0d68a2e-7687-4d8f-a376-20d870e7708a |
| metadata | |
| migration_status | error |
| multiattach | False |
| name | None |
| os-vol-host-attr:host | fedora25dev.localdomain@lvmdriver-1#lvmdriver-1 |
| os-vol-mig-status-attr:migstat | error |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | bb2142eaba20414d87583fe4b0d3efdf |
| replication_status | None |
| size | 1 |
| snapshot_id | None |
| source_volid | None |
| status | available |
| updated_at | 2017-10-02T21:22:55.000000 |
| user_id | b7ada84e4fa247ef8de596c0e66d3c91 |
| volume_image_metadata | checksum : f8ab98ff5e73ebab884d80c9dc9c7290 |
| | container_format : bare |
| | disk_format : qcow2 |
| | image_id : 43afb53b-2f4c-4d14-a3f1-e08712e9cf34 |
| | image_name : cirros-0.3.5-x86_64-disk |
| | min_disk : 0 |
| | min_ram : 0 |
| | size : 13267968 |
| volume_type | lvmdriver-1 |
+--------------------------------+-------------------------------------------------+

$ cinder migrate --force-host-copy True d0d68a2e-7687-4d8f-a376-20d870e7708a fedora25dev.localdomain@lvmdriver-2

2017-10-02 17:16:08.973 TRACE cinder.volume.manager Traceback (most recent call last):
2017-10-02 17:16:08.973 TRACE cinder.volume.manager File "/opt/stack/cinder/cinder/volume/manager.py", line 2094, in _migrate_volume_generic
2017-10-02 17:16:08.973 TRACE cinder.volume.manager self._copy_volume_data(ctxt, volume, new_volume, remote='dest')
2017-10-02 17:16:08.973 TRACE cinder.volume.manager File "/opt/stack/cinder/cinder/volume/manager.py", line 2011, in _copy_volume_data
2017-10-02 17:16:08.973 TRACE cinder.volume.manager {'src': src_vol['id'], 'dest': dest_vol['id']})
2017-10-02 17:16:08.973 TRACE cinder.volume.manager File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-02 17:16:08.973 TRACE cinder.volume.manager self.force_reraise()
2017-10-02 17:16:08.973 TRACE cinder.volume.manager File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-10-02 17:16:08.973 TRACE cinder.volume.manager six.reraise(self.type_, self.value, self.tb)
2017-10-02 17:16:08.973 TRACE cinder.volume.manager File "/opt/stack/cinder/cinder/volume/manager.py", line 2006, in _copy_volume_data
2017-10-02 17:16:08.973 TRACE cinder.volume.manager sparse=sparse_copy_volume)
2017-10-02 17:16:08.973 TRACE cinder.volume.manager File "/opt/stack/cinder/cinder/volume/utils.py", line 585, in copy_volume
2017-10-02 17:16:08.973 TRACE cinder.volume.manager sparse=sparse)
2017-10-02 17:16:08.973 TRACE cinder.volume.manager File "/opt/stack/cinder/cinder/volume/utils.py", line 469, in _copy_volume_with_path
2017-10-02 17:16:08.973 TRACE cinder.volume.manager execute(*cmd, run_as_root=True)
2017-10-02 17:16:08.973 TRACE cinder.volume.manager File "/opt/stack/cinder/cinder/utils.py", line 123, in execute
2017-10-02 17:16:08.973 TRACE cinder.volume.manager return processutils.execute(*cmd, **kwargs)
2017-10-02 17:16:08.973 TRACE cinder.volume.manager File "/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 400, in execute
2017-10-02 17:16:08.973 TRACE cinder.volume.manager cmd=sanitized_cmd)
2017-10-02 17:16:08.973 TRACE cinder.volume.manager ProcessExecutionError: Unexpected error while running command.
2017-10-02 17:16:08.973 TRACE cinder.volume.manager Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf dd if=/dev/disk/by-id/scsi-36001405dae6f7cd109c44a8a280c0ed3 of=/dev/disk/by-id/scsi-3600140557a05fbd6a5749abb7b47c58a count=1073741824 bs=1M iflag=count_bytes,direct oflag=direct conv=sparse
2017-10-02 17:16:08.973 TRACE cinder.volume.manager Exit code: 1
2017-10-02 17:16:08.973 TRACE cinder.volume.manager Stdout: u''
2017-10-02 17:16:08.973 TRACE cinder.volume.manager Stderr: u"/bin/dd: error writing '/dev/disk/by-id/scsi-3600140557a05fbd6a5749abb7b47c58a': No space left on device\n1022+1 records in\n1022+0 records out\n1071644672 bytes (1.1 GB, 1022 MiB) copied, 38.9663 s, 27.5 MB/s\n"
2017-10-02 17:16:08.973 TRACE cinder.volume.manager

See original description
Eric Harney (eharney)
description: updated
Eric Harney (eharney)
tags: added: encryption
Revision history for this message
Vishakha Agarwal (vishakha.agarwal) wrote :

Hi all

Is any one working on it? I am willing to take this issue.

Thanks

Revision history for this message
Sean McGinnis (sean-mcginnis) wrote :

Vishakha, if you are still able to work on this, please reassign the owner to you and proceed. Thanks!

XF4jVw5nyfgRK (cnsg2zd7oyphe)
Changed in cinder:
assignee: Eric Harney (eharney) → XF4jVw5nyfgRK (cnsg2zd7oyphe)
Revision history for this message
yenai (yenai2008) wrote :

The reason is luks-volume contains LUKS header in the device, it costs 2M (512bytes * 4096sectors).

[root@localhost ~]# cryptsetup status /dev/mapper/crypt-scsi-36001405eb821aa8e66e44ecbfec83dfa
/dev/mapper/crypt-scsi-36001405eb821aa8e66e44ecbfec83dfa is active and is in use.
  type: LUKS1
  cipher: aes-xts-plain64
  keysize: 512 bits
  key location: dm-crypt
  device: /dev/sde
  sector size: 512
  offset: 4096 sectors
  size: 2093056 sectors
  mode: read/write

[root@localhost ~]# cryptsetup luksDump /dev/sde
LUKS header information for /dev/sde

Version: 1
Cipher name: aes
Cipher mode: xts-plain64
Hash spec: sha256
Payload offset: 4096
MK bits: 512
MK digest: 5d 6a d7 86 41 d0 f4 0a dd 55 c7 c6 ff f5 ed 63 76 e9 61 fd
MK salt: 77 c2 88 b6 a0 7e cb 18 f2 79 63 09 19 14 dc 76
                40 8b 0f d5 eb aa 0d ef ba 85 e3 58 75 48 bc df
MK iterations: 15784
UUID: 83431299-e4c5-4999-b007-973425725ace

Key Slot 0: ENABLED
        Iterations: 279470
        Salt: 8e 57 e5 c6 d2 65 d2 da 94 bb cb a7 65 e3 2c 6d
                                d1 46 7d 30 0e 1f d1 33 89 39 f9 97 30 54 aa 17
        Key material offset: 8
        AF stripes: 4000
Key Slot 1: DISABLED
Key Slot 2: DISABLED
Key Slot 3: DISABLED
Key Slot 4: DISABLED
Key Slot 5: DISABLED
Key Slot 6: DISABLED
Key Slot 7: DISABLED

Yeshwanth Allampati (yeshwanth-allampati)
Changed in cinder:
assignee: XF4jVw5nyfgRK (cnsg2zd7oyphe) → Yeshwanth Allampati (yeshwanth-allampati)
assignee: Yeshwanth Allampati (yeshwanth-allampati) → nobody
assignee: nobody → Yeshwanth Allampati (yeshwanth-allampati)
Yeshwanth Allampati (yeshwanth-allampati)
description: updated
Revision history for this message
Yeshwanth Allampati (yeshwanth-allampati) wrote :

If this is an Encrypted Volume, why is the volumetype lvmdriver and not LUKS?

Revision history for this message
Yeshwanth Allampati (yeshwanth-allampati) wrote :
Download full text (4.7 KiB)

I configured an LVM Backend and tried this out and this worked fine for me.

cinder show 3f83570d-b1a0-4c91-9773-2b697c1e982d

+--------------------------------+--------------------------------------+
| Property | Value |
 +--------------------------------+--------------------------------------+
| attached_servers | [] |
| attachment_ids | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2019-08-02T08:17:09.000000 |
| description | None |
| encrypted | True |
| id | 3f83570d-b1a0-4c91-9773-2b697c1e982d |
| metadata | |
| migration_status | success |
| multiattach | False |
| name | encrypted volume |
| os-vol-host-attr:host | infra@lvmdriver-1#lvmdriver-1 |
| os-vol-mig-status-attr:migstat | success |
| os-vol-mig-status-attr:name_id | e9b415df-1ec7-4e41-a7c8-d9f4c8a39ba5 |
| os-vol-tenant-attr:tenant_id | 5cb1426e43cd4f2db6bb428a6051622a |
| replication_status | None |
| size | 1 |
| snapshot_id | None |
| source_volid | None |
| status | available |
| updated_at | 2019-08-02T09:56:23.000000 |
| user_id | d645bb574c394feeb2390140ed981f8c |
| volume_type | LUKS |
 +--------------------------------+--------------------------------------+

I executed the command

cinder migrate --force-host-copy True 3f83570d-b1a0-4c91-9773-2b697c1e982d infra@lvmdriver-3
Request to migrate volume 3f83570d-b1a0-4c91-9773-2b697c1e982d has been accepted.

After the command.. I got

cinder show 3f83570d-b1a0-4c91-9773-2b697c1e982d
+--------------------------------+--------------------------------------+
| Property | Value |
 +--------------------------------+--------------------------------------+
| attached_servers | [] |
| attachment_ids | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2019-08-02T08:17:09.00...

Read more...

Revision history for this message
Yeshwanth Allampati (yeshwanth-allampati) wrote :

@Eric Harney .. The /etc/cinder/cinder.conf is very much needed to understand the root cause of the problem.

Analysis of lvm backend configuration is needed.

Yeshwanth Allampati (yeshwanth-allampati)
Changed in cinder:
status: New → In Progress
Revision history for this message
Yeshwanth Allampati (yeshwanth-allampati) wrote :

This worked fine for me but the Duplicate of this bug i.e.,

https://bugs.launchpad.net/cinder/+bug/1824321

got reproduced. Working on it.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.opendev.org/682339

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/682355

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on cinder (stable/stein)

Change abandoned by Sean McGinnis (<email address hidden>) on branch: stable/stein
Review: https://review.opendev.org/682355
Reason: Abandoning since there are issues here that are just going to cause test failures, and needs to be proposed to master first before backporting to stein.

Yeshwanth Allampati (yeshwanth-allampati)
Changed in cinder:
status: In Progress → Confirmed
Yeshwanth Allampati (yeshwanth-allampati)
Changed in cinder:
assignee: Yeshwanth Allampati (yeshwanth-allampati) → nobody
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on cinder (master)

Change abandoned by "Yeshwanth Allampati <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/cinder/+/682339

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.