Cinder

Properties of an attached volume are lost after live migration

Bug #1782714 reported by Viktor Tikkanen
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
New
Medium
Unassigned
OpenStack Compute (nova)
Invalid
Undecided
Unassigned

Bug Description

Steps to reproduce the problem:

1. Launch an instance.
2. Create a volume and attach it to the instance. The volume will have "attached_mode='rw'" -property:

[cloudadmin@controller-1 ~(admin)]$ openstack volume list --long
+--------------------------------------+----------+--------+------+------+----------+------------------------------------+--------------------+
| ID | Name | Status | Size | Type | Bootable | Attached to | Properties |
+--------------------------------------+----------+--------+------+------+----------+------------------------------------+--------------------+
| 4394bec9-3b87-4a6e-b977-cb56719b0d2a | test_vol | in-use | 1 | None | false | Attached to cirros-01 on /dev/vdb | attached_mode='rw' |
+--------------------------------------+----------+--------+------+------+----------+------------------------------------+--------------------+

3. Start live migration of the instance:

[cloudadmin@controller-1 ~(admin)]$ openstack server migrate --live compute-2 --block-migration cirros-01

4. After completion of the migration volume properties are lost:

[cloudadmin@controller-1 ~(admin)]$ openstack volume list --long
+--------------------------------------+----------+--------+------+------+----------+----------------------------------------------------------------------+--------------------+
| ID | Name | Status | Size | Type | Bootable | Attached to | Properties |
+--------------------------------------+----------+--------+------+------+----------+----------------------------------------------------------------------+--------------------+
| 4394bec9-3b87-4a6e-b977-cb56719b0d2a | test_vol | in-use | 1 | None | false | Attached to cirros-01 on /dev/vdb Attached to cirros-01 on /dev/vdb | attached_mode='rw' |
+--------------------------------------+----------+--------+------+------+----------+----------------------------------------------------------------------+--------------------+
[cloudadmin@controller-1 ~(admin)]$
[cloudadmin@controller-1 ~(admin)]$ openstack volume list --long
+--------------------------------------+----------+--------+------+------+----------+------------------------------------+------------+
| ID | Name | Status | Size | Type | Bootable | Attached to | Properties |
+--------------------------------------+----------+--------+------+------+----------+------------------------------------+------------+
| 4394bec9-3b87-4a6e-b977-cb56719b0d2a | test_vol | in-use | 1 | None | false | Attached to cirros-01 on /dev/vdb | |
+--------------------------------------+----------+--------+------+------+----------+------------------------------------+------------+
[cloudadmin@controller-1 ~(admin)]$

Version information:

[cloudadmin@controller-1 ~(admin)]$ sudo rpm -qa|grep nova
openstack-ansible-os_nova-17.0.2-1.el7.centos.ncir.2.noarch
python-nova-17.0.2-1.el7.centos.ncir.2.noarch
openstack-nova-placement-api-17.0.2-1.el7.centos.ncir.2.noarch
openstack-nova-novncproxy-17.0.2-1.el7.centos.ncir.2.noarch
nova-inventory-c2.g8617a07-1.el7.centos.ncir.noarch
openstack-nova-scheduler-17.0.2-1.el7.centos.ncir.2.noarch
openstack-nova-conductor-17.0.2-1.el7.centos.ncir.2.noarch
python2-novaclient-10.1.0-1.el7.noarch
openstack-nova-compute-17.0.2-1.el7.centos.ncir.2.noarch
openstack-nova-api-17.0.2-1.el7.centos.ncir.2.noarch
openstack-nova-common-17.0.2-1.el7.centos.ncir.2.noarch
openstack-nova-console-17.0.2-1.el7.centos.ncir.2.noarch

[cloudadmin@controller-1 ~(admin)]$ sudo rpm -qa|grep cinder
python-cinder-12.0.2-2.el7.noarch
openstack-cinder-12.0.2-2.el7.noarch
python2-cinderclient-3.5.0-1.el7.noarch
openstack-ansible-os_cinder-17.0.2-1.el7.centos.ncir.1.noarch
[cloudadmin@controller-1 ~(admin)]$

ceph: 12.2.5
libvirt: 3.9.0

Revision history for this message
Matt Riedemann (mriedem) wrote :

Adding cinder since cinder controls the volume metadata and nova shouldn't be changing anything during live migration with respect to the attached_mode.

tags: added: live-migration volumes
Revision history for this message
Matt Riedemann (mriedem) wrote :

Note the "Attached to cirros-01 on /dev/vdb Attached to cirros-01 on /dev/vdb" during the live migration. That's probably while the volume has 2 attachments to the same server, one on the source host and one on the destination host. My guess is the attached_mode is lost during the creation/update of the 2nd attachment for the destination host.

Are you able to get the attachment details before and after the live migration and see what those look like using this API?

https://developer.openstack.org/api-ref/block-storage/v3/#list-attachments-with-details

or https://developer.openstack.org/api-ref/block-storage/v3/#list-attachments-with-details

Unfortunately GET /v3/{project_id}/attachments/detail doesn't support filtering by volume_id.

Revision history for this message
Matt Riedemann (mriedem) wrote :

From the volume details itself, there is an attachments entry which would have the attachment_id for each attachment to the volume, which you could use to get the attachment detail information in the attachments API.

https://developer.openstack.org/api-ref/block-storage/v3/#show-a-volume-s-details

Revision history for this message
Viktor Tikkanen (viktor-tikkanen) wrote :
Download full text (4.5 KiB)

It seems that new attachment details are OK but in the volume details metadata is lost:

Attachments details before live migration:

{"attachment": {"status": "attached", "detached_at": "", "connection_info": {"attachment_id": "52113892-d425-4686-ac09-cce6b8529f17", "encrypted": false, "driver_volume_type": "rbd", "secret_uuid": "2eee3a45-5bbd-48c6-b940-6c0b67dc8347", "qos_specs": null, "volume_id": "4394bec9-3b87-4a6e-b977-cb56719b0d2a", "auth_username": "cinder", "secret_type": "ceph", "name": "volumes/volume-4394bec9-3b87-4a6e-b977-cb56719b0d2a", "keyring": null, "cluster_name": "ceph", "auth_enabled": true, "hosts": ["192.168.12.21", "192.168.12.22", "192.168.12.23"], "discard": true, "access_mode": "rw", "ports": ["6789", "6789", "6789"]}, "attached_at": "2018-07-25T05:17:47.000000", "attach_mode": "rw", "instance": "ff97a18d-cf12-4133-9df1-ee22a7304fc7", "volume_id": "4394bec9-3b87-4a6e-b977-cb56719b0d2a", "id": "52113892-d425-4686-ac09-cce6b8529f17"}}

Attachments details after live migration:

{"attachment": {"status": "attached", "detached_at": "", "connection_info": {"attachment_id": "8d229721-a25e-48ea-b4b9-39a2c90611ab", "encrypted": false, "driver_volume_type": "rbd", "secret_uuid": "2eee3a45-5bbd-48c6-b940-6c0b67dc8347", "qos_specs": null, "volume_id": "4394bec9-3b87-4a6e-b977-cb56719b0d2a", "auth_username": "cinder", "secret_type": "ceph", "name": "volumes/volume-4394bec9-3b87-4a6e-b977-cb56719b0d2a", "keyring": null, "cluster_name": "ceph", "auth_enabled": true, "hosts": ["192.168.12.21", "192.168.12.22", "192.168.12.23"], "discard": true, "access_mode": "rw", "ports": ["6789", "6789", "6789"]}, "attached_at": "2018-07-25T05:53:52.000000", "attach_mode": "rw", "instance": "ff97a18d-cf12-4133-9df1-ee22a7304fc7", "volume_id": "4394bec9-3b87-4a6e-b977-cb56719b0d2a", "id": "8d229721-a25e-48ea-b4b9-39a2c90611ab"}}

Volume details before live migration:

{"volume": {"migration_status": null, "attachments": [{"server_id": "ff97a18d-cf12-4133-9df1-ee22a7304fc7", "attachment_id": "52113892-d425-4686-ac09-cce6b8529f17", "attached_at": "2018-07-25T05:17:47.000000", "host_name": "compute-2", "volume_id": "4394bec9-3b87-4a6e-b977-cb56719b0d2a", "device": "/dev/vdb", "id": "4394bec9-3b87-4a6e-b977-cb56719b0d2a"}], "links": [{"href": "https://10.7.25.34:8776/v3/b742e570d76d496e8d3d0cca8f2f039a/volumes/4394bec9-3b87-4a6e-b977-cb56719b0d2a", "rel": "self"}, {"href": "https://10.7.25.34:8776/b742e570d76d496e8d3d0cca8f2f039a/volumes/4394bec9-3b87-4a6e-b977-cb56719b0d2a", "rel": "bookmark"}], "availability_zone": "nova", "os-vol-host-attr:host": "controller-2@rbd#volumes_hdd", "encrypted": false, "updated_at": "2018-07-25T05:17:49.000000", "replication_status": null, "snapshot_id": null, "id": "4394bec9-3b87-4a6e-b977-cb56719b0d2a", "size": 1, "user_id": "545682b2c9b7466694bca136518a4568", "os-vol-tenant-attr:tenant_id": "b742e570d76d496e8d3d0cca8f2f039a", "os-vol-mig-status-attr:migstat": null, "metadata": {"attached_mode": "rw"}, "status": "in-use", "description": "", "multiattach": false, "source_volid": null, "consistencygroup_id": null, "os-vol-mig-status-attr:name_id": null, "name": "test_vol", "bootable": "false", "created_at": "2018-07...

Read more...

Revision history for this message
Viktor Tikkanen (viktor-tikkanen) wrote :

And here are the volume details from the time when both attachments exist during live migration (metadata is still OK at this point):

{"volume": {"migration_status": null, "attachments": [{"server_id": "ff97a18d-cf12-4133-9df1-ee22a7304fc7", "attachment_id": "2f9944ba-9aa4-4e3f-a58f-7d60faae6ed5", "attached_at": "2018-07-25T06:15:42.000000", "host_name": "compute-1", "volume_id": "4394bec9-3b87-4a6e-b977-cb56719b0d2a", "device": "/dev/vdb", "id": "4394bec9-3b87-4a6e-b977-cb56719b0d2a"}, {"server_id": "ff97a18d-cf12-4133-9df1-ee22a7304fc7", "attachment_id": "e4a1918e-c3f1-429d-a006-84b0e27a9eea", "attached_at": "2018-07-25T06:28:17.000000", "host_name": "compute-3", "volume_id": "4394bec9-3b87-4a6e-b977-cb56719b0d2a", "device": "/dev/vdb", "id": "4394bec9-3b87-4a6e-b977-cb56719b0d2a"}], "links": [{"href": "https://10.7.25.34:8776/v3/b742e570d76d496e8d3d0cca8f2f039a/volumes/4394bec9-3b87-4a6e-b977-cb56719b0d2a", "rel": "self"}, {"href": "https://10.7.25.34:8776/b742e570d76d496e8d3d0cca8f2f039a/volumes/4394bec9-3b87-4a6e-b977-cb56719b0d2a", "rel": "bookmark"}], "availability_zone": "nova", "os-vol-host-attr:host": "controller-2@rbd#volumes_hdd", "encrypted": false, "updated_at": "2018-07-25T06:28:34.000000", "replication_status": null, "snapshot_id": null, "id": "4394bec9-3b87-4a6e-b977-cb56719b0d2a", "size": 1, "user_id": "545682b2c9b7466694bca136518a4568", "os-vol-tenant-attr:tenant_id": "b742e570d76d496e8d3d0cca8f2f039a", "os-vol-mig-status-attr:migstat": null, "metadata": {"attached_mode": "rw"}, "status": "in-use", "description": "", "multiattach": false, "source_volid": null, "consistencygroup_id": null, "os-vol-mig-status-attr:name_id": null, "name": "test_vol", "bootable": "false", "created_at": "2018-07-20T06:16:14.000000", "volume_type": null}}

Revision history for this message
Gorka Eguileor (gorka) wrote :

I think this could be related to how things are seen now with the multi-attach feature.

Prior to having multi-attach we stored the "attached_mode" in the volume's metadata (which is presented as the "Properties" column by the OpenStack client.

But now that we have multi-attach we store the "attached_mode" as a specific column in our "volume_attachment" DB table.

So I think that after we do a migration of a volume we may lose that metadata.

Please set use the following command to see the attachments available on the volume:
  "cinder --os-volume-api-version=3.27 attachment-list --volume-id=4394bec9-3b87-4a6e-b977-cb56719b0d2a"

And once you have the ID of the attachment, check the details with
  "cinder --os-volume-api-version=3.27 attachment-show $ATTACHMENT_UUID"

And see if the right attachment mode is present there.

Revision history for this message
Gorka Eguileor (gorka) wrote :

If that is the case, could you try with a volume that has a different attached_mode? As I believe we may have a bug there.

It looks like we are redoing all attachments as "rw" [1]:

            for attachment in volume_attachments:
                LOG.debug('Re-attaching: %s', attachment)
                # This is just a db state toggle, the volume is actually
                # already attach and in-use, new attachment flow won't allow
                # this
                rpcapi.attach_volume(ctxt, volume,
                                     attachment.instance_uuid,
                                     attachment.attached_host,
                                     attachment.mountpoint,
                                     'rw')

[1]: https://github.com/openstack/cinder/blob/37f2bdcdec85f27651b91a9a2d0fddb66e7bfe8a/cinder/volume/manager.py#L2312-L2316

Revision history for this message
Jay Bryant (jsbryant) wrote :

@Viktor, are you able to answer Gorka's questions?

Changed in cinder:
importance: Undecided → Medium
Revision history for this message
Viktor Tikkanen (viktor-tikkanen) wrote :
Download full text (7.1 KiB)

Hi!

Below is the output:

[cloudadmin@controller-1 ~(admin)]$ openstack server add volume cirros-01 tst-volume
[cloudadmin@controller-1 ~(admin)]$
[cloudadmin@controller-1 ~(admin)]$ cinder --os-volume-api-version=3.27 attachment-list --volume-id=df03bb89-7383-4d21-bc67-1da89b9fa1cf
+--------------------------------------+--------------------------------------+----------+-----------+
| ID | Volume ID | Status | Server ID |
 +--------------------------------------+--------------------------------------+----------+-----------+
| e7e2fda8-b742-46c5-a758-3559c923d1cb | df03bb89-7383-4d21-bc67-1da89b9fa1cf | attached | |
 +--------------------------------------+--------------------------------------+----------+-----------+
[cloudadmin@controller-1 ~(admin)]$
[cloudadmin@controller-1 ~(admin)]$ cinder --os-volume-api-version=3.27 attachment-show e7e2fda8-b742-46c5-a758-3559c923d1cb
+-------------+--------------------------------------+
| Property | Value |
 +-------------+--------------------------------------+
| attach_mode | rw |
| attached_at | 2018-09-27T04:46:00.000000 |
| detached_at | |
| id | e7e2fda8-b742-46c5-a758-3559c923d1cb |
| instance | cd8d1af1-24ec-4539-bdc6-82bdd6988851 |
| status | attached |
| volume_id | df03bb89-7383-4d21-bc67-1da89b9fa1cf |
 +-------------+--------------------------------------+
+--------------------+--------------------------------------------------------+
| Property | Value |
 +--------------------+--------------------------------------------------------+
| access_mode | rw |
| attachment_id | e7e2fda8-b742-46c5-a758-3559c923d1cb |
| auth_enabled | True |
| auth_username | cinder |
| cluster_name | ceph |
| discard | True |
| driver_volume_type | rbd |
| encrypted | False |
| hosts | [u'192.168.12.21', u'192.168.12.25', u'192.168.12.26'] |
| keyring | None |
| name | volumes/volume-df03bb89-7383-4d21-bc67-1da89b9fa1cf |
| ports | [u'6789', u'6789', u'6789'] |
| qos_specs | None |
| secret_type | ceph |
| secret_uuid | ba484b64-4de7-47cf-8af0-96abfbec84cd |
| volume_id | df03bb89-7383-4d21-bc67-1da89b9fa1cf |
 +--------------------+--------------------------------------------------------+
[cloudadmin@controller-1 ~(admin)]...

Read more...

Revision history for this message
Viktor Tikkanen (viktor-tikkanen) wrote :
Download full text (10.3 KiB)

And if I change attach_mode value (e.g. rw -> ro directly into database) and make live migration, the attach_mode is reset back to rw:

[cloudadmin@controller-1 ~(admin)]$ cinder --os-volume-api-version=3.27 attachment-list --volume-id=df03bb89-7383-4d21-bc67-1da89b9fa1cf
+--------------------------------------+--------------------------------------+----------+-----------+
| ID | Volume ID | Status | Server ID |
 +--------------------------------------+--------------------------------------+----------+-----------+
| b20146fd-2d75-4288-afb8-c4d1a3415d2b | df03bb89-7383-4d21-bc67-1da89b9fa1cf | attached | |
 +--------------------------------------+--------------------------------------+----------+-----------+
[cloudadmin@controller-1 ~(admin)]$ cinder --os-volume-api-version=3.27 attachment-show b20146fd-2d75-4288-afb8-c4d1a3415d2b
+-------------+--------------------------------------+
| Property | Value |
 +-------------+--------------------------------------+
| attach_mode | rw |
| attached_at | 2018-09-27T05:45:31.000000 |
| detached_at | |
| id | b20146fd-2d75-4288-afb8-c4d1a3415d2b |
| instance | cd8d1af1-24ec-4539-bdc6-82bdd6988851 |
| status | attached |
| volume_id | df03bb89-7383-4d21-bc67-1da89b9fa1cf |
 +-------------+--------------------------------------+
+--------------------+--------------------------------------------------------+
| Property | Value |
 +--------------------+--------------------------------------------------------+
| access_mode | rw |
| attachment_id | b20146fd-2d75-4288-afb8-c4d1a3415d2b |
| auth_enabled | True |
| auth_username | cinder |
| cluster_name | ceph |
| discard | True |
| driver_volume_type | rbd |
| encrypted | False |
| hosts | [u'192.168.12.21', u'192.168.12.25', u'192.168.12.26'] |
| keyring | None |
| name | volumes/volume-df03bb89-7383-4d21-bc67-1da89b9fa1cf |
| ports | [u'6789', u'6789', u'6789'] |
| qos_specs | None |
| secret_type | ceph |
| secret_uuid | ba484b64-4de7-47cf-8af0-96abfbec84cd |
| volume_id | df03bb89-7383-4d21-bc67-1da89b9fa1cf |
 +--------------------+--------------------------------------------------------+
[cloudadmin@controller-1 ~(admin)]$
...

MariaDB [cinder]> update volume_attachm...

Revision history for this message
Gorka Eguileor (gorka) wrote :

Viktor, as you can see, we don't really lose the metadata, the "rw" attribute from the connection is still present after the migration, it's just in some other place (since now we can do multi-attach). So this is not really a bug.

Though there is a bug, as you showed on your last comment. A "ro" volume becomes "rw" after the migration.

Revision history for this message
Gorka Eguileor (gorka) wrote :

Created a bug for the "ro" issue: https://bugs.launchpad.net/cinder/+bug/1802155

Revision history for this message
Balazs Gibizer (balazs-gibizer) wrote :

I'm removing Nova from the bug as it seems from the comments this is a Cinder issue.

Changed in nova:
status: New → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.