Cinder

cinder retype fails with "no host supplied" for SolidFire driver

  1. Series yoga
  2. Bug #2018001
Bug #2018001 reported by Jonathan Williams
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
Status tracked in Zed
Yoga
New
Low
Felipe Rodrigues
Zed
New
Low
Felipe Rodrigues

Bug Description

When I perform a cinder retype from a Solidfire volume type to a ONTAP volume type, I see
a "No host supplied" error.

Command run: cinder retype --migration-policy on-demand solidfire_vol ontap-nfs

Releases affected: Yoga, Zed

Environments:

1. Running packstack / kolla-ansible deployments

2. A Stein controller (using a very similar cinder.conf and connected to the same storage arrays)
does not have this issue. cinder retype completes without issue.

3. My volume type only specifies the volume_backend_name. There are no extra_specs

4. My SolidFire volume backend is pretty simple:

[solidfire]
volume_driver = cinder.volume.drivers.solidfire.SolidFireDriver
san_ip = xxxx
san_login = xxxx
san_password = xxxx
sf_svip=xxxxx
sf_account_prefix = ''
volume_backend_name=solidfire

In the cinder volume.log the exception I see is:

2023-04-25 13:37:19.428 32 DEBUG cinder.volume.manager [req-88faa23e-b5b8-4636-b1fe-003291e2594a req-7db290b9-f0b6-4594-be53-9c5276904af7 040e9a0e099846d898ec2675de2054c8 da32384368a4472a9c4
a48f1fdce8057 - - - -] Issue driver.migrate_volume. migrate_volume /var/lib/kolla/venv/lib/python3.10/site-packages/cinder/volume/manager.py:2671
2023-04-25 13:37:19.429 32 INFO cinder.volume.drivers.solidfire [req-88faa23e-b5b8-4636-b1fe-003291e2594a req-7db290b9-f0b6-4594-be53-9c5276904af7 040e9a0e099846d898ec2675de2054c8 da32384368
a4472a9c4a48f1fdce8057 - - - -] Migrate volume 6b2fc7f8-d0be-4641-8d5a-fd7b558b89aa to xxxx@ontap-nfs#xxxx:/cinder_ha_vol.
2023-04-25 13:37:19.431 32 INFO cinder.volume.drivers.solidfire [req-88faa23e-b5b8-4636-b1fe-003291e2594a req-7db290b9-f0b6-4594-be53-9c5276904af7 040e9a0e099846d898ec2675de2054c8 da32384368
a4472a9c4a48f1fdce8057 - - - -] Source and destination clusters are different. A cluster migration will be performed.
2023-04-25 13:37:19.432 32 DEBUG cinder.volume.drivers.solidfire [req-88faa23e-b5b8-4636-b1fe-003291e2594a req-7db290b9-f0b6-4594-be53-9c5276904af7 040e9a0e099846d898ec2675de2054c8 da3238436
8a4472a9c4a48f1fdce8057 - - - -] Active cluster: [solidfire_array], Destination: [] migrate_volume /var/lib/kolla/venv/lib/python3.10/site-packages/cinder/volume/drivers/solidfire.py:2445
2023-04-25 13:37:19.432 32 DEBUG cinder.volume.drivers.solidfire [req-88faa23e-b5b8-4636-b1fe-003291e2594a req-7db290b9-f0b6-4594-be53-9c5276904af7 040e9a0e099846d898ec2675de2054c8 da3238436
8a4472a9c4a48f1fdce8057 - - - -] Start migrating volume [volume-6b2fc7f8-d0be-4641-8d5a-fd7b558b89aa] to cluster [xxxx@ontap-nfs#xxxx:/cinder_ha_vol] _do_intercluster_volume_migration /var/lib/kolla/venv/lib/python3.10/site-packages/cinder/volume/drivers/solidfire.py:2376
..
2023-04-25 13:37:19.432 32 DEBUG cinder.volume.drivers.solidfire [req-88faa23e-b5b8-4636-b1fe-003291e2594a req-7db290b9-f0b6-4594-be53-9c5276904af7 040e9a0e099846d898ec2675de2054c8 da3238436
8a4472a9c4a48f1fdce8057 - - - -] Destination cluster mvip is [] _do_intercluster_volume_migration /var/lib/kolla/venv/lib/python3.10/site-packages/cinder/volume/drivers/solidfire.py:2381
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server [req-88faa23e-b5b8-4636-b1fe-003291e2594a req-7db290b9-f0b6-4594-be53-9c5276904af7 040e9a0e099846d898ec2675de2054c8 da32384368a4472
a9c4a48f1fdce8057 - - - -] Exception during message handling: requests.exceptions.InvalidURL: Invalid URL 'https://:443/json-rpc/1.0/': No host supplied
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/cinder/volume/manager.py", line 3143, in retype
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception():
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server self.force_reraise()
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server raise self.value
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/cinder/volume/manager.py", line 3140, in retype
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server self.migrate_volume(context, volume, host,
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/cinder/volume/manager.py", line 2693, in migrate_volume
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception():
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server self.force_reraise()
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server raise self.value
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/cinder/volume/manager.py", line 2672, in migrate_volume
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server moved, model_update = self.driver.migrate_volume(ctxt,
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/cinder/volume/drivers/solidfire.py", line 2450, in migrate_volume
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server updates = self._do_intercluster_volume_migration(volume, host,
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/cinder/volume/drivers/solidfire.py", line 2383, in _do_intercluster_volume
_migration
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server dst_cluster_ref = self._create_cluster_reference(dst_endpoint)
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/cinder/volume/drivers/solidfire.py", line 544, in _create_cluster_referenc
e
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server cluster_info = (self._issue_api_request(
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/cinder/volume/drivers/solidfire.py", line 182, in func_retry
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/cinder/volume/drivers/solidfire.py", line 678, in _issue_api_request
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server req = requests.post(url,
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/requests/api.py", line 115, in post
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server return request("post", url, data=data, json=json, **kwargs)
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/requests/api.py", line 59, in request
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server return session.request(method=method, url=url, **kwargs)
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/requests/sessions.py", line 573, in request
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server prep = self.prepare_request(req)
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/requests/sessions.py", line 484, in prepare_request
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server p.prepare(
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/requests/models.py", line 368, in prepare
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server self.prepare_url(url, params)
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/requests/models.py", line 445, in prepare_url
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server raise InvalidURL(f"Invalid URL {url!r}: No host supplied")
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server requests.exceptions.InvalidURL: Invalid URL 'https://:443/json-rpc/1.0/': No host supplied
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server

The issue appears to be this in the workflow:

Destination cluster mvip is [] _do_intercluster_volume_migration

Looking at https://github.com/openstack/cinder/blob/master/cinder/volume/drivers/solidfire.py, is the
SF driver only looking for destination "solidfire" addresses? And the retype is failing because
it's a ONTAP destination?

Sofia Enriquez (lsofia-enriquez)
tags: added: drivers solidfire
Sofia Enriquez (lsofia-enriquez)
tags: added: retype
Revision history for this message
Jonathan Williams (jon-at-netapp) wrote :

It looks like there are two types of migrations available:

1. storage assisted migration

2. host assisted migration

As I understand it, during the Victoria release storage assisted migration was added to the Solidfire driver.
I believe that is what the Solidfire driver is currently trying to do. Perform a storage assisted
migration between two different storage vendor's backends (Solidfire to ONTAP). I believe what it should
be doing is performing a host assisted migration which would be successful.

cinder migrate has the "--force-host-copy" option. However cinder retype does not have an option like this.
Unless there is a way to force a host assisted migration with cinder retype, it looks like any attempt to
retype from a Solidfire volume type to a volume type used by another storage vendor's backend will fail.

Revision history for this message
Jonathan Williams (jon-at-netapp) wrote :

I think we've figured this out. In Victoria, storage assisted migration was added to Solidfire:
https://review.opendev.org/c/openstack/cinder/+/746941. So prior to Victoria, cinder retype
was a host assisted migration using the generic cinder driver.

After Victoria, it's now handled by the Solidfire driver as a storage assisted migration. And
it looks like cinder retype (when run as a storage assisted migration) on works for volume migrations
between two solidfire clusters: https://github.com/openstack/cinder/commit/b1d6210d454c05aeca2fbe944c6b6e82c508372f#diff-a62544f589ca8b9ea1f17078ec5fb30aa687f3dde2b8125b1928cf244832f6b9R2425.

To workaround this, you can't use a volume type that has the "volume_backend_name" property set.
Essentially you can't refer to the backend at all it seems. If you instead create a volume type
with the "vendor_name='NetApp'" property, it looks like it reverts to a host assisted migration and the cinder retype works.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.