[Pure Storage] Missing replication pod can cause driver failure on restart

Bug #2035404 reported by Simon Dodsley
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
In Progress
Undecided
Simon Dodsley

Bug Description

If synchronous replication is configured, under certain circumstances, if the replication pod defined by the parameter `pure_replication_pod_name` does not exist on the backend at driver restart. Forexample, if someone accidentally deletes the pod, this will cause the restart of the driver to fail.

The driver should be capable of correctly recreating the pod.

Error traceback example:

Sep 13 20:36:13 sn1-pool-c07-03 cinder-volume[2654547]: ERROR cinder.volume.manager Traceback (most recent call last):
Sep 13 20:36:13 sn1-pool-c07-03 cinder-volume[2654547]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/pure.py", line 2864, in _get_current_array
Sep 13 20:36:13 sn1-pool-c07-03 cinder-volume[2654547]: ERROR cinder.volume.manager pod_info = target_array.get_pod(self._replication_pod_name)
Sep 13 20:36:13 sn1-pool-c07-03 cinder-volume[2654547]: ERROR cinder.volume.manager File "/opt/stack/.local/lib/python3.10/site-packages/purestorage/purestorage.py", line 3291, in get_pod
Sep 13 20:36:13 sn1-pool-c07-03 cinder-volume[2654547]: ERROR cinder.volume.manager return self._request("GET", "pod/{0}".format(pod), kwargs)
Sep 13 20:36:13 sn1-pool-c07-03 cinder-volume[2654547]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/pure.py", line 1884, in wrapper
Sep 13 20:36:13 sn1-pool-c07-03 cinder-volume[2654547]: ERROR cinder.volume.manager ret = fn(*args, **kwargs)
Sep 13 20:36:13 sn1-pool-c07-03 cinder-volume[2654547]: ERROR cinder.volume.manager File "/opt/stack/.local/lib/python3.10/site-packages/purestorage/purestorage.py", line 202, in _request
Sep 13 20:36:13 sn1-pool-c07-03 cinder-volume[2654547]: ERROR cinder.volume.manager raise PureHTTPError(self._target, str(self._rest_version), response)
Sep 13 20:36:13 sn1-pool-c07-03 cinder-volume[2654547]: ERROR cinder.volume.manager purestorage.purestorage.PureHTTPError: PureHTTPError status code 400 returned by REST version 1.19 at 10.21.228.28: BAD REQUEST
Sep 13 20:36:13 sn1-pool-c07-03 cinder-volume[2654547]: ERROR cinder.volume.manager [{"msg": "Pod does not exist.", "ctx": "simon-cinder"}]
Sep 13 20:36:13 sn1-pool-c07-03 cinder-volume[2654547]: ERROR cinder.volume.manager
Sep 13 20:36:13 sn1-pool-c07-03 cinder-volume[2654547]: ERROR cinder.volume.manager During handling of the above exception, another exception occurred:
Sep 13 20:36:13 sn1-pool-c07-03 cinder-volume[2654547]: ERROR cinder.volume.manager
Sep 13 20:36:13 sn1-pool-c07-03 cinder-volume[2654547]: ERROR cinder.volume.manager Traceback (most recent call last):
Sep 13 20:36:13 sn1-pool-c07-03 cinder-volume[2654547]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/manager.py", line 524, in _init_host
Sep 13 20:36:13 sn1-pool-c07-03 cinder-volume[2654547]: ERROR cinder.volume.manager self.driver.do_setup(ctxt)
Sep 13 20:36:13 sn1-pool-c07-03 cinder-volume[2654547]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/pure.py", line 432, in do_setup
Sep 13 20:36:13 sn1-pool-c07-03 cinder-volume[2654547]: ERROR cinder.volume.manager self.do_setup_replication()
Sep 13 20:36:13 sn1-pool-c07-03 cinder-volume[2654547]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/pure.py", line 525, in do_setup_replication
Sep 13 20:36:13 sn1-pool-c07-03 cinder-volume[2654547]: ERROR cinder.volume.manager self._setup_replicated_pods(
Sep 13 20:36:13 sn1-pool-c07-03 cinder-volume[2654547]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/pure.py", line 200, in wrapper
Sep 13 20:36:13 sn1-pool-c07-03 cinder-volume[2654547]: ERROR cinder.volume.manager backend_name = driver._get_current_array().backend_id
Sep 13 20:36:13 sn1-pool-c07-03 cinder-volume[2654547]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/pure.py", line 2877, in _get_current_array
Sep 13 20:36:13 sn1-pool-c07-03 cinder-volume[2654547]: ERROR cinder.volume.manager raise purestorage.PureError('No functional arrays '
Sep 13 20:36:13 sn1-pool-c07-03 cinder-volume[2654547]: ERROR cinder.volume.manager purestorage.purestorage.PureError: PureError: No functional arrays available

summary: - [Pure Storage] Replication Failover with sync-repl cvolumes faiure
+ [Pure Storage] Missing replication pod can cause driver failure on
+ restart
Changed in cinder:
assignee: nobody → Simon Dodsley (simon-dodsley)
description: updated
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/cinder/+/895008

Changed in cinder:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on cinder (master)

Change abandoned by "Simon Dodsley <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/cinder/+/895008
Reason: This code is going to be included in https://review.opendev.org/c/openstack/cinder/+/895731

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.