Manila Flasblade driver does not reconnect sessions

Bug #2032681 reported by Matthew Heler
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Shared File Systems Service (Manila)
Fix Released
Medium
Simon Dodsley

Bug Description

If there is a network disconnect, or a condition which closes the session to the Flashblade, the Manila driver for FB does not reconnect unless the manila-share service is restart.

2023-08-22 16:50:22.241 665 ERROR oslo_service.periodic_task Traceback (most recent call last):
2023-08-22 16:50:22.241 665 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_service/periodic_task.py", line 216, in run_periodic_tasks
2023-08-22 16:50:22.241 665 ERROR oslo_service.periodic_task task(self, context)
2023-08-22 16:50:22.241 665 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/manila/utils.py", line 579, in wrapper
2023-08-22 16:50:22.241 665 ERROR oslo_service.periodic_task return func(self, *args, **kwargs)
2023-08-22 16:50:22.241 665 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/manila/share/manager.py", line 4029, in _report_driver_status
2023-08-22 16:50:22.241 665 ERROR oslo_service.periodic_task share_stats = self.driver.get_share_stats(refresh=True)
2023-08-22 16:50:22.241 665 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/manila/share/driver.py", line 892, in get_share_stats
2023-08-22 16:50:22.241 665 ERROR oslo_service.periodic_task self._update_share_stats()
2023-08-22 16:50:22.241 665 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/manila/share/drivers/purestorage/flashblade.py", line 153, in _update_share_stats
2023-08-22 16:50:22.241 665 ERROR oslo_service.periodic_task ) = self._get_available_capacity()
2023-08-22 16:50:22.241 665 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/manila/share/drivers/purestorage/flashblade.py", line 197, in _get_available_capacity
2023-08-22 16:50:22.241 665 ERROR oslo_service.periodic_task space = self._sys.arrays.list_arrays_space()
2023-08-22 16:50:22.241 665 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/purity_fb/purity_fb_1dot12/apis/arrays_api.py", line 713, in list_arrays_space
2023-08-22 16:50:22.241 665 ERROR oslo_service.periodic_task (data) = self.list_arrays_space_with_http_info(**kwargs)
2023-08-22 16:50:22.241 665 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/purity_fb/purity_fb_1dot12/apis/arrays_api.py", line 786, in list_arrays_space_with_http_info
2023-08-22 16:50:22.241 665 ERROR oslo_service.periodic_task return self.api_client.call_api('/1.12/arrays/space', 'GET',
2023-08-22 16:50:22.241 665 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/purity_fb/api_client/api_client.py", line 327, in call_api
2023-08-22 16:50:22.241 665 ERROR oslo_service.periodic_task return self.__call_api(resource_path, method,
2023-08-22 16:50:22.241 665 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/purity_fb/api_client/api_client.py", line 151, in __call_api
2023-08-22 16:50:22.241 665 ERROR oslo_service.periodic_task response_data = self.request(method, url,
2023-08-22 16:50:22.241 665 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/purity_fb/api_client/api_client.py", line 350, in request
2023-08-22 16:50:22.241 665 ERROR oslo_service.periodic_task return self.rest_client.GET(url,
2023-08-22 16:50:22.241 665 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/purity_fb/api_client/rest.py", line 240, in GET
2023-08-22 16:50:22.241 665 ERROR oslo_service.periodic_task return self.request("GET", url,
2023-08-22 16:50:22.241 665 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/purity_fb/api_client/rest.py", line 235, in request
2023-08-22 16:50:22.241 665 ERROR oslo_service.periodic_task raise ApiException(http_resp=r)
2023-08-22 16:50:22.241 665 ERROR oslo_service.periodic_task purity_fb.api_client.rest.ApiException: (403)
2023-08-22 16:50:22.241 665 ERROR oslo_service.periodic_task Reason:
2023-08-22 16:50:22.241 665 ERROR oslo_service.periodic_task HTTP response headers: HTTPHeaderDict({'Server': 'nginx', 'Date': 'Tue, 22 Aug 2023 16:50:22 GMT', 'Content-Type': 'application/json', 'Transfer-Encoding': 'chunked', 'Connection': 'keep-alive', 'Vary': 'Origin, Access-Control-Request-Method, Access-Control-Request-Headers', 'X-Content-Type-Options': 'nosniff', 'X-XSS-Protection': '1; mode=block', 'Cache-Control': 'no-cache, no-store, max-age=0, must-revalidate', 'Pragma': 'no-cache', 'Expires': '0', 'X-Frame-Options': 'DENY', 'x-auth-token': 'XXXXXXXX-XXXX-XXX-XXXXX-XXXXXXXXX', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains;'})
2023-08-22 16:50:22.241 665 ERROR oslo_service.periodic_task HTTP response body: {"error":[{"code":null,"context":"/api/1.12/arrays/space","message":"Access Denied"}]}
2023-08-22 16:50:22.241 665 ERROR oslo_service.periodic_task
2023-08-22 16:50:22.241 665 ERROR oslo_service.periodic_task

Tags: driver
Vida Haririan (vhariria)
tags: added: driver
Vida Haririan (vhariria)
Changed in manila:
importance: Undecided → Medium
assignee: nobody → Simon Dodsley (simon-dodsley)
Revision history for this message
Matthew Heler (mheler) wrote :

(manila-share)[manila@homelab-node003 /]$ pip show purity-fb
Name: purity-fb
Version: 1.12.3
Summary: Pure Storage FlashBlade REST 1.12 Python SDK
Home-page:
Author:
Author-email: <email address hidden>
License:
Location: /var/lib/kolla/venv/lib/python3.10/site-packages
Requires: certifi, python-dateutil, six, urllib3
Required-by:

Revision history for this message
Vida Haririan (vhariria) wrote :
Revision history for this message
Simon Dodsley (simon-dodsley) wrote :

@matthew heller - can you provide an example of a reproduceable event that causes this issue to occur?

Vida Haririan (vhariria)
Changed in manila:
status: New → Triaged
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to manila (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/manila/+/892795

Changed in manila:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to manila (master)

Reviewed: https://review.opendev.org/c/openstack/manila/+/892795
Committed: https://opendev.org/openstack/manila/commit/279b8e2861681a16ae3bb3b5c5766e331895b434
Submitter: "Zuul (22348)"
Branch: master

commit 279b8e2861681a16ae3bb3b5c5766e331895b434
Author: Simon Dodsley <email address hidden>
Date: Fri Aug 25 10:40:49 2023 -0400

    [Pure Storage] Fix connection issue on network failure.

    Retry the login process if a network issue has caused the connection to
    the FlashBlade to fail.

    Use the periodic task to update share stats to catch this.

    Closes-bug: #2032681
    Change-Id: Ic4be90aba729978e24bdd808ba1130d05aed9985

Changed in manila:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to manila (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/manila/+/899521

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on manila (master)

Change abandoned by "Goutham Pacha Ravi <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/manila/+/899521
Reason: Bad rebase

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/manila 18.0.0.0rc1

This issue was fixed in the openstack/manila 18.0.0.0rc1 release candidate.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.