Manila Flasblade driver does not reconnect sessions

Bug #2032681 reported by Matthew Heler
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Shared File Systems Service (Manila)
In Progress
Medium
Simon Dodsley

Bug Description

If there is a network disconnect, or a condition which closes the session to the Flashblade, the Manila driver for FB does not reconnect unless the manila-share service is restart.

2023-08-22 16:50:22.241 665 ERROR oslo_service.periodic_task Traceback (most recent call last):
2023-08-22 16:50:22.241 665 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_service/periodic_task.py", line 216, in run_periodic_tasks
2023-08-22 16:50:22.241 665 ERROR oslo_service.periodic_task task(self, context)
2023-08-22 16:50:22.241 665 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/manila/utils.py", line 579, in wrapper
2023-08-22 16:50:22.241 665 ERROR oslo_service.periodic_task return func(self, *args, **kwargs)
2023-08-22 16:50:22.241 665 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/manila/share/manager.py", line 4029, in _report_driver_status
2023-08-22 16:50:22.241 665 ERROR oslo_service.periodic_task share_stats = self.driver.get_share_stats(refresh=True)
2023-08-22 16:50:22.241 665 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/manila/share/driver.py", line 892, in get_share_stats
2023-08-22 16:50:22.241 665 ERROR oslo_service.periodic_task self._update_share_stats()
2023-08-22 16:50:22.241 665 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/manila/share/drivers/purestorage/flashblade.py", line 153, in _update_share_stats
2023-08-22 16:50:22.241 665 ERROR oslo_service.periodic_task ) = self._get_available_capacity()
2023-08-22 16:50:22.241 665 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/manila/share/drivers/purestorage/flashblade.py", line 197, in _get_available_capacity
2023-08-22 16:50:22.241 665 ERROR oslo_service.periodic_task space = self._sys.arrays.list_arrays_space()
2023-08-22 16:50:22.241 665 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/purity_fb/purity_fb_1dot12/apis/arrays_api.py", line 713, in list_arrays_space
2023-08-22 16:50:22.241 665 ERROR oslo_service.periodic_task (data) = self.list_arrays_space_with_http_info(**kwargs)
2023-08-22 16:50:22.241 665 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/purity_fb/purity_fb_1dot12/apis/arrays_api.py", line 786, in list_arrays_space_with_http_info
2023-08-22 16:50:22.241 665 ERROR oslo_service.periodic_task return self.api_client.call_api('/1.12/arrays/space', 'GET',
2023-08-22 16:50:22.241 665 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/purity_fb/api_client/api_client.py", line 327, in call_api
2023-08-22 16:50:22.241 665 ERROR oslo_service.periodic_task return self.__call_api(resource_path, method,
2023-08-22 16:50:22.241 665 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/purity_fb/api_client/api_client.py", line 151, in __call_api
2023-08-22 16:50:22.241 665 ERROR oslo_service.periodic_task response_data = self.request(method, url,
2023-08-22 16:50:22.241 665 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/purity_fb/api_client/api_client.py", line 350, in request
2023-08-22 16:50:22.241 665 ERROR oslo_service.periodic_task return self.rest_client.GET(url,
2023-08-22 16:50:22.241 665 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/purity_fb/api_client/rest.py", line 240, in GET
2023-08-22 16:50:22.241 665 ERROR oslo_service.periodic_task return self.request("GET", url,
2023-08-22 16:50:22.241 665 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/purity_fb/api_client/rest.py", line 235, in request
2023-08-22 16:50:22.241 665 ERROR oslo_service.periodic_task raise ApiException(http_resp=r)
2023-08-22 16:50:22.241 665 ERROR oslo_service.periodic_task purity_fb.api_client.rest.ApiException: (403)
2023-08-22 16:50:22.241 665 ERROR oslo_service.periodic_task Reason:
2023-08-22 16:50:22.241 665 ERROR oslo_service.periodic_task HTTP response headers: HTTPHeaderDict({'Server': 'nginx', 'Date': 'Tue, 22 Aug 2023 16:50:22 GMT', 'Content-Type': 'application/json', 'Transfer-Encoding': 'chunked', 'Connection': 'keep-alive', 'Vary': 'Origin, Access-Control-Request-Method, Access-Control-Request-Headers', 'X-Content-Type-Options': 'nosniff', 'X-XSS-Protection': '1; mode=block', 'Cache-Control': 'no-cache, no-store, max-age=0, must-revalidate', 'Pragma': 'no-cache', 'Expires': '0', 'X-Frame-Options': 'DENY', 'x-auth-token': 'XXXXXXXX-XXXX-XXX-XXXXX-XXXXXXXXX', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains;'})
2023-08-22 16:50:22.241 665 ERROR oslo_service.periodic_task HTTP response body: {"error":[{"code":null,"context":"/api/1.12/arrays/space","message":"Access Denied"}]}
2023-08-22 16:50:22.241 665 ERROR oslo_service.periodic_task
2023-08-22 16:50:22.241 665 ERROR oslo_service.periodic_task

Tags: driver
Vida Haririan (vhariria)
tags: added: driver
Vida Haririan (vhariria)
Changed in manila:
importance: Undecided → Medium
assignee: nobody → Simon Dodsley (simon-dodsley)
Revision history for this message
Matthew Heler (mheler) wrote :

(manila-share)[manila@homelab-node003 /]$ pip show purity-fb
Name: purity-fb
Version: 1.12.3
Summary: Pure Storage FlashBlade REST 1.12 Python SDK
Home-page:
Author:
Author-email: <email address hidden>
License:
Location: /var/lib/kolla/venv/lib/python3.10/site-packages
Requires: certifi, python-dateutil, six, urllib3
Required-by:

Revision history for this message
Vida Haririan (vhariria) wrote :
Revision history for this message
Simon Dodsley (simon-dodsley) wrote :

@matthew heller - can you provide an example of a reproduceable event that causes this issue to occur?

Vida Haririan (vhariria)
Changed in manila:
status: New → Triaged
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to manila (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/manila/+/892795

Changed in manila:
status: Triaged → In Progress
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.