OpenStack Shared File Systems Service (Manila)

Force delete does not work if share service is down

Bug #1867030 reported by Goutham Pacha Ravi
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Shared File Systems Service (Manila)
Confirmed
Low
Unassigned

Bug Description

The ability to force delete a share [1][2] was provided as an admin API [3] to remove shares from manila despite backend failures. However, this API does not account for times when the backend share service is not responding. For example:

$ manila service-list --binary manila-share
+----+--------------+------------------+---------+---------+-------+----------------------------+
| Id | Binary | Host | Zone | Status | State | Updated_at |
+----+--------------+------------------+---------+---------+-------+----------------------------+
| 8 | manila-share | ostk-train@beta | chicago | enabled | down | 2020-03-11T17:15:58.000000 |
| 9 | manila-share | ostk-train@delta | chicago | enabled | down | 2020-03-11T17:21:15.000000 |
| 10 | manila-share | ostk-train@gamma | dallas | enabled | down | 2020-03-11T17:15:49.000000 |
| 11 | manila-share | ostk-train@alpha | dallas | enabled | down | 2020-03-11T17:15:54.000000 |
+----+--------------+------------------+---------+---------+-------+----------------------------+

In this situation, if force deleting a share on a backend reporting as "down" leaves the share in "deleting" state forever.

The error in the API logs suggests that the host binary was not found. This error isn't being handled:

ERROR manila.api.middleware.fault [None req-c3c161dc-ddc3-478f-a538-6f36ecd98bb6 None None]
Caught error: Could not find binary ostk-train@delta on host manila-share.: manila.exception.HostBinaryNotFound: Could not find binary ostk-train@delta on host man
ila-share.
ERROR manila.api.middleware.fault Traceback (most recent call last):
ERROR manila.api.middleware.fault File "/opt/stack/manila/manila/api/middleware/fault.py",
 line 77, in __call__
ERROR manila.api.middleware.fault return req.get_response(self.application)
ERROR manila.api.middleware.fault File "/usr/local/lib/python3.6/dist-packages/webob/reque
st.py", line 1314, in send
ERROR manila.api.middleware.fault application, catch_exc_info=False)
ERROR manila.api.middleware.fault File "/usr/local/lib/python3.6/dist-packages/webob/reque
st.py", line 1278, in call_application
ERROR manila.api.middleware.fault app_iter = application(self.environ, start_response)
ERROR manila.api.middleware.fault File "/usr/local/lib/python3.6/dist-packages/webob/dec.p
y", line 129, in __call__
ERROR manila.api.middleware.fault resp = self.call_func(req, *args, **kw)
ERROR manila.api.middleware.fault File "/usr/local/lib/python3.6/dist-packages/webob/dec.p
y", line 193, in call_func
ERROR manila.api.middleware.fault return self.func(req, *args, **kwargs)
ERROR manila.api.middleware.fault File "/usr/local/lib/python3.6/dist-packages/oslo_middle
ware/base.py", line 130, in __call__
ERROR manila.api.middleware.fault response = req.get_response(self.application)
ERROR manila.api.middleware.fault File "/usr/local/lib/python3.6/dist-packages/webob/reque
st.py", line 1314, in send
ERROR manila.api.middleware.fault application, catch_exc_info=False)
ERROR manila.api.middleware.fault File "/usr/local/lib/python3.6/dist-packages/webob/reque
st.py", line 1278, in call_application
ERROR manila.api.middleware.fault app_iter = application(self.environ, start_response)
ERROR manila.api.middleware.fault File "/usr/local/lib/python3.6/dist-packages/webob/dec.p
y", line 143, in __call__
y", line 143, in __call__
ERROR manila.api.middleware.fault return resp(environ, start_response)
ERROR manila.api.middleware.fault File "/usr/local/lib/python3.6/dist-packages/webob/dec.p
y", line 129, in __call__
ERROR manila.api.middleware.fault resp = self.call_func(req, *args, **kw)
ERROR manila.api.middleware.fault File "/usr/local/lib/python3.6/dist-packages/webob/dec.p
y", line 193, in call_func
ERROR manila.api.middleware.fault return self.func(req, *args, **kwargs)
ERROR manila.api.middleware.fault File "/usr/local/lib/python3.6/dist-packages/keystonemid
dleware/auth_token/__init__.py", line 341, in __call__
ERROR manila.api.middleware.fault response = req.get_response(self._app)
ERROR manila.api.middleware.fault File "/usr/local/lib/python3.6/dist-packages/webob/reque
st.py", line 1314, in send
ERROR manila.api.middleware.fault application, catch_exc_info=False)
ERROR manila.api.middleware.fault File "/usr/local/lib/python3.6/dist-packages/webob/reque
st.py", line 1278, in call_application
ERROR manila.api.middleware.fault app_iter = application(self.environ, start_response)
ERROR manila.api.middleware.fault File "/usr/local/lib/python3.6/dist-packages/webob/dec.p
y", line 143, in __call__
ERROR manila.api.middleware.fault return resp(environ, start_response)
ERROR manila.api.middleware.fault File "/usr/local/lib/python3.6/dist-packages/webob/dec.p
y", line 143, in __call__
ERROR manila.api.middleware.fault return resp(environ, start_response)
ERROR manila.api.middleware.fault File "/usr/local/lib/python3.6/dist-packages/routes/midd
leware.py", line 141, in __call__
ERROR manila.api.middleware.fault response = self.app(environ, start_response)
ERROR manila.api.middleware.fault File "/usr/local/lib/python3.6/dist-packages/webob/dec.p
y", line 143, in __call__
ERROR manila.api.middleware.fault return resp(environ, start_response)
ERROR manila.api.middleware.fault File "/usr/local/lib/python3.6/dist-packages/webob/dec.p
y", line 129, in __call__
ERROR manila.api.middleware.fault resp = self.call_func(req, *args, **kw)
ERROR manila.api.middleware.fault File "/usr/local/lib/python3.6/dist-packages/webob/dec.p
y", line 193, in call_func
ERROR manila.api.middleware.fault return self.func(req, *args, **kwargs)
ERROR manila.api.middleware.fault File "/opt/stack/manila/manila/api/openstack/wsgi.py", l
ine 766, in __call__
ERROR manila.api.middleware.fault content_type, body, accept)
ERROR manila.api.middleware.fault File "/opt/stack/manila/manila/api/openstack/wsgi.py", l
ine 830, in _process_stack
ERROR manila.api.middleware.fault action_result = self.dispatch(meth, request, action_ar
gs)
ERROR manila.api.middleware.fault File "/opt/stack/manila/manila/api/openstack/wsgi.py", line 925, in dispatch
ERROR manila.api.middleware.fault return method(req=request, **action_args)
ERROR manila.api.middleware.fault File "/opt/stack/manila/manila/api/openstack/wsgi.py", line 1068, in version_select
ERROR manila.api.middleware.fault return func.func(self, *args, **kwargs)
ERROR manila.api.middleware.fault File "/opt/stack/manila/manila/api/v2/services.py", line 125, in update
ERROR manila.api.middleware.fault return self._update(req, id, body)
ERROR manila.api.middleware.fault File "/opt/stack/manila/manila/api/openstack/wsgi.py", line 1162, in wrapper
ERROR manila.api.middleware.fault return f(self, req, *args, **kwargs)
ERROR manila.api.middleware.fault File "/opt/stack/manila/manila/api/v2/services.py", line 89, in _update
ERROR manila.api.middleware.fault svc = db.service_get_by_args(context, data['host'], data['binary'])
ERROR manila.api.middleware.fault File "/opt/stack/manila/manila/db/api.py", line 119, in service_get_by_args
ERROR manila.api.middleware.fault return IMPL.service_get_by_args(context, host, binary)
ERROR manila.api.middleware.fault File "/opt/stack/manila/manila/db/sqlalchemy/api.py", line 152, in wrapper
ERROR manila.api.middleware.fault return f(*args, **kwargs)
ERROR manila.api.middleware.fault File "/opt/stack/manila/manila/db/sqlalchemy/api.py", line 462, in service_get_by_args
ERROR manila.api.middleware.fault raise exception.HostBinaryNotFound(host=host, binary=binary)
ERROR manila.api.middleware.fault manila.exception.HostBinaryNotFound: Could not find binary ostk-train@delta on host manila-share.
ERROR manila.api.middleware.fault

Victoria Martinez de la Cruz (vkmc)
Changed in manila:
importance: Undecided → Medium
status: New → Confirmed
Revision history for this message
Vida Haririan (vhariria) wrote :

Additional comments http://eavesdrop.openstack.org/meetings/manila/2020/manila.2020-03-12-15.01.log.html

Victoria Martinez de la Cruz (vkmc)
Changed in manila:
milestone: none → queens-3
milestone: queens-3 → none
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to manila (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/713745

Revision history for this message
Goutham Pacha Ravi (gouthamr) wrote :

https://review.opendev.org/713745 is not meant to be a solution, at least not yet. I put it up to start the discussion on one way to resolve this issue.

Manila's share service going down is an anomaly. We shouldn't work around such an anomaly in my opinion. Differing opinions?

Revision history for this message
Vida Haririan (vhariria) wrote :

Additional comments http://eavesdrop.openstack.org/meetings/manila/2021/manila.2021-02-11-15.00.log.html

'a "manila-manage share delete" would be a better solution, basically a tool to clean up a zombie share without ever hitting the backend'

Goutham Pacha Ravi (gouthamr)
tags: added: low-hanging-fruit
Changed in manila:
importance: Medium → Low
Revision history for this message
Vida Haririan (vhariria) wrote (last edit ):

See additional related discussions at https://meetings.opendev.org/meetings/manila/2023/manila.2023-04-13-15.00.log.html

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.