OpenStack Shared File Systems Service (Manila)

Unable to delete CephFS manila share if nfs-ganesha export rule doesn't exist

Bug #2035572 reported by Goutham Pacha Ravi
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Shared File Systems Service (Manila)
Triaged
Medium
Goutham Pacha Ravi
OpenStack Shared File Systems Service (Manila) caracal-2

Bug Description

User report:

We have some manila shares which are stuck in 'error_deleting' state and cannot be removed unless their status is reset to 'available'. One thing in common with these shares is that the instances associated with all these shares (share_instance_id) do not exist in the environment. Even so, the 'access_rules_status' is still 'active'.
After the status is reset to active, the 'access_rules_status' gets set to 'error' state after which we are able to delete it.

For example, share ID: 67d12f8b-5bb8-4486-83ce-b3a1b4030a9c

$ manila show 67d12f8b-5bb8-4486-83ce-b3a1b4030a9c
+---------------------------------------+---------------------------------------------------------------------------+
| Property | Value |
+---------------------------------------+---------------------------------------------------------------------------+
| id | 67d12f8b-5bb8-4486-83ce-b3a1b4030a9c |
| size | 5 |
| availability_zone | nova |
| created_at | 2022-12-06T12:31:40.000000 |
| status | error_deleting |
| name | pvc-ce7985c3-c3d0-493b-aa70-37253922b057 |
| description | provisioned-by=manila.csi.openstack.org |
| project_id | 7b4c0133359d44a4ba633f7db8357524 |
| snapshot_id | None |
| share_network_id | None |
| share_proto | NFS |
| metadata | {'manila.csi.openstack.org/cluster': 'cicd-f58nc'} |
| share_type | f6b11e95-1072-44a6-8d89-b660993a505e |
| is_public | False |
| snapshot_support | True |
| task_state | None |
| share_type_name | ceph |
| access_rules_status | active |
| replication_type | None |
| has_replicas | False |
| user_id | 0a3daf14f6aecb825dbf414fb7eac3504d538f3ad0e53a2d0df84ce2e4ac3af9 |
| create_share_from_snapshot_support | False |
| revert_to_snapshot_support | False |
| share_group_id | None |
| source_share_group_snapshot_member_id | None |
| mount_snapshot_support | False |
| share_server_id | None |
| host | hostgroup@cephfs#cephfs |
| export_locations | |
| | id = a6b373a8-4c77-42e1-ad03-7dbbab48782a |
| | path = 172.16.33.4:/volumes/_nogroup/a257d35b-d254-4dff-9dac-657bcb1625fc |
| | preferred = False |
| | share_instance_id = a257d35b-d254-4dff-9dac-657bcb1625fc |
| | is_admin_only = False |
+---------------------------------------+---------------------------------------------------------------------------+

$ manila reset-state --state available 67d12f8b-5bb8-4486-83ce-b3a1b4030a9c

$ manila show 67d12f8b-5bb8-4486-83ce-b3a1b4030a9c
+---------------------------------------+---------------------------------------------------------------------------+
| Property | Value |
+---------------------------------------+---------------------------------------------------------------------------+
| id | 67d12f8b-5bb8-4486-83ce-b3a1b4030a9c |
| size | 5 |
| availability_zone | nova |
| created_at | 2022-12-06T12:31:40.000000 |
| status | available |
| name | pvc-ce7985c3-c3d0-493b-aa70-37253922b057 |
| description | provisioned-by=manila.csi.openstack.org |
| project_id | 7b4c0133359d44a4ba633f7db8357524 |
| snapshot_id | None |
| share_network_id | None |
| share_proto | NFS |
| metadata | {'manila.csi.openstack.org/cluster': 'cicd-f58nc'} |
| share_type | f6b11e95-1072-44a6-8d89-b660993a505e |
| is_public | False |
| snapshot_support | True |
| task_state | None |
| share_type_name | ceph |
| access_rules_status | error |
| replication_type | None |
| has_replicas | False |
| user_id | 0a3daf14f6aecb825dbf414fb7eac3504d538f3ad0e53a2d0df84ce2e4ac3af9 |
| create_share_from_snapshot_support | False |
| revert_to_snapshot_support | False |
| share_group_id | None |
| source_share_group_snapshot_member_id | None |
| mount_snapshot_support | False |
| share_server_id | None |
| host | hostgroup@cephfs#cephfs |
| export_locations | |
| | id = a6b373a8-4c77-42e1-ad03-7dbbab48782a |
| | path = 172.16.33.4:/volumes/_nogroup/a257d35b-d254-4dff-9dac-657bcb1625fc |
| | preferred = False |
| | share_instance_id = a257d35b-d254-4dff-9dac-657bcb1625fc |
| | is_admin_only = False |
+---------------------------------------+---------------------------------------------------------------------------+

Some of the logs indicate that manila is not able to update the rules of the shares which we are trying to delete where the share_instance_id for that share does not exist in the environment.

For example:

Share ID: 5b5d2a38-2a46-4d81-8690-ba9cc94629a6
Request ID: req-d7e38b71-85db-444c-8689-f0d6298ffbce

~~~
manila-api.log.2.gz:2023-01-30 10:19:00.712 21 INFO manila.api.openstack.wsgi [req-d7e38b71-85db-444c-8689-f0d6298ffbce 0a3daf14f6aecb825dbf414fb7eac3504d538f3ad0e53a2d0df84ce2e4ac3af9 7b4c0133359d44a4ba633f7db8357524 - - -] DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13786/v2/7b4c0133359d44a4ba633f7db8357524/shares/5b5d2a38-2a46-4d81-8690-ba9cc94629a6
manila-api.log.2.gz:2023-01-30 10:19:00.713 21 INFO manila.api.v1.shares [req-d7e38b71-85db-444c-8689-f0d6298ffbce 0a3daf14f6aecb825dbf414fb7eac3504d538f3ad0e53a2d0df84ce2e4ac3af9 7b4c0133359d44a4ba633f7db8357524 - - -] Delete share with id: 5b5d2a38-2a46-4d81-8690-ba9cc94629a6

manila-share.log:2023-01-30 10:19:01.742 42 DEBUG manila.share.access [req-d7e38b71-85db-444c-8689-f0d6298ffbce 0a3daf14f6aecb825dbf414fb7eac3504d538f3ad0e53a2d0df84ce2e4ac3af9 7b4c0133359d44a4ba633f7db8357524 - - -] Updating access rules for share instance cf864d1b-8d5f-4eea-96a0-454ad90ab8b3 belonging to share 5b5d2a38-2a46-4d81-8690-ba9cc94629a6. update_access_rules /usr/lib/python3.6/site-packages/manila/share/access.py:281
manila-share.log:2023-01-30 10:19:02.120 42 DEBUG oslo_concurrency.processutils [req-d7e38b71-85db-444c-8689-f0d6298ffbce 0a3daf14f6aecb825dbf414fb7eac3504d538f3ad0e53a2d0df84ce2e4ac3af9 7b4c0133359d44a4ba633f7db8357524 - - -] Running cmd (subprocess): dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.RemoveExport uint16:11406 execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:372
manila-share.log:2023-01-30 10:19:02.149 42 DEBUG oslo_concurrency.processutils [req-d7e38b71-85db-444c-8689-f0d6298ffbce 0a3daf14f6aecb825dbf414fb7eac3504d538f3ad0e53a2d0df84ce2e4ac3af9 7b4c0133359d44a4ba633f7db8357524 - - -] CMD "dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.RemoveExport uint16:11406" returned: 1 in 0.029s execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:409
manila-share.log:2023-01-30 10:19:02.149 42 DEBUG oslo_concurrency.processutils [req-d7e38b71-85db-444c-8689-f0d6298ffbce 0a3daf14f6aecb825dbf414fb7eac3504d538f3ad0e53a2d0df84ce2e4ac3af9 7b4c0133359d44a4ba633f7db8357524 - - -] 'dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.RemoveExport uint16:11406' failed. Not Retrying. execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:457
manila-share.log:2023-01-30 10:19:02.150 42 ERROR manila.share.drivers.ganesha.manager [req-d7e38b71-85db-444c-8689-f0d6298ffbce 0a3daf14f6aecb825dbf414fb7eac3504d538f3ad0e53a2d0df84ce2e4ac3af9 7b4c0133359d44a4ba633f7db8357524 - - -] Error while executing management command on Ganesha node <no name>: dbus call exportmgr.RemoveExport.: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
~~~

Tags: cephfs ganesha
Goutham Pacha Ravi (gouthamr)
Changed in manila:
assignee: nobody → Goutham Pacha Ravi (gouthamr)
Vida Haririan (vhariria)
tags: added: ganesha
tags: added: cephfs
Vida Haririan (vhariria)
Changed in manila:
status: New → Triaged
status: Triaged → New
Revision history for this message
Goutham Pacha Ravi (gouthamr) wrote :
Download full text (8.4 KiB)

Some steps to reproduce this bug by causing a "NotFound" error.. (although the error doesn't appear to be the same in the log in the description) :

Set manila's debug logging

$ manila create nfs 1 --name testshare
$ manila access-allow testshare ip 10.10.10.10

check manila logs for the export ID of the rule that was just created; for example if it is 1001; run:

 sudo dbus-send --print-reply \
    --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr \
    org.ganesha.nfsd.exportmgr.RemoveExport uint16:1001

on the node where ganesha is running; then try to delete the share:

$ manila delete testshare

list the shares, the share must be "error_deleting"

+--------------------------------------+-----------+------+-------------+----------------+-----------+-----------------+---------------------------------------+-------------------+
| ID | Name | Size | Share Proto | Status | Is Public | Share Type Name | Host | Availability Zone |
 +--------------------------------------+-----------+------+-------------+----------------+-----------+-----------------+---------------------------------------+-------------------+
| b0b24d51-7785-415e-9c10-86218ad727a0 | testshare | 1 | NFS | error_deleting | False | cephfsnfstype | zorilla-cephnfs-rpm@cephfsnfs1#cephfs | manila-zone-0 |
 +--------------------------------------+-----------+------+-------------+----------------+-----------+-----------------+---------------------------------------+-------------------+

manila-share log shows the following:
DEBUG oslo_concurrency.lockutils [None req-cc0acbf6-22af-4810-9159-64b7fc6eda98 demo None] Lock "locked_access_rules_operation_by_share_instance_97bf7384-a6ad-43ea-a2da-1b19803babf7" "released" by "manila.share.access.locked_access_rules_operation.<locals>.wrapped.<locals>.locked_operation" :: held 0.020s {{(pid=593653) inner /usr/local/lib/python3.9/site-packages/oslo_concurrency/lockutils.py:423}}
INFO manila.message.api [None req-cc0acbf6-22af-4810-9159-64b7fc6eda98 demo None] Creating message record for request_id = req-cc0acbf6-22af-4810-9159-64b7fc6eda98
ERROR oslo_messaging.rpc.server [None req-cc0acbf6-22af-4810-9159-64b7fc6eda98 demo None] Exception during message handling: manila.exception.GaneshaCommandFailure: Ganesha management command failed.
Command: sudo manila-rootwrap /etc/manila/rootwrap.conf dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.RemoveExport uint16:1002
Exit code: 1
Stdout: ''
Stderr: 'Error org.freedesktop.DBus.Error.InvalidArgs: lookup_export failed with Export id not found\n'
ERROR oslo_messaging.rpc.server Traceback (most recent call last):
ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/drivers/ganesha/manager.py", line 233, in _execute
ERROR oslo_messaging.rpc.server return execute(*args, **kwargs)
ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/drivers/ganesha/utils.py", line 59, in __call__
ERROR oslo_messaging.rpc.server return self.execute(*args, **exkwargs)
ERROR oslo_messaging.rpc.server File "...

Read more...

Vida Haririan (vhariria)
Changed in manila:
status: New → Triaged
importance: Undecided → Medium
milestone: none → caracal-2
Revision history for this message
Vida Haririan (vhariria) wrote :

Adding workaround steps provided by Goutham Pacha Ravi
    "admins can reset the state of the share, and re-attempt share deletion and it'll work.."

Revision history for this message
Vida Haririan (vhariria) wrote :

See additional related discussions at https://meetings.opendev.org/meetings/manila/2023/manila.2023-10-05-15.00

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.