Some steps to reproduce this bug by causing a "NotFound" error.. (although the error doesn't appear to be the same in the log in the description) :
Set manila's debug logging
$ manila create nfs 1 --name testshare $ manila access-allow testshare ip 10.10.10.10
check manila logs for the export ID of the rule that was just created; for example if it is 1001; run:
sudo dbus-send --print-reply \ --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr \ org.ganesha.nfsd.exportmgr.RemoveExport uint16:1001
on the node where ganesha is running; then try to delete the share:
$ manila delete testshare
list the shares, the share must be "error_deleting"
+--------------------------------------+-----------+------+-------------+----------------+-----------+-----------------+---------------------------------------+-------------------+ | ID | Name | Size | Share Proto | Status | Is Public | Share Type Name | Host | Availability Zone | +--------------------------------------+-----------+------+-------------+----------------+-----------+-----------------+---------------------------------------+-------------------+ | b0b24d51-7785-415e-9c10-86218ad727a0 | testshare | 1 | NFS | error_deleting | False | cephfsnfstype | zorilla-cephnfs-rpm@cephfsnfs1#cephfs | manila-zone-0 | +--------------------------------------+-----------+------+-------------+----------------+-----------+-----------------+---------------------------------------+-------------------+
manila-share log shows the following: DEBUG oslo_concurrency.lockutils [None req-cc0acbf6-22af-4810-9159-64b7fc6eda98 demo None] Lock "locked_access_rules_operation_by_share_instance_97bf7384-a6ad-43ea-a2da-1b19803babf7" "released" by "manila.share.access.locked_access_rules_operation.<locals>.wrapped.<locals>.locked_operation" :: held 0.020s {{(pid=593653) inner /usr/local/lib/python3.9/site-packages/oslo_concurrency/lockutils.py:423}} INFO manila.message.api [None req-cc0acbf6-22af-4810-9159-64b7fc6eda98 demo None] Creating message record for request_id = req-cc0acbf6-22af-4810-9159-64b7fc6eda98 ERROR oslo_messaging.rpc.server [None req-cc0acbf6-22af-4810-9159-64b7fc6eda98 demo None] Exception during message handling: manila.exception.GaneshaCommandFailure: Ganesha management command failed. Command: sudo manila-rootwrap /etc/manila/rootwrap.conf dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.RemoveExport uint16:1002 Exit code: 1 Stdout: '' Stderr: 'Error org.freedesktop.DBus.Error.InvalidArgs: lookup_export failed with Export id not found\n' ERROR oslo_messaging.rpc.server Traceback (most recent call last): ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/drivers/ganesha/manager.py", line 233, in _execute ERROR oslo_messaging.rpc.server return execute(*args, **kwargs) ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/drivers/ganesha/utils.py", line 59, in __call__ ERROR oslo_messaging.rpc.server return self.execute(*args, **exkwargs) ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/utils.py", line 115, in execute ERROR oslo_messaging.rpc.server return processutils.execute(*cmd, **kwargs) ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.9/site-packages/oslo_concurrency/processutils.py", line 438, in execute ERROR oslo_messaging.rpc.server raise ProcessExecutionError(exit_code=_returncode, ERROR oslo_messaging.rpc.server oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command. ERROR oslo_messaging.rpc.server Command: sudo manila-rootwrap /etc/manila/rootwrap.conf dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.RemoveExport uint16:1002 ERROR oslo_messaging.rpc.server Exit code: 1 ERROR oslo_messaging.rpc.server Stdout: '' ERROR oslo_messaging.rpc.server Stderr: 'Error org.freedesktop.DBus.Error.InvalidArgs: lookup_export failed with Export id not found\n' ERROR oslo_messaging.rpc.server ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred: ERROR oslo_messaging.rpc.server ERROR oslo_messaging.rpc.server Traceback (most recent call last): ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.9/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/manager.py", line 236, in wrapped ERROR oslo_messaging.rpc.server return f(self, *args, **kwargs) ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/utils.py", line 579, in wrapper ERROR oslo_messaging.rpc.server return func(self, *args, **kwargs) ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/manager.py", line 3529, in delete_share_instance ERROR oslo_messaging.rpc.server self.message_api.create( ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__ ERROR oslo_messaging.rpc.server self.force_reraise() ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise ERROR oslo_messaging.rpc.server raise self.value ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/manager.py", line 3507, in delete_share_instance ERROR oslo_messaging.rpc.server self.access_helper.update_access_rules( ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/access.py", line 299, in update_access_rules ERROR oslo_messaging.rpc.server self._update_access_rules(context, share_instance_id, ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/access.py", line 336, in _update_access_rules ERROR oslo_messaging.rpc.server driver_rule_updates = self._update_rules_through_share_driver( ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/access.py", line 401, in _update_rules_through_share_driver ERROR oslo_messaging.rpc.server driver_rule_updates = self.driver.update_access( ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/drivers/cephfs/driver.py", line 546, in update_access ERROR oslo_messaging.rpc.server return self.protocol_helper.update_access( ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/drivers/ganesha/__init__.py", line 314, in update_access ERROR oslo_messaging.rpc.server self.ganesha.remove_export(share['name']) ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/drivers/ganesha/manager.py", line 516, in remove_export ERROR oslo_messaging.rpc.server self._remove_export_dbus(confdict["EXPORT"]["Export_Id"]) ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/drivers/ganesha/manager.py", line 417, in _remove_export_dbus ERROR oslo_messaging.rpc.server self._dbus_send_ganesha("RemoveExport", "uint16:%d" % xid) ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/drivers/ganesha/manager.py", line 410, in _dbus_send_ganesha ERROR oslo_messaging.rpc.server self.execute("dbus-send", "--print-reply", "--system", ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/drivers/ganesha/manager.py", line 240, in _execute ERROR oslo_messaging.rpc.server raise exception.GaneshaCommandFailure( ERROR oslo_messaging.rpc.server manila.exception.GaneshaCommandFailure: Ganesha management command failed. ERROR oslo_messaging.rpc.server Command: sudo manila-rootwrap /etc/manila/rootwrap.conf dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.RemoveExport uint16:1002 ERROR oslo_messaging.rpc.server Exit code: 1 ERROR oslo_messaging.rpc.server Stdout: '' ERROR oslo_messaging.rpc.server Stderr: 'Error org.freedesktop.DBus.Error.InvalidArgs: lookup_export failed with Export id not found\n' ERROR oslo_messaging.rpc.server
Some steps to reproduce this bug by causing a "NotFound" error.. (although the error doesn't appear to be the same in the log in the description) :
Set manila's debug logging
$ manila create nfs 1 --name testshare
$ manila access-allow testshare ip 10.10.10.10
check manila logs for the export ID of the rule that was just created; for example if it is 1001; run:
sudo dbus-send --print-reply \ org.ganesha. nfsd /org/ganesha/ nfsd/ExportMgr \ ganesha. nfsd.exportmgr. RemoveExport uint16:1001
--system --dest=
org.
on the node where ganesha is running; then try to delete the share:
$ manila delete testshare
list the shares, the share must be "error_deleting"
+------ ------- ------- ------- ------- ----+-- ------- --+---- --+---- ------- --+---- ------- -----+- ------- ---+--- ------- ------- +------ ------- ------- ------- ------- -----+- ------- ------- ----+ ------- ------- ------- ------- ----+-- ------- --+---- --+---- ------- --+---- ------- -----+- ------- ---+--- ------- ------- +------ ------- ------- ------- ------- -----+- ------- ------- ----+ 7785-415e- 9c10-86218ad727 a0 | testshare | 1 | NFS | error_deleting | False | cephfsnfstype | zorilla- cephnfs- rpm@cephfsnfs1# cephfs | manila-zone-0 | ------- ------- ------- ------- ----+-- ------- --+---- --+---- ------- --+---- ------- -----+- ------- ---+--- ------- ------- +------ ------- ------- ------- ------- -----+- ------- ------- ----+
| ID | Name | Size | Share Proto | Status | Is Public | Share Type Name | Host | Availability Zone |
+------
| b0b24d51-
+------
manila-share log shows the following: y.lockutils [None req-cc0acbf6- 22af-4810- 9159-64b7fc6eda 98 demo None] Lock "locked_ access_ rules_operation _by_share_ instance_ 97bf7384- a6ad-43ea- a2da-1b19803bab f7" "released" by "manila. share.access. locked_ access_ rules_operation .<locals> .wrapped. <locals> .locked_ operation" :: held 0.020s {{(pid=593653) inner /usr/local/ lib/python3. 9/site- packages/ oslo_concurrenc y/lockutils. py:423} } 22af-4810- 9159-64b7fc6eda 98 demo None] Creating message record for request_id = req-cc0acbf6- 22af-4810- 9159-64b7fc6eda 98 rpc.server [None req-cc0acbf6- 22af-4810- 9159-64b7fc6eda 98 demo None] Exception during message handling: manila. exception. GaneshaCommandF ailure: Ganesha management command failed. rootwrap. conf dbus-send --print-reply --system --dest= org.ganesha. nfsd /org/ganesha/ nfsd/ExportMgr org.ganesha. nfsd.exportmgr. RemoveExport uint16:1002 .DBus.Error. InvalidArgs: lookup_export failed with Export id not found\n' rpc.server Traceback (most recent call last): rpc.server File "/opt/stack/ manila/ manila/ share/drivers/ ganesha/ manager. py", line 233, in _execute rpc.server return execute(*args, **kwargs) rpc.server File "/opt/stack/ manila/ manila/ share/drivers/ ganesha/ utils.py" , line 59, in __call__ rpc.server return self.execute(*args, **exkwargs) rpc.server File "/opt/stack/ manila/ manila/ utils.py" , line 115, in execute rpc.server return processutils. execute( *cmd, **kwargs) rpc.server File "/usr/local/ lib/python3. 9/site- packages/ oslo_concurrenc y/processutils. py", line 438, in execute rpc.server raise ProcessExecutio nError( exit_code= _returncode, rpc.server oslo_concurrenc y.processutils. ProcessExecutio nError: Unexpected error while running command. rpc.server Command: sudo manila-rootwrap /etc/manila/ rootwrap. conf dbus-send --print-reply --system --dest= org.ganesha. nfsd /org/ganesha/ nfsd/ExportMgr org.ganesha. nfsd.exportmgr. RemoveExport uint16:1002 rpc.server Exit code: 1 rpc.server Stdout: '' rpc.server Stderr: 'Error org.freedesktop .DBus.Error. InvalidArgs: lookup_export failed with Export id not found\n' rpc.server rpc.server During handling of the above exception, another exception occurred: rpc.server rpc.server Traceback (most recent call last): rpc.server File "/usr/local/ lib/python3. 9/site- packages/ oslo_messaging/ rpc/server. py", line 165, in _process_incoming rpc.server res = self.dispatcher .dispatch( message) rpc.server File "/usr/local/ lib/python3. 9/site- packages/ oslo_messaging/ rpc/dispatcher. py", line 309, in dispatch rpc.server return self._do_ dispatch( endpoint, method, ctxt, args) rpc.server File "/usr/local/ lib/python3. 9/site- packages/ oslo_messaging/ rpc/dispatcher. py", line 229, in _do_dispatch rpc.server result = func(ctxt, **new_args) rpc.server File "/opt/stack/ manila/ manila/ share/manager. py", line 236, in wrapped rpc.server return f(self, *args, **kwargs) rpc.server File "/opt/stack/ manila/ manila/ utils.py" , line 579, in wrapper rpc.server return func(self, *args, **kwargs) rpc.server File "/opt/stack/ manila/ manila/ share/manager. py", line 3529, in delete_ share_instance rpc.server self.message_ api.create( rpc.server File "/usr/local/ lib/python3. 9/site- packages/ oslo_utils/ excutils. py", line 227, in __exit__ rpc.server self.force_ reraise( ) rpc.server File "/usr/local/ lib/python3. 9/site- packages/ oslo_utils/ excutils. py", line 200, in force_reraise rpc.server raise self.value rpc.server File "/opt/stack/ manila/ manila/ share/manager. py", line 3507, in delete_ share_instance rpc.server self.access_ helper. update_ access_ rules( rpc.server File "/opt/stack/ manila/ manila/ share/access. py", line 299, in update_access_rules rpc.server self._update_ access_ rules(context, share_instance_id, rpc.server File "/opt/stack/ manila/ manila/ share/access. py", line 336, in _update_ access_ rules rpc.server driver_rule_updates = self._update_ rules_through_ share_driver( rpc.server File "/opt/stack/ manila/ manila/ share/access. py", line 401, in _update_ rules_through_ share_driver rpc.server driver_rule_updates = self.driver. update_ access( rpc.server File "/opt/stack/ manila/ manila/ share/drivers/ cephfs/ driver. py", line 546, in update_access rpc.server return self.protocol_ helper. update_ access( rpc.server File "/opt/stack/ manila/ manila/ share/drivers/ ganesha/ __init_ _.py", line 314, in update_access rpc.server self.ganesha. remove_ export( share[' name']) rpc.server File "/opt/stack/ manila/ manila/ share/drivers/ ganesha/ manager. py", line 516, in remove_export rpc.server self._remove_ export_ dbus(confdict[ "EXPORT" ]["Export_ Id"]) rpc.server File "/opt/stack/ manila/ manila/ share/drivers/ ganesha/ manager. py", line 417, in _remove_export_dbus rpc.server self._dbus_ send_ganesha( "RemoveExport" , "uint16:%d" % xid) rpc.server File "/opt/stack/ manila/ manila/ share/drivers/ ganesha/ manager. py", line 410, in _dbus_send_ganesha rpc.server self.execute( "dbus-send" , "--print-reply", "--system", rpc.server File "/opt/stack/ manila/ manila/ share/drivers/ ganesha/ manager. py", line 240, in _execute rpc.server raise exception. GaneshaCommandF ailure( rpc.server manila. exception. GaneshaCommandF ailure: Ganesha management command failed. rpc.server Command: sudo manila-rootwrap /etc/manila/ rootwrap. conf dbus-send --print-reply --system --dest= org.ganesha. nfsd /org/ganesha/ nfsd/ExportMgr org.ganesha. nfsd.exportmgr. RemoveExport uint16:1002 rpc.server Exit code: 1 rpc.server Stdout: '' rpc.server Stderr: 'Error org.freedesktop .DBus.Error. InvalidArgs: lookup_export failed with Export id not found\n' rpc.server
DEBUG oslo_concurrenc
INFO manila.message.api [None req-cc0acbf6-
ERROR oslo_messaging.
Command: sudo manila-rootwrap /etc/manila/
Exit code: 1
Stdout: ''
Stderr: 'Error org.freedesktop
ERROR oslo_messaging.
ERROR oslo_messaging.
ERROR oslo_messaging.
ERROR oslo_messaging.
ERROR oslo_messaging.
ERROR oslo_messaging.
ERROR oslo_messaging.
ERROR oslo_messaging.
ERROR oslo_messaging.
ERROR oslo_messaging.
ERROR oslo_messaging.
ERROR oslo_messaging.
ERROR oslo_messaging.
ERROR oslo_messaging.
ERROR oslo_messaging.
ERROR oslo_messaging.
ERROR oslo_messaging.
ERROR oslo_messaging.
ERROR oslo_messaging.
ERROR oslo_messaging.
ERROR oslo_messaging.
ERROR oslo_messaging.
ERROR oslo_messaging.
ERROR oslo_messaging.
ERROR oslo_messaging.
ERROR oslo_messaging.
ERROR oslo_messaging.
ERROR oslo_messaging.
ERROR oslo_messaging.
ERROR oslo_messaging.
ERROR oslo_messaging.
ERROR oslo_messaging.
ERROR oslo_messaging.
ERROR oslo_messaging.
ERROR oslo_messaging.
ERROR oslo_messaging.
ERROR oslo_messaging.
ERROR oslo_messaging.
ERROR oslo_messaging.
ERROR oslo_messaging.
ERROR oslo_messaging.
ERROR oslo_messaging.
ERROR oslo_messaging.
ERROR oslo_messaging.
ERROR oslo_messaging.
ERROR oslo_messaging.
ERROR oslo_messaging.
ERROR oslo_messaging.
ERROR oslo_messaging.
ERROR oslo_messaging.
ERROR oslo_messaging.
ERROR oslo_messaging.
ERROR oslo_messaging.
ERROR oslo_messaging.
ERROR oslo_messaging.
ERROR oslo_messaging.
ERROR oslo_messaging.
ERROR oslo_messaging.
ERROR oslo_messaging.
ERROR oslo_messaging.