ZFSonLinux driver fails when removing access rule

Bug #1869340 reported by Jacolex
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Shared File Systems Service (Manila)
Triaged
Undecided
Unassigned

Bug Description

When removing access rule, driver throws error:

2020-03-26 19:35:36.482 194 ERROR oslo_messaging.rpc.server [req-934c39da-b94b-4ccd-95f7-81b25759ad92 13622fba11a84996a70019d553060c8c 8a937ac1fc3e41c6b95a83961001e06a - - -] Exception during message handling: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
Command: sudo exportfs -u 172.16.180.5:/zfspool01/manila_share_ce6e9292_76c7_404e_b8c9_ab42062e1308
Exit code: 1
Stdout: ''
Stderr: "exportfs: Could not find '172.16.180.5:/zfspool01/manila_share_ce6e9292_76c7_404e_b8c9_ab42062e1308' to unexport.\n"
2020-03-26 19:35:36.482 194 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2020-03-26 19:35:36.482 194 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2020-03-26 19:35:36.482 194 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2020-03-26 19:35:36.482 194 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 274, in dispatch
2020-03-26 19:35:36.482 194 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2020-03-26 19:35:36.482 194 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
2020-03-26 19:35:36.482 194 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2020-03-26 19:35:36.482 194 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/manila/share/manager.py", line 187, in wrapped
2020-03-26 19:35:36.482 194 ERROR oslo_messaging.rpc.server return f(self, *args, **kwargs)
2020-03-26 19:35:36.482 194 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/manila/utils.py", line 568, in wrapper
2020-03-26 19:35:36.482 194 ERROR oslo_messaging.rpc.server return func(self, *args, **kwargs)
2020-03-26 19:35:36.482 194 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/manila/share/manager.py", line 3554, in update_access
2020-03-26 19:35:36.482 194 ERROR oslo_messaging.rpc.server share_server=share_server)
2020-03-26 19:35:36.482 194 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/manila/share/access.py", line 283, in update_access_rules
2020-03-26 19:35:36.482 194 ERROR oslo_messaging.rpc.server share_server=share_server)
2020-03-26 19:35:36.482 194 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/manila/share/access.py", line 322, in _update_access_rules
2020-03-26 19:35:36.482 194 ERROR oslo_messaging.rpc.server share_server)
2020-03-26 19:35:36.482 194 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/manila/share/access.py", line 390, in _update_rules_through_share_driver
2020-03-26 19:35:36.482 194 ERROR oslo_messaging.rpc.server share_server=share_server
2020-03-26 19:35:36.482 194 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/manila/share/drivers/zfsonlinux/driver.py", line 138, in wrap
2020-03-26 19:35:36.482 194 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kwargs)
2020-03-26 19:35:36.482 194 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/manila/share/drivers/zfsonlinux/driver.py", line 687, in update_access
2020-03-26 19:35:36.482 194 ERROR oslo_messaging.rpc.server executor=executor)
2020-03-26 19:35:36.482 194 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/manila/share/drivers/zfsonlinux/utils.py", line 48, in wrapped_func
2020-03-26 19:35:36.482 194 ERROR oslo_messaging.rpc.server return source_func(self, *args, **kwargs)
2020-03-26 19:35:36.482 194 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py", line 328, in inner
2020-03-26 19:35:36.482 194 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
2020-03-26 19:35:36.482 194 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/manila/share/drivers/zfsonlinux/utils.py", line 46, in source_func
2020-03-26 19:35:36.482 194 ERROR oslo_messaging.rpc.server return f(self, *args, **kwargs)
2020-03-26 19:35:36.482 194 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/manila/share/drivers/zfsonlinux/utils.py", line 319, in update_access
2020-03-26 19:35:36.482 194 ERROR oslo_messaging.rpc.server executor=executor,
2020-03-26 19:35:36.482 194 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/manila/share/drivers/zfsonlinux/utils.py", line 97, in execute
2020-03-26 19:35:36.482 194 ERROR oslo_messaging.rpc.server return executor(*cmd, **kwargs)
2020-03-26 19:35:36.482 194 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/manila/share/drivers/zfsonlinux/utils.py", line 97, in execute
2020-03-26 19:35:36.482 194 ERROR oslo_messaging.rpc.server return executor(*cmd, **kwargs)
2020-03-26 19:35:36.482 194 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/manila/share/drivers/ganesha/utils.py", line 79, in __call__
2020-03-26 19:35:36.482 194 ERROR oslo_messaging.rpc.server ret = processutils.ssh_execute(ssh, cmd, **kwargs)
2020-03-26 19:35:36.482 194 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_concurrency/processutils.py", line 570, in ssh_execute
2020-03-26 19:35:36.482 194 ERROR oslo_messaging.rpc.server cmd=sanitized_cmd)
2020-03-26 19:35:36.482 194 ERROR oslo_messaging.rpc.server oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
2020-03-26 19:35:36.482 194 ERROR oslo_messaging.rpc.server Command: sudo exportfs -u 172.16.180.5:/zfspool01/manila_share_ce6e9292_76c7_404e_b8c9_ab42062e1308
2020-03-26 19:35:36.482 194 ERROR oslo_messaging.rpc.server Exit code: 1
2020-03-26 19:35:36.482 194 ERROR oslo_messaging.rpc.server Stdout: ''
2020-03-26 19:35:36.482 194 ERROR oslo_messaging.rpc.server Stderr: "exportfs: Could not find '172.16.180.5:/zfspool01/manila_share_ce6e9292_76c7_404e_b8c9_ab42062e1308' to unexport.\n"
2020-03-26 19:35:36.482 194 ERROR oslo_messaging.rpc.server

It seems that this part of code is responsible for that:

        # NOTE(vponomaryov): Setting of ZFS share options does not remove rules
        # that were added and then removed. So, remove them explicitly.
        if delete_rules and access_rules:
            mountpoint = self.get_zfs_option(dataset_name, 'mountpoint')
            for rule in delete_rules:
                if rule['access_type'].lower() != 'ip':
                    continue
                access_to = self._get_parsed_access_to(rule['access_to'])
                export_location = access_to + ':' + mountpoint
                self.execute(
                    'sudo', 'exportfs', '-u', export_location,
                    executor=executor,
                )

Revision history for this message
Goutham Pacha Ravi (gouthamr) wrote :

Hi,

A few questions for you:
What release version of OpenStack Manila are you running? What distribution?

At the outset, the driver shouldn't be erring on this path when attempting to delete a non-existing rule. But I'm confused as to why this rule was non-existing. We haven't seen this issue in our testing, so it'd be useful if you have the complete manila-share log that you can provide so we can backtrack and see what happened with the rule for your client (172.16.180.5).

Thanks!

Revision history for this message
Jacolex (jacolex) wrote :

Hello!
It is Train Release.
I see that problem is with handle multiple access rules.
When there is only one access rule - no problem. Error is throwing when multple access rules are applied to a share and trying to remove one rule.
So what I see on the zfs server.

1. Create share
zfs list -o sharenfs,mountpoint /zfspool01/manila_share_6f6bba49_a601_4c5b_a60c_1545af60faa3
SHARENFS MOUNTPOINT
off /zfspool01/manila_share_6f6bba49_a601_4c5b_a60c_1545af60faa3

2. Add first rule.
zfs list -o sharenfs,mountpoint /zfspool01/manila_share_6f6bba49_a601_4c5b_a60c_1545af60faa3
SHARENFS MOUNTPOINT
rw=10.10.10.1,no_root_squash /zfspool01/manila_share_6f6bba49_a601_4c5b_a60c_1545af60faa3

3. Add second rule.
zfs list -o sharenfs,mountpoint /zfspool01/manila_share_6f6bba49_a601_4c5b_a60c_1545af60faa3
SHARENFS MOUNTPOINT
rw=10.10.10.2:10.10.10.1,no_root_squash /zfspool01/manila_share_6f6bba49_a601_4c5b_a60c_1545af60faa3

4. Remove second rule. Error throwed but the rule is removed.

Stderr: "exportfs: Could not find '10.10.10.2:/zfspool01/manila_share_6f6bba49_a601_4c5b_a60c_1545af60faa3' to unexport.\n"

zfs list -o sharenfs,mountpoint /zfspool01/manila_share_6f6bba49_a601_4c5b_a60c_1545af60faa3
SHARENFS MOUNTPOINT
rw=10.10.10.1,no_root_squash /zfspool01/manila_share_6f6bba49_a601_4c5b_a60c_1545af60faa3

As you can see, there is probably problem with handling access rules on zfs, because driver is looking for '10.10.10.2:/zfspool01/manila_share_6f6bba49_a601_4c5b_a60c_1545af60faa3', but there was rw=10.10.10.2:10.10.10.1,no_root_squash /zfspool0/manila_share_6f6bba49_a601_4c5b_a60c_1545af60faa3

Version of zfs storage os:
cat /etc/system-release
CentOS Linux release 7.6.1810 (Core)

Vida Haririan (vhariria)
Changed in manila:
status: New → Triaged
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.