Comment 0 for bug 1929699

Revision history for this message
Vladimir Grevtsev (vlgrevtsev) wrote : manila-ganesha fails to initialize CephFS driver along with cephfs charm usage

== Environment

bionic/ussuri cloud, latest stable charms. The same behaviour was observed on the focal/ussuri cloud as well.

== Problem statement

Ganesha is trying to initialize the CephFS driver, but it fails at the client eviction step.

== Analysis

The full log output of the manila-share service: https://paste.ubuntu.com/p/hgDwzYZFrZ/

In the meanwhile, the ceph-fs unit contains the following:

2021-05-26T11:36:29.675+0000 7faafe573700 1 mds.juju-551dcc-3-lxd-0 handle_command: received command from client without `tell` capability: 172.27.84.251:0/2721661976

And indeed, there is no "mds" capability defined for the client.manila-ganesha user created in Ceph:

ubuntu@OrangeBox84:~/fce-demo$ j ssh ceph-mon/0

Last login: Wed May 26 11:06:58 2021 from 172.27.84.1
ubuntu@juju-551dcc-1-lxd-0:~$ sudo ceph auth get client.manila-ganesha
exported keyring for client.manila-ganesha
[client.manila-ganesha]
 key = AQC8Ja5gbC1ZNhAAZEcfuaaJrrx/CuMbOvg89A==
 caps mon = "allow r; allow command \"osd blacklist\""
 caps osd = "allow rwx"

The "mds" capability has only been granted for the mon units, ceph-fs units, and the "client.admin" and "client.bootstrap-mds" users: http://paste.ubuntu.com/p/bcwPD9psMY/

The above issue is completely gone after issuing the manual capability grant command:

ubuntu@juju-551dcc-1-lxd-0:~$ sudo ceph auth caps client.manila-ganesha mon 'allow r; allow command "osd blacklist"' mds 'allow *' osd 'allow rwx'
updated caps for client.manila-ganesha
ubuntu@juju-551dcc-1-lxd-0:~$ sudo ceph auth get client.manila-ganesha
exported keyring for client.manila-ganesha
[client.manila-ganesha]
 key = AQC8Ja5gbC1ZNhAAZEcfuaaJrrx/CuMbOvg89A==
 caps mds = "allow *"
 caps mon = "allow r; allow command \"osd blacklist\""
 caps osd = "allow rwx"

After issuing the above, manila-share service finally was able to initialize the driver:

2021-05-26 11:44:32.807 18296 INFO manila.share.manager [req-22acddb5-b684-4172-bd90-31c4123f3676 - - - - -] Finished initialization of driver: 'CephFSDriver@juju-551dcc-5-lxd-2@cephfsnfs1'

(full log: https://paste.ubuntu.com/p/2Gb48B9yXs/)