hacluster for manila-ganesha get stuck with "Resource: res_ganesha_xxx_vip not running"

I have attached the log of pacemaker as well.
In this log, I could see that the pacemaker resource is failing with the following error.

Could not issue start for res_manila_share_manila_share: Unit manila-share.service is masked.

This symptom is seen every time I deploy this environment.
BTW, since Vault is used as TLS termination for API endpoints, the reproducible steps are

1. Deploy OpenStack with Vault
2. Unlock and activate Vault. Before this, hacluster-manila-ganesha is all fine
3. After the relation with Vault is completed, hacluster-manila-ganesha gets stuck. I have waited for more than 30 minutes, but it was still stuck.

Hi Yoshi, I will try to reproduce this issue.

Yoshi, I was trying to reproduce without success, the specific bundle I'm using can be found at https://paste.ubuntu.com/p/q6G3xhKRDd/

In the latest stable version (21.10) landed this commit https://opendev.org/openstack/charm-manila-ganesha/commit/b1cb18391ce8f913c5c64de1a8df88ae0308bbae that precisely re-enables/unmask manila-share service, and the juju status posted in the bug description points out you are using cs:manila-ganesha-20 (21.10).

I will give it another try using focal-wallaby, because with focal-xena things work ok.

do you have /var/log/juju/ directory from one of the units where you found this issue?

focal-wallaby gave me the same results - https://pastebin.ubuntu.com/p/myrbqtFX9m/ - what I did find surprising though was that after unlocking Vault the manila-ganesha units had manila-share.service running where before unlocking it only one of them was running.

I will set this as incomplete until the /var/log/juju from manila-ganesha units (or a juju-crashdump) is provided.

Hi Felipe,

Thank you for looking into this.
I can reproduce this with the following bundle.

https://pastebin.ubuntu.com/p/X79hSMdfYw/

As you can see from the bundle, I have configured a VIP for manila and manila-ganesha and also added relation with Vault.
Before unsealing Vault, manila-ganesha works ok, however, after unsealing Vault which update the endpoint to SSL, then this symptom is seen.

I will add the juju-crashdump later once I have collected it.

On Fri, 2022-02-04 at 02:29 +0000, Yoshi Kadokawa wrote:
> Hi Felipe,
>
> Thank you for looking into this.
> I can reproduce this with the following bundle.
>
> https://pastebin.ubuntu.com/p/X79hSMdfYw/
>
> As you can see from the bundle, I have configured a VIP for manila and
> manila-ganesha and also added relation with Vault.
> Before unsealing Vault, manila-ganesha works ok, however, after
> unsealing Vault which update the endpoint to SSL, then this symptom is
> seen.

I can reproduce the issue now, the piece of the puzzle that I was
missing was the (direct) relation between manila-ganesha and vault.

I was going through the source of the charm and I find no indication of
what should do when it's related to vault, what it did was to install
apache2 and render an empty configuration file:

$ sudo wc -l /etc/apache2/sites-enabled/openstack_https_frontend.conf
0 /etc/apache2/sites-enabled/openstack_https_frontend.conf

What's the expected setup when these 2 charms are related?, because I
have the feeling this is just an interface inherited from a layer and
manila-ganesha doesn't use this functionality.

https://opendev.org/openstack/charm-manila-ganesha/src/branch/master/src/layer.yaml#L16-L18

Currently the manila-ganesha charm inherits from the openstack-api base layer - this implies access to keystone, endpoint registration and TLS ca certs for trust.

Upstream documentation does detail configuration for access to the identity service so lets assume this is actually required:

https://docs.openstack.org/manila/latest/install/install-share-ubuntu.html

That said there is no need to enable an apache configuration with no config so it looks like there is a bug in here somewhere.

The relation to vault would supply the CA cert for the deployment to enable trust - the charm looks to be using the default handler for this relation so I suspect that is somewhat confusing things.

As Felipe was able to reproduce this issue I'm marking this bug as Triaged and High.

I have encountered this bug in my current deployment. This scenario occurs on deployment when vault is related to the manila-ganesha application but it also re-occurs when the signed cert from the CSR is uploaded into vault.

The workaround mentioned by Yoshi does indeed resolve the issue:

  juju run --unit manila-ganesha/leader -- sudo systemctl unmask manila-share

Happened to me as well on an offline deployment...

After more testing, it appears that manila-share service is masked on all manila-ganesha units.
It is consequently required to unmask the service on all of them, otherwise if the leader unit goes down, the whole service is down as well.

  juju run -a manila-ganesha systemctl unmask manila-share

And for Corosync/Pacemaker to try to start right away the service. Doing on one unit :
  juju ssh manila-ganesha/leader sudo crm resource cleanup

This is a very common issue in the SQA lab as well. We have also seen the hacluster units going into this state after the OpenStack cluster is deployed already. Let me know if we can do anything in terms of extra data collection to help debug the issue.